Data - concept and best practices
Learn the concept behind AI Tools data
Learn the concept behind AI Tools data
Last updated 11th April 2022
OVHcloud Object Storage should be used to persist any data needed by AI Training jobs or AI notebooks.
There are two ways to manage your data:
ovhaiCLI. Next, find out how to manage your data with the CLI.
AI Training jobs and AI Notebooks can read and write data from and to the OVHcloud Object Storage. Here is what is happening under the hood.
Users can give 3 different access rights on Filesystem volumes attached on jobs and notebooks:
RO): you are only able to read data in your job or notebook. It is forbidden to write, modify or delete data in this container.
When to attach a volume in read-only
The purpose of read-only permissions is to ensure that you do not modify your data by mistake. This permission is recommended for your input data: images, videos, sounds, text or csv files, etc.
Volumes which are in read-only mode are not synchronized with the Object Storage during the
STOPPING phase because there is no point in synchronizing data which has not changed during the life of a job or notebook.
If you have directly added data from the Object Storage to the container connected in
RO, you can request an intermediate synchronization. Otherwise, your data will be synchronized at the next start of your job or notebook.
RW): you have full access to your data in your job or notebook. You can therefore write, modify or add data in this container.
When to attach a volume in read-write
The purpose of the read-write permission is to be able to modify the contents of the object container. This permission is recommended if your container contains code (e.g. a Python file), a notebook or if you want to save a model or connection weights after your training.
Volumes which are in read-write do not allow you to delete data from your Object Storage. If you delete them from a job or notebook, they will still be present in your object container, even after synchronisation.
If you need to delete data from your object container, you can mount your volume in
RWD): you have full access to your data in your job or notebook. You can therefore write, modify, add or delete data in this container.
When to attach a volume in read-write-delete
The purpose of the read-write-delete permission is to be able to modify or delete the object container contents. You can not only make changes to the different files in your container (Python code, requirements file, ...) but also, and especially, delete data from it. The RWD will allow you to permanently delete data you no longer need from your object container. This permission can be used in particular if you want to delete old codes or connection weights which can be heavy files.
After the synchronisation phase, your data deleted from your notebook or job will be permanently deleted in the Object Storage.
For volume in
RWD: as long as your job or notebook is in the
STOPPING state, this means that the upload is still in progress. Once the state changes to
STOPPED, it means all the data was uploaded to your Object Storage.
By default filesystem volumes are created and deleted on the fly for each job needing it.
Users can enable a
cache feature that allow jobs and notebooks to re-use available volumes instead of creating a new one each time. That feature serves several purposes:
Unused volume data is regularly deleted. Users should not rely on this
cache feature for long time persistence.
You can check the OVHcloud documentation on how to create a data container.
You can check how to use your data in an AI Notebook.
You can check how to launch an AI Training job with attached volumes.
Please send us your questions, feedback and suggestions to improve the service:
Please feel free to give any suggestions in order to improve this documentation.
Whether your feedback is about images, content, or structure, please share it, so that we can improve it together.
Your support requests will not be processed via this form. To do this, please use the "Create a ticket" form.
Thank you. Your feedback has been received.
Access your community space. Ask questions, search for information, post content, and interact with other OVHcloud Community members.Discuss with the OVHcloud community