AI Training - Job concept
Learn the concept behind AI Training jobs
Last updated 18th May, 2021.
Definition
A job in AI Training is the workload unit submitted to the cluster. A job runs as a Docker container within OVHcloud infrastructure.
Each job is linked to a Public Cloud project and specifies an amount of resources to use to run the training task along with a Docker image either publicly available, in the AI Training shared registry scoped to your project or the private registry of your choosing that you added. For the latter, see the OVHcloud documentation on how to add a private registry.
Considerations
- A job will run indefinitely until completion or manual interruption.
- Data can be attached to a job to serve either/both as input for your training workload or output (e.g. model weights).
- If you do not customise you resource request, the default requested is 1 GPU. Memory is not customisable.
- Billing for jobs is minute-based and starts at job initialisation until completion. Each commenced minute is billed completely.
- You can read further on job limitations here.
Under the hood
Jobs in AI Training are Docker containers within OVHcloud infrastructure.
Job lifecycle
During its lifetime the job will transition between the following statuses:
- Only jobs that reach the
RUNNINGstatus are billed. Billing starts with theINITIALIZINGstep and ends when theFINALIZINGstep starts. - Only jobs in states
QUEUED,INITIALIZING,PENDINGandRUNNINGare included in the quota computation.
QUEUEDthe job run request is about to be processedINITIALIZINGthe job instance is created and the data is synchronised from the Object Storage. To know more about the data synchronisation check out the Data How it works section.PENDINGjob is being startedRUNNINGthe job is runningINTERRUPTINGthe job is still running but an interruption order was received and is about to be processedFINALIZINGthe job instance is deleted and the data is synchronised back to the Object Storage. To know more about the data synchronisation check out the Data How it works section.DONEthe job ended normallyTIMEOUTthe job is still running but is about to be interrupted because the timeout was reachedINTERRUPTEDthe job is ended and was interruptedFAILEDthe job ended with an error, e.g. the process in the job finished with a non 0 exit code, Docker image could not be pulled, ...ERRORthe job ended due to a backend error
Going further
- You can check the OVHcloud documentation on how to create a data container.
- You can check the OVHcloud documentation on how to submit a job
Feedback
Please send us your questions, feedback and suggestions to improve the service:
- On the OVHcloud Discord server
Czy ten przewodnik był pomocny?
Zachęcamy do przesyłania sugestii, które pomogą nam ulepszyć naszą dokumentację.
Obrazy, zawartość, struktura - podziel się swoim pomysłem, my dołożymy wszelkich starań, aby wprowadzić ulepszenia.
Zgłoszenie przesłane za pomocą tego formularza nie zostanie obsłużone. Skorzystaj z formularza "Utwórz zgłoszenie" .
Dziękujemy. Twoja opinia jest dla nas bardzo cenna.