AI Training - Troubleshooting
Tutorial about how to debug your jobs
Last updated 04th October, 2021.
Objective
This tutorial gives you some hints on how to debug your jobs if things go wrong.
Requirements
- an AI Training Job you would like to start
- the OVHcloud AI CLI installed
Which commands and arguments can I use to debug?
A lot of options and sub-commands are available in the ovhai tool.
To get a list of available sub-commands and arguments, just start run:
ovhai --help
Further details on each sub-command can be accessed by:
ovhai <subcommand> --help
Where to find the UUID of my job?
First you need the UUID of your job, so use:
ovhai job list
If your job is not listed, you may use:
$ ovhai job list -a
to list all jobs.
Why has my job FAILED?
First, check the return-code / error-code of my job
You can find the return-code of your job by running:
ovhai job get <UUID>
Your return-code is listed in the "Infos"-field in the "Status"-section:
Status:
...
Infos: Job failed with code 1
...
The following info is returned if there was an issue with downloading/pulling your image. Check for typos and access issues if you try to access a non-public image.
Infos: Error image pull
Check if there are any error-messages
Your stdout (Output) and stderr (Error) messages can be read with:
ovhai job logs <UUID>
Debug interactively
If the answers above don't help you solving your issue, it may help running your job a bit more interactively.
To skip any "autostart" of your image, you may use a bash with infinite sleep and connect to this by SSH.
ovhai job run --ssh-public-keys ~/.ssh/id_rsa.pub <Image> -- bash -c 'sleep inf'
Verify you can connect to the SSH host by running the following command:
ssh <job-id>@gra.training.ai.cloud.ovh.net
Welcome to OVHcloud AI Training Jobs SSH
$
You may now start your commands and/or use the typical commandline utils to debug your issue within the container.
Debug your Code
The easiest way to debug your code may be using above interactive debug-session and run/compile your code interactively checking for:
- error-messages
- syntax errors
- missing libs
- wrong versions
- ...
f.e. by running (parts) of your python-code with:
python -i
or using any other debugger
Feedback
Please send us your questions, feedback and suggestions to improve the service:
- On the OVHcloud Discord server
Czy ten przewodnik był pomocny?
Zachęcamy do przesyłania sugestii, które pomogą nam ulepszyć naszą dokumentację.
Obrazy, zawartość, struktura - podziel się swoim pomysłem, my dołożymy wszelkich starań, aby wprowadzić ulepszenia.
Zgłoszenie przesłane za pomocą tego formularza nie zostanie obsłużone. Skorzystaj z formularza "Utwórz zgłoszenie" .
Dziękujemy. Twoja opinia jest dla nas bardzo cenna.