AI Training - Troubleshooting

Tutorial about how to debug your jobs

Last updated 04th October, 2021.

Objective

This tutorial gives you some hints on how to debug your jobs if things go wrong.

Requirements

Which commands and arguments can I use to debug?

A lot of options and sub-commands are available in the ovhai tool.

To get a list of available sub-commands and arguments, just start run:

ovhai --help

Further details on each sub-command can be accessed by:

ovhai <subcommand> --help

Where to find the UUID of my job?

First you need the UUID of your job, so use:

ovhai job list

If your job is not listed, you may use:

$ ovhai job list -a

to list all jobs.

Why has my job FAILED?

First, check the return-code / error-code of my job

You can find the return-code of your job by running:

ovhai job get <UUID>

Your return-code is listed in the "Infos"-field in the "Status"-section:

Status:
  ...
  Infos:          Job failed with code 1
  ...

The following info is returned if there was an issue with downloading/pulling your image. Check for typos and access issues if you try to access a non-public image.

  Infos:          Error image pull

Check if there are any error-messages

Your stdout (Output) and stderr (Error) messages can be read with:

ovhai job logs <UUID>

Debug interactively

If the answers above don't help you solving your issue, it may help running your job a bit more interactively.

To skip any "autostart" of your image, you may use a bash with infinite sleep and connect to this by SSH.

ovhai job run --ssh-public-keys ~/.ssh/id_rsa.pub <Image> -- bash -c 'sleep inf'

Verify you can connect to the SSH host by running the following command:

ssh <job-id>@gra.training.ai.cloud.ovh.net

Welcome to OVHcloud AI Training Jobs SSH
$

You may now start your commands and/or use the typical commandline utils to debug your issue within the container.

Debug your Code

The easiest way to debug your code may be using above interactive debug-session and run/compile your code interactively checking for:

  • error-messages
  • syntax errors
  • missing libs
  • wrong versions
  • ...

f.e. by running (parts) of your python-code with:

python -i

or using any other debugger

Feedback

Please send us your questions, feedback and suggestions to improve the service:


Did you find this guide useful?

Please feel free to give any suggestions in order to improve this documentation.

Whether your feedback is about images, content, or structure, please share it, so that we can improve it together.

Your support requests will not be processed via this form. To do this, please use the "Create a ticket" form.

Thank you. Your feedback has been received.


These guides might also interest you...

OVHcloud Community

Access your community space. Ask questions, search for information, post content, and interact with other OVHcloud Community members.

Discuss with the OVHcloud community

In accordance with the 2006/112/CE Directive, modified on 01/01/2015, prices incl. VAT may vary according to the customer's country of residence
(by default, the prices displayed are inclusive of the UK VAT in force).