AI Training - Monitor your cloud resources

Learn the resources provided by AI Training jobs

Last updated 3rd May, 2021.

Description

AI-Training provides users with access to several types of resources. Resource usage can be fetched through a dedicated UI on Grafana.

Requirements

  • a running job on AI-Training

UI Access

The URL to access the monitoring is built like this : https://monitoring.<REGION>.training.ai.cloud.ovh.net/d/gpu?var-job=<JOB-ID>

It can be fetched in the CLI using this command:

ovhai job get <JOB-ID>

---
Name:       ai-training-transformers-heuristic
ID:         xxx
Created At: 2021-04-14 09:30:02.709449 UTC
Updated At: 2021-04-14 09:30:09.328392 UTC
User:       user-xxx
Spec:       
  Image:        ovhcom/ai-training-transformers:3.1.0
  Command:      
  Env:          
  Default Port: 8080
  Unsecure:     false
  Resources:    
    GPU:       1
    GPU Model: Tesla-V100S
    CPU:       13
  Volumes:      
  Timeout:      0
  Labels:       
  SSH Keys:     
Status:     
  State:          RUNNING
  Queued At:      14-04-21 09:30
  Started At:     14-04-21 09:30
  Stopped At:     
  IP:             10.42.165.21
  Duration:       2s
  Job Url:        https://xxx.job.console.preprod.training.ai.cloud.ovh.net
  SSH Url:        
  Monitoring Url: https://monitoring.console.preprod.training.ai.cloud.ovh.net/d/gpu?var-job=xxx&from=1618392608000
  Infos:          
  Data Sync:      
  Sync Progress:  
  History:        
    State         Date
    -----         ----
    QUEUED        2021-04-14 09:30:02.708117 UTC
    INITIALIZING  2021-04-14 09:30:02.802995 UTC
    PENDING       2021-04-14 09:30:02.824278 UTC
    RUNNING       2021-04-14 09:30:09.327003 UTC

It is displayed in the line Monitoring Url.

You can also retrieve it using the UI by clicking on the Resource usage link in the job details.

image

Panel Details

We will now go through each panel to give a short description. image

GPU Usage

  • This panel will only be present on GPU jobs.

This panel displays the usage of each GPU allocated to your job.

GPU Memory

  • This panel will only be present on GPU jobs.

This panel displays the usage and limit of memory for each GPU allocated to your Job

CPU Usage

This panel displays the overall CPU usage of your job.

Memory Usage

This panel displays the usage and limit of Memory allocated to your job.

Network Usage

This panel displays input and output traffic on your job.

Ephemeral storage usage

This panel shows the usage and limit of ephemeral storage allocated to your job. Jobs can use ephemeral storage for data not within a synchronised container.

If your usage go beyond the limit of the ephemeral storage, your job will be evicted.

Feedback

Please send us your questions, feedback and suggestions to improve the service:


Esta documentação foi-lhe útil?

Não hesite em propor-nos sugestões de melhoria para fazer evoluir este manual.

Imagens, conteúdo, estrutura... Não hesite em dizer-nos porquê para evoluirmos em conjunto!

Os seus pedidos de assistência não serão tratados através deste formulário. Para isso, utilize o formulário "Criar um ticket" .

Obrigado. A sua mensagem foi recebida com sucesso.


Estes manuais também podem ser úteis...

OVHcloud Community

Aceda ao seu espaço comunitário. Coloque as suas questões, procure informações e interaja com outros membros do OVHcloud Community.

Discuss with the OVHcloud community

Em conformidade com a alteração à Diretiva 2006/112/CE, os preços com IVA podem variar de acordo com o país de residência do cliente
(por defeito, os preços com IVA apresentados incluem o IVA português em vigor).