OVH Guides

Monitoring your jobs using Grafana and the Apache Spark dashboard

Find out how to monitor your jobs using Grafana and the Apache Spark dashboard

Last updated 06th March, 2020

Objective

This guide will tell you how you can monitor you jobs after submitting them.

In this guide, we are assuming that you're using the OVHcloud Manager to use the Data Processing platform and you have already submitted a job.

To read an introduction about Data Processing service you can visit Data Processing Overview.

Requirements

  • Access to the OVHcloud Manager
  • A job that has been already submitted to Data Processing

Instructions

Apache Spark Dashboard

After submitting your job, follow these steps to open its Apache Spark dashboard:

  • Find your job in the list of jobs in the Data Processing page of the OVHcloud Manager.
  • Click on the name of the job to open its dashboard.

Data Processing Job dashboard

  • Click on Open job user interface in the Actions panel to open the Apache Spark dashboard. If you do this right after submitting a job, you will see a warm up page for a few seconds while your job is starting.

Warming up

The Apache Spark dashboard for your job will then start automatically.

Apache Spark dashboard

Grafana Dashboard

After you submit a job to the Data Processing platform, follow these steps to monitor your job using a Grafana dashboard.

  • Find your job in the list of jobs in the Data Processing page in the OVHcloud Manager.
  • Click on the name of the job to open its dashboard.
  • In the dashboard, click on Monitor job in the Actions panel.

Metrics token

grafana

  • In the import dashboard page, paste the Grafana ID that you copied before and click on the Load button.

grafana import

  • Write a name for your dashboard
  • Paste the metrics token that you copied before.
  • For datasource, select My Data Source 1 from the menu.
  • Click on the Import button to open your dashboard.

grafana import

  • You can now select a job ID from the list of all the jobs you have submitted so far to see its Grafana dashboard.

grafana charts

Go further

To learn more about using Data Processing and how to create cluster and process your data, we invite you to look at Data Processing documentations page.

You can send your questions, suggestions or feedbacks in our community of users on https://community.ovh.com/en/ or in our public Gitter


These guides might also interest you...