Apache Spark Overview

Introduction to Apache Spark

Last updated 06th March, 2020

What is Apache Spark?

Apache Spark is an open-source distributed, general-purpose cluster-computing framework that is much faster than the previous cluster computing framework, Hadoop MapReduce, thanks to features like in-memory processing and lazy evaluation.

Apache Spark is the leading platform for large-scale SQL, batch processing, stream processing and machine learning, with an easy-to-use API, and for coding in Spark, you have the options of using different programming languages, including Java, Scala, Python, R and SQL. It can be run locally in a single machine or in a cluster of computers to distribute its tasks.

How Apache Spark works

Apache Spark stores data in RDD (Resilient Distributed Datasets), which is an immutable distributed collection of objects, and then divides it into different logical partitions, so it can process each part in parallel, in different nodes of the cluster. Task parallelism and in-memory computing are the key to being ultra-fast in Apache Spark.

Apache Spark creates a core process called driver program in driver node and several executors in worker nodes and then driver program will distribute the processing task among executors to be executed in parallel. The number of the workers can be scaled up and down to optimise the workload based on available resources and computation nodes.

Apache Spark Components

Different components of Apache Spark are:

  • Apache Spark Core, which provides in-memory computing, and forms the basis of other components.
  • Spark SQL, which provides structured and semi-structured data processing.
  • Spark Streaming, which performs streaming analysis using RDD (Resilient Distributed Datasets) transformation.
  • MLlib (Machine Learning Library), which is a distributed machine learning framework above Spark.
  • GraphX, which is a distributed graph processing framework on top of Spark.

Some use cases of Apache Spark?

  • With Apache Spark stream processing you can analyse real-time data by consuming data of other streams like Apache Kafka and write the results in your data storage.
  • Netflix uses Apache Spark as its recommendation engine to provide better recommendation to its members.
  • Uber uses Apache Spark to create an ETL pipeline over the huge data it receives everyday from its mobile users to convert raw data into structured data and run different kinds of queries and analysis.
  • Pinterest uses Apache Spark to analyse user engagement and give a better recommendations while they are navigating in their platform in real-time.


Please send us your questions, feedback and suggestions to improve the service:

Did you find this guide useful?

Please feel free to give any suggestions in order to improve this documentation.

Whether your feedback is about images, content, or structure, please share it, so that we can improve it together.

Your support requests will not be processed via this form. To do this, please use the "Create a ticket" form.

Thank you. Your feedback has been received.

OVHcloud Community

Access your community space. Ask questions, search for information, post content, and interact with other OVHcloud Community members.

Discuss with the OVHcloud community