OVH Queue and Kafka 101
Welcome to the getting started guide of the OVH Queue. This guide will help you to understand the core concepts behind the OVH Queue and how to produce your first message to the platform and consume it.
What is the OVH Queue ?
It provides you a shared queuing system based on Apache Kafka.
It powers fast, scalable and global low-latency messaging allowing you to connect anything to everything.
It is a solution to send and receive messages between independent applications, to decouple systems and components or to connect, build, and scale realtime applications.
What is Kafka ?
Kafka is a high-throughput, distributed, publish-subscribe messaging system. It allows you to connect producers and consumers through topics with fault tolerance as well as journalized events. All messages are persisted on disk and replicated within the cluster to prevent data loss.
What is a producer and a consumer ?
Kafka maintains feeds of messages in topics. Producers write data to topics and consumers read from topics. Since Kafka is a distributed system, topics are partitioned and replicated across multiple nodes. Each message in a partition is assigned a unique offset. A consumer can save its offset position to remember the messages read.
Joining the lab
Before starting to use the Queue lab you need to make sure that you have an OVH.com account.
Go to ovh.com and select "Create Account".
Create an application
The first step to use OVH Queue is to create an application.
Go to the Queue lab page and click on the Start Now button. Note that you can also order a new application in the Sunrise Manager section of the OVH Manager.
Follow the order steps. You will receive an email as soon as your application is ready.
Go to the OVH Manager Sunrise and spot the DBaaS Queue section.
In the Sunrise manager, a new 'Not configured app' is available under the Queue DBaaS section.
Select the application. Choose its name, select a region and save it. Pay attention to the name you choose since it will be used as a namespace prefix for all your topics and users (eg.
Once your application is configured, 3 users are automatically created along with their generated password. Each user is assigned a specific role:
readfor read access on all topics,
writefor write access on all topics, and
adminfor read access, write access and topic auto-creation.
You can retrieve the Kafka SASL/SSL URL on the Info tab of the Sunrise Manager.
Produce and consume
Important: to be authenticated to the OVH Queue, you must:
- Set the proper username (eg.
my-app.admin) and password in the
SASLconfiguration of your Kafka client.
- Produce and consume on topics that your user can access, which are correctly prefixed by your namespace (eg.
- Use any consumer group name that is prefixed by your username (eg
Download the golang queue example.
Start a consumer
./kafka-client-$(uname -s)-amd64 consume --brokers $HOST:9093 --username $SASL_USERNAME --password $SASL_PASSWORD --topic $TOPIC --consumer-group $CONSUMER_GROUP
Example with the application created above:
./kafka-client-Linux-amd64 consume --brokers kafka.p1.sbg.queue.ovh.net:9093 --username my-app.admin --password aRT3u7R2TuRzhMmnLl --topic my-app.my-topic --consumer-group my-app.admin.my-consumer-group
Start a producer and write something to stdin to produce one message (the topic is automatically created).
./kafka-client-$(uname -s)-amd64 produce --brokers $HOST:9093 --username $SASL_USERNAME --password $SASL_PASSWORD --topic $TOPIC
Then write to STDIN to send a message. Example with the application created above:
./kafka-client-Linux-amd64 produce --brokers kafka.p1.sbg.queue.ovh.net:9093 --username my-app.admin --password aRT3u7R2TuRzhMmnLl --topic my-app.my-topic
The OVH Queue supports any Kafka standard client.
In the queue examples repository, you can find examples for: