Kafka - Getting started

Find out how to set up and manage your Public Cloud Databases for Kafka

Last updated 5th January 2022

Objectives

Apache Kafka is an open-source and highly resilient event streaming platform based on 3 main capabilities:

  • write or read data to/from stream events;
  • store streams of events;
  • process streams of events.

You can get more information on Kafka from the official Kafka website{.external).

This guide explains how to successfully configure Public Cloud Databases for Kafka via the OVHcloud Control Panel.

Requirements

Instructions

Subscribe to the service

Log in to your OVHcloud Control Panel and switch to Public Cloud in the top navigation bar. After selecting your Public Cloud project, click on Databases in the left-hand navigation bar under Storage.

Click the Create a database instance button. (Create a service if your project already contains databases.)

Step 1: Select your database type

Click on the type of database you want to use and then select the version to install from the respective drop-down menu. Click Next to continue.

Choose database

Step 2: Select a solution

In this step, choose an appropriate service plan. If needed, you will be able to upgrade the plan after creation.

Choose plan

Please visit the capabilities page of your selected database type for detailed information on each plan's properties.

Click Next to continue.

Step 3: Select a location

Choose the geographical region of the datacenter where your service will be hosted.

Choose region

Click Next to continue.

Step 4: Size the database nodes

You can increase the number of nodes and choose the node template in this step. The minimum and maximum amount of nodes depends on the solution chosen in step 2.

Order nodes

Please visit the capabilities page of your selected database type for detailed information on hardware resources and other properties of the database installation.

Take note of the pricing information and click Next to continue.

Step 5: Configure your options

You can name your database in this step and decide to attach a public or private network. Please note that attaching a private network is a feature not yet available at this time.

Configure options

Step 6: Review and confirm

The final section will display a summary of your order as well as the API equivalent of creating this database instance with the OVHcloud API.

Confirm order

In a matter of minutes, your new Apache Kafka service will be deployed. Messages in the OVHcloud Control Panel will inform you when the streaming tool is ready to use.

Configure the Apache Kafka service

Once the Public Cloud Databases for Kafka service is up and running, you will have to define at least one user and one authorised IP in order to fully connect to the service (as producer or consumer).

Users

The General information tab should inform you to create users and authorized IPs.

Kafka General information

Step 1 (mandatory): Set up a user

Switch to the Users tab. An admin user is preconfigured during the service installation. You can add more users by clicking the Add user button.

Users

Enter a username, then click Create User.

Once the user is created, the password is generated. Please keep it securely as it will not be shown again.

Passwords can be reset for the admin user or changed afterwards for other users in the Users tab.

Password reset

Step 2 (mandatory): Configure authorised IPs

For security reasons the default network configuration doesn't allow any incoming connections. It is thus critical to authorize the suitable IP addresses in order to successfully access your Kafka cluster.

Switch to the Authorized IPs tab. At least one IP address must be authorised here before you can connect to your database. It can be your laptop IP for example.

Authorised IP

Clicking on Add an IP address or IP address block (CIDR) opens a new window in which you can add single IP addresses or blocks to allow access to the database.

Add IP

You can edit and remove database access via the ... button in the IP table.

If you don't know how to get your IP, please visit a website like www.WhatismyIP.com. Copy the IP address shown on this website and keep it for later.

Your Apache Kafka service is now fully accessible!

Optionally, you can configure access control lists (ACL) for granular permissions and create something called topics, as shown below.

Optional: Create Kafka topics

Topics can be seen as categories, allowing you to organize your Kafka records. Producers write to topics, and consumers read from topics.

To create Kafka topics, click on the Add a topic button:

Add a topic

In advanced configuration you can change the default value for the following parameters:

  • Replication (3 brokers by default)
  • Partitions (1 partition by default)
  • Retention size in bytes (-1: no limitation by default)
  • Retention time in hours (-1: no limitation by default)
  • Minimum in-sync replica (2 by default)
  • Deletion policy

Create a topic

Optional: Configure ACLs on topics

Public Cloud Databases for Kafka supports access control lists (ACLs) to manage permissions on topics. This approach allows you to limit the operations that are available to specific connections and to restrict access to certain data sets, which improves the security of your data.

By default the admin user has access to all topics with admin privileges. You can define some additional ACLs for all users / topics, click on Add a new entry button:

Enable ACLs

For a particular user, and one topic (or all with '*'), define the ACL with the the following permissions:

  • admin: full access to APIs and topic
  • read: allow only searching and retrieving data from a topic
  • write: allow updating, adding, and deleting data from a topic
  • readwrite: full access to the topic

Define ACLs

Note: Write permission allows the service user to create new indexes that match the pattern, but it does not allow deletion of those indexes.

When multiple rules match, they are applied in the order listed above. If no rules match, access is denied.

First CLI connection to your Kafka service

Verify that the IP address visible from your browser application is part of the "Authorised IPs" defined for this Kafka service.

Check also that the user has granted ACLs for the target topics.

Download server and user certificates

In order to connect to the Apache Kafka service, it is required to use server and user certificates.

1 - Server certificate

The server CA (Certificate Authority) certificate can be downloaded from the General information tab:

Kafka server certificate

2 - User certificate

The user certificate can be downloaded from the Users tab:

User informations User certificate

3 - User access key

Also download the user access key.

User access key

Install an Apache Kafka CLI

As part of the Apache Kafka official installation, you will get different scripts that will also allow you to connect to Kafka in a Java 8+ environment: Apache Kafka Official Quickstart.

We propose to use a generic producer and consumer client instead: Kcat (formerly known as kafkacat). Kcat is more lightweight since it does not require a JVM.

Install Kcat

For this client installation, please follow the instructions available at: Kafkacat Official Github.

Kcat configuration file

Let's create a configuration file to simplify the CLI commands to act as Kafka Producer and Consumer:

kafkacat.conf :

bootstrap.servers=kafka-f411d2ae-f411d2ae.database.cloud.ovh.net:20186
enable.ssl.certificate.verification=false
ssl.ca.location=/home/user/kafkacat/ca.pem
security.protocol=ssl
ssl.key.location=/home/user/kafkacat/service.key
ssl.certificate.location=/home/user/kafkacat/service.cert

In our example, the cluster address and port are kafka-f411d2ae-f411d2ae.database.cloud.ovh.net:20186 and the previously downloaded CA certificates are in the /home/user/kafkacat/ folder.

Change theses values according to your own configuration.

Kafka producer

For this first example let's push the "test-message-key" and its "test-message-content" to the "my-topic" topic.

echo test-message-content | kcat -F kafkacat.conf -P -t my-topic -k test-message-key

Note: depending on the installed binary, the CLI command can be either kcat or kafkacat.

Kafka consumer

The data can be retrieved from "my-topic".

kcat -F kafkacat.conf -C -t my-topic -o -1 -e

Note: depending on the installed binary, the CLI command can be either kcat or kafkacat.

Conclusion

Congratulations, you now have an up and running Apache Kafka cluster, fully managed and secured. You are able to push and retrieve data easily via CLI.

Go further

Kafka capabilities

Kafka Official documentation

Kafka clients

Some UI tools for Kafka are also available:

Visit our dedicated Discord channel: https://discord.gg/PwPqWUpN8G. Ask questions, provide feedback and interact directly with the team that builds our databases services.


Questa documentazione ti è stata utile?

Prima di inviare la valutazione, proponici dei suggerimenti per migliorare la documentazione.

Immagini, contenuti, struttura... Spiegaci perché, così possiamo migliorarla insieme!

Le richieste di assistenza non sono gestite con questo form. Se ti serve supporto, utilizza il form "Crea un ticket" .

Grazie per averci inviato il tuo feedback.


Potrebbero interessarti anche...

OVHcloud Community

Accedi al tuo spazio nella Community Fai domande, cerca informazioni, pubblica contenuti e interagisci con gli altri membri della Community OVHcloud

Discuss with the OVHcloud community

Conformemente alla Direttiva 2006/112/CE e successive modifiche, a partire dal 01/01/2015 i prezzi IVA inclusa possono variare in base al Paese di residenza del cliente
(i prezzi IVA inclusa pubblicati includono di default l'aliquota IVA attualmente in vigore in Italia).