Python 3.x - Push logs with logging-ldp

Last updated 16th January, 2023

Objective

This guide will show you how to push your logs to Logs Data Platform using Python 3.x.

logging-ldp is intended to be a high performance logging formatter and handler to send log entries into Logs Data Platform.

This package includes:

  • a TCP/TLS handler to send log entries over TCP with TLS support.
  • a formatter to convert logging record into GELF(1.1).
  • a facility to ensure fields suits the LDP naming conventions.

Requirements

To complete this guide you will need:

Instructions

Install

Using pip

You can use pip to install logging-ldp, make sure you have the latest version:

$ pip3 install --upgrade pip
[...]
Successfully installed pip-<version>
$ pip3 install --upgrade logging-ldp
[...]
Successfully installed logging-ldp-<version> setuptools-18.3.1

Using sources

logging-ldp is available on the OVH github repository and can be installed manually:

$ git clone git@github.com:ovh/python-logging-ldp.git
Cloning into 'python-logging-ldp'...
remote: Counting objects: 58, done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 58 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (58/58), 9.62 KiB | 0 bytes/s, done.
Resolving deltas: 100% (26/26), done.
Checking connectivity... done.

$ cd python-logging-ldp
$ python3 setup.py install
[...]
Using /usr/lib/python3.x/site-packages
Finished processing dependencies for logging-ldp==<version>

How to send logs

The following example shows how to send log in Graylog TCP input:

import logging
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler

def setup_logging():
    handler = LDPGELFTCPSocketHandler(hostname="gra1.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    logging.info("Test !")

Send additional static meta data

To automatically append meta data on all your logs, you can implement an alternate Schema:

import logging
from marshmallow import fields
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler
from logging_ldp.schemas import LDPSchema

def setup_logging():
    # Load you config there
    config = dict(name="myapp", version="0.0.1")

    # Define a custom Schema
    class MyApp(LDPSchema):
        app_name = fields.Constant(config['name'])
        app_version = fields.Constant(config['version'])

    handler = LDPGELFTCPSocketHandler(hostname="gra1.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", schema=MyApp))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)


if __name__ == '__main__':
    setup_logging()

    logging.info("Test !")

The log entry sent to Graylog will be something like:

{
  "_app_name": "myapp",
  "_app_version": "0.0.1",
  "_X-OVH-TOKEN": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "file": "test_thread.py",
  "host": "cdumay-desk",
  "level": 6,
  "line": 36,
  "short_message": "Test !",
  "timestamp": 1556036745.6493497,
  "version": "1.1"
}

Note: The result is "pretty printed" only for the documentation.

Send additional intermittent meta data

To define occasional meta data, you can define a Schema with Nested sub-items:

import logging
from marshmallow import Schema, fields
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler
from logging_ldp.schemas import LDPSchema

# Define a sub-schema
class User(Schema):
    name = fields.String(required=True)
    age = fields.Integer()

# Define a custom schema to apply on log entries
class AccountInfo(LDPSchema):
    user = fields.Nested(User, required=True)
    manager = fields.Nested(User)

def setup_logging():
    handler = LDPGELFTCPSocketHandler(hostname="gra1.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(
        token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        schema=AccountInfo
    ))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    current_user = dict(name="John Doe")
    boss = dict(name="Roger Smith", age=51)
    logging.info("User has logged", extra=dict(user=current_user, manager=boss))

The log entry sent will be:

{
  "_manager_age_int": 51,
  "_manager_name": "Roger Smith",
  "_user_name": "John Doe",
  "_X-OVH-TOKEN": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "file": "test_thread.py",
  "host": "cdumay-desk",
  "level": 6,
  "line": 40,
  "short_message": "User has logged",
  "timestamp": 1556037587.4444475,
  "version": "1.1"
}

As we can see:

  • Objects are transformed to flatten dictionaries: manager.name is renamed manager_name.
  • Fields are types using the LDP naming convention: manager.age is renamed manager_age_int
  • Null values are not sent: user.age.
  • You can set required=True to force meta data value or default=xxx to add data automatically.

Go further

Did you find this guide useful?

Please feel free to give any suggestions in order to improve this documentation.

Whether your feedback is about images, content, or structure, please share it, so that we can improve it together.

Your support requests will not be processed via this form. To do this, please use the "Create a ticket" form.

Thank you. Your feedback has been received.

These guides might also interest you...

Logs Data Platform
Python 2.x - Push logs with Djehouty
Logs Data Platform
Rust - Push logs with gelf_logger or log4rs-gelf
Most viewed tutorials
IP Management
Customer data protection
Instance management
Managing dedicated servers
VPS management
Managing Hosted Private Cloud

OVHcloud Community

Access your community space. Ask questions, search for information, post content, and interact with other OVHcloud Community members.

Discuss with the OVHcloud community