Shipping logs to Logs Data Platform with Filebeat

Last updated 24th July, 2020


Filebeat is an open source file harvester, used to fetch logs files and can be easily setup to feed them into Logs Data Platform.

The main benefits of Filebeat are it's resilient protocol to send logs, and a variety of modules ready-to-use for the most common applications.

This guide will describe how to setup Filebeat OSS on your system for forwarding your logs on Logs Data Platform. It will also present you with some configuration setup useful to further structure your logs.


Note that in order to complete this tutorial, you should have at least:


Setup Filebeat OSS 7.X in your system

Filebeat supports many platforms as listed here

You can decide to setup Filebeat OSS from a package or to compile it from source (you will need the latest go compiler to compile it) or just download the binary to start immediately.

For this part, head to Filebeat OSS download website to download the best version for your distribution.

The following configuration files have been tested on the latest version of Filebeat OSS available at the time of writing (7.9).

The package will install the config file in the following directory: /etc/filebeat/filebeat.yml.

Configure Filebeat OSS 7.X on your system

In the following example we will enable Apache and Syslog support, but you can easily prospect anything else.

Filebeat expect a configuration file named filebeat.yml .

  1. For the configuration to work, the important part is to replace hosts: ["<your_cluster>"] with the hostname given by Logs Data Platform.
  2. You should also ensure to specify the X-OVH-TOKEN of the related stream.

Filebeat configuration

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
# You can find the full configuration reference here:

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#=========================== Filebeat inputs =============================


# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #  level: debug
  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

#============================= Filebeat modules ===============================

  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
fields_under_root: true
  X-OVH-TOKEN: 'xxxxxxxxxxxxxxxxxxxxx'

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#----------------------------- Logstash output --------------------------------
  # Boolean flag to enable or disable the output module.
  enabled: true

  # The Logstash hosts
  hosts: ["<your_cluster>"]

  # Set gzip compression level.
  compression_level: 3

  # Enable SSL support. SSL is automatically enabled if any SSL setting is set.
  ssl.enabled: true

  # Optional SSL configuration options. SSL is off by default.
  # List of root certificates for HTTPS server verifications
  # ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

  - add_host_metadata: ~
  - add_cloud_metadata: ~

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

You can also use our Elasticsearch endpoint to send your logs. This endpoint support ingest and then ensures a higher performance and a higher compatibility with the modules selected. For legal reasons, we do not support X-Pack modules on this endpoint but any OSS module is supported. To enable this endpoint, replace the Logstash Output configuration with the following snippet:

#==================== Elasticsearch template setting ==========================

setup.template.enabled: false
setup.ilm.enabled: false

#-------------------------- Elasticsearch output ------------------------------
  # Array of hosts to connect to.
  hosts: ["<your-cluster>"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  username: "<username>"
  password: "<password>"
  index: "ldp-logs"

This configuration deactivates the template configuration (unneeded for our endpoint). You need to provide your credentials and of your account. Like all Logs Data Platform APIs you can also use tokens. Don't change ldp-logs since it is our special destination index.

When you use Elasticsearch endpoint with filebeat, it will use the ingest module to parse and structure the logs.

Enable Apache Filebeat module

To enable the apache2 support on Filebeat, call the following command:

$ ldp@ubuntu:~$ sudo filebeat modules enable apache

It will generate a new module file: /etc/filebeat/modules.d/apache.yml, please change it to include all your apache2 access/error path files:

- module: apache
  # Access logs
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/apache2/access.log*","/var/log/apache2/ssl_access.log*"]

  # Error logs
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/LOG/apache2/error_log*","/var/log/apache2/ssl_error_log*"]

Enable System Filebeat module

Syslog and authentication supports are part of the system Filebeat module, to enable it:

$ ldp@ubuntu:~$ sudo filebeat modules enable system

Once again, it will generate a file /etc/filebeat/modules.d/system.yml

- module: system
  # Syslog
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/syslog*"]

  # Authorization logs
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/auth.log*"]

Ensure both file path exists on your system.

Enable pipelines

If you use the Elasticsearch output, be sure to setup the pipelines by using the following command:

$ filebeat setup --pipelines --modules apache,system

Filebeat will then connect to Elasticsearch and setup the pipelines needed by your modules.

Launch Filebeat

Launch the Filebeat binary or service to test your config file and head to your apache website for an example of how to send some logs. You will see this kind of log in Graylog:

$ ldp@ubuntu:~$ sudo systemctl restart filebeat.service


$ ldp@ubuntu:~$ sudo /etc/init.d/filebeat restart


Note the type value (apache or syslog or apache-error) that indicates the source file of the log message. You can easily display only your apache access logs for example by typing in the search bar fileset_module:apache2.

Conclusion and useful resources

Filebeat is a handy tool to send the content of your current log files to Logs Data Platform. It offers a clean and easy way to send your logs without changing the configuration of your software. Don't hesitate to check the links below to master this tool.

Going further

Cette documentation vous a-t-elle été utile ?

N’hésitez pas à nous proposer des suggestions d’amélioration afin de faire évoluer cette documentation.

Images, contenu, structure… N’hésitez pas à nous dire pourquoi afin de la faire évoluer ensemble !

Vos demandes d’assistance ne seront pas traitées par ce formulaire. Pour cela, utilisez le formulaire "Créer un ticket" .

Merci beaucoup pour votre aide ! Vos retours seront étudiés au plus vite par nos équipes..

Ces guides pourraient également vous intéresser...

OVHcloud Community

Accedez à votre espace communautaire. Posez des questions, recherchez des informations, publiez du contenu et interagissez avec d’autres membres d'OVHcloud Community.

Echanger sur OVHcloud Community