Getting Logs Into Loki With Promtail

eric | Nov. 16, 2021, 3:31 p.m.

This is a quick guide to getting started with Promtail for Loki. Promtail is Grafana's native solution for getting logs into Loki and, as you should expect, is nicely integrated with it.

First Steps

Get the files you need to get going, the configuration file and the binary, respectively:

wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml

wget https://github.com/grafana/loki/releases/download/v2.3.0/promtail-linux-amd64.zip

(An overview of all the latest releases: https://github.com/grafana/loki/releases/ )

Expand the binary and move both the binary and the configuration file to the right place in the folder structure:

$ sudo unzip promtail-linux-amd64.zip

$ sudo mv promtail-linux-amd64 /usr/local/bin/

$ sudo mv promtail-local-config.yaml /etc/loki/promtail.yaml

Create a promtail user and give it access to logs:

$ sudo adduser --system promtail

$ sudo setfacl -R -m u:promtail:rX /var/log

$ sudo usermod -a -G systemd-journal promtail

$ sudo usermod -a -G adm promtail

Create a temporary file needed for promtail and give the promtail user ownership:

$ sudo touch /tmp/positions.yaml

$ sudo chown promtail:promtail /tmp/positions.yaml

Edit the promtail config-file:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://10.0.0.20:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: "/var/log/*log"

Call the job_name whatever you like (here: system) and use an adequate label for the job (here: varlogs).

Test the solution

First, start Promtail:

$ sudo runuser -l promtail -c "promtail -log.level=debug -config.file=path-to-config-file"

Make sure there are actual events in the logs. See if anything appears in Loki. If not, after at least 60 seconds, start (or restart) Loki. From the command line:

$ sudo loki -log.level=debug -config.file=path-to-config-file

Next, see if Promtail is working with:

http://server-ip-address:9080/targets

http://server-ip-address:9080/service-discovery

Promtail Service

Set up a promtail service by creating and editing a promtail service file (/etc/systemd/system/promtail.service):

[Unit]

Description=Promtail for Loki

After=network.target

[Service]

Type=simple

User=promtail

ExecStart=/usr/local/bin/./promtail-linux-amd64 -config.file /etc/promtail/promtail.yaml

Restart=on-abort

[Install]

WantedBy=multi-user.target

$ sudo systemctl start promtail && sudo systemctl enable promtail

$ sudo systemctl status promtail.service

Troubleshooting

Permission denied for log-files:

Dec  7 15:51:40 server01 promtail-linux-amd64[2266]: level=error ts=2021-12-07T14:51:40.709501865Z caller=filetarget.go:287 msg="failed to start tailer" error="open /var/log/fail2ban.log: permission denied" filename=/var/log/fail2ban.log

The logfile's access rights:

$ getfacl /var/log/fail2ban.log
getfacl: Removing leading '/' from absolute path names
# file: var/log/fail2ban.log
# owner: root
# group: adm
user::rw-
group::r--
other::---

This can be solved by adding the promtail-user to the group that has read access to the log, in this case the adm-group:

sudo usermod -a -G adm promtail

References

Introduction:

https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/

Configuring Promtail as a service:

https://sbcode.net/grafana/install-promtail-service/

General configuration of promtail:

https://grafana.com/docs/loki/latest/clients/promtail/configuration/

Troubleshooting:

https://grafana.com/docs/loki/latest/clients/promtail/troubleshooting/

Grafana Community:

https://community.grafana.com/

About Me

Experienced dev and PM. Data science, DataOps, Python and R. DevOps, Linux, clean code and agile. 10+ years working remotely. Polyglot. Startup experience.
LinkedIn Profile

By Me

Statistics & R - a blog about - you guessed it - statistics and the R programming language.
R-blog

Erlang Explained - a blog on the marvelllous programming language Erlang.
Erlang Explained