Getting Logs Into Loki With Promtail

eric | Nov. 16, 2021, 3:31 p.m.

This is a quick guide to getting started with Promtail for Loki. Promtail is Grafana's native solution for getting logs into Loki and, as you should expect, is nicely integrated with it.

First Steps

Before installing promtail, make sre you have loki installed and working. See "Install and run Grafana Loki locally" for details (opens in a new window).

In OpenSUSE, just install the latest promtail with the package manager:

$ sudo zypper install promtail

In other Linux flavours, get the files you need to get going, the configuration file and the binary, respectively:

wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml

wget https://github.com/grafana/loki/releases/download/v2.8.2/promtail-linux-amd64.zip

(An overview of all the latest releases: https://github.com/grafana/loki/releases/ )

Expand the binary and move both the binary and the configuration file to the right place in the folder structure:

$ unzip promtail-linux-amd64.zip

$ sudo mv promtail-linux-amd64 /usr/local/bin/

$ sudo mkdir /etc/loki

$ sudo mv promtail-local-config.yaml /etc/loki/promtail.yaml

In openSUSE, the binary is in /usr/bin, and the configuration file in /etc/loki.

Create a promtail user and give it access to logs (use YaST and setfacl below in openSUSE):

$ sudo adduser --system promtail

$ sudo setfacl -R -m u:promtail:rX /var/log

OBS: You may have to install the acl package to run setfacl.

$ sudo usermod -a -G systemd-journal promtail

$ sudo usermod -a -G adm promtail

Create a temporary file needed for promtail and give the promtail user ownership:

$ sudo touch /tmp/positions.yaml

$ sudo chown promtail:promtail /tmp/positions.yaml

Edit the promtail config-file (/etc/loki/promtail.yaml):

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://{url-of-loki-server}:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: "/var/log/*log"

Call the job_name whatever you like (here: system) and use an adequate label for the job (here: varlogs).

Open ports in the firewall

You need access to the ports that both loki and promtail are using. These are - given the setup above - 3000, 3100, 9080, 9093, and 9096.

In openSuse:

$ sudo firewall-cmd --zone=public --add-port=3100/tcp

$ sudo firewall-cmd --zone=public --list-all

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: bond0 eth0 eth1 eth2
  sources:
  services: dhcpv6-client mqtt ssh
  ports: 3000/tcp 3100/tcp 9080/tcp 9093/tcp 9096/tcp
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

In Ubuntu:

$ sudo  ufw allow 3100/tcp

$ $ sudo ufw reload

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
80,443/tcp                 ALLOW       Anywhere                  
22                         ALLOW       Anywhere                  
3100/tcp                   ALLOW       Anywhere                  
3000/tcp                   ALLOW       Anywhere                  
9093/tcp                   ALLOW       Anywhere                  
9096/tcp                   ALLOW       Anywhere                  
9080/tcp                   ALLOW       Anywhere                  
80,443/tcp (v6)            ALLOW       Anywhere (v6)             
22 (v6)                    ALLOW       Anywhere (v6)             
3100/tcp (v6)              ALLOW       Anywhere (v6)             
3000/tcp (v6)              ALLOW       Anywhere (v6)             
9093/tcp (v6)              ALLOW       Anywhere (v6)             
9096/tcp (v6)              ALLOW       Anywhere (v6)             
9080/tcp (v6)              ALLOW       Anywhere (v6)     

Test the solution

First, start Promtail:

$ sudo runuser -l promtail -c "/usr/local/bin/./promtail-linux-amd64 -log.level=debug -config.file=/etc/loki/promtail.yaml"

Note: If you try to run promtail with runuser and you get

This account is currently not available.

there is no valid shell for the promtail user. Add a shell with the chsh-command:

$ sudo chsh -s /bin/bash promtail

Make sure there are actual events in the logs. See if anything appears in Loki. If not, after at least 60 seconds, start (or restart) Loki. From the command line:

$ sudo loki -log.level=debug -config.file=path-to-config-file

Next, see if Promtail is working with:

http://server-ip-address:9080/targets

http://server-ip-address:9080/service-discovery

Promtail Service

Set up a promtail service by creating and editing a promtail service file (/etc/systemd/system/promtail.service):

[Unit]

Description=Promtail for Loki

After=network.target

[Service]

Type=simple

User=promtail

ExecStart=/usr/local/bin/./promtail-linux-amd64 -config.file /etc/loki/promtail.yaml

Restart=on-abort

NoNewPrivileges=true

PrivateTmp=yes

RestrictNamespaces=uts ipc pid user cgroup

ProtectKernelTunables=yes

ProtectKernelModules=yes

ProtectControlGroups=yes

ProtectSystem=strict

PrivateUsers=strict

CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_DAC_READ_SEARCH

[Install]

WantedBy=multi-user.target

$ sudo systemctl start promtail && sudo systemctl enable promtail

$ sudo systemctl status promtail.service
● promtail.service - Promtail for Loki
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2023-01-07 16:48:21 CET; 11s ago
   Main PID: 50858 (promtail-linux-)
      Tasks: 11 (limit: 6940)
     Memory: 612.1M
        CPU: 1.106s
     CGroup: /system.slice/promtail.service
             └─50858 /usr/local/bin/./promtail-linux-amd64 -config.file /etc/loki/promtail.yaml
...

 

Troubleshooting

Permission denied for log-files

Dec  7 15:51:40 server01 promtail-linux-amd64[2266]: level=error ts=2021-12-07T14:51:40.709501865Z caller=filetarget.go:287 msg="failed to start tailer" error="open /var/log/fail2ban.log: permission denied" filename=/var/log/fail2ban.log

The logfile's access rights:

$ getfacl /var/log/fail2ban.log
getfacl: Removing leading '/' from absolute path names
# file: var/log/fail2ban.log
# owner: root
# group: adm
user::rw-
group::r--
other::---

This is typically he result of adding a log after running setfacl above. Running setfacl again gives the promtail-user read access to the log:

sudo setfacl -R -m u:promtail:rX /var/log

Account not available

If you try to run promtail with runuser

$ sudo runuser -l promtail -c "promtail -log.level=debug -config.file=/etc/loki/promtail.yaml"

and you get

This account is currently not available.

there is no valid shell for the promtail user. Add a shell with the chsh-command:

$ sudo chsh -s /bin/bash promtail

Connection refused

This is usually a firewall-related problem. Check that you have opened the right ports.

References

Introduction:

https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/

Configuring Promtail as a service:

https://sbcode.net/grafana/install-promtail-service/

General configuration of promtail:

https://grafana.com/docs/loki/latest/clients/promtail/configuration/

Troubleshooting:

https://grafana.com/docs/loki/latest/clients/promtail/troubleshooting/

Grafana Community:

https://community.grafana.com/