eric | Nov. 16, 2021, 3:31 p.m.
This is a quick guide to getting started with Promtail for Loki. Promtail is Grafana's native solution for getting logs into Loki and, as you should expect, is nicely integrated with it.
First Steps
Before installing promtail, make sre you have loki installed and working. See "Install and run Grafana Loki locally" for details (opens in a new window).
In OpenSUSE, just install the latest promtail with the package manager:
$ sudo zypper install promtail
In other Linux flavours, get the files you need to get going, the configuration file and the binary, respectively:
wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml
wget https://github.com/grafana/loki/releases/download/v2.8.2/promtail-linux-amd64.zip
(An overview of all the latest releases: https://github.com/grafana/loki/releases/ )
Expand the binary and move both the binary and the configuration file to the right place in the folder structure:
$ unzip promtail-linux-amd64.zip
$ sudo mv promtail-linux-amd64 /usr/local/bin/
$ sudo mkdir /etc/loki
$ sudo mv promtail-local-config.yaml /etc/loki/promtail.yaml
In openSUSE, the binary is in /usr/bin, and the configuration file in /etc/loki.
Create a promtail user and give it access to logs (use YaST and setfacl below in openSUSE):
$ sudo adduser --system promtail
$ sudo setfacl -R -m u:promtail:rX /var/log
OBS: You may have to install the acl package to run setfacl.
$ sudo usermod -a -G systemd-journal promtail
$ sudo usermod -a -G adm promtail
Create a temporary file needed for promtail and give the promtail user ownership:
$ sudo touch /tmp/positions.yaml
$ sudo chown promtail:promtail /tmp/positions.yaml
Edit the promtail config-file (/etc/loki/promtail.yaml):
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://{url-of-loki-server}:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: "/var/log/*log"
Call the job_name whatever you like (here: system) and use an adequate label for the job (here: varlogs).
Open ports in the firewall
You need access to the ports that both loki and promtail are using. These are - given the setup above - 3000, 3100, 9080, 9093, and 9096.
In openSuse:
$ sudo firewall-cmd --zone=public --add-port=3000/tcp --permanent
$ sudo firewall-cmd --zone=public --add-port=3100/tcp --permanent
$ sudo firewall-cmd --zone=public --add-port=9080/tcp --permanent
$ sudo firewall-cmd --zone=public --add-port=9093/tcp --permanent
$ sudo firewall-cmd --zone=public --add-port=9096/tcp --permanent
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --zone=public --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: bond0 eth0 eth1 eth2
sources:
services: dhcpv6-client mqtt ssh
ports: 3000/tcp 3100/tcp 9080/tcp 9093/tcp 9096/tcp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
In Ubuntu:
$ sudo ufw allow 3000/tcp
$ sudo ufw allow 3100/tcp
$ sudo ufw allow 9080/tcp
$ sudo ufw allow 9093/tcp
$ sudo ufw allow 9096/tcp
$ $ sudo ufw reload
$ sudo ufw status
Status: active
To Action From
-- ------ ----
80,443/tcp ALLOW Anywhere
22 ALLOW Anywhere
3100/tcp ALLOW Anywhere
3000/tcp ALLOW Anywhere
9093/tcp ALLOW Anywhere
9096/tcp ALLOW Anywhere
9080/tcp ALLOW Anywhere
80,443/tcp (v6) ALLOW Anywhere (v6)
22 (v6) ALLOW Anywhere (v6)
3100/tcp (v6) ALLOW Anywhere (v6)
3000/tcp (v6) ALLOW Anywhere (v6)
9093/tcp (v6) ALLOW Anywhere (v6)
9096/tcp (v6) ALLOW Anywhere (v6)
9080/tcp (v6) ALLOW Anywhere (v6)
Test the solution
First, start Promtail:
$ sudo runuser -l promtail -c "/usr/local/bin/./promtail-linux-amd64 -log.level=debug -config.file=/etc/loki/promtail.yaml"
Note: If you try to run promtail with runuser and you get
This account is currently not available.
there is no valid shell for the promtail user. Add a shell with the chsh-command:
$ sudo chsh -s /bin/bash promtail
Make sure there are actual events in the logs. See if anything appears in Loki. If not, after at least 60 seconds, start (or restart) Loki. From the command line:
$ sudo loki -log.level=debug -config.file=path-to-config-file
Next, see if Promtail is working with:
http://server-ip-address:9080/targets
http://server-ip-address:9080/service-discovery
Promtail Service
Set up a promtail service by creating and editing a promtail service file (/etc/systemd/system/promtail.service):
[Unit]
Description=Promtail for Loki
After=network.target
[Service]
Type=simple
User=promtail
ExecStart=/usr/local/bin/./promtail-linux-amd64 -config.file /etc/loki/promtail.yaml
Restart=on-abort
NoNewPrivileges=true
PrivateTmp=yes
RestrictNamespaces=uts ipc pid user cgroup
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectSystem=strict
PrivateUsers=strict
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_DAC_READ_SEARCH
[Install]
WantedBy=multi-user.target
$ sudo systemctl start promtail && sudo systemctl enable promtail
$ sudo systemctl status promtail.service
● promtail.service - Promtail for Loki
Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2023-01-07 16:48:21 CET; 11s ago
Main PID: 50858 (promtail-linux-)
Tasks: 11 (limit: 6940)
Memory: 612.1M
CPU: 1.106s
CGroup: /system.slice/promtail.service
└─50858 /usr/local/bin/./promtail-linux-amd64 -config.file /etc/loki/promtail.yaml
...
Troubleshooting
Permission denied for log-files
Dec 7 15:51:40 server01 promtail-linux-amd64[2266]: level=error ts=2021-12-07T14:51:40.709501865Z caller=filetarget.go:287 msg="failed to start tailer" error="open /var/log/fail2ban.log: permission denied" filename=/var/log/fail2ban.log
The logfile's access rights:
$ getfacl /var/log/fail2ban.log
getfacl: Removing leading '/' from absolute path names
# file: var/log/fail2ban.log
# owner: root
# group: adm
user::rw-
group::r--
other::---
This is typically he result of adding a log after running setfacl above. Running setfacl again gives the promtail-user read access to the log:
sudo setfacl -R -m u:promtail:rX /var/log
Account not available
If you try to run promtail with runuser
$ sudo runuser -l promtail -c "promtail -log.level=debug -config.file=/etc/loki/promtail.yaml"
and you get
This account is currently not available.
there is no valid shell for the promtail user. Add a shell with the chsh-command:
$ sudo chsh -s /bin/bash promtail
Connection refused
This is usually a firewall-related problem. Check that you have opened the right ports.
Failed to locate executable
If you get something like this:
[root@localhost ~]# systemctl status promtail.service
× promtail.service - Promtail for Loki
Loaded: loaded (/etc/systemd/system/promtail.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Tue 2024-01-09 05:21:23 EST; 5s ago
Duration: 22ms
Process: 3633351 ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file /etc/loki/promtail.yaml (code=exited, status=203/EXEC)
Main PID: 3633351 (code=exited, status=203/EXEC)
CPU: 21ms
Jan 09 05:21:23 localhost.localdomain systemd[1]: Started Promtail for Loki.
Jan 09 05:21:23 localhost.localdomain systemd[3633351]: promtail.service: Failed to locate executable /usr/local/bin/promtail-linux-amd64: Permission denied
Jan 09 05:21:23 localhost.localdomain systemd[3633351]: promtail.service: Failed at step EXEC spawning /usr/local/bin/promtail-linux-amd64: Permission denied
Jan 09 05:21:23 localhost.localdomain systemd[1]: promtail.service: Main process exited, code=exited, status=203/EXEC
Jan 09 05:21:23 localhost.localdomain systemd[1]: promtail.service: Failed with result 'exit-code'.
it either due to the executable not being in the right place or to file permissions. For the latter, do:
[root@localhost ~]# ls -al /usr/local/bin/
total 165048
drwxr-xr-x+ 3 root root 128 Jan 2 12:35 .
drwxr-xr-x. 12 root root 131 May 30 2023 ..
-rwxr-xr-x+ 1 loki loki 59424768 May 3 2023 loki-linux-amd64
-rw-r--r--+ 1 root root 18930096 May 31 2023 loki-linux-amd64.zip
-rwxr-xr-x. 1 root root 233 Nov 6 11:53 normalizer
-rwxrwxr--+ 1 promtail promtail 90640576 May 3 2023 promtail-linux-amd64
drwxr-xr-x. 7 root root 4096 Jan 4 06:47 server_heartbeat
We can see that promtail is the user for the executable, and it appears to have the correct permissions. Let's take a look at the ACL:
[root@localhost ~]# getfacl /usr/local/bin/promtail-linux-amd64
getfacl: Removing leading '/' from absolute path names
# file: usr/local/bin/promtail-linux-amd64
# owner: promtail
# group: promtail
user::rwx
group::r-x
other::r--
That looks good as well. However, we can see from the directory listing that SELinux is controlling access (some of the files has "." at the end). Is SELinux active and what SELinux context does the file have?
[root@localhost ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33
[root@localhost ~]# ls -Z /usr/local/bin/loki-linux-amd64
unconfined_u:object_r:admin_home_t:s0 /usr/local/bin/loki-linux-amd64
Here is the problem. promtail cannot run files in the admin_home_t. Other running services have bin_t. Let's change that:
semanage fcontext -a -t bin_t /usr/local/bin/promtail-linux-amd64
restorecon -v /usr/local/bin/promtail-linux-amd64
Restart the service and look at the status.
References
Introduction:
https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/
Configuring Promtail as a service:
https://sbcode.net/grafana/install-promtail-service/
General configuration of promtail:
https://grafana.com/docs/loki/latest/clients/promtail/configuration/
Troubleshooting:
https://grafana.com/docs/loki/latest/clients/promtail/troubleshooting/
Grafana Community:
Experienced dev and PM. Data science, DataOps, Python and R. DevOps, Linux, clean code and agile. 10+ years working remotely. Polyglot. Startup experience.
LinkedIn Profile
Statistics & R - a blog about - you guessed it - statistics and the R programming language.
R-blog
Erlang Explained - a blog on the marvelllous programming language Erlang.
Erlang Explained