My First 60 Minutes With A Server

eric | Jan. 3, 2023, 5:25 p.m.

There are a few posts with titles like "My First 5 Minutes With A Server", perhaps even 10 or 15 minutes. Most are helpful for a minimal setup, but I can't remember spending less than 60 to get everything up and going and secured. Maybe I am just slow, but a minimal setup involves: A working firewall, passwordless ssh, automatic banning of malicious IPs, unattended security upgrades, malware scanning, an up-to-date security auditing system, systematic backups, and a monitoring system.

Recipes for:

- AlmaLinux

- Debian

- openSUSE

- Ubuntu

AlmaLinux / Rocky Linux

Preparation

Several packages are not available from the standard repository. Add the EPEL repo (Extra packages for Enterprise Linux) to get access to all packages referenced below (adding this is free):

# dnf install epel-release -y

# dnf config-manager --set-enabled epel

# dnf update -y

Set up cron and crontab [1]:

# dnf install cronie-noanacron

and

# dnf remove cronie-anacron

Firewall With firewall-cmd

The firewall should be enabled as part of the installation process, as you presumably have direct terminal access. You can do anything in this list via ssh, but the firewall must be active to let ssh through, which means that it will initially block ssh. Catch 22!

Get the firewall going:

# systemctl start firewalld.service

# systemctl enable firewalld.service

# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2023-01-02 12:28:18 CET; 2 days ago
       Docs: man:firewalld(1)
   Main PID: 1177 (firewalld)
      Tasks: 2 (limit: 4915)
     CGroup: /system.slice/firewalld.service
             └─1177 /usr/bin/python3 /usr/sbin/firewalld --nofork --nopid

Now, you can add the ssh service (and perhaps also other services, such as http, https, etc):

# firewall-cmd --zone=public --add-service=ssh --permanent

# firewall-cmd --reload

Take a look at what you have so far:

# firewall-cmd --zone=public --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp3s0
  sources:
  services: cockpit dhcpv6-client ssh
  ports:
  protocols:
  forward: yes
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Block ssh Password Login

Allowing only certificate-based login to ssh will make it much harder to gain access for unauthorized users. If you think this is overkill, check the logs of a public-facing server, and see how many login attempts there are on port 22.

First, copy public keys from all clients that are supposed to have access to the authorized_keys file. From the client (with the user you will use to connect to the server):

$ ssh-copy-id -i ~/.ssh/id_rsa.pub username@server
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/eric/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
(username@server) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@server'"
and check to make sure that only the key(s) you wanted were added.

Before you try to log in from the client, restart ssh on the server to make sure that the changed authorized_keys file is loaded:

# systemctl restart sshd.service

Next, log in from the client with:

$ ssh user@server

You should be logged directly in without being prompted for a password.

Next, edit the ssh configuration file,

# vi /etc/ssh/sshd_config

Find the following line, and change the "yes" to "no":

PasswordAuthentication yes

Then you need to restart the ssh service:

# systemctl restart ssh

System Audit

For most systems, lynis is the go-to package for system audits. With epel enabled:

# dnf install lynis

# lynis audit system

A long report is shown, where the most important part is what is shown under Lynis *.*.* Results. Anything appearing under Warnings should be dealt with immediately. There is also a long list of suggestions that you may want to take a look at.

Note: You can get quite a few things done with auditd. Take a look at this excellent article to get going.

Malware Scan With clamav

Install, enable and start clamav packages. Freshclam keeps the clamav-database up to date and should run continuously as a service:

# dnf install clamav clamd clamav-update

# systemctl enable clamav-freshclam

# systemctl start clamav-freshclam

# systemctl status clamav-freshclam

Clamav scans for malware with clamscan, which should run at regular intervals. This can be done with cron. Add clamscan to cron:

# crontab -e

# m     h       dom     mon     dow     command
*       3       *       *       1,3,5   nice -n 15 clamscan && clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

So what is going on here?

*       3       *       *       1,3,5

The command runs every Monday, Wednesday, and Friday at 3 am.

nice -n 15 clamscan

Clamscan is resource intensive. nice makes clamscan run with an adjusted niceness; It affects process scheduling, and a positive number is unfavorable to the process, thus avoiding that the process hogs the server.

clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

This is a mouthful. We only want warnings of infected files (i), recursive search (r), and all findings logged to /var/log/clamav.log. We exclude directories that are unlikely to contain malware by using exclude-dir, and we set the maximum file size for individual files to 50M.

Filtering Malicious IPs With fail2ban

Fail2ban scans log files and bans IPs that show certain signs of malicious intent. Filters for many services are ready out of the box [2]. Simply:

# dnf install fail2ban

None of the jails within fail2ban are enabled by default. To enable, make a custom configuration file and add the services you want to watch:

# vim /etc/fail2ban/jail.d/customization.local

Add the options you wish to customize and the services you would like to include. For instance, to increase ban time to one hour (default is 10 minutes) and add sshd:

[DEFAULT]
bantime = 1h

[sshd]
enabled = true

For more details, see the default configuration file at /etc/fail2ban/jail.conf.

Unattended Security Upgrades

Run the following command to update security-related packages only:

# dnf update --security

To run the command at regular intervals (below every Monday, Wednesday, and Friday at 3 am), add it to cron.

# crontab -e

# m     h       dom     mon     dow     command

*       3       *       *       1,3,5   yum update --security

Backup

The best backup solution for you will depend a lot on your setup. My favorite solution is — hands down — BorgBackup. It includes deduplication, compression, and authenticated encryption. Moreover, it works well for both desktops and servers. For the former, the Vorta GUI is really handy. With a separate machine holding the backups, the installation looks like this on both server and remote backup server (if both are Almalinux, Rocky Linux, or similar — for other distributions, see [3]):

# dnf install borgbackup

First, initialize a repository on the remote backup server:

> borg init --encryption=repokey /path/to/repo

Next, enter a passphrase when prompted, and then copy the repo key (you will need both to access the repo):

> borg key export /path/to/repo text.txt
> cat text.txt
BORG_KEY e036...

Now, you can create a backup using the repo. Let's call it Tuesday:

$ borg create ssh://user@server//path/to/repo::Tuesday ~/Documents/
Enter passphrase for key ssh://user@server//path/to/repo:

> borg list /path/to/repo/
Enter passphrase for key /path/to/repo:
Tuesday                              Tue, 2023-01-10 20:03:47 [3b096c...........]

Monitoring With Grafana, Loki and Promtail

Services such as clamav don't do much good if you are not keeping an eye on logs. That also goes for the server maintenance as such. There are many ways to look at logs. Personally, I really like the grafana log-stack, with grafana as a front end, combined with the loki database, and the promtail client-side agent that ships logs to loki.

A normal setup is to have loki and grafana on a separate machine, and then use promtail to shovel logs into loki from each client. To get started quick, look at this webinar on loki and grafana. To get going with promtail, all the details you need are here.

Debian

Preparation

Most of the following commands require root / sudo access, so on Debian it makes sense to change to root user (as users are by default not sudoers on Debian installations):

$ su root

Firewall

There are many firewalls to choose from [4], but the simplest choice is to go with Ubuntu Uncomplicated Firewall (ufw) [5].

# apt update && sudo apt upgrade -y

# apt install ufw -y

When using the root user, the path to ufw may not be included in PATH. Add it to PATH, or use the whole path when running the command (ufw is usually in /usr/sbin ).

If you are accessing the server via ssh, you'd better allow ssh before enabling the firewall:

# ./usr/sbin/ufw allow ssh

Next, enable ufw:

# systemctl enable ufw --now

See the status:

# systemctl status ufw

List of current rules:

# ufw status numbered

Other typical services to add are http and https. For a full list of available apps:

# ufw app list

Block ssh Password Login

Allowing only certificate-based login to ssh will make it much harder to gain access for unauthorized users. If you think this is overkill, check the logs of a public-facing server, and see how many login attempts there are on port 22.

First, copy public keys from all clients that are supposed to have access to the authorized_keys file. From the client (with the user you will use to connect to the server):

# ssh-copy-id -i ~/.ssh/id_rsa.pub username@server
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/eric/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
(username@server) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@server'"
and check to make sure that only the key(s) you wanted were added.

Before you try to log in from the client, restart ssh on the server to make sure that the changed authorized_keys file is loaded:

# systemctl restart sshd.service

Next, log in from the client with:

# ssh user@server

You should be logged directly in without being prompted for a password.

Next, edit the ssh configuration file,

# vi /etc/ssh/sshd_config

Find the following line, and change the "yes" to "no":

PasswordAuthentication yes

Then you need to restart the ssh service:

# systemctl restart ssh

System Audit With lynis

Install lynis by adding the lynis repository. First, make a directory for the public key for the repository:

# mkdir -p /etc/apt/keyrings/

Next, download the key:

# wget -O - https://packages.cisofy.com/keys/cisofy-software-public.key | sudo tee /etc/apt/keyrings/cisofy-software-public.key > /dev/null

# cat /etc/apt/keyrings/cisofy-software-public.key

Then, add the repository to apt sources:

# echo "deb [signed-by=/etc/apt/keyrings/cisofy-software-public.key] https://packages.cisofy.com/community/lynis/deb/ stable main" | sudo tee /etc/apt/sources.list.d/cisofy-lynis.list

Now, you are ready to install lynis:

# apt update

# apt install lynis

# lynis show version

Time to run lynis and get some results:

# lynis audit system

You will get a long list of results and recommendations. The most important part is the beginning of the result list. If you are lucky, you get this message:
...
  -[ Lynis 3.?.? Results ]-

  Great, no warnings
...

However, if you get warnings instead, immediate action is probably required. Typically, lynis suggests a reboot to apply changes. Other warnings are followed by links to further explanations and recommendations.

Scanning For Malware With clamav

Install clamav:

# apt install clamav

Test that clamscan works with:

# nice -n 15 clamscan && clamscan -ir

clamscan can be really resource intensive, and nice ensures that clamscan doesn't hog the server.

Freshclam keeps the clamav-database up to date and should run continuously as a service:

# systemctl enable clamav-freshclam

# systemctl start clamav-freshclam

# systemctl status clamav-freshclam

Clamav scans for malware with clamscan, which should run at regular intervals. This can be done with cron. Add clamscan to cron:

# crontab -e

# m     h       dom     mon     dow     command
*       3       *       *       1,3,5   nice -n 15 clamscan && clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

So what is going on here?

*       3       *       *       1,3,5

This means that the command runs every Monday, Wednesday, and Friday at 3 am.

nice -n 15 clamscan

Clamscan is resource intensive. nice makes clamscan run with an adjusted niceness; It affects process scheduling, and a positive number is unfavorable to the process, thus avoiding that the process hogs the server.

clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

This is a mouthful. We only want warnings of infected files (i), recursive search (r), and all findings logged to /var/log/clamav.log. We exclude directories that are unlikely to contain malware by using exclude-dir, and we set the maximum file size for individual files to 50M.

Filtering Malicious IPs With fail2ban

Fail2ban scans log files and bans IPs that show certain signs of malicious intent. Filters for many services are ready out of the box []. Simply:

# apt install fail2ban

Unattended Security Upgrades

Minimum setup:

# vim /etc/apt/apt.conf.d/50unattended-upgrades

Change:

Unattended-Upgrade::Mail "";

Unattended-Upgrade::MailReport "only-on-error";

Create an apt configuration file:

# vim /etc/apt/apt.conf.d/02periodic

with the following contents:

// Control parameters for cron jobs by /etc/cron.daily/apt-compat //

// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "1";

// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "1";

// Do "apt-get upgrade --download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "1";

// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "1";

// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "21";

// Send report mail to root
//     0:  no report             (or null string)
//     1:  progress report       (actually any string)
//     2:  + command outputs     (remove -qq, remove 2>/dev/null, add -d)
//     3:  + trace on
APT::Periodic::Verbose "2";

Check status:

# systemctl restart unattended-upgrades.service

# systemctl status unattended-upgrades.service

Monitoring With Grafana, Loki and Promtail

Services such as clamav doesn't do much good if you are not keeping an eye on logs. That also goes for the server maintenance as such. There are many ways to look at logs. Personally, I really like the grafana log-stack, with grafana as a front end, combined with the loki database, and the promtail client-side agent that ships logs to loki.

A normal setup is to have loki and grafana on a separate machine, and then use promtail to shovel logs into loki from each client. To get started quick, look at this webinar on loki and grafana. To get going with promtail, all the details you need are here.

openSUSE

Firewall With firewall-cmd

The firewall should be enabled as part of the installation process, as you presumably have direct terminal access. You can do anything in this list via ssh, but the firewall must be active to let ssh through, which means that it will initially block ssh. Catch 22!

Get the firewall going:

$ sudo systemctl start firewalld.service

$ sudo systemctl enable firewalld.service

$ sudo systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2023-01-02 12:28:18 CET; 2 days ago
       Docs: man:firewalld(1)
   Main PID: 1177 (firewalld)
      Tasks: 2 (limit: 4915)
     CGroup: /system.slice/firewalld.service
             └─1177 /usr/bin/python3 /usr/sbin/firewalld --nofork --nopid

Now, you can add the ssh service (and perhaps also other services, such as http, https, etc):

$ sudo firewall-cmd --zone=public --add-service=ssh --permanent

$ sudo firewall-cmd --reload

Block ssh Password Login

Allowing only certificate-based login to ssh will make it much harder to gain access for unauthorized users. If you think this is overkill, check the logs of a public-facing server, and see how many login attempts there are on port 22.

First, copy public keys from all clients that are supposed to have access to the authorized_keys file. From the client (with the user you will use to connect to the server):

$ ssh-copy-id -i ~/.ssh/id_rsa.pub username@server
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/eric/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
(username@server) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@server'"
and check to make sure that only the key(s) you wanted were added.

Before you try to log in from the client, restart ssh on the server to make sure that the changed authorized_keys file is loaded:

# systemctl restart sshd.service

Next, log in from the client with:

$ ssh user@server

You should be logged directly in without being prompted for a password.

Next, edit the ssh configuration file,

# vi /etc/ssh/sshd_config

Find the following line, and change the "yes" to "no":

PasswordAuthentication yes

Then you need to restart the ssh service:

# systemctl restart ssh

Lynis

Install lynis by adding the lynis repository. First, import the GPG key. This ensures the signed repository can be checked:

# rpm --import https://packages.cisofy.com/keys/cisofy-software-rpms-public.key

# zypper addrepo --gpgcheck --name "CISOfy Lynis repository" --priority 1 --refresh --type rpm-md https://packages.cisofy.com/community/lynis/rpm/ lynis

Check that the repository has been added:

# zypper repos

Now, you are ready to install lynis:

# zypper refresh

# zypper install lynis

# lynis show version

Time to run lynis and get some results:

# lynis audit system

You will get a long list of results and recommendations. The most important part is the beginning of the result list. If you are lucky, you get this message:
...
  -[ Lynis 3.?.? Results ]-

  Great, no warnings
...

However, if you get warnings instead, immediate action is probably required.

Scanning For Malware With clamav

Install clamav:

$ sudo zypper install clamav

Next, run freshclam in order to get virus definitions to work with:

$ sudo freshclam

Test that clamscan works with:

$ sudo nice -n 15 clamscan && clamscan -ir

clamscan can be really resource intensive, and nice ensures that clamscan doesn't hog the server.

Freshclam keeps the clamav-database up to date and should run continuously as a service:

$ sudo systemctl enable clamav-freshclam

$ sudo systemctl start clamav-freshclam

$ sudo systemctl status clamav-freshclam

Clamav scans for malware with clamscan, which should run at regular intervals. This can be done with cron. Add clamscan to cron:

$ sudo crontab -e

# m     h       dom     mon     dow     command
*       3       *       *       1,3,5   nice -n 15 clamscan && clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

So what is going on here?

*       3       *       *       1,3,5

This means that the command runs every Monday, Wednesday, and Friday at 3 am.

nice -n 15 clamscan

Clamscan is resource intensive. nice makes clamscan run with an adjusted niceness; It affects process scheduling, and a positive number is unfavorable to the process, thus avoiding that the process hogs the server.

clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

This is a mouthful. We only want warnings of infected files (i), recursive search (r), and all findings logged to /var/log/clamav.log. We exclude directories that are unlikely to contain malware by using exclude-dir, and we set the maximum file size for individual files to 50M.

Filtering Malicious IPs With fail2ban

Fail2ban scans log files and bans IPs that show certain signs of malicious intent. Filters for many services are ready out of the box [2]. Simply:

$ sudo zypper install fail2ban

Unattended Security Upgrades

Install the yast2 online update tool:

> sudo zypper install yast2-online-update-configuration

Run the tool:

> sudo yast2 online_update_configuration

My choices look like this:

These choices give you a weekly update of non-interactive security patches. For other options, tweak the configuration with the tool or edit the configuration file.

If you want automatic reboots whenever a package installation requires a reboot, you can do this using the Reboot Manager:

> sudo zypper install rebootmgr

> sudo systemctl enable --now rebootmgr.service
Created symlink /etc/systemd/system/multi-user.target.wants/rebootmgr.service → /usr/lib/systemd/system/rebootmgr.service.

Monitoring With Grafana, Loki and Promtail

Services such as clamav don't do much good if you are not keeping an eye on logs. That also goes for the server maintenance as such. There are many ways to look at logs. Personally, I really like the grafana log-stack, with grafana as a front end, combined with the loki database, and the promtail client-side agent that ships logs to loki.

A normal setup is to have loki and grafana on a separate machine, and then use promtail to shovel logs into loki from each client. To get started quick, look at this webinar on loki and grafana. To get going with promtail, all the details you need are here.

Ubuntu

Firewall With ufw

Ubuntu Uncomplicated Firewall (ufw) is easy to use []. If you are accessing the server via ssh, you'd better allow ssh before enabling the firewall:

$ sudo ufw allow ssh

Next, enable ufw:

$ sudo systemctl enable ufw --now

See the status:

$ sudo systemctl status ufw

List of current rules:

$ sudo ufw status numbered

Other typical services to add are http and https. For a full list of available apps:

$ sudo ufw app list

 

Block ssh Password Login

Allowing only certificate-based login to ssh will make it much harder to gain access for unauthorized users. If you think this is overkill, check the logs of a public-facing server, and see how many login attempts there are on port 22.

First, copy public keys from all clients that are supposed to have access to the authorized_keys file. From the client (with the user you will use to connect to the server):

$ ssh-copy-id -i ~/.ssh/id_rsa.pub username@server
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/eric/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
(username@server) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@server'"
and check to make sure that only the key(s) you wanted were added.

Before you try to log in from the client, restart ssh on the server to make sure that the changed authorized_keys file is loaded:

# systemctl restart sshd.service

Next, log in from the client with:

$ ssh user@server

You should be logged directly in without being prompted for a password.

Next, edit the ssh configuration file,

# vi /etc/ssh/sshd_config

Find the following line, and change the "yes" to "no":

PasswordAuthentication yes

Then you need to restart the ssh service:

# systemctl restart ssh

System Audit With lynis

Install lynis by adding the lynis repository. First, make a directory for the public key for the repository:

$ sudo mkdir -p /etc/apt/keyrings/

Next, download the key:

$ sudo wget -O - https://packages.cisofy.com/keys/cisofy-software-public.key | sudo tee /etc/apt/keyrings/cisofy-software-public.key > /dev/null

$ cat /etc/apt/keyrings/cisofy-software-public.key

Then, add the repository to apt sources:

$ echo "deb [signed-by=/etc/apt/keyrings/cisofy-software-public.key] https://packages.cisofy.com/community/lynis/deb/ stable main" | sudo tee /etc/apt/sources.list.d/cisofy-lynis.list

Now, you are ready to install lynis:

$ sudo apt update

$ sudo apt install lynis

$ sudo lynis show version

Time to run lynis and get som results:

$ sudo lynis audit system

You will get a long list of results and recommendations. The most important part is the beginning of the result list. If you are lucky, you get this message:
...
  -[ Lynis 3.0.8 Results ]-

  Great, no warnings
...

However, if you get warnings instead, immediate action is required.

Scanning For Malware With clamav

Install clamav:

$ sudo apt install clamav

Try it out. From the terminal, you could do something like this:

$ sudo clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

It limits CPU hogging by excluding directories that are unlikely to contain problems and by limiting file size. Running it after-hours also helps. Set up regular scans after-hours with cron and cpulimit:

$ sudo crontab -e

# m h  dom mon dow   command
*       3       *       *       *       1,3,5   cpulimit -l 30 -- clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

"*       3       *       *       *       1,3,5" means that cron runs the command at 3 am every Monday, Wednesday, and Friday.

The l-optionfor cpulimit limits the maximum resources a clamscan process can use. 30 limits the amount of CPU usage to 30%. Use a value that works for you.

Make sure that cpulimit is installed:

$ sudo apt install cpulimit

Filtering Malicious IPs With fail2ban

Fail2ban scans log files and bans IPs that show certain signs of malicious intent. Filters for many services are ready out of the box [2]. Simply:

$ sudo apt install fail2ban

Unattended Security Upgrades

To get going with unattended security upgrades on Ubuntu is straightforward:

$ sudo apt update && sudo apt upgrade

$ sudo apt install unattended-upgrades -y

For more options, take a look at the configuration file:

$ sudo vi /etc/apt/apt.conf.d/50unattended-upgrades

Furthermore, this article provides a lot of detail on commands and options.

Monitoring With Grafana, Loki and Promtail

Services such as clamav don't do much good if you are not keeping an eye on logs. That also goes for the server maintenance as such. There are many ways to look at logs. Personally, I really like the grafana log-stack, with grafana as a front end, combined with the loki database, and the promtail client-side agent that ships logs to loki.

A normal setup is to have loki and grafana on a separate machine, and then use promtail to shovel logs into loki from each client. To get started fast, look at this webinar on loki and grafana. To get going with promtail, all the details you need are here.

References

[1] Cron in Rocky Linux - https://docs.rockylinux.org/en/guides/automation/cron_jobs_howto/

[2] Fail2ban Wiki - http://www.fail2ban.org/wiki/index.php/Main_Page

[3] Borg 1.24 Installation - https://borgbackup.readthedocs.io/en/1.2-maint/installation.html

[4] Firewalls (for Debian) - https://wiki.debian.org/Firewalls

[5] Install/Configure UFW Firewall on Debian 11 Bullseye - https://www.linuxcapable.com/how-to-setup-and-configure-ufw-firewall-on-debian-11-bullseye/

Other useful references:

How to Use the ssh-copy-id Command - https://bytexd.com/how-to-use-the-ssh-copy-id-command/

How to Disable SSH Login With Password - https://linuxhandbook.com/ssh-disable-password-authentication/

CISOFY Software Repository (Community) - https://packages.cisofy.com/community/

How to stop ClamAV/Clamscan from hogging the CPU/ draining the battery in Ubuntu 18.04? https://askubuntu.com/questions/1158514/how-to-stop-clamav-clamscan-from-hogging-the-cpu-draining-the-battery-in-ubuntu

UFW Essentials: Common Firewall Rules and Commands - https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands