diff --git a/README.md b/README.md index 3225b3f..ed59dce 100644 --- a/README.md +++ b/README.md @@ -128,6 +128,10 @@ docker run --restart unless-stopped \ ghcr.io/analogj/scrutiny:latest-collector ``` +### Hub rootless installation using Podman Quadlets + +See [docs/INSTALL_ROOTLESS_PODMAN.md](docs/INSTALL_ROOTLESS_PODMAN.md) for instructions. + ## Manual Installation (without-Docker) While the easiest way to get started with [Scrutiny is using Docker](https://github.com/AnalogJ/scrutiny#docker), diff --git a/docs/INSTALL_MANUAL.md b/docs/INSTALL_MANUAL.md index 6746add..b9f101e 100644 --- a/docs/INSTALL_MANUAL.md +++ b/docs/INSTALL_MANUAL.md @@ -122,6 +122,11 @@ So you'll need to install the v7+ version using one of the following commands: - `dnf install smartmontools` - **FreeBSD:** `pkg install smartmontools` +The following additional dependencies are needed if you want to run the collector as an unprivileged user: + +- systemd version > 235 +- a restricted user account + ### Directory Structure Now let's create a directory structure to contain the Scrutiny collector binary. @@ -133,40 +138,337 @@ mkdir -p /opt/scrutiny/bin ### Download Files -Next, we'll download the Scrutiny collector binary from the [latest Github release](https://github.com/analogj/scrutiny/releases). -The file you need to download is named: +Next, we'll download the Scrutiny collector binary from the [latest Github release](https://github.com/analogj/scrutiny/releases). You are looking for the one titled **scrutiny-collector-metrics-linux-amd64** unless you know you are on arm. -- **scrutiny-collector-metrics-linux-amd64** - save this file to `/opt/scrutiny/bin` +```sh +wget -O /tmp/scrutiny-collector-metrics https://github.com/AnalogJ/scrutiny/releases/latest/download/scrutiny-collector-metrics-linux-amd64 +``` + +Optional, but recommended: Before continuing it's recommended you compare the sha from the release page with the downloaded file to ensure it's the same file and not corrupted/tampered with. The command to do this is: + +`echo "SHA_GOES_HERE /tmp/scrutiny-collector-metrics" | sha256sum -c` + +example for the v0.8.6 release: + +`echo "4c163645ce24e5487f4684a25ec73485d77a82a57f084808ff5aad0c11499ad2 /tmp/scrutiny-collector-metrics" | sha256sum -c` + +followed by: + +`sudo mv /tmp/scrutiny-collector-metrics /opt/scrutiny/bin/` + +to move the binary to its final resting place ### Prepare Scrutiny Now that we have downloaded the required files, let's prepare the filesystem. -``` +```sh # Let's make sure the Scrutiny collector is executable. -chmod +x /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 +chmod +x /opt/scrutiny/bin/scrutiny-collector-metrics ``` +if you are using SELinux, you may need to also do the following: + +```sh +# tell SELinux to allow these binaries +sudo semanage fcontext -a -t bin_t "/opt/scrutiny/bin(/.*)?" +# update labels +sudo restorecon -Rv /opt/scrutiny/bin +``` + + ### Start Scrutiny Collector, Populate Webapp Next, we will manually trigger the collector, to populate the Scrutiny dashboard: > NOTE: if you need to pass a config file to the scrutiny collector, you can provide it using the `--config` flag. -``` -/opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 run --api-endpoint "http://localhost:8080" +```sh +/opt/scrutiny/bin/scrutiny-collector-metrics run --api-endpoint "http://localhost:8080" ``` -### Schedule Collector with Cron +### Schedule Collector with (root) Cron Finally you need to schedule the collector to run periodically. This may be different depending on your OS/environment, but it may look something like this: -``` +```sh # open crontab -crontab -e +sudo crontab -e # add a line for Scrutiny -*/15 * * * * . /etc/profile; /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 run --api-endpoint "http://localhost:8080" +*/15 * * * * . /etc/profile; /opt/scrutiny/bin/scrutiny-collector-metrics run --api-endpoint "http://localhost:8080" ``` + +### Schedule Collector with Systemd (rootless) + +Alternatively you can run `scrutiny-collector-metrics` as non-root so long as the relevant capabilities and permissions are granted. + + +#### Creating a Restricted Service Account + +This is the account that will run `scrutiny-collector-metrics`. Note this isn't strictly needed for all setups, but is useful from a logging/auditing perspective. + +- Debian-based distros: + - `sudo adduser --system scrutiny-svc --group --home /opt/scrutiny-svc` +- RHEL-based distros: + - `sudo useradd --system --home-dir /opt/scrutiny-svc --shell /sbin/nologin scrutiny-svc` + +Next, add the user to the `disk` group: + +```sh +sudo usermod -aG disk scrutiny-svc +``` + + +#### Creating a Restricted Systemd Service using AmbientCapabilities (easier) + +This is the simpler setup, which allows you to run scrutiny rootless, but depending on what you want, may require granting more permissions to scrutiny than you would like to. + +1. go to `/etc/systemd/system` +2. create scrutiny-collector.service with the following contents: + + +```ini +[Unit] +Description=Daily Restricted Scrutiny Collector +After=network.target + +[Service] +[Unit] +Description=Daily Restricted Scrutiny Collector +After=network.target + +[Service] +Type=oneshot +User=scrutiny-svc +Group=disk +ExecStart=/opt/scrutiny/bin/scrutiny-collector-metrics run --api-endpoint "http://localhost:8080" + +# --- PRIVILEGE LOCKDOWN --- +## CAP_SYS_RAWIO is needed for SATA drives +AmbientCapabilities=CAP_SYS_RAWIO +CapabilityBoundingSet=CAP_SYS_RAWIO +## unfortunately nvme drives require CAP_SYS_ADMIN +## if you want nvme drives you must do the following: +#AmbientCapabilities=CAP_SYS_RAWIO CAP_SYS_ADMIN +#CapabilityBoundingSet= + +NoNewPrivileges=yes + +# Security/sandboxing settings +KeyringMode=private +LockPersonality=yes +MemoryDenyWriteExecute=yes +ProtectSystem=strict +ProtectHome=yes +PrivateDevices=no +## you can restrict devices using: +#DevicePolicy=closed +#DeviceAllow=/dev/sda r +#DeviceAllow=/dev/nvme0 r +ProtectKernelModules=yes +ProtectKernelTunables=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +ProtectKernelLogs=yes +RemoveIPC=yes +RestrictSUIDSGID=true + + +# --- NETWORK LOCKDOWN +## use these to restrict what scrutiny can talk to over the network +## if using a hub on a different host you will need to change the values accordingly +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +IPAddressDeny=any +IPAddressAllow=localhost + +[Install] +WantedBy=multi-user.target + +``` + +Additionally, for nvme drives you may need to create a udev rule on many systems, as /dev/nvme* is often owned only by root: + +##### add udev rule `/etc/udev/rules.d/99-nvme.rules` with contents: + +``` +KERNEL=="nvme[0-9]*", GROUP="disk", MODE="0640" +``` + +then run the following commands to load the udev rule: + +```sh +sudo udevadm control --reload-rules +sudo udevadm trigger --subsystem-match=nvme --action=add +``` + + +##### Pros: + +- easy to maintain +- much better than running as root (especially if you don't need nvme drives) +- there are no privilege escalations needed + + +##### Cons: + +NOTE: These cons basically only apply if a major supply-chain attack happens against scrutiny, and reflect a worst-case scenario that is unlikely to ever occur: + +- CAP_SYS_RAWIO allows for data exfiltration/modification from SATA drives (ssh keys, /etc/shadow, etc) +- CAP_SYS_ADMIN would theoretically allow for significant system compromise +- nvme drives requires a udev rule for reliable access + + +If you are happy with that, you can jump to [Create a Systemd Timer to run scrutiny-collector.service](#create-a-systemd-timer-to-run-scrutiny-collectorservice) + + +#### Creating a Restricted Systemd Service using sudo and Shim Script + +If granting scrutiny `CAP_SYS_RAWIO` and/or `CAP_SYS_ADMIN` exceeds your risk appetite, you have another option, though one more complicated and with its own set of pros/cons + +1. run `sudo mkdir -p /opt/smartctl-shim/bin` +2. edit `/opt/smartctl-shim/bin/smartctl` with the following content: + +```sh +#!/bin/bash +# Shim for accounts to use smartctl without being root +# for automation requires the account be in sudoers +exec /usr/bin/sudo /usr/sbin/smartctl "$@" +``` + +3. create a new `scrutiny-collector` file in `/etc/sudoers.d/` +4. inside `/etc/sudoers.d/scrutiny-collector` add the following: + +```sh +scrutiny-svc ALL=(root) NOPASSWD: /usr/sbin/smartctl * +``` + +5. go to `/etc/systemd/system` +6. create scrutiny-collector.service with the following contents: + + +```ini +[Unit] +Description=Daily Restricted Scrutiny Collector +After=network.target + +[Service] +Type=oneshot +User=scrutiny-svc +Environment="PATH=/opt/smartctl-shim/bin:/usr/bin:/bin" +ExecStart=/opt/scrutiny/bin/scrutiny-collector-metrics run --api-endpoint "http://localhost:8080" + +# --- PRIVILEGE LOCKDOWN --- +## we use sudo to elevate privileges for smartctl only, so no Ambient Capabilities are needed +AmbientCapabilities= +## CAP_SYS_RAWIO is needed for SATA drives +CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_AUDIT_WRITE CAP_SYS_RAWIO CAP_SYS_RESOURCE +## unfortunately nvme drives require CAP_SYS_ADMIN +## if you want nvme drives you must do the following: +# CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_AUDIT_WRITE CAP_SYS_RAWIO CAP_SYS_ADMIN CAP_SYS_RESOURCE + +## since sudo needs to be used to elevate permissions in this setup, we need to allow new privileges +NoNewPrivileges=no + +# Security/sandboxing settings +KeyringMode=private +LockPersonality=yes +MemoryDenyWriteExecute=yes +ProtectSystem=strict +ProtectHome=yes +PrivateDevices=no +ProtectKernelModules=yes +ProtectKernelTunables=yes +ProtectControlGroups=yes +ProtectClock=yes +ProtectHostname=yes +ProtectKernelLogs=yes +RemoveIPC=yes +RestrictSUIDSGID=true + + +# --- NETWORK LOCKDOWN +## use these to restrict what scrutiny can talk to over the network +## if using a hub on a different host you will need to change the values accordingly +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +IPAddressDeny=any +IPAddressAllow=localhost + +[Install] +WantedBy=multi-user.target +``` + + +##### Pros: + +- the scrutiny binary itself will not have permissions like CAP_SYS_ADMIN +- much better than running as root (especially if you don't need nvme drives) +- `sudo` restricts privilege escalation to just `smartctl` +- no udev rule needed + + +##### Cons: + +NOTE: These cons basically only apply if a major supply-chain attack happens against scrutiny, and reflect a worst-case scenario that is unlikely to ever occur: + +- Any sort of privilege escalation attack in sudo could theoretically allow a compromised scrutiny to gain additional privileges, since the process has permission to escelate privileges in general +- Even though sudo only allows `smartctl`, it still has `CAP_SYS_RAWIO` and `CAP_SYS_ADMIN` so in theory the same attacks from the first method are possible, though now only with an exploit using smartctl instead of scrutiny directly +- even though you don't need a udev rule, this adds a lot of additional administrative overhead +- while the scrutiny binary itself isn't elevated, it has a sub-process that is running as root (systemctl) + +#### Create a Systemd Timer to run scrutiny-collector.service + +First, lets test our service. It doesn't matter which method you used above, as either way you need to load and run it. + +```sh +# reload changes for systemd services +sudo systemctl daemon-reload + +# enable the service +sudo systemctl enable scrutiny-collector.service + +# now run the service +sudo systemctl start scrutiny-collector.service +``` + +You should see the data in your hub instance of scrutiny now. If your run into issues I recommend turning on debug logging for scrutiny and checking your system logs using journalctl. It may be a permission is missing or wrong. + +Now that things have been validated, lets create the systemd timer to run the service for us on a schedule: + +1. if you are not still there, go to `/etc/systemd/system` +2. create scrutiny-collector.timer with the following contents: + +```ini +[Unit] +Description=Run Scruitiny Collector daily at 2am + +[Timer] +# Standard calendar trigger +OnCalendar=*-*-* 02:00:00 +# Ensures the job runs if the computer was off at 2am +Persistent=true +# Minimizes I/O spikes by staggering start time +RandomizedDelaySec=30 + +[Install] +WantedBy=timers.target + +``` + +Update the schedule as you see fit for your needs + +Once you are satisfied with our timer, you'll need to load and enable it: + +```sh +# reload changes for systemd services +sudo systemctl daemon-reload + +# now enable the timer +sudo systemctl enable --now scrutiny-collector.timer +``` + +That's it! you're done. You can check the status of the timer using `sudo systemctl status scrutiny-collector.timer +` \ No newline at end of file diff --git a/docs/INSTALL_ROOTLESS_PODMAN.md b/docs/INSTALL_ROOTLESS_PODMAN.md new file mode 100644 index 0000000..4b6e30d --- /dev/null +++ b/docs/INSTALL_ROOTLESS_PODMAN.md @@ -0,0 +1,170 @@ +# Rootless Podman Quadlet Install + +Note: These instructions are written with Podman 4.9 in mind, as that's what's available on Ubuntu 24.04. Podman 5+ can simplify the process using a .pod file to run both the hub and influxdb instance in the same pod, sharing localhost. This is a fairly trivial change should anyone want to add the documentation for it. While this document isn't Ubuntu-specific, this is being purposefully done to allow it to apply to the vast majority of Podman users, regardless of what Linux distro they use. + + +### Dependencies + +- Podman > 4.9 +- Systemd > 250 (for quadlet support) +- a restricted service account + + +### Creating a Service Account + +See [Creating a Restricted Service Account](INSTALL_MANUAL.md#creating-a-restricted-service-account) for instructions. + +While you do not need to use the same account as the collector, this guide will assume you will be for all its examples. + +In addition to those steps, you will need to create sub ids and enable lingering for the user: + +```sh +# add sub-uids and sub-gids, you may need to adjust numbers if you have other rootless quadlets running for other users already +# it is not recommended to go below 100000 +# we choose to start at 500000 in the event you have some other podman accounts +sudo usermod --add-subuids 500000-565535 scrutiny-svc +sudo usermod --add-subgids 500000-565535 scrutiny-svc + +# We want the quadlets to stay running even if the user isn't logged in +sudo loginctl enable-linger scrutiny-svc +``` + + +### Directory Structure + +Once the account is created, you will need to grab its id to create a few drectories for the data files and rootless quadlet files: + +```sh +# create folders for config and influxdb +sudo mkdir -p /opt/scrutiny-svc/scrutiny/{config,influxdb} + +# get the config file for scrutiny hub +sudo wget -O /opt/scrutiny-svc/scrutiny/config/scrutiny.yaml https://raw.githubusercontent.com/AnalogJ/scrutiny/refs/heads/master/example.scrutiny.yaml + +# set permissions on everything +sudo chown -R scrutiny-svc:scrutiny-svc /opt/scrutiny-svc + +# Get the ID of scrutiny-svc so you know it for your own record-keeping +id -u scrutiny-svc + +# create a directory +sudo mkdir -p /etc/containers/systemd/users/$(id -u scrutiny-svc) + +## go into the directory you just created for the rest of the guide +cd /etc/containers/systemd/users/$(id -u scrutiny-svc) +``` + + +### Quadlet Files + +Now that everything is set up and configured for the account to run quadlets, we just need to create a few quadlet files. + +All remaining system actions will take place in `/etc/containers/systemd/users/$(id -u scrutiny-svc)` which is why we had you cd into it. + + +#### Networking + +We need the hub and influxdb instances to be able to talk to each other, and in the case of Podman 4.9, they will run separately not sharing a localhost, and as such we need to configure a network for them to share. The file is pretty simple: + + +##### scrutiny-net.network + +```ini +[Network] +NetworkName=scrutiny-net +``` + + +#### Containers + +Now we're ready for creating the containers + + +##### influxdb.container + +```ini +[Unit] +Description=influxdb + +[Container] +ContainerName=influxdb +Image=docker.io/library/influxdb:2.2 +AutoUpdate=registry +Timezone=local +## not strictly necessary, but keeps file permission sane for influxdb +PodmanArgs=--group-add keep-groups +## versions of podman after 5.1 should do the below instead +#GroupAdd=keep-groups +Volume=/opt/scrutiny-svc/scrutiny/influxdb:/var/lib/influxdb2:Z +Network=scrutiny-net + +[Service] +Restart=on-failure + +[Install] +# Start by default on boot +WantedBy=default.target +``` + + +##### scrutiny-web.container + +```ini +[Unit] +Description=scrutiny-web +After=influxdb.service +Requires=influxdb.service + +[Container] +ContainerName=scrutiny-web +Image=ghcr.io/analogj/scrutiny:latest-web +AutoUpdate=registry +Timezone=local +Volume=/opt/scrutiny-svc/scrutiny/config:/opt/scrutiny/config:Z +Network=scrutiny-net +PublishPort=8080:8080/tcp + +[Service] +Restart=on-failure + +[Install] +# Start by default on boot +WantedBy=default.target +``` + +#### Update scrutiny config + +Since our containers are running separately, we need to update `/opt/scrutiny-svc/scrutiny/config/scrutiny.yaml` to the new influxdb host: + +1. edit `/opt/scrutiny-svc/scrutiny/config/scrutiny.yaml` +2. under `influxdb` section, change `host: 0.0.0.0` to `host: influxdb` -- remember that yaml is whitespace-sensitive! so be mindful of the indents + +```yaml + influxdb: +# scheme: 'http' + host: influxdb + port: 8086 +``` + +# Running the hub and doing the + +With that done, we're now ready to start up the services: + +```sh +# reload all the systemd user files for scrutiny-svc +sudo systemctl --user -M scrutiny-svc@ daemon-reload + +# start the scrutiny-net network: +sudo systemctl --user -M scrutiny-svc@ start scrutiny-net-network.service + +# start influxdb first and wait for it to come up +sudo systemctl --user -M scrutiny-svc@ start influxdb.service + +# check if it's fully up +sudo systemctl --user -M scrutiny-svc@ status influxdb.service + +# now start scrutiny +sudo systemctl --user -M scrutiny-svc@ start scrutiny-web.service +``` + +You are now ready to run the collector, if you would like to run that rootless as well, see the guide at [Schedule Collector with Systemd (rootless)](INSTALL_MANUAL.md#schedule-collector-with-systemd-rootless) \ No newline at end of file