Night lockdown: keeping the servers up through the curfew

How a self-discipline shutdown stopped killing seven services every night — and the four bugs that surfaced on the way there.
3ebb56c · 2026-07-17 · linux_configuration/periodic_background/digital_wellbeing/

1. Background

This machine leads a double life. By day it is a desktop. Around the clock it is a small server: Gitea, a public Caddy TLS edge on ports 80/443, a personal website, Open WebUI, Joplin, dufs, plus ollama, dnsmasq, wg-quick@wg0, nftables and sshd.

It also runs a digital-wellbeing curfew: a deliberately hard-to-defeat system that stopped you using the PC late at night. It is built as a fortress, and the distinction between its parts is the key to this whole change:

The action, until this change, was to power the machine off — or, on nights before a wake-alarm day, to arm an RTC timer and hibernate:

tomorrow_dow=$(date -d "tomorrow" +%u)
case "$tomorrow_dow" in
    1|5|6|7)   # tomorrow is Mon/Fri/Sat/Sun
        wake_epoch=$(( $(printf '%(%s)T' -1) + 8 * 3600 ))
        /usr/bin/sudo /usr/sbin/rtcwake -m no -t "$wake_epoch"
        /usr/bin/systemctl hibernate -i
        ;;
    *)  /usr/bin/systemctl poweroff -i ;;
esac

That worked perfectly as self-discipline and was a disaster for the servers: at 21:00 every night, Gitea and everything beside it went dark for eight hours.

2. Intuition

The old design conflated two things that only looked like one:

But “you cannot use the PC” never actually required the power to be off. It required the human interface to be gone. A server has no interface — it is a process listening on a socket. So the two can be separated:

This is exactly why the change is small. The fortress guards the schedule. The action was always just the last few lines of a generated script. Swap those lines and the curfew keeps every one of its guarantees, while the servers never notice:

-  systemctl poweroff -i        # or rtcwake + hibernate -i
+  DRY_RUN="${DRY_RUN:-}" /usr/local/bin/night-lockdown-enter.sh

A wrinkle: the reversal must outlive the fortress

Power-off had one accidental virtue — it was atomic. It either happened or it did not. The new action is a multi-step teardown, and a teardown can fail halfway: getty masked, lightdm stopped, and no way back in. So the unlock deliberately lives outside the fortress: not immutable, not guarded by the monitor, on its own timer with five staggered triggers (05:00, 05:15, 05:30, 06:00, 07:00) and Persistent=true. A bug in the lock path must never be able to take the recovery path down with it.

The alarm bug, made concrete

Recall that the alarm rides on a resume event. Remove hibernate and this happens — silently, with no error anywhere:

There is a second trap hiding inside the first. The which days logic lived in the deleted branch (case 1|5|6|7), not in the alarm. The alarm's own gate fires every day. So a naive “just start it from a timer” fix would ring on Tuesdays too. The unlock therefore re-implements the Mon/Fri/Sat/Sun gate itself.

Play with the schedule rule

The check script compares now against two thresholds. One character in that comparison — -le versus -lt — decides whether 05:00 belongs to the night or the morning. Drive the clock to 05:00 and watch the two verdicts disagree:

At 05:00 the old rule still said lock, while the unlock timer was firing at that same instant — the 30-minute tick and the unlock racing each other over the same minute. Making the boundary strict (-lt) means the last lock tick is 04:30, 05:00 is a clean no-op, and the unlock owns the morning uncontested.

3. Code

Grouped by concern rather than by file. The new installer setup_night_lockdown.sh generates two scripts (night-lockdown-enter.sh, night-lockdown-unlock.sh), a config, an unlock timer family and a boot-time RGB service; setup_midnight_shutdown.sh only had its action swapped.

3.1 The swap, and the boundary

Both edits are in the heredoc that generates the check script, so they take effect when the installer regenerates it. The action becomes a delegation, with DRY_RUN passed through so the whole path is testable without locking:

-    tomorrow_dow=$(date -d "tomorrow" +%u)
-    case "$tomorrow_dow" in
-        1|5|6|7) ... rtcwake ... systemctl hibernate -i ;;
-        *)       ... systemctl poweroff -i ;;
-    esac
+    logger -t day-specific-shutdown "Entering night lockdown (servers stay up)"
+    DRY_RUN="${DRY_RUN:-}" /usr/local/bin/night-lockdown-enter.sh

  # both day branches — the off-by-one from the micro-world
-  if [[ $current_time_minutes -ge $shutdown_start ]] || [[ $current_time_minutes -le $morning_end_minutes ]]
+  if [[ $current_time_minutes -ge $shutdown_start ]] || [[ $current_time_minutes -lt $morning_end_minutes ]]

3.2 Ordering: persist intent before you destroy your own session

The lock script runs inside the session it is about to kill. If it wrote its state afterwards, a kill mid-teardown would leave state=UNLOCKED while the machine was, in fact, locked — and the unlock no-ops when it reads UNLOCKED. That combination strands the machine until 05:00. So the intent is persisted first:

if [[ "$DRY_RUN" != "1" ]]; then
    echo LOCKED >"$STATE_FILE"     # <- before, not after
fi
run systemctl mask getty@.service   # only getty@: autovt@ is an ALIAS of it,
run systemctl stop 'getty@*.service'  #   and `mask autovt@` errors on the existing symlink
run systemctl stop lightdm.service  # <- this kills the session running this script
blank_console

The same reasoning gives the lock its order overall: hardware and user-session work first (RGB, audio, tray apps), then the login surface is masked before lightdm stops so logind cannot spawn a getty into the gap, and only then the GUI goes.

3.3 Darkness, without killing a monitor

Stopping lightdm hands the VT back to fbcon, which unblanks it and prints kernel and systemd messages — a bright wall of log spam rather than a dark screen. Two sources are silenced, then the console is blanked:

silence_console() {
    run kill -s RTMIN+21 1                  # systemd status messages off (RTMIN+20 re-enables)
    cat /proc/sys/kernel/printk >"$STATE_DIR/printk.prev"   # saved for the unlock
    run sysctl -q -w "kernel.printk=1 4 1 4"
}

The darkness matters for a reason beyond aesthetics, which is worth stating because it is not obvious: a lit screen invites you to switch the monitors off by hand — and the monitor's HDMI speaker is one of only two audio sinks the alarm has. A bright console at 21:00 is, indirectly, a silent alarm at 05:00.

3.4 Re-firing the alarm the lockdown displaced

The unlock restores the login surface and the GUI, then — on alarm days only — starts the routine that the vanished resume event used to start. The day gate is re-implemented here because the alarm's own gate fires every day:

maybe_start_morning_alarm() {
    dow="$(date +%u)"                       # 1=Mon .. 7=Sun
    for d in $ALARM_DAYS_DOW; do [[ "$d" == "$dow" ]] && today_is_alarm_day=true; done
    [[ "$today_is_alarm_day" == true ]] || { logg "not an alarm day (dow=$dow)"; return 0; }

    for ((i = 0; i < 60; i++)); do          # bounded wait: lightdm -> autologin -> session
        sudo -u "$LOCK_USER" env "XDG_RUNTIME_DIR=/run/user/$LOCK_UID" \
            systemctl --user is-active graphical-session.target >/dev/null 2>&1 && break
        sleep 1
    done
    run sudo -u "$LOCK_USER" env "XDG_RUNTIME_DIR=/run/user/$LOCK_UID" \
        systemctl --user start --no-block "$MORNING_ROUTINE_UNIT"
}

It is called after the state token flips to UNLOCKED, so the four later unlock triggers dedupe on the token and cannot double-start it during the 60-second wait.

3.5 RGB, and a lesson about exit codes

The RGB step originally gated on a device list and always found zero devices, so it silently did nothing while reporting success. The cause was not the hardware but the build: openrgb-git 0.9.2148's CLI returned in ~70 ms, before its asynchronous detection (~1.9 s) had populated anything — so every invocation acted on an empty device list and still exited 0. Stable openrgb 1.0rc3 waits properly. Two device quirks then remained:

Because the preference is simply no lighting, ever, the unlock re-asserts black rather than restoring anything, and a rgb-off.service re-applies it at every boot — the board firmware lights everything back up on power-on, and nothing else would turn it off again until the next 21:00.

3.6 A guard against a bug that bit twice

The config heredoc must stay unquoted, because it interpolates the detected user, uid and ALSA cards. That means any stray backtick or $ anywhere inside it — including inside a comment — is executed or expanded at write time. It happened twice; the second time it ran openrgb mid-write and pasted its stderr into the config, producing invalid shell. The lock script then failed to source it and skipped every configured step, silently. The fix is to make that class of mistake loud:

if ! bash -n "$CONF_FILE" 2>/dev/null; then
    log_error "Generated $CONF_FILE is not valid shell — refusing to continue."
    exit 1
fi

3.7 One line that is not about lockdown

SCHEDULE_THU_SUN_HOUR moved 22 → 21. This is not a policy change: screen_locker's sick-day feature had ratcheted the canonical schedule to 21 back on 2026-07-03, and the ratchet only permits same-or-stricter. The script's own constant of 22 therefore read as relaxing the schedule, and the installer refused to run at all — meaning the check script could not be regenerated. Aligning the source with the schedule already in force unblocked it.

4. Quiz

Five questions on the reasoning behind the change. Score: 0 / 0