testsAndMisc/docs/superpowers/evidence/thorium-crash-logging-2026-08-09.json

{
  "intent": "Make Thorium (and every Chromium-family browser) record why it died, so 'the browser closes randomly' stops being undiagnosable. Previously there was no evidence at all: Crashpad was never enabled (no crash_reporting consent key, empty crash DB) and stderr from a dmenu/.desktop launch reached no terminal and no journal.",
  "scope": [
    "linux_configuration/scripts/periodic_background/system-maintenance/bin/browser-preexec-wrapper.sh",
    "linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/pacman_whitelist.txt",
    "Non-goal: fixing the crash itself — root cause is still unconfirmed by design; this change is what produces the evidence to confirm it.",
    "Non-goal: changing behaviour for non-Chromium browsers (firefox/librewolf are exec'd unchanged)."
  ],
  "changes": [
    "browser-preexec-wrapper.sh: for Chromium-family names only, launch the browser in the background instead of exec'ing, so the wrapper survives to record how it died. Adds --enable-logging=stderr --v=1 and writes a per-launch log to $XDG_STATE_HOME/browser-crash-logs.",
    "Each log records a PRE-LAUNCH and POST-EXIT snapshot (VRAM used/total, RAM, kernel, nvidia module version) plus the exit code and decoded signal. The two VRAM snapshots are the discriminator between the VRAM-exhaustion and stale-binary hypotheses.",
    "Exit recording moved into record_exit() called from HUP/INT/TERM traps, so the verdict is written even when the wrapper itself is killed; an inline block after `wait` would silently leave the log ending mid-stream.",
    "Count rotation (LOG_KEEP=20) plus a byte watchdog (LOG_MAX_BYTES=32 MiB) that trims to the newest half. Measured 2026-08-09: ~2 MB per 45s of navigation, ~0 while idle — count alone would not bound disk use, and a full disk causes exactly the random deaths under investigation.",
    "Deliberately a direct redirect rather than a pipe into a size filter: in a pipeline `wait` reports the last element's status, which would discard the browser's signal — the one fact the wrapper exists to capture.",
    "pacman_whitelist.txt: allow the exact package name `librewolf` (the `librewolf` keyword stays blocked, as do all other Firefox forks). LibreWolf is LeechBlock-enforced via /etc/librewolf/policies, so this is not an unfiltered escape hatch; needed because Chromium's MV2 sunset ends full uBlock Origin with all filter lists."
  ],
  "verification": [
    {
      "command": "shellcheck + shfmt -w on browser-preexec-wrapper.sh",
      "result": "pass",
      "evidence": "shellcheck clean (no SC codes); shfmt applied repo tab style. No lint suppressions added."
    },
    {
      "command": "TEST A: fake browser killed with SIGSEGV under the wrapper",
      "result": "pass",
      "evidence": "log tail: 'exit code: 139' / 'died on signal: 11 (SEGV)' plus both VRAM snapshots."
    },
    {
      "command": "TEST B: wrapper itself SIGTERMed",
      "result": "pass",
      "evidence": "trap fired: 'exit code: 143' / 'died on signal: 15 (TERM)' — the case an inline post-wait block would have missed."
    },
    {
      "command": "TEST C: orphan check after both death modes",
      "result": "pass",
      "evidence": "no surviving fake-browser processes (initial FAIL traced to a leftover from an aborted setsid run started before the test, and to the pgrep pattern matching the test's own shell)."
    },
    {
      "command": "TEST D: trim watchdog with LOG_MAX_BYTES=4096 against a 4000-line spammer",
      "result": "pass",
      "evidence": "log bounded to 2101 bytes with 1 'log trimmed' marker; newest bytes retained."
    },
    {
      "command": "Real binary: kill -SEGV on live /opt/thorium-browser/thorium (PID 1613592)",
      "result": "pass",
      "evidence": "'exit code: 139' / 'died on signal: 11 (SEGV)' written to the real log. cmdline confirmed the 2026-07-25 mitigation flags AND --enable-logging=stderr --v=1 were both applied."
    },
    {
      "command": "pacman -S librewolf (previously blocked) after whitelist entry + install_pacman_wrapper.sh reinstall",
      "result": "pass",
      "evidence": "install succeeded; policy integrity checksums regenerated by the installer rather than edited in place. LeechBlock present as force_installed and uBlock Origin as normal_installed in the loaded profile (avpem8cg.default-default, per profiles.ini Locked=1); both active=True."
    }
  ],
  "risks": [
    "The five auto-generated app-mode .desktop files under ~/.local/share/applications were rewritten from Exec=/opt/thorium-browser/thorium-browser to the wrapper. Chromium regenerates these, so re-creating an app shortcut silently reverts them — the same gap that left the 2026-07-25 flags un-applied. Not yet automated in setup_periodic_system.sh.",
    "--v=1 is verbose; the watchdog bounds a single log but 20 retained logs can still reach ~640 MB worst case. Drop --v=1 if that proves too costly — the signal + VRAM discriminator does not depend on it.",
    "The watchdog trims with tail -c, which can cut mid-line at the trim boundary.",
    "Whitelisting librewolf widens the digital-wellbeing blocklist by exactly one package; enforcement depends on the LibreWolf policies.json remaining in place."
  ],
  "rollback": [
    "git revert the commit, then re-run install_pacman_wrapper.sh (regenerates policy checksums) and re-deploy the wrapper via the same sed substitution to /usr/local/bin/browser-preexec-wrapper.",
    "Restore the .desktop files from the *.bak-precrashlog copies left alongside them.",
    "After rollback, validate: `pacman -S --print librewolf` is blocked again, and a Thorium launch no longer creates files in ~/.local/state/browser-crash-logs."
  ]
}