steam-backlog-enforcer/DOCS-steam-installer.md
Krzysztof kuhy Rudnicki a76e726e36 Merge steam-game-installer into the repo
The installer ships as the steam_installer package beside
steam_backlog_enforcer, with its tests, docs and Proton lock. The
enforcer's fast install now starts the installer from this checkout.

Both appinfo.vdf readers share one key-value decoder
(steam_installer.appinfo_kv), and the duplicated test fixtures and
helpers are folded into conftest and _cli_helpers so the whole-tree
duplicate-code check is clean. check_requirements_sync treats sibling
packages as first-party.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzxdWZCxnNWLtpVwFLVk9w
2026-11-22 20:25:39 +01:00

233 lines
12 KiB
Markdown

# Steam game installer
Download an owned Steam game with `depotdownloader` (which saturates a link far
better than the Steam client), then make the Steam client accept the result as
a real install so **Play** works.
```bash
steam_installer/run.sh BeamNG.drive
```
That is the whole thing. `steam_installer/run.sh` installs its own dependencies (`libibus` via
pacman, DepotDownloader from the SteamRE release and the native
Proton described below) on first use, then works out what you meant. It never
runs `steamcmd`: app info comes from Steam's own cache and the tool writes the
manifest itself. All four of these install the same game:
```bash
steam_installer/run.sh 284160 # an AppID
steam_installer/run.sh https://steamcommunity.com/app/284160/guides/ # any Steam URL
steam_installer/run.sh BeamNG.drive # the exact name
steam_installer/run.sh BeamNG # part of the name
```
An exact title always wins, so `steam_installer/run.sh Portal` installs Portal and not
Portal 2 — even though Steam's own search ranks the sequel first. An ambiguous
name shows a numbered picker, and in a non-interactive run it lists the
candidates and exits 2 rather than guessing.
Useful flags: `--dry-run` (resolve and report, change nothing), `--yes`
(no prompts), `--repair-ghosts` (see below), `--valve-proton` (keep this game on
Valve's Proton for this run), `--map-installed` (see below), `--username`
(remembered after the first run, in `~/.config/steam-game-installer/config`;
never a password).
## Games run on a FOSS Proton
A plain Windows game is mapped to `proton-cachyos-native`, a Proton that
CachyOS builds from source against the host's own libraries. Steam then runs
it with no Steam Linux Runtime container, and Proton Experimental, Steam Linux
Runtime 4.0 and the Steamworks Redistributables are not needed while only such
games are installed (steam-backlog-enforcer removes them then).
- **The tool.** `steam_installer/install_deps.sh` downloads the exact build pinned in
`steam_installer/data/proton-native.lock` from the CachyOS mirror. It checks
the package's `.sig` against the CachyOS key pinned in
`steam_installer/data/cachyos_signing_key.asc` (fingerprint and sources in
`proton_pkg.py`), and its sha256 against the lock. Then it extracts only the
tool folder into `compatibilitytools.d/` and writes a `VERSION` stamp; the
package's `.INSTALL` is never run. A copy already on disk is adopted when
every file matches the package's `.MTREE`. An update is swapped in by rename,
only while Steam and every game are closed. The dependency-freshness gate
checks the lock against `cachyos.db`.
- **Which games.** Only a game proven safe: Windows-only (the app, every depot,
every launch option), no shared depots, no owned DLC depot, not in
AreWeAntiCheatYet's `games.json` (fetched every run, last good copy cached;
no list means nothing is mapped), and not in
`steam_installer/data/compat_overrides.json` (`{"<appid>": "<reason>"}` for a
game that breaks on the native build). Anything else keeps Valve's default,
so it runs exactly as before. The run prints every reason.
- **The mapping.** With Steam closed, the game gets a `CompatToolMapping`
entry in `config/config.vdf` (`"name" "proton-cachyos-native"`, priority
250). The file is edited as text, and none of its other lines is ever logged.
An entry Valve chose (priority 100) is replaced; one you picked is kept.
- **Games already installed.** `steam_installer/run.sh --map-installed` prints the decision
for every installed game and writes nothing; add `--apply` to close Steam and
write the safe ones. `--appids 730,1201270` decides any cached app instead.
Proven live on 2026-11-22 with A Space for the Unbound (1201270). Not tested
yet, so they stay on Valve's default or are fixture-tested only: a game with
shared depots (its from-scratch manifest records `SharedDepots`), an owned DLC
depot, a native Linux game, an anti-cheat game, a global default mapping and a
package update.
**Steam gets closed for you.** Steam owns `appmanifest_<appid>.acf` while it
runs and rewrites it from its own in-memory state, so anything written
underneath a live client is discarded. The tool asks, shuts it down with
`steam -shutdown`, and restarts it afterwards. It never sends a signal: a client
that refuses to exit aborts the run instead.
## It takes over a download Steam already started
Steam stages an in-progress download into `steamapps/downloading/<appid>/`, laid
out exactly like the finished game. Those bytes are yours — you waited for them
— so the tool **hardlinks them into place** before calling depotdownloader,
which hash-checks what is already on disk and fetches only the remainder.
Hardlinks, not moves and not copies. Nothing is deleted up front: the staging
tree is cleared only at the very end, so a failed run still leaves Steam able to
resume its own download. Linking is free, where copying a 47 GB tree would cost
47 GB.
Because the two trees share inodes, depotdownloader patching a partial file in
place also rewrites Steam's copy. That is deliberate — both names converge on
the same corrected content — but it does mean the staged bytes are working data,
not a frozen backup. On ext4 there is no reflink that would give you both.
## Owned DLC comes down fast too
Some DLC keeps its content under an app id of its own rather than in the
game's depot list -- Cyberpunk 2077's Phantom Liberty is app 2138330, about
25 GB. A depotdownloader run for the game never sees it, and Steam, restarted
onto a manifest without it, downloads all of it at its own pace.
So every DLC in the game's `listofdlc` that has depots of its own gets its own
depotdownloader run into the same install dir. Depotdownloader's licence check
decides ownership: its "not available from this account" is skipped as
expected, any other failure aborts before the manifest is touched. Each DLC
depot is then written into `InstalledDepots` with its `dlcappid`, exactly as
Steam records it.
## Which depotdownloader
Always `~/.local/bin/depotdownloader` (the copy `steam_installer/install_deps.sh` manages) when it
exists, never "whatever is first on PATH". Each depotdownloader binary keeps
its remembered login in its own storage slot, keyed by its path; with an AUR
copy in `/usr/bin` too, a terminal run and a `sudo`'d run (whose PATH lacks
`~/.local/bin`) used different binaries -- one logged in, one prompting for a
password nobody was there to type.
## Ghost installs
Older versions of this tool ran steamcmd. `steamcmd +login anonymous +app_update
<paid appid>` has no licence for the app.
It does not say so. It transfers nothing, prints `Success! App '...' fully
installed.`, exits 0, and writes a manifest claiming `StateFlags=4` with
`SizeOnDisk=0` and an empty `InstalledDepots`.
That manifest is not harmless: a running Steam client reads it, concludes the
app is installed but out of date, and downloads the entire game by itself.
```bash
steam_installer/run.sh --repair-ghosts
```
finds them across every library and clears them away — but only the *cold* ones.
A ghost with staged content behind it is a real partial download; it is reported
and left alone, to be adopted by a normal install instead. Detection requires
all of: the manifest claims installed, `InstalledDepots` is empty, `SizeOnDisk`
is `0`, there are zero content bytes on disk, and there is no `.DepotDownloader`
state. Repair uses `rmdir`, never `rmtree`, so a non-empty directory cannot be
deleted even if the diagnosis were wrong.
## Run logs
Every run writes a full log:
```
~/.local/state/steam-game-installer/run-<timestamp>-<appid>.log
~/.local/state/steam-game-installer/latest.log -> most recent
```
The path is printed at the start and end of every run, and on every error path.
It contains the full argv, exit code and complete output of each subprocess, the
manifest verbatim before and after every write, the depot cache listing,
expected-vs-actual depots, and whether Steam was running. Send `latest.log` when
something goes wrong. `--verbose` mirrors all of it to the console.
Subprocess output is teed through a pty rather than captured, because
depotdownloader only prompts for a password / Steam Guard code when it believes
it is on a terminal — with a plain pipe the prompt never appears and the run looks hung.
## How it works
1. App info (name, installdir, depots, build) is read from Steam's own cache,
`appcache/appinfo.vdf`. An app missing from it fails with the fix: start
Steam once, let it sync, rerun. A DLC missing from it counts as not owned.
2. `depotdownloader` pulls the content (fast path).
Then each owned DLC with depots of its own gets a run of its own.
3. The manifest is finished by hand, or written from scratch with the 15 keys
Steam accepted when none exists, but only when every precondition below
holds. Steam adds `LastOwner`, `UserConfig` and the rest itself.
4. The game is mapped to the native Proton when it is safe (see above).
## Fail-closed preconditions
The tool refuses to mark an app installed, and exits non-zero, unless:
- `depotdownloader` exited 0;
- every required depot has a cached manifest under `.DepotDownloader/`
(required = this app's depots for the chosen language and OS, excluding DLC,
optional and shared depots — KCD2 lists 12 depots but an english/windows
install only ever fetches 3);
- every required depot holds the public gid from Steam's cache, which proves
the build (a differing gid means the cache is stale: start Steam, let it
sync, rerun), or else the manifest carries a matching real `buildid`;
- no Steam client is running (it is closed first, and restarted afterwards);
- the manifest reads back as a real install *after* being written.
This matters: an earlier version wrote `StateFlags=4` and reported success after
a steamcmd run that had done nothing at all. A from-scratch manifest is built in
memory and only written once every check has passed.
## Failure modes this encodes
Each of these cost a real debugging session; the tests pin them.
**A wrong `installdir` makes Steam wipe the game.** Steam treated the correctly
downloaded folder as orphaned, deleted it and started a fresh download into the
folder its manifest named. The field is fixed first, as soon as Steam is closed.
**Steam rejects an internally inconsistent manifest.** A file that says
`StateFlags=4` while `BytesDownloaded` (22 GB) still differs from
`BytesToDownload` (92 GB) is self-contradictory; Steam discards the "installed"
claim and resumes downloading. Across every genuinely installed manifest the
counters are equal in pairs and `StagingSize` is `0`. `DownloadType` is *kept* —
it is present on every real install, so removing it is not the fix.
**Staging must be cleared too.** Steam stages chunks in
`steamapps/downloading/<appid>/`. Declaring the app installed while tens of GB
sit there pending makes Steam resume the update. Installed apps have an empty
(4 KB) staging dir.
**Shared depots are not installed depots.** `228989` belongs to app 228980
(Steamworks Common Redistributables); listing it in `InstalledDepots`
contradicts `SharedDepots`.
**`SizeOnDisk` excludes `.DepotDownloader/`** — its cache and staging tree are
not game content and overstate the total by gigabytes.
## Development
The installer shares this repo's gate (ruff, mypy, pylint, bandit, 250-line cap)
and its test run, which enforces **100% of statements and branches** for both
packages (`fail_under = 100`):
```bash
python3 -m pytest -q # both suites, coverage gated
scripts/test_changed.sh # only the tests of changed files
```
The steam-backlog-enforcer daemon runs the installer from this checkout as
`python3 -m steam_installer.cli`; it needs no virtualenv.