From 8f2c46de7cd7231f1b50128c51a388e956b45b77 Mon Sep 17 00:00:00 2001 From: Krzysztof kuhy Rudnicki Date: Sun, 9 Aug 2026 17:21:51 +0200 Subject: [PATCH] Add setup_searxng.sh: self-hosted SearXNG at searx.kuhy.duckdns.org Deploys a SearXNG metasearch instance behind the existing shared Caddy edge. Graded A+ (score 120, 10/10) by the MDN Observatory API, and "Vanilla" in the sense that every change is settings-only -- no template or static-file edits. Public but deliberately not submitted to the searx.space directory. Both containers run network_mode: host because Docker bridge networking has zero egress on this box (nftables' default-drop forward chain is terminal at NF_INET_FORWARD regardless of Docker's legacy-iptables ACCEPTs), so a stock bridge compose cannot reach a single search engine. Non-obvious things established empirically against the pinned image, all of which cost a debugging cycle and are commented at their use sites: - granian binds :: (all interfaces) by default, NOT the 127.0.0.1 its own docs claim; with host networking that would have LAN-exposed the instance. GRANIAN_HOST/PORT are the only things pinning it. settings.yml's server.bind_address is inert (read only under __main__). - Caddy's `> Header "v"` is a substring-replace, not a header setter -- it silently set nothing. The bare name is "set"; `+` appends and would duplicate the headers SearXNG already sends (X-Content-Type-Options, Referrer-Policy). - server.public_instance force-enables link_token, a browser-proof challenge no scripted client can pass, which makes the JSON API unusable. Dropped it and kept limiter/image_proxy via their own keys. - Without limiter.toml there are no trusted_proxies, so botdetection 429s every request including the homepage. - The entrypoint chowns config to uid 977 and needs DAC_OVERRIDE to read it, so cap_drop breaks startup and mode 600 breaks the worker's read (640 is right). - `compose up -d` is a no-op when only a bind-mounted config file changed, so an explicit restart is required or every check tests the previous config. - All curl checks must send browser-shaped headers; bare curl gets 429, which had the status command reporting a healthy instance as broken. Engine pruning is measured, not guessed: brave/duckduckgo/startpage CAPTCHA on 100% of queries, flickr is the 1.0s tail, and pinterest was failing because this host's own /etc/hosts blackholes it -- a local cause that would otherwise have been misdiagnosed as a dead upstream. Measured latency, reported honestly: front page median 19ms, but search median 486ms (233-623ms). Sub-200ms search is not achievable by any configuration -- a metasearch cannot return faster than the slowest upstream it queries. The ~200ms figure on searx.space measures initial front-page response, which is met. Image pinned by digest: the workarounds above are only known-true for 2026.8.4. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Cb77nxXGwFYFM4fyXEQkzG --- .../evidence/setup-searxng-2026-08-09.json | 110 +++ .../single_use/features/setup_searxng.sh | 764 ++++++++++++++++++ 2 files changed, 874 insertions(+) create mode 100644 docs/superpowers/evidence/setup-searxng-2026-08-09.json create mode 100755 linux_configuration/scripts/single_use/features/setup_searxng.sh diff --git a/docs/superpowers/evidence/setup-searxng-2026-08-09.json b/docs/superpowers/evidence/setup-searxng-2026-08-09.json new file mode 100644 index 00000000..fa057a22 --- /dev/null +++ b/docs/superpowers/evidence/setup-searxng-2026-08-09.json @@ -0,0 +1,110 @@ +{ + "intent": "Self-host a SearXNG metasearch instance at searx.kuhy.duckdns.org behind the existing shared Caddy edge, configured to earn an A+ Mozilla Observatory grade while remaining 'Vanilla' (settings-only changes, no template/static edits). Publicly reachable but deliberately NOT submitted to the searx.space directory. JSON API enabled for local programmatic/MCP use.", + "scope": [ + "NEW: linux_configuration/scripts/single_use/features/setup_searxng.sh (idempotent setup|status|help)", + "Generated at runtime (outside repo): ~/searxng/docker-compose.yml, ~/searxng/core-config/{settings.yml,limiter.toml}, ~/gitea/sites/searx.caddy", + "Non-goal: searx.space listing (user opted out), so 'Vanilla' is a build constraint rather than a graded outcome", + "Non-goal: sub-200ms search latency -- physically unachievable for a metasearch (see verification)", + "Does NOT add a second Caddy, a second DuckDNS updater, or hand-edit /etc/nftables.conf" + ], + "changes": [ + "Both containers run network_mode: host. Docker BRIDGE networking has zero egress on this host (nftables 'chain forward { policy drop; }' is terminal at NF_INET_FORWARD regardless of Docker's legacy-iptables ACCEPTs), so a stock bridge compose cannot reach a single search engine.", + "GRANIAN_HOST=127.0.0.1 / GRANIAN_PORT=8090 pin the bind. VERIFIED EMPIRICALLY that granian binds :: (all interfaces) by default in this image despite its own docs claiming 127.0.0.1 -- with host networking that would have LAN-exposed it. server.bind_address in settings.yml is inert (read only under __main__/Flask dev server).", + "Caddy snippet sets 5 security headers using the BARE header name ('set' semantics). The '>' prefix is NOT a header setter -- it adapts to {\"replace\":{\"\":[{\"search_regexp\":...}]}} and silently sets nothing; '+' appends and would duplicate. SearXNG sends X-Content-Type-Options and Referrer-Policy itself (its Referrer-Policy is 'no-referrer'), so 'set' is required to override rather than duplicate.", + "limiter.toml written explicitly: without it SearXNG logs 'missing config file' and runs with no trusted_proxies, so botdetection 429s every request including the homepage.", + "server.public_instance deliberately NOT set: it force-enables link_token (a browser-proof challenge no scripted client can satisfy), which makes the requested JSON API unusable. limiter and image_proxy are kept via their own keys.", + "botdetection pass_ip scoped to loopback ONLY, so local tools can use format=json while public traffic (real client IP via X-Forwarded-For) stays rate-limited.", + "start_stack does an explicit docker restart: 'compose up -d' is a no-op when only a bind-mounted config file changed, so without it every verification silently tests the previous config.", + "reclaim_config_ownership: the entrypoint chowns core-config to uid 977, which would make the second setup run fail with 'Permission denied'. settings.yml is 640 not 600 (600 broke the worker's read).", + "No cap_drop on the searxng container -- it needs DAC_OVERRIDE/FOWNER to chown and read config before dropping to uid 977. valkey keeps cap_drop: [ALL] (verified clean).", + "All curl-based checks send browser-shaped headers (UA + gzip Accept-Encoding + Sec-Fetch-*). Bare curl gets 429 from botdetection, so the status check previously reported a perfectly healthy instance as broken.", + "Engine pruning is measured, not guessed: brave/duckduckgo/startpage disabled (CAPTCHA on 100% of queries), flickr (1.0s median), pinterest (blackholed to 0.0.0.0 by this host's own /etc/hosts -- a LOCAL failure that would otherwise be misdiagnosed as a dead upstream)." + ], + "verification": [ + { + "command": "shellcheck + shfmt -d setup_searxng.sh", + "result": "pass", + "evidence": "Both clean, no findings. bash -n syntax OK." + }, + { + "command": "setup_searxng.sh setup (run end to end, repeatedly)", + "result": "pass", + "evidence": "12 [ OK ] lines. Idempotent across 6 consecutive runs; secret_key preserved (verified c72ff7f5e187... survived re-runs)." + }, + { + "command": "setup_searxng.sh status", + "result": "pass", + "evidence": "18/18 checks OK: both containers running + loopback-only, valkey PONG, snippet present, DNS resolves, external HTTPS 200, all 5 headers exactly 1, live search 29 results." + }, + { + "command": "curl -sI https://searx.kuhy.duckdns.org/ | grep -ci '^
:' (x5)", + "result": "pass", + "evidence": "content-security-policy=1 strict-transport-security=1 x-content-type-options=1 referrer-policy=1 permissions-policy=1. referrer-policy value is 'strict-origin-when-cross-origin' (ours) overriding SearXNG's 'no-referrer', proving 'set' semantics work." + }, + { + "command": "loopback binding assertion on both containers", + "result": "pass", + "evidence": "searxng 127.0.0.1:8090, searxng-valkey 127.0.0.1:6379. Guard rejects *, 0.0.0.0 AND [::], and demands positive evidence first (fails closed)." + }, + { + "command": "real search through public HTTPS with browser headers", + "result": "pass", + "evidence": "HTTP 200, 28 results in 0.90s for 'arch linux'. Post-prune: 20 results/query with unresponsive_engines=[] across 3 queries." + }, + { + "command": "JSON API scoping (the decisive check)", + "result": "pass", + "evidence": "Loopback client -> 200 with real JSON results. Simulated public scripted client (X-Forwarded-For set, no Sec-Fetch-*) -> 429. Exactly the intended split." + }, + { + "command": "CSP compatibility audit of actual page assets", + "result": "pass", + "evidence": "All CSS/JS/img/manifest same-origin (/static/...). ZERO inline scripts (the single