testsAndMisc/docs/superpowers/evidence/setup-searxng-2026-08-09.json

{
  "intent": "Self-host a SearXNG metasearch instance at searx.kuhy.duckdns.org behind the existing shared Caddy edge, configured to earn an A+ Mozilla Observatory grade while remaining 'Vanilla' (settings-only changes, no template/static edits). Publicly reachable but deliberately NOT submitted to the searx.space directory. JSON API enabled for local programmatic/MCP use.",
  "scope": [
    "NEW: linux_configuration/scripts/single_use/features/setup_searxng.sh (idempotent setup|status|help)",
    "Generated at runtime (outside repo): ~/searxng/docker-compose.yml, ~/searxng/core-config/{settings.yml,limiter.toml}, ~/gitea/sites/searx.caddy",
    "Non-goal: searx.space listing (user opted out), so 'Vanilla' is a build constraint rather than a graded outcome",
    "Non-goal: sub-200ms search latency -- physically unachievable for a metasearch (see verification)",
    "Does NOT add a second Caddy, a second DuckDNS updater, or hand-edit /etc/nftables.conf"
  ],
  "changes": [
    "Both containers run network_mode: host. Docker BRIDGE networking has zero egress on this host (nftables 'chain forward { policy drop; }' is terminal at NF_INET_FORWARD regardless of Docker's legacy-iptables ACCEPTs), so a stock bridge compose cannot reach a single search engine.",
    "GRANIAN_HOST=127.0.0.1 / GRANIAN_PORT=8090 pin the bind. VERIFIED EMPIRICALLY that granian binds :: (all interfaces) by default in this image despite its own docs claiming 127.0.0.1 -- with host networking that would have LAN-exposed it. server.bind_address in settings.yml is inert (read only under __main__/Flask dev server).",
    "Caddy snippet sets 5 security headers using the BARE header name ('set' semantics). The '>' prefix is NOT a header setter -- it adapts to {\"replace\":{\"\":[{\"search_regexp\":...}]}} and silently sets nothing; '+' appends and would duplicate. SearXNG sends X-Content-Type-Options and Referrer-Policy itself (its Referrer-Policy is 'no-referrer'), so 'set' is required to override rather than duplicate.",
    "limiter.toml written explicitly: without it SearXNG logs 'missing config file' and runs with no trusted_proxies, so botdetection 429s every request including the homepage.",
    "server.public_instance deliberately NOT set: it force-enables link_token (a browser-proof challenge no scripted client can satisfy), which makes the requested JSON API unusable. limiter and image_proxy are kept via their own keys.",
    "botdetection pass_ip scoped to loopback ONLY, so local tools can use format=json while public traffic (real client IP via X-Forwarded-For) stays rate-limited.",
    "start_stack does an explicit docker restart: 'compose up -d' is a no-op when only a bind-mounted config file changed, so without it every verification silently tests the previous config.",
    "reclaim_config_ownership: the entrypoint chowns core-config to uid 977, which would make the second setup run fail with 'Permission denied'. settings.yml is 640 not 600 (600 broke the worker's read).",
    "No cap_drop on the searxng container -- it needs DAC_OVERRIDE/FOWNER to chown and read config before dropping to uid 977. valkey keeps cap_drop: [ALL] (verified clean).",
    "All curl-based checks send browser-shaped headers (UA + gzip Accept-Encoding + Sec-Fetch-*). Bare curl gets 429 from botdetection, so the status check previously reported a perfectly healthy instance as broken.",
    "Engine pruning is measured, not guessed: brave/duckduckgo/startpage disabled (CAPTCHA on 100% of queries), flickr (1.0s median), pinterest (blackholed to 0.0.0.0 by this host's own /etc/hosts -- a LOCAL failure that would otherwise be misdiagnosed as a dead upstream)."
  ],
  "verification": [
    {
      "command": "shellcheck + shfmt -d setup_searxng.sh",
      "result": "pass",
      "evidence": "Both clean, no findings. bash -n syntax OK."
    },
    {
      "command": "setup_searxng.sh setup (run end to end, repeatedly)",
      "result": "pass",
      "evidence": "12 [ OK ] lines. Idempotent across 6 consecutive runs; secret_key preserved (verified c72ff7f5e187... survived re-runs)."
    },
    {
      "command": "setup_searxng.sh status",
      "result": "pass",
      "evidence": "18/18 checks OK: both containers running + loopback-only, valkey PONG, snippet present, DNS resolves, external HTTPS 200, all 5 headers exactly 1, live search 29 results."
    },
    {
      "command": "curl -sI https://searx.kuhy.duckdns.org/ | grep -ci '^<header>:' (x5)",
      "result": "pass",
      "evidence": "content-security-policy=1 strict-transport-security=1 x-content-type-options=1 referrer-policy=1 permissions-policy=1. referrer-policy value is 'strict-origin-when-cross-origin' (ours) overriding SearXNG's 'no-referrer', proving 'set' semantics work."
    },
    {
      "command": "loopback binding assertion on both containers",
      "result": "pass",
      "evidence": "searxng 127.0.0.1:8090, searxng-valkey 127.0.0.1:6379. Guard rejects *, 0.0.0.0 AND [::], and demands positive evidence first (fails closed)."
    },
    {
      "command": "real search through public HTTPS with browser headers",
      "result": "pass",
      "evidence": "HTTP 200, 28 results in 0.90s for 'arch linux'. Post-prune: 20 results/query with unresponsive_engines=[] across 3 queries."
    },
    {
      "command": "JSON API scoping (the decisive check)",
      "result": "pass",
      "evidence": "Loopback client -> 200 with real JSON results. Simulated public scripted client (X-Forwarded-For set, no Sec-Fetch-*) -> 429. Exactly the intended split."
    },
    {
      "command": "CSP compatibility audit of actual page assets",
      "result": "pass",
      "evidence": "All CSS/JS/img/manifest same-origin (/static/...). ZERO inline scripts (the single <script> has src= and passes config via a base64 client_settings attribute). ZERO inline style= attributes. Autocomplete is same-origin /autocompleter (200). JS bundle references zero absolute external origins. Image results use data-src=\"/image_proxy?url=...\" (780 occurrences, same-origin lazy-load) -- external URLs appear only as encoded query params, never as fetch targets, so img-src 'self' data: is safe."
    },
    {
      "command": "engine pruning round 3 -- FIX: disabled -> inactive",
      "result": "pass -- pruned engines now actually stop executing",
      "evidence": "Round 2 used `disabled: true`, which per enginelib only sets the DEFAULT for a fresh visitor ('will allow the user to manually activate it in the settings') -- any browser session with an existing preferences cookie kept running the pruned engines, and yandex images was still throwing JSONDecodeErrors minutes after being 'disabled'. `inactive: true` is the key that actually removes an engine. Converted 84 entries. Verified behaviourally: an images search now serves from google cse images ALONE (20 results, unresponsive=[]) and NO pruned engine appears in the logs after restart. EXCEPTION: brave and qwant keep `disabled`, because sibling engines reference their outgoing pool by name (network: brave <- brave.images/.videos/.news; network: qwant <- qwant news/images/videos) and `inactive` removes the definition, crashing startup with KeyError: 'qwant'. Pool owners in this image: brave, qwant, yandex, piped, yacy."
    },
    {
      "command": "public-path 429 investigation (turned out to be the test harness, not the deployment)",
      "result": "resolved -- public search works: general 200/21 results/0.71s, images 200/40 proxied thumbs/0.58s",
      "evidence": "Public /search returned 429 while loopback returned 200, which looked like a real regression. Root cause was a MISSING Accept-Language header in the curl invocation, not the deployment: botdetection runs one check per header and rejects on the first failure. Found it by setting `debug: true` under general:, which logs the exact rejecting method -- 'NOT OK (searx.botdetection.http_accept_language): 79.184.24.6/32'. That log line also confirmed X-Forwarded-For/X-Real-IP arrive correctly through the edge. Earlier theories (rate limiting, method: POST, stale config) were all wrong and were discarded on evidence. wait_for_site and check_headers were missing Accept-Language too and would have false-failed the same way; both fixed, and the required header set is now documented at check_http."
    },
    {
      "command": "engine pruning round 2 (kuhy-selected from /preferences, 9 groups, all categories)",
      "result": "pass -- 84 engines disabled, 0 unresponsive in every category",
      "evidence": "Disabled every erroring engine (CAPTCHA/access denied/timeout/parsing error/crash) plus the slow tail across general, images, videos, news, music, it, science, files and social media. Three engines NOT in kuhy's list surfaced as unresponsive only under load and were pruned with the rest: brave.images, startpage images, kickass. Post-prune per-category (q=linux): general 572ms/20, images 406ms/20, videos 1378ms/119, news 455ms/52, music 461ms/48, it 1698ms/114, science 1338ms/50, files 584ms/125, social media 1074ms/120 -- unresponsive_engines=[] for ALL nine."
    },
    {
      "command": "images category under 1 second (kuhy's explicit ask)",
      "result": "pass -- 406-900ms, was 2.1s",
      "evidence": "12 queries measured post-prune: cat 900, dog 343, arch+linux 501, mountain 446, coffee 687, bridge 511, sunset 612, laptop 450, guitar 461, ocean 504, forest 624, city 450 ms -- all sub-1s, still 20 results each. ROOT CAUSE of the old 2.1s was NOT google cse images being slow (1.1s): SearXNG waits for the slowest engine in the category, so quark images at 2.9s set the floor. Removing the other image engines removed the ceiling, and also freed connection-pool contention."
    },
    {
      "command": "general-search single-point-of-failure check (correcting an earlier wrong claim)",
      "result": "documented in-file + guarded in status",
      "evidence": "Initially asserted general search retained 'three working sources' (google cse + google + bing) because !go returned 20 and !bi returned 10. That was WRONG: upstream marks both google and bing 'inactive: true', so they answer ONLY to an explicit bang and were never in the merged view. The merged general view is served by google cse alone, and was before this pruning too -- so the pruning removed only zero-contribution engines, but general search does have a single point of failure. Comment corrected in settings.yml and status_cmd now prints the !go/!bi fallback if a live search returns 0 results."
    },
    {
      "command": "real browser load + search (Thorium with the Claude extension)",
      "result": "pass -- CSP confirmed by a rendering engine, closing the last gap",
      "evidence": "Front page renders fully styled (logo, search box, About/Preferences, footer 2026.8.4+cd3835bd2). Text search 'arch linux wiki' returned ArchWiki/Wikipedia/Reddit/LWN/GitHub results in 0.9s with favicons + thumbnails loading. Image search returned hundreds of proxied thumbnails in 2.1s. Autocomplete dropdown fired on typing (a fetch to /autocompleter), proving connect-src 'self'. UNFILTERED console over the whole session: 4 messages, all benign '[PLUGIN] calculator: Running/Done' DEBUG lines from SearXNG's own bundle -- ZERO 'Refused to load/execute/apply' CSP violations. The '?' placeholder tiles in image results are dead upstream URLs (origin 404s), not blocked requests; a CSP block would have logged a Refused-to-load per image."
    },
    {
      "command": "Mozilla Observatory scan (MDN Observatory API v2)",
      "result": "pass -- A+ GRADED, not inferred",
      "evidence": "POST https://observatory-api.mdn.mozilla.net/api/v2/scan?host=searx.kuhy.duckdns.org -> {\"grade\":\"A+\",\"score\":120,\"tests_passed\":10,\"tests_failed\":0,\"tests_quantity\":10,\"scanned_at\":\"2026-08-09T15:20:15Z\"}. Score exactly matches the predicted 120 (100 base +10 default-src 'none' +5 frame-ancestors +5 Referrer-Policy)."
    },
    {
      "command": "external ingress proof (ACME cert on disk)",
      "result": "pass",
      "evidence": "/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/searx.kuhy.duckdns.org/*.crt exists; openssl s_client shows issuer 'C=US, O=Let's Encrypt, CN=YE1', subject CN=searx.kuhy.duckdns.org, valid Aug 9 - Nov 7 2026. Stock caddy:2.8 has no DNS-provider module, so this had to be HTTP-01 -- i.e. Let's Encrypt's validator reached this host from the public internet on :80. External ingress is therefore verified, not assumed.",
      "note": "This is the durable evidence; the gitea-caddy log line had already rotated away."
    },
    {
      "command": "image pinned by digest",
      "result": "pass",
      "evidence": "searxng/searxng@sha256:f4c8e59de166ed71f6380c0847c312ca51f0d41996e31d0559163b6b09ecde52 (== :latest on 2026-08-09, release 2026.8.4). Confirmed live: docker inspect searxng --format '{{.Config.Image}}' returns the digest. Re-ran setup + status after pinning: 12 [ OK ] / 18 status OK, unchanged."
    },
    {
      "command": "no regression on the shared Caddy edge",
      "result": "pass",
      "evidence": "All five pre-existing sites still answer after the snippet + reload: kuhy 200, gitea 200, dice 200, kuhy-cloud 401 (its normal auth challenge), kuhy-sync 200."
    },
    {
      "command": "reboot survival",
      "result": "pass",
      "evidence": "docker/nftables/cronie all 'enabled'; both containers restart=unless-stopped. No systemd unit needed."
    },
    {
      "command": "latency measurement (the backlog note's '~200ms' line)",
      "result": "measured, reported honestly -- NOT met for search, and not meetable",
      "evidence": "Front page: median 19ms (min 18, max 157) over 10 runs via public HTTPS. SEARCH: median 486ms (min 233, max 623) over 8 distinct queries. Sub-200ms search is not achievable by any configuration -- a metasearch cannot return faster than the slowest upstream engine it queries. The ~200ms figures on searx.space measure initial (front-page) response, which IS met here."
    }
  ],
  "risks": [
    "JSON API is LOCAL-ONLY by design. format=json works for processes on this host (pass_ip exempts loopback); a scripted client from anywhere else gets 429 because it cannot send Sec-Fetch-* headers. This satisfies an MCP server running on this box, but NOT one calling in from the phone or another machine. Widening it would mean adding that client's IP to pass_ip.",
    "RESOLVED 2026-08-09: the strict CSP is now confirmed by a real rendering engine (Thorium + Claude extension), not just static analysis -- see verification. No CSP violations on front page, text search, image search, or autocomplete.",
    "NOT YET VERIFIED by the user: the phone-on-cellular acceptance test. Note this is now about user acceptance, not about whether the public path works -- see the ACME evidence in verification.",
    "brave/duckduckgo/startpage are disabled due to CAPTCHA from this IP. If the IP's reputation changes they could be re-enabled -- re-check with the unresponsive_engines command in settings.yml's comment.",
    "pass_ip exempts loopback from ALL botdetection. Any process on this host can query the JSON API unthrottled. Acceptable for a single-user box; would need revisiting if untrusted local services existed.",
    "The instance is publicly reachable and will attract scrapers. The limiter mitigates but does not eliminate this."
  ],
  "rollback": [
    "docker compose -f ~/searxng/docker-compose.yml down",
    "rm ~/gitea/sites/searx.caddy && docker exec gitea-caddy caddy reload --config /etc/caddy/Caddyfile",
    "Validate after rollback: the other five sites on the shared edge (gitea, kuhy, dice, cloud, sync) still answer -- the snippet is the only shared-state change, and reload_caddy removes it automatically if caddy validate fails.",
    "rm -rf ~/searxng to drop config + the valkey volume (docker volume rm searxng_valkey-data)."
  ]
}