Commit Graph

2 Commits

Author SHA1 Message Date
dc45779d38 Prune 84 SearXNG engines; images category now sub-1s
Some checks are pending
Pre-commit checks / pre-commit (push) Waiting to run
Python tests / test (push) Waiting to run
Shell tests / shell-tests (push) Waiting to run
Shell tests / arch-tests (push) Waiting to run
kuhy went through the /preferences engine table and selected every erroring
engine (CAPTCHA / access denied / timeout / parsing error / crash) plus the
slow tail, across all nine categories. Applied all of them, keeping only
google cse images for the images category as requested.

Result: unresponsive_engines is now [] in EVERY category. Per-category
latency (q=linux): general 572ms, images 406ms, videos 1378ms, news 455ms,
music 461ms, it 1698ms, science 1338ms, files 584ms, social media 1074ms.

Images went 2.1s -> 406-900ms across 12 queries, still 20 results. The root
cause was not google cse images being slow (1.1s): SearXNG waits for the
SLOWEST engine in a category, so quark images at 2.9s set the floor. Cutting
the others removed the ceiling.

Three engines that were not in the list surfaced as unresponsive only under
load and were pruned with the rest: brave.images, startpage images, kickass.

Corrects a wrong claim from the first pass. I had said general search retained
three working sources because !go returned 20 results and !bi returned 10.
Upstream marks both google and bing "inactive: true", so they answer ONLY to
an explicit bang and were never part of the merged view -- which google cse
has been serving alone all along. The pruning therefore removed only engines
already contributing zero, but general search does have a single point of
failure, so status now prints the !go/!bi fallback when a live search returns
nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cb77nxXGwFYFM4fyXEQkzG
2026-08-09 17:42:56 +02:00
8f2c46de7c Add setup_searxng.sh: self-hosted SearXNG at searx.kuhy.duckdns.org
Some checks are pending
Pre-commit checks / pre-commit (push) Waiting to run
Shell tests / shell-tests (push) Waiting to run
Shell tests / arch-tests (push) Waiting to run
Deploys a SearXNG metasearch instance behind the existing shared Caddy edge.
Graded A+ (score 120, 10/10) by the MDN Observatory API, and "Vanilla" in the
sense that every change is settings-only -- no template or static-file edits.
Public but deliberately not submitted to the searx.space directory.

Both containers run network_mode: host because Docker bridge networking has
zero egress on this box (nftables' default-drop forward chain is terminal at
NF_INET_FORWARD regardless of Docker's legacy-iptables ACCEPTs), so a stock
bridge compose cannot reach a single search engine.

Non-obvious things established empirically against the pinned image, all of
which cost a debugging cycle and are commented at their use sites:

- granian binds :: (all interfaces) by default, NOT the 127.0.0.1 its own docs
  claim; with host networking that would have LAN-exposed the instance.
  GRANIAN_HOST/PORT are the only things pinning it. settings.yml's
  server.bind_address is inert (read only under __main__).
- Caddy's `> Header "v"` is a substring-replace, not a header setter -- it
  silently set nothing. The bare name is "set"; `+` appends and would duplicate
  the headers SearXNG already sends (X-Content-Type-Options, Referrer-Policy).
- server.public_instance force-enables link_token, a browser-proof challenge no
  scripted client can pass, which makes the JSON API unusable. Dropped it and
  kept limiter/image_proxy via their own keys.
- Without limiter.toml there are no trusted_proxies, so botdetection 429s every
  request including the homepage.
- The entrypoint chowns config to uid 977 and needs DAC_OVERRIDE to read it, so
  cap_drop breaks startup and mode 600 breaks the worker's read (640 is right).
- `compose up -d` is a no-op when only a bind-mounted config file changed, so an
  explicit restart is required or every check tests the previous config.
- All curl checks must send browser-shaped headers; bare curl gets 429, which
  had the status command reporting a healthy instance as broken.

Engine pruning is measured, not guessed: brave/duckduckgo/startpage CAPTCHA on
100% of queries, flickr is the 1.0s tail, and pinterest was failing because
this host's own /etc/hosts blackholes it -- a local cause that would otherwise
have been misdiagnosed as a dead upstream.

Measured latency, reported honestly: front page median 19ms, but search median
486ms (233-623ms). Sub-200ms search is not achievable by any configuration --
a metasearch cannot return faster than the slowest upstream it queries. The
~200ms figure on searx.space measures initial front-page response, which is met.

Image pinned by digest: the workarounds above are only known-true for 2026.8.4.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cb77nxXGwFYFM4fyXEQkzG
2026-08-09 17:21:51 +02:00