mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-11 19:54:01 +01:00
1d7271f023
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| d9c315e3ef |
Split setup_searxng.sh 912 -> 184
Six libs: the script has four phases and write_stack alone was 342 lines because it emits three config files from heredocs. That became three functions, one per file, each keeping the comment that explains it - including why settings.yml is mode 640 and not 600. For a script whose job is writing config, "the functions still exist" is not enough. Compared the sorted non-comment lines of the original write_stack against its three successors: identical hash, so every line of emitted compose, settings.yml and limiter.toml is byte-for-byte the same and only the wrappers were added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w2Pj4Ek3ZNMQVYcFHgXDY |
|||
| 7cfe188bbe |
Fix engine pruning: disabled -> inactive (they were still running)
The previous commit's pruning did not actually take effect. Per enginelib,
"disabled" only sets the DEFAULT for a fresh visitor -- "will allow the user
to manually activate it in the settings" -- so any browser session with an
existing preferences cookie kept running every pruned engine. yandex images
was still throwing JSONDecodeErrors in the log minutes after being
"disabled". "inactive" is the key that actually removes an engine.
Converted 84 entries. Verified behaviourally rather than by reading config:
an images search now serves from google cse images ALONE (20 results,
unresponsive=[]), and no pruned engine appears in the logs after a restart.
brave and qwant deliberately keep "disabled". Sibling engines reference
their outgoing connection pool by name (network: brave <- brave.images/
.videos/.news; network: qwant <- qwant news/images/videos), and "inactive"
removes the definition entirely, so the container dies at startup with
KeyError: 'qwant'. "disabled" keeps the pool defined while hiding the engine.
Pool owners in this image: brave, qwant, yandex, piped, yacy.
Also fixes two status checks that could report a healthy instance as broken.
Public /search was returning 429 while loopback returned 200, which looked
like a regression but was a missing Accept-Language header in the request:
botdetection runs one check per header and rejects on the first failure.
Setting "debug: true" under general: makes SearXNG log the exact rejecting
method ("NOT OK (searx.botdetection.http_accept_language)"), which settled it
in one run after several wrong theories. wait_for_site and check_headers had
the same omission and are fixed; the full required header set is documented
at check_http.
Public path confirmed working end to end: general 200 with 21 results in
0.71s, images 200 with 40 proxied thumbnails in 0.58s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cb77nxXGwFYFM4fyXEQkzG
|
|||
| dc45779d38 |
Prune 84 SearXNG engines; images category now sub-1s
kuhy went through the /preferences engine table and selected every erroring engine (CAPTCHA / access denied / timeout / parsing error / crash) plus the slow tail, across all nine categories. Applied all of them, keeping only google cse images for the images category as requested. Result: unresponsive_engines is now [] in EVERY category. Per-category latency (q=linux): general 572ms, images 406ms, videos 1378ms, news 455ms, music 461ms, it 1698ms, science 1338ms, files 584ms, social media 1074ms. Images went 2.1s -> 406-900ms across 12 queries, still 20 results. The root cause was not google cse images being slow (1.1s): SearXNG waits for the SLOWEST engine in a category, so quark images at 2.9s set the floor. Cutting the others removed the ceiling. Three engines that were not in the list surfaced as unresponsive only under load and were pruned with the rest: brave.images, startpage images, kickass. Corrects a wrong claim from the first pass. I had said general search retained three working sources because !go returned 20 results and !bi returned 10. Upstream marks both google and bing "inactive: true", so they answer ONLY to an explicit bang and were never part of the merged view -- which google cse has been serving alone all along. The pruning therefore removed only engines already contributing zero, but general search does have a single point of failure, so status now prints the !go/!bi fallback when a live search returns nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cb77nxXGwFYFM4fyXEQkzG |
|||
| 8f2c46de7c |
Add setup_searxng.sh: self-hosted SearXNG at searx.kuhy.duckdns.org
Deploys a SearXNG metasearch instance behind the existing shared Caddy edge. Graded A+ (score 120, 10/10) by the MDN Observatory API, and "Vanilla" in the sense that every change is settings-only -- no template or static-file edits. Public but deliberately not submitted to the searx.space directory. Both containers run network_mode: host because Docker bridge networking has zero egress on this box (nftables' default-drop forward chain is terminal at NF_INET_FORWARD regardless of Docker's legacy-iptables ACCEPTs), so a stock bridge compose cannot reach a single search engine. Non-obvious things established empirically against the pinned image, all of which cost a debugging cycle and are commented at their use sites: - granian binds :: (all interfaces) by default, NOT the 127.0.0.1 its own docs claim; with host networking that would have LAN-exposed the instance. GRANIAN_HOST/PORT are the only things pinning it. settings.yml's server.bind_address is inert (read only under __main__). - Caddy's `> Header "v"` is a substring-replace, not a header setter -- it silently set nothing. The bare name is "set"; `+` appends and would duplicate the headers SearXNG already sends (X-Content-Type-Options, Referrer-Policy). - server.public_instance force-enables link_token, a browser-proof challenge no scripted client can pass, which makes the JSON API unusable. Dropped it and kept limiter/image_proxy via their own keys. - Without limiter.toml there are no trusted_proxies, so botdetection 429s every request including the homepage. - The entrypoint chowns config to uid 977 and needs DAC_OVERRIDE to read it, so cap_drop breaks startup and mode 600 breaks the worker's read (640 is right). - `compose up -d` is a no-op when only a bind-mounted config file changed, so an explicit restart is required or every check tests the previous config. - All curl checks must send browser-shaped headers; bare curl gets 429, which had the status command reporting a healthy instance as broken. Engine pruning is measured, not guessed: brave/duckduckgo/startpage CAPTCHA on 100% of queries, flickr is the 1.0s tail, and pinterest was failing because this host's own /etc/hosts blackholes it -- a local cause that would otherwise have been misdiagnosed as a dead upstream. Measured latency, reported honestly: front page median 19ms, but search median 486ms (233-623ms). Sub-200ms search is not achievable by any configuration -- a metasearch cannot return faster than the slowest upstream it queries. The ~200ms figure on searx.space measures initial front-page response, which is met. Image pinned by digest: the workarounds above are only known-true for 2026.8.4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cb77nxXGwFYFM4fyXEQkzG |
