kuhyx/testsAndMisc
1
0
Fork 0
mirror of https://github.com/kuhyx/testsAndMisc.git synced 2026-11-10 20:27:02 +01:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity

Prune 84 SearXNG engines; images category now sub-1s
Some checks are pending
Pre-commit checks / pre-commit (push) Waiting to run
Details
Python tests / test (push) Waiting to run
Details
Shell tests / shell-tests (push) Waiting to run
Details
Shell tests / arch-tests (push) Waiting to run
Details

Browse Source 
kuhy went through the /preferences engine table and selected every erroring
engine (CAPTCHA / access denied / timeout / parsing error / crash) plus the
slow tail, across all nine categories. Applied all of them, keeping only
google cse images for the images category as requested.

Result: unresponsive_engines is now [] in EVERY category. Per-category
latency (q=linux): general 572ms, images 406ms, videos 1378ms, news 455ms,
music 461ms, it 1698ms, science 1338ms, files 584ms, social media 1074ms.

Images went 2.1s -> 406-900ms across 12 queries, still 20 results. The root
cause was not google cse images being slow (1.1s): SearXNG waits for the
SLOWEST engine in a category, so quark images at 2.9s set the floor. Cutting
the others removed the ceiling.

Three engines that were not in the list surfaced as unresponsive only under
load and were pruned with the rest: brave.images, startpage images, kickass.

Corrects a wrong claim from the first pass. I had said general search retained
three working sources because !go returned 20 results and !bi returned 10.
Upstream marks both google and bing "inactive: true", so they answer ONLY to
an explicit bang and were never part of the merged view -- which google cse
has been serving alone all along. The pruning therefore removed only engines
already contributing zero, but general search does have a single point of
failure, so status now prints the !go/!bi fallback when a live search returns
nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cb77nxXGwFYFM4fyXEQkzG
This commit is contained in:
Krzysztof Rudnicki 2026-08-09 17:42:56 +02:00
parent 84d1c6e897
commit dc45779d38
3 changed files with 151 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.pre-commit-config.yaml
View File

@ -311,7 +311,10 @@ repos:
# English breaks the parsers that match on them.
# `daa` is MediaTek's Download Agent Authentication, a BROM security
# feature named in the mtk_root notes - not a misspelling of "data".
- --ignore-words-list=als,ans,ect,nd,som,sur,te,nam,numer,lew,sie,wil,postion,clen,ther,folow,derrive,ony,tje,noe,theses,crate,doubleclick,wile,tabel,pary,blok,bloc,proces,serwer,parametr,adres,hart,dout,metod,tekst,synonim,grup,mosty,lokal,skalar,milion,nowe,tre,hel,alph,iif,dokument,daa
# `artic` is SearXNG's engine id for the Art Institute of Chicago
# (setup_searxng.sh), not a misspelling of "arctic" - renaming it
# would stop matching the engine.
- --ignore-words-list=als,ans,ect,nd,som,sur,te,nam,numer,lew,sie,wil,postion,clen,ther,folow,derrive,ony,tje,noe,theses,crate,doubleclick,wile,tabel,pary,blok,bloc,proces,serwer,parametr,adres,hart,dout,metod,tekst,synonim,grup,mosty,lokal,skalar,milion,nowe,tre,hel,alph,iif,dokument,daa,artic
exclude: ^(Bash/ffmpeg-build/|LaTeX/|.*\.geojson$)
# ===========================================================================

15
docs/superpowers/evidence/setup-searxng-2026-08-09.json
View File

@ -61,6 +61,21 @@
"result": "pass",
"evidence": "All CSS/JS/img/manifest same-origin (/static/...). ZERO inline scripts (the single <script> has src= and passes config via a base64 client_settings attribute). ZERO inline style= attributes. Autocomplete is same-origin /autocompleter (200). JS bundle references zero absolute external origins. Image results use data-src=\"/image_proxy?url=...\" (780 occurrences, same-origin lazy-load) -- external URLs appear only as encoded query params, never as fetch targets, so img-src 'self' data: is safe."
},
{
"command": "engine pruning round 2 (kuhy-selected from /preferences, 9 groups, all categories)",
"result": "pass -- 84 engines disabled, 0 unresponsive in every category",
"evidence": "Disabled every erroring engine (CAPTCHA/access denied/timeout/parsing error/crash) plus the slow tail across general, images, videos, news, music, it, science, files and social media. Three engines NOT in kuhy's list surfaced as unresponsive only under load and were pruned with the rest: brave.images, startpage images, kickass. Post-prune per-category (q=linux): general 572ms/20, images 406ms/20, videos 1378ms/119, news 455ms/52, music 461ms/48, it 1698ms/114, science 1338ms/50, files 584ms/125, social media 1074ms/120 -- unresponsive_engines=[] for ALL nine."
},
{
"command": "images category under 1 second (kuhy's explicit ask)",
"result": "pass -- 406-900ms, was 2.1s",
"evidence": "12 queries measured post-prune: cat 900, dog 343, arch+linux 501, mountain 446, coffee 687, bridge 511, sunset 612, laptop 450, guitar 461, ocean 504, forest 624, city 450 ms -- all sub-1s, still 20 results each. ROOT CAUSE of the old 2.1s was NOT google cse images being slow (1.1s): SearXNG waits for the slowest engine in the category, so quark images at 2.9s set the floor. Removing the other image engines removed the ceiling, and also freed connection-pool contention."
},
{
"command": "general-search single-point-of-failure check (correcting an earlier wrong claim)",
"result": "documented in-file + guarded in status",
"evidence": "Initially asserted general search retained 'three working sources' (google cse + google + bing) because !go returned 20 and !bi returned 10. That was WRONG: upstream marks both google and bing 'inactive: true', so they answer ONLY to an explicit bang and were never in the merged view. The merged general view is served by google cse alone, and was before this pruning too -- so the pruning removed only zero-contribution engines, but general search does have a single point of failure. Comment corrected in settings.yml and status_cmd now prints the !go/!bi fallback if a live search returns 0 results."
},
{
"command": "real browser load + search (Thorium with the Claude extension)",
"result": "pass -- CSP confirmed by a rendering engine, closing the last gap",

145
linux_configuration/scripts/single_use/features/setup_searxng.sh
View File

@ -279,22 +279,132 @@ ui:
# Re-check periodically with:
# curl -s 'http://127.0.0.1:${SEARX_PORT}/search?q=test&format=json' \\
# | grep -o '"unresponsive_engines":.*'
# Engine pruning, kuhy-selected 2026-08-09 from the /preferences engine table:
# every engine that was erroring (CAPTCHA / access denied / timeout / parsing
# error / crash) plus the slow tail, across all categories.
#
# Retained on purpose: google cse, wikipedia/wikidata and the specialist
# lookups (arch linux wiki, github, stackexchange, pypi, ...).
#
# IMPORTANT about general search: the merged general view is served by
# google cse ALONE, and was before this pruning too. The plain google and bing
# engines are marked "inactive: true" upstream, meaning they only run when
# explicitly invoked with a bang (!go -> 20 results, !bi -> 10, both
# verified working). So the pruning removed only engines that were already
# contributing zero results -- but it also means general search has a single
# point of failure. If google cse ever starts erroring, general search returns
# nothing until another engine is enabled or !go/!bi is used.
#
# pinterest is blackholed to 0.0.0.0 by this host's own /etc/hosts
# (generate_hosts_file.sh) -- a LOCAL failure that looks identical to a dead
# upstream. Disabled to stop the pointless request, not because the engine is bad.
#
# Re-check periodically:
# curl -s 'http://127.0.0.1:8090/search?q=test&format=json' | grep -o '"unresponsive_engines":.*'
engines:
# --- CAPTCHA / actively blocking this IP (100% failure across every query) ---
- name: brave
disabled: true # "Suspended: too many requests" on every query
- name: duckduckgo
disabled: true # CAPTCHA on every query
- name: startpage
disabled: true # "Suspended: CAPTCHA" on every query
# === general ===
- { name: brave, disabled: true } # too many requests
- { name: duckduckgo, disabled: true } # CAPTCHA
- { name: startpage, disabled: true } # Suspended: CAPTCHA
- { name: dogpile, disabled: true } # access denied
- { name: encyclosearch, disabled: true } # timeout
- { name: fastbot, disabled: true } # access denied
- { name: fireball, disabled: true } # access denied
- { name: quark, disabled: true } # unexpected crash
- { name: qwant, disabled: true } # CAPTCHA
- { name: sogou, disabled: true } # CAPTCHA
- { name: tusksearch, disabled: true } # HTTP error
- { name: yahoo, disabled: true } # HTTP protocol error
- { name: yep, disabled: true } # access denied
- { name: naver, disabled: true } # 1.9s
- { name: baidu, disabled: true } # 1.7s
- { name: abcnyheter, disabled: true } # 1.5s
- { name: boardreader, disabled: true } # 1.5s
- { name: crowdview, disabled: true } # 1.0s
# --- blocked locally, not upstream ---
- name: pinterest
disabled: true # 0.0.0.0 in /etc/hosts on this host
# === images === (keeping ONLY google cse images)
- { name: dogpile images, disabled: true } # access denied
- { name: findfiles images, disabled: true } # unexpected crash
- { name: library of congress, disabled: true } # parsing error
- { name: pinterest, disabled: true } # 0.0.0.0 in /etc/hosts (local)
- { name: qwant images, disabled: true } # CAPTCHA
- { name: tusksearch images, disabled: true } # HTTP error
- { name: wikicommons.images, disabled: true } # too many requests
- { name: yandex images, disabled: true } # parsing error
- { name: quark images, disabled: true } # 2.9s
- { name: 1x, disabled: true } # 2.5s
- { name: baidu images, disabled: true } # 2.5s
- { name: picjumbo, disabled: true } # 2.4s
- { name: naver images, disabled: true } # 2.3s
- { name: sogou images, disabled: true } # 2.0s
- { name: unsplash, disabled: true } # 1.2s
- { name: duckduckgo images, disabled: true }
- { name: flickr, disabled: true } # 1.0s
- { name: bing images, disabled: true } # cut for the sub-1s images goal
- { name: deviantart, disabled: true } # cut for the sub-1s images goal
- { name: openverse, disabled: true } # cut for the sub-1s images goal
- { name: pexels, disabled: true } # cut for the sub-1s images goal
- { name: artic, disabled: true } # cut for the sub-1s images goal
- { name: devicons, disabled: true } # cut for the sub-1s images goal
- { name: lucide, disabled: true } # cut for the sub-1s images goal
- { name: wallhaven, disabled: true } # cut for the sub-1s images goal
# Not in the /preferences list kuhy sent -- these only surfaced as
# unresponsive under load, same failure pattern, so pruned with the rest.
- { name: brave.images, disabled: true } # too many requests
- { name: startpage images, disabled: true } # Suspended: CAPTCHA
# --- slow tail (>= 1.0s median, measured) ---
- name: flickr
disabled: true # 1.0s median, image-only, redundant with the rest
# === videos ===
- { name: 360search videos, disabled: true } # unexpected crash
- { name: acfun, disabled: true } # timeout
- { name: brave.videos, disabled: true } # too many requests
- { name: dogpile videos, disabled: true } # access denied
- { name: fireball videos, disabled: true } # access denied
- { name: niconico, disabled: true } # HTTP connection error
- { name: pixabay videos, disabled: true } # timeout
- { name: tusksearch videos, disabled: true } # HTTP error
- { name: vimeo, disabled: true } # access denied
- { name: youtube, disabled: true } # HTTP connection error
- { name: naver videos, disabled: true } # 2.4s
- { name: sogou videos, disabled: true } # 2.0s
- { name: iqiyi, disabled: true } # 1.3s
- { name: rumble, disabled: true } # 1.3s
# === news ===
- { name: bing news, disabled: true } # parsing error
- { name: brave.news, disabled: true } # too many requests
- { name: dogpile news, disabled: true } # access denied
- { name: fireball news, disabled: true } # access denied
- { name: google news, disabled: true } # Suspended: CAPTCHA
- { name: startpage news, disabled: true } # Suspended: CAPTCHA
- { name: tusksearch news, disabled: true } # HTTP error
- { name: sogou wechat, disabled: true } # 1.7s
- { name: naver news, disabled: true } # 1.7s
# === music ===
- { name: radio browser, disabled: true } # HTTP error
- { name: yandex music, disabled: true } # HTTP error
# === it ===
- { name: codeberg, disabled: true } # timeout
- { name: metacpan, disabled: true } # HTTP error
- { name: nixos wiki, disabled: true } # timeout
- { name: baidu kaifa, disabled: true } # 2.0s
- { name: gitea.com, disabled: true } # 1.8s
- { name: rubygems, disabled: true } # 1.1s
# === science ===
- { name: openairepublications, disabled: true } # 2.7s
- { name: openairedatasets, disabled: true } # 2.7s
# === files ===
- { name: 1337x, disabled: true } # HTTP connection error
- { name: btdigg, disabled: true } # 1.5s
- { name: findfiles, disabled: true } # 1.2s
- { name: kickass, disabled: true } # HTTP connection error (surfaced under load)
# === social media ===
- { name: 9gag, disabled: true } # access denied
- { name: tootfinder, disabled: true } # access denied
EOF
# 640, NOT 600. The entrypoint chowns this file to the container's searxng
# user (uid 977), but granian's worker runs as a different uid and only has
@ -737,6 +847,15 @@ status_cmd() {
dead="$(grep -oE '"unresponsive_engines": \[\[[^]]*' <<<"$json" | head -c 300 || true)"
[[ -n $dead ]] && log_info "unresponsive engines: ${dead#*: }"
# After pruning, general search is served by google cse alone (google/bing
# are inactive: true upstream and only answer to !go / !bi). That is a single
# point of failure worth naming explicitly rather than discovering as "no
# results", so check the engine that is actually carrying the category.
if [[ ${hits:-0} -eq 0 ]]; then
log_warn "general search has no working engine — try: !go <query> / !bi <query>,"
log_warn "or re-enable an engine in the 'engines:' block of ${SEARX_SETTINGS}"
fi
log_info "Not checked here (needs another machine): the JSON API under bot"
log_info "detection. Run from your phone on cellular, not from this host:"
log_info " curl 'https://${SEARX_DOMAIN}/search?q=test&format=json'"