From dc45779d382b76337f2596ce6344979148035775 Mon Sep 17 00:00:00 2001
From: Krzysztof kuhy Rudnicki <krzysztofrudnicki0@gmail.com>
Date: Sun, 9 Aug 2026 17:42:56 +0200
Subject: [PATCH] Prune 84 SearXNG engines; images category now sub-1s

kuhy went through the /preferences engine table and selected every erroring
engine (CAPTCHA / access denied / timeout / parsing error / crash) plus the
slow tail, across all nine categories. Applied all of them, keeping only
google cse images for the images category as requested.

Result: unresponsive_engines is now [] in EVERY category. Per-category
latency (q=linux): general 572ms, images 406ms, videos 1378ms, news 455ms,
music 461ms, it 1698ms, science 1338ms, files 584ms, social media 1074ms.

Images went 2.1s -> 406-900ms across 12 queries, still 20 results. The root
cause was not google cse images being slow (1.1s): SearXNG waits for the
SLOWEST engine in a category, so quark images at 2.9s set the floor. Cutting
the others removed the ceiling.

Three engines that were not in the list surfaced as unresponsive only under
load and were pruned with the rest: brave.images, startpage images, kickass.

Corrects a wrong claim from the first pass. I had said general search retained
three working sources because !go returned 20 results and !bi returned 10.
Upstream marks both google and bing "inactive: true", so they answer ONLY to
an explicit bang and were never part of the merged view -- which google cse
has been serving alone all along. The pruning therefore removed only engines
already contributing zero, but general search does have a single point of
failure, so status now prints the !go/!bi fallback when a live search returns
nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cb77nxXGwFYFM4fyXEQkzG
---
 .pre-commit-config.yaml                       |   5 +-
 .../evidence/setup-searxng-2026-08-09.json    |  15 ++
 .../single_use/features/setup_searxng.sh      | 145 ++++++++++++++++--
 3 files changed, 151 insertions(+), 14 deletions(-)

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 2342f850..d5a7fc8f 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -311,7 +311,10 @@ repos:
           # English breaks the parsers that match on them.
           # `daa` is MediaTek's Download Agent Authentication, a BROM security
           # feature named in the mtk_root notes - not a misspelling of "data".
-          - --ignore-words-list=als,ans,ect,nd,som,sur,te,nam,numer,lew,sie,wil,postion,clen,ther,folow,derrive,ony,tje,noe,theses,crate,doubleclick,wile,tabel,pary,blok,bloc,proces,serwer,parametr,adres,hart,dout,metod,tekst,synonim,grup,mosty,lokal,skalar,milion,nowe,tre,hel,alph,iif,dokument,daa
+          # `artic` is SearXNG's engine id for the Art Institute of Chicago
+          # (setup_searxng.sh), not a misspelling of "arctic" - renaming it
+          # would stop matching the engine.
+          - --ignore-words-list=als,ans,ect,nd,som,sur,te,nam,numer,lew,sie,wil,postion,clen,ther,folow,derrive,ony,tje,noe,theses,crate,doubleclick,wile,tabel,pary,blok,bloc,proces,serwer,parametr,adres,hart,dout,metod,tekst,synonim,grup,mosty,lokal,skalar,milion,nowe,tre,hel,alph,iif,dokument,daa,artic
         exclude: ^(Bash/ffmpeg-build/|LaTeX/|.*\.geojson$)
 
   # ===========================================================================
diff --git a/docs/superpowers/evidence/setup-searxng-2026-08-09.json b/docs/superpowers/evidence/setup-searxng-2026-08-09.json
index 6d2b32cc..9c6d668f 100644
--- a/docs/superpowers/evidence/setup-searxng-2026-08-09.json
+++ b/docs/superpowers/evidence/setup-searxng-2026-08-09.json
@@ -61,6 +61,21 @@
       "result": "pass",
       "evidence": "All CSS/JS/img/manifest same-origin (/static/...). ZERO inline scripts (the single <script> has src= and passes config via a base64 client_settings attribute). ZERO inline style= attributes. Autocomplete is same-origin /autocompleter (200). JS bundle references zero absolute external origins. Image results use data-src=\"/image_proxy?url=...\" (780 occurrences, same-origin lazy-load) -- external URLs appear only as encoded query params, never as fetch targets, so img-src 'self' data: is safe."
     },
+    {
+      "command": "engine pruning round 2 (kuhy-selected from /preferences, 9 groups, all categories)",
+      "result": "pass -- 84 engines disabled, 0 unresponsive in every category",
+      "evidence": "Disabled every erroring engine (CAPTCHA/access denied/timeout/parsing error/crash) plus the slow tail across general, images, videos, news, music, it, science, files and social media. Three engines NOT in kuhy's list surfaced as unresponsive only under load and were pruned with the rest: brave.images, startpage images, kickass. Post-prune per-category (q=linux): general 572ms/20, images 406ms/20, videos 1378ms/119, news 455ms/52, music 461ms/48, it 1698ms/114, science 1338ms/50, files 584ms/125, social media 1074ms/120 -- unresponsive_engines=[] for ALL nine."
+    },
+    {
+      "command": "images category under 1 second (kuhy's explicit ask)",
+      "result": "pass -- 406-900ms, was 2.1s",
+      "evidence": "12 queries measured post-prune: cat 900, dog 343, arch+linux 501, mountain 446, coffee 687, bridge 511, sunset 612, laptop 450, guitar 461, ocean 504, forest 624, city 450 ms -- all sub-1s, still 20 results each. ROOT CAUSE of the old 2.1s was NOT google cse images being slow (1.1s): SearXNG waits for the slowest engine in the category, so quark images at 2.9s set the floor. Removing the other image engines removed the ceiling, and also freed connection-pool contention."
+    },
+    {
+      "command": "general-search single-point-of-failure check (correcting an earlier wrong claim)",
+      "result": "documented in-file + guarded in status",
+      "evidence": "Initially asserted general search retained 'three working sources' (google cse + google + bing) because !go returned 20 and !bi returned 10. That was WRONG: upstream marks both google and bing 'inactive: true', so they answer ONLY to an explicit bang and were never in the merged view. The merged general view is served by google cse alone, and was before this pruning too -- so the pruning removed only zero-contribution engines, but general search does have a single point of failure. Comment corrected in settings.yml and status_cmd now prints the !go/!bi fallback if a live search returns 0 results."
+    },
     {
       "command": "real browser load + search (Thorium with the Claude extension)",
       "result": "pass -- CSP confirmed by a rendering engine, closing the last gap",
diff --git a/linux_configuration/scripts/single_use/features/setup_searxng.sh b/linux_configuration/scripts/single_use/features/setup_searxng.sh
index 31c46b6f..ef84367d 100755
--- a/linux_configuration/scripts/single_use/features/setup_searxng.sh
+++ b/linux_configuration/scripts/single_use/features/setup_searxng.sh
@@ -279,22 +279,132 @@ ui:
 # Re-check periodically with:
 #   curl -s 'http://127.0.0.1:${SEARX_PORT}/search?q=test&format=json' \\
 #     | grep -o '"unresponsive_engines":.*'
+# Engine pruning, kuhy-selected 2026-08-09 from the /preferences engine table:
+# every engine that was erroring (CAPTCHA / access denied / timeout / parsing
+# error / crash) plus the slow tail, across all categories.
+#
+# Retained on purpose: google cse, wikipedia/wikidata and the specialist
+# lookups (arch linux wiki, github, stackexchange, pypi, ...).
+#
+# IMPORTANT about general search: the merged general view is served by
+# google cse ALONE, and was before this pruning too. The plain google and bing
+# engines are marked "inactive: true" upstream, meaning they only run when
+# explicitly invoked with a bang (!go -> 20 results, !bi -> 10, both
+# verified working). So the pruning removed only engines that were already
+# contributing zero results -- but it also means general search has a single
+# point of failure. If google cse ever starts erroring, general search returns
+# nothing until another engine is enabled or !go/!bi is used.
+#
+# pinterest is blackholed to 0.0.0.0 by this host's own /etc/hosts
+# (generate_hosts_file.sh) -- a LOCAL failure that looks identical to a dead
+# upstream. Disabled to stop the pointless request, not because the engine is bad.
+#
+# Re-check periodically:
+#   curl -s 'http://127.0.0.1:8090/search?q=test&format=json' | grep -o '"unresponsive_engines":.*'
 engines:
-  # --- CAPTCHA / actively blocking this IP (100% failure across every query) ---
-  - name: brave
-    disabled: true          # "Suspended: too many requests" on every query
-  - name: duckduckgo
-    disabled: true          # CAPTCHA on every query
-  - name: startpage
-    disabled: true          # "Suspended: CAPTCHA" on every query
+  # === general ===
+  - { name: brave, disabled: true }             # too many requests
+  - { name: duckduckgo, disabled: true }        # CAPTCHA
+  - { name: startpage, disabled: true }         # Suspended: CAPTCHA
+  - { name: dogpile, disabled: true }           # access denied
+  - { name: encyclosearch, disabled: true }     # timeout
+  - { name: fastbot, disabled: true }           # access denied
+  - { name: fireball, disabled: true }          # access denied
+  - { name: quark, disabled: true }             # unexpected crash
+  - { name: qwant, disabled: true }             # CAPTCHA
+  - { name: sogou, disabled: true }             # CAPTCHA
+  - { name: tusksearch, disabled: true }        # HTTP error
+  - { name: yahoo, disabled: true }             # HTTP protocol error
+  - { name: yep, disabled: true }               # access denied
+  - { name: naver, disabled: true }             # 1.9s
+  - { name: baidu, disabled: true }             # 1.7s
+  - { name: abcnyheter, disabled: true }        # 1.5s
+  - { name: boardreader, disabled: true }       # 1.5s
+  - { name: crowdview, disabled: true }         # 1.0s
 
-  # --- blocked locally, not upstream ---
-  - name: pinterest
-    disabled: true          # 0.0.0.0 in /etc/hosts on this host
+  # === images === (keeping ONLY google cse images)
+  - { name: dogpile images, disabled: true }    # access denied
+  - { name: findfiles images, disabled: true }  # unexpected crash
+  - { name: library of congress, disabled: true } # parsing error
+  - { name: pinterest, disabled: true }         # 0.0.0.0 in /etc/hosts (local)
+  - { name: qwant images, disabled: true }      # CAPTCHA
+  - { name: tusksearch images, disabled: true } # HTTP error
+  - { name: wikicommons.images, disabled: true } # too many requests
+  - { name: yandex images, disabled: true }     # parsing error
+  - { name: quark images, disabled: true }      # 2.9s
+  - { name: 1x, disabled: true }                # 2.5s
+  - { name: baidu images, disabled: true }      # 2.5s
+  - { name: picjumbo, disabled: true }          # 2.4s
+  - { name: naver images, disabled: true }      # 2.3s
+  - { name: sogou images, disabled: true }      # 2.0s
+  - { name: unsplash, disabled: true }          # 1.2s
+  - { name: duckduckgo images, disabled: true }
+  - { name: flickr, disabled: true }            # 1.0s
+  - { name: bing images, disabled: true }       # cut for the sub-1s images goal
+  - { name: deviantart, disabled: true }        # cut for the sub-1s images goal
+  - { name: openverse, disabled: true }         # cut for the sub-1s images goal
+  - { name: pexels, disabled: true }            # cut for the sub-1s images goal
+  - { name: artic, disabled: true }             # cut for the sub-1s images goal
+  - { name: devicons, disabled: true }          # cut for the sub-1s images goal
+  - { name: lucide, disabled: true }            # cut for the sub-1s images goal
+  - { name: wallhaven, disabled: true }         # cut for the sub-1s images goal
+  # Not in the /preferences list kuhy sent -- these only surfaced as
+  # unresponsive under load, same failure pattern, so pruned with the rest.
+  - { name: brave.images, disabled: true }      # too many requests
+  - { name: startpage images, disabled: true }  # Suspended: CAPTCHA
 
-  # --- slow tail (>= 1.0s median, measured) ---
-  - name: flickr
-    disabled: true          # 1.0s median, image-only, redundant with the rest
+  # === videos ===
+  - { name: 360search videos, disabled: true }  # unexpected crash
+  - { name: acfun, disabled: true }             # timeout
+  - { name: brave.videos, disabled: true }      # too many requests
+  - { name: dogpile videos, disabled: true }    # access denied
+  - { name: fireball videos, disabled: true }   # access denied
+  - { name: niconico, disabled: true }          # HTTP connection error
+  - { name: pixabay videos, disabled: true }    # timeout
+  - { name: tusksearch videos, disabled: true } # HTTP error
+  - { name: vimeo, disabled: true }             # access denied
+  - { name: youtube, disabled: true }           # HTTP connection error
+  - { name: naver videos, disabled: true }      # 2.4s
+  - { name: sogou videos, disabled: true }      # 2.0s
+  - { name: iqiyi, disabled: true }             # 1.3s
+  - { name: rumble, disabled: true }            # 1.3s
+
+  # === news ===
+  - { name: bing news, disabled: true }         # parsing error
+  - { name: brave.news, disabled: true }        # too many requests
+  - { name: dogpile news, disabled: true }      # access denied
+  - { name: fireball news, disabled: true }     # access denied
+  - { name: google news, disabled: true }       # Suspended: CAPTCHA
+  - { name: startpage news, disabled: true }    # Suspended: CAPTCHA
+  - { name: tusksearch news, disabled: true }   # HTTP error
+  - { name: sogou wechat, disabled: true }      # 1.7s
+  - { name: naver news, disabled: true }        # 1.7s
+
+  # === music ===
+  - { name: radio browser, disabled: true }     # HTTP error
+  - { name: yandex music, disabled: true }      # HTTP error
+
+  # === it ===
+  - { name: codeberg, disabled: true }          # timeout
+  - { name: metacpan, disabled: true }          # HTTP error
+  - { name: nixos wiki, disabled: true }        # timeout
+  - { name: baidu kaifa, disabled: true }       # 2.0s
+  - { name: gitea.com, disabled: true }         # 1.8s
+  - { name: rubygems, disabled: true }          # 1.1s
+
+  # === science ===
+  - { name: openairepublications, disabled: true } # 2.7s
+  - { name: openairedatasets, disabled: true }  # 2.7s
+
+  # === files ===
+  - { name: 1337x, disabled: true }             # HTTP connection error
+  - { name: btdigg, disabled: true }            # 1.5s
+  - { name: findfiles, disabled: true }         # 1.2s
+  - { name: kickass, disabled: true }           # HTTP connection error (surfaced under load)
+
+  # === social media ===
+  - { name: 9gag, disabled: true }              # access denied
+  - { name: tootfinder, disabled: true }        # access denied
 EOF
 	# 640, NOT 600. The entrypoint chowns this file to the container's searxng
 	# user (uid 977), but granian's worker runs as a different uid and only has
@@ -737,6 +847,15 @@ status_cmd() {
 	dead="$(grep -oE '"unresponsive_engines": \[\[[^]]*' <<<"$json" | head -c 300 || true)"
 	[[ -n $dead ]] && log_info "unresponsive engines: ${dead#*: }"
 
+	# After pruning, general search is served by google cse alone (google/bing
+	# are inactive: true upstream and only answer to !go / !bi). That is a single
+	# point of failure worth naming explicitly rather than discovering as "no
+	# results", so check the engine that is actually carrying the category.
+	if [[ ${hits:-0} -eq 0 ]]; then
+		log_warn "general search has no working engine — try: !go <query> / !bi <query>,"
+		log_warn "or re-enable an engine in the 'engines:' block of ${SEARX_SETTINGS}"
+	fi
+
 	log_info "Not checked here (needs another machine): the JSON API under bot"
 	log_info "detection. Run from your phone on cellular, not from this host:"
 	log_info "  curl 'https://${SEARX_DOMAIN}/search?q=test&format=json'"