From 0f824c441e2a623a9030f9796a30334845652aa3 Mon Sep 17 00:00:00 2001 From: Krzysztof kuhy Rudnicki Date: Sun, 30 Aug 2026 19:32:41 +0200 Subject: [PATCH] feat: block binaries from a public repo, and write down the real attachment frame Two halves of the same ask: the message database is private and must stay that way. The database was already gitignored, and the event table is pinned by a test that asserts its exact column set so no message body can be quietly added. What was missing was any defence against a binary reaching this *public* repo at all -- and the proof it was missing was sitting in the tree: three `bottles/**/indexeddb.sqlite` files, committed in "register the bot as its own Signal account". Their contents turned out to be harmless (one row, 120 bytes, a userPreferences store from developer.android.com), but the path into history was real. So the shared gate from ~/utils is wired in, the three files are untracked, and .gitignore now blocks *.sqlite3 along with its -wal and -shm companions rather than only the one filename. The second half is the attachment envelope, captured from the live API rather than assumed -- which is what 0.0.3 taught. A caption-less PNG posted in the real group confirmed `attachments[].contentType`, and settled something that was not guessable: a photo with no caption sends `message: null`. That is exactly the frame `should_count` used to value at zero. Verified end to end on production data, not a fixture: the frame at 19:31:16 was recorded as media_kind=image, media_count=1, char_len=0, filed under 2026-08-30 hour 19. The doc also records the fields nothing reads yet, and one open question left deliberately open: `caption` is a field on the attachment and was null here, so whether a captioned photo puts its text in `message` or in `attachments[].caption` has not been observed. If it is the latter, char_len is recording 0 for those. DEBUG stays on until a `caption: str` settles it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015d48t4qsQLBJnDwDeo2Nom --- .gitignore | 9 ++++ .pre-commit-config.yaml | 10 ++++ DOCS-signal-envelopes.md | 46 ++++++++++++++++++ .../indexeddb.sqlite | Bin 4096 -> 0 bytes .../indexeddb.sqlite-shm | Bin 32768 -> 0 bytes .../indexeddb.sqlite-wal | Bin 119512 -> 0 bytes scripts/check_no_binaries.sh | 26 ++++++++++ 7 files changed, 91 insertions(+) delete mode 100644 bottles/1/7e946f8d-f1a3-40f9-8d1e-b9f4139ab051/indexeddb.sqlite delete mode 100644 bottles/1/7e946f8d-f1a3-40f9-8d1e-b9f4139ab051/indexeddb.sqlite-shm delete mode 100644 bottles/1/7e946f8d-f1a3-40f9-8d1e-b9f4139ab051/indexeddb.sqlite-wal create mode 100755 scripts/check_no_binaries.sh diff --git a/.gitignore b/.gitignore index cd0cf62..6063b50 100644 --- a/.gitignore +++ b/.gitignore @@ -166,3 +166,12 @@ cython_debug/ *.jpg *.gif nohup.out + +# The message database is private and never leaves this machine. A live +# SQLite database is three files, so all three are named. +*.sqlite3 +*.sqlite3-wal +*.sqlite3-shm + +# Wine/Bottles scratch dir, never part of the bot. +bottles/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 976358d..ae547ee 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -45,6 +45,16 @@ repos: language: pygrep types: [python] + # This repository is public and holds a private message database. The + # database is gitignored, but three binary SQLite files reached the + # history anyway before this gate existed, so .gitignore is evidently + # not the whole defence. Exceptions go in .binary-allowlist. + - id: no-binaries + name: no binary or image files + entry: scripts/check_no_binaries.sh + language: script + pass_filenames: true + # The shared 250-line cap. Checks only the files in the commit, so an # unrelated commit never breaks. No baseline, no allowlist -- those are # suppressions. The exemption list lives with the checker in ~/utils so diff --git a/DOCS-signal-envelopes.md b/DOCS-signal-envelopes.md index 3cdc780..c3f8a4d 100644 --- a/DOCS-signal-envelopes.md +++ b/DOCS-signal-envelopes.md @@ -28,6 +28,52 @@ Reference for the two JSON frames `signal-cli-rest-api` pushes down } ``` +## A message with an attachment + +Captured from the live API on 2026-08-30 with `LOG_LEVEL=DEBUG`, which makes +the dispatcher log each frame's shape (keys and types, never values -- see +`signal_bot/envelope_shape.py`). This is a caption-less PNG posted in the real +group, reproduced exactly as the logger described it: + +``` +{account: str, envelope: { + dataMessage: { + attachments: [{caption: null, contentType: 'image/png', filename: null, + height: int, id: str, size: int, uploadTimestamp: int, + width: int} x1], + expiresInSeconds: int, + groupInfo: {groupId: str, groupName: str, revision: int, type: 'DELIVER'}, + isExpirationUpdate: bool, + message: null, + timestamp: int, + viewOnce: bool}, + serverDeliveredTimestamp: int, serverReceivedTimestamp: int, + source: str, sourceDevice: int, sourceName: str, + sourceNumber: str, sourceUuid: str, timestamp: int}} +``` + +Three things this settles, none of which were guessable: + +- `attachments` is a list of objects and the type lives in `contentType`. That + is what `models._attachments` reads and what `counting.media_kind` maps. +- **A caption-less photo sends `message: null`**, not `""` and not an absent + key. Before 0.1.0 `should_count` ended in `bool(message.text)`, so every one + of these was silently worth nothing. +- Read them off the *resolved* payload from `_payload()`, never off + `envelope["dataMessage"]` -- see "The trap this caused" below. + +Fields nothing reads yet, recorded so the next person does not have to capture +them again: `expiresInSeconds`, `isExpirationUpdate`, `viewOnce`, +`groupInfo.groupName`, `groupInfo.revision`, and the attachment's `caption`, +`filename`, `width`, `height`, `size` and `uploadTimestamp`. + +**Open question, deliberately not assumed.** `caption` is a field on the +attachment, and it was `null` here because this photo had none. Whether a +photo *with* a caption puts the text in `message` or in `attachments[].caption` +has not been observed yet. `char_len` currently measures `message` only, so if +it turns out to be the latter, captioned photos are recording a length of 0. +Leave `LOG_LEVEL=DEBUG` on and watch for `caption: str` to settle it. + ## Somebody changed the membership: `groupInfo.type` is `UPDATE` The only frame that names *who* added the bot to a group, which is what diff --git a/bottles/1/7e946f8d-f1a3-40f9-8d1e-b9f4139ab051/indexeddb.sqlite b/bottles/1/7e946f8d-f1a3-40f9-8d1e-b9f4139ab051/indexeddb.sqlite deleted file mode 100644 index 4b95fc958807c7957018d03deb09746df8b1b7d9..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 4096 zcmWFz^vNtqRY=P(%1ta$FlG>7U}9o$P*7lCU|@t|AVoG{WYFsp;st3JAlr;ljiVtj n8UmvsFd71*Aut*OqaiRF0;3@?8UmvsFd71*Aut*O6ovo*OJ5CG6g$mYu@`~w7u3q%}&3(!zdLPg7YsHivtoPi1<(Na-x4X~w+v@OkM z#Xd{c*sHZ?<{iMZuSG0JCt`YdZIFXJG^VYwcaah&2L&0KRnpJ(-azGHc=y3ThR1PBly zK!5-N0t5&UAV7cs0RjXF5FkK+009C72oNAZfB*pk1PBlyK!5-N0t5&UAV7cs0RjXF z5FkK+009C72oNAZfB*pks}|^_w;~2HO0P$3uDTTY3Dhjmjh)!7Ij3|4njz4Oy=D}m z3IeqY6tQ1>P6-J#L!ciAaTv$VC`J_oY84npevQ!}&_;oA%;Kbtjq8~}dMo!R<{SvL zQedN%U2B>^7TCf-ptS)DMLH?iYy?AHca5P$##AOHafKmY;|fWTZJaCDKE%(b+5Uu_*R9j4Rt zn}XRVYr&@ub6}FsiM!2iv&C#Qd*wYV?;Fh~)8?;756zmDW}f5;{syGrq7-O9AC%9- zrpxsDsomqOH}>Yvonmhi2J&ziAb6+sqErE@@VX?(ny5 z)=}sG&bRV{S6{zx#eYtv>gMU5K$M_sED(SI1Rwwb2tWV=5P$##AOHafT%QCy?Ut1b zeDl?_W4Gh_`fd&VK$`xv}7g$&@^-W2!7ijwaf=jZ(0s#m>00Izz00bZafon@( z?56r;uC>)0TM=q9519R;i0czoT*2ghUFLpK%@sr?*Cp?j_cp|9IV&HyQi>1soU6vG zY&K8rO(b(0Hh9O^g^e2&-QlCsj4o-OHf}^Tio^0bZhKvPgW?V;RloL$deW**%Tra4 zDyJTK4OhsgN0zHJqvcJmmd*OAQSGSK*5!9s6~bDxt7NqRqooF)t1EEB& zaiZhGZ9||)&YK}=^N6UZ!!mU(XoD($wXLdp&imSPJriyzstVTr<|pZK*5BqbkE)Ni zifX;h?D19W-dlSbBr!7o-OtiVSL7y*4x0pN}11d9yBdpZ;h}>GvCZfyhA6 z!4N$)?f|$d_5yW!YOGvf#k!B@U)l8bYPIplUO*wRKmY;|fB*y_009U<00Izz00gdU z0=}-_$_2J9UDDICe(MML5nR{joX$Z20uX=z1Rwwb2tWV=5P$##ECKZiXs~jDjW3Zmw~jiIhGDa<=dn76cgsst>%pFlb=h^DPy=6PTTXeZalf^_ zc2Be|ZdqY(ZSK-^DrI#^tpmR6`>=RX2Sl=Fx%gZ^W?CXWv-K`_LVMeqVI) zUbX3d+&U+ZFkLz1uv=>ocRC3D?4IRLQOXR8*0&&f*I-&?cW<;uuKu?;KDlw9D0G6z z-9hwgJ#)66sgNGxqdHBl8jH#1nSdxM`)^_f?sXz@Hz`+6z4R z>)+h`tBZI4k@x~L|5pwzga8B}009U<00Izz00bZa0SHV30TtP#tX$yGk<86MdEv(Q z@gtZ9vJ?RU2tWV=5P$##AOHafKmY;|nE3=s{0KgL_N^xl-F5DvDt-iMr&~IoLCt;y zDX$puBajVqZ8_GRO6G3A-8)$b&6`~tN*jG=vTf;j^X191N6nS3*{BVS)M~byy=IH~ zg1mN%eam*ULq2QTZrbF}HHuQxt9~&^Ey(^&vTujHS3aWD2-$WkKOP3Cwo*#XvelHi zwm#(jFic&3pWGg~J>DryTFt#?qofZ=P4@^TO;`HilL7r97tQFB# zA-P*hZk4mJy<#mIG)Z%A_0QH8$$hu1?J29Rl#xB>+Uk8`{_FNIzV6cYmzKCs%9u(I zucXa(bWCR$Df|fJG~q`undx5CY%lye)@BnLe3=95RQw3~%!nU^%Jo&Tm&-~Q7JI3} zkAS#?_V#Ggxa-(--Y|$eNZdg?-oiBgUSadpcF(AEw{RcQ>{T@IpeptGE01le*V%4) ztR0mH+!49|ZmpCS;gchN1YM?APRtDTBhcUY zX=^X=^vJHQ=ei1C#9m;o2`A_;1Rwwb2tWV=5P$##AOHaf%uxcK_S?z@_7$If@51xX ztf^uzFyCRN^BL4+FQA47UgnY(H9vEyDVaK>aRL9fb17 z|Ig+DdjS=8V=piw^6g-TozULqN>rM}9rR~n(?O)*R6pD!uX15{xsJkWS%^D0xs^ho z7qamHEv>A~F-P1%oeC@bq7rv796`h#B<`SGhJ(9V+#QU#gL1#pzd%=CNr4+)cjND1 zHV^g!6Higt3+SMjP4g5Q)1<%YebWFd7x>e>&%F8a@|PdLUSJN%CFm*yAOHafKmY;| zfB*y_009WhWdcpwc`Fy#w&>4?|G9Jdwgoa_U}M=Ff^$%pe>Fx1_5#=o5MMxKs-5Oh z83}=!hxh{HnhdLhh^Dn_<>UNvt=FQKTk5mu`ZkuHq;g{&ZDp-3!(JfN_PQD;S0nC@ zf|+J7AclTNnWr$kgq43gqTh__n#bC|6W9x2FMz#3ctx!|mo4@J{StpbXPYIyK;*%z z`bB#3N2=1treS$3>hT|q^cG%yxJ7GqOS>xb6sk|ad@C2&{Oz857W`=aOFBjjt;+txOV14=xA`^d+=wn^;cc0c0uZduc9n+@tK)K{plu7dh%%p@iBW3GL=LO;3} zzIe9|_kO5z6ei>~As6`hlZ!8Z>d1#X)sMi-e8zHAt)R(hjy zfeAl?d{58z8`k__x5@<)nV*SVK;Bp&009U<00Izz00bZa0SG_<0uY#C1a3~$wa5P4 zcjp|aCKpI*$E+X0@7E5WDgNbOivxRs3np`6hIO0fKmY;|fB*y_009U<00Izz00bZ~ zbpkhfNt0f-G@TldnF)##YhWV2fX-Do;YaXh?)>v7-gxl0L41LWCUbG>RdECaAOHaf zKmY;|fB*y_009U<00J|YKvUha^n`RE;zw}wrG}qge&qf6fgiz(>PPV6%A9-;JOhcrsPL(ar3UB&3j&16vzeM5I=&<8*`K} z-G%@JAOHafKmY;|fB*y_009WhHUdi%;w_N7DU~`X-U9t*RQ&}S4tDg97Ov2b;KZRP z)<636Ym3#7Ad&f-_z}n(3j`nl0SG_<0uX=z1Rwwb2tWV=vzNfax||mZ3K9`Nf=21s zM0|lyp8DNgO%H#sA&4(fG@0V;?KUlk00bZa0SG_<0uX=z1Rwwb2+Sw~V>dLUKRtGM zZLxE(_h4sl_m}n`=y7zxy`%#jMbY^ivt;~pf5$-2 sXh%;WKQvtEE({k2y8PWb+n}G{Hx7{eQl9_0FrY^V%MvNiu8*hx0~812jsO4v diff --git a/scripts/check_no_binaries.sh b/scripts/check_no_binaries.sh new file mode 100755 index 0000000..13ecefd --- /dev/null +++ b/scripts/check_no_binaries.sh @@ -0,0 +1,26 @@ +#!/bin/bash + +# ============================================================================ +# Thin delegate to the shared gate in ~/utils. +# +# Copying the logic here is what lets one repo's idea of the rule drift from +# every other repo's, so this script only locates the shared checker and +# forwards its arguments. +# ============================================================================ + +set -euo pipefail + +readonly SHARED_GATE="${UTILS_ROOT:-$HOME/utils}/scripts/check_no_binaries.sh" + +main() { + if [[ ! -x "$SHARED_GATE" ]]; then + echo "Error: shared gate not found at $SHARED_GATE" >&2 + echo " Clone github.com/kuhyx/utils to ~/utils, or set" >&2 + echo " UTILS_ROOT to where it lives." >&2 + exit 1 + fi + + exec bash "$SHARED_GATE" "$@" +} + +main "$@"