signal-bot/DOCS-signal-envelopes.md
Krzysztof kuhy Rudnicki faeb756cdd feat: block binaries from a public repo, and write down the real attachment frame
Two halves of the same ask: the message database is private and must
stay that way.

The database was already gitignored, and the event table is pinned by a
test that asserts its exact column set so no message body can be quietly
added. What was missing was any defence against a binary reaching this
*public* repo at all -- and the proof it was missing was sitting in the
tree: three `bottles/**/indexeddb.sqlite` files, committed in "register
the bot as its own Signal account". Their contents turned out to be
harmless (one row, 120 bytes, a userPreferences store from
developer.android.com), but the path into history was real.

So the shared gate from ~/utils is wired in, the three files are
untracked, and .gitignore now blocks *.sqlite3 along with its -wal and
-shm companions rather than only the one filename.

The second half is the attachment envelope, captured from the live API
rather than assumed -- which is what 0.0.3 taught. A caption-less PNG
posted in the real group confirmed `attachments[].contentType`, and
settled something that was not guessable: a photo with no caption sends
`message: null`. That is exactly the frame `should_count` used to value
at zero.

Verified end to end on production data, not a fixture: the frame at
19:31:16 was recorded as media_kind=image, media_count=1, char_len=0,
filed under 2026-08-30 hour 19.

The doc also records the fields nothing reads yet, and one open question
left deliberately open: `caption` is a field on the attachment and was
null here, so whether a captioned photo puts its text in `message` or in
`attachments[].caption` has not been observed. If it is the latter,
char_len is recording 0 for those. DEBUG stays on until a `caption: str`
settles it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015d48t4qsQLBJnDwDeo2Nom
2026-08-30 19:32:41 +02:00

6.2 KiB

Signal envelope shapes

Reference for the two JSON frames signal-cli-rest-api pushes down ws://…/v1/receive/{number}. signal_bot/models.py flattens both into one IncomingMessage; these are the shapes it flattens.

From another account: dataMessage

{
  "envelope": {
    "source": "NAME",
    "sourceNumber": "PHONE_NUMBER",
    "sourceUuid": "SOURCE_ID",
    "sourceName": "USER_DEFINED_NAME",
    "sourceDevice": 69,
    "timestamp": 1719177917717,
    "dataMessage": {
      "timestamp": 1719177917717,
      "message": "MESSAGE_CONTENT",
      "expiresInSeconds": 0,
      "viewOnce": false,
      "sticker": { "packId": "166dd1b5…", "stickerId": 6 },
      "groupInfo": { "groupId": "ID_OF_GROUP", "type": "DELIVER" }
    }
  },
  "account": "BOT_ACCOUNT_PHONE_NUMBER"
}

A message with an attachment

Captured from the live API on 2026-08-30 with LOG_LEVEL=DEBUG, which makes the dispatcher log each frame's shape (keys and types, never values -- see signal_bot/envelope_shape.py). This is a caption-less PNG posted in the real group, reproduced exactly as the logger described it:

{account: str, envelope: {
  dataMessage: {
    attachments: [{caption: null, contentType: 'image/png', filename: null,
                   height: int, id: str, size: int, uploadTimestamp: int,
                   width: int} x1],
    expiresInSeconds: int,
    groupInfo: {groupId: str, groupName: str, revision: int, type: 'DELIVER'},
    isExpirationUpdate: bool,
    message: null,
    timestamp: int,
    viewOnce: bool},
  serverDeliveredTimestamp: int, serverReceivedTimestamp: int,
  source: str, sourceDevice: int, sourceName: str,
  sourceNumber: str, sourceUuid: str, timestamp: int}}

Three things this settles, none of which were guessable:

  • attachments is a list of objects and the type lives in contentType. That is what models._attachments reads and what counting.media_kind maps.
  • A caption-less photo sends message: null, not "" and not an absent key. Before 0.1.0 should_count ended in bool(message.text), so every one of these was silently worth nothing.
  • Read them off the resolved payload from _payload(), never off envelope["dataMessage"] -- see "The trap this caused" below.

Fields nothing reads yet, recorded so the next person does not have to capture them again: expiresInSeconds, isExpirationUpdate, viewOnce, groupInfo.groupName, groupInfo.revision, and the attachment's caption, filename, width, height, size and uploadTimestamp.

Open question, deliberately not assumed. caption is a field on the attachment, and it was null here because this photo had none. Whether a photo with a caption puts the text in message or in attachments[].caption has not been observed yet. char_len currently measures message only, so if it turns out to be the latter, captioned photos are recording a length of 0. Leave LOG_LEVEL=DEBUG on and watch for caption: str to settle it.

Somebody changed the membership: groupInfo.type is UPDATE

The only frame that names who added the bot to a group, which is what signal_bot/trust.py needs. It carries no message at all, so it counts toward nothing and answers nothing; source/sourceNumber is the person who made the change. Captured live on 2026-08-29:

{
  "envelope": {
    "source": "+48516501330",
    "sourceNumber": "+48516501330",
    "sourceUuid": "ffcae71a-…",
    "sourceName": "kuhy",
    "sourceDevice": 1,
    "timestamp": 1788006851503,
    "dataMessage": {
      "timestamp": 1788006851503,
      "message": null,
      "groupInfo": { "groupId": "6/gEwRkqs/…", "groupName": "testbot", "type": "UPDATE" }
    }
  },
  "account": "+420720355763"
}

Note source here is a phone number, not the display name this document used to show for dataMessage. Both spellings have been observed, which is why models.py reads the sender's number from sourceNumber and nothing trusts source.

From the bot's own account: syncMessage.sentMessage

When the account is deployed as a linked device, its own outgoing messages come back as sync messages. Same fields, one level deeper.

This deployment never receives them. The bot registers its own number as its own primary device and has no linked devices, so Signal sends it no copy of its own traffic. The consequence to know before you try to test anything: the bot cannot self-trigger a command. Posting !cat from the bot's number into a group does nothing whatsoever — a human has to type it, and journalctl --user -u signal-bot.service showing running command … is how you confirm it landed. The parsing below stays because the linked-device deployment is still supported, not because it fires here.

{
  "envelope": {
    "sourceUuid": "SOURCE_ID",
    "sourceName": "USER_DEFINED_NAME",
    "timestamp": 1719177728639,
    "syncMessage": {
      "sentMessage": {
        "destination": "DEST_NAME",
        "destinationNumber": "DEST_PHONE_NUMBER",
        "timestamp": 1719177728639,
        "message": "MESSAGE_CONTENT",
        "sticker": { "packId": "166dd1b5…", "stickerId": 6 },
        "groupInfo": { "groupId": "ID_OF_GROUP", "type": "DELIVER" }
      }
    }
  },
  "account": "BOT_ACCOUNT_PHONE_NUMBER"
}

The trap this caused

The original bot read envelope.dataMessage.sticker even when the frame was a syncMessage. Its own stickers therefore never matched the sticker filter and were counted as messages. Four consecutive commits (f1931b8, b325126, 62352e5, d35aa59) fought the symptom.

models.py resolves the payload first — dataMessage or syncMessage.sentMessage — and reads sticker, reaction, message and groupInfo off whichever it found. tests/test_models.py asserts both shapes.

No groupInfo means a direct message

A frame without groupInfo arrived in a one-to-one conversation. A linked device would receive every one of those on the account, so counting.watched_group rejects them explicitly rather than relying on a falsy comparison. See the scoping scenario in fake_signal/scenarios/.

Frames the bot ignores

Receipts, typing indicators and anything else with no dataMessage and no syncMessage.sentMessage parse to None and are dropped.