Phase 5: IMAP Bounce Processing

Overview

When the downstream MTA cannot deliver a message it returns a Delivery Status Notification (DSN). This phase closes that loop automatically: bounces are parsed, the message row is flipped to bounced with a transient/permanent classification (RFC 3463), and — for permanent failures — the recipient is inserted into the per-mailbox suppressions table. Future send attempts to a suppressed address are rejected at the API layer with 422 before any SMTP connection is made. The entire pipeline is VERP-authenticated: a forged DSN cannot flip a legitimate message’s status, and unsigned bounces are silently quarantined.

Design ADR: docs/adr/0002-phase5-bounce-processing.md is the binding design record. This spec is the implementor-facing checklist; where the two disagree, the ADR wins.

Goal

Flip outbound_messages rows to bounced when the downstream MTA returns a Delivery Status Notification (DSN), classify the bounce as transient vs. permanent (RFC 3463), and maintain a per-mailbox suppressions table so recipients with permanent failures are blocked from future /messages/send requests. The bounce identification path is VERP-authenticated end-to-end — no header-heuristic trust, no subject-keyword matching.

Process topology

The bounce processor is a separate, synchronous, one-shot CLI command (cli bounces:poll-once), scheduled by an external periodic mechanism (Docker Swarm periodic task / system cron) at an initial cadence of every 5 minutes. It is not a long-running scheduler and it is not a task inside the async outbound worker. This honors the CONTRIBUTING.md §IMAP Parsing sync-def-for-imaplib invariant without thread-pool carve-outs. See ADR 0002 §2.

TDD Acceptance Criteria

All must fail before implementation and pass after, in listed order:

  1. pytest tests/services/test_verp.py::test_sign_verp_round_trips
  2. pytest tests/services/test_verp.py::test_verify_rejects_unsigned_and_tampered
  3. pytest tests/services/test_bounce_parser.py::test_extracts_original_message_id_from_dsn
  4. pytest tests/services/test_bounce_parser.py::test_classifies_status_4xx_as_transient
  5. pytest tests/services/test_bounce_parser.py::test_classifies_status_5xx_as_permanent
  6. pytest tests/services/test_bounce_parser.py::test_classifies_missing_status_as_unknown
  7. pytest tests/workers/test_bounce_cron.py::test_verp_authenticated_bounce_flips_row_to_bounced_permanent
  8. pytest tests/workers/test_bounce_cron.py::test_transient_bounce_does_not_insert_suppression
  9. pytest tests/workers/test_bounce_cron.py::test_permanent_bounce_inserts_suppression_row
  10. pytest tests/workers/test_bounce_cron.py::test_forged_bounce_with_valid_message_id_is_rejected
  11. pytest tests/workers/test_bounce_cron.py::test_non_multipart_report_email_is_rejected
  12. pytest tests/workers/test_bounce_cron.py::test_bounce_email_is_moved_to_processed_folder_on_success
  13. pytest tests/workers/test_bounce_cron.py::test_bounce_email_is_moved_to_rejected_folder_on_hmac_failure
  14. pytest tests/api/test_outbound_send.py::test_422_rejects_suppressed_recipient
  15. pytest tests/commands/test_suppression_commands.py::test_cli_remove_clears_suppression_row
  16. pytest tests/workers/test_smtp_delivery.py::test_return_path_uses_verp_when_bounce_imap_configured
  17. pytest tests/workers/test_smtp_delivery.py::test_return_path_falls_back_to_mailbox_email_when_bounce_imap_unconfigured
  18. pytest tests/test_alembic.py::test_single_head (still green after adding 0004_bounce_imap_and_verp)

Technical Specifications

VERP addressing

  • New module src/app/utils/verp.py:
    • sign_verp(message_id: UUID) -> str returns bounce+{message_id}.{hmac16}@placeholder template; the caller substitutes the domain. Prefer a build_verp_return_path(mailbox, message_id) -> str helper that inspects mailbox.bounce_verp_domain (falling back to mailbox.email.split('@', 1)[1]) and returns the fully qualified Return-Path.
    • verify_verp(local_part: str) -> UUID | None — constant-time HMAC comparison. Returns the message_id on success, None on any of: missing bounce+ prefix, malformed, bad HMAC, wrong length. Never raises.
  • HMAC input: b"bounce-verp:" + str(message_id).encode() signed with the existing TRACKING_HMAC_SECRET. The domain-separation prefix prevents a signed tracking token from being reinterpretable as a VERP token.
  • HMAC output: SHA-256, hex, first 16 chars (64 bits). Matches the existing sign_message_id precedent in src/app/utils/tracking_token.py.
  • Worker change (src/app/workers/smtp_delivery.py, _bind_sender_headers): when mailbox.bounce_imap_host is set, Return-Path and the SMTP MAIL FROM envelope sender become the VERP address. When it is NULL, both stay as <{mailbox.email}> (current behavior). From / Sender / Reply-To remain pinned to mailbox.email regardless — VERP does NOT weaken the sender-binding invariant (ADR 0001 §4, FEEDBACK.md §1.4).

Schema — Alembic revision 0004_bounce_imap_and_verp

On mailboxes:

  • bounce_imap_host text NULL
  • bounce_imap_port int NULL (DB default 993 documented; no server default — NULL means “feature not configured”)
  • bounce_imap_username text NULL
  • bounce_imap_password_encrypted bytea NULL
  • bounce_imap_tls_mode mailboxtlsmode NULL (reuses the enum from 0003_smtp_tls_mode; default on the model layer is implicit because bounce IMAP is almost always 993)
  • bounce_imap_folder text NULL (default string 'INBOX' at the model layer)
  • bounce_verp_domain text NULL (explicit override; worker falls back to mailbox.email domain when NULL)

New enum bouncetype(transient, permanent, unknown).

On outbound_messages:

  • bounce_type bouncetype NULL
  • bounce_diagnostic text NULL (scrubbed via redact_pii before persist, same rules as error_log)
  • bounced_at timestamptz NULL

New enum suppressionreason(hard_bounce, unsubscribe, complaint). Phase 5 only writes hard_bounce; the other two are reserved for §1.10 pre-GA.

New table suppressions:

CREATE TABLE suppressions (
  id                 uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  mailbox_id         uuid NOT NULL REFERENCES mailboxes(id) ON DELETE CASCADE,
  recipient_email    citext NOT NULL,
  reason             suppressionreason NOT NULL,
  source_message_id  uuid NULL REFERENCES outbound_messages(id) ON DELETE SET NULL,
  created_at         timestamptz NOT NULL DEFAULT now(),
  notes              text NULL,
  UNIQUE (mailbox_id, recipient_email)
);
CREATE INDEX suppressions_mailbox_idx ON suppressions(mailbox_id);

If citext is not already enabled, the migration adds CREATE EXTENSION IF NOT EXISTS citext; before the table. The email-address comparison must be case-insensitive (ALICE@EX.COM == alice@ex.com).

Guard: tests/test_alembic.py still asserts a single head after 0004 is merged.

Bounce parser (src/app/services/bounce_parser.py, sync)

  • Input: raw .eml bytes (or a pathlib.Path to one, for tests).
  • parse(bytes) -> ParsedBounce | None where ParsedBounce carries:
    • recipient_local_part (the bounce+{token} piece of the envelope To:, pulled from the IMAP-delivered message’s Delivered-To: / Envelope-To: / Return-Path: — IMAP servers vary; the parser tries all three in order and returns the first valid VERP match; no header-only heuristic match)
    • verp_message_id: UUID (HMAC-verified via verp.verify_verp; if verification fails, parse returns None)
    • bounce_type: Literal['transient', 'permanent', 'unknown']
    • diagnostic_raw: str (the Diagnostic-Code: line from the message/delivery-status part, before redact_pii; the caller runs redact_pii before persisting to bounce_diagnostic)
    • original_recipient_email: str (from Final-Recipient: / Original-Recipient: in the message/delivery-status part — this is what goes into suppressions.recipient_email for permanent bounces)
  • Requires the message have Content-Type: multipart/report; report-type=delivery-status. Anything else → None.
  • Classification: Status: line in the delivery-status part, first digit: 4 → transient, 5 → permanent, other / missing → unknown.

Bounce cron worker (src/app/workers/bounce_cron.py, sync)

  • Entry point poll_once() called by cli bounces:poll-once.
  • For each mailbox with bounce_imap_host IS NOT NULL:
    1. Build a sync IMAPClient (reuse existing helpers in src/app/services/imap_*) against bounce_imap_host:bounce_imap_port, TLS per bounce_imap_tls_mode, creds decrypted from bounce_imap_password_encrypted.
    2. SELECT bounce_imap_folder, search UNSEEN.
    3. For each message UID:
      • Fetch full RFC822 (BODY.PEEK[] — small, DSNs are tiny).
      • parsed = bounce_parser.parse(raw).
      • If parsed is None → move to Rejected-Bounces folder (create if missing), increment bounce_cron_rejected_total{reason=malformed|hmac} metric, continue.
      • Look up outbound_messages by id = parsed.verp_message_id. If absent → move to Rejected-Bounces, metric bounce_cron_rejected_total{reason=unknown_message_id}, continue. (A VERP token whose message_id does not match any row is either a pre-retention DSN or a forgery that happened to guess a valid HMAC — both are non-events.)
      • Authoritative bounce. In one DB transaction:
        • UPDATE outbound_messages SET status='bounced', bounce_type=$1, bounce_diagnostic=redact_pii($2), bounced_at=now() WHERE id=$msg.
        • If bounce_type == 'permanent': INSERT INTO suppressions (mailbox_id, recipient_email, reason, source_message_id) VALUES ($mbx, $rcpt, 'hard_bounce', $msg) ON CONFLICT (mailbox_id, recipient_email) DO NOTHING.
      • On commit success: move the IMAP message to Processed-Bounces (create if missing), mark \\Seen.
      • On any DB error: leave the message unprocessed (UNSEEN), next poll retries. Do NOT move to Processed- or Rejected- until the DB transaction commits.
    4. LOGOUT inside a try / finally (existing IMAP discipline).
  • Exit code 0 on clean pass; non-zero on per-mailbox exception (caught, logged with redacted stderr line per mailbox, but bubbled up so the external scheduler alarms).
  • DB session: sync SQLAlchemy engine. Factor src/app/db/sync_session.py alongside the existing async session factory, reading the same DATABASE_URL and translating postgresql+asyncpg → postgresql+psycopg2 (or postgresql:// bare) at engine build time.

Send path (src/app/api/…/send) — suppressions enforcement

  • Before row insertion in the /messages/send handler (Phase 2 already owns this handler), query suppressions by (mailbox_id, recipient_email) for every combined to / cc / bcc recipient.
  • If any recipient matches: reject the whole request with 422 and error.details.suppressed: [<addr>, …] (not a partial success). No row is inserted for any recipient when the request is rejected. No MIME is built.
  • The lookup is a single SELECT recipient_email FROM suppressions WHERE mailbox_id = $1 AND recipient_email = ANY($2::citext[]). No per-recipient round-trip.
  • This check happens after the existing Pydantic caps and the sender-binding validation — order does not matter for correctness (all of these paths reject the request), but suppressions last means a caller sending a malformed payload gets the malformed error, not the suppressed-recipient error.

CLI surface

  • cli bounces:poll-once — runs the cron once (above). Prints a one-line redacted summary: bounces: processed=N(perm=X, trans=Y, unk=Z), rejected=R, errors=E.
  • cli suppressions:list [--mailbox <email>] [--full] — paginated, defaults to redacted output (hash-prefix + domain only). --full requires interactive confirmation.
  • cli suppressions:remove <mailbox-email> <recipient-email> — deletes the (mailbox_id, recipient_email) row. Prints a one-line confirmation.
  • cli mailboxes:create / cli mailboxes:update gain the bounce-IMAP prompt block. An empty response for bounce_imap_host disables bounce processing for that mailbox and the CLI prints: warning: bounce processing disabled for this mailbox — DSNs will be black-holed at the upstream MTA.

Deployment

  • Docker Swarm service email.bounce-cron:
    • Image: same as email.worker (single-image repo).
    • Command: cli bounces:poll-once inside a wrapper that runs it on a schedule (either cron in-container, or a Swarm replicated-job with --detach=false fired by an external scheduler — chosen at deploy time, orthogonal to this phase).
    • Resource limits + health check + restart policy declared alongside email.api and email.worker per the §Docker Swarm rule in CONTRIBUTING.md.
  • .env.example adds: # bounce cron polls mailboxes with bounce_imap_host configured; default cadence 5 minutes, set via Swarm scheduler.

Observability (ADR 0002 §Observability, routed through PLAN.md §3.4)

New Prometheus metrics (exported wherever the pre-GA §3.4 work eventually lands — Phase 5 exposes them in-process whether or not the scrape endpoint is wired):

  • outbound_messages_bounced_total{type=permanent|transient|unknown}
  • bounce_cron_polled_total{mailbox_hash}
  • bounce_cron_rejected_total{reason=malformed|hmac|unknown_message_id}
  • bounce_cron_suppressions_inserted_total
  • bounce_cron_duration_seconds{mailbox_hash} (histogram)

mailbox_hash is sha256(mailbox.email)[:12] — operator can correlate by running cli mailboxes:list --hash. Raw email addresses as Prometheus labels would re-introduce the PII leak that §1.5 closes.

Testing discipline

Every DSN fixture under tests/fixtures/dsn/ is a real RFC-3464-shaped .eml. Hand-crafted for coverage:

  • valid_5xx_user_unknown.eml — permanent bounce, clean VERP.
  • valid_4xx_mailbox_full.eml — transient, clean VERP.
  • missing_status_line.eml — unknown classification.
  • malformed_multipart.eml — not multipart/report, rejected.
  • forged_from_mailer_daemon.eml — correct From: MAILER-DAEMON, correct subject, but unsigned VERP: must not flip any row. This is the regression test for §1.3.
  • valid_verp_unknown_message_id.eml — HMAC is valid because the attacker computed it against a random UUID, but the UUID does not match any outbound_messages row. Must be rejected (moved to Rejected-Bounces), no DB write.

The IMAP interaction is mocked at the imaplib.IMAP4_SSL layer, not at the helper layer, so the test exercises the real folder moves (COPY + STORE +FLAGS \\Deleted + EXPUNGE).

Required reading before coding

  1. ADR 0002 (docs/adr/0002-phase5-bounce-processing.md) — binding design record.
  2. FEEDBACK.md §1.3, §2.1, §2.4, §2.6 — the problems this phase closes.
  3. CONTRIBUTING.md §IMAP Parsing + §Outbound SMTP & Tracking + §Queue, Worker & Migrations — the invariants this phase must not break.
  4. src/app/workers/smtp_delivery.py::_bind_sender_headers — current Return-Path construction that the VERP rewrite slots into.
  5. src/app/utils/tracking_token.py — HMAC-signing precedent that the VERP token format mirrors (same secret, domain-separated prefix).

Anti-patterns (do NOT ship any of these)

  • ANTI-PATTERN: matching From: MAILER-DAEMON / subject keyword anywhere in the bounce pipeline, even as a “helpful fallback”. An unsigned bounce is not a bounce. §1.3 is a CRITICAL finding.
  • ANTI-PATTERN: calling imaplib from inside the async outbound worker via run_in_threadpool. ADR 0002 §2 chose the separate- process option; do not re-open that decision inside the worker.
  • ANTI-PATTERN: silent-skip of suppressed recipients on /messages/send. The request is rejected with 422 and the suppressed addresses surface in the error response; partial successes hide deliverability problems.
  • ANTI-PATTERN: logging raw recipient addresses or raw DSN Diagnostic-Code lines. Both go through redact_pii before persisting (§1.5 rule); Prometheus labels use the mailbox hash, never the address.
  • ANTI-PATTERN: cross-mailbox suppression. The unique constraint is (mailbox_id, recipient_email) for a reason — each mailbox is a separate sending identity.

Sourced from docs/features/05_imap_bounce_processing in the repo. Edits go through the same review as code.