Feature 7: Raw Email Forwarding & Scheduled Rules Engine

Overview

AI agents and integrations often need a robust way to forward existing incoming emails to external partners or specialized webhooks without losing data (such as original HTML formatting, nested attachments, inline media, or headers).

Directly editing or reconstructing MIME structures is fragile and violates DMARC, SPF, and DKIM alignment if sent from a different server.

To solve this, me.squibble.email implements a clean, high-performance, stream-backed Raw Email Forwarding Engine that:

1. Encapsulates the original message as a standard MIME attachment of type message/rfc822 (.eml).
2. Streams bytes directly from IMAP to local disk storage (__ROOTFS/var/lib/squibble-email/attachments/) to bypass Python RAM constraints.
3. Queues standard outbound delivery envelopes with loop-prevention protections.
4. Provides a scheduled Rules Engine CLI command to scan, match, and forward matched emails automatically.

Architecture Flow

                      ┌──────────────────────────┐
                      │     IMAP Mailbox         │
                      └─────────────┬────────────┘
                                    │ (UID Fetch BODY.PEEK[])
                                    ▼ (Stream raw bytes)
                      ┌──────────────────────────┐
                      │    FileStorageService    │ ──► Saved on persistent disk
                      └─────────────┬────────────┘
                                    │ (Create outbound attachment)
                                    ▼
                      ┌──────────────────────────┐
                      │ Postgres Outbound Queue  │
                      └─────────────┬────────────┘
                                    │ (Worker claims)
                                    ▼
                      ┌──────────────────────────┐
                      │    Outbound SMTP MTA     │ ──► Deliver as message/rfc822
                      └──────────────────────────┘

API Reference

Forward Email Action

Directly trigger a forward for a specific IMAP Message UID.

• Endpoint: POST /api/v1/email/account/{mailbox}/messages/{uid}/forward
• Token Scope Required: messages:send
• Request Headers:
  • Authorization: Bearer <JWT_TOKEN>
• Request Body (ForwardMessageRequest):

  {
    "to": ["recipient@external-domain.com"]
  }

• Response (202 Accepted):

  {
    "message_ids": ["c29bc529-7fe8-4a65-a5e4-6be2b4fda722"]
  }

Technical Specifications & Safeguards

1. Loop Prevention & RFC Headers

To prevent devastating infinite forwarding mail loops, the outer SMTP envelope enforces these immutable headers when compiling a forward:

• X-Forwarded-By: email.squibble.ch
• X-Loop-Prevention: <mailbox_id>
• Auto-Submitted: auto-generated (Instructs auto-responders to drop reply loops)

Any incoming email carrying X-Forwarded-By with our hostname or X-Loop-Prevention matching the active mailbox ID is instantly discarded and logged as a quarantined loop prevention event.

2. Stream-Based Storage (__ROOTFS)

Instead of loading potentially massive (e.g., 20MB) base64 strings directly into SQLAlchemy text columns, attachments are written directly to a persistent local Docker volume directory.

• Default Path: __ROOTFS/var/lib/squibble-email/attachments/
• Configuration: Configurable via the SQUIBBLE_STORAGE_ROOT environment variable.
• During worker claim loops, files are streamed block-by-block using non-blocking file readers (aiofiles), maintaining the worker’s resident memory footprint under 50MB even when processing gigabytes of raw files.

Inbound fetch caveat — imaplib buffers the full payload. Reading a message from IMAP is not incrementally streamable: imaplib’s UID FETCH BODY.PEEK[] materialises the entire payload in RAM before we can chunk it to disk. The block-by-block streaming above therefore applies to the write and replay paths, not the IMAP read. To bound that one-shot footprint, the engine probes RFC822.SIZE first and rejects any message larger than FORWARDING_MAX_MESSAGE_BYTES (default 25 MiB, set 0 to disable) before downloading the body. A size probe that fails is fail-open: the forward proceeds and the failure is logged.

3. Connection & Concurrency Lockout Guard

IMAP servers aggressively throttle multi-connection concurrency per account. me.squibble.email enforces a strict SQLite or Redis-backed active lockout that limits background forwarding/polling workers to a maximum of 1 concurrent IMAP connection per Mailbox ID, queuing subsequent actions to avoid server bans.

Scheduled Rules Engine

For hands-off, automated routing of specific alerts (e.g., matching security or invoice headers), administrators can configure database-driven forwarding rules.

Database Schema (forwarding_rules)

CLI Command: forwarding:poll-once

This script executes as a sync poller (intended to be scheduled via cron or host supervisor every 5 minutes):

1. Connects to the active IMAP accounts.
2. Queries the server for UNSEEN messages that also lack the custom $Forwarded IMAP keyword.
3. Selective fetches the headers only (BODY.PEEK[HEADER]) to evaluate active forwarding_rules.
4. If matched, it streams the full raw MIME to disk, queues the standard OutboundMessage, and applies the $Forwarded flag to the original message on the server using STORE +FLAGS ($Forwarded) to guarantee it is never processed again.