4. Queuing subsystem

Halon provides control and flexibility on how messages are handled and delivered, according to policies that you define. The MTA queue is central to this.

The default behavior of an MTA is to spool messages, which are being received, to disk, then pick them up and deliver them by using the queuing subsystem. This allows the MTA to accept messages even if the next hop is unavailable, or if a message has multiple recipients on different next hop destinations.

This figure illustrates the typical states and movements inside the queue.

flowchart LR smtp{{SMTP}} --> spool[(Spool)] http{{HTTP}} --> spool api{{API}} --> spool start((Startup)) --> spool spool --> spoolin(Spool-in) spoolin --> pre(Pre-delivery hook) spoolin -->|if now < next retry|defer[(Defer queue)] spoolin -->|if message on hold|hold[(Hold queue)] pre <--> defer pre <--> hold pre --> dns(DNS resolving) dns -->|DNS error|post(Post-delivery hook) dns --> active[(Active queue)] active-->|SMTP delivery attempt|post post --> defer post --> hold post --> done{{Message Done}}

Each message in the queue can be in one of three states:

Active

Messages that are not on hold, and are scheduled for delivery now (or in the past). Messages that are put in queue() for delivery end up directly in the active queue (unless the “delay” option is used). Messages are also moved automatically from the defer queue into the active queue on their scheduled time of delivery. The active queue is the by far most advanced part of the queuing subsystem, and contains many sub-states. This stems from the fact that concurrency, rate and connectinterval can be limited based on many dynamic parameters, essentially creating a virtually unlimited number of sub-queues.

Defer

Messages that are not on hold, and are scheduled for delivery in the future. The default behavior, for every message that fails delivery (with a non-permanent error), is to be put in the defer queue; scheduled to be re-tried sometime in the future. It is a queue in the sense that messages are ordered by their scheduled delivery time.

Hold

Messages that are on permanent hold. This is the most simple, and typically least used, state. It is not a queue; rather a collection of messages. It can be used to, for example, implement a spam/virus quarantine. Messages can be put in hold list directly during the reception of a message, using the end-of-DATA script queue() function’s “hold” option.

• 4.1. Active queue
  • 4.1.1. Active queue pickup
• 4.2. Delivery settings
• 4.3. Queue policies
  • 4.3.1. Available fields
  • 4.3.2. Loading policy
  • 4.3.3. Policy counter thresholds
    • 4.3.3.1. Policy conditions
    • 4.3.3.2. Policy properties
  • 4.3.4. Policy counter groups
• 4.4. Cluster-aware policy
  • 4.4.1. Example architecture
  • 4.4.2. Delivery Orchestrator: clusterd
  • 4.4.3. Writing policy for clusters
    • 4.4.3.1. Advanced usage: selective non-clustered properties
  • 4.4.4. Observing clustered policy in operation
  • 4.4.5. Moving messages between instances
• 4.5. Message suspension
  • 4.5.1. Excluding IPs from pool
• 4.6. Changing message states
  • 4.6.1. Messages on hold
• 4.7. Spool startup
• 4.8. On-disk format
• 4.9. Queue statistics via CLI/API