4.1. Active queue
Message delivery goes through five stages:
- Pre-delivery script
The pre-delivery script is executed (if it exists). The script allows the administrator to implement per-attempt logic, such as dynamic routing.
- DNS
All necessary domain name information is resolved. If that succeeds, the message is put in the active queue. If not, it skips directly to the post-delivery script.
- Active queue
The message remains in the active queue until allowed by the active queue’s pickup policies (that control concurrency, rate, connectinterval, etc). The pickup policies define the virtually unlimited number of sub-queues, and will be described later in detail.
- Delivery
A delivery attempt (SMTP or LMTP) is made.
- Post-delivery script
Regardless of the outcome of the delivery attempt, the post-delivery script is executed (if it exists). It can be used by the administrator to override the default logic or to implement logging.
4.1.1. Active queue pickup
The active queue pickup subsystem determines when a message should be picked up, usually based on concurrency, rate or connectinterval settings on properties such as local IP or destination. The queue pickup takes into account message suspensions that are in force.
The Policy configuration file defines combinations of policy fields, forming virtual sub-queues for messages. This is useful for separating email of different classes, so that one class of email that is stuck or moving slowly does not block others. Those sub-queues are created based on any properties; pre-defined ones like recipient domain, or custom fields that can be populated from script.
For example, consider the case where the system’s total concurrency is set to 20 000, and the system has two local IP addresses that can be used as source IPs when sending email. With a pickup policy limiting the concurrency to 10 000 connections per local IP, we can be certain that even if traffic from one of the IPs jams up, traffic from the other IP will not be disturbed. This concept can be extended to, for example, customers in a multi-tenant system (making sure that abuse from one customer does not jam the queue for others) or recipient domains/MXs (so that one slow destination does not jam the queue for other destinations). Finally, combinations of those can be created.
As explained above the queue is not a FIFO (first-in first-out) type of queue, as messages are divided into virtual queue and delivered in the fastest possible way without exceeding the queue policies (of concurrency and rates). However there is a concept of priority messages (0 - normal, 1 - high), these will be considered candidates for pickup/delivery before any other messages. They also have precedence when handled by the DNS resolver and pre- and post-delivery threads. However given the explained concept there are no guarantees that they will be delivered before a low priority message in case of suspending rules (exceeding concurrency or rates). Also within each queue it employs FIFO. That guarantees that no message should be stuck in the queue because of unfair treatment.
Below is an illustration of how the queue looks like.