4.3. Queue policies
Halon provides powerful policy features to control the flow of emails from the queue to recipients.
Concurrency, rate and connectinterval limits are measured with counters, which can have one or multiple fields that define a unique entry.
4.3.1. Available fields
tenantid
A single value free-text field that can be used for any purpose. Email service providers (ESPs) typically have many customers (tenants) using the same system. This field enables you to create policy on a per-tenant basis when a message is queued, to handle responses relating to the tenant’s reputation. One way is to set the
tenantid
to the header From domain when queuing the message in HSL.jobid
A single value free-text field that can be used for any purpose, for example to identify the current campaign for bulk mailing, or message stream for transactional mailing. One way is to use the value of an incoming email header such as X-Job. This enables you to create policy to manage traffic flow on a per-job basis.
transportid
A single value referring to a
transportgroups[].transports[]
, similar to “binding” in other systems. Atransportid
can map to one or morelocalip
(s), or to other settings, such as specific custom destinations.localip
This is a list, so that an email can be queued with multiple alternatives for source IP. The same source IP may be specified multiple times to distribute the traffic according to its ratio.
grouping
A single value used for “rolling up” queued emails. It’s set after DNS resolving, it may either default to empty, remotemx or recipientdomain. Any grouping configured using the
queues.grouping
setting will apply and fill out this field (the grouping id will be prefixed with&
throughout the queue).remoteip
This single value is known after DNS resolving. In some cases it’s preferable to build policies on the
grouping
field instead together with thequeues.grouping.groupings[].remoteip[]
setting.remotemx
This single value is known after DNS resolving. In some cases it’s preferable to build policies on the
grouping
field instead together with thequeues.grouping.groupings[].remotemx[]
setting.recipientdomain
This single value is a basic property of the message
RCPT TO
address. In some cases it’s preferable to build policies on thegrouping
field instead together with thequeues.grouping.groupings[].recipientdomain[]
setting.
4.3.2. Loading policy
The default
start configuration’s
environment.policyconf
directive
loads it from /etc/halon/smtpd-policy.yaml.
It is described by, and can be validated with, the
smtpd-policy.schema.json
JSON schema (included in our Visual Studio Code integration).
If reloading the environment.policyconf
configuration during runtime, active suspensions based on (now) removed or previous exceeded
rate, concurrency or connectinterval limits are not automatically removed (as a side effect of the reload)
instead the suspension is removed when the next message for that rate, concurrency or connectinterval is allow to be sent.
If adding a new counter to the configuration during a reload, the counter will only count new messages (and not those being sent).
In addition to the configuration file on disk, policy conditions can be added on the fly over the
Protocol Buffer API’s PolicyConditionAddRequest
function, command line interface,
web administration,
as well as from the pre- and post-delivery script.
4.3.3. Policy counter thresholds
Thresholds for concurrency limit the number of emails in the delivery state. Thresholds for rate limit the number of emails, X, passing through the delivery state over a given time interval, Y, specified in seconds, as X/Y. Thresholds for connectinterval limit the number of connections to be open for a specific destination.
The very simplistic example from above (with two local IPs) can be described using the following YAML pickup policy configuration:
policies:
- fields:
- localip
default:
concurrency: 10000
Each time an email is picked up from the active queue, the “localip” concurrency counter entry with that email’s source IP is incremented. When the delivery attempt is done, the same counter entry is decremented. If 10 000 email for the same source IP is being delivered at the same time, the default threshold will be exceeded, and the suspension list will be populated with an entry saying that any email with that source IP should not be picked up.
Note that all configured queue pickup policies (counters) are taken into consideration when an email is picked up from the active queue, and the lowest allowed rate, concurrency and connectinterval applies. And that it’s possible to leave out the default value for a counter (so that only a specific condition is applied).
4.3.3.1. Policy conditions
Different thresholds can be set by using conditions with the desired field values. Conditions are evaluated first-to-last, with the first matching threshold winning. Consequently, if a threshold in a more general conditions is placed above a more specific one, the latter might never match (because the former always wins).
The example below limits the concurrency based on a combination of source IP and destination domain, with an override for the domain “halon.io”:
Per every field in a if
condition (eg. recipientdomain
) multiple values (eg. domains) may be given as an array,
and matched as or-conditions.
Lists may be used to reference and match multiple values at the same time.
policies:
- fields:
- localip
- recipientdomain
conditions:
- if:
recipientdomain:
- halon.io
- halon.se
then:
concurrency: 2
default:
concurrency: 5
The above policy will be exceeded if two emails are being delivered to the recipient domain “halon.io” from the same source IP.
4.3.3.2. Policy properties
When a policy condition (if
) is matched, the policies given in the then
are applied.
There are three counter thresholds that can be applied; the concurrency
, the rate
and the connectinterval
policies, that will affect the message delivery.
If you don’t want apply specify specific concurrency
, rate
or connectinterval
, nor the default
matching, you can configure them as null.
There is also a tag
property which can be used to identity which condition were matched.
In addition custom properties may by specified in a properties
object as key and value, these custom properties
are available eg. in the post-delivery scripting hook.
policies:
- fields:
- localip
- recipientdomain
conditions:
- if:
recipientdomain: halon.io
then:
concurrency: 2
rate: 1/60
tag: this_rule
properties:
foo: bar
baz: 5
boz: true
default:
concurrency: 5
4.3.4. Policy counter groups
Note
Please note that it’s recommended to use the grouping
field described in Queue policies together with the queues.grouping
setting if you don’t need per policy groupings.
Counters can be aggregated based on wild-card, subnets or regular expression matching, so that different field values count against the same entry. Groups are given IDs, and conditions are matched against the grouped entry by prefixing with “#”. The example below has two counters, with multiple fields per counter. One limits both rate and concurrency based on destination MX (with rollup for Google G-suite) in combination with source IP. The other also limits the concurrency per source IP, but destination IP instead of MX, and only enforces a threshold for emails to recipient domains with a Microsoft Outlook MX.
policies:
- fields:
- localip
- remotemx:
gsuite:
- '*.google.com'
- '*.googlemail.com'
- '*.smtp.goog'
conditions:
- if:
remotemx: '#gsuite'
then:
concurrency: 10
rate: 50
default:
concurrency: 5
rate: 10
- fields:
- localip
- remotemx:
o365:
- '*.protection.outlook.com'
- remoteip
conditions:
- if:
remotemx: '#o365'
then:
concurrency: 10
rate: 30