4.3. Queue policies

Halon provides powerful policy features to control the flow of emails from the queue to recipients.

Concurrency, rate and connectinterval limits are measured with counters, which can have one or multiple fields that define a unique entry.

4.3.1. Available fields

tenantid

A single value free-text field that can be used for any purpose. Email service providers (ESPs) typically have many customers (tenants) using the same system. This field enables you to create policy on a per-tenant basis when a message is queued, to handle responses relating to the tenant’s reputation. One way is to set the tenantid to the header From domain when queuing the message in HSL.

jobid

A single value free-text field that can be used for any purpose, for example to identify the current campaign for bulk mailing, or message stream for transactional mailing. One way is to use the value of an incoming email header such as X-Job. This enables you to create policy to manage traffic flow on a per-job basis.

transportid

A single value referring to a transportgroups[].transports[], similar to “binding” in other systems. A transportid can map to one or more localip(s), or to other settings, such as specific custom destinations.

localip

This is a list, so that an email can be queued with multiple alternatives for source IP. The same source IP may be specified multiple times to distribute the traffic according to its ratio.

grouping

A single value used for “rolling up” queued emails. It’s set after DNS resolving, it may either default to empty, remotemx or recipientdomain. Any grouping configured using the queues.grouping setting will apply and fill out this field (the grouping id will be prefixed with & throughout the queue).

remoteip

This single value is known after DNS resolving. In some cases it’s preferable to build policies on the grouping field instead together with the queues.grouping.groupings[].remoteip[] setting.

remotemx

This single value is known after DNS resolving. In some cases it’s preferable to build policies on the grouping field instead together with the queues.grouping.groupings[].remotemx[] setting.

recipientdomain

This single value is a basic property of the message RCPT TO address. In some cases it’s preferable to build policies on the grouping field instead together with the queues.grouping.groupings[].recipientdomain[] setting.

4.3.2. Loading policy

The default start configuration’s environment.policyconf directive loads it from /etc/halon/smtpd-policy.yaml. It is described by, and can be validated with, the smtpd-policy.schema.json JSON schema (included in our Visual Studio Code integration).

If reloading the environment.policyconf configuration during runtime, active suspensions based on (now) removed or previous exceeded rate, concurrency or connectinterval limits are not automatically removed (as a side effect of the reload) instead the suspension is removed when the next message for that rate, concurrency or connectinterval is allow to be sent. If adding a new counter to the configuration during a reload, the counter will only count new messages (and not those being sent).

In addition to the configuration file on disk, policy conditions can be added on the fly over the Protocol Buffer API’s PolicyConditionAddRequest function, command line interface, web administration, as well as from the pre- and post-delivery script.

4.3.3. Policy counter thresholds

Thresholds for concurrency limit the number of emails in the delivery state. Thresholds for rate limit the number of emails, X, passing through the delivery state over a given time interval, Y, specified in seconds, as X/Y. Thresholds for connectinterval limit the number of connections to be open for a specific destination.

The very simplistic example from above (with two local IPs) can be described using the following YAML pickup policy configuration:

policies:
  - fields:
      - localip
    default:
      concurrency: 10000

Each time an email is picked up from the active queue, the “localip” concurrency counter entry with that email’s source IP is incremented. When the delivery attempt is done, the same counter entry is decremented. If 10 000 email for the same source IP is being delivered at the same time, the default threshold will be exceeded, and the suspension list will be populated with an entry saying that any email with that source IP should not be picked up.

Note that all configured queue pickup policies (counters) are taken into consideration when an email is picked up from the active queue, and the lowest allowed rate, concurrency and connectinterval applies. And that it’s possible to leave out the default value for a counter (so that only a specific condition is applied).

4.3.3.1. Policy conditions

Different thresholds can be set by using conditions with the desired field values. Conditions are evaluated first-to-last, with the first matching threshold winning. Consequently, if a threshold in a more general conditions is placed above a more specific one, the latter might never match (because the former always wins).

The example below limits the concurrency based on a combination of source IP and destination domain, with an override for the domain “halon.io”:

Per every field in a if condition (eg. recipientdomain) multiple values (eg. domains) may be given as an array, and matched as or-conditions.

Lists may be used to reference and match multiple values at the same time.

policies:
  - fields:
      - localip
      - recipientdomain
    conditions:
      - if:
          recipientdomain:
            - halon.io
            - halon.se
        then:
          concurrency: 2
    default:
      concurrency: 5

The above policy will be exceeded if two emails are being delivered to the recipient domain “halon.io” from the same source IP.

4.3.3.2. Policy properties

When a policy condition (if) is matched, the policies given in the then are applied. There are three counter thresholds that can be applied; the concurrency, the rate and the connectinterval policies, that will affect the message delivery. If you don’t want apply specify specific concurrency, rate or connectinterval, nor the default matching, you can configure them as null. There is also a tag property which can be used to identity which condition were matched. In addition custom properties may by specified in a properties object as key and value, these custom properties are available eg. in the post-delivery scripting hook.

policies:
  - fields:
      - localip
      - recipientdomain
    conditions:
      - if:
          recipientdomain: halon.io
        then:
          concurrency: 2
          rate: 1/60
          tag: this_rule
          properties:
            foo: bar
            baz: 5
            boz: true
    default:
      concurrency: 5

4.3.4. Policy counter groups

Note

Please note that it’s recommended to use the grouping field described in Queue policies together with the queues.grouping setting if you don’t need per policy groupings.

Counters can be aggregated based on wild-card, subnets or regular expression matching, so that different field values count against the same entry. Groups are given IDs, and conditions are matched against the grouped entry by prefixing with “#”. The example below has two counters, with multiple fields per counter. One limits both rate and concurrency based on destination MX (with rollup for Google G-suite) in combination with source IP. The other also limits the concurrency per source IP, but destination IP instead of MX, and only enforces a threshold for emails to recipient domains with a Microsoft Outlook MX.

policies:
  - fields:
      - localip
      - remotemx:
          gsuite:
            - '*.google.com'
            - '*.googlemail.com'
            - '*.smtp.goog'
    conditions:
      - if:
          remotemx: '#gsuite'
        then:
          concurrency: 10
          rate: 50
    default:
      concurrency: 5
      rate: 10
  - fields:
      - localip
      - remotemx:
          o365:
            - '*.protection.outlook.com'
      - remoteip
    conditions:
      - if:
          remotemx: '#o365'
        then:
          concurrency: 10
          rate: 30