4.3. Queue policies

Halon provides powerful policy features to control the flow of emails from the queue to recipients.

Concurrency, rate and connectinterval limits are measured with counters, which can have one or multiple fields that define a unique entry.

4.3.1. Available fields

tenantid: A single value free-text field that can be used for any purpose. Email service providers (ESPs) typically have many customers (tenants) using the same system. This field enables you to create policy on a per-tenant basis when a message is queued, to handle responses relating to the tenant’s reputation. One way is to set the tenantid to the header From domain when queuing the message in HSL.
jobid: A single value free-text field that can be used for any purpose, for example to identify the current campaign for bulk mailing, or message stream for transactional mailing. One way is to use the value of an incoming email header such as X-Job. This enables you to create policy to manage traffic flow on a per-job basis.
transportid: A single value referring to a transportgroups[].transports[], similar to “binding” in other systems. A transportid can map to one or more localip(s), or to other settings, such as specific custom destinations.
localip: This is a list, so that an email can be queued with multiple alternatives for source IP. The same source IP may be specified multiple times to distribute the traffic according to its ratio.
grouping: A single value used for “rolling up” queued emails. It’s set after DNS resolving, it may either default to empty, remotemx or recipientdomain. Any grouping configured using the queues.grouping setting will apply and fill out this field (the grouping id will be prefixed with & throughout the queue).
remoteip: This single value is known after DNS resolving. In some cases it’s preferable to build policies on the grouping field instead together with the queues.grouping.groupings[].remoteip[] setting.
remotemx: This single value is known after DNS resolving. In some cases it’s preferable to build policies on the grouping field instead together with the queues.grouping.groupings[].remotemx[] setting.
recipientdomain: This single value is a basic property of the message RCPT TO address. In some cases it’s preferable to build policies on the grouping field instead together with the queues.grouping.groupings[].recipientdomain[] setting.

4.3.2. Loading policy

The default start configuration’s environment.policyconf directive loads it from /etc/halon/smtpd-policy.yaml. It is described by, and can be validated with, the smtpd-policy.schema.json JSON schema (included in our Visual Studio Code integration).

If reloading the environment.policyconf configuration during runtime, active suspensions based on (now) removed or previous exceeded rate, concurrency or connectinterval limits are not automatically removed (as a side effect of the reload) instead the suspension is removed when the next message for that rate, concurrency or connectinterval is allow to be sent. If adding a new counter to the configuration during a reload, the counter will only count new messages (and not those being sent).

In addition to the configuration file on disk, policy conditions can be added on the fly over the Protocol Buffer API’s PolicyConditionAddRequest function, command line interface, web administration, as well as from the pre- and post-delivery script.

4.3.3. Policy counter thresholds

Thresholds for concurrency limit the number of emails in the delivery state. Thresholds for rate limit the number of emails, X, passing through the delivery state over a given time interval, Y, specified in seconds, as X/Y. Thresholds for connectinterval limit the number of connections to be open for a specific destination.

The very simplistic example from above (with two local IPs) can be described using the following YAML pickup policy configuration:

policies:
  - fields:
      - localip
    default:
      concurrency: 10000

Each time an email is picked up from the active queue, the “localip” concurrency counter entry with that email’s source IP is incremented. When the delivery attempt is done, the same counter entry is decremented. If 10 000 email for the same source IP is being delivered at the same time, the default threshold will be exceeded, and the suspension list will be populated with an entry saying that any email with that source IP should not be picked up.

Note that all configured queue pickup policies (counters) are taken into consideration when an email is picked up from the active queue, and the lowest allowed rate, concurrency and connectinterval applies. And that it’s possible to leave out the default value for a counter (so that only a specific condition is applied).

4.3.3.1. Policy conditions

Different thresholds can be set by using conditions with the desired field values. Conditions are evaluated first-to-last, with the first matching threshold winning. Consequently, if a threshold in a more general conditions is placed above a more specific one, the latter might never match (because the former always wins).

The example below limits the concurrency based on a combination of source IP and destination domain, with an override for the domain “halon.io”:

Per every field in a if condition (eg. recipientdomain) multiple values (eg. domains) may be given as an array, and matched as or-conditions.

Lists may be used to reference and match multiple values at the same time.

policies:
  - fields:
      - localip
      - recipientdomain
    conditions:
      - if:
          recipientdomain:
            - halon.io
            - halon.se
        then:
          concurrency: 2
    default:
      concurrency: 5

The above policy will be exceeded if two emails are being delivered to the recipient domain “halon.io” from the same source IP.

4.3.3.2. Policy properties

When a policy condition (if) is matched, the policies given in the then are applied. There are three counter thresholds that can be applied; the concurrency, the rate and the connectinterval policies, that will affect the message delivery. If you don’t want apply specify specific concurrency, rate or connectinterval, nor the default matching, you can configure them as null. There is also a tag property which can be used to identity which condition were matched. In addition custom properties may by specified in a properties object as key and value, these custom properties are available eg. in the post-delivery scripting hook.

policies:
  - fields:
      - localip
      - recipientdomain
    conditions:
      - if:
          recipientdomain: halon.io
        then:
          concurrency: 2
          rate: 1/60
          tag: this_rule
          properties:
            backoff-concurrency: 1 # Slow down messages when in backoff
            backoff-retry-intervals: 600,600,1200,1200,1800
            backoff-rate: 1/10
            backoff-ttl: 3600
            backoff-disableable: true
            backoff-suspendable: true
            foo: bar # Can set other custom properties too
    default:
      concurrency: 5

4.3.4. Backoff policy

Mailbox providers can send SMTP diagnostic responses (i.e. 4xx or 5xx codes) that indicate message streams are being sent too quickly. The halon-extras-backoff package provides functions that can efficiently detect these responses and act on them. When regular expressions in the backoff.csv file match, the system will apply specific backoff policy settings. This allows for fine-grained control over behavior such as how quickly deliveries are attempted.

The scope of the backoff is defined by the fields passed to the enable_backoff() function. Usually these are applied to (localip + grouping) combinations, but they could also be applied to other field combinations using tenantid or jobid, to slow down specific senders or message streams, while keeping others unaffected.

The backoff settings in Policy properties can set a specific backoff-rate, backoff-concurrency, backoff-connectinterval, and backoff-retryintervals to override the normal settings. The time to live backoff-ttl setting makes the backoff auto-expire after a number of seconds. The backoff-disableable setting allows the backoff to be disabled by a successful delivery occurring. It is usually set true. The backoff-suspendable setting allows the applicable sub-queue to be fully suspended by backoff patterns that have the action suspend=.., and is usually set true.

Other features of the backoff.csv format:

Rules can have a tag, this is a human-readable name that can be seen when viewing active backoffs in the web UI and CLI, to identify which condition was matched
Rules can be scoped to a mailbox provider, by using specific grouping values (such as &google)
The rule scope can be limited to a specific SMTP conversation state (e.g. EOD)
Rules can be written to require more than one matching response in a time period to trigger, using events=n/m
Rules can completely suspend a sub-queue, using the action suspend=.. (if backoff-suspendable is set to true)

The backoff rules are evaluated in the order they are defined in the CSV file, and the first matching rule will be applied. For a full description of the CSV backoff rule format, backoff policy settings, and how to use the HSL functions, see the repository README.

4.3.5. Policies with uniqueip

The queues.grouping.groupings[].unique setting is used for certain mailbox providers where policy should be applied separately per unique remote IP address. The if conditions used to match these groupings in the -policy, -delivery, and -suspend files need to be written as regexes.

Each regex should begin with /^ and end with ;.*/ to fully match the expanded policy id, which will be of the form &grouping-name;<remoteip>, for example:

smtpd-app.yaml

queues:
  grouping:
    groupings:
      - id: pphosted
        unique: remoteip
        remotemx:
          - "*.pphosted.com"

smtpd-policy.yaml

policies:
  fields:
    conditions:
    - if:
        grouping:
          - "/^&pphosted;.*/"
      then:
        concurrency: 15

smtpd-delivery.yaml

delivery:
  conditions:
  - if:
      grouping:
        - "/^&pphosted;.*/"
    then:
      pooling:
        transactions: 300

4.3.6. Policy precedence

Policies which are created by Web, CLI or API calls while the system is running are known as dynamic policies. These have have a time-to-live (TTL) and are reset if the service is restarted. Because there are now several sources of dynamic policies, these are the policy types, and the precedence when individual conditions are evaluated by the system:

Dynamic user-created policy (highest precedence)

Dynamic backoff policy

Dynamic warmup policy

Static policy

4.3.7. Policy counter groups

Note

Please note that it’s recommended to use the grouping field described in Queue policies together with the queues.grouping setting if you don’t need per policy groupings.

Counters can be aggregated based on wild-card, subnets or regular expression matching, so that different field values count against the same entry. Groups are given IDs, and conditions are matched against the grouped entry by prefixing with “#”. The example below has two counters, with multiple fields per counter. One limits both rate and concurrency based on destination MX (with rollup for Google G-suite) in combination with source IP. The other also limits the concurrency per source IP, but destination IP instead of MX, and only enforces a threshold for emails to recipient domains with a Microsoft Outlook MX.

policies:
  - fields:
      - localip
      - remotemx:
          gsuite:
            - '*.google.com'
            - '*.googlemail.com'
            - '*.smtp.goog'
    conditions:
      - if:
          remotemx: '#gsuite'
        then:
          concurrency: 10
          rate: 50
    default:
      concurrency: 5
      rate: 10
  - fields:
      - localip
      - remotemx:
          o365:
            - '*.protection.outlook.com'
      - remoteip
    conditions:
      - if:
          remotemx: '#o365'
        then:
          concurrency: 10
          rate: 30