<a id="cluster-aware-policy"></a>

<a id="id1"></a>

# Cluster-aware policy

<a id="example-architecture"></a>

## Example architecture

```mermaid
flowchart LR
  GEN(["Message
  Generation"]) --> LB
  LB["Load
  balancer"] --> H1
  LB--> H2
  LB --> H3
  H1[Halon smtpd 1] --> C
  H2[Halon smtpd 2] --> C
  H3[Halon smtpd n] --> C

  HAP1[HAProxy 1] --> |SMTP| MBP
  HAP2[HAProxy 2] --> |SMTP| MBP

  C(( )) --> HAP1
  C --> HAP2

  MBP(["Network
  egress"])
```

Generated messages are network load-balanced across all Halon instances, which are configured identically.
The delivery (*localip*) addresses are located on the proxies; any Halon instance can choose any *localip*.
The HAProxy layer can use `keepalived` or similar, to provide resilience in the event of proxy host outage.

A basic approach is to statically divide the overall concurrency and rates across Halon instances.
For Kubernetes deployments, a plugin *halon-extras-k8s-policydivide* is available which calls [`HalonMTA_queue_policy_divide()`](plugins_native.html.md#c.HalonMTA_queue_policy_divide) to adjust the traffic shaping.
This is acceptable for mailbox providers that allow concurrency greater than your number of instances, but some mailbox providers expect lower concurrency.

To solve for all cases, a different approach to traffic shaping is needed for equitable sharing of connections. This is provided by the `clusterd` service.

<a id="delivery-orchestrator-clusterd"></a>

## Delivery Orchestrator: `clusterd`

`clusterd` works in dynamic scaling situations with Kubernetes, as well as static scaling with virtual machines.

```mermaid
flowchart LR
    H1[Halon smtpd 1]
    H2[Halon smtpd 2]
    H3[Halon smtpd n]
    CLUSTERD[clusterd] <--> H1
    CLUSTERD <--> H2
    CLUSTERD <--> H3
```

It allocates effective policy values to instances:

> [`policies[].conditions[].then.rate`](config_policy.html.md#confval-policies-.conditions-.then.rate)
> : Rates are shared across the cluster, depending on the chosen algorithm; more on this below.
> 
> [`policies[].conditions[].then.concurrency`](config_policy.html.md#confval-policies-.conditions-.then.concurrency)
> : For each (*localip* + *grouping*) combination, outbound connections are allocated to instances in the cluster, ensuring fair rotation of concurrent connections, even for low concurrency destinations.
>   It ensures each instance with messages to deliver will receive a turn.
> 
> [`policies[].conditions[].then.connectinterval`](config_policy.html.md#confval-policies-.conditions-.then.connectinterval)
> : For each (*localip* + *grouping*) combination, set a minimum time interval between connections, shared across the cluster.

`clusterd` registers connected Halon instances, counts them and announces the total number back to each instance.
See [Clustering directives](config_startup.html.md#clustering-directives) for the required *smtpd* startup configuration.
To configure the `clusterd` service itself, see [Delivery Orchestrator](/clusterd/).

If a Halon instance loses connection to `clusterd`, it will revert to division of slots (min 1 connection per node) based on the last number of known hosts.
It will automatically try to reconnect and re-establish dynamic sharing.

The service supports TLS authentication/verification of both the server and client, over TCP.

<a id="writing-policy-for-clusters"></a>

## Writing policy for clusters

A system with `clusterd` active has policies clustered by default.

**Concurrent connections** are granted to instances in the cluster.
The overall cluster follows the *concurrency* properties for (*localip + grouping*) combinations in the *smtpd-policy.yaml* [`policies[].fields[]`](config_policy.html.md#confval-policies-.fields).
Because connections are opened, closed and pooled between specific `localip` and `grouping`, these fields must be used when writing cluster-aware concurrency policy.

*smtpd-app.yaml*

```yaml
groupings:
- id: orange
  remotemx:
    - "smtp-in.orange.fr"
```

*smtpd-policy.yaml*

```yaml
rate:
  algorithm: tokenbucket
policies:
  - fields:
    - jobid
  - fields:
    - localip
  - fields:
    - tenantid
  - fields:
    - tenantid
    - grouping
  - fields: # cluster-wide policies here
    - localip
    - grouping
  default:
    concurrency: 20
  conditions:
    if:
      grouping: &orange
    then:
      concurrency: 2 # Cluster-wide rate for all localip, see https://postmaster.orange.fr/
```

**Rates** for the cluster can be set on *any* field condition, because rates are always divisible.
How they are divided depends on your chosen [`rate.algorithm`](config_policy.html.md#confval-rate.algorithm) setting.

> Algorithm `tokenbucket` (default) gives smooth traffic flow over the interval by scaling the denominator, i.e.
> `tokens / (interval * number_of_hosts)`
> 
> Algorithm `fixedwindow` gives traffic bursts at the start of each interval, by allocating the tokens between hosts, with each getting at least 1, i.e.
> `max(1, tokens / number_of_hosts) / interval`

For fair rotation of connections between your instances, set `delivery.pooling.timeout` to small values such as 5 to 10 seconds, and `delivery.pooling.transactions` to 100 or less.

If you wish to disable rotation of connections on an entire smtpd instance, set [`cluster.policy.sharedconcurrency`](config_startup.html.md#confval-cluster.policy.sharedconcurrency) to `false`.
This will give basic “divide by n” concurrency, with at least 1 connection per instance.

<a id="advanced-usage-selective-non-clustered-properties"></a>

### Advanced usage: selective non-clustered properties

Rules can selectively disengage clustering, for example to give more throughput to time-critical message streams.

*smtpd-policy.yaml*

```yaml
conditions:
  if:
    localip: 1.2.3.4
    grouping: &yahoo
  then:
    concurrency: 5
    cluster: false # each instance will get 5 connections
  default:
    rate: 5/1 # default rate applies cluster-wide
```

The `cluster: false` property applies to all properties in `then` scope.
To create policy that has some properties clustered, and others non-clustered, create two `if` clauses:

```yaml
conditions:
if:
  localip: 1.2.3.4
  grouping: &yahoo
then:
  concurrency: 5
  cluster: false # affects properties in “then” scope
if:
  localip: 1.2.3.4
  grouping: &yahoo
then:
  rate: 10/1 # put (clustered) rate property in a separate if clause
default:
  rate: 5/1 # default rate applies cluster-wide
```

<a id="observing-clustered-policy-in-operation"></a>

## Observing clustered policy in operation

The [web user interface](https://docs.halon.io/web/) Delivery Insights view shows cluster behavior.
For each policy setting, it shows:

* the cluster’s concurrency and rate
* the effective concurrency and rate in use on this instance.

For example, a policy rate of `1/10` (one message every 10 seconds) with the *tokenbucket* algorithm, will be divided by the number of instances in the cluster
giving an effective rate on a three-instance cluster of `1/30`.

The [halontop](cli_stats.html.md#halontop) command has a panel showing the cluster status and number of hosts found.

<a id="cluster-moving"></a>

<a id="moving-messages-between-instances"></a>

## Moving messages between instances

Messages can be moved from the current MTA to another, using the `hqfmove` command-line utility. See also [Moving the queue files with hqfmove](cli_queue.html.md#cli-hqfmove).

Before moving messages, either:

> - shut down the `smtpd` process, or
> - unload specific messages using `halonctl queue unload` and move messages from the [`spool.path`](config_startup.html.md#confval-spool.path) folder to a separate folder, so new messages arriving are not mixed with the messages to transfer.

Run the command on the source MTA, specifying the folder of messages you wish to move with the `--directory` argument. This will move all the messages in the specified folder and its sub-folders.

Container-based deployments make use of `hqfmove` to handle scaling-down requests, moving messages from the container being removed to another container.
For more information, see the [Halon Docker](https://github.com/halon/halon-docker/) repository.