# Halon Delivery Guru - Warmup Version: 1.1-stable This is the documentation for Halon Delivery Guru *halon-policyd* service. The package consists of one service called *halon-policyd* which can be used to automate warmup of IP-addresses. ## Table of contents * [Introduction](#intro) * [Connectivity and state](#intro:connectivity-and-state) * [Tips for successful warmup](#intro:tips-for-successful-warmup) * [Recommended approach](#intro:recommended-approach) * [Alternate approach](#intro:alternate-approach) * [Example configuration](#intro:example-configuration) * [Installation](#install) * [Prerequisites](#install:prerequisites) * [Distributions](#install:distributions) * [Ubuntu](#install:ubuntu) * [RHEL](#install:rhel) * [Startup](#install:startup) * [Docker & K8s](#install:docker-k8s) * [Upgrade](#upgrade) * [Configuration](#config) * [Startup configuration](#config_startup) * [Listener directives](#config_startup:listener-directives) * [WebSocket directives](#config_startup:websocket-directives) * [Elasticsearch directives](#config_startup:elasticsearch-directives) * [Running configuration](#config_running) * [Authentication directives](#config_running:authentication-directives) * [IP directives](#config_running:ip-directives) * [Elasticsearch](#elastic) * [Install Elasticsearch](#elastic:install-elasticsearch) * [Import an index template](#elastic:import-an-index-template) * [Action events](#elastic:action-events) * [HTTP API](#api) * [Troubleshooting](#api:troubleshooting) * [*Config* endpoints](#api:config-endpoints) * [Get all config](#api:get-all-config) * [Get config for an IP](#api:get-config-for-an-ip) * [Get config for an IP and grouping](#api:get-config-for-an-ip-and-grouping) * [Add IP to config](#api:add-ip-to-config) * [Update IP config](#api:update-ip-config) * [Delete IP from config](#api:delete-ip-from-config) * [Pause/resume all groupings for an IP](#api:pause-resume-all-groupings-for-an-ip) * [Pause/resume a grouping within an IP](#api:pause-resume-a-grouping-within-an-ip) * [Delete grouping from IP](#api:delete-grouping-from-ip) * [*Status* endpoints](#api:status-endpoints) * [Get all status](#api:get-all-status) * [Get status for an IP](#api:get-status-for-an-ip) * [Get status for an IP and grouping](#api:get-status-for-an-ip-and-grouping) * [Changelog](#changelog) # Introduction The purpose of the `halon-policyd` service is to orchestrate the flow of email from the cluster, enabling source IPs to be warmed up with selected traffic according to planned schedules. The service is hosted separately to the Halon [smtpd](https://docs.halon.io/manual/) instances, in a similar manner to the [halon-clusterd](https://docs.halon.io/clusterd/) service. Because these services impose a small load on the host, they can be co-located; in a [containerized environment](https://github.com/halon/halon-docker/) it is usual to dedicate a pod to each service. The warmup schedules are easily defined by the user on the [Halon web UI](https://docs.halon.io/web/) Delivery Guru / IP Warmup page. You can also directly edit the `/etc/halon/policyd-app.yaml` [Running configuration](#config_running), or use the [HTTP API](#api). For each IP address being warmed, the user selects one or more recipient groupings (`&yahoo`, `&gmail`, `µsoft` and so on). For each grouping, the user can define: * The schedule mode (automatic or manual) * The start date for the warmup * How many days to run * The target volume per day, when the warmup is done. In manual mode, the user can adjust additional settings as the warmup progresses. Daily warmup progress can be monitored by the user on the web UI, in both graphical and tabular form. ## Connectivity and state ```mermaid graph LR B[halon-policyd] A[(Elasticsearch)] D[Halon Engage Web UI] C[smtpd] D -->|reads/writes| B D -->|reads| A B -->|writes| A C -->|writes| A C -->|subscribes to events| B ``` Components: : - **Elasticsearch:** Stores action events (rate policies and suspensions) for warmup progress tracking and analytics. - **halon-policyd:** The main service that manages warmup schedules, policies, and state. - **smtpd:** The SMTP daemon responsible for sending email, subscribing to policy updates from `halon-policyd`. - **Halon Engage Web UI:** The user interface for monitoring and configuring warmup schedules and viewing progress. Each `smtpd` subscribes to events on a websocket, which provides a bidirectional stream of updates to rate policies and suspensions to achieve warmup goals. The Halon smtpd plugin [policyd-client](https://github.com/halon-extras/policyd-client) is configured in the startup configuration `plugins` section, and must match the `halon-policyd` startup configuration [WebSocket directives](#config_startup:websocket), see [example configuration](#intro:example). Elasticsearch is a required component. Indexes are used to hold: - **action events** - a record of the actions taken by the `halon-policyd` service. To set up Elasticsearch, see [Elasticsearch](#elastic). The `halon-policyd` service requires persistent state of the schedules over many days. This is held in the [Running configuration](#config_running). The `halon-policyd` service updates the running schedules every 24 hours. This is used to ensure that the warmup schedules are always up-to-date and reflect the current state of the warmup process. Schedules are also updated when the user makes changes to the warmup schedules on the web UI. The Halon web UI connects to `halon-policyd` via an API listener. This can be secured with [`listeners[].pki`](#config_startup:confval-listeners-.pki) , and access control with [`authentication.apikeys[]`](#config_running:confval-authentication.apikeys). ## Tips for successful warmup > **Important** > > Add the cold IPs into a warmup schedule before those IPs are active for delivery, otherwise they will be used immediately which can lead to poor warmup results. When adding a new IP, define it first in the warmup menu. Set the grouping schedules to begin on a day in the future. Then add the IP to a transport in your `smtpd` configuration. This prevents the IP from being used before you are ready. To ensure that messages are delivered in a timely manner during warming, there must be an alternative, **already warm IP** in the same transport to carry overflow email. Effective warmup needs enough eligible messages in the queue, for most of the day, to reach the day’s target volume. If not enough messages are available, then the deliveries made via the warming IP will naturally fall below the daily target, taking more days to reach the desired volume. ### Recommended approach As the already-warm IP will have a fast delivery rate, adjust this down a little in static policy, so that some queued email is eligible for attempts via the warming IPs. ### Alternate approach You can also add the warmup IP first in the [transportgroups[].connection.sourceip.ipv4](https://docs.halon.io/manual/config_smtpd.html#confval-transportgroups-.connection.sourceip.ipv4) list or [transportgroups[].connection.sourceip.ipv6](https://docs.halon.io/manual/config_smtpd.html#confval-transportgroups-.connection.sourceip.ipv6) list and then disable [transportgroups[].connection.sourceip.random](https://docs.halon.io/manual/config_smtpd.html#confval-transportgroups-.connection.sourceip.random) (defaults `true`). Then it will always **try the warmup IP first** if it is allowed to deliver traffic. If you have more than one already warm IP (in addition to the one being warmed-up) disabling `random` is not ideal, as it may never use the IPs further down in the list, unless you have a lot of traffic. ## Example configuration > **Note** > > This example uses `wss://` to establish a secure (TLS) WebSocket connection from `smtpd` to `halon-policyd`. Use `ws://` for an unencrypted (plain) WebSocket connection. *smtpd.yaml* ```yaml plugins: - id: policyd-client config: address: wss://my-cluster.company.com:8091 # talk to the halon-policyd websocket listener ``` *policyd.yaml* ```yaml version: "1.1" listeners: - port: 8090 # talk to the web UI # omit address: to listen on all interfaces, ipv4 and ipv6 pki: privatekey: # your private key here certificate: # your certificate here websocket: listener: port: 8091 # talk to the smtpd policyd-clients # omit address: to listen on all interfaces, ipv4 and ipv6 pki: privatekey: # your private key here certificate: # your certificate here elasticsearch: index: actions: "halon-policyd-actions" # written by halon-policyd nodes: - url: https://your-elastichost.com:9200 # adjust for your ElasticSearch service auth: username: elastic password: # adjust for your ElasticSearch service tls: verify: false # set if your ElasticSearch has self-signed certs ``` *policyd-app.yaml* ```yaml version: "1.1" authentication: apikeys: - badsecret # matches policyd.secret[] in web.yaml ``` *web.yaml* ```yaml elasticsearch: index: policyd: "halon-policyd-actions" nodes: - url: https://your-elastichost.com:9200 auth: username: elastic password: # adjust for your ElasticSearch service tls: rejectUnauthorized: false # set if your ElasticSearch has self-signed certs policyd: address: your.policyd.com # where halon-policyd is running port: 8090 # talk to halon-policyd secret: badsecret # matches authentication.apikeys[] in policyd-app.yaml tls: enabled: true verify: false ``` # Installation ## Prerequisites Make sure you have followed the [instructions](https://docs.halon.io/manual/install.html#installation) in our manual to add our package repository. ## Distributions ### Ubuntu To install the .deb package use the following commands. ```console $ apt-get update $ apt-get install halon-policyd ``` ### RHEL To install the .rpm package use the following commands. ```console $ yum install halon-policyd ``` ## Startup When the initial [configuration](#config) is in place, the program can be started with: ```console $ systemctl start halon-policyd ``` ## Docker & K8s For instructions on how to build a container image of the program as well as deploying it on K8s (Kubernetes), see our [halon-docker](https://github.com/halon/halon-docker) GitHub repository. # Upgrade In order to upgrade the package, install the new package over the previous installed one (see the installation chapter). After the upgrade the service needs to be manually restarted. # Configuration The configuration consists of files in YAML format on disk. The package comes with an example configuration in */opt/halon/policyd/share*. The [running configuration](#config_running) contains the bulk of the configuration data, including all the schedules. It can be softly reloaded. * [Startup configuration](#config_startup) * [Listener directives](#config_startup:listener-directives) * [WebSocket directives](#config_startup:websocket-directives) * [Elasticsearch directives](#config_startup:elasticsearch-directives) * [Running configuration](#config_running) * [Authentication directives](#config_running:authentication-directives) * [IP directives](#config_running:ip-directives) # Startup configuration This is the non-reloadable part of the configuration. *halon-policyd* loads the non-reloadable configuration from `/etc/halon/policyd.yaml`. It is described by, and can be validated with, the [policyd.schema.json](http://lbovet.github.io/docson/index.html#https://docs.halon.io/json-schemas/policyd/1.1/policyd.schema.json). ## Listener directives This listener is used to receive requests from applications such as the [Halon web UI](https://docs.halon.io/web/). ### listeners[].port Which TCP port to listen on. **Required** ### listeners[].address IPv4 or IPv6 address to listen on. The default is to listen to all IPv4 and IPv6 addresses. ### listeners[].pki The X.509 certificate and private key to use for transport security. The private key and certificate should have either a `path` or `data` property. ```yaml pki: certificate: data: |- -----BEGIN CERTIFICATE----- ... privatekey: data: |- -----BEGIN PRIVATE KEY----- ... ``` ### listeners[].pki.certificate The X.509 certificate to use for transport security. The public certificate should have either a `path` or `data` property. ### listeners[].pki.privatekey The X.509 private key to use for transport security. The private key should have either a `path` or `data` property. ## WebSocket directives This listener is used to to communicate with the [smtpd](https://docs.halon.io/manual/) instances. ### websocket.listener.port Which TCP port to listen on. **Required** ### websocket.listener.address IPv4 or IPv6 address to listen on. The default is to listen to all IPv4 and IPv6 addresses. ### websocket.listener.pki The X.509 certificate and private key to use for transport security. The private key and certificate should have either a `path` or `data` property. ```yaml pki: certificate: data: |- -----BEGIN CERTIFICATE----- ... privatekey: data: |- -----BEGIN PRIVATE KEY----- ... ``` ### websocket.listener.pki.certificate The X.509 certificate to use for transport security. The public certificate should have either a `path` or `data` property. ### websocket.listener.pki.privatekey The X.509 private key to use for transport security. The private key should have either a `path` or `data` property. ## Elasticsearch directives ### elasticsearch The settings for Elasticsearch. **Required** ### elasticsearch.index.actions The index for storing action events. **Required** ### elasticsearch.nodes[] A list of Elasticsearch nodes. **Required** ### elasticsearch.nodes[].url The Elasticsearch URL to connect to. **Required** ### elasticsearch.auth Specify any credentials for authentication. ### elasticsearch.auth.username The username for basic authentication. ### elasticsearch.auth.password The password for basic authentication. ### elasticsearch.tls.verify If the server certificate should be verified. The default is **true**. # Running configuration This is the reloadable part of the configuration. *halon-policyd* loads the reloadable configuration from `/etc/halon/policyd-app.yaml`. It is described by, and can be validated with, the [policyd-app.schema.json](http://lbovet.github.io/docson/index.html#https://docs.halon.io/json-schemas/policyd/1.1/policyd-app.schema.json). ## Authentication directives The authentication settings for the HTTP [HTTP API](#api). ### authentication.apikeys[] The API keys to allow. ## IP directives ### ips{} The IP addresses. ### ips{}.groupings[] The groupings. ### ips{}.groupings[].grouping The name of the grouping. **Required** ### ips{}.groupings[].mode The mode. Can be either `AUTO` or `MANUAL`. **Required** ### ips{}.groupings[].paused Indicates whether the grouping is paused for that IP address. The default is `false`. ### ips{}.groupings[].start IP warmup start configuration for the grouping. **Required** ### ips{}.groupings[].start.date The date when the IP warmup starts for the grouping. **Required** ### ips{}.groupings[].start.volume The starting volume for IP warmup for the grouping. ### ips{}.groupings[].end IP warmup end configuration for the grouping. ### ips{}.groupings[].end.date The date goal when the IP warmup should reach the target volume for the grouping. ### ips{}.groupings[].end.volume The target volume for IP warmup for the grouping. ### ips{}.groupings[].algorithm The rate-limiting algorithm for the grouping. Can be either `tokenbucket` or `fixedwindow`. ### ips{}.groupings[].interval The rate-limiting interval (in seconds). The default is 86400 seconds (1 day). ### ips{}.groupings[].schedule[] The schedules. ### ips{}.groupings[].schedule[].day The day number. **Required** ### ips{}.groupings[].schedule[].messages The amount of messages to attempt to send out. **Required** ### ips{}.groupings[].properties{} The custom properties to add to the policies that are created. # Elasticsearch To enable actions logging there are some steps that need to be completed which are outlined below. ## Install Elasticsearch Elastic has an article on installing Elasticsearch [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html). ## Import an index template Import the template and policy below into Elasticsearch. ### Action events For this template you should set the value for [`elasticsearch.index.actions`](#config_startup:confval-elasticsearch.index.actions) to match. > ```console > curl -k -X PUT -H "Content-Type: application/json" -d @/opt/halon/policyd/share/elasticsearch/actions/halon-policyd-actions-timeseries-policy.json http://elastic:elastic@127.0.0.1:9200/_ilm/policy/halon-policyd-actions-timeseries-policy > curl -k -X PUT -H "Content-Type: application/json" -d @/opt/halon/policyd/share/elasticsearch/actions/halon-policyd-actions.json http://elastic:elastic@127.0.0.1:9200/_index_template/halon-policyd-actions > ``` # HTTP API This is the documentation for the [Listener directives](#config_startup:api-listener), shown in the form of `curl` examples. For the address `127.0.0.1:8090`, use your own specific host and port. ## Troubleshooting - If you see `curl: (1) Received HTTP/0.9 when not allowed`, you are probably making plain `http` requests to an `https` listener that is configured with [`listeners[].pki`](#config_startup:confval-listeners-.pki). Adjust your request method to `https:` - In these examples, `` and `` are placeholders. Replace them with specific values. - Grouping names start with a literal `&` which has special meaning in HTTP. Escape them in the request path, e.g. `%26yahooapac`. - Dotted-decimal IP addresses are accepted in literal form. You can also encode the `.` in the request path as `%2E`. Possible error responses : - **400** (Bad Request): Missing parameters, invalid data, or grouping already exists. - **401** (Unauthorized): You have [`authentication.apikeys[]`](#config_running:confval-authentication.apikeys). Pass in a valid key: `-H "X-Api-Key: badsecret"` - **404** (Not Found) JSON response is in compact format; examples shown below are pretty-printed for ease of reading. ## *Config* endpoints ### Get all config ```bash curl http://127.0.0.1:8090/config ``` Returns all IPs and their groupings. ### Get config for an IP ```bash curl http://127.0.0.1:8090/config/ ``` ### Get config for an IP and grouping ```bash curl http://127.0.0.1:8090/config// ``` *example response* ```json { "end": { "date": "2025-09-08", "volume": 26396 }, "grouping": "&yahooapac", "mode": "AUTO", "paused": false, "algorithm":"TOKENBUCKET", "interval": 86400, "start": { "date": "2025-07-31", "volume": 233 } } ``` ### Add IP to config ```bash curl -iX POST http://127.0.0.1:8090/config/ -H "Content-Type: application/json" -d '{ ... }' ``` Returns `200` OK on success, `400` Bad Request if IP already exists or if the data is invalid. Required keys and values for each grouping are start date and volume, end date and volume, and schedule mode. For manual schedules an array of days and target volumes for each day is also required. > **Tip** > > Use single-quotes as shown, to surround data with `-d`, then the inner `"` don’t need to be escaped. *example request* ```bash curl -iX POST http://127.0.0.1:8090/config/10.0.0.1 -H "Content-Type: application/json" -d '{ "groupings": [ { "grouping": "&auto_example", "start": { "date": "2025-08-01", "volume": 100 }, "end": { "date": "2025-08-30", "volume": 20000 }, "mode": "AUTO", "paused": false, "algorithm":"TOKENBUCKET", "interval": 86400, }, { "grouping": "&manual_example", "start": { "date": "2025-08-01" }, "mode": "MANUAL", "paused": false, "algorithm":"TOKENBUCKET", "interval": 86400, "schedule": [ { "day": 1, "messages": 300 }, { "day": 2, "messages": 400 }, { "day": 3, "messages": 500 } ] } ] }' ``` *Success (200) response* ```json {"status":"success"} ``` *Error (400) response* ```json {"error":"IP already exists"} ``` ### Update IP config ```bash curl -iX PUT http://127.0.0.1:8090/config/ -H "Content-Type: application/json" -d '{ ... }' ``` Request data follows the same pattern as [Add IP to config](#api:add-ip-config) example. Returns `200` OK on success, `400` Bad Request on validation errors, `404` if IP does not exist. ### Delete IP from config ```bash curl -iX DELETE http://127.0.0.1:8090/config/ ``` Returns `200` OK on success, `404` Not Found if IP does not exist. ### Pause/resume all groupings for an IP ```bash curl -iX PUT http://127.0.0.1:8090/config//pause curl -iX PUT http://127.0.0.1:8090/config//resume ``` Returns `200` OK on success, `404` if IP does not exist. ### Pause/resume a grouping within an IP ```bash curl -iX PUT http://127.0.0.1:8090/config///pause curl -iX PUT http://127.0.0.1:8090/config///resume ``` Returns `200` OK on success, `404` if the IP or named grouping does not exist. ### Delete grouping from IP ```bash curl -iX DELETE http://127.0.0.1:8090/config// ``` Returns `200` OK on success, `404` if the IP or named grouping does not exist. ## *Status* endpoints ### Get all status ```bash curl http://127.0.0.1:8090/status ``` Returns the configuration and current status of every IP and grouping in the configuration. Status indicates if the schedule is currently on track to meet the target volumes, finished, or paused. Information about the current day in the schedule is also included. *Success (200) response* ```json { "ips": { "10.0.0.2": { "groupings": [ { "current_day": 1, "end": { "date": "2025-08-03", "volume": 30000 }, "mode": "MANUAL", "paused": false, "algorithm":"TOKENBUCKET", "interval": 86400, "properties": null, "schedule": [ { "date": "2025-08-01", "day": 1, "messages": 500 }, { "date": "2025-08-02", "day": 2, "messages": 600 }, { "date": "2025-08-03", "day": 3, "messages": 30000 } ], "start": { "date": "2025-08-01", "volume": 500 }, "status": "ONSCHEDULE" } ] } } } ``` ### Get status for an IP ```bash curl http://127.0.0.1:8090/status/ ``` This returns the data for a single IP, e.g. *Success (200) response* ```json { "groupings": [ { "current_day": 1, "end": { "date": "2025-08-03", "volume": 30000 }, "grouping": "&manual_example", "mode": "MANUAL", "paused": false, "algorithm":"TOKENBUCKET", "interval": 86400, "properties": null, "schedule": [ { "date": "2025-08-01", "day": 1, "messages": 500 }, { "date": "2025-08-02", "day": 2, "messages": 600 }, { "date": "2025-08-03", "day": 3, "messages": 30000 } ], "start": { "date": "2025-08-01", "volume": 500 }, "status": "ONSCHEDULE" } ], "timezone_offset": 0 } ``` ### Get status for an IP and grouping ```bash curl http://127.0.0.1:8090/status// ``` *Success (200) response* ```json { "current_day": 1, "end": { "date": "2025-08-03", "volume": 30000 }, "mode": "MANUAL", "paused": false, "properties": null, "algorithm":"TOKENBUCKET", "interval": 86400, "schedule": [ { "date": "2025-08-01", "day": 1, "messages": 500 }, { "date": "2025-08-02", "day": 2, "messages": 600 }, { "date": "2025-08-03", "day": 3, "messages": 30000 } ], "start": { "date": "2025-08-01", "volume": 500 }, "status": "ONSCHEDULE" } ``` # Changelog ## 1.1.1 Released 2026-11-13 - **Fix** Performance improvements ## 1.1.0 Released 2026-05-25 - **Imp** Add support for configuring interval for schedules - **Imp** Add support for configuring rate algorithm for schedules - **Imp** Add TCP keep-alive to websockets - **Bug** Fix issue with timezones - **Bug** Improved validation of requests for API endpoints ## 1.0.10 Released 2026-04-27 - **Fix** Updated third-party dependencies ## 1.0.9 Released 2026-04-10 - **Bug** Fix issue where it refused to start if no Elasticsearch index had been created yet - **Bug** Fix issue with policy synchronization ## 1.0.8 Released 2026-03-23 - **Pkg** Updated third-party dependencies ## 1.0.7 Released 2026-03-05 - **Fix** `stats.path` no longer required, set it to write stats files; leave it empty to turn stats off - **Pkg** Updated third-party dependencies ## 1.0.6 Released 2026-02-09 - **Bug** Fixed rare panic in Elasticsearch client library - **Pkg** Updated third-party dependencies ## 1.0.5 Released 2026-01-07 - **Pkg** Updated third-party dependencies ## 1.0.4 Released 2025-10-03 - **Sec** Replaced `serde_yml` dependency with `serde_norway` to address a security vulnerability ## 1.0.3 Released 2025-07-29 - **Bug** Bundle Elasticsearch index pattern ## 1.0.2 Released 2025-07-24 - **Bug** Fixed issue with schedule calculation ## 1.0.1 Released 2025-07-12 - **Bug** Fixed issue with config update API endpoint ## 1.0.0 Released 2025-07-04 - **New** Initial release