3. Command line interface

There are three primary ways of managing Halon MTA hosts (or clusters of); the web administration, API (Protocol Buffer or REST) or CLI. The Halon MTA package comes with a program called halonctl that can be used to manage any aspect of the MTA. This section is a guide for that program.

Every command have a --help flag to display available options.

The halonctl program communicates with the Halon MTA server processes over control sockets as described by the Protocol Buffer schemas. The raw response data can be useful when using the CLI for scripting with tools like jq, and the request data can be useful when learning about the API. Therefore, the commands have a --json output flag making it return the raw response and a --json-request flag that prints the raw request (JSON encoded).

3.1. Queue management

These sub-commands operate on the queue (which contains queued email messages) in various ways. Please see the queue section for more information about what the queue is, and how it works. Because the metadata for all messages in queue is loaded into memory, the MTA can quickly summarise and modify large number of messages. The list, groupby and update sub-commands share the same condition arguments, which are used to filter the result:

$ halonctl queue list -h
...
Conditions:
--transportid               Filter on transport ID (repeatable) <string>
--jobid                     Filter on job ID (repeatable) <string>
--senderdomain              Filter on sender domain (repeatable) <domain>
--recipientdomain           Filter on recipient domain (repeatable) <domain>
--sender                    Filter on sender (repeatable) <address>
--recipient                 Filter on recipient (repeatable) <address>
--id                        Filter on ID (repeatable) <id>
--state                     Filter on state (repeatable) <state>
--retrycount                Filter on retry count (operators, repeatable) <number>
--retrydelay                Filter on next retry (operators, repeatable) <seconds>
--age                       Filter on age (operators, repeatable) <seconds>
--ts                        Filter on recevied time (operators, repeatable) <iso8601>
--metadata                  Filter on metadata <key=value>

Available states are:
queue:   ACTIVE, DEFER
working: SENDING, UNKNOWN
freeze:  HOLD, UPDATE

Number and date format:
operators: >, >=, <, <= and = (eg ">2020-01-01T01:02:03")
range:     .. (eg "10..20")
...

3.1.1. List messages

This command shows information about messages in queue, filtered by the conditions described above. There are a few pre-defined views for showing the defer queue, active queue and open connections with relevant table fields pre-selected.

$ halonctl queue list --age '>3600' --state DEFER -F id -F retrycount -F retrydelay -F recipient
┌────────────────────────────────────────┬────────────┬────────────┬─────────────────┐
│ id                                     │ retrycount │ retrydelay │ recipient       │
├────────────────────────────────────────┼────────────┼────────────┼─────────────────┤
│ c6321f4c-1671-11ea-885e-005056914940:1 │         20 │        243 │ [email protected]
│ f640a91a-1671-11ea-b6ea-005056914940:1 │         21 │         70 │ [email protected]
...

3.1.2. Take actions on messages

The queue update command can take various actions on messages, filtered by the conditions described above.

  • --delete remove messages, without sending bounces
  • --bounce remove messages, and generate bounce messages to the sender
  • --active move messages (that are deferred or on hold) to the active queue for imminent delivery
  • --defer move messages (that are in active queue or on hold) to the defer queue for the specified number of seconds (number)
  • --hold put queued messages on hold
  • --updatetransportid changes the messages’ transport ID to the one specified by the argument (string)
  • --updatemetadata modify or add auxiliary metadata to the messages as specified by the argument (in format field=value)
$ halonctl queue update --age '>3600' --state DEFER --state ACTIVE --bounce
230 messages affected
$ halonctl queue update --transportid t1 --state DEFER --updatetransportid t2
27854 messages affected

3.1.2.1. Actions and working states

Messages in a working state (delivery attempts, script execution, DNS resolution, etc) cannot be directly modified, and they will be reported as unaffected by the command. Sometimes it is important to perform a task on all matching messages, even those that are in a working state. For example, you might want to delete every possible queued message from a certain sender domain. To do this, re-run the command with the --freeze flag. This will capture those messages once they are no longer in a working state and put them in the so-called UPDATE state (given that they still in in queue; messages that have been delivered or deleted can not be captured). No messages will be reported as unaffected; they will instead be reported as freezing. You should then monitor the queue.freeze.update.pending process counter and wait for it to reach 0. When it does, every messages that could not be immediately modified (but which is still in queue) will be in the UPDATE state, ready for you to modify them.

$ halonctl queue update --delete --senderdomain example.com --all-states
100 messages affected
30 messages unaffected
$ halonctl queue update --delete --senderdomain example.com --all-states --freeze
4 messages affected
26 messages freezing
$ halonctl process-stats | grep queue.freeze.update
queue.freeze.update.size: 6
queue.freeze.update.pending: 20
$ halonctl process-stats | grep queue.freeze.update
queue.freeze.update.size: 22
queue.freeze.update.pending: 0
$ halonctl queue update --delete --senderdomain example.com --state UPDATE
22 messages affected

3.1.3. Show messages distribution

This command shows distribution of messages in queue, filtered by the conditions described above. There are a few pre-defined views for showing the distribution by for example age.

$ halonctl queue groupby --view state
┌─────────────┬─────────────────┬───────┬────────┬───────┬──────┬─────────┬─────────┐
│ transportid │ recipientdomain │ total │ ACTIVE │ DEFER │ HOLD │ UNKNOWN │ SENDING │
├─────────────┼─────────────────┼───────┼────────┼───────┼──────┼─────────┼─────────┤
│ inbound     │ example.com     │    19 │      0 │    17 │    1 │       0 │       1 │
...

3.1.4. Active queue policies

These sub-commands can add, list and delete dynamic active queue policies that overrides the ones in the environment.policyconf configuration, as well as reload that configuration file.

$ halonctl queue policy add -F remotemx --remotemx *.example.com --rate 10/600 --ttl 3600
23244363005953
$ halonctl queue policy list
┌────────────────┬──────────┬─────────────┬─────────┬──────────┬────────────────┬─────────────────┬───────┬─────────────┬────────┬─────────┬─────┐
│             id │ fields   │ transportid │ localip │ remoteip │ remotemx       │ recipientdomain │ jobid │ concurrency │   rate │     ttl │ tag │
├────────────────┼──────────┼─────────────┼─────────┼──────────┼────────────────┼─────────────────┼───────┼─────────────┼────────┼─────────┼─────┤
│ 23244363005953 │ remotemx │             │         │          │ *.example.com  │                 │       │             │ 10/600 │ 3592.44 │     │
└────────────────┴──────────┴─────────────┴─────────┴──────────┴────────────────┴─────────────────┴───────┴─────────────┴────────┴─────────┴─────╯
$ halonctl queue policy delete 23244363005953

It is also possible to clear or partially refill active queue policy rates, for a specific policy rate “bucket” (matched exactly by the optional condition arguments). If you run it without specifying a bucket, it will clear all rate buckets, created by any policy, and return the number of buckets it cleared/refilled.

$ halonctl queue policy rate refill --localip 198.51.100.4 --remotemx mx1.example.org
1

3.1.5. Active queue suspensions

These sub-commands can add, list and delete dynamic active queue suspension that overrides the ones in the environment.suspendconf configuration, as well as reload that configuration file.

$ halonctl queue suspend add --remotemx *.example.com --ttl 60
23244363005953
$ halonctl queue suspend list
┌────────────────┬─────────┬─────────────┬─────────┬──────────┬───────────────┬─────────────────┬───────┬─────────┬─────┐
│             id │ source  │ transportid │ localip │ remoteip │ remotemx      │ recipientdomain │ jobid │     ttl │ tag │
├────────────────┼─────────┼─────────────┼─────────┼──────────┼───────────────┼─────────────────┼───────┼─────────┼─────┤
│ 23244363005953 │ dynamic │             │         │          │ *.example.com │                 │       │ 57.7753 │     │
└────────────────┴─────────┴─────────────┴─────────┴──────────┴───────────────┴─────────────────┴───────┴─────────┴─────╯
$ halonctl queue suspend delete 23244363005953

3.1.6. Active queue delivery settings

This sub-command can reload the active queue delivery settings configuration file environment.deliveryconf.

$ halonctl queue delivery reload

3.1.7. Moving the queue files

If you need to move the queued message files (for example from one MTA to another) it is important to first “unload” them. If you want to move the entire queue, the easiest way is to simply shut down the server process. To move specific messages, it is recommended to use the queue unload command, filtered by the conditions described above.

  • --partial match a queued message, even if only one of its recipients matches
  • --freeze freeze messages that are in a working state
$ halonctl queue unload --transportid t1
/storage/mail/spool/13/13ef04db-1e06-11ea-927e-0050569a4c9c.hqf unloaded
/storage/mail/spool/1b/1b5a5e80-1da2-11ea-927e-0050569a4c9c.hqf unloaded
...
$ halonctl queue unload
No messages unloaded

3.2. Configuration management

These sub-commands can reload the running configuration (including its Halon script) and control the built-in blue-green testing that we call “live stage”. See the configuration section for more information.

3.2.1. Script syntax and packing

The script is normally edited as individual files using the integrated package’s IDE, the Visual Studio Code plugin or simply a text editor. Those are then checked and “packed” into an actual configuration file by the halonconfig script as described in the script directive section.

Given that the folder test contains halonconfig’s standard file structure (see /opt/halon/share/examples) you can pack, validate and reload the configuration directly on the MTA by:

$ cd test
(editing configuration files)
$ halonconfig
$ sudo cp dist/* /etc/halon/
$ sudo service halon reload

3.2.2. Reloading configurations

You can also invoke the individual reload commands directly:

$ halonctl config reload --program smtpd   # /etc/halon/smtpd-app.yaml
$ halonctl config reload --program rated   # /etc/halon/rated-app.yaml
$ halonctl config reload --program dlpd    # /etc/halon/dlpd-app.yaml
$ halonctl queue policy reload             # /etc/halon/smtpd-policy.yaml
$ halonctl queue suspend reload            # /etc/halon/smtpd-suspend.yaml
$ halonctl queue delivery reload           # /etc/halon/smtpd-delivery.yaml

3.2.3. Blue-green testing

The Halon MTA has built-in blue-green testing, which allows you to deploy a parallel running configuration for only a selection of the traffic, for example connections from certain IP addresses or a certain percentage chosen by random. The following example will deploy a parallel configuration for 10% of connecting clients during one hour:

$ halonctl config livestage start --id t1 --file dist/smtpd-app.yaml --probability 0.1 --time 3600
$ halonctl config livestage status
t1: 2 connections(s), 30/3600 runtime
$ halonctl config livestage cancel

3.2.4. Remote configuration

The halonconfig script can be used to pack and validate configuration folders on any computer that runs Python. You can copy it from the MTA:

$ python -m venv env
$ source env/bin/activate
$ pip install pyyaml
$ pip install jsonschema
$ scp mta:/opt/halon/bin/halonconfig .
$ scp mta:/opt/halon/share/json-schemas json

and run it by specifying the JSON schema path and disabling the script linter:

$ python halonconfig --schema-dir json --no-check-hsl

You can start a live stage directly from a remote computer by using SSH stdin redirection:

$ cat dist/smtpd-app.yaml | ssh mta "halonctl config livestage start --file /dev/stdin --id t2"
$ ssh mta "halonctl config livestage status"
t2: 2 connections(s), 3 runtime

3.3. Script engine

These sub-commands can work on various parts of the Halon script engine. The actual script is managed via the configuration commands.

3.3.1. Rate function

These sub-commands can list and clear the built-in and cluster-aware rate limit functionality implemented by the rated program and available via the script engine’s rate() function.

$ halonctl hsl rates list
┌───────────────────┬────────────┬───────┬─────────────┬────────┬────────┬────────────┐
│ Namespace         │ Entry      │ Count │ Local count │ Oldest │ Newest │ Rate (#/s) │
├───────────────────┼────────────┼───────┼─────────────┼────────┼────────┼────────────┤
│ delivery-failures │ customer0  │     1 │           1 │      8 │      8 │          0 │
│ delivery-failures │ customer1  │     5 │           5 │     12 │      4 │      0.625 │
│ delivery-failures │ customer10 │     1 │           1 │      8 │      8 │          0 │
...
$ halonctl hsl rates clear --entry customer10
1 entries deleted

3.3.2. Shared memory functions

These sub-commands can store, fetch and delete key/value pairs from the the script engine’s shared memory API. In order to be able to work with complex Halon script data structures, the values are JSON encoded.

$ halonctl hsl memory store theanswer 42
$ halonctl hsl memory fetch theanswer
42
$ halonctl hsl memory delete theanswer

3.3.3. Cache statement

These sub-commands can list and clear the script engine’s function cache buckets.

$ halonctl hsl cache list
┌───────────┬──────────────────┬──────────┬──────┬────────┬────────┬────────┬──────────┐
│ Namespace │ Function         │ Max size │ Size │   Hits │ Misses │ Evicts │ Hit rate │
├───────────┼──────────────────┼──────────┼──────┼────────┼────────┼────────┼──────────┤
│           │ api_call_http    │   100000 │  845 │ 122799 │   6041 │      0 │  95.3112 │
│           │ http             │    16384 │  747 │   5670 │   2850 │      0 │  66.5493 │
│ test      │ smtp_lookup_rcpt │    16384 │    2 │      1 │     31 │      0 │    3.125 │
└───────────┴──────────────────┴──────────┴──────┴────────┴────────┴────────┴──────────╯
$ halonctl hsl cache clear --function http
1 buckets cleared

3.4. DNS resolver

These commands can clear and view statistics about the asynchronous DNS resolver’s built-in cache.

$ halonctl process-stats | grep resolver.cache
resolver.cache.size: 5
resolver.cache.hits: 897
resolver.cache.misses: 1624
resolver.cache.expires: 1566
resolver.cache.evicts: 0
resolver.cache.skips: 0
$ halonctl resolver clear

3.5. Statistics and information

Shows information about process statistics (cache hits, internal queue sizes) and other information such as email queue size.

$ halonctl process-stats
process.pid: 48535
process.runtime: 3638
...
Name Description
process.pid The smtpd process PID
process.runtime Seconds that smtpd has been running
resolver.pending DNS queries waiting in line to be sent
resolver.running DNS queries waiting for response, max is resolver.concurrency
resolver.cache.size Entries in cache, max is resolver.cache.size
resolver.cache.hits DNS requests that were found in cached
resolver.cache.misses DNS requests not found in cache
resolver.cache.expires Entries dropped due to TTL, see resolver.cache.ttl.min
resolver.cache.evicts Entries dropped due to LRU eviction
resolver.cache.skips Entries not cached (eg. bad responses)
servers[].serverid The virtual servers[] ID
servers[].connections.concurrent Connected clients, max is servers[].concurrency.total
servers[].scripts.connect.pending Waiting in line for servers[].phases.connect.hook
servers[].scripts.connect.running Executing the above, max is servers[].threads.script
servers[].scripts.connect.finished Completed script executions
servers[].scripts.proxy.pending
servers[].scripts.proxy.running
servers[].scripts.proxy.finished
servers[].scripts.helo.pending
servers[].scripts.helo.running
servers[].scripts.helo.finished
servers[].scripts.auth.pending
servers[].scripts.auth.running
servers[].scripts.auth.finished
servers[].scripts.mailfrom.pending
servers[].scripts.mailfrom.running
servers[].scripts.mailfrom.finished
servers[].scripts.rcptto.pending
servers[].scripts.rcptto.running
servers[].scripts.rcptto.finished
servers[].scripts.eod.pending
servers[].scripts.eod.running
servers[].scripts.eod.finished
queue.loader.count Email loaded into the message queue
queue.loader.pending Email waiting in line to be loaded
queue.scripts.predelivery.pending Waiting in line for scripting.hooks.predelivery
queue.scripts.predelivery.running Executing the above, max is queues.threads.script
queue.scripts.predelivery.finished Completed script executions
queue.scripts.postdelivery.pending
queue.scripts.postdelivery.running
queue.scripts.postdelivery.finished
queue.queue.defer.size Messages in defer queue
queue.queue.active.size Messages in active queue
queue.freeze.hold.size Messages on hold
queue.freeze.update.size Messages frozen for update
queue.freeze.update.pending Messages in working state waiting to be frozen for update
queue.policy.concurrency.counters Concurrency counters in memory created by active queue policies
queue.policy.concurrency.suspends Suspensions created as a result of exceeded concurrency
queue.policy.rate.buckets Rate buckets in memory created by active queue policies
queue.policy.rate.suspends Suspensions created as a result of exceeded rate
queue.policy.dynamic.suspends Number of suspends added via API/CLI or script
queue.policy.dynamic.conditions Number of policy conditions added via API/CLI or script
queue.pickup.skips Number of skips in queue pickup algorithm due to suspends
queue.pickup.misses Number of times no were picked up because all were suspended
queue.quota.size Number of quota name entries
queue.connections.concurrent Open connections, max is queues.concurrency.total
queue.connections.pooling.size Cached connections in pool for reuse, max is queues.pooling.size
queue.connections.pooling.hits Message deliveries that could re-use a cached connections
queue.connections.pooling.misses Message deliveries that had to open a new conenctions
queue.connections.pooling.expires Connections closed due to idle timeout
queue.connections.pooling.evicts Connections closed due to LRU eviction
queue.connections.pooling.skips If the pooling cache is full and all items are non-evictable

3.6. Script interpreter

The hsh program is a Halon script interpreter and interactive shell (REPL). It allows you to easily write, run and test Halon script without having to deploy it into the Halon MTA smtpd server. It implements the full standard library, but not any smtpd server or queue hooks. It also doesn’t provide an interface for halonctl’s script commands.

3.6.1. Options

-c, --config path

In order to make the hsh script environment resemble the environment of smtpd, it supports loading smtpd’s startup configuration file (default /etc/halon/smtpd.yaml) which in turn load the running configuration. This allows all paths and permission to be set up correctly, as well as virtual files scripting.files[] and certificates pki.private[]. Since that file usually drop privileges and change user, it might require hsh -P to be run as root. If you don’t need the full environment, you may run without a startup configuration: /dev/null.

-P, --privdrop

Drop privileges (setuid / setgid) to the user specificed in the startup configuration. This may allow you to have access to various sockets, but requires root permission. The recommended setup would be to have shared group permission with the smtpd server on various sockets and paths.

-F, --ffi

Enable FFI support, which is equivalent to scripting.ffi, without having to use hsh -c to load a startup a configuration.

-R, --rootpath path

Set the script root path, which is equivalent to scripting.rootpath, without having to use hsh -c to load a startup a configuration.

-A, --appconf path

Load a specific running configuration which is equivalent to environment.appconf, without having to use hsh -c to load a startup a configuration.

-s, --syntax

Check the script syntax only.

-v, --version

Display Halon MTA version and exit.

3.6.2. Script interpreter

The hsh program may be executed with a script file as the entry point for the script execution. It should be given as the last argument to the hsh command line.

$ cat test.hsl
echo "Hello World";
$ hsh test.hsl
Hello World

Setting the root to the current directory may be useful when implementing unit testing of modules:

$ cat module.hsl
function foo() {
   echo "Hello World";
}
$ cat test.hsl
import { foo } from "file://module.hsl";
foo();
$ hsh -R . test.hsl
Hello World

3.6.3. Interactive shell

The hsh program may be executed without a file, in which case it enters the interactive shell mode (REPL). Statements are executed line-by-line.

$ hsh
HSH> echo "Hello World";
Hello World
HSH>