5. Command line interface

There are three primary ways of managing hosts (or clusters of); the web administration, API (Protocol Buffer) or CLI. The smtpd package comes with a program called halonctl that can be used to manage any aspect of the MTA. This section is a guide for that program.

Every command have a --help flag to display available options.

The halonctl program communicates with the smtpd processes over control sockets as described by the Protocol Buffer schemas. The raw response data can be useful when using the CLI for scripting with tools like jq, and the request data can be useful when learning about the API. Therefore, the commands have a --json output flag making it return the raw response and a --json-request flag that prints the raw request (JSON encoded).

5.1. Queue management

These sub-commands operate on the queue (which contains queued email messages) in various ways. Please see the queue section for more information about what the queue is, and how it works. Because the metadata for all messages in queue is loaded into memory, the MTA can quickly summarise and modify large number of messages. The list, groupby and update sub-commands share the same condition arguments, which are used to filter the result:

$ halonctl queue list -h
...
Conditions:
--transportid               Filter on transport ID (repeatable) <string>
--jobid                     Filter on job ID (repeatable) <string>
--senderdomain              Filter on sender domain (repeatable) <domain>
--recipientdomain           Filter on recipient domain (repeatable) <domain>
--sender                    Filter on sender (repeatable) <address>
--recipient                 Filter on recipient (repeatable) <address>
--id                        Filter on ID (repeatable) <id>
--state                     Filter on state (repeatable) <state>
--retrycount                Filter on retry count (operators, repeatable) <number>
--retrydelay                Filter on next retry (operators, repeatable) <seconds>
--age                       Filter on age (operators, repeatable) <seconds>
--ts                        Filter on recevied time (operators, repeatable) <iso8601>
--metadata                  Filter on metadata <key=value>

Available states are:
queue:   ACTIVE, DEFER
working: SENDING, UNKNOWN
freeze:  HOLD, UPDATE

Number and date format:
operators: >, >=, <, <= and = (eg ">2020-01-01T01:02:03")
range:     .. (eg "10..20")
...

5.1.1. List messages

This command shows information about messages in queue, filtered by the conditions described above. There are a few pre-defined views for showing the defer queue, active queue and open connections with relevant table fields pre-selected.

$ halonctl queue list --age '>3600' --state DEFER -F id -F retrycount -F retrydelay -F recipient
┌────────────────────────────────────────┬────────────┬────────────┬─────────────────┐
│ id                                     │ retrycount │ retrydelay │ recipient       │
├────────────────────────────────────────┼────────────┼────────────┼─────────────────┤
│ c6321f4c-1671-11ea-885e-005056914940:1 │         20 │        243 │ [email protected]
│ f640a91a-1671-11ea-b6ea-005056914940:1 │         21 │         70 │ [email protected]
...

5.1.2. Export messages

This command exports a messages in queue in “message/rfc822” format. If only a messageid (and no :queueid is provided) the original message (as received) will be exported.

$ halonctl queue export c6321f4c-1671-11ea-885e-005056914940
...
$ halonctl queue export c6321f4c-1671-11ea-885e-005056914940:1
...

5.1.3. Take actions on messages

The queue update command can take various actions on messages, filtered by the conditions described above.

  • --delete remove messages, without sending bounces

  • --reason set the DSN diagnosticcode when deleting the message (“Message could not be delivered in the allotted time frame”)

  • --bounce remove messages, and generate bounce messages to the sender

  • --dsn-status set the DSN status when bouncing the message (5.4.7)

  • --dsn-diagnosticcode set the DSN diagnosticcode when bouncing the message (“Message could not be delivered in the allotted time frame”)

  • --active move messages (that are deferred or on hold) to the active queue for imminent delivery

  • --defer move messages (that are in active queue or on hold) to the defer queue for the specified number of seconds (number)

  • --retrytsjitter apply an optional jitter so that messages are spread evenly between the defer time and an additional jitter in seconds

  • --hold put queued messages on hold

  • --updatetransportid changes the messages’ transport ID to the one specified by the argument (string)

  • --updatemetadata modify or add auxiliary metadata to the messages as specified by the argument (in format field=value)

  • --updatepriority change the priority of the message (0 - normal, 1 - high, number)

$ halonctl queue update --age '>3600' --state DEFER --state ACTIVE --bounce
230 messages affected
$ halonctl queue update --transportid t1 --state DEFER --updatetransportid t2
27854 messages affected

5.1.3.1. Actions and working states

Messages in a working state (delivery attempts, script execution, DNS resolution, etc) cannot be directly modified, and they will be reported as unaffected by the command. Sometimes it is important to perform a task on all matching messages, even those that are in a working state. For example, you might want to delete every possible queued message from a certain sender domain. To do this, re-run the command with the --freeze flag. This will capture those messages once they are no longer in a working state and put them in the so-called UPDATE state (given that they still in in queue; messages that have been delivered or deleted can not be captured). No messages will be reported as unaffected; they will instead be reported as freezing. You should then monitor the queue.freeze.update.pending process counter and wait for it to reach 0. When it does, every messages that could not be immediately modified (but which is still in queue) will be in the UPDATE state, ready for you to modify them.

$ halonctl queue update --delete --senderdomain example.com --all-states
100 messages affected
30 messages unaffected
$ halonctl queue update --delete --senderdomain example.com --all-states --freeze
4 messages affected
26 messages freezing
$ halonctl process-stats | grep queue.freeze.update
queue.freeze.update.size: 6
queue.freeze.update.pending: 20
$ halonctl process-stats | grep queue.freeze.update
queue.freeze.update.size: 22
queue.freeze.update.pending: 0
$ halonctl queue update --delete --senderdomain example.com --state UPDATE
22 messages affected

5.1.4. Show messages distribution

This command shows distribution of messages in queue, filtered by the conditions described above. There are a few pre-defined views for showing the distribution by for example age.

$ halonctl queue groupby --view state
┌─────────────┬─────────────────┬───────┬────────┬───────┬──────┬─────────┬─────────┐
│ transportid │ recipientdomain │ total │ ACTIVE │ DEFER │ HOLD │ UNKNOWN │ SENDING │
├─────────────┼─────────────────┼───────┼────────┼───────┼──────┼─────────┼─────────┤
│ inbound     │ example.com     │    19 │      0 │    17 │    1 │       0 │       1 │
...

5.1.5. Active queue policies

These sub-commands can add, list and delete dynamic active queue policies that overrides the ones in the environment.policyconf configuration, as well as reload that configuration file.

$ halonctl queue policy add -F remotemx --remotemx *.example.com --rate 10/600 --ttl 3600
30be002e-7436-11eb-ae42-5254004d77d3
$ halonctl queue policy list
┌──────────────────────────────────────┬──────────┬─────────────┬─────────┬──────────┬────────────────┬─────────────────┬───────┬──────────┬─────────────┬────────┬─────────┬─────┐
│                                   id │ fields   │ transportid │ localip │ remoteip │ remotemx       │ recipientdomain │ jobid │ grouping │ concurrency │   rate │     ttl │ tag │
├──────────────────────────────────────┼──────────┼─────────────┼─────────┼──────────┼────────────────┼─────────────────┼───────┼──────────┼─────────────┼────────┼─────────┼─────┤
│ 30be002e-7436-11eb-ae42-5254004d77d3 │ remotemx │             │         │          │ *.example.com  │                 │       │          │             │ 10/600 │ 3592.44 │     │
└──────────────────────────────────────┴──────────┴─────────────┴─────────┴──────────┴────────────────┴─────────────────┴───────┴──────────┴─────────────┴────────┴─────────┴─────╯
$ halonctl queue policy delete 30be002e-7436-11eb-ae42-5254004d77d3

It is also possible to clear or partially refill active queue policy rates, for a specific policy rate “bucket” (matched exactly by the optional condition arguments). If you run it without specifying a bucket, it will clear all rate buckets, created by any policy, and return the number of buckets it cleared/refilled.

$ halonctl queue policy rate refill --localip 198.51.100.4 --remotemx mx1.example.org
1

5.1.6. Active queue suspensions

These sub-commands can add, list and delete dynamic active queue suspension that overrides the ones in the environment.suspendconf configuration, as well as reload that configuration file.

$ halonctl queue suspend add --remotemx *.example.com --ttl 60
803c41d5-4a8e-424a-97e9-f8db55f69179
$ halonctl queue suspend list
┌──────────────────────────────────────┬─────────┬─────────────┬─────────┬──────────┬───────────────┬─────────────────┬───────┬──────────┬─────────┬─────┐
│                                   id │ source  │ transportid │ localip │ remoteip │ remotemx      │ recipientdomain │ jobid │ grouping │     ttl │ tag │
├──────────────────────────────────────┼─────────┼─────────────┼─────────┼──────────┼───────────────┼─────────────────┼───────┼──────────┼─────────┼─────┤
│ 803c41d5-4a8e-424a-97e9-f8db55f69179 │ dynamic │             │         │          │ *.example.com │                 │       │          │ 57.7753 │     │
└──────────────────────────────────────┴─────────┴─────────────┴─────────┴──────────┴───────────────┴─────────────────┴───────┴──────────┴─────────┴─────╯
$ halonctl queue suspend delete 803c41d5-4a8e-424a-97e9-f8db55f69179

5.1.7. Active queue delivery settings

This sub-command can reload the active queue delivery settings configuration file environment.deliveryconf.

$ halonctl queue delivery reload

5.1.8. Moving the queue files

If you need to move the queued message files (for example from one MTA to another) it is important to first “unload” them. If you want to move the entire queue, the easiest way is to simply shut down the server process and use the hqfmove tool. To move specific messages, it is recommended to use the queue unload command, filtered by the conditions described above.

  • --partial match a queued message, even if only one of its recipients matches

  • --freeze freeze messages that are in a working state

$ halonctl queue unload --transportid t1
/storage/mail/spool/13/13ef04db-1e06-11ea-927e-0050569a4c9c.hqf unloaded
/storage/mail/spool/1b/1b5a5e80-1da2-11ea-927e-0050569a4c9c.hqf unloaded
...
$ halonctl queue unload
No messages unloaded

5.1.9. Trace messages

This command allows you to get debug or trace information regarding message deliveries in queue, filtered by the queue pickup fields available (eg. --remotemx '*.example.com') or a specific transaction id.

Additional options are available to control the pooling (--connect) and PIPELINING (--no-pipelining) behaviour.

$ halonctl queue trace
2023-03-28 21:59:23.076 Begin trace of message 18096a8d-cba3-11ed-b464-930f4c7d94ae:1
2023-03-28 21:59:23.076 Connecting from [1.2.3.4] to [5.6.7.8]:25 (mx.example.com) (DNSSEC)
2023-03-28 21:59:23.079 Connected
2023-03-28 21:59:23.083 [SMTP] < 220 mx.example.com ESMTP
2023-03-28 21:59:23.083 [SMTP] > EHLO mx.example.org
...

5.1.10. Loading queue files

If you need to manually import a queue file into a running MTA instance you can use the queue load command. The “load” command will copy the .hqf specified to the spool folder leaving the original file in place. The message ID (in the HQF file’s ID field) has to be an unique and will be used as the target file name (regardless of the imported source file name). If the HQF file’s ID field is an all zero UUID (00000000-0000…) then a new UUID will be generated.

  • --hqf path to the queue file

$ halonctl queue load --hqf 13ef04db-1e06-11ea-927e-0050569a4c9c.hqf
Message loaded

5.2. Configuration management

These sub-commands can reload the running configuration (including its Halon script) and control the built-in blue-green testing that we call “live stage”. See the configuration section for more information.

5.2.1. Script syntax and packing

The script is normally edited as individual files using the Visual Studio Code integration or simply a text editor. Those are then checked and “packed” into an actual configuration file by the halonconfig script as described in the script directive section.

Given that the folder test contains halonconfig’s standard file structure (see /opt/halon/share/examples) you can pack, validate and reload the configuration directly on the MTA by:

$ cd test
(editing configuration files)
$ halonconfig
$ sudo cp dist/* /etc/halon/
$ sudo service halon reload

If you want to exclude any files inside the src/files folder from being packed you can add a .halonignore file which supports regular glob patterns.

The halonconfig command may also be used to “unpack” the configuration, reversing the described process above.

5.2.2. Reloading configurations

You can also invoke the individual reload commands directly:

$ halonctl config reload                   # /etc/halon/smtpd-app.yaml
$ halonctl queue policy reload             # /etc/halon/smtpd-policy.yaml
$ halonctl queue suspend reload            # /etc/halon/smtpd-suspend.yaml
$ halonctl queue delivery reload           # /etc/halon/smtpd-delivery.yaml
$ ratectl reload                           # /etc/halon/rated-app.yaml
$ dlpctl reload                            # /etc/halon/dlpd-app.yaml

5.2.3. Blue-green testing

Halon has built-in blue-green testing, which allows you to deploy a parallel running configuration for only a selection of the traffic, for example connections from certain IP addresses or a certain percentage chosen by random. The following example will deploy a parallel configuration for 10% of connecting clients during one hour:

$ halonctl config livestage start --id t1 --file dist/smtpd-app.yaml --probability 0.1 --time 3600
$ halonctl config livestage status
t1: 2 connections(s), 30/3600 runtime
$ halonctl config livestage cancel

5.2.4. Remote configuration

The halonconfig script can be used to pack and validate configuration folders on any computer that runs Python. You can copy it from the MTA:

$ python -m venv env
$ source env/bin/activate
$ pip install pyyaml
$ pip install jsonschema
$ scp mta:/opt/halon/bin/halonconfig .
$ scp mta:/opt/halon/share/json-schemas json

and run it by specifying the JSON schema path and disabling the script linter:

$ python halonconfig --schema-dir json --no-check-hsl

You can start a live stage directly from a remote computer by using SSH stdin redirection:

$ cat dist/smtpd-app.yaml | ssh mta "halonctl config livestage start --file /dev/stdin --id t2"
$ ssh mta "halonctl config livestage status"
t2: 2 connections(s), 3 runtime

5.3. Script engine

These sub-commands can work on various parts of the Halon script engine. The actual script is managed via the configuration commands.

5.3.1. Rate function

These sub-commands can list and clear the cluster-aware rate limit functionality implemented by the rated program and available via the rate() function exported by the rate limiting client plugin.

$ ratectl list
┌───────────────────┬────────────┬───────┬─────────────┬────────┬────────┬────────────┐
│ Namespace         │ Entry      │ Count │ Local count │ Oldest │ Newest │ Rate (#/s) │
├───────────────────┼────────────┼───────┼─────────────┼────────┼────────┼────────────┤
│ delivery-failures │ customer0  │     1 │           1 │      8 │      8 │          0 │
│ delivery-failures │ customer1  │     5 │           5 │     12 │      4 │      0.625 │
│ delivery-failures │ customer10 │     1 │           1 │      8 │      8 │          0 │
...
$ ratectl clear --entry customer10
1 entries deleted

5.3.2. Shared memory functions

These sub-commands can store, list, fetch and delete key/value pairs from the the script engine’s shared memory API. In order to be able to work with complex Halon script data structures, the values are JSON encoded.

$ halonctl hsl memory store theanswer 42
$ halonctl hsl memory list
theanswer
$ halonctl hsl memory fetch theanswer
42
$ halonctl hsl memory delete theanswer

5.3.3. Cache statement

These sub-commands can list and clear the script engine’s function cache buckets.

$ halonctl hsl cache list
┌───────────┬──────────────────┬──────────┬──────┬────────┬────────┬────────┬──────────┐
│ Namespace │ Function         │ Max size │ Size │   Hits │ Misses │ Evicts │ Hit rate │
├───────────┼──────────────────┼──────────┼──────┼────────┼────────┼────────┼──────────┤
│           │ api_call_http    │   100000 │  845 │ 122799 │   6041 │      0 │  95.3112 │
│           │ http             │    16384 │  747 │   5670 │   2850 │      0 │  66.5493 │
│ test      │ smtp_lookup_rcpt │    16384 │    2 │      1 │     31 │      0 │    3.125 │
└───────────┴──────────────────┴──────────┴──────┴────────┴────────┴────────┴──────────╯
$ halonctl hsl cache clear --function http
1 buckets cleared

5.4. DNS resolver

These commands can clear and view statistics about the asynchronous DNS resolver’s built-in cache.

$ halonctl process-stats | grep resolver.cache
resolver.cache.size: 5
resolver.cache.hits: 897
resolver.cache.misses: 1624
resolver.cache.expires: 1566
resolver.cache.evicts: 0
resolver.cache.skips: 0
$ halonctl resolver clear

There is also a command to test the DNS resolvers behaviour as for choosing the next-hop based on a transactionid and retry count.

$ halonctl resolver query --seed 67b06545-2592-11eb-a941-f1a44e94c1b9 --retry 3 halon.io
address: 2a02:750:7:3305::ee
hostname: mx.halon.io
dnssec: true
tlsa:
  dnssec: true
  rr:
    - usage: 3
      selector: 1
      mtype: 1
      data: 42568063726264388f17d494cd3d01079ba1df4e60b1448e5670544cd7739218

5.5. Statistics and information

Shows information about process statistics (cache hits, internal queue sizes) and other information such as email queue size.

5.5.1. halontop

The halontop program shows a visual view of all counters with live updates and rates calcuations.

_images/halontop.png

5.5.2. halonctl

Text based output for counters may be extracted using halonctl suited for data collection.

$ halonctl process-stats
process.pid: 48535
process.runtime: 3638
...

Name

Description

process.pid

The smtpd process PID

process.runtime

Seconds that smtpd has been running

process.elapsed

Seconds that smtpd has been running (with fractions)

resolver.pending

DNS queries waiting in line to be sent

resolver.running

DNS queries waiting for response, max is resolver.concurrency

resolver.maxrunning

Maximum allowed DNS queries waiting for response, max is resolver.concurrency

resolver.cache.size

Entries in cache, max is resolver.cache.size

resolver.cache.maxsize

Maximum allowed entries in cache, max is resolver.cache.size

resolver.cache.hits

DNS requests that were found in cached

resolver.cache.misses

DNS requests not found in cache

resolver.cache.expires

Entries dropped due to TTL, see resolver.cache.ttl.min

resolver.cache.evicts

Entries dropped due to LRU eviction

resolver.cache.skips

Entries not cached (eg. bad responses)

servers[].serverid

The virtual servers[] ID

servers[].connections.concurrent

Connected clients, max is servers[].concurrency.total

servers[].connections.maxconcurrent

Maximum allowed connected clients, max is servers[].concurrency.total

servers[].scripts.connect.pending

Waiting in line for servers[].phases.connect.hook

servers[].scripts.connect.running

Executing the above, max is servers[].threads.script

servers[].scripts.connect.finished

Completed script executions

servers[].scripts.connect.errors

Script execution errors

servers[].scripts.proxy.pending

servers[].scripts.proxy.running

servers[].scripts.proxy.finished

servers[].scripts.proxy.errors

servers[].scripts.helo.pending

servers[].scripts.helo.running

servers[].scripts.helo.finished

servers[].scripts.helo.errors

servers[].scripts.auth.pending

servers[].scripts.auth.running

servers[].scripts.auth.finished

servers[].scripts.auth.errors

servers[].scripts.mailfrom.pending

servers[].scripts.mailfrom.running

servers[].scripts.mailfrom.finished

servers[].scripts.mailfrom.errors

servers[].scripts.rcptto.pending

servers[].scripts.rcptto.running

servers[].scripts.rcptto.finished

servers[].scripts.rcptto.errors

servers[].scripts.eod.pending

servers[].scripts.eod.running

servers[].scripts.eod.finished

servers[].scripts.eod.errors

servers[].scripts.disconnect.pending

servers[].scripts.disconnect.running

servers[].scripts.disconnect.finished

servers[].scripts.disconnect.errors

queue.loader.count

Email loaded into the message queue

queue.loader.pending

Email waiting in line to be loaded

queue.loader.active

Total amount of emails loaded into memory

queue.loader.maxactive

Max amount of emails loaded into memory (see queues.maxmessages)

queue.scripts.predelivery.pending

Waiting in line for scripting.hooks.predelivery

queue.scripts.predelivery.running

Executing the above, max is queues.threads.script

queue.scripts.predelivery.finished

Completed script executions

queue.scripts.predelivery.errors

Script execution errors

queue.scripts.postdelivery.pending

queue.scripts.postdelivery.running

queue.scripts.postdelivery.finished

queue.scripts.postdelivery.errors

queue.queue.defer.size

Messages in defer queue

queue.queue.active.size

Messages in active queue

queue.queue.active.priorities[].size

Messages in with the different priorities (0 - normal, 1 - high) queue

queue.freeze.hold.size

Messages on hold

queue.freeze.update.size

Messages frozen for update

queue.freeze.update.pending

Messages in working state waiting to be frozen for update

queue.policy.concurrency.counters

Concurrency counters in memory created by active queue policies

queue.policy.concurrency.suspends

Suspensions created as a result of exceeded concurrency

queue.policy.rate.buckets

Rate buckets in memory created by active queue policies

queue.policy.rate.suspends

Suspensions created as a result of exceeded rate

queue.policy.dynamic.suspends

Number of suspends added via API/CLI or script

queue.policy.dynamic.conditions

Number of policy conditions added via API/CLI or script

queue.pickup.count

Number of messages picked up from queue

queue.pickup.skips

Number of skips in queue pickup algorithm due to suspends

queue.pickup.misses

Number of times no were picked up because all were suspended

queue.quota.size

Number of quota name entries

queue.connections.concurrent

Open connections, max is queues.concurrency.total

queue.connections.maxconcurrent

Maximum allowed open connections, max is queues.concurrency.total

queue.connections.pooling.size

Cached connections in pool for reuse, max is queues.pooling.size

queue.connections.pooling.maxsize

Maximum allowed cached connections in pool for reuse, max is queues.pooling.size

queue.connections.pooling.hits

Message deliveries that could re-use a cached connections

queue.connections.pooling.misses

Message deliveries that had to open a new conenctions

queue.connections.pooling.expires

Connections closed due to idle timeout

queue.connections.pooling.evicts

Connections closed due to LRU eviction

queue.connections.pooling.skips

If the pooling cache is full and all items are non-evictable

queue.delivery.delivered

Number of successfully deliveries

queue.delivery.delayed

Number of temporarily failed delivery attempts

queue.delivery.failed

Number of permanently failed delivery attempts

queue.threads.scripts[].id

Name of the script thread

queue.threads.scripts[].pending

New script waiting in line for execution

queue.threads.scripts[].rescheduled

Rescheduled scripts waiting in line for execution

queue.threads.scripts[].running

Number of scripts running

queue.threads.scripts[].maxrunning

Maximum allowed number of scripts running

queue.threads.scripts[].scripts

Number of scripts started

queue.threads.scripts[].maxscripts

Maximum allowed number of scripts started

5.6. Script interpreter

The hsh program is a Halon script interpreter and interactive shell (REPL). It allows you to easily write, run and test Halon script without having to deploy it into the smtpd server. It implements the full standard library, but not any smtpd server or queue hooks. It also doesn’t provide an interface for halonctl’s script commands.

5.6.1. Options

-c, --config path

In order to make the hsh script environment resemble the environment of smtpd, it supports loading smtpd’s startup configuration file (default /etc/halon/smtpd.yaml) which in turn load the running configuration. This allows all paths and permission to be set up correctly, as well as virtual files scripting.files[] and certificates pki.private[]. Since that file usually drop privileges and change user, it might require hsh -P to be run as root. If you don’t need the full environment, you may run without a startup configuration: /dev/null.

-P, --privdrop

Drop privileges (setuid / setgid) to the user specificed in the startup configuration. This may allow you to have access to various sockets, but requires root permission. The recommended setup would be to have shared group permission with the smtpd server on various sockets and paths.

-F, --ffi

Enable FFI support, which is equivalent to scripting.ffi, without having to use hsh -c to load a startup a configuration.

-p, --plugin id

Load a specific HSL plugin. This option is repeatable.

-R, --rootpath path

Set the script root path, which is equivalent to scripting.rootpath, without having to use hsh -c to load a startup a configuration.

-A, --appconf path

Load a specific running configuration which is equivalent to environment.appconf, without having to use hsh -c to load a startup a configuration.

-B, --binary

Print output as binary (not as UTF-8), and also the output of echo will not automatically include a newline (\n).

-s, --syntax

Check the script syntax only.

-v, --version

Display Halon version and exit.

5.6.2. Script interpreter

The hsh program may be executed with a script file as the entry point for the script execution. It should be given as the last argument to the hsh command line.

$ cat test.hsl
echo "Hello World";
$ hsh test.hsl
Hello World

Setting the root to the current directory may be useful when implementing unit testing of modules:

$ cat module.hsl
function foo() {
   echo "Hello World";
}
$ cat test.hsl
import { foo } from "file://module.hsl";
foo();
$ hsh -R . test.hsl
Hello World

5.6.3. Interactive shell

The hsh program may be executed without a file, in which case it enters the interactive shell mode (REPL). Statements are executed line-by-line.

$ hsh
HSH> echo "Hello World";
Hello World
HSH>