5. Command line interface
There are three primary ways of managing hosts (or clusters of);
the web administration, API (Protocol Buffer) or CLI.
The smtpd
package comes with a program called halonctl
that
can be used to manage any aspect of the MTA. This section is a guide for
that program.
Every command have a --help
flag to display available options.
The halonctl program communicates with the smtpd
processes over control sockets as
described by the Protocol Buffer schemas.
The raw response data can be useful
when using the CLI for scripting with tools like jq,
and the request data can be useful when learning about the API.
Therefore, the commands have a --json
output flag making it return the raw response
and a --json-request
flag that prints the raw request (JSON encoded).
5.1. Queue management
These sub-commands operate on the queue (which contains queued email messages) in various ways. Please see the queue section for more information about what the queue is, and how it works. Because the metadata for all messages in queue is loaded into memory, the MTA can quickly summarise and modify large number of messages. The list, groupby and update sub-commands share the same condition arguments, which are used to filter the result:
$ halonctl queue list -h
...
Conditions:
--transportid Filter on transport ID (repeatable) <string>
--jobid Filter on job ID (repeatable) <string>
--senderdomain Filter on sender domain (repeatable) <domain>
--recipientdomain Filter on recipient domain (repeatable) <domain>
--sender Filter on sender (repeatable) <address>
--recipient Filter on recipient (repeatable) <address>
--id Filter on ID (repeatable) <id>
--state Filter on state (repeatable) <state>
--retrycount Filter on retry count (operators, repeatable) <number>
--retrydelay Filter on next retry (operators, repeatable) <seconds>
--age Filter on age (operators, repeatable) <seconds>
--ts Filter on recevied time (operators, repeatable) <iso8601>
--metadata Filter on metadata <key=value>
Available states are:
queue: ACTIVE, DEFER
working: SENDING, UNKNOWN
freeze: HOLD, UPDATE
Number and date format:
operators: >, >=, <, <= and = (eg ">2020-01-01T01:02:03")
range: .. (eg "10..20")
...
5.1.1. List messages
This command shows information about messages in queue, filtered by the conditions described above. There are a few pre-defined views for showing the defer queue, active queue and open connections with relevant table fields pre-selected.
$ halonctl queue list --age '>3600' --state DEFER -F id -F retrycount -F retrydelay -F recipient
┌────────────────────────────────────────┬────────────┬────────────┬─────────────────┐
│ id │ retrycount │ retrydelay │ recipient │
├────────────────────────────────────────┼────────────┼────────────┼─────────────────┤
│ c6321f4c-1671-11ea-885e-005056914940:1 │ 20 │ 243 │ [email protected] │
│ f640a91a-1671-11ea-b6ea-005056914940:1 │ 21 │ 70 │ [email protected] │
...
5.1.2. Export messages
This command exports a messages in queue in “message/rfc822” format. If only a messageid (and no :queueid is provided) the original message (as received) will be exported.
$ halonctl queue export c6321f4c-1671-11ea-885e-005056914940
...
$ halonctl queue export c6321f4c-1671-11ea-885e-005056914940:1
...
5.1.3. Take actions on messages
The queue update command can take various actions on messages, filtered by the conditions described above.
--delete
remove messages, without sending bounces
--reason
set the DSN diagnosticcode when deleting the message (“Message could not be delivered in the allotted time frame”)
--bounce
remove messages, and generate bounce messages to the sender
--dsn-status
set the DSN status when bouncing the message (5.4.7)
--dsn-diagnosticcode
set the DSN diagnosticcode when bouncing the message (“Message could not be delivered in the allotted time frame”)
--active
move messages (that are deferred or on hold) to the active queue for imminent delivery--defer
move messages (that are in active queue or on hold) to the defer queue for the specified number of seconds (number)
--retrytsjitter
apply an optional jitter so that messages are spread evenly between the defer time and an additional jitter in seconds
--hold
put queued messages on hold--updatetransportid
changes the messages’ transport ID to the one specified by the argument (string)--updatemetadata
modify or add auxiliary metadata to the messages as specified by the argument (in format field=value)--updatepriority
change the priority of the message (0 - normal, 1 - high, number)
$ halonctl queue update --age '>3600' --state DEFER --state ACTIVE --bounce
230 messages affected
$ halonctl queue update --transportid t1 --state DEFER --updatetransportid t2
27854 messages affected
5.1.3.1. Actions and working states
Messages in a working state
(delivery attempts, script execution, DNS resolution, etc)
cannot be directly modified, and they will be reported as unaffected by the command.
Sometimes it is important to perform a task on all matching messages, even those that are in a working state.
For example, you might want to delete every possible queued message from a certain sender domain.
To do this, re-run the command with the --freeze
flag.
This will capture those messages once they are no longer in a working state
and put them in the so-called UPDATE state
(given that they still in in queue;
messages that have been delivered or deleted can not be captured).
No messages will be reported as unaffected;
they will instead be reported as freezing.
You should then monitor the queue.freeze.update.pending
process counter
and wait for it to reach 0.
When it does, every messages that could not be immediately modified (but which is still in queue)
will be in the UPDATE state, ready for you to modify them.
$ halonctl queue update --delete --senderdomain example.com --all-states
100 messages affected
30 messages unaffected
$ halonctl queue update --delete --senderdomain example.com --all-states --freeze
4 messages affected
26 messages freezing
$ halonctl process-stats | grep queue.freeze.update
queue.freeze.update.size: 6
queue.freeze.update.pending: 20
$ halonctl process-stats | grep queue.freeze.update
queue.freeze.update.size: 22
queue.freeze.update.pending: 0
$ halonctl queue update --delete --senderdomain example.com --state UPDATE
22 messages affected
5.1.4. Show messages distribution
This command shows distribution of messages in queue, filtered by the conditions described above. There are a few pre-defined views for showing the distribution by for example age.
$ halonctl queue groupby --view state
┌─────────────┬─────────────────┬───────┬────────┬───────┬──────┬─────────┬─────────┐
│ transportid │ recipientdomain │ total │ ACTIVE │ DEFER │ HOLD │ UNKNOWN │ SENDING │
├─────────────┼─────────────────┼───────┼────────┼───────┼──────┼─────────┼─────────┤
│ inbound │ example.com │ 19 │ 0 │ 17 │ 1 │ 0 │ 1 │
...
5.1.5. Active queue policies
These sub-commands can add, list and delete dynamic
active queue policies that overrides
the ones in the environment.policyconf
configuration,
as well as reload that configuration file.
$ halonctl queue policy add -F remotemx --remotemx *.example.com --rate 10/600 --ttl 3600
30be002e-7436-11eb-ae42-5254004d77d3
$ halonctl queue policy list
┌──────────────────────────────────────┬──────────┬─────────────┬─────────┬──────────┬────────────────┬─────────────────┬───────┬──────────┬─────────────┬────────┬─────────┬─────┐
│ id │ fields │ transportid │ localip │ remoteip │ remotemx │ recipientdomain │ jobid │ grouping │ concurrency │ rate │ ttl │ tag │
├──────────────────────────────────────┼──────────┼─────────────┼─────────┼──────────┼────────────────┼─────────────────┼───────┼──────────┼─────────────┼────────┼─────────┼─────┤
│ 30be002e-7436-11eb-ae42-5254004d77d3 │ remotemx │ │ │ │ *.example.com │ │ │ │ │ 10/600 │ 3592.44 │ │
└──────────────────────────────────────┴──────────┴─────────────┴─────────┴──────────┴────────────────┴─────────────────┴───────┴──────────┴─────────────┴────────┴─────────┴─────╯
$ halonctl queue policy delete 30be002e-7436-11eb-ae42-5254004d77d3
It is also possible to clear or partially refill active queue policy rates, for a specific policy rate “bucket” (matched exactly by the optional condition arguments). If you run it without specifying a bucket, it will clear all rate buckets, created by any policy, and return the number of buckets it cleared/refilled.
$ halonctl queue policy rate refill --localip 198.51.100.4 --remotemx mx1.example.org
1
5.1.6. Active queue suspensions
These sub-commands can add, list and delete dynamic
active queue suspension that overrides
the ones in the environment.suspendconf
configuration,
as well as reload that configuration file.
$ halonctl queue suspend add --remotemx *.example.com --ttl 60
803c41d5-4a8e-424a-97e9-f8db55f69179
$ halonctl queue suspend list
┌──────────────────────────────────────┬─────────┬─────────────┬─────────┬──────────┬───────────────┬─────────────────┬───────┬──────────┬─────────┬─────┐
│ id │ source │ transportid │ localip │ remoteip │ remotemx │ recipientdomain │ jobid │ grouping │ ttl │ tag │
├──────────────────────────────────────┼─────────┼─────────────┼─────────┼──────────┼───────────────┼─────────────────┼───────┼──────────┼─────────┼─────┤
│ 803c41d5-4a8e-424a-97e9-f8db55f69179 │ dynamic │ │ │ │ *.example.com │ │ │ │ 57.7753 │ │
└──────────────────────────────────────┴─────────┴─────────────┴─────────┴──────────┴───────────────┴─────────────────┴───────┴──────────┴─────────┴─────╯
$ halonctl queue suspend delete 803c41d5-4a8e-424a-97e9-f8db55f69179
5.1.7. Active queue delivery settings
This sub-command can reload the
active queue delivery settings configuration file
environment.deliveryconf
.
$ halonctl queue delivery reload
5.1.8. Moving the queue files
If you need to move the queued message files (for example from one MTA to another)
it is important to first “unload” them. If you want to move the entire queue, the easiest way is to simply
shut down the server process and use the hqfmove
tool. To move specific messages, it is recommended to use the
queue unload command, filtered by the conditions described above.
--partial
match a queued message, even if only one of its recipients matches--freeze
freeze messages that are in a working state
$ halonctl queue unload --transportid t1
/storage/mail/spool/13/13ef04db-1e06-11ea-927e-0050569a4c9c.hqf unloaded
/storage/mail/spool/1b/1b5a5e80-1da2-11ea-927e-0050569a4c9c.hqf unloaded
...
$ halonctl queue unload
No messages unloaded
5.1.9. Trace messages
This command allows you to get debug or trace information regarding message deliveries in queue,
filtered by the queue pickup fields available (eg. --remotemx '*.example.com'
) or a specific transaction id.
Additional options are available to control the pooling (--connect
) and PIPELINING (--no-pipelining
) behaviour.
$ halonctl queue trace
2023-03-28 21:59:23.076 Begin trace of message 18096a8d-cba3-11ed-b464-930f4c7d94ae:1
2023-03-28 21:59:23.076 Connecting from [1.2.3.4] to [5.6.7.8]:25 (mx.example.com) (DNSSEC)
2023-03-28 21:59:23.079 Connected
2023-03-28 21:59:23.083 [SMTP] < 220 mx.example.com ESMTP
2023-03-28 21:59:23.083 [SMTP] > EHLO mx.example.org
...
5.1.10. Loading queue files
If you need to manually import a queue file into a running MTA instance you can use the queue load command. The “load” command will copy the .hqf specified to the spool folder leaving the original file in place. The message ID (in the HQF file’s ID field) has to be an unique and will be used as the target file name (regardless of the imported source file name). If the HQF file’s ID field is an all zero UUID (00000000-0000…) then a new UUID will be generated.
--hqf
path to the queue file
$ halonctl queue load --hqf 13ef04db-1e06-11ea-927e-0050569a4c9c.hqf
Message loaded
5.2. Configuration management
These sub-commands can reload the running configuration (including its Halon script) and control the built-in blue-green testing that we call “live stage”. See the configuration section for more information.
5.2.1. Script syntax and packing
The script is normally edited as individual files using the Visual Studio Code integration or simply a text editor. Those are then checked and “packed” into an actual configuration file by the halonconfig script as described in the script directive section.
Given that the folder test contains halonconfig’s standard file structure (see /opt/halon/share/examples) you can pack, validate and reload the configuration directly on the MTA by:
$ cd test
(editing configuration files)
$ halonconfig
$ sudo cp dist/* /etc/halon/
$ sudo service halon reload
If you want to exclude any files inside the src/files folder from being packed you can add a .halonignore file which supports regular glob patterns.
The halonconfig command may also be used to “unpack” the configuration, reversing the described process above.
5.2.2. Reloading configurations
You can also invoke the individual reload commands directly:
$ halonctl config reload # /etc/halon/smtpd-app.yaml
$ halonctl queue policy reload # /etc/halon/smtpd-policy.yaml
$ halonctl queue suspend reload # /etc/halon/smtpd-suspend.yaml
$ halonctl queue delivery reload # /etc/halon/smtpd-delivery.yaml
$ ratectl reload # /etc/halon/rated-app.yaml
$ dlpctl reload # /etc/halon/dlpd-app.yaml
5.2.3. Blue-green testing
Halon has built-in blue-green testing, which allows you to deploy a parallel running configuration for only a selection of the traffic, for example connections from certain IP addresses or a certain percentage chosen by random. The following example will deploy a parallel configuration for 10% of connecting clients during one hour:
$ halonctl config livestage start --id t1 --file dist/smtpd-app.yaml --probability 0.1 --time 3600
$ halonctl config livestage status
t1: 2 connections(s), 30/3600 runtime
$ halonctl config livestage cancel
5.2.4. Remote configuration
The halonconfig
script can be used to pack and validate configuration
folders on any computer that runs Python. You can copy it from the MTA:
$ python -m venv env
$ source env/bin/activate
$ pip install pyyaml
$ pip install jsonschema
$ scp mta:/opt/halon/bin/halonconfig .
$ scp mta:/opt/halon/share/json-schemas json
and run it by specifying the JSON schema path and disabling the script linter:
$ python halonconfig --schema-dir json --no-check-hsl
You can start a live stage directly from a remote computer by using SSH stdin redirection:
$ cat dist/smtpd-app.yaml | ssh mta "halonctl config livestage start --file /dev/stdin --id t2"
$ ssh mta "halonctl config livestage status"
t2: 2 connections(s), 3 runtime
5.3. Script engine
These sub-commands can work on various parts of the Halon script engine. The actual script is managed via the configuration commands.
5.3.1. Rate function
These sub-commands can list and clear the cluster-aware rate
limit functionality implemented by the rated
program and available via
the rate() function
exported by the rate limiting client plugin.
$ ratectl list
┌───────────────────┬────────────┬───────┬─────────────┬────────┬────────┬────────────┐
│ Namespace │ Entry │ Count │ Local count │ Oldest │ Newest │ Rate (#/s) │
├───────────────────┼────────────┼───────┼─────────────┼────────┼────────┼────────────┤
│ delivery-failures │ customer0 │ 1 │ 1 │ 8 │ 8 │ 0 │
│ delivery-failures │ customer1 │ 5 │ 5 │ 12 │ 4 │ 0.625 │
│ delivery-failures │ customer10 │ 1 │ 1 │ 8 │ 8 │ 0 │
...
$ ratectl clear --entry customer10
1 entries deleted
5.3.3. Cache statement
These sub-commands can list and clear the script engine’s function cache buckets.
$ halonctl hsl cache list
┌───────────┬──────────────────┬──────────┬──────┬────────┬────────┬────────┬──────────┐
│ Namespace │ Function │ Max size │ Size │ Hits │ Misses │ Evicts │ Hit rate │
├───────────┼──────────────────┼──────────┼──────┼────────┼────────┼────────┼──────────┤
│ │ api_call_http │ 100000 │ 845 │ 122799 │ 6041 │ 0 │ 95.3112 │
│ │ http │ 16384 │ 747 │ 5670 │ 2850 │ 0 │ 66.5493 │
│ test │ smtp_lookup_rcpt │ 16384 │ 2 │ 1 │ 31 │ 0 │ 3.125 │
└───────────┴──────────────────┴──────────┴──────┴────────┴────────┴────────┴──────────╯
$ halonctl hsl cache clear --function http
1 buckets cleared
5.4. DNS resolver
These commands can clear and view statistics about the asynchronous DNS resolver’s built-in cache.
$ halonctl process-stats | grep resolver.cache
resolver.cache.size: 5
resolver.cache.hits: 897
resolver.cache.misses: 1624
resolver.cache.expires: 1566
resolver.cache.evicts: 0
resolver.cache.skips: 0
$ halonctl resolver clear
There is also a command to test the DNS resolvers behaviour as for choosing the next-hop based on a transactionid and retry count.
$ halonctl resolver query --seed 67b06545-2592-11eb-a941-f1a44e94c1b9 --retry 3 halon.io
address: 2a02:750:7:3305::ee
hostname: mx.halon.io
dnssec: true
tlsa:
dnssec: true
rr:
- usage: 3
selector: 1
mtype: 1
data: 42568063726264388f17d494cd3d01079ba1df4e60b1448e5670544cd7739218
5.5. Statistics and information
Shows information about process statistics (cache hits, internal queue sizes) and other information such as email queue size.
5.5.1. halontop
The halontop program shows a visual view of all counters with live updates and rates calcuations.
5.5.2. halonctl
Text based output for counters may be extracted using halonctl suited for data collection.
$ halonctl process-stats
process.pid: 48535
process.runtime: 3638
...
Name |
Description |
---|---|
process.pid |
The smtpd process PID |
process.runtime |
Seconds that smtpd has been running |
process.elapsed |
Seconds that smtpd has been running (with fractions) |
resolver.pending |
DNS queries waiting in line to be sent |
resolver.running |
DNS queries waiting for response, max is |
resolver.maxrunning |
Maximum allowed DNS queries waiting for response, max is |
resolver.cache.size |
Entries in cache, max is |
resolver.cache.maxsize |
Maximum allowed entries in cache, max is |
resolver.cache.hits |
DNS requests that were found in cached |
resolver.cache.misses |
DNS requests not found in cache |
resolver.cache.expires |
Entries dropped due to TTL, see |
resolver.cache.evicts |
Entries dropped due to LRU eviction |
resolver.cache.skips |
Entries not cached (eg. bad responses) |
servers[].serverid |
The virtual |
servers[].connections.concurrent |
Connected clients, max is |
servers[].connections.maxconcurrent |
Maximum allowed connected clients, max is |
servers[].scripts.connect.pending |
Waiting in line for |
servers[].scripts.connect.running |
Executing the above, max is |
servers[].scripts.connect.finished |
Completed script executions |
servers[].scripts.connect.errors |
Script execution errors |
servers[].scripts.proxy.pending |
” |
servers[].scripts.proxy.running |
” |
servers[].scripts.proxy.finished |
” |
servers[].scripts.proxy.errors |
” |
servers[].scripts.helo.pending |
” |
servers[].scripts.helo.running |
” |
servers[].scripts.helo.finished |
” |
servers[].scripts.helo.errors |
” |
servers[].scripts.auth.pending |
” |
servers[].scripts.auth.running |
” |
servers[].scripts.auth.finished |
” |
servers[].scripts.auth.errors |
” |
servers[].scripts.mailfrom.pending |
” |
servers[].scripts.mailfrom.running |
” |
servers[].scripts.mailfrom.finished |
” |
servers[].scripts.mailfrom.errors |
” |
servers[].scripts.rcptto.pending |
” |
servers[].scripts.rcptto.running |
” |
servers[].scripts.rcptto.finished |
” |
servers[].scripts.rcptto.errors |
” |
servers[].scripts.eod.pending |
” |
servers[].scripts.eod.running |
” |
servers[].scripts.eod.finished |
” |
servers[].scripts.eod.errors |
” |
servers[].scripts.disconnect.pending |
” |
servers[].scripts.disconnect.running |
” |
servers[].scripts.disconnect.finished |
” |
servers[].scripts.disconnect.errors |
” |
queue.loader.count |
Email loaded into the message queue |
queue.loader.pending |
Email waiting in line to be loaded |
queue.loader.active |
Total amount of emails loaded into memory |
queue.loader.maxactive |
Max amount of emails loaded into memory (see |
queue.scripts.predelivery.pending |
Waiting in line for |
queue.scripts.predelivery.running |
Executing the above, max is |
queue.scripts.predelivery.finished |
Completed script executions |
queue.scripts.predelivery.errors |
Script execution errors |
queue.scripts.postdelivery.pending |
” |
queue.scripts.postdelivery.running |
” |
queue.scripts.postdelivery.finished |
” |
queue.scripts.postdelivery.errors |
” |
queue.queue.defer.size |
Messages in defer queue |
queue.queue.active.size |
Messages in active queue |
queue.queue.active.priorities[].size |
Messages in with the different priorities (0 - normal, 1 - high) queue |
queue.freeze.hold.size |
Messages on hold |
queue.freeze.update.size |
Messages frozen for update |
queue.freeze.update.pending |
Messages in working state waiting to be frozen for update |
queue.policy.concurrency.counters |
Concurrency counters in memory created by active queue policies |
queue.policy.concurrency.suspends |
Suspensions created as a result of exceeded concurrency |
queue.policy.rate.buckets |
Rate buckets in memory created by active queue policies |
queue.policy.rate.suspends |
Suspensions created as a result of exceeded rate |
queue.policy.dynamic.suspends |
Number of suspends added via API/CLI or script |
queue.policy.dynamic.conditions |
Number of policy conditions added via API/CLI or script |
queue.pickup.count |
Number of messages picked up from queue |
queue.pickup.skips |
Number of skips in queue pickup algorithm due to suspends |
queue.pickup.misses |
Number of times no were picked up because all were suspended |
queue.quota.size |
Number of quota name entries |
queue.connections.concurrent |
Open connections, max is |
queue.connections.maxconcurrent |
Maximum allowed open connections, max is |
queue.connections.pooling.size |
Cached connections in pool for reuse, max is |
queue.connections.pooling.maxsize |
Maximum allowed cached connections in pool for reuse, max is |
queue.connections.pooling.hits |
Message deliveries that could re-use a cached connections |
queue.connections.pooling.misses |
Message deliveries that had to open a new conenctions |
queue.connections.pooling.expires |
Connections closed due to idle timeout |
queue.connections.pooling.evicts |
Connections closed due to LRU eviction |
queue.connections.pooling.skips |
If the pooling cache is full and all items are non-evictable |
queue.delivery.delivered |
Number of successfully deliveries |
queue.delivery.delayed |
Number of temporarily failed delivery attempts |
queue.delivery.failed |
Number of permanently failed delivery attempts |
queue.threads.scripts[].id |
Name of the script thread |
queue.threads.scripts[].pending |
New script waiting in line for execution |
queue.threads.scripts[].rescheduled |
Rescheduled scripts waiting in line for execution |
queue.threads.scripts[].running |
Number of scripts running |
queue.threads.scripts[].maxrunning |
Maximum allowed number of scripts running |
queue.threads.scripts[].scripts |
Number of scripts started |
queue.threads.scripts[].maxscripts |
Maximum allowed number of scripts started |
5.6. Script interpreter
The hsh program is a Halon script interpreter and interactive shell (REPL).
It allows you to easily write, run and test Halon script
without having to deploy it into the smtpd
server.
It implements the full standard library, but not any smtpd server or queue hooks.
It also doesn’t provide an interface for halonctl’s script
commands.
5.6.1. Options
- -c, --config path
In order to make the hsh script environment resemble the environment of smtpd, it supports loading smtpd’s startup configuration file (default /etc/halon/smtpd.yaml) which in turn load the running configuration. This allows all paths and permission to be set up correctly, as well as virtual files
scripting.files[]
and certificatespki.private[]
. Since that file usually drop privileges and change user, it might require hsh -P to be run as root. If you don’t need the full environment, you may run without a startup configuration: /dev/null.
- -P, --privdrop
Drop privileges (setuid / setgid) to the user specificed in the startup configuration. This may allow you to have access to various sockets, but requires root permission. The recommended setup would be to have shared group permission with the smtpd server on various sockets and paths.
- -F, --ffi
Enable FFI support, which is equivalent to
scripting.ffi
, without having to usehsh -c
to load a startup a configuration.
- -p, --plugin id
Load a specific HSL plugin. This option is repeatable.
- -R, --rootpath path
Set the script root path, which is equivalent to
scripting.rootpath
, without having to usehsh -c
to load a startup a configuration.
- -A, --appconf path
Load a specific running configuration which is equivalent to
environment.appconf
, without having to usehsh -c
to load a startup a configuration.
- -B, --binary
Print output as binary (not as UTF-8), and also the output of
echo
will not automatically include a newline (\n
).
- -s, --syntax
Check the script syntax only.
- -v, --version
Display Halon version and exit.
5.6.2. Script interpreter
The hsh program may be executed with a script file as the entry point for the script execution. It should be given as the last argument to the hsh command line.
$ cat test.hsl
echo "Hello World";
$ hsh test.hsl
Hello World
Setting the root to the current directory may be useful when implementing unit testing of modules:
$ cat module.hsl
function foo() {
echo "Hello World";
}
$ cat test.hsl
import { foo } from "file://module.hsl";
foo();
$ hsh -R . test.hsl
Hello World
5.6.3. Interactive shell
The hsh program may be executed without a file, in which case it enters the interactive shell mode (REPL). Statements are executed line-by-line.
$ hsh
HSH> echo "Hello World";
Hello World
HSH>