4. History logging

To enable history logging there are some steps that need to be completed which are outlined below.

4.1. Install Elasticsearch

Elastic has an article on installing Elasticsearch here.

4.2. Import an index template

Import one of the index templates below (Delivery attempts or Transactions) into Elasticsearch. The differences between these index templates are described in each section.

4.2.1. Delivery attempts

This template will log all delivery attempts and the same message may appear multiple times, once for each delivery attempt. For this template you should set the value for elasticsearch.index to halon-delivery-attempts in the web.yaml file.

curl -k -X PUT -H "Content-Type: application/json" -d @/opt/halon/web/share/history/elasticsearch/delivery-attempts/halon_timeseries_policy.json http://elastic:[email protected]:9200/_ilm/policy/halon-timeseries-policy
curl -k -X PUT -H "Content-Type: application/json" -d @/opt/halon/web/share/history/elasticsearch/delivery-attempts/halon_delivery_attempts.json http://elastic:[email protected]:9200/_index_template/halon-delivery-attempts

4.2.2. Transactions

This template will log all transactions and update the state for them instead of creating a new log entry after each delivery attempt. For this template you should set the value for elasticsearch.index to halon-transactions-* in the web.yaml file.

curl -k -X PUT -H "Content-Type: application/json" -d @/opt/halon/web/share/history/elasticsearch/transactions/halon_transactions.json http://elastic:[email protected]:9200/_index_template/halon-transactions

When using the transactions template you also need to periodically remove old indices in Elasticsearch, you can use Elasticsearch’s Curator CLI as a cron job. This sample configuration will remove indices older than 120 days based on the index and date syntax (e.g. halon-transactions-%Y-%m-%d).

Curator configuration ~/.curator/curator.yml sample file:

elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9200
    ca_certs:
    client_cert:
    client_key:
    verify_certs: False
    request_timeout: 30
  other_settings:
    username: elastic
    password: elastic
    master_only: False
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elastic_transport', 'urllib3']

Curator action ~/.curator/delete_indices.yml sample file:

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 120 days (based on index name)
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: halon-transactions-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 120

Run the action file with this command:

curator delete_indices.yml

4.3. Add configuration

The history feature requires an additional configuration file. This configuration file is used to define fields in Elasticsearch and how they should be rendered.

halon-web loads the non-reloadable configuration from /etc/halon/web-history.yaml. It is described by, and can be validated with, the web-history.schema.json JSON schema. The default startup configuration file in /opt/halon/web/share can normally be used as a base (only choose one of them):

The following configuration is based on the Elasticsearch index template called halon-delivery-attempts:

# cp /opt/halon/web/share/history/delivery-attempts.yaml /etc/halon/web-history.yaml

The following configuration is based on the Elasticsearch index template called halon-transactions:

# cp /opt/halon/web/share/history/transactions.yaml /etc/halon/web-history.yaml

Use the elasticsearch directives to enable history logging in the web administration. Sample web.yaml config using the Delivery attempts template:

elasticsearch:
  index: 'halon-delivery-attempts'
  nodes:
    - url: http://127.0.0.1:9200
  auth:
    username: elastic
    password: elastic
  tls:
    rejectUnauthorized: false

4.4. Start sending data

To start sending data to Elasticsearch from the smtpd process using HSL you can use the history-elastic plugin (supports both index templates).