Command Line Parameters

Synopsis

logagent [options] [file list]

Options Description
Genernal options
-h, --help output logagent help
-V, --version output logagent version
-v, --verbose output activity report every minute
--config path to logagent config file (see below)
--geoipEnabled true/false to enable/disable geoip lookups in patterns.
--diskBufferDir path directory to store status and buffered logs (during network outage)
--includeOriginalLine includes the original message in parsed logs
-f, --file file with pattern definitions, use multiple -f options multiple files
-s, --suppress silent, print no logs to stdout, prints only stats on exit
--printStats print processing stats in the given interval in seconds, e.g. --print_stats 30 to stderr. Usefull with -s to see logagent activity on the console without printing the parsed logs to stdout.
Log input options
-g glob-pattern use a glob pattern to watch log files e.g. -g "{/var/log/*.log,/Users/stefan/myapp/*.log}". The complete glob expression must be quoted, to avoid interpretation of special characters by the linux shell.
+ --tailStartPosition bytes
--stdin read from stdin, default if no other input like files or UDP are set
-n name name for the log source only when stdin is used, important to make multi-line patterns working on stdin because the status is tracked by the log source name.
-u UDP_PORT starts a syslogd UDP listener on the given port to act as syslogd
--heroku PORT listens for Heroku logs (http drain / framed syslog over http)
--cfhttp PORT listens for Cloud Foundry logs (syslog over http)
list of files Every argument after the options list is interpreted as file name. All files in the file list (e.g. /var/log/*.log) are watched by tail-forever starting at end of file
Output options
standard output stream combine logagent with any unix tool via pipes
-y, --yaml prints parsed messages in YAML format to stdout
-p, --pretty prints parsed messages in pretty json format to stdout
-j, --ldjson print parsed messages in line delimited JSON format to stdout
Elasticsearch / Logsene Log storage
-e, --elasticsearch-host Elasticsearch url e.g. http://localhost:9200, default htpps://logsene-receiver.sematext.com:443'
-t, --index Logsene App Token to insert parsed records into Logsene or Elasticsearch index (see --elasticsearch-host)
--httpProxy HTTP proxy url
--httpsProxy HTTPS proxy url
rtail Realtime log viewer
--rtailPort forwards logs via UDP to rtail server
--rtailHost hostname rtail server (UI for realtime logs), default: localhost
--rtailWebPort starts rtail UI webserver (if not installed install with: - npm i rtail -g)
--rtailWebHost rtail UI webserver and bind hostname. E.g. logagent --rtailWebPort 9000 --rtailPort 8989 --rtailWebHost $(hostname) -g \'/var/log/**/*.log

The default output is line delimited JSON for parsed log lines, as long as no format options like -yaml (YAML format), -p (pretty JSON), or -s (silent, no output to console) are specified.

Config File

The config file needs to be in YAML format.

Section: options

# Global options
options:
  # print stats every 60 seconds 
  printStats: 60
  # don't write parsed logs to stdout
  suppress: false
  # Enalbe/disable GeoIP lookups
  # Startup of logagent might be slower, when downloading the GeoIP database
  geoipEnabled: false
  # Directory to store Logagent status nad temporary files
  diskBufferDir: ./tmp

Section: input

input:
  # a list of glob patterns to watch files to tail
  files:
      - '/var/log/**/*.log'
      - '/opt/myapp/logs/*.log'
  # listen to udp syslog protocol  
  #syslog: 
  #  port: 514
  # listen to http to receive data from Heroku  log drains  
  #heroku: 
  #  port: 9999
  # listen to http to receive data from Cloud Foundry drains  
  #cloudFoundry:
  #  port: 8888

Section: parser

In this section defines loading of custom pattern files or inline pattern definitions for the log parser.

# optional, if not specified default patterns are used
parser:
  patternFiles:
    # load a list of pattern files to parse logs
    # later files overwrite settings from previous files
    # a 'hot reload' is done as soon one of the listed fiels changes on disk
    - patterns1.yml
    - patterns2.yml
  # inline pattern definitions, to put on top of patterns list
  # loaded from files or default librarary  
  patterns:
    - # timestamped messages from /var/log/*.log on Mac OS X
      sourceName: !!js/regexp /\system\.log/ # catch all system.log files  
      match:
        -
          type: system_log
          regex: !!js/regexp /([\w|\s]+\s+\d{2}\s[\d|\:]+)\s(.+?)\s(.+?)\s<(.+)>(.*)/
          fields: [ts,host,service,severity,message]
          dateFormat: MMM DD HH:mm:ss

Section: output

Logs could be shipped to Elasticsearch or to rtail for realtime log view. The Elasticsearch output supports HTTPS, username/password in the url. In addtion it is possible to route logs from different files to different indicies in Elasticsearch. All logs, which don't match the rules in the indices section are routed to the default index (elasticsearch.index).

output:
  # index logs in Elasticsearch or Logsene
  elasticsearch: 
    # URL to Elasticearch server, defaults to Logsene SaaS if not set
    url: https://logsene-receiver.sematext.com

    # Proxy settings behind firewalls
    #httpProxy:  http://localProxy:port
    #httpsProxy: https://localHttpsProxy:port

    # default index to use, for all logs that don't match later in indices section
    # for Logsene use the Logsene App Token here
    index: 0a835c75-9847-4f74-xxxx

    # specific index to use per logSource field of parsed logs
    # logSource is by default the file name of the log file
    # but it can be modified by JS transforms functions in the patterns.yml file
    indices: 
      4f70a0c7-9458-43e2-bbc5-xxxx: 
      # list of RegEx mathich logSource / filename  
      # all logs matching logSOurce name will be indexed to above index
        - .*wifi.*
        - .*bluetooth.*
      999532c9-18f1-4c4b-8753-xxxx: 
        - system\.log
        - access\.log
        - auth\.log
  # print parsed logs in YAML format to stdout (only if options.supress is set to false)    
  stdout: yaml # use 'pretty' for pretty json and 'ldjson' for line delimited json (default)

  # forward logs to rtail realtime log viewer
  #rtail:
    # rtail host to send logs to
    #host: localhost
    # rtails port to send logs to 
    #udpPort: 3434
    # start rtail Server with given http port and bind to address of hostname
    #webPort: 8080
    #webHost: localhost

Environment variables

Variable Description
LOGSENE_TMP_DIR Directory to store failed bulk requests, for later re-transmission.
LOGSENE_LOG_INTERVAL Time to batch logs before a bulk request is done. Default 10000 ms (10 seconds)
LOGSENE_BULK_SIZE Maximum size of a bulk request. Default 1000.
LOGSENE_URL URL for the Logsene receiver. For a local Elasticsearch server or for On-Premise version of Logsene. Defaults to Sematext Logsene SaaS receiver https://logsene-receiver.sematext.com/_bulk. Example for Elasticsearch: LOGSENE_URL=http://localhost:9200/_bulk
HTTPS_PROXY Proxy URL for HTTPS endpoints, like Logsene receiver. export HTTPS_PROXY=http://my-proxy.example
HTTP_PROXY Proxy URL for HTTP endpoints (e.g. On-Premises or local Elasticsearch). export HTTP_PROXY=http://my-proxy.example
LOGAGENT_CONFIG Filename to read logagent CLI parameters from a file, defaults to `/etc/sematext/logagent.conf
PATTERN_MATCHING_ENABLED Default is 'true'. The value 'false' disables parsing of logs.

Command Line Examples

# Be Evil: parse all logs 
# stream logs to Logsene 1-Click ELK stack 
logagent -i LOGSENE_TOKEN /var/log/*.log 

# Act as syslog server on UDP and forward messages to Logsene
logagent -u 514 -i LOGSENE_TOKEN  

# Act as syslog server on UDP and write YAML formatted messages to console
logagent -u 514 -y  

Use a glob pattern to build the file list

logagent -i LOGSENE_TOKEN -g '/var/log/**/*.log'
# pass multiple glob patterns
logagent -i LOGSENE_TOKEN -g '{/var/log/*.log,/opt/myapp/*.log}'

Watch selective log output on console by passing logs via stdin and format in YAML

tail -f /var/log/access.log | logagent -y -n httpd
tail -f /var/log/system.log | logagent -f my_own_patterns.yml  -y 

Ship logs to rtail and Logsene to view logs in real-time in rtail and store logs in Logsene

# rtail don't need to be installed, logagent uses the rtail protocol
logagent -i $LOGSENE_TOKEN --rtail-host myrtailserver --rtail-port 9999 /var/log/*.log

Logagent can start the rtail web-server (in-process, saving memory), open browser with http://localhost:8080

# logagent has no dependency to rtail, to keep the package small
sudo npm i rtail -g
logagent -s -i $LOGSENE_TOKEN --rtail-web-port 8080 --rtail-port 9999 /var/log/*.log

And of course you can combine rtail and Logagent in the traditional way, simply connect both via unix pipes. An example with rtail and Logsene storage and charts: