Sumo Logic – example json’ery

When installing a Sumo Logic collector, you have 2 methods of configuration:

  • You can continue to configure the collector via the web interface
    • Easy to do and many options are just a click or drop-down away
  • You can continue to configure the collector via an installed JSON file
    • Full control of the configuration, version control, and the ability to use an automation system like Ansible to handle standardized/consistent deployment

But you cannot do both.

There are not a lot of good examples that I have found for configuring the collector via the JSON method and a few options available. For this blog post I am going to use one of my collectors’ configuration files (sumologic-collector.json) and break down things a bit to help out the community.

But before we dive into the JSON file let’s take a quick look at the sumo.conf file (no, these are not my real access credentials for registering a collector):

name=syslog-receiver
email=insert-email-here
accessid=wbevLDb8wEFPV1
accesskey=d1b6766f3fcb005QAijDD4LPQzcbGGaFVHqYBQEfUMoleIm4bff9274afc4aac31
syncSources=/etc/sumologic-collector.json

Very straight forward options above.

  • name
    • the hostname of the system, and will default to the local hostname if available
  • email
    • email address associated with a Sumo Logic account, and used if you decide to use the authentication version of registration instead of the accessid/accesskey way
  • accessid
  • accesskey
    • self-explanatory and you generate these credentials via the web interface for collectors
  • syncSources
    • where the JSON file is read from

Now let’s look at the basic configuration file we can start the process of breaking down the included basic JSON configuration file. The Sumo Logic documentation has the full list of options you can set.

Variable Name Example Comment
name
apache_logs
This is  the _source directive and names the source
automaticDateParsing
true
Tells the collector that it should try to do automatic date parsing. This works well for most normal items though the Fortigate systems log with a key/value layout that requires a little more work
multilineProcessingEnabled
false
Defaults to false and is very useful for Java/Ruby stack traces
forceTimeZone
false
If you need to force the timezone to match something that the collector is unable to decide itself
timeZone
America/Chicago
UTC
Set the timezone of the source
defaultDateFormat
'date='yyyy-MM-dd 'time='HH:mm:ss
Specifically define how to parse the date – this example is for a Fortigate
 encoding
UTF-8
How should the incoming data be interpreted
protocol
TCP
UDP
If you are using a port than which protocol would you like to use
port
514
Port number to listen on
category
unix_logs
Sets the _sourceCategory flag for searching
sourceType
Syslog
LocalFile
Type of item to inject – syslog (over a port) or localfile (a local file to read

One hint: values of true and false should not have double-quotes added but everything else requires double-quotes.

And if you did not click above to download the JSON file it is below to browse.

{
    "api.version": "v1",
    "sources": [
        {
            "name": "syslog_udp_514",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "encoding": "UTF-8",
            "protocol": "UDP",
            "port": 514,
            "category": "unix_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "syslog_tcp_514",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "encoding": "UTF-8",
            "protocol": "TCP",
            "port": 514,
            "category": "unix_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "syslog_udp_1514",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "encoding": "UTF-8",
            "protocol": "UDP",
            "port": 1514,
            "category": "juniper_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "syslog_tcp_1514",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "encoding": "UTF-8",
            "protocol": "TCP",
            "port": 1514,
            "category": "juniper_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "syslog_udp_1515",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "encoding": "UTF-8",
            "protocol": "UDP",
            "port": 1515,
            "category": "bigip_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "syslog_tcp_1515",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "encoding": "UTF-8",
            "protocol": "TCP",
            "port": 1515,
            "category": "bigip_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "syslog_udp_3514",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "encoding": "UTF-8",
            "protocol": "UDP",
            "port": 3514,
            "category": "unix_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "syslog_tcp_3514",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "encoding": "UTF-8",
            "protocol": "TCP",
            "port": 3514,
            "category": "unix_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "fgt_udp_4514",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "defaultDateFormat": "'date='yyyy-MM-dd 'time='HH:mm:ss",
            "encoding": "UTF-8",
            "protocol": "UDP",
            "port": 4514,
            "category": "fw_security",
            "sourceType": "Syslog"
        },
        {
            "name": "fgt_tcp_4514",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "defaultDateFormat": "'date='yyyy-MM-dd 'time='HH:mm:ss",
            "encoding": "UTF-8",
            "protocol": "TCP",
            "port": 4514,
            "category": "fw_security",
            "sourceType": "Syslog"
        },
        {
            "name": "ubuntu_audit_log",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "os_security",
            "pathExpression": "/var/log/audit/*.log"
        },
        {
            "name": "ubuntu_auth_log",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "os_security",
            "pathExpression": "/var/log/auth.log"
        },
        {
            "name": "ubuntu_cron",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/cron"
        },
        {
            "name": "ubuntu_dmesg",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/dmesg"
        },
        {
            "name": "ubuntu_daemon_log",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/daemon.log"
        },
        {
            "name": "ubuntu_kern_log",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/kern.log"
        },
        {
            "name": "ubuntu_mail_log",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/mail.log"
        },
        {
            "name": "ubuntu_mail_err",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/mail.err"
        },
        {
            "name": "ubuntu_mail_info",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/mail.info"
        },
        {
            "name": "ubuntu_messages",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/messages"
        },
        {
            "name": "ubuntu_syslog",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "unix_logs",
            "pathExpression": "/var/log/syslog"
        },
        {
            "name": "ubuntu_secure",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "America/Chicago",
            "category": "os_security",
            "pathExpression": "/var/log/secure"
        },
        {
            "name": "nginx_logs",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "UTC",
            "category": "nginx_logs",
            "pathExpression": "/var/log/nginx/*.log"
        },
        {
            "name": "mirrors_udp_1516",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": false,
            "forceTimeZone": false,
            "timeZone": "UTC",
            "encoding": "UTF-8",
            "protocol": "UDP",
            "hostName": "mirrors-server",
            "port": 1516,
            "category": "nginx_logs",
            "sourceType": "Syslog"
        },
        {
            "name": "apache_logs",
            "sourceType": "LocalFile",
            "automaticDateParsing": true,
            "multilineProcessingEnabled": false,
            "useAutolineMatching": true,
            "forceTimeZone": false,
            "timeZone": "UTC",
            "category": "apache_logs",
            "pathExpression": "/var/log/apache2/*.log"
        }
    ]
}