You’re reading this to learn how to create a set of reports for how your users (or servers!) are using the Internet and what type of websites (or HTTP endpoints) are being hit.
To have this work you will need to configure your Fortigate firewall to capture this kind of information and log it to a system, like Sumo Logic, for later processing and analysis. I wrote a blog post on how to do this.
These are basic searches that you could turn into a dashboard, add these searches as panels to existing dashboards, or create scheduled searches to email you regularly. One item of note - this is returning results where the amount of data transferred is over 500 bytes. In my example I had to generate traffic through my firewall to pick up different types of categories as I don’t use this type of filtering in my lab (it is all servers). If I were using this in an enterprise, or at home, I would instead use 500 KiB as my limit to cut false positives or drive-by advertising that might be questionable and not the users fault.
Usage by classification sorted by bytes transferred
_index=security_logs _sourceCategory=fw_security "subtype=webfilter" | parse " catdesc=\"*\"" as category nodrop | keyvalue regex "=(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}) " keys "srcip" | keyvalue regex "=(\d+) " keys "sentbyte", "rcvdbyte" | sentbyte+rcvdbyte as bytes | sum(bytes) as bytes by srcip, category | where bytes >=500 // >=500000 | sort by bytes | limit 20 | fields - srcip // strip out the srcip field for this result set
Usage by IP address by category sorted by bytes transferred
_index=security_logs _sourceCategory=fw_security "subtype=webfilter" | parse " catdesc=\"*\"" as category nodrop | keyvalue regex "=(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}) " keys "srcip" | keyvalue regex "=(\d+) " keys "sentbyte", "rcvdbyte" | sentbyte+rcvdbyte as bytes | sum(bytes) as bytes by srcip, category | where bytes >=500 // >=500000 | sort by bytes | limit 20
Potentially liable categories by IP address sorted by usage
_index=security_logs _sourceCategory=fw_security "subtype=webfilter" ( ("catdesc=\"Pornography\"") OR ("catdesc=\"Other Adult Materials\"") OR ("catdesc=\"Lingerie and Swimsuit\"") OR ("catdesc=\"Nudity and Risque\"") OR ("catdesc=\"Hacking\"") OR ("catdesc=\"Illegal or Unethical\"") OR ("catdesc=\"Proxy Avoidance\"") OR ("catdesc=\"Malicious Websites\"") OR ("catdesc=\"Phishing\"") OR ("catdesc=\"Spam URLs\"") OR ("catdesc=\"Unrated\"") ) | parse " catdesc=\"*\"" as category nodrop | keyvalue regex "=(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}) " keys "srcip" | keyvalue regex "=(\d+) " keys "sentbyte", "rcvdbyte" | sentbyte+rcvdbyte as bytes | sum(bytes) as bytes by srcip, category | where bytes >=500000 | sort by bytesThis last search will come in very handy for policing environments where the business is potentially liable for content. Scheduling this search (see below) can help you track down problems before they become a HR issue.
You can create scheduled emails with these results easily. Click on ‘Save as’ at the bottom of the search box, then click the blue link named ‘Schedule this search’ and fill in the parameters. My example, which runs at midnight, returns data from the previous day then emails it to me once completed.
Scheduled search configuration
The query above is modified below to look for servers acting in a way that isn’t normal. This updated query will help track down breaches, root kits, or other malicious activity. Apply the policy to your outbound interface and change the query (see below) to look for non-server type activity.
_index=security_logs _sourceCategory=fw_security "subtype=webfilter" ( ("catdesc=\"Hacking\"") OR ("catdesc=\"Proxy Avoidance\"") OR ("catdesc=\"Malicious Websites\"") OR ("catdesc=\"Phishing\"") OR ("catdesc=\"Spam URLs\"") OR ("catdesc=\"File Sharing and Storage\"") OR ("catdesc=\"Peer-to-peer File Sharing\"") ) | parse " catdesc=\"*\"" as category nodrop | keyvalue regex "=(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}) " keys "srcip", "dstip" | keyvalue regex "=(\d+) " keys "sentbyte", "rcvdbyte" | sentbyte+rcvdbyte as bytes | sum(bytes) as bytes by srcip, category, dstip | sort by bytesFor a list of Fortigate categories check out this link.