If you really need to work on this topic, I suggest you read the book of Anton Chuvakin – Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management. It has helped me tremendously in several projects and I would like to pass on the fact that it is an incredibly useful book.
Besides that, I have a few ideas to share which might be useful to you and are not present in the aforementioned book.
Before even starting to collect logs you need to understand your environment. That is why audits and assessments are useful. But that is not all – you need to understand how much data your devices are producing in log format every second, minute, hour, 24 hours – then you need to understand how to optimize what is logged and what is not.
It is very easy to lose yourself in the amount of logging information generated on each of your devises, especially if you add them all at once to your SIEM (even if you add only servers/switches and firewalls and omit the desktops).
The noisiest devices in my experience are Windows systems – especially domain controllers and mail servers. Second would come your proxy servers and firewalls.
To understand how logging in a Microsoft Environment works, I suggest you read these two links:
- Advanced Audit Policy Configuration FAQ http://technet.microsoft.com/en-us/library/ff182311(v=ws.10).aspx,
- HOWTO: http://technet.microsoft.com/en-us/library/dd408940(v=ws.10).aspx
Once you’ve read the above and understand the difference between basic and advanced audit policy configuration, move on to the next link:
It explains in detail the changes you need to make in order to have an optimized for detection logging environment. It also explains which EventIDs you should focus on.
In order to clean up your logs and only collect what is needed, you need to know what can be discarded. To know what is being logged, a good idea would be to export the logs from your noisiest server for a day to a CSV file, open it in Excel and filter by EventID. It should be easy to distinguish by percentage which events are happening most frequently and which are not.
In one case I saw thousands of events per second with the word “Filtering engine” in them. Turns out packet logging was turned on – and every single network packet coming through the internal Windows firewall was being logged! You can imagine the amount of logs generated by this server per hour and the usefulness of these logs.
Unfortunately such level of detail can render your SIEM and your storage incapacitated – and you should be very careful what you log and what you discard.
If you want a good start with configuring your SIEM, you could only configure it to log the EventIDs in the aforementioned “Spotting the Adversary with Windows Event Log Monitoring” paper by NSA – and expand to other systems / events once you are sure you got that one right. Don’t just plug in your SIEM and all your devices into it in a shotgun manner – all your money spend on storage and SIEM will go to waste if the system is not optimized and tuned for effectiveness.
You should also focus on the following EventIDs:
- 4688 Process Create (after going to https://technet.microsoft.com/en-us/library/dn535776.aspx?f=255&MSPPError=-2147217396 and enabling logging)
- 4663 File/Registry Auditing
- 4075 Service Created
- 4070 Service Changed
- 4624 User Login Success
- 5140 Share accessed
(List taken from http://www.slideshare.net/Hackerhurricane/ask-aalware-archaeologist)
The same process can and should be repeated for all your systems / devices. Only plug a device into your SIEM solution once you fully understand the format of logging it uses, the amount / type of events and how you could fine-tune the amount of details in these logs.
A really good additional guide on configuring Windows Logging is this one: https://malwarearchaeology.squarespace.com/s/Windows-Logging-Cheat-Sheet.pdf – the “Windows Logging Cheatsheet”, if the link breaks and you find the need to Google it.
Tools
These links will be useful during your work on optimizing your logging environment.
http://sourceforge.net/projects/syslogserverwindows/
http://sourceforge.net/projects/nxlog-ce/
https://indihiang.codeplex.com/
https://eventloganalyzer.codeplex.com/
https://logexpert.codeplex.com/ – especially this one!
https://www.mandiant.com/resources/download/highlighter
Using open source tools for centralized logging management
Before even beginning to think about buying anything, think about optimizing your logs and storage – as storage is utilized very quickly and becomes expensive if not managed properly.
There are multiple log storage calculation options online – what I used back in the days were a set of spreadsheets with pre-configured EPS (events per second) per device or operating system, or one of the online tools, such as http://codepen.io/packetinspector/details/vxjbL/
Another way of obtaining proper log storage calculation spreadsheets is asking SIEM vendors for them in a pre-sales meeting – their engineers have them and are willing to share.
There are a few really good open source solutions for centralized log storage and analysis (well, not as in a SIEM, but with a really good searching and filtering capability which is still good) for smaller budgets.
One of them – a classic – is Syslog-NG – https://www.balabit.com/network-security/syslog-ng/opensource-logging-system
They offer free and commercial versions.
In my opinion though the next 2 projects are offering better user interface (at least I like them more):
Logstash, ElasticSearch and Kibana
ElasticSearch has recently acquired Logstash under its wing – which is a good thing from a compatibility / support point of view.
Logstash is the engine used to receive, process and then forward your logs for storage.
ElasticSearch has a very good performance, compared to conventional database log storage – and Kibana is probably the most beautiful visualization / search engine developed.
Together they form a really nice combination – and if you don’t have the budget for a SIEM (which is the case with many companies) – is a really good evaluation option.
Just remember to make sure you calculate your storage requirements – as it might turn out that your storage will cost as much as a SIEM with good archiving capabilities (archiving (compression ratio) is one weakness of the above combination).
Yet another good tutorial from DigitalOcean on setting it up: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04
GrayLog2
Another really good open source log management product is GrayLog2 (https://www.graylog.org/) .
It is very similar in operation and performance to the ES/Logstash/Kibana – with the difference that it is a bit simpler to set up and get up and running and that the user interface of GrayLog2 does not use Kibana – which is a weakness, as Kibana is incredibly powerful in terms of what you can do with its dashboards and custom filtering / reporting capabilities.
I will spare you the screenshots / detailed technical capabilities – and I’m just mentioning the projects so you could have them in your mind when comparing to commercial solutions.
Security Onion
I cannot possibly write a chapter on security monitoring without mentioning this security linux distribution.
It is installable on your own hardware – and for a small (500 endpoints) would require at least 16 gigs of RAM and many TB of storage (as much as you can / want to afford) to store all the network traffic passing through your gateway.
It operates by running all network traffic through a set of open source intrusion detection systems. As with all of them turned on the machine would need significant amount of RAM and CPU power as well as the maximum IO you can push it is not recommended to run it on a VM. It is also recommended that your storage is built out of high-performance disks.
The distro allows you to look back in time in a way, and if an incident happens, to extract the exact packets containing the attack. The longer period you can store your network traffic for, the better.
http://blog.securityonion.net/p/securityonion.html
Feeding your SIEM with external threat intelligence data
The value of having threat intelligence data fed into your SIEM can be seen when events from your IDS/IPS/Firewalls/other devices properly correlate with it. When unknown binaries or traffic patterns enter your network this new data will help make sense of the unknowns and generate alerts when a patterns matches with a known threat indicator.
Since it is humanely impossible to read all the information passing through your SIEM you would probably rely on its alerting capability whenever it detects anything suspicious. Let us leave aside anomaly detection and spike detection for a moment and focus on threat intelligence as that is a very good source of alerts whenever something matches with your network.
External threat intelligence feeds usually comprise of file hashes, IP addresses, hostnames, domain names, Indicators of Compromise (IOCs), matches to YARA rules, etc.
Companies make a living from dissecting botnets, malware and breach investigations to produce valuable information on detecting even small portions of malware based on similarity and functionality rather than hashes (so called fuzzy hashing, for example). Then they convert that data into actionable form and sell access to it – so you could plug it into your SIEM.
Once your SIEM is fed with one or more threat intelligence feeds it could notice any similar activities, files, portions of files, accesses to suspicious networks or hosts – and alert you.
Below I will list some free and commercial threat intelligence feeds. It is your choice which ones to use, but I would put my focus on the commercial ones (after reading reviews from their customers and comparing them).
Commercial threat intelligence providers:
Kaspersky Security Intelligence Services:
http://www.kaspersky.com/enterprise-it-security/security-intelligence-services/
TrendMicro Security Threats Connect: http://www.trendmicro.com/us/security-intelligence/current-threat-activity/threat-connect/
Norse Intelligence Service: http://www.norse-corp.com/products/norse-intelligence-service/index.html (they also have a pretty cool (funny to look at) map of ongoing attack traffic here: http://map.ipviking.com/ )
iSightPartners ThreatScape®: http://www.isightpartners.com/products/threatscape/
VeriSign Security Intelligence: http://www.verisigninc.com/en_US/cyber-security/security-intelligence/threat-intelligence/index.xhtml
CrowdStrike Falcon Intelligence: http://www.crowdstrike.com/falcon-intelligence/
And finally, one service by Microsoft, which is focused on internal data analytics rather than external:
http://www.microsoft.com/en-us/server-cloud/products/advanced-threat-analytics/
Open Source / Free Threat Intelligence providers:
AlienVault Open Threat Exchange (OTX): https://www.alienvault.com/open-threat-exchange
Thanks to http://cyberwarzone.com/30-malicious-ip-list-and-block-lists-providers-2015/, we have a list of 31 (30 active) malicious IP/domain list providers:
- CriticalStack – https://intel.criticalstack.com/
- ATLAS – https://atlas.arbor.net/
- BLADE Malicious (outdated)
- CLEAN-MX Realtime Database http://support.clean-mx.de/clean-mx/viruses
- CYMRU Bogon List http://www.cymru.com/Documents/bogon-dd.html
- DShield Blocklist http://www.dshield.org/ipsascii.html
- EmergingThreats Lists http://www.emergingthreats.net/index.php/rules-mainmenu-38.html
- Google Safe Browsing API http://code.google.com/apis/safebrowsing/
- hpHosts File http://hosts-file.net/
- Malc0de Database http://malc0de.com/database/
- Malware Domain Blocklist http://www.malwaredomains.com/wordpress/?page_id=66
- Malware Patrol’s Malware Block Lists
- Malware-Control Blacklist http://www.malware-control.com/
- Malwared http://malwared.ru/database.php?page=1
- MalwareDomainList.com http://www.malwaredomainlist.com/hostslist/hosts.txt
- MalwareURL List http://www.malware.com.br/lists.shtml
- Malwr https://malwr.com/
- Nictatech http://www.nictasoft.com/viruslib/
- Norse Darklist http://www.norse-corp.com/darklist.html
- OpenPhish http://openphish.com/
- ParetoLogic URL Clearing House http://mdl.paretologic.com/
- PhishTank Phish Archive http://www.phishtank.com/phish_archive.php
- Project Honey Pot’s Directory of Malicious IPs http://www.projecthoneypot.org/list_of_ips.php
- Scumware.org http://www.scumware.org/
- Shadowserver http://www.shadowserver.org/wiki/pmwiki.php?n=Services/Reports
- Sourcefire Vulnerability Research http://labs.snort.org/iplists/
- SRI Threat Intelligence Lists http://mtc.sri.com/
- Sucuri Blacklists http://sucuri.net/?page=tools&title=blacklist
- ThreatStop http://www.threatstop.com/
- URL Blacklist http://urlblacklist.com/
- ZeuS Tracker Blocklist https://zeustracker.abuse.ch/blocklist.php , https://zeustracker.abuse.ch/monitor.php?browse=binaries
You can also use these to plug into your Web Filter appliance / proxy server blacklist.
