Do I need log management?
If you need to work on this topic, I suggest you read the book of Anton Chuvakin – Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management. It has helped me tremendously in several projects, and I would like to pass on the fact that it is an incredibly useful book.
Besides that, we have a few ideas to share, which might be useful to you and are not present in the book above.
Before even starting to collect logs, you need to understand your environment. That is why audits and assessments are useful. But that is not all – you need to know how much data your devices are producing in log format every second, minute, hour, 24 hours – then you need to understand how to optimize what is logged and what is not.
It is straightforward to lose yourself in the amount of logging information generated on each of your devices, especially if you add them all at once to your SIEM (even if you add only servers/switches and firewalls and omit the desktops).
The noisiest devices in our experience are Windows systems – especially domain controllers and mail servers. Second would be your proxy servers and firewalls.
To understand how logging in a Microsoft Environment works, I suggest you read these two links:
Once you’ve read the above and understand the difference between basic and advanced audit policy configuration, move on to the next link:
It explains in detail the changes you need to make to have an optimized for detection logging environment. It also emphasizes which EventIDs you should target.
To clean up your logs and only collect what is needed, you need to know what can be discarded. To understand what is being logged, a good idea would be to export the logs from your noisiest server for a day to a CSV file, open it in Excel, and filter by EventID. It should be easy to distinguish by a percentage which events are happening most frequently and which are not.
In one case, I saw thousands of events per second with the word “Filtering engine” in them. It turns out packet logging was turned on – and every single network packet coming through the internal Windows firewall was being logged! You can imagine the number of logs generated by this server per hour and the usefulness of these logs.
Unfortunately, such a level of detail can render your SIEM and your storage incapacitated – and you should be very careful what you log and what you discard.
If you want a good start with configuring your SIEM, you could only set it to log the events in as in the book which we mentioned earlier: “Spotting the Adversary with Windows Event Log Monitoring” paper by NSA – and expand to other systems/events once you are sure you got that one right. Don’t just plug in your SIEM and all your devices into it in a shotgun manner – all your money spend on storage and SIEM will go to waste if the system is not optimized and tuned for effectiveness.
It would help if you also focused on the following events:
- 4688 Process Create (after going to Command line process auditing and enabling logging)
- 4663 File/Registry Auditing
- 4075 Service Created
- 4070 Service Changed
- 4624 User Login Success
- 5140 Share accessed
(List is taken from http://www.slideshare.net/Hackerhurricane/ask-aalware-archaeologist)
The same process can and should be repeated for all your systems/devices. Only plug a device into your SIEM solution once you fully understand the format of logging it uses, the amount/type of events, and how you could fine-tune the number of details in these logs.
You can find an excellent set of guides on configuring Windows Logging here, too.
Tools
These links will be useful during your work on optimizing your logging environment.
http://sourceforge.net/projects/syslogserverwindows/
http://sourceforge.net/projects/nxlog-ce/
LogExpert – especially this one!
One of my favorite tools to analyze logs quickly and filter them is Mandiant Highlighter.