SIEM
Linux SecOps – Look Who’s Knocking
This is a tutorial I posted on Anti-Online back in 2006 – just thought I’d update it and pass it along to the SecOps community to show you how easy it is in 2021 to do something as mundane with a SIEM that required lots of scripting back before SIEMs were common in the security field. It makes me laugh when I see some of this old scripting “Kung Fu” I had to do with Grep, Awk, Sed in order to do something that takes seconds with a good CLM or SIEM tool!
DISCLAIMER: This is a tutorial of sorts that takes you through a day-to-day problem and solution that I was often faced with in my Security Planning / Operations role for a large Telecommunications company. I am not making any assumption as to where in the curve people reading this will be situated and I don’t even guarantee this will be a good read. In fact, given my exposure and expertise of the tools used in this article, I may be missing the plot and some may find an easier, softer way of doing what I was tasked to do. Having said all of this, for those I’ve confused, sorry, I tried to provide links for further reading. For those I’ve disgusted with my simplicity or seeming Lamer approach, well, like you, I’m always learning and I’m open to criticism and advice.
Introduction
Why is it when you Google for something you absolutely need you can never find it? Well, case in fact, I had a Squid proxy server left over from a decommissioning project that was still seeing tons of traffic when it shouldn’t be seeing any! The Linux server was locked down using sudo and no one knew the root password so we had very little choices as to what programs we could run to view activity. The server was flaky and Netstat would never finish outputting the current activity. So the server folks approached me and asked if there was any way to find out what unique IP addresses internally were connecting to the five pre-configured proxy ports (8080, 8082, 8084, 8086, 8888).
As it turns out, the Squid admin user had access to the Tcpdump application and could run the application against Eth0. I got him to run Tcpdump and output it to a dump file for three hours worth of activity during the lunch hour web traffic spike. This produced a 470MB text file that I had to SFTP from his server to my Linux box.
Alrighty then! What do I do with a honkin’ text file that repeats the same info endlessly? We have hits from employees and internal servers hitting the proxy ports, the proxy itself establishing connections to the web, the foreign sites replying to the proxy and then, finally, the proxy returns the data to the corporate host. One conversation from an internal host connecting to the homepage of their favorite security tutorial site could warrant four times the number of HTTP flows. I needed to strip out extraneous information and narrow down the million+ lines of data to something sensible. So, I started thinking of the commands that would be required so that eventually I could write a shell script.
Calculating Peak EPS for Security Log Monitoring
Much of the challenge in sizing and planning Centralized Log Management (CLM), Security Intelligence Systems and Security Information and Event Management (SIEM) solutions is determining an adequate amount of storage for storing logs for real-time analysis and archiving the log data to meet long-term retention requirements. The biggest challenge most customers face is determining the required metrics needed in sizing a solution. My post “Basic Log Storage Calculations” http://www.buzzcircuit.com/?p=208 can assist in the sizing and variables needed and my post “Guessing Game – Planning & Sizing SIEM Based on EPS” http://www.buzzcircuit.com/?p=231 can help with guessing the EPS averages for each device types. Finally, I have a couple of cool calculators at http://www.buzzcircuit.com/?p=408 and http://www.buzzcircuit.com/?p=378 that can actually assist with the final calculations.
At this point you have probably guessed that log storage calculations and storage planning is somewhat of an art, rather than a science – there’s a lot of guesswork involved, especially if you don’t have access to the systems or network devices hosting the logs. While I have done a good job (I think) in helping you dispel some of the myths and “guesstimating” an overall log capacity in previous posts, one area that is often overlooked in planning log management or SIEM is the concept of Normal EPS (NE) vs Peak EPS (PE) and ensuring your daily calculation provide a necessary contingency for consistent peaks in your event logging throughout the day.
Normal vs Peak Logging
There are two basic calculations when combining normal + peak EPS, which by no means is a hard rule. The idea is that there is the NUMBER_OF_PEAKS multiplied by the DURATION_OF_EACH_PEAK, which is then multiplied by the DEVIATION_FACTOR. To describe each of these points:
- NUMBER_OF_PEAKS: calculating Peak EPS (PE) is required to factor in Normal EPS with Peaks (expressed as NE+PE) to ensure their is sufficient licensing and storage to accommodate periods of deviation from the normal EPS throughout the day. The default in the calculator below assumes there will be at least 3 peaks a day (morning logins, lunchtime web surfing, evening logoffs/backups). This value will vary based on network throttling, congestion, attacks, etc.
- DURATION_OF_EACH_PEAK: This setting works in conjunction with the previous PE setting and assumes that each peak lasts for approximately 1 hour (3600 seconds) – this may vary given many factors such as how congested the network is, how busy the logging device is or other scenarios such as DDoS attacks.
- DEVIATION_FACTOR: is generally 2-5x the average EPS for that period. While in reality the EPS spikes almost 20x the average EPS for only seconds, we are building in contingency for attacks such as perimeter devices under DDoS or excessive IT Operational errors that go unnoticed for hours. NOTE: again, this is an art, not a science and we’ll sound like we know more than our competitors if we think to include contingency into our calculations!
Hope you enjoy!
Event Log Convergence = Business Intelligence
Problem
I have come across many prospects over the last 15 years that are only trying to acquire a SIEM solution to satisfy a compliance requirement, or what we call in the industry, “check-box purchasing” – they have a minimum set of requirements specific to only one business unit or compliance mandate that is completely siloed from the rest of the organization.
Here is how this conversation usually goes:
Client: “we would like a SIEM tool that will help us monitor our 200+ Windows Servers for PCI-DSS compliance”
Me: “What other event sources are you going to be monitoring with the solution?”
Client: (Stunned look) “we only need to monitor our servers.”
Me: “PCI-DSS requirement 10 states you have to monitor the logs from all of your security devices and servers that are deemed critical assets.”
Client: “our department is only responsible for the servers we listed.”
Me: “To get value out of a SIEM solution and monitor all 12 PCI requirements you need audit logs from all of your devices and contextual information regarding your network, asset and vulnerability data – and that will just get you started.”
Client: “Perhaps we need to increase the scope – we’ll get back to you.”
While the Centralized Log Management (CLM) and Security Information and Event Management (SIEM) vendors will be lined up around the block to influence the sale, the vendor you choose should be a trusted advisor. They will be interested in providing you the most value from your investment and assist you in designing a solution to satisfy many business problems that goes beyond a traditional security-centric SIEM. This is why you will need to identify key device types and the value that can be derived by cross correlating the log data with business context to align monitoring with your governance, security and compliance initiatives.
The SIEM Value Derived from Heterogeneous Device Logging
While each of the event sources you collect events from will provide distinct reporting and alerting value, combining many different “types” of event sources will derive immediate intelligence about the business and help analysts establish baselines of threat activity. One of the other benefits is that incidents can be prioritized by business value, threat classification and the additional context can help reduce the plethora of false-positives or false-negatives that plagues every CLM solution.
Additionally, the multitude of the various technologies have their own management and reporting solutions that become “silos of information” that only the black-belts responsible for each of the device types are able to decipher. This makes security intelligence and investigations near impossible when an analyst has to request log data from the owners or log in to many different systems to find the evidence to support their cause. Essentially, they would have to piece together the clues and manually normalize the data using a technique called “Spreadsheet Kung Fu”, which would be fraught with assumptions and inference.
Below is a list of different device types and the value that can be derived from each when correlated together. The list isn’t exhaustive and I’m not suggesting you need everyone mentioned to successfully deploy a SIEM, but the more data feeds you can correlate, the more intelligence you will have available in the future to expand and grow with your business (click “more…” for complete article):
Are you a Security PreSales Ninja?
Security Ninja Quiz
Take this quiz and determine if you are a Security PreSales Ninja.
NOTE: this quiz has a 20 minute time limit to complete.
Enter your full name and email address in the results table to save to the leaderboard!
|
You must specify a text. |
|
|
You must specify an email address. |
How to Become a C.S.I. – Enterprise Forensics using a SIEM
Gary Freeman – SecTor 2013 Sponsor Session
Many Security Analysts are tasked with assisting in Corporate Governance. This session explores the concept of network forensic investigations using a SIEM, and how security analysts can use it to assist in Governance, HR or law enforcement with network interception to gather evidence that must preserve chain-of-custody. With the challenges of cloud-based computing and mobile devices, the need for well-defined workflow and the use of industry-accepted tools is even more essential than ever. Get familiar with Using integration Commands on-demand to gather external data for an investigation.
NetCerebral’s Device EPS Calculator
Hi folks, this post is another form I created using the Calculated Fields Form plugin for WordPress. Basically, this simple form calculates the number of devices input in the form fields and multiplies the number of devices by the designated Events Per Second (EPS) average for each device type. It then provides a live calculation of total number of devices, total average EPS and total average Events Per Day (EPD).
This handy calculation can then be used on my other calculator NetCerebral’s Simple Log Storage Calculator as the average EPS, need as the primary input to calculate amount of storage and IOPs required for the EPD and retention periods defined.
RFP = Really Fast Paperwork
Now I am in a management role that balances “individual contributor” with a healthy portion of “resource leadership” I have a better understanding of the impact sales has on the SE organization.
While we may have a great library of RFP responses, every new RFP has those challenging questions that will require creative writing. They’re always scenarios that no vendor (including the competitor) can address because prospects are looking to combine functionality from multiple security projects and have the SIEM tool save them budget or provide a “Swiss Army Knife” solution. These sorts of questions require clarification, advanced technical writing skills and consume hours while you ponder the final response.
An average RFP will be between 100-200 questions in length and take each resource approximately 20 minutes per question to research, cut and paste the answer, embellish, format and correct grammar before moving to the next question. With that in mind, it’s no wonder why a 100 question RFP would take an SE 30-40 hours to complete, much to the sales reps chagrin. Final formatting, cover letters and waiting for 5-10% of responses that have been farmed out to engineering, marketing or sales teams to complete, usually adds another 16-20 hours, thus totalling more than a week’s worth of work for the SE.
Now, if you don’t have an in-house, dedicated RFP response team, extensive knowledgebase or boilerplates, and you have limited SE resources that are busy with customer meetings, demos or POCs, now you have to increase the response time given that the SEs will only have time in the evenings to work on the RFP.
Having written RFPs in the past, I know it takes months to put together the requirements and agree on what product features you seek from vendors and then do your homework so you know what vendors to include in the response – yet the response deadline is usually between 7 to 15 days.
To streamline and provide appropriate resource coverage here are some things we tend to do:
- Assess the product fit and decline to respond if the RFP is clearly influenced with competitor differentiators
- Immediately ask for an extension (sometimes this is best done by the SE manager – never say you are too busy!)
- If working with a channel partner, ask them to assist in the responses (provide them with boiler-plates, past RFP responses)
- Evaluate the schedules of your SE team and determine who has the most cycles to contribute to the bulk of the work
- Consistently update a central repository of RFP knowledge with any unique questions discovered during an RFP
- Build response templates and distribute to the SE team immediately
- Get the Sales Rep involved during the RFP event, providing cover letters, company background, perhaps some of the easy technical questions
- Farm out a portion of the RFP to other SE organizations outside of your region
- Seek out internal project management teams that may be dedicated to RFP responses – they may be able to answer the easy questions, manage formatting, printing, binding and can manage the resource deadlines
- Establish and maintain relationships with some of the prospect’s technical owners, getting them to assist in wording, proposals or additional clarification questions after the question deadline (this is a gamble)
Log Management Planning Calculator
I was using the cloud service by EditGrid but they went offline – Use the three calculators I built below instead
http://www.buzzcircuit.com/determining-peak-eps-calculations-in-logging/
http://www.buzzcircuit.com/netcerebrals-device-eps-calculator/
http://www.buzzcircuit.com/simple-log-storage-calculator/
NOTE: EDITGRID IS NO LONGER IN BUSINESS…
Select the “click to edit” button at the top of the spreadsheet to start entering data. Select the drop-down button in the top left corner for features such as full-screen, download as excel and info related to EditGrid.
To use, just enter total quantity of each device type into the “Device Quantity” column. The “Per Device EPS” column provides industry averages for the event per second (eps) rate from each device type and you can change the values with your own. Next, modify the values next to the text highlighted in red under the “Event Capacity Planning” section to finish your planning.
You may want to do this separately for every remote site you plan on aggregating event for to model the bandwidth and storage planning. Continue reading
Who’s In Your Cloud?
Wikipedia cites:
Cloud Computing describes systems that provide computation, software, and data access services without requiring end-user knowledge of or dependence on the system’s physical location and configuration
As an example, take an online vacation reservation system that may be a hosted cloud model such as Software as a Service (SaaS), in which your business would host an application that consists of a web front-end, database, storage and billing services.
While the cloud provider provides an Application Programming Interface (API) and access to the various components through traditional interfaces such as SSH, FTP or SOAP, there is limited access to the underlying systems as they are usually multi-tenancy in which multiple customers share their applications on the same system. This creates challenges for monitoring and controlling the security controls governing your application.
Cloud providers will provide SLAs and frequent security reports but there is no visibility into who is administering the systems hosting your application or what vulnerabilities may be present that will allow attackers to successfully compromise the systems using SQL injection or Cross-site scripting attacks.
Cloud providers will usually allow you to conduct third-party web application penetration testing against your own URL but will not allow you to monitor their servers nor will they send you events from their network security devices (IDS/IPS, firewalls, etc), which would allow real-time correlation and threat mitigation. Essentially, you lose control of your sensitive data and who may be accessing the systems in adherence to your security policies.
With the rise of Botnets, Scareware, Phishing, Brand theft, social network vulnerabilities and many other forms of evolving malware, Cloud Computing companies that will be most successful will be those that offer security monitoring services with logical segregation that uses context regarding your business, such as:
- Real-time threat feeds
- Lists of nefarious IP addresses
- Countries of concern
- Export control
- Software vulnerabilities
- Geo-spatial disparity
- Customer activity profiling
- Privileged user accountability
- Perimeter threat baselining
- Terminated employee monitoring
With this context information correlated with real-time events gathered from all of the control points between the cloud components, customers could receive real-time alerts from the cloud and would access a GUI to drill-down and conduct post-analysis of threats and then create their own dashboards or reports regarding attackers, application issues and administration accountability.
This model would alleviate the loss of visibility by placing applications into the cloud and ensure your auditors have access to the security and compliance data they need during an assessment.
CyberCrime Investigator: Forensic Use of SIEM Tools
Wikipedia states that Network Forensics is “…proven techniques to collect, fuse, identify, examine, correlate, analyze, and document digital evidence from multiple, actively processing and transmitting digital sources for the purpose of uncovering facts related to the planned intent, or measured success of unauthorized activities meant to disrupt, corrupt, and or compromise system components as well as providing information to assist in response to or recovery from these activities…”
This business case requires a number of different tools, the most important of which is an enterprise-class Security Information and Event Management (SIEM) tool, which becomes the epicenter of all investigations and workflow. The SIEM must have some mandatory features which I will cover later in this article. But first, I would like to tell you how it’s done without SIEM.
In a previous job as a Network Security Specialist, I was in charge of tapping the wire for employee investigations and handling the data with chain-of-custody. This served as a daunting task as I would start my data captures with Open-Source software and use the spread-sheet kung-fu method of mapping all of the user activity and log data into digitally-signed archives, pending possible litigation. I established all of the guidelines and processes with support from our Legal and Corporate Fraud teams and built the procedures around the following processes:
