epd
Calculating Peak EPS for Security Log Monitoring
Much of the challenge in sizing and planning Centralized Log Management (CLM), Security Intelligence Systems and Security Information and Event Management (SIEM) solutions is determining an adequate amount of storage for storing logs for real-time analysis and archiving the log data to meet long-term retention requirements. The biggest challenge most customers face is determining the required metrics needed in sizing a solution. My post “Basic Log Storage Calculations” http://www.buzzcircuit.com/?p=208 can assist in the sizing and variables needed and my post “Guessing Game – Planning & Sizing SIEM Based on EPS” http://www.buzzcircuit.com/?p=231 can help with guessing the EPS averages for each device types. Finally, I have a couple of cool calculators at http://www.buzzcircuit.com/?p=408 and http://www.buzzcircuit.com/?p=378 that can actually assist with the final calculations.
At this point you have probably guessed that log storage calculations and storage planning is somewhat of an art, rather than a science – there’s a lot of guesswork involved, especially if you don’t have access to the systems or network devices hosting the logs. While I have done a good job (I think) in helping you dispel some of the myths and “guesstimating” an overall log capacity in previous posts, one area that is often overlooked in planning log management or SIEM is the concept of Normal EPS (NE) vs Peak EPS (PE) and ensuring your daily calculation provide a necessary contingency for consistent peaks in your event logging throughout the day.
Normal vs Peak Logging
There are two basic calculations when combining normal + peak EPS, which by no means is a hard rule. The idea is that there is the NUMBER_OF_PEAKS multiplied by the DURATION_OF_EACH_PEAK, which is then multiplied by the DEVIATION_FACTOR. To describe each of these points:
- NUMBER_OF_PEAKS: calculating Peak EPS (PE) is required to factor in Normal EPS with Peaks (expressed as NE+PE) to ensure their is sufficient licensing and storage to accommodate periods of deviation from the normal EPS throughout the day. The default in the calculator below assumes there will be at least 3 peaks a day (morning logins, lunchtime web surfing, evening logoffs/backups). This value will vary based on network throttling, congestion, attacks, etc.
- DURATION_OF_EACH_PEAK: This setting works in conjunction with the previous PE setting and assumes that each peak lasts for approximately 1 hour (3600 seconds) – this may vary given many factors such as how congested the network is, how busy the logging device is or other scenarios such as DDoS attacks.
- DEVIATION_FACTOR: is generally 2-5x the average EPS for that period. While in reality the EPS spikes almost 20x the average EPS for only seconds, we are building in contingency for attacks such as perimeter devices under DDoS or excessive IT Operational errors that go unnoticed for hours. NOTE: again, this is an art, not a science and we’ll sound like we know more than our competitors if we think to include contingency into our calculations!
Hope you enjoy!
NetCerebral’s Device EPS Calculator
Hi folks, this post is another form I created using the Calculated Fields Form plugin for WordPress. Basically, this simple form calculates the number of devices input in the form fields and multiplies the number of devices by the designated Events Per Second (EPS) average for each device type. It then provides a live calculation of total number of devices, total average EPS and total average Events Per Day (EPD).
This handy calculation can then be used on my other calculator NetCerebral’s Simple Log Storage Calculator as the average EPS, need as the primary input to calculate amount of storage and IOPs required for the EPD and retention periods defined.
NetCerebral’s Simple Log Storage Calculator
The following form assumes you have done the preliminary math of determining your number of devices and the total anticipated Events Per Second (EPS) you will be collecting from all of your logging devices. The calculator uses EPS to determine the Events Per Day (EPD), amount of raw and normalized log data you will generate daily and then use retention and compression values you set to determine the required amount of storage as well as IOPs required.
Stay tuned and I will be building another calculator that will allow you to specify number of devices by device type, which will give you your estimated EPS (needed as a starting point for this calculator).
Guessing Game – Planning & Sizing SIEM Based on EPS
Many of the competing log management and SIEM tools on the market these days use some variation 0f the Events Per Second (EPS) metric to determine the licensing, sizing and storage requirements for scalable solution. Unfortunately, none of the devices that are to be monitored have a specification associated with the amount of logging which will be generated per second (or volume for day, for that matter!) by the device. Moreover, many of the same device type from the same vendor will generate varying amounts of log volume daily and it’s more of an art than a science when determining what the total volume all of the corporate devices will generate daily.
Determining EPS isn’t a problem for existing log management or SIEM customers looking to upgrade to a new solution as they can generate reports from the old log management/SIEM tool and provide a break-down of device type and the daily volumes generated by each device category. However, prospects looking for a proposal for a net-new solution are plagued with the following tasks to properly design a log management or SIEM solution:
- Complete inventory of all assets they plan on monitoring
- Determining average, sustained event rates expressed as an EPS metric
- Understanding how logging levels impact the volume of logs that are generated
- Retention periods, storage options, use cases, regulatory requirements, ad infinitum
Fortunately, once you have a device count and can determine the EPS generated on average by each of the different device categories you need to monitor, the math is easy to determine the licensing, storage, system performance and archiving needs. My post “Basic Log Storage Calculations” http://www.netcerebral.com/?p=208 can assist in the sizing, as this post is geared more towards guessing the EPS averages for each device types.
In my roles as a presales SE that sold log management and SIEM we often were asked by prospects for budgetary quotes, proposals and architecture with little to no empirical data. In most cases the best we could get out of the prospect is an itemized inventory of the number and types of systems they would like to monitor. Without an understanding of the log volumes generated by devices, unique to every customer’s environment, we had to come up with a system of determining the EPS for the different device classes and using this as a starting point for calculating daily storage (EPS * Event_Size * 84600 / Compression Ratio).
The list below is an example of lessons learned in the field from actual customer environments and a document provided by SANS (sponsored by NitroSecurity – now McAfee) called “Benchmarking Security Information Event Management (SIEM)” (found at http://www.sans.org/reading_room/analysts_program/eventMgt_Feb09.pdf). With the information we collected we devised a list, which is a cross-section of averages per event source.
I hope you find this helpful:
Basic Log Storage Calculations
Determining the sizes of log management systems requires knowledge of the number of devices being monitored and the anticipated event rates for each class of system. In many customer engagements, Professional Services time may be required to measure the event rate calculations from all of the monitored devices. This is important since there are too many variables to predict the average or peak Events Per Day (EPS) of any given system. I would caution any customer that if the vendor they are working with gives them “magic” calculations and pricing without gathering the necessary information regarding customer-specific speeds and feeds, they can expect to spend a lot more money later once the vendor gets their foot in the door. Basically, poor planning will result in unavoidable OP/EX costs later.
EPS is one metric used by many log management and SIEM vendors to determine such factors as licensing, storage and peak system loads. Another variable used could be Events Per Day (EPD), especially when it relates to storage sizing and license enforcement. This is why it’s imperative that accurate device counts and product types are audited when planning a centralized log management or SIEM solution.
Log Management Planning Calculator
I was using the cloud service by EditGrid but they went offline – Use the three calculators I built below instead
http://www.buzzcircuit.com/determining-peak-eps-calculations-in-logging/
http://www.buzzcircuit.com/netcerebrals-device-eps-calculator/
http://www.buzzcircuit.com/simple-log-storage-calculator/
NOTE: EDITGRID IS NO LONGER IN BUSINESS…
Select the “click to edit” button at the top of the spreadsheet to start entering data. Select the drop-down button in the top left corner for features such as full-screen, download as excel and info related to EditGrid.
To use, just enter total quantity of each device type into the “Device Quantity” column. The “Per Device EPS” column provides industry averages for the event per second (eps) rate from each device type and you can change the values with your own. Next, modify the values next to the text highlighted in red under the “Event Capacity Planning” section to finish your planning.
You may want to do this separately for every remote site you plan on aggregating event for to model the bandwidth and storage planning. Continue reading
- Mom’s Meals issues “Notice of Data Event”: What to know and what to do
- S3 Ep149: How many cryptographers does it take to change a light bulb?
- Using WinRAR? Be sure to patch against these code execution bugs…
- Smart light bulbs could give away your password secrets
- “Snakes in airplane mode” – what if your phone says it’s offline but isn’t?
- S3 Ep148: Remembering crypto heroes
- FBI warns about scams that lure you in as a mobile beta-tester
- “Grab hold and give it a wiggle” – ATM card skimming is still a thing
- Crimeware server used by NetWalker ransomware seized and shut down
- S3 Ep147: What if you type in your password during a meeting?