siem sizing

Determining Peak EPS Calculations in Logging

Published January 21, 2014 | By netcerebral

Much of the challenge in sizing and planning Centralized Log Management (CLM) and Security Information and Event Management (SIEM) solutions is determining an adequate amount of storage for storing logs for real-time analysis and archiving the log data to meet long-term retention requirements. The biggest challenge most customers face is determining the required metrics needed in sizing a solution. My post “Basic Log Storage Calculations” http://www.buzzcircuit.com/?p=208 can assist in the sizing and variables needed and my post “Guessing Game – Planning & Sizing SIEM Based on EPS” http://www.buzzcircuit.com/?p=231 can help with guessing the EPS averages for each device types. Finally, I have a couple of cool calculators at http://www.buzzcircuit.com/?p=408 and http://www.buzzcircuit.com/?p=378 that can actually assist with the final calculations.

At this point you have probably guessed that log storage calculations and storage planning is somewhat of an art, rather than a science – there’s a lot of guesswork involved, especially if you don’t have access to the systems or network devices hosting the logs. While I have done a good job (I think) in helping you dispel some of the myths and “guesstimating” an overall log capacity in previous posts, one area that is often overlooked in planning log management or SIEM is the concept of Normal EPS (NE) vs Peak EPS (PE) and ensuring your daily calculation provide a necessary contingency for consistent peaks in your event logging throughout the day.

Normal vs Peak Logging

There are two basic calculations when combining normal + peak EPS, which by no means is a hard rule. The idea is that their is the NUMBER_OF_PEAKS multiplied by the DURATION_OF_EACH_PEAK, which is then multiplied by the DEVIATION_FACTOR. To describe each of these points:

NUMBER_OF_PEAKS: calculating Peak EPS (PE) is required to factor in Normal EPS with Peaks (expressed as NE+PE) to ensure their is sufficient licensing and storage to accommodate periods of deviation from the normal EPS throughout the day. The default in the calculator below assumes there will be at least 3 peaks a day (morning logins, lunchtime web surfing, evening logoffs/backups). This value will vary based on network throttling, congestion, attacks, etc.
DURATION_OF_EACH_PEAK: This setting works in conjunction with the previous PE setting and assumes that each peak lasts for approximately 1 hour (3600 seconds) – this may vary given many factors such as how congested the network is, how busy the logging device is or other scenarios such as DDoS attacks.
DEVIATION_FACTOR: is generally 2-5x the average EPS for that period. While in reality the EPS spikes almost 20x the average EPS for only seconds, we are building in contingency for attacks such as perimeter devices under DDoS or excessive IT Operational errors that go unnoticed for hours. NOTE: again, this is an art, not a science and we’ll sound like we know more than our competitors if we think to include contingency into our calculations!

Hope you enjoy!

Posted in Log Management, Pre-Sales, SIEM | Tagged 86400, buzzcircuit.com, Calculated Fields Form, Centralized Log Management (CLM), epd, eps, events per second, log calculator, log capacity planning, log management, NE, NE+PE, Normal Events Per Second, PE, Peak Events Per Second, SIEM, siem sizing, sizing siem | Leave a comment

NetCerebral’s Device EPS Calculator

Published January 21, 2014 | By netcerebral

Hi folks, this post is another form I created using the Calculated Fields Form plugin for WordPress. Basically, this simple form calculates the number of devices input in the form fields and multiplies the number of devices by the designated Events Per Second (EPS) average for each device type. It then provides a live calculation of total number of devices, total average EPS and total average Events Per Day (EPD).

This handy calculation can then be used on my other calculator NetCerebral’s Simple Log Storage Calculator as the average EPS, need as the primary input to calculate amount of storage and IOPs required for the EPD and retention periods defined.

Posted in Log Management, Security, SIEM, Use Cases | Tagged baseline, Centralized Log Management (CLM), device logging, epd, eps, events per day, events per second, iops, log calculator, log capacity planning, log management, log storage calculator, normalized logs, raw logs, retention, security, SIEM, siem sizing, siem storage calculator, sizing siem | 1 Comment

NetCerebral’s Simple Log Storage Calculator

Published January 17, 2014 | By netcerebral

The following form assumes you have done the preliminary math of determining your number of devices and the total anticipated Events Per Second (EPS) you will be collecting from all of your logging devices. The calculator uses EPS to determine the Events Per Day (EPD), amount of raw and normalized log data you will generate daily and then use retention and compression values you set to determine the required amount of storage as well as IOPs required.

Stay tuned and I will be building another calculator that will allow you to specify number of devices by device type, which will give you your estimated EPS (needed as a starting point for this calculator).

Posted in Log Management, Security, SIEM, Templates | Tagged Application Logging, baseline, Centralized Log Management (CLM), epd, eps, iops, log calculator, log capacity planning, log management, log storage calculator, normalized logs, raw logs, retention, siem sizing, siem storage calculator, sizing siem | 1 Comment