Determining Peak EPS Calculations in Logging

Published January 21, 2014 | By netcerebral

Much of the challenge in sizing and planning Centralized Log Management (CLM) and Security Information and Event Management (SIEM) solutions is determining an adequate amount of storage for storing logs for real-time analysis and archiving the log data to meet long-term retention requirements. The biggest challenge most customers face is determining the required metrics needed in sizing a solution. My post “Basic Log Storage Calculations” http://www.buzzcircuit.com/?p=208 can assist in the sizing and variables needed and my post “Guessing Game – Planning & Sizing SIEM Based on EPS” http://www.buzzcircuit.com/?p=231 can help with guessing the EPS averages for each device types. Finally, I have a couple of cool calculators at http://www.buzzcircuit.com/?p=408 and http://www.buzzcircuit.com/?p=378 that can actually assist with the final calculations.

At this point you have probably guessed that log storage calculations and storage planning is somewhat of an art, rather than a science – there’s a lot of guesswork involved, especially if you don’t have access to the systems or network devices hosting the logs. While I have done a good job (I think) in helping you dispel some of the myths and “guesstimating” an overall log capacity in previous posts, one area that is often overlooked in planning log management or SIEM is the concept of Normal EPS (NE) vs Peak EPS (PE) and ensuring your daily calculation provide a necessary contingency for consistent peaks in your event logging throughout the day.

Normal vs Peak Logging

There are two basic calculations when combining normal + peak EPS, which by no means is a hard rule. The idea is that their is the NUMBER_OF_PEAKS multiplied by the DURATION_OF_EACH_PEAK, which is then multiplied by the DEVIATION_FACTOR. To describe each of these points:

NUMBER_OF_PEAKS: calculating Peak EPS (PE) is required to factor in Normal EPS with Peaks (expressed as NE+PE) to ensure their is sufficient licensing and storage to accommodate periods of deviation from the normal EPS throughout the day. The default in the calculator below assumes there will be at least 3 peaks a day (morning logins, lunchtime web surfing, evening logoffs/backups). This value will vary based on network throttling, congestion, attacks, etc.
DURATION_OF_EACH_PEAK: This setting works in conjunction with the previous PE setting and assumes that each peak lasts for approximately 1 hour (3600 seconds) – this may vary given many factors such as how congested the network is, how busy the logging device is or other scenarios such as DDoS attacks.
DEVIATION_FACTOR: is generally 2-5x the average EPS for that period. While in reality the EPS spikes almost 20x the average EPS for only seconds, we are building in contingency for attacks such as perimeter devices under DDoS or excessive IT Operational errors that go unnoticed for hours. NOTE: again, this is an art, not a science and we’ll sound like we know more than our competitors if we think to include contingency into our calculations!