Pre-Sales
Linux SecOps – Look Who’s Knocking
This is a tutorial I posted on Anti-Online back in 2006 – just thought I’d update it and pass it along to the SecOps community to show you how easy it is in 2021 to do something as mundane with a SIEM that required lots of scripting back before SIEMs were common in the security field. It makes me laugh when I see some of this old scripting “Kung Fu” I had to do with Grep, Awk, Sed in order to do something that takes seconds with a good CLM or SIEM tool!
DISCLAIMER: This is a tutorial of sorts that takes you through a day-to-day problem and solution that I was often faced with in my Security Planning / Operations role for a large Telecommunications company. I am not making any assumption as to where in the curve people reading this will be situated and I don’t even guarantee this will be a good read. In fact, given my exposure and expertise of the tools used in this article, I may be missing the plot and some may find an easier, softer way of doing what I was tasked to do. Having said all of this, for those I’ve confused, sorry, I tried to provide links for further reading. For those I’ve disgusted with my simplicity or seeming Lamer approach, well, like you, I’m always learning and I’m open to criticism and advice.
Introduction
Why is it when you Google for something you absolutely need you can never find it? Well, case in fact, I had a Squid proxy server left over from a decommissioning project that was still seeing tons of traffic when it shouldn’t be seeing any! The Linux server was locked down using sudo and no one knew the root password so we had very little choices as to what programs we could run to view activity. The server was flaky and Netstat would never finish outputting the current activity. So the server folks approached me and asked if there was any way to find out what unique IP addresses internally were connecting to the five pre-configured proxy ports (8080, 8082, 8084, 8086, 8888).
As it turns out, the Squid admin user had access to the Tcpdump application and could run the application against Eth0. I got him to run Tcpdump and output it to a dump file for three hours worth of activity during the lunch hour web traffic spike. This produced a 470MB text file that I had to SFTP from his server to my Linux box.
Alrighty then! What do I do with a honkin’ text file that repeats the same info endlessly? We have hits from employees and internal servers hitting the proxy ports, the proxy itself establishing connections to the web, the foreign sites replying to the proxy and then, finally, the proxy returns the data to the corporate host. One conversation from an internal host connecting to the homepage of their favorite security tutorial site could warrant four times the number of HTTP flows. I needed to strip out extraneous information and narrow down the million+ lines of data to something sensible. So, I started thinking of the commands that would be required so that eventually I could write a shell script.
Calculating Peak EPS for Security Log Monitoring
Much of the challenge in sizing and planning Centralized Log Management (CLM), Security Intelligence Systems and Security Information and Event Management (SIEM) solutions is determining an adequate amount of storage for storing logs for real-time analysis and archiving the log data to meet long-term retention requirements. The biggest challenge most customers face is determining the required metrics needed in sizing a solution. My post “Basic Log Storage Calculations” http://www.buzzcircuit.com/?p=208 can assist in the sizing and variables needed and my post “Guessing Game – Planning & Sizing SIEM Based on EPS” http://www.buzzcircuit.com/?p=231 can help with guessing the EPS averages for each device types. Finally, I have a couple of cool calculators at http://www.buzzcircuit.com/?p=408 and http://www.buzzcircuit.com/?p=378 that can actually assist with the final calculations.
At this point you have probably guessed that log storage calculations and storage planning is somewhat of an art, rather than a science – there’s a lot of guesswork involved, especially if you don’t have access to the systems or network devices hosting the logs. While I have done a good job (I think) in helping you dispel some of the myths and “guesstimating” an overall log capacity in previous posts, one area that is often overlooked in planning log management or SIEM is the concept of Normal EPS (NE) vs Peak EPS (PE) and ensuring your daily calculation provide a necessary contingency for consistent peaks in your event logging throughout the day.
Normal vs Peak Logging
There are two basic calculations when combining normal + peak EPS, which by no means is a hard rule. The idea is that there is the NUMBER_OF_PEAKS multiplied by the DURATION_OF_EACH_PEAK, which is then multiplied by the DEVIATION_FACTOR. To describe each of these points:
- NUMBER_OF_PEAKS: calculating Peak EPS (PE) is required to factor in Normal EPS with Peaks (expressed as NE+PE) to ensure their is sufficient licensing and storage to accommodate periods of deviation from the normal EPS throughout the day. The default in the calculator below assumes there will be at least 3 peaks a day (morning logins, lunchtime web surfing, evening logoffs/backups). This value will vary based on network throttling, congestion, attacks, etc.
- DURATION_OF_EACH_PEAK: This setting works in conjunction with the previous PE setting and assumes that each peak lasts for approximately 1 hour (3600 seconds) – this may vary given many factors such as how congested the network is, how busy the logging device is or other scenarios such as DDoS attacks.
- DEVIATION_FACTOR: is generally 2-5x the average EPS for that period. While in reality the EPS spikes almost 20x the average EPS for only seconds, we are building in contingency for attacks such as perimeter devices under DDoS or excessive IT Operational errors that go unnoticed for hours. NOTE: again, this is an art, not a science and we’ll sound like we know more than our competitors if we think to include contingency into our calculations!
Hope you enjoy!
Event Log Convergence = Business Intelligence
Problem
I have come across many prospects over the last 15 years that are only trying to acquire a SIEM solution to satisfy a compliance requirement, or what we call in the industry, “check-box purchasing” – they have a minimum set of requirements specific to only one business unit or compliance mandate that is completely siloed from the rest of the organization.
Here is how this conversation usually goes:
Client: “we would like a SIEM tool that will help us monitor our 200+ Windows Servers for PCI-DSS compliance”
Me: “What other event sources are you going to be monitoring with the solution?”
Client: (Stunned look) “we only need to monitor our servers.”
Me: “PCI-DSS requirement 10 states you have to monitor the logs from all of your security devices and servers that are deemed critical assets.”
Client: “our department is only responsible for the servers we listed.”
Me: “To get value out of a SIEM solution and monitor all 12 PCI requirements you need audit logs from all of your devices and contextual information regarding your network, asset and vulnerability data – and that will just get you started.”
Client: “Perhaps we need to increase the scope – we’ll get back to you.”
While the Centralized Log Management (CLM) and Security Information and Event Management (SIEM) vendors will be lined up around the block to influence the sale, the vendor you choose should be a trusted advisor. They will be interested in providing you the most value from your investment and assist you in designing a solution to satisfy many business problems that goes beyond a traditional security-centric SIEM. This is why you will need to identify key device types and the value that can be derived by cross correlating the log data with business context to align monitoring with your governance, security and compliance initiatives.
The SIEM Value Derived from Heterogeneous Device Logging
While each of the event sources you collect events from will provide distinct reporting and alerting value, combining many different “types” of event sources will derive immediate intelligence about the business and help analysts establish baselines of threat activity. One of the other benefits is that incidents can be prioritized by business value, threat classification and the additional context can help reduce the plethora of false-positives or false-negatives that plagues every CLM solution.
Additionally, the multitude of the various technologies have their own management and reporting solutions that become “silos of information” that only the black-belts responsible for each of the device types are able to decipher. This makes security intelligence and investigations near impossible when an analyst has to request log data from the owners or log in to many different systems to find the evidence to support their cause. Essentially, they would have to piece together the clues and manually normalize the data using a technique called “Spreadsheet Kung Fu”, which would be fraught with assumptions and inference.
Below is a list of different device types and the value that can be derived from each when correlated together. The list isn’t exhaustive and I’m not suggesting you need everyone mentioned to successfully deploy a SIEM, but the more data feeds you can correlate, the more intelligence you will have available in the future to expand and grow with your business (click “more…” for complete article):
Are you a Security PreSales Ninja?
Security Ninja Quiz
Take this quiz and determine if you are a Security PreSales Ninja.
NOTE: this quiz has a 20 minute time limit to complete.
Enter your full name and email address in the results table to save to the leaderboard!
|
You must specify a text. |
|
|
You must specify an email address. |
How to Become a C.S.I. – Enterprise Forensics using a SIEM
Gary Freeman – SecTor 2013 Sponsor Session
Many Security Analysts are tasked with assisting in Corporate Governance. This session explores the concept of network forensic investigations using a SIEM, and how security analysts can use it to assist in Governance, HR or law enforcement with network interception to gather evidence that must preserve chain-of-custody. With the challenges of cloud-based computing and mobile devices, the need for well-defined workflow and the use of industry-accepted tools is even more essential than ever. Get familiar with Using integration Commands on-demand to gather external data for an investigation.
Guessing Game – Planning & Sizing SIEM Based on EPS
Many of the competing log management and SIEM tools on the market these days use some variation 0f the Events Per Second (EPS) metric to determine the licensing, sizing and storage requirements for scalable solution. Unfortunately, none of the devices that are to be monitored have a specification associated with the amount of logging which will be generated per second (or volume for day, for that matter!) by the device. Moreover, many of the same device type from the same vendor will generate varying amounts of log volume daily and it’s more of an art than a science when determining what the total volume all of the corporate devices will generate daily.
Determining EPS isn’t a problem for existing log management or SIEM customers looking to upgrade to a new solution as they can generate reports from the old log management/SIEM tool and provide a break-down of device type and the daily volumes generated by each device category. However, prospects looking for a proposal for a net-new solution are plagued with the following tasks to properly design a log management or SIEM solution:
- Complete inventory of all assets they plan on monitoring
- Determining average, sustained event rates expressed as an EPS metric
- Understanding how logging levels impact the volume of logs that are generated
- Retention periods, storage options, use cases, regulatory requirements, ad infinitum
Fortunately, once you have a device count and can determine the EPS generated on average by each of the different device categories you need to monitor, the math is easy to determine the licensing, storage, system performance and archiving needs. My post “Basic Log Storage Calculations” http://www.netcerebral.com/?p=208 can assist in the sizing, as this post is geared more towards guessing the EPS averages for each device types.
In my roles as a presales SE that sold log management and SIEM we often were asked by prospects for budgetary quotes, proposals and architecture with little to no empirical data. In most cases the best we could get out of the prospect is an itemized inventory of the number and types of systems they would like to monitor. Without an understanding of the log volumes generated by devices, unique to every customer’s environment, we had to come up with a system of determining the EPS for the different device classes and using this as a starting point for calculating daily storage (EPS * Event_Size * 84600 / Compression Ratio).
The list below is an example of lessons learned in the field from actual customer environments and a document provided by SANS (sponsored by NitroSecurity – now McAfee) called “Benchmarking Security Information Event Management (SIEM)” (found at http://www.sans.org/reading_room/analysts_program/eventMgt_Feb09.pdf). With the information we collected we devised a list, which is a cross-section of averages per event source.
I hope you find this helpful:
RFP = Really Fast Paperwork
Now I am in a management role that balances “individual contributor” with a healthy portion of “resource leadership” I have a better understanding of the impact sales has on the SE organization.
While we may have a great library of RFP responses, every new RFP has those challenging questions that will require creative writing. They’re always scenarios that no vendor (including the competitor) can address because prospects are looking to combine functionality from multiple security projects and have the SIEM tool save them budget or provide a “Swiss Army Knife” solution. These sorts of questions require clarification, advanced technical writing skills and consume hours while you ponder the final response.
An average RFP will be between 100-200 questions in length and take each resource approximately 20 minutes per question to research, cut and paste the answer, embellish, format and correct grammar before moving to the next question. With that in mind, it’s no wonder why a 100 question RFP would take an SE 30-40 hours to complete, much to the sales reps chagrin. Final formatting, cover letters and waiting for 5-10% of responses that have been farmed out to engineering, marketing or sales teams to complete, usually adds another 16-20 hours, thus totalling more than a week’s worth of work for the SE.
Now, if you don’t have an in-house, dedicated RFP response team, extensive knowledgebase or boilerplates, and you have limited SE resources that are busy with customer meetings, demos or POCs, now you have to increase the response time given that the SEs will only have time in the evenings to work on the RFP.
Having written RFPs in the past, I know it takes months to put together the requirements and agree on what product features you seek from vendors and then do your homework so you know what vendors to include in the response – yet the response deadline is usually between 7 to 15 days.
To streamline and provide appropriate resource coverage here are some things we tend to do:
- Assess the product fit and decline to respond if the RFP is clearly influenced with competitor differentiators
- Immediately ask for an extension (sometimes this is best done by the SE manager – never say you are too busy!)
- If working with a channel partner, ask them to assist in the responses (provide them with boiler-plates, past RFP responses)
- Evaluate the schedules of your SE team and determine who has the most cycles to contribute to the bulk of the work
- Consistently update a central repository of RFP knowledge with any unique questions discovered during an RFP
- Build response templates and distribute to the SE team immediately
- Get the Sales Rep involved during the RFP event, providing cover letters, company background, perhaps some of the easy technical questions
- Farm out a portion of the RFP to other SE organizations outside of your region
- Seek out internal project management teams that may be dedicated to RFP responses – they may be able to answer the easy questions, manage formatting, printing, binding and can manage the resource deadlines
- Establish and maintain relationships with some of the prospect’s technical owners, getting them to assist in wording, proposals or additional clarification questions after the question deadline (this is a gamble)
Essential Tools Every SE Needs
I have only been doing the SE gig for three years now and I have learned a lot from my own mistakes or from other SEs I work with. Here’s a compilation of things that I regard as being the “trade-craft” for Sales Engineers:
Laptop: Your company will probably outfit you with your own laptopwith enough juice to run a VMWare demo and PowerPoint slide deck simultaneously, while projecting your desktop onto a presentation screen. Make sure you do the regular maintenance so your system will keep up with the demands. There’s nothing more frustrating than being in the middle of a 3 hour demo and your system freezes for no reason and you have to reboot, thus delaying the demo and creating a 10 minute window when the customer loses interest in what you are presenting. Additionally, if your company allows you to claim business related expenses, drop the cash on a solid-state drive for your demo VMs and install the maximum amount of RAM your laptop can handle.
SIEM Sales Engineer Top 5 Trial Tips
In my role as an SE there are many challenges during the sales cycle that need to be addressed and I have had my share of dents and bruises because I didn’t adhere to any part of the 5 tips I’ve detailed below. These are my top 5 tips that have been instrumental in making trials run smoothly:
Scoping: Winston Churchill once said “Let our advance worrying become advance thinking and planning”. Too often in my early days as an SE would I anticipate all that could go wrong on a trial because I didn’t have enough experience with what could go “right”. Once you have a few successful trials under your belt you begin to see the trends and tasks that are most effective.
One of the best tools in preparation for any product trial is the Proof of Concept (POC) scoping call. Arrange a quick conference call with your prospect and ask them to extend the meeting to all of the various technology and security stake-holders in their organization. During the call, establish the duration, shipping addresses for equipment, contacts, event sources, success criteria, use cases, custom content, etc. My company has actually devised a POC scoping document that must be filled out before any equipment gets shipped to the prospect and anyone books their travel.
Finally, you have to accommodate for scope creep and the best way to do this is to limit the scope up front. You will have a lot on your plate, especially if it’s only a 5 day POC. Then, while you are on site during the trial you are going to create excitement about your product (if everything goes as planned).
Come Prepared: Not only does this include printing out the POC scoping document, configuration guides and address/contact for the site, but this also includes any work you can get done prior to the trial. Some such tasks are; map out each of the use cases and how you plan on satisfying the criteria; If there are custom rules, reports and dashboards, use demo events to pre-create this on your laptop to minimize how much effort you’ll spend on developing this in front of the customer (risking how complex the product is to use).
There are many other preparation steps including: building the necessary integration into unsupported event sources and; scheduling other teams for a WebEx session during the POC to speak about product roadmaps, customer support, customer success, advanced use cases, to name a few.
Ask For Help: This point lends itself to the “preparation” phase, whereby you may discover during the scoping exercise that there is a difficult event source to connect to or use cases that you don’t have the first clue on how to address. Reach out to the SE thread by email or call other SEs direct who may be senior and may have encountered this complexity in the past. At the very least, request assistance from the product support team and perhaps schedule either onsite or remote assistance during the POC so that the issues can be addressed with some extra hand-holding. In some instances, speaking with the prospect and suggesting alternatives may be the answer. This job is about being abreast of all sorts of technology and you must be humble and resourceful to be successful.
Be a Consultant: We are not just representing the product sales team in our roll. We are teachers, mentors, architects, IT specialists, project managers, to name a few of the required roles during a POC. We must advocate the value proposition as well as provide guidance of how our system is going to retrofit into an existing enterprise environment. Schedule planning sessions with the architecture, security planning and compliance teams during the POC to whiteboard the proposed design and capture as much data as possible. Share planning questionnaires, proposed diagrams, key differentiators, use case matrixes, product installation documents and anything else you feel would benefit the prospect in determining that you are an authority on Log Management and SIEM technology. We are the authority on SIEM and we are not just there to let the prospect put our feet to the fire and satisfy use cases – we are their to inform them on making the right decision.
Share the Wealth: The final point here is one of the most overlooked aspects amongst SEs and is instrumental in your development. Great responsibility comes with being a specialist in your profession. Whether you’ve created custom content or worked day and night fixing a problem, you need to share this with other SEs. Beyond the obvious point of helping your colleagues be successful is the paradox “you can’t keep it unless you give it away”. What this means is that some of the most complex concepts need to be articulated for you, yourself, to truly understand how they work. When sharing content or templates you’ve created it forces you to figure out how to deliver your ideas in a way that can be easily digested by your peers. The more you do it, the more you will master your profession.
These tips are by no means the silver bullet for being successful during trials but they have worked for me and as you can tell, the center around being prepared and having all of the facts.
Good luck and happy selling!
Gary Freeman
Getting them to drink the Kool-aid…
I have to say that one of the most incredible feelings is walking away from a demo or presentation knowing that the audience got what our product does and how flexible it is to deploy. I just get a rush when technical folks start drinking the kool-aid and ask questions or regurgitate our product’ terminology. Even better is one of your prospects has done their homework and starts advocating the benefits or the value proposition to their own colleagues!
I work with a product that is a niche product that is both software and hardware and has many flexible options. The sky’s the limit and when a serious prospect starts formulating how our solution will retro-fit their environment and helps you map out the architecture before the meeting is finished, you know there’s no denying the fact they are onboard and ready to get their hands dirty with your product.
In one such engagement recently, I walked into the room, one immediate credibility with the tech team as had been on the other side of the table with the same pains before doing a Sales Engineering role. I opted not to do “death-by-PowerPoint” and went straight to the white board and started mapping out their requirements and architecture while explaining the product and features at each tier of the design. Customers respond well to a reference architecture as it provides a mapping to their existing infrastructure and allows them to absorb your infiltration of product knowledge in digestible chunks.
Some of the things I do to engage them are:
- Ask them tons of questions about thier pain and what they are doing to deal with the problems today
- Let them come up to the whiteboard and draw their architecture
- Build trust with the technical folks but also try to deliver the techno babble to the middle ground (there are always non-tech people present who’ll get bored if you don’t engage them)
- Offer to give them whitepapers, admin guides, specs, etc – techies love to read!
- Let them take you off script and drive the demo (show them what they want to see)
- Manage difficult questions by qualifying the purpose for knowing the answer
Being an SE lets me sell, train, demostrate, propose, design and essentially talk people into buying a solution to suit their needs, but the job is so much easier when I’ve established a repore with the right people.
- Mom’s Meals issues “Notice of Data Event”: What to know and what to do
- S3 Ep149: How many cryptographers does it take to change a light bulb?
- Using WinRAR? Be sure to patch against these code execution bugs…
- Smart light bulbs could give away your password secrets
- “Snakes in airplane mode” – what if your phone says it’s offline but isn’t?
- S3 Ep148: Remembering crypto heroes
- FBI warns about scams that lure you in as a mobile beta-tester
- “Grab hold and give it a wiggle” – ATM card skimming is still a thing
- Crimeware server used by NetWalker ransomware seized and shut down
- S3 Ep147: What if you type in your password during a meeting?