The Inevitable Evolution of SIEMs
August 5, 2019 • Zane Pokorny
Our guest today is Monzy Merza. He’s the VP and head of security research at Splunk.
He shares his journey into tech and security, including leadership positions in both the government and private sectors, his thoughts on threat intelligence and the maturity companies need to properly implement it, as well as his perspective on the current state of SIEMs, and how they’ll need to evolve to keep up with the changes happening in the industry and the world at large.
This podcast was produced in partnership with the CyberWire.
For those of you who’d prefer to read, here’s the transcript:
This is Recorded Future, inside threat intelligence for cybersecurity.
Hello everyone, and welcome to episode 119 of the Recorded Future podcast. I’m Dave Bittner from the CyberWire.
Our guest today is Monzy Merza. He’s the VP and head of security research at Splunk. He shares his journey into tech and security, including leadership positions in both the government and private sectors, his thoughts on threat intelligence and the maturity companies need to properly implement it, as well as his perspective on the current state of SIEMs and how they’ll need to evolve to keep up with the changes happening in the industry and the world at large.
Stay with us.
I took my first “programming” class, I was the only person in that class because my dad was able to arrange a thing with the institution when I was in fifth grade, so I learned to program in BASIC. You can think of it as a professional tutor almost, so that’s how I learned it. I was too young, but they were like, “How can this kid do this?” My dad said, “No, no. He understands these things. He likes to tinker.” That was my start.
Then, I had a really sort of meandering career. I was going to go to college for other things, not computer science-related. Then I dropped out of college and I started my own little business doing web development and stuff for a little while. After that, I started doing odds and ends kind of things. The Y2K stuff happening, I started doing a lot of migrations for just consulting, contracting work.
Then, ultimately, I landed working for the Department of Energy. My first job was with them at Los Alamos National Labs. That’s when I really started to get into a little bit of security. There’s a big security culture at the Department of Energy, so just started learning things there and then eventually moved into the security teams there. I like to say, you know, my entry to the “security space” was by demonstration. We’ll leave that for another conversation over coffee. But …
I see where you’re going with that.
But, someone decided that it was better to have this person be in an organized environment.
I was fortunate for that. I had good mentors and teachers and so that’s how I got into the security stuff. Then I learned to do a lot of interesting things, worked on some pretty amazing teams. It was a big privilege working for the government doing a lot of different things, both on the offensive side and the defensive side, and also not just trying to break things, but I was also trying to build tools, both hardware and software tools. It was a lot of fun, and now I’ve been at Splunk for about eight years and a quarter. Things are pretty exciting. I really, I like to say I really only have had two real jobs. Because, my work in government was probably about 13 years long, 14 years long almost, and then my other part of my life has been at Splunk for about eight years. Prior to that it was just a lot of odds and ends kinds of things that I did.
What is your day to day like there at Splunk?
My job responsibilities are really three part. One, as a leader on the security market group, which by itself is a pretty sizable chunk of the $2 billion revenue company that is Splunk, Incorporated. As a leader in that business unit, or in that market group, we don’t call them business units. In that market group, I participate with the rest of the leadership team on making, on all the strategic decisions that you can imagine one requires for the business, whether it’s M&A related, whether it’s directionally-related, whether it’s product-related or people-related. That’s a lot of internal facing stuff that I participate in. I spend significant time there.
The other two thirds of, or another third of my time is spent, because I am the head of research and the vice president of security research for Splunk, I have my own teams and I spend a lot of time with those teams and collaborating and leading the research initiatives, both from a people management point of view and also from a technology and research point of view. I like to say I still keep my hands dirty, so I just don’t talk about stuff, I do things as well on those teams and explore areas.
The last third of my job as part of being with Splunk and given my background and my curiosities, I spend a lot of time with our customers, with Splunk’s customers, talking about what Splunk is doing, where Splunk is going and also really learning from customers in terms of what is it that they’d like to see, what are the challenges that they face, not just technological but operational issues and so on, so we can bring that back and reflect that and be able to serve the customers with the amazing products that Splunk creates and continue to keep them amazing and be of service to customers.
Well, for folks who might not be familiar with Splunk, obviously a well known name in the industry, can you give us an overview of the type of things that you all handle there?
Yeah, so Splunk’s mission in life is, what we like to say, is to make machine data accessible, usable, and valuable for everybody. If you think of machine data, it’s everything. It’s this digital exhaust of everything from firewalls to point of sale systems to IoT devices to trains and all sorts of these things. All these machines, when they talk to each other and communicate with each other, they generate machine data.
Splunk makes that machine data usable and accessible and valuable for people. What ends up happening is, when you have this data, what can you do with it? Well, you can create reports so you can understand how many devices you have, or if you get more advanced you can start to do troubleshooting because you can figure out how an email system or a retail system, web service is working in core correlation with the backend storage systems.
In the security space, Splunk can be utilized for SIEM operations, so you could do security incident and event management or threat detection or automated response by way of Splunk’s capability. Of course, those are the use cases. Underneath the hood, Splunk is this amazing platform that can collect information from anywhere, any kind of system, and can openly integrate with other systems and has capabilities for machine learning and analysis and searching and so on, so that you can get value out of the things that you’re trying to do. Essentially, at the end of the day, really allows you to investigate, monitor, analyze, and act on the data that you’re collecting so that you can move your business or your mission forward.
Well, let’s dig in a bit and talk about threat intelligence and your take on that, how it fits into an organization’s defenses.
I think threat intelligence, when I talk to our customers there are a couple of different classes in which they evaluate threat intelligence or they see threat intelligence, and then, I’ll share their view first. I like to be customer led, and then I’ll share a little bit of my view from my own history in practice.
From a customer point of view, threat intelligence has, I think, two major facets. The first one is, they’re taking in threat intelligence in the form of indicators of compromise, in most cases, whether those are IP addresses or email addresses or host names, et cetera. They’re enriching the data that they’re collecting so that, essentially, they can make a decision to say, “I have an IP address in a log file somewhere, or I have an authentication event someplace and is this bad or does this belong to or has this been related to some threat activity?” That, I would say, is a, let’s call that in an enrichment related activity for threat intelligence so that they can, they can raise the confidence of a particular event or a particular alert.
Another mechanism, the more mature organizations, what they’re doing is they’re creating threat intelligence. When there is an incident or when there is an alert or event, they go through their security operation processes and the investigative processes and ultimately yield some set of, really, intelligence at that point because then they have context in terms of there was a certain attack, it was targeting a certain system, and what was the system? Who was the source? What kind of exploit was used or what sort of a compromise was attempted? They are creating threat intelligence for themselves. Those are the two big classes that I see.
Now, over time what’s happening is, as the industry matures, so now this is a little bit of my view, a color with what I’m seeing customers do is, they are maturing to the point where they’re saying, “Okay, we get this notion of IOCs, if you will, indicators of compromises, but we want more context and better, really, intelligence. Not just indicators but intelligence in almost like, a little bit of the military intelligence style where you say, well, intelligence, the idea behind intelligence is to gain an unfair advantage against your adversary.” Now, I think a lot of organizations are expanding their definition of threat intelligence, what used to be essentially threat feeds or IOCs into threat intelligence to try to understand what are the kinds of actors that are coming after them.
Also, then trying to understand what sort of vulnerabilities exist within their systems, not just from a configuration management vulnerability point of view, but more broadly, strategically what vulnerabilities exist and try to understand it. I think those organizations are not expanding beyond IOCs and going into understanding what are people tweeting about them or what kind of people are interacting with their websites or what sort of locations. Also, in terms of, people talk about dark web stuff, but even more practically, what are people saying about their organizations or what are the things that might happen as a result of let’s say a merger or acquisition or as a result of a lawsuit or something like that. It’s expanding for the more mature organizations.
Do you find that there are any common misperceptions that people have when they’re trying to spin up or make better use of threat intelligence within their organization? Are there areas that they need to be filled in on that you all … You help provide clarification with?
A lot of organizations, I think they don’t recognize that there is a journey to this, and sometimes I myself struggled with this term journey. But, the idea is that not everyone is ready for threat intelligence in any form at day one as they’re starting their security operations. I mean, there is a maturity level in the sense that, both from a technological point of view and from a process point of view as well, in the sense that if we go back to, historically and you say well, let’s say somebody just showed up at your desk and gave you a threat intel report and it’s very comprehensive. It has actors and locations and intent and IOCs as well and all the different things in the TTPs, that the techniques and tools and procedures or tactics that an adversary might use. Let’s say you have all that.
Now the question is, what would you do with it? I think that’s where the maturity comes into play, is for an organization to understand the usefulness of it and for organizations to be ready to receive threat intelligence so that they can action the threat intelligence. I think that is an important maturity point that maybe not all organizations realize, but I think as the industry matures, and by industry here I don’t mean the technology or vendor community, but the actual industry itself of organizations and governments and so on using and maturing their cybersecurity operations. I think people are getting to understand that a little bit more and more.
I want to switch gears a little bit and talk about SIEMs. First of all, where are we today? Where do we find ourselves with the current state of SIEMs, how they’re being used, how people are implementing them? What’s your take there?
I think it’s useful when we talk about SIEMs to just take a step back and think about where SIEMs came from. I think as the digital explosion started happening, even in the ’90s or maybe in the early 2000s, people started deploying firewalls and antiviruses and they had this diversity of different technologies, let’s just call them sensors, that they were detecting things or they were needed to detect things. SIEMs came about. The early SIEMs were essentially collectors of alerts, and they almost hijacked the term correlation because they weren’t really correlating anything, but the idea of a correlation was more in the sense that I have an AV that sends me information, I have lots of different AVs, so I’m going to collect and aggregate that information. That’s kind of where SIEMs came about. They were reporting engines. They were sort of these things where you can get started from.
Fast forward to today, and you know, they have evolved a lot. The big thing that happened in that middle phase, being that reporting phase, and now where SIEMs are today is this idea that it’s not just about collecting a bunch of detections, which is what the early SIEMs did, but it’s about being able to investigate it. From that investigation notion then this idea came about, well you don’t necessarily just need to investigate. You need to respond to these things as well. Response has these notions of part of automation but also these heavy, heavy requirements for integration. Now, we fast forward to today where the SIEMs that … What the customer is asking for from SIEMs to do is the ability to collect data from anywhere, whether it’s in the cloud, whether it’s on prem, whether it’s multi-cloud, and collecting any kind of information. That’s a first capability any SIEM has to have now. It’s not just being able to collect alerts from somewhere.
The second ability it has to have is this analytical capability, which is not just the ability to search, but the analytical capability to apply machine learning models to be able to enrich and contextualize information. Then, the other element at the top layer is to act and operate on whatever the information is so that that information can be used to respond. SIEM is becoming more of … The requirement for SIEM now is to be more of a system than to be this one thing that it can just collect information from one place. I mean, it really is becoming … The expectation now for SIEM is to be the security operations center’s nerve center, and that’s where the action happens because analysts actually operate and use a SIEM. They don’t just look at it, which was the historical or … This is what they did with the first SIEMs.
Do you think that people have realistic expectations of what’s coming from their SIEMs?
Well, I think, this is where I would maybe be a little controversial and I would say that the security industry at large has done a little bit of a disservice to our community. The marketing tends to be a little bit farther ahead of what the technology capabilities are in general. As a consequence, what happens is, customers on the one hand have expectations because they have their own imagination or they have their own pain points and that’s why they have expectations. But, the other side of it is, they have expectations because the industry, regardless of whether it’s the same or any other kind of technologies, the industry is telling them or the vendors are telling them that they can deliver certain capabilities. In some cases, I would say maybe the vendors are not necessarily “lying,” but it’s when they use a term, the customer has an expectation from that term, which is not the same as what maybe the vendor is saying.
I think there is a little bit of a mismatch, and I think the customer expectations are not necessarily, in some cases the only … I would say unrealistic expectations, are the ones where they skip these, what I would describe as a maturity level of the customer itself. For example, you can’t really have a good security operations program if you don’t have a basic underlying vulnerability management program in your organization. You can’t say, “I’m going to build a threat intel team.” Then you say, “Well what’s your vulnerability management program look like?” Say, “Well we have one person in 100,000 employee company that’s responsible for vulnerability management.” Well, you’re missing the development gap here.
Well, I mean, looking forward towards the future, how do you see SIEMs evolving? What do you think we’re going to see looking towards the horizon?
The biggest thing that would happen for SIEMs and that is going to happen, is that SIEM is going to, I mean, this is like a SIEM survival almost. Otherwise, the organizations, the SIEM producers who don’t do the following will not survive the next five years. The one biggest capability, the underlying piece is having a platform that is not just focused on “security events,” but it’s focused on event data in general. Because, when you think about the way the world is changing from a digital point of view, whether it’s IoT, whether it’s cloud computing, whether it’s apps and services, containerization technologies, when you see the world in this highly connected way, it’s very difficult to argue what is not a security significant event or security useful event. That’s number one. That platform capability has to exist.
I think the other piece is, it has to be resident. That platform capability has to be very open because connectivity is the key to connecting all these different environments. By open, I don’t just mean that it’s open to connecting from one kind of technology. By open, I mean the platforms themselves have to be open so they can connect to each other. This is where I like this concept of a nerve center, is that it is a bi-directional connectivity. I think they’re going to have to have that capability.
Then, this notion that there is a lot of hype around machine learning and AI, people use those terms. SIEMs are going to have to make those things consumable. You cannot expect that organizations are going to have teams and teams of data scientists in security operations to be able to manipulate the tools. The SIEMs are going to have to work the way that people work, rather than having the people work the system. I have some friends who say, “Well I’ve been doing machine learning my whole life. Every time I get a new machine I have to learn it.” I think it has to be the other way around a little bit, that these systems have to be built in a manner and machine learning has to be used in practical ways.
Then, the last piece is that this whole notion of operational expediency and highly integrated automation, and automation not just in the perspective of configuration management which is what most people think about when they think about automation. Automation in things like, how do they accelerate the human decision making process? How do they create more context? How do they pull information on the fly? How do they enable people to collaborate with each other or have recommendation systems to say, “Oh, you know, the last time Sally who was an expert,” in a security operation center for example, “the last time Sally saw this alert, she did A, B, C, D steps, and, John, it looks like the thing that you’re working on is 87% like what Sally worked on. Do you want me to just do what Sally did?” Give you those results? I think it’s those kinds of things that SIEMs are going to have to evolve to, because the complexity and the challenges that people dealing with with are very extreme.
Do you have the sense that things are headed in that direction? Are those problems people are hammering away at?
I think there are organizations that in the industry, there are companies that are taking this very seriously. There are some that are still talking about the legacy thing and they’re stuck in some hype to try to compensate for technological deficiencies. I think there are others who are casting a broader vision and saying, “Look, we need to get someplace.” Then, there will always be gaps and I think this is where the startup community is going to come in and fill the gaps. Then, the bigger players will either get disrupted or they will acquire some of these startups to accelerate their own roadmaps. I mean, look, I am a paranoid security practitioner, but at the same time I’m a security practitioner, so I am generally an optimist. I think there is hope and I think things will continue to grow and evolve for the better.
At the end of the day, it’s about the people. Whether those people are your customers or whether those people are the employees in an organization, and from a security point of view, the security operator. It’s ultimately about that security operator. I think the more the technology industry and the security industry focuses on that operator and says, “What does this operator care about and how does this operator’s life get better?” I think that’s … The more we will create better solutions. I think the converse of that for the operators is to be more vocal about what it is that they need, and also be more in tune with what the business or mission requires for them. I think we can do better. The more we anchor in on the human aspect of this, the more that technology translation will happen in a meaningful way.
Our thanks to Monzy Mezra from Splunk for joining us.
Don’t forget to sign up for the Recorded Future Cyber Daily email, where every day you’ll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.
We hope you’ve enjoyed the show and that you’ll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast team includes Coordinating Producer Zane Pokorny, Executive Producer Greg Barrette. The show is produced by the CyberWire, with Editor John Petrik, Executive Producer Peter Kilpe, and I’m Dave Bittner.
Thanks for listening.