Leadership and Buy-In Help Protect BT’s Global Networks

December 17, 2018 • Amanda McKeon

BT is a global telecommunications giant, headquartered in London with over 100,000 employees all over the world. In addition to telephone services in Great Britain, BT provides broadband internet, fiber-optic communications, digital television, and even supply chain management services. They also provide IT and network security services.

Our guest today is Mark Hughes, who served as the CEO of BT Security from 2013 to 2018. He oversaw the security of BT’s internal networks and assets, as well as the services they provide to outside clients. He shares with us his techniques for building effective, collaborative security teams, how he earns buy-in from both his colleagues and his board of directors, and the importance of threat intelligence. He’ll also share his experience gearing up for the 2012 London Olympic Games, and why he thinks it was a milestone moment for cybersecurity.

This podcast was produced in partnership with the CyberWire.

For those of you who’d prefer to read, here’s the transcript:

This is Recorded Future, inside threat intelligence for cybersecurity.

Dave Bittner:

Hello everyone, and welcome to episode 87 of the Recorded Future podcast. I’m Dave Bittner from the CyberWire.

BT is the global telecommunications giant, headquartered in London with over 100,000 employees all over the world. In addition to telephone services in Great Britain, BT provides broadband internet, fiber optic communications, digital television, and even supply chain management services. They also provide IT and network security services.

Our guest today is Mark Hughes, CEO of BT Security. He oversees the security of BT’s internal networks and assets, as well as the security services they provide to outside clients. He shares with us his techniques for building effective collaborative security teams, how he earns buy-in from both his colleagues and his board of directors, and the importance of threat intelligence. He’ll also share his experience gearing up for the 2012 London Olympic Games, and why he thinks that was a milestone moment for cybersecurity. Stay with us.

Mark Hughes:

I started off in security some years ago, and I basically was responsible for the security of BT itself. So I am still the CISO and CSO, if you want, brought together. I decided to reshape the team so that we brought both the information security teams and also the physical security teams as well. BT is one of the largest network communication service providers in the world. We’re one of the largest global MPLS networks. So, we’re very distributed. We’re operating in 180 countries. So, a lot of people in a lot of places, a lot of buildings and sites, because telecom’s infrastructure needs locations for those nodes to be where they need to be positioned.

I brought the security of both of those things together. So clearly, there’s a big need to ensure that we protect not just the physical locations, but all the information that’s running around them. I did that for a number of years. We have changed the approach that we’ve taken on that, for our business, as I said, it operates globally. We have 130,000 staff across the globe. So it’s a big deal. And the deal is also quite big because the U.K., traditionally, has been a very important pairing point for internet services across the globe.

So, I brought this team together. As the threat environment has been changing, as we know, dramatically over the last few years, there was a need to really re-engineer how we did security in our organization, right away through the actual security activity itself, but also into the various parts of the IT stacks.

Then, thereafter, a few years ago, we thought hard about how we were approaching the market in security. I decided that I would take a really fresh look about how we were providing security services and re-engineer that part of our business, take that under my wing, and then really drive and grow the business as well.

I have a role which, basically, leads me to both be protecting the organization as the CSO, but also equally driving our market opportunity and the growth in the market as well.

Dave Bittner:

Can you give us some insight into what your day to day is like? With an organization as large as BT and the scope of the challenges that you’re up against, as you just described, how do you set your priorities?

Mark Hughes:

Priority one is always going to be that I have to maintain the security of our business and make sure that we have a very proportionate level of control based upon the type of risk that we are carrying as a business, on behalf of our customers as well, to ensure that we are really delivering to our customers’ requirements. That is my absolute number one priority, and it will always be my number one priority. What I mean by that is, importantly, we have to really unpick that. As an organization, I think long and hard about how to, as a communication service provider, to ensure that I understand what our customer requirements are, and therefore, what role we play in ensuring that those requirements are met. They may be from consumers using, for example, wireless-type services and cell-type services, all the way through to some of the largest global Fortune 500 enterprises who take services from us, and making sure that all of our data that we carry on behalf of our customers, the data that we carry on behalf of our business, to make sure our business work is protected. And most importantly, in the telecommunications context, that it works.

As you can imagine, when thinking about how we design a security strategy for an organization of our size, availability is very important when you think about the confidentiality and integrity and availability triumvirate, availability sits out there very importantly as a thing that is dear to our hearts. Not to say that confidentiality isn’t important and integrity, but availability, when it comes to telecommunications, is really, really important. So that’s always my first priority.

My second priority then is, how do I take that and work closely with our customers to ensure that where they want specific services relating to security — they may well consume network services from us — but if they want specific security-related services, how can I take some of that expertise that we have from running the sort of security operations that we run for our business, and then share that with our customers and provide them services that potentially allows them to manage their risk in a proportionate way as well, on behalf of the business that they’re conducting with their customers.

So, priority one is BT itself. Priority two is making sure that our customers take advantage of that, and we provide the services that we really need. They are very much symbiotic. If we can’t do the first one well, our customers aren’t going to be getting the best service that we can possibly offer them. Obviously, as a business, we need to ensure that we are really generating and driving not just new business, but the business we have already, and maintaining that. A lot of that now is really based upon having the right security in place to do that.

Dave Bittner:

And how would you describe your management style? How do you work with your team?

Mark Hughes:

I think, in security, this is something that we don’t necessarily pay enough attention to. I would say, and I’m going to broaden it out a bit, as a person who’s been in security now for a number of years, I think the way in which organizations work is ever more important, in that we have to be able to react quickly. When you think about our layered defense approach that we put in place, increasingly with the types of attacks that we’re seeing, where we know that we can be infiltrated and that there can be persistency in those types of attacks when identifying and working out what’s going on, then we have to be able to respond ever more quickly. That, to me, relies a lot upon having a style where one is collaborative, where you understand where the right touchpoints are in the organization, and really using that ability through collaboration to actually then get things done, oftentimes very, very quickly. So that’s one point around this style, generally, that you need.

That then extends into my management style with my team, that we have to be able to … That person who happens to be in that spot at that time, who’s identified that particular piece of activity, needs to be able to feel confident that they can operate in the right time frame without having to be worried about referring up and layers of hierarchy. It’s really about creating that management environment where people feel that they can act. Not just feel that they can act, but they will actually act as well, independently, faced with what they’ve got. That they have the ability and the autonomy to do that, that they have the guidelines within which they have to operate, but once they know that, they know that they can act and that they will be backed up. And that’s the management style that I really try to foster in my organization. Because without that, what we will end up with is the situation where people feel that they have to refer up all the time, things slow down, and then the bad guys, essentially, will exploit that and then be able to take advantage of us in a way in which is potentially quite serious for an organization like ours.

The other key thing I’d say about my management style is, I’ve come to realize in security that when you look at the types of professionals that we have in our org, that I have in my team — and I have over almost 3,000 people in my team — is that often, the people who know the most about something at any one time are not the people who are, in any way, shape, or form, the most senior or high up in the management chain. It’s that deep expertise that you need in certain given situations and having a management framework where people can be listened to at every level in the organization and feel that their … Not just their opinion counts, but also when they say that this is what needs to be done, they get backed up. That’s really, really important. So it’s a great question because getting the right environment through the right management style is something that I think has actually helped us be really quite successful, and hopefully continue to be successful in defending against a lot of these types of attacks that we see.

Dave Bittner:

What’s your interaction with the folks above you in the organization, with the board of directors? How do you get buy-in from them for your activities?

Mark Hughes:

Well, all the time, I have very regular interactions with the chief executive chairman, other members of the board. Sometimes on a more formal basis, a regular appearance at the Board Order and Risk Committee and other board committees as well. But also, on quite an infrequent basis. The CEO and myself will swap notes on most days. He will ask me questions about certain things that might be coming up, or I will obviously go to him and say, “Look, you need to be aware that this is happening.” And quite often, that will be a commentary on stuff that’s happening in the news, more broadly in the security space.

The need for formal reporting is also very significant as well. I have a risk-based approach to the security here at BT. I’m continually reevaluating that based upon the new risks that are emerging. For that, having the right threat intelligence in place to be able to achieve that is very, very important, and actually sharing some of that threat intel, even up to and including the board of directors, so that I can contextualize why it is that we’re taking the steps to do what we’re doing in terms of the control framework that we’re running within the organization.

Lots of frequent updates, but also set piece updates as well, which revolve around, “This is what potential consequences and impacts are to the organization, and then this is what we’re doing about it, and these are the things that really bother me the most,” with some evidence to back that up. And the way I do that is, I ensure that I’m completely aligned with the other risk areas in the organization as well. I assist on what we call the “group risk panel,” so that we have a look and feel about the reporting in the security space, which is very similar to the look and feel for reporting for all of our other risks. I find that really helps with the conversation because it means that it doesn’t need to be seen as something special, but it also contextualizes it within the type of metrics that they’re used to seeing. It makes a conversation flow much more easily.

Dave Bittner:

I’m curious, with your own board, but also the interactions you have with BT’s customers and dealing with their boards of directors, where do you think we find ourselves today when it comes to boards of directors and their understanding of the challenges that those of us in cybersecurity face?

Mark Hughes:

It’s improving, certainly improving. I think most board members that you would speak to would say that they have a much better awareness and an understanding that this is something that they have to consider very carefully. I think that there is still this thing in cyber, of the fact that this is somehow different. Although I’m saying it doesn’t necessarily have to be different, there are a couple of things about it that are. My experience is that they are different. Number one, it’s normally done to you, you don’t do it to yourselves. A lot of risk in organizations manifests itself because you make strategic decisions that take you down a certain route. You do things with certain customers in the way in which you manage and deliver your services. You get that, maybe, not right, and then things can happen and consequences can emerge from that.

Often, what happens with security is that things coming from outside the organization with that malicious intent are the things that can trip you up. And, of course, that is a characteristic of security — it’s malicious in its motivation, where other risks that manifest themselves are not necessarily maliciously motivated. That’s one key characteristic.

The second one is the velocity. You can go from having a control framework in place that can be very quickly undermined by a new vulnerability and exploit pairing. That, I would say, is different. Therefore … It’s not entirely different. There are things that happen in our organizations that also happen very quickly as well, but making sure that those couple of characteristics are taken into account when presenting it … The reality is that if you can drive a framework for the board of directors, where they understand that framework already through other risk presentations that they might have, that, to me, has been really helpful in unlocking the fact that this risk is characteristically its own risk, but is similar to many other risks. That, therefore, moves away, I think, from this thing of, “Well, cyber is a big unknown. It’s something that’s separate. It’s something I don’t really know enough about, and it’s something I can’t even begin to understand.” Not that at all. Take it into the normal risk methodology that any large organization will have, even any small organization will have, to be able to help make those trade-off judgment risks around, what is the proportionate level of control that the organization needs. That’s my experience as a way in which it helps them. I think it is really improving, and I think people are getting that.

The one thing beyond anything that really affects the design is, you can get to the dollar sign in terms of what the impact and likelihood is, combining those two things to actually arrive at a risk exposure dollar sign. That does help drive the conversation to something where there is a great deal of understanding already.

Dave Bittner:

Let’s talk about threat intelligence and the part that you feel it plays in your own organization, the importance of it there.

Mark Hughes:

Threat intelligence, to me, is those key important things — how do you observe what is going on over the increasingly complex environments that we operate in? When I say increasingly complex, the traditional way of managing a corporate IT environment of a large data center environment with branch offices, corporate head offices, and redundancy in between different data centers is really changing. Why? Because people are consuming services from the car, be they hyper scaled, software as a service, or even platform as a service as well, and, of course, that is changing the environment dramatically.

Add to that the increasing prevalence now of SSL traffic and TLS 1.3, stick those things together, and the visibility that we’ve traditionally had in the security space across the network is diminishing rapidly. Obviously, combined with the fact that a lot of that traffic doesn’t exist within the corporate envelope any longer in any case.

To me, therefore, helping to be informed by threat intelligence to get visibility across those different environments now is absolutely fundamental. Understanding what’s out there, what’s going on, being able to take reliable sources and correlate that into the types of tools that we’re running, the seam-type tools that allows us then to correlate that into the events that we’re seeing on our own traffic is fundamental. So, threat intelligence is really driving our ability to have our antennae up, our eyes and ears open to contextualize what’s actually going on. It’s really fundamental to be successful and having an end-to-end monitoring view over those different environments.

Dave Bittner:

How much information sharing goes on among your organization and your competitors? You’re still competitors, but obviously, there’s benefits for you all to keep in touch when it comes to threats.

Mark Hughes:

Absolutely. Yesterday, I was actually at the World Economic Forum, and we had a big discussion around threat intelligence. We all … Well, there are lots of large organizations there, 140 information security professionals or cyber professionals from many different large organizations across the globe, they universally said that sharing of this type of information, threat intelligence, is not something that is going to drive competitive advantage. It’s something that’s going to allow us all collectively to get better. So what we do, specifically, because we can observe across our networks an enormous amount of information, we have an information sharing partnership, the Malware Information Sharing Partnership, which we’ve put in place in the U.K., and now, increasingly internationally as well, called MISP. What that allows us to do is to share a whole range of parameters of information with our peer group in the telecommunication space, and increasingly to our customers who come from all different types of sectors as well. It’s a pretty straightforward process to get signed up to it. We can do real-time sharing using some of the protocols that are now understood, like STIX/TAXII, but we can also do different types of sharing depending upon how organizations want to share that information with us.

We put that in place. We see it is very important that we share that information as well as consuming other people’s information because that does help create that, as I said earlier on, situation awareness.

Dave Bittner:

You have your day-to-day operations there, but also, there are special events that come up. I want to touch on the work that you did with the 2012 Olympic Games because that was certainly a special event for your organization, but obviously, a little local pride there that it was taking place in London. What was the gearing up for that like? What was that experience like for you?

Mark Hughes:

Well, 2012 feels like a long time ago now, certainly in the cyber world. I think that really kick-started the approach that I was talking about earlier on about how we would become much more hunters rather than simply sitting there and watching and then waiting to react. So we really took the opportunity to say, “Fundamentally, we’re going to have to collect a lot of data from around our networks and around, specifically, the networks that were operating in the 2012 environment, and we’re then going to use that data in a way in which we’re going to correlate activity to go looking for bad stuff before it had a bad impact on our ability to execute against the 2012 Olympics.” And, of course, it goes without saying that those environments are so time-critical, and have to be. The reliability requirements are so high that there’s very little room to make errors there, as you can imagine.

Dave Bittner:

And the world is watching.

Mark Hughes:

Absolutely right. And you know, there are a lot of people who target those environments, and so, therefore, we had to be very mindful of the ability to understand all the different types of risks and threats that were out there and how they would actually manifest themselves as risks.

2012 provided us with a complete catalyst to say, “How are we going to get that situation awareness? What threat intelligence do we need to be able to suck in from different sources and then be able to correlate that with the real-time games operating environments, and then, actually then spot stuff before it then turns itself into having an impact adversely on our ability to execute on that mission, in a world where many of the things that we take for granted today were fairly nascent?” WiFi usage and coverage, for example, cell usage, all of those things were relatively small then, but nonetheless could still, as we all know, bring in different types of malicious activity to interrupt our ability to deliver service.

The 2012 games, for us, acted as a huge catalyst to think about how we go about collecting data. We put our first big data solutions in place at that point, where we brought a lot of that data into Hadoop clusters and then used a number of tools across that to then be able to start analyzing that. And that really built the bedrock for how we now deliver security, not just for BT, but for our customers as well. That has been really, really influential. 2012 has been a seminal event in our maturity that we have today, and we’ve learned an enormous amount from that enormous responsibility that we had at that time.

Dave Bittner:

For those people out there who are running smaller organizations who are CISOs at companies around the world, what sort of advice do you have for them? What insights do you have for the scale of organization that you’re in charge of? Do you have any tips for them to keep up with the day-to-day challenges they face?

Mark Hughes:

Yep. I absolutely do. The first one is, use what you’ve got, use what comes out of the box. Most organizations now are either on a journey to or are using … If you’re an Office 365 user, a Microsoft 365 user, there’s a lot of capability that comes in there straight off. Get your identities right, and spend time really thinking about that and how you manage your active directory, be that as your on-prem. So that would be a couple of tips I’d have around that. I know it sounds very familiar, but to me, more and more and more making sure that that works well and getting identity and access management right is really fundamental with some out-of-bound authentication as well. That would be my second tip.

Next is, even if you’re in a small organization, you’ve got big data sets in whatever shape or form, make sure you compartmentalize them, make them less of a risk, because we see time and time again that malicious actors are often after getting into those big data sets, especially if there’s confidential information there or payment-related information there, for example.

Third is, then, also make sure that those privileged users in your organization — even small organizations have privileged users — make sure you keep their access appropriate and proportionate for what they need to do. No one, even the users themselves, those privileged users, will thank you for giving them unlimited access to stuff when that’s not really required. Most malicious actors are exploiting privileged use when they’re going through these types of things.

And the last thing I would say, my fourth tip is to make sure you’ve got a plan in place so that if something does go wrong, ransomware attack or whatever it may be, you sort through how to do it. And what I mean by that is, not just sorting through how to deal with specific scenarios, but you just sort through how you’re going to communicate in an event. Even in an organization of our size, we spend a lot of time just simply thinking about how we work together in those types of moments, and practicing it as well. So the scenarios, often, are less of an issue. It’s more about thinking about how you’re going to work together, and then when the scenario comes along, because it’s never going to be as you will necessarily rehearse. You can do some things to try and predict what it might be because we can see what’s happening to organizations, but make sure that you then rehearse that as well.

And above all else, get your antennae out. And what I mean by that is, my last tip is to make sure you’ve got the situation awareness. Think about the partners who you might think of as competitors, but in this case might have some really rich sources of information. Think about the threat intelligence you’re consuming. Think about the law enforcement agencies and other organizations that you might have to work with if something goes wrong. The time to establish and foster those relationships is before something goes wrong, not when it actually happens.

Dave Bittner:

Our thanks to Mark Hughes from BT for joining us.

If you enjoyed this podcast, we hope you’ll take the time to rate it and leave a review on iTunes. It really does help people find the show.

Don’t forget to sign up for the Recorded Future Cyber Daily email, where every day you’ll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.

We hope you’ve enjoyed the show and that you’ll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast team includes Coordinating Producer Amanda McKeon, Executive Producer Greg Barrette. The show is produced by Pratt Street Media, with Editor John Petrik, Executive Producer Peter Kilpe, and I’m Dave Bittner.

Thanks for listening.