How to Improve Your Next Pen Test Result With a Purple Team Engagement
Speakers: Michael Oglesby, Chris Clements, and Josh Bozarth
[0:10] Hi and welcome to today’s webcast. I’m Lindsey Watts, Vice President of marketing at Cerberus Sentinel, and I’m here with Josh Bozarth, Chris Clements, and Michael Oglesby, our expert penetration testing team.
I should say a sampling of our expert team [as] we have a pretty deep bench. We are going to discuss purple team penetration testing and dig into a little bit about what it is what it isn’t … how it helps … anything you need to know… questions you might have about that. So, gents if you’d like to introduce yourselves and tell us a little bit about your pen testing careers
[0:49] My name is Josh Bozarth. I’m the Director of Security Testing Services here at Cerberus Sentinel. I’ve been doing penetration testing for easily 10 plus years. It gets to the point where you kind of keep track, [but] I lose track but … been involved in lots of different types of exercises whether it’s mobile app network you know red team, purple team, rainbow team whatever teams … [H]ave a team of … six plus folks [who] regularly rotate in and out of projects to get those fresh eyes on things
[1:24] I’m Chris Clement, Vice President of Solutions Architecture for Cerberus. I’ve been in cybersecurity for over 20 years now. I’ve worked both on the defensive side and on the offensive side. I’ve done penetration testing [for] about eight years in my career and had a lot of fun doing it.
[1:43] I’m Michael Oglesby, the Executive Vice President of Security Services here at Cerberus. I’ve been pen testing pretty much my entire career; you know 20 plus years. If it’s got an IP address, I’ve probably attacked it at some point in my career; these days, in addition to … leading penetration testing, I’m also an adjunct professor teaching the next generation of pen testers up and coming. And excited to talk about some new and exciting areas in pen testing today.
[2:18] I like to think of it using the term … demystifying what’s going on. There’s a lot of … quote unquote magic going on when penetration testers … hack into your systems. It can be unclear on what the tools and techniques and processes they’re using to attack your systems and you might learn about it at the end of the engagement or they might try to bring it you know bring it up to you as they’re finding vulnerabilities along the way as they’re attacking your system.
A purple team engagement’s … goal is to really get you the defender, you the IT person, involved in that process from the get-go, from the very beginning. You’re involved in how the attackers are attacking your systems. You’re watching them firsthand. You’re experiencing the tactics techniques along with the tester which makes it a much more interactive, much more valuable service …
Today’s … defenders really break [the] kind of the traditional mold of a tackle versus defender; it’s really a team-based approach; it’s really thinking about we are all really on the same team together both attackers and defenders against … the real threat actors that are out there. So, let’s start working as a team, and we kind of coined that term trying to combine the red versus the blue into the purple team color. Just building on that a little bit – really the collaborative approach is what the big difference is in my mind in a purple team.
[3:47] The outcome of a penetration test can be … a lot of times be adversarial in nature, where the purple team is much more collaborative an approach and as a result of that one of the key outcomes of a purple team that often doesn’t happen during a penetration test is the penetration test will give you results that are vulnerabilities attack vectors – potential attack pathways – and it’s well and good to remediate those…
[4:36] In a traditional pen test the attacker is going after vulnerabilities, and you’re going to learn about those vulnerabilities after the fact. You can go fix those …. and go patch it, but after the test is over you can’t really test the process anymore because the test is over.
So, the purple team really gets your people involved while the test is going on so that in real time you can start improving and testing your people skills, your processes, your playbooks; ensuring those are working, making sort of real-time updates to those while the test is going on. It’s really hard to do that after the fact. You have to think about what I would have done if I was doing that a week ago. When the tester was doing their things in a purple team engagement, you just can sit down with the attacker right then [and] watch what they’re doing; learn from them and make those process improvements while the engagement’s going on. So much more real time, much more immediacy than you can get from a traditional pen test.
[5:38] A blue team to me is anyone that’s going to be responsible for the defense of an organization and in some organizations that can be a dedicated team that’s responsible for information security and in other organizations that can be just the IT staff that is responsible for implementing and monitoring those controls.
[6:04] Have a red team they’re goal-oriented right … and as Michael alluded to that those real-time communications with the blue team and the red team as they’re doing you know the attack and defense in a red team engagement it is really a lot of one-sided attack. It’s designed to find a goal and get to a specific situation, but it also will have longer term type aspects of testing than a normal, maybe as what we would call, a standard penetration test. It’s a very time boxed time … constrained that maybe it’s for like a one week or a two-week type of situation, where that again we go back to the point in time aspect of penetration testing.
A red team engagement can be longer and designed to be…to do an advanced threat persistent type situations where there might be something with more social engineering or maybe some physical penetration testing involved.
[7:02] But in the end, it is very one-sided. It’s attack and then you come back after and look at it from a defense standpoint: what could be done differently. As opposed to a purple team, where it’s you know everyone’s talking at the same time…whereas iI guess if you’re doing a point in time approach if you can get in that’s great; you make note of that if you can’t get in that’s great. You make note of it, but if the processes after that point were not strong enough or wouldn’t protect an organization … to the best of what they could be protecting themselves to best practices. If you’re testing beyond that, then the point in time, you know, that could be limited in a one-time engagement where you’re okay “that’s great” – we must be solid because hey nobody had passwords that have been leaked. That’s great, good for us but then that one time…
[7:59] When it happens, you’re going to get owned I suppose. So, I can definitely see the benefit there and thinking about this from … a businessperson’s terms and how they would approach this and think about it from a non-technical standpoint, it’s just nice to have that reassurance
[8:24] You know as people are navigating what kind of tests they need. To get that, you get on the phone with clients or … in a call … in a meeting and they feel like I absolutely have to have a pen test but the reality is a vulnerability assessment might be better for them.
[8:42] In this case … how does someone know which one to approach the team with like I think maybe a purple team engagement would be good for me that kind of pen test but maybe standard? How do I know as a client, or as a businessperson, [or] as a non-technical person? … How do I decide? … It’s easy to fall into the trap of what’s the new hotness right, you know. It’s like, oh, purple team we ought to be doing that.
Well, when I’m on the calls with clients … they’re .. you brought the great scenario up where they wanted a pen test … I’m like you’re trying to run before you’ve even learned to walk here folks. We need you to step back, and let’s look at things from an assessment standpoint; let’s identify that low-hanging fruit and kind of make it a challenge for our testers to do their work to raise …. up those security levels for you; know the security maturity that we want. It really is just an escalation of …
[9:46] Doing a purple team can be more beneficial than a penetration test. If you’ve got … what you’re looking for … you … need to have those folks that are somewhat dedicated to defense right? You don’t want to be a company of five people and say I want to do I want a purple to do a purple team pen test when it’s just 5 and … maybe you’re just using cloud services for your email and whatnot … you know those types of things fall more in line.
[10:17] If you’re new to security doing vulnerability assessments … maybe doing a pen test … but I’m going to lean towards a vulnerability assessment … first…. As the organization grows, matures, creates these opportunities for avenues of attack that can be demonstrated in contests or purple team engagements that’s when the business has to understand what their maturity level is.
[10:43] It’s like we don’t know what we don’t know; and that’s when they have to kind of rely on us to kind of steer them in that right path and sometimes it makes sense [and] sometimes it doesn’t. Sometimes folks want a pen test and we give them a pen test but I think … you need to make sure you have the blue capability to some degree, whether it’s an outsourced MSSP or if it’s your own in-house – you know guy who that’s all he does is monitoring. You need to have that capability to participate because otherwise it’s just red; there’s no purple creating at that point.
[11:25] I outsource my operational IT or at least the part of the environment
that probably needs to be addressed or tested. How does that work if you know if I’ve outsourced some part of my environment or what if I’m outsourcing some of my security management? I’ve got somebody doing SIEM for me or you know how does that work in a purple team engagement because those aren’t my teams necessarily …They’re an extension, right, but how does that work? Yeah that’s probably … one of the best examples of when a purple team can be real valuable if you can get all the different players …. who might be [with] multiple companies together operating under the same umbrella … the same engagement to ensure those lines of communication are open.
So, making sure your partners, your providers, and your internal staff … even at a fundamental level … do they even know if an attack is going on? Who they would call over there? Do they have the right points of contact at each entity? Do they know if they needed to make a change in the cloud environment? …. How would they do that? Could they do that? They need to call the provider …. to do …. to try to game plan those out ahead of time you know before an actual ransomware event is ongoing….
[12:35] You know where a purple team can really shine is that it’s let you practice before you actually go out in the real world and do it.
[12:41] Yeah, you can test … your vendor’s effectiveness. You know I think it’s one thing we’ve talked about in the past, if you’re not sure that’s a good way but the idea also is that it’s not an us versus them mentality. It is something that we’re everyone’s working together on and we still struggle with that…. we’re coming in to do a purple team engagement; the client will still act you know standoffish and like you know we got to brace ourselves for this. Like, yeah, you do need to brace yourselves, but we’re kind of telling you what we’re doing as we’re going along we’re trying to communicate as best as we can. We don’t want to like just tell them everything we’re doing but we want them to take into account what could happen …Then the idea is if you don’t catch it that’s okay; we’re going to go back through those motions with you. We’re going to talk about those attack vectors we’re doing those threats. We’ve … know the tactics, the techniques, the procedures: you know their …
[13:34] We’re going to do those and then we’re going to say did you see this … we’ve finished one of these up a few weeks ago and we streamlined communications. We created a portal for them to document what they’re finding and that can correspond … that we’re doing we had uh you know communication channels streamlining and like for example in slack you know where we had chats going where we had three different groups, we had the red team folks we had the clients you know defenses and then we had the third party MSSP um who happened to be our SOC….
[14:10] At Cerberus … we kept those all three areas distinct, but they are all together in streamline communication so that we could …. update us on what they’re finding and maybe the problems they’re running up against ….
[14:27] Shutting things down and are taking machines off the network and the SOCs haven’t scrambled to bring things back online. We’re hearing all that chatter; seeing all that’s going on, and we’re like okay, yeah, that was us. We’re going to pause for a moment while you … get things settled down and then we’ll come back and readdress it and see where … the tools were working but it’s more than tools right. But the tools will do what they can do. We need the people to be able to respond as they need to as well and not just off the cuff. we’re getting it
[14:55] We’re getting it … you have to kind of strike that balance. We were finding where the client was …. just dumping it into the chat; we’re like great [but] what are you doing about it? We have to kind of keep zeroing them back into on what’s the purpose of this. This isn’t a test; we’re not going to give you an A or an F. You know based on the results. it’s really just … let’s make this better….The C level folks – they understood that concept they love that concept every time we get on the call… They’re like, yeah, we want to make things better. What are you guys doing to make things better? ….
[15:30] They’re sensitive … they don’t want to look bad quote unquote and we have we work really hard to say no. It’s not that kind of engagement; it’s an engagement where we’re bringing value to the table with our expertise and coordinating all of this and then it’s your value that we’re going to make you better in the long run. It’s not a grade; you’re not gonna’ get a black eye for this.
[15:58] It’s about when the situation happens in real time – in real life – you know – later you know what to do, and you can find what’s going on and you’re not in the dark. So, it takes a little bit of effort but in the end, it ends up being a valuable experience for clients
[16:18] You know another thing: I think it’s wonderful for is introducing and validating non-obvious attack scenarios. So, i think most people that work in security or even IT in general would say, hey you know, if someone’s trying to guess 100 passwords against an account, that’s something i need to be able to guard against and so maybe I’ll implement say an account lockout or put some sort of alert mechanism. If I see that … but an attacker may think very differently.
[16:50] An attacker might say, well, instead of trying 100 different passwords against one account which I know is probably going to get me locked out. Instead I’m going … to try one password across 100 different accounts and so that is a that’s a very valid attack scenario. … I think for a lot of folks [it’s] non-obvious but that’s the type of thing that a purple team will be able to walk you through.
[17:13] And say this is why we think this is a risk; let’s talk about how to defend against this and that makes sense. I think that [this] does get back to what Michael said earlier. Kind of demystifies the process. I’m sure and I have heard that it can be, but I’m sure it’s intimidating to have somebody come and test your environment.
[17:32] If it were me, you know, any kind of type a person is just going to be nervous thinking like you know are you what are you going to find … and have I really done my job even though … you’ve done due diligence, you’re still going to be worried about it; but I think that collaborative nature … makes a lot of sense… We’re all on the same team; we’re here to just fix things and make them better knowing that everybody’s doing their job right. It’s just a matter of improving things or going the extra mile figuring out what an attacker is going to do that I didn’t think of.
[18:12] Reality is there’s still a lot of people who are getting penetration tests because there is a requirement. There’s a compliance mandate that they have to do this. So where does purple team fit with that? If I get a purple team pen test will that meet compliance? Is it the same thing? Is the report different? What does that look like?
It can definitely meet your pen testing compliance requirements right; the reports, you know, can definitely look similar so you definitely want to work with your provider [to] know [you’re] make sure the outcome is going to meet … whatever specific kind of needs you have … and [that] your pen testing is definitely not going away it’s still a needed service at the end of the day.
[A] real red team attack from malicious entities on the internet so you gotta’ be ready for that and make sure your environment is secure from that as well …. See them as sort of replacing each other. They’re really very complementary to each other right there you know you’re practicing your different skills, but you know both are still needed right yeah and I mean it’s important to note that a purple team penetration test is still a penetration test. At least at least ours is, so I mean if you have that compliance requirement it is going to meet that.
[19:32] I think it fits most companies. I think Josh … said it earlier, you do need to have your fundamentals down right, you do need to have some sort of security defenses, some tool sets in place to be able to exercise, to be able to participate right. You can’t go out on a baseball field without a bat or a helmet or a glove and expect to do anything, so you do need to get some of those basic security controls. You know some password management, some endpoint detection products out there to at least be able to get the most value out of your purple team engagement right off the bat.
But I think most clients would see a lot of real value because it’s a huge educational opportunity for your IT security team as well you know even just going through the motions of a test is just tremendous value on both sides. You know the pen testers get to learn more about your environment can help you know give you suggestions. Your IT team will probably learn more about their own environment than they knew you know even beforehand.
You know a lot of times they’re learning stuff that’s on their network they hadn’t even thought of like … very often we hear about … there’s a tool over here that could have detected them and people are like oh we have that tool on the network; it still runs … oh yeah there’s data over there that could have helped.
Okay so there’s a lot of collaboration … you know even on the blue team talking to each other learning about their own capabilities and different departments that they might not even be aware of and … my viewpoint on that is that once you have those base security controls in place, I don’t necessarily see a blue team operation as being something that only the most advanced and most highly secure organizations can benefit from. I think really at the point that you’re ready for even a basic penetration test, you would receive a lot more value honestly out of purple team penetration tests.
[21:22]…I’m gonna’ do this … what teams what on my side … you need to show up to this thing or be ready … anyone who’s involved in you know responding to security alerts if you’re a big organization that might include some dedicated teams like the firewall team as an example. If you know the endpoint, help desk team people who are watching workstations that are part of the scope, definitely the server team, and definitely the it security team some other groups you may think about including maybe some of your application teams if they manage their own servers … If you were doing like a brute force attack against a particular application, they might be seeing that attack inside their application log somewhere and those may or may not be going to your security fabric so identifying gaps like that is one of the huge benefits of the purple theme so maybe including some of those business application owners in the test as well…
[22:34] [T]hat’s a great point because you may run into scenarios where you are trying techniques against a particular application that your organization’s built and as part of that process the blue team identifies we don’t have the capability of recognizing this right now so that would be on the application owners to then go and uh make it a priority for them to build out okay and like you said Lindsey if you have any outsourced providers and you know outsourced MSSPs who are, you know part, of your program definitely include those people as well. … If you’re able to, there are definitely … places we think we can predict where that your organization will need to include but I think one of the important parts of the process is that the that’s a discovery that will happen in real time during the purple team engagement so as we’re saying okay this is the methodology that we’re using. You need to be able to do this. that’s where you…
[23:28] That’s where … the light bulb often goes off where they say oh that means I need to talk to this group or that’s going to be this group and so that is that that’s a process that happens very organically during a purple team engagement and that’s one of the big benefits because you don’t have to game out every potential defensive scenario beforehand. You’re doing that in real time, and I presume that you all would probably coach me through that if I’m looking. You would kind of say hey you might want to have these people.
It’s not going to be just a blind eye. I think I want a purple team penetration test and … I assume there’s going to be some kind of prep ahead of that, and I guess that kind of leads me to my next question which is if I’m wanting to do this and I’m sure … there are a lot of different avenues to go down looking for a purple team pen test and different providers. How do I, how would I, select … who should help me with this? Who’s going to be a good provider? What do i look for? Or do I just go to their offering? Do I evaluate the offering? How do I find the best provider for me?
[24:45] You want to look for a provider who says they’re doing a purple team first off the bat. So don’t assume that if your provider just says we do, for example, network pen testing that that’s going to be a purple team. Those are two very different styles of engagement, so you really want to look for a provider who has done some purple teams. … Knows how to talk … understands it’s a collaborative process and not just a red versus blue type engagement … so definitely look at who you’re going to be working with.
This is a collaborative approach … what type of tools what type of people are you going to be interacting with … Making sure that you’re working with individuals who have some time and experience. Really look for a provider who is including, you the client, in on the scoping and the preparation and the testing. If it looks like their methodology and their approach is all on them, they’re going to do everything, and they may just let you know what’s going on. That’s not a purple team. You should be highly engaged during this engagement; that means they should be asking you a lot of questions about who can you bring to the table who are you going to involve, how many people on your side, can you guys use slack, you guys use email? … Your provider should be asking you a lot of questions about how they can work with you best because that’s how they’re going to structure the engagement. If they don’t ask you any questions, if they just look like they’re going to do it all, and you’re barely going to be involved [those] would be some red flags that you might want to look for. A provider who is more collaborative with you. Because that’s the main purpose of this test …
[26:24] I think another big thing to look for … is that they’re basing it on some sort of comprehensive framework that they’re not just trying the one attack of the day and measuring against that. Josh, I know you’ve got a lot of experience on that … don’t know what you might be able to add. … We like to uh look at MITRE’s attack framework. … It’s well documented, and it evolves over time. We use that as the framework when we’re going through purple teaming engagements, right. … It’s designed for us to … we’ll go through those ttps in those areas … like lateral movement or initial access … or privilege escalation. These are all known attack vectors that exist, and we want to make sure we’re exercising those for detection capabilities …
[27:13] Like I said we use a common framework … but we also use a common tool to allow everyone to log in and see what’s going on. … It notes when we did the attack; it notes when we stopped the attack; it notes … what we did and then there’s a section for the defensive side to say yeah we detected this or no we did not or maybe we kind of did. … Or they put in some notes; everyone’s documenting and collaborating together. ….
[27:46] … Into showing here’s what we did, here’s what was detected, yay nay… Then everyone has marching orders on what to do better; to go forward; and that usually means having various meetings to discuss what could be done to make it better. You know, whether it’s a SOC. You know we’ve met … with our own SOC when we go through purple team exercise, and they’ve identified gaps that we need to address. We stand up labs; we redo a lot of those types of tests with them to make sure that you know whatever changes they’ve made it’s now detectable by what we’re doing …. Not a huge fan of throwing colors around, you know. It’s like what color is your lightsaber guys; yeah as long as it’s not red, you’re usually pretty good in the end right but even then … I mean I’m a huge Star Wars fan … there’s lots of scenarios where red’s not going to make the end of the world. You just kind of have to know who are you dealing with, and that’s kind of what you want to do when you’re dealing with … you’re selecting a provider on doing a pen test. …They’re going to use buzzwords or whatnot … are you dealing with people that actually care and want you to do better with your security. And I think that’s the differentiator. I think anyone can agree with. Obviously, we would say hey that’s us, but in the end, we want you to make educated decisions on what you’re doing. … When it comes to security testing, make sure you’re involved … I mean that’s key; like what Michael said. that’s key point, number one; if you’re not involved, it’s a red flag because it’s a red team operation at that point instead of the purple team.
[29:37] My last question is this, and this is from a marketing person’s standpoint. Right I go out, and I like to research things that are out on the market and I do a little bit of reading about purple team, and I hear about orange teams, yellow teams, green teams, and it gets a little confusing for me. Are these terms I need to be familiar with [or] is this something I should be concerned with?
[30:04] I think as far as just the different terminology, that folks might run across the different color teams. I think the big differentiator in a lot of those when it delves down to the minutia: they’re breaking out a separate team from the red and the blue team. In the yellow team or what they would call their build team and to me … that can make a lot of sense if you’re working with an organization where those teams. So you’ve got a hard and fast blue team; you’ve got a hard and fast yellow team; they really don’t interact with each other a lot then yeah that might make sense.
[30:43] But I think for most organizations, you want just that straight collaborative effort so that the organization can learn where they need to develop those capabilities and how to implement them to protect themselves. I mean the
the numbers are just staggering – the amount of organizations that ….
is just astronomically low. It almost always happens when they’re contacted by an outsourced organization like a security researcher or law enforcement. Says hey you guys really got a problem or when the attacker makes their presence unmistakable when they launch the ransomware. Right, it’s hard to mistake that you’ve been attacked once the ransomware launches, but in general you know my thoughts, is … if you if you’re not able to catch your penetration tester at work your likelihood of catching a real attacker [is] just vanishingly small. And the purple team is just the way you’re doing that right. It’s not saying hey we’re addressing a particular vector; it’s not saying that we’re addressing a particular vulnerability; it’s really saying we want to be able to catch the hacker at work no matter what they’re doing. Well, that makes sense. All of it and I think you’ve answered questions.
Definitely our team is here to answer any additional questions anybody has. Please reach out to us we’re happy to help you. If you’ve done a purple team penetration test, and you just want to talk about it and ask questions about it we’d be more than happy to do that.
[32:03] Just reach out to us. You’ll have contact information available, and we’ve got a blog on our website at www.cerberussentinel.com. You’re welcome to review…as an additional resource. And thank you everybody for being here. Thank you gents for helping walk us through this. Appreciate it.