Finding the silver lining in getting your teeth kicked in

Lots of pen test and red team blogs follow the same model: we came, we saw, we conquered, blue team tears flowed. This is not one of those blogs.

TL/DR; Pen testing isn’t about finding vulnerabilities. It’s about finding opportunities for your client to improve, even when they did seemingly everything right.

On a lot of assumed breach tests, it’s the same old story. Fire up PowerUp, get an elevated beacon, fire up BloodHound, and quickly tunnel towards domain admin and all the data. Every once in a long while you’ll encounter what Casey Smith calls “apex defenders.” Apex defenders are teams of security analysts, incident responders, and system admins that have hardened their infrastructure and instrumented incredible visibility into their networks and endpoints. These apex defenders can, and will, kick your teeth in if you slip up.

Recently, I had the opportunity to come up against one of these apex defender teams. While we were able to get custom malware running on endpoints, once we had our beacons we quickly realized that we had an uphill battle. On first inspection, we found that workstation endpoints were running with multiple AV and EDR packages which both prevented most code execution attempts but also increased the likelihood of being detected with every command. We also had been warned that PowerShell was heavily logged, monitored, and alerted.

Our initial malware didn’t work. We originally thought this was a problem with our redirectors, so we rebuilt our C2 infrastructure to use domain fronting. A short time later, we had a beacon. Turns out, the initial malware used userinit.exe to spawn our beacon and EDR stomped on it. It also alerted the IR team, who isolated the box and burned our delivery and C2 domains.

No problem. We spun up new C2 and delivery infrastructure and our point of contact launched several new beacons for us. Our initial optimism at our malware evading all detection quickly dissolved into despair as we watched beacon after beacon die. We built a custom PowerLine executable and uploaded it. EDR quarantined the binary and a few minutes later the IR team isolated the host.

We determined that they didn’t have visibility into running .NET assemblies via execute-assembly, so we used that method to perform reconnaissance. Unfortunately, when running SharpHound, EDR detected BloodHound.bin being dropped to disk. Lesson learned: always use the –NoSaveCache option.

We tried Kerberoasting and did manage to crack a couple of hashes. Unfortunately, none of those accounts seemed to be able to log in anywhere. We didn’t manage to crack any additional hashes during the engagement. Long and strong passwords FTW.

We went old school and looked for passwords stored in various AD attributes like description, comment, and info schema. No joy there.

This particular client has dedicated pen test and red teams. They work continuously with their blue teams to improve detection capabilities, identify root causes and implement fixes across the organization. Over the course of the engagement, the IR team detected us through network behavior analysis, EDR detecting lateral movement techniques, EDR detecting files dropped to disk, and alerting based on PowerShell logging. For the most part, detection and eradication was swift. So, after all this, getting stomped on by the blue team repeatedly, where’s the silver lining?

After a weekend of dejection and licking my wounds, I stared at the report and thought about the engagement. What could we report other than the blue team beat us up? What recommendations could we offer? We analyzed the IR team actions and found a few nuggets that could help an already impressive IR team improve.

While they found and blocked our original delivery and C2 domains, curiously, the IR teams did not isolate the new infrastructure and we continued to use it throughout the engagement. This presents an opportunity to review procedures, and improve training.

We also noted that the time from an action that caused detection to isolation was considerably longer after hours. We discussed this with the client and learned that they have US and EMEA SOC teams. The EMEA SOC took a lot longer to isolate hosts after receiving an alert. Yet another opportunity for training and improvement.

Our job, as pen testers and red teamers isn’t to find vulnerabilities. Our job is to help the customer improve. Helping a customer improve processes and identifying opportunities to improve training is just as important and finding holes that need patching. The next time you come up short on vulnerabilities, give some thought to the other opportunities for improvement.