In this installment of Tales from the Incident Response Cliff Face, we’ll take a look at a recent engagement, which involved a string of events that spanned several months.
The focus: ransomware with assisted initial access.
Summary
An initial attacker broke into the victim’s environment, established a foothold throughout, and then advertised it for sale on the dark net.
The victim, however, was an NGO that cared for the most vulnerable. For some reason – perhaps the weak financial incentive, perhaps a sense of morals – the access rights to the victim’s environment sat on the marketplace for a full eight months before they were eventually purchased by a second attacker.
In this report, we will elaborate on how the unprepared victim was set up for this ransomware event and, as usual, share some insights to help you avoid being caught in the same trap.
In a nutshell, we will cover:
- How threat actors obtained initial foothold by exploiting vulnerabilities on public-facing servers.
- A well-known technique that allows attackers to steal credentials practically on a domain scale and goes under the radar of leading AV/EDR solutions.
- The actions that we were able to observe of the NoEscape ransomware operators.
Proxywhat?
Our engagement started nearly two weeks after the ransomware detonation. While the client, fortunately, managed to leverage spare backups to perform restoration and avoid business disruption, it did mean that their intervention contaminated the crime scene and that full visibility into the cyber kill chain would be impossible.
Initial triage of forensic collections revealed that the on-premise Exchange server played a central role in the threat actor’s operation by being the source of the RDP internal lateral movement.
To make matters worse, the lack of patching since 2021 coupled with exposure to the internet, would have almost certainly meant that it had already been hit by the Proxysomethingwave.
This hunch was quickly confirmed to be correct. While the usual logs to hunt for this family of Exchange CVEs have only been available since mid-August 2023 and did not show any traces of exploitation, we did manage to find two webshells on disk. I could not determine the previously installed version of Exchange and therefore can’t say with confidence which of these were exploited:
- ProxyShell (CVE-2021-34473, CVE-2021-34523 and CVE-2021-31207)
- ProxyLogon (CVE-2021-26855, CVE-2021-27065)
- ProxyNotShell (CVE-2022-41040 and CVE-2022-41082).
However, working on the assumption that this was a known CVE, it was safe to assume that it must have been one of these.
The two webshells were lightly obfuscated to avoid static disk detection but remained simple (as shown by Figures 1-4, below). They would execute whatever was in the cadataKey request parameter and then either force a 404 or redirect to another page.
My assumption is that this was a way to make it harder for Blue Team operators to identify webshell interactions and reduce the risk of being discovered by their usual technique of using the filter on aspxcoupled with a 200-return code.
The timestamps on these two files indicate that they were created back in December 2022. Furthermore, as shown in Figures 5, I discovered that a file had been created a couple of seconds before the first webshell.
This file masked as the ApacheBench command line utility but proved quickly to be a Meterpreter stager. The Meterpreter stager is built on a template executable, which allows the malicious shellcode to be injected into the .text section.
This stager then communicates to 103.112.232.44:443 to get the rest of the Meterpreter payload. In our investigation, however, it became evident that it was not the only payload to have reached out (see Figure 6). Behind that particular IP, was a compromised Exchange server that was used to compromise other vulnerable Exchange servers.
This is a classic case of launching an attack from another victim – so the hackers can hide their identity.
The Eye of Sauron
The forensic analysis revealed that NoEscape managed to perform lateral movement via RDP with the domain admin account. After confirming that the latter was not performed in restricted admin mode, which would require the NT Hash only, the usual question was on the table once again: How did the attacker obtain the clear text password?
A fair guess would be that the attacker got hold of it by exploiting the previously mentioned CVEs that give local system privileges on the Exchange server. If, as was the case here, a decent AV/EDR solution was missing, dumping credentials from the LSASS process would become child’s play and provide easy access to the domain admins credentials, which would likely be cached on the Exchange server.
But a fair guess, in this case, is not a correct guess!
In this case, the LSASS did not contain cleartext credentials since WDigest was not enabled. Furthermore, the NTLM hash would not have been easily cracked either since the customer confirmed he was using complex 18-character passwords. Finally, the password did not appear to be stored in common credential stores nor in simple text or script files.
Suddenly, answering how the attacker achieved this privilege escalation became really interesting. To find answers, we had to go back to the textbooks.
On the Matter of Stealing Cleartext Credentials
What we know about cleartext passwords is that their lifetime in modern systems is short. A cleartext password only exists when it is being acquired, but as soon the credential needs to be validated (be it locally or remotely), the password is hashed and the cleartext form is basically lost for good. This means that it’s only possible to obtain the cleartext form in a small number of other ways. The most basic involves listening to the interactive logon process. This is what keyloggers do (as their name suggests) by collecting keystrokes. More refined techniques specifically target the Windows interactive logon process. A famously documented technique involves Windows Authentication Providers, which comprise two groups: Security Support Providers and Network Providers. We will focus on the latter in our tale.
If you are curious to know more about Security support providers and the different techniques for stealing cleartext credentials, read more here:
https://www.scip.ch/en/?labs.20220217
The technique to steal cleartext credentials by abusing Network Providers has a 2020 implementation under the name NPPSPY
https://github.com/gtworek/PSBits/tree/master/PasswordStealing/NPPSpy (I also found descriptions of the technique dating back to the early 2000s)
https://www.giac.org/paper/gcih/117/microsoft-network-provider-exploit/101145
TLDR: A neat visual explains how the technique works. The dll needs to be “recorded” in the registry for it to be called, by declaring a network provider.
Back to the Case (Study) at Hnd: Applying the Theory
The documented steps of the technique would mean that the attacker would have had to make some changes in the registry, so this gave us a lead to test the NPPSPY hypothesis. When we queried the registry for traces of the exploit, we found a new network provider had been introduced (see Figure 7).
As shown in Figure 7, we found from our investigation that the ntoskrnl.dll had been maliciously placed in the system32 folder (Windows does not use ntoskrnl.dll, only ntoskrnl.exe) and a dummy network provider named credman had been created, which pointed to the dll. Interesting to note, the last WriteTime of the registry was on February 28th, 2023.
Given that this technique requires admin privileges that allow a user to make changes in the registry (something the exploitation of the Exchange vulnerabilities can grant) we made the link between the two exploitation events.
In fact, malicious .NET compilation artefacts around the same time as the registry modification led us to believe that the exchange vulnerability may have been exploited multiple times.
The next step was to analyze the malicious dll (ntoskrnl.dll) to determine where it had stored the looted files. Existing threat intelligence on this dll gave us this information (Figure 8).
Most importantly, we can see that it generated the following file “C:\ProgramData\Package Cache\Windows10.0-KB5009543-x64.msu”. A quick inspection of the file’s content in the compromised server revealed the stolen passwords in cleartext:
A lot of what we do on the investigation front is driven by our intuition. Our hunches. We explore these and see if they deliver fruit. So, what was supposed to be a simple confirmation of the NPPSPY technique with some admin users’ passwords sparked my curiosity: There were a lot of accounts in the file, far more than you’d normally expect to see in a usual exchange server (which should be solely admin users). It seemed that every regular domain user had their credentials stored here. The idea that they had all interactively logged onto it would be totally implausible.
Or so we thought…
What was the probable explanation?
Going from ‘How was it possible for the attacker to find the cleartext password?’ we moved onto the question ‘Why were so many regular domain user credentials on the Exchange server?’
We suspected that the scope of the technique covered in the documentation could – in certain conditions – be widely extended. The intuitive explanation was that somehow, it must have captured mailbox-related authentications. Looking up NPPSPY in conjunction with Exchange Server I found only one blog post that gave me the information I needed:
This post suggests that using NPPSPY on an Exchange server would swoop all mail-related authentications. Interesting as this may be, the blog post didn’t cover why, how, or under what conditions.
This left me on the hunt for further validation, so I proceeded to reproduce it in my lab against a fresh default installation of Exchange Server 2019.
Reproducing the technique in my lab
My approach was to interactively login using different ways (OWA, ECP, Powershell, etc.) and see which login attempts would interact with the loot file, thus suggesting credential theft.
For example, when logging in via OWA, I get the following hit (see Figure 10).
Figure 11 shows that the initiating process was the Microsoft Exchange Outlook WebApp (OWA). Based on what I found, I checked the stack trace (Figure 12).
The stack trace confirmed that OWA authentication went through the Malicious Network Provider and called the NPLogonNotify API for loot.dll.
Interestingly, the stack trace showed also that authbas.dll was involved. As shown in figure 13 below, this proves that this dll was responsible for the IIS BasicAuthenticationModule
As soon as I realized this, the fog cleared and it all became crystal clear.
The authentication process had proceeded by invoking LogonUserExW (see Figure 46.). This literally behaved as if authentication was being made to a local computer, just as we would expect to see from the documentation of NPPSPY with Interactive/Remote Interactive Logons.
After this step,Mpr.dll was then called. By way of reminder, Mpr.dll stands for the Multiple Router Provider that handles communication with the installed network providers. Mpr.dll checks the registry to determine which providers are registered and then goes through them, one by one.
So just as we thought, our malicious network provider was called and our dll was triggered via the exported NPLogonNotify. The credentials were then passed down to it and ended up in the text file as indicated by the call to WriteFile (Figure 11, above). Note that the functionality of NPLogonNotify is arbitrarily set by the attacker. In this case, the attacker deemed it sufficient just to write credentials to a text file, but they could have done something more complex, e.g., send the credentials over the network, which would have removed any trace on the disk.
Thus, we were able to confirm that the threat actor did abuse IIS Basic authentication with the NPPSPY technique to place himself into every Exchange web authentication.
The corresponding event log for this Exchange Web Authentication corroborates our observations about IIS Basic Authentication as the logon attempt is Type 8: network clear text (Figure 15).
In Focus: detection and response – how to deal with this attack
The reason I focused on this attack was to explore the new elements it comprised and offer guidance on detection and prevention.
Let’s talk about detection avenues
Unfortunately, the dll used in this test is based on the NPPSPY repository but was recompiled to avoid AV/EDR detection with literally no sophistication at all. It did evade endpoint security solutions successfully which says a lot about the power of this technique, even though the threat model is expensive as it requires admin privileges. Note that I am working with the assumption that admin privileges should never give an attacker the ability to roam freely if a proper AV/EDR is in place.
So, if AV/EDRs seem to be blind to it for now, I suggest focusing on the registry changes that enabled this attack. Any attacker will have to announce his network provider along with the DLL. This means that such changes need to be closely monitored. We appreciate that monitoring requires some resource commitment, as shown by the number of registry changes that are made in our customer base, but this is where something like long-tail analysis can weed out false positives. Sysmon and EDRs have telemetry for commands, process execution, and registry changes and can be leveraged for hunting and detections. Windows Event Logs can be leveraged as well but require explicit auditing.
For more information read the following article:
And now, onto the matter of prevention
We want to prevent credential collection of users that log into the Exchange Server (or any Windows host), including those of users authenticating via OWA/ECP.
Credential theft from interactive logons
For users of Windows 11, Microsoft has released the 2H22 Security Baseline which includes a new setting “Enable MPR notifications for the system”. As described above, Microsoft recommends setting it to ‘disabled’ to prevent password disclosure to providers (exceptions apply).
This is a sniper fix – useful if you don’t use providers.
If you don’t use Windows 11, one way is to use the recommended Protected Users Group in Active Directory. I tested this against the technique and can confirm that it inhibits the MPR notifications (the mpnotify.exe process is not spawned after login). Unfortunately, I cannot explain the internals of it and can only deduce from what Microsoft says about “Protected Users Group” that cleartext credentials die too early (Figure 16).
There’s no such thing as a silver bullet though. So, keep in mind that there are side effects to adding users to this group.
This recommendation applies to LSASS credential dumping as well since they are no longer cached.
Credential theft from ISS Basic Authentication
Regarding the credentials stolen from IIS Basic Authentication by using Outlook on the Web. It is enabled by default and is necessary for OWA/ECP to work (Figure 17).
Disclaimer: I am no Exchange administrator
Obviously, I am not an Exchange administrator. But like many incident responders, I can consume Microsoft documentation. So, II tried disabling Basic Authentication and enabling the other types of authentication but this broke functionality. I did not spend time digging into alternative ways to determine if I could make it work or not.
However, the Microsoft recommendation for on-prem non-hybrid environments is to move from Basic Authentication to Modern Authentication with ADFS for Outlook on the Web (OWA/ECP). This would also allow the usage of stronger authentication features like enabling MFA.
If you do this, be sure not to use Basic Auth in ADFS as it would only shift the problem to another server.
Ransomware with a Red Carpet
Eight months after the first Exchange compromise, we observed interactions with the webshells that were followed by Anydesk installation in unattended access mode so that user approval would not be required (Figure 18).
From the Exchange server, the threat actor then used Nmap/Zenmap to scan the network and powershell, and then to enumerate the shares in the domain using Invoker-ShareFinderThreaded from PowerView.
Note that this is not the first threat actor to use such a technique; it has been widely used not only for host discovery but also to discover the location of company data.
The threat actor then checked who was logged into a server before RDPing into it with the domain admin account (remember, he successfully stole the clear text password via the malicious NPPSPY dll and wrote it on disk). Forensic analysis on the hosts where lateral movement was performed shows that the threat actor identified data and inspected some of it manually by opening some files. The restored environment would not show us how data exfiltration was performed exactly but the firewall logs did show significant outgoing traffic towards IPs that belonged to the mega.co domain.
Given the large delay (8 months) between the first evidence of NPPSPY usage and NoEscape active engagement with the target, it is unlikely that the ransomware actors are behind the exploitation attempts cited so far. We believe NoEscape bought access in the form of cleartext passwords and webshell locations from Initial Access Brokers.
If you are curious about the economics, read https://www.kelacyber.com/the-secret-life-of-an-initial-access-broker/
5-Recommendations
In this Incident Response Cliff Face tale we investigated a ransomware operator that managed to act on a victim’s environment quickly and efficiently, but whose privileged access was facilitated by a much earlier compromise.
Here are some simple things you can do to prevent this chain of events happening in your organization:
- Patch your environment, especially against critical vulnerabilities
- Reduce attack surface by reducing exposed services and hiding them behind VPNs
- Deploy state-of-the-art AV/EDR solutions (in this case they would have helped with cve exploitation and encryption)
- Invest in security event logging and monitoring
- Monitor remote access tool usage
Click here to download the full case study, including the 5 recommendations.
Kudelski Security Research 