Active exploitation of Citrix ADC and Gateway Critical Remote Code Execution Vulnerability by Suspected Chinese APT5 (CVE-2022-27518)

Written by Harish Segar of the Kudelski Security Threat Detection & Research Team

Summary

On December 13, 2022, The U.S. National Security Agency (NSA) release an advisory warning of in-the-wild exploitation of Citrix products by APT5, a threat actor attributed to China (also known as UNC2630 and MANGANESE). This APT has been actively exploiting a zero-day flaw in Citrix Application Delivery Controller (ADC) and Gateway to take over affected systems. This Chinese state-backed threat actor that has been known to target telecommunications and technology companies and has also previously exploited vulnerabilities in Pulse Secure VPNs.

According to Citrix, this critical remote code execution vulnerability, identified as CVE-2022-27518, could allow an unauthenticated attacker to execute commands remotely on vulnerable devices and seize control.

APT5 has demonstrated capabilities against Citrix Application Delivery Controller deployments. Targeting Citrix ADCs can facilitate illegitimate access to targeted organizations by bypassing normal authentication controls.

Due to the critical nature of this vulnerability, it’s active exploitation in attack campaigns by Nation State Adversaries, and the fact that this vulnerability exists on appliances designed to be directly exposed to the internet, Kudelski Security strongly recommends that organizations who leverage Citrix ADC and Gateway products and have configured them to use SAML Service Providers or Identity Providers apply available patches immediately.

Affected Systems and/or Applications

The vulnerability described in this advisory has been present in all Citrix ADC and Gateway versions between 12.1 (including FIPS and NDcPP) and 13.0 (but before 13.0-58.32).

In order to be impacted by this vulnerability, the device must be configured with a SAML Service Provider (SP) or Identity Provider (IdP).

The following supported versions of Citrix ADC and Citrix Gateway are affected by this vulnerability:

Citrix ADC and Citrix Gateway 13.0 before 13.0-58.32
Citrix ADC and Citrix Gateway 12.1 before 12.1-65.25
Citrix ADC 12.1-FIPS before 12.1-55.291
Citrix ADC 12.1-NDcPP before 12.1-55.291

All Citrix Gateway and Citrix ADC products with a version earlier than 12.1 are End of Life (EoL) and have NOT received patches for this vulnerability. Citrix has recommended that organizations running versions of that are EOL update to newer versions in order to mitigate this vulnerability.

Customers can determine if their Citrix ADC or Citrix Gateway is configured as a SAML SP or a SAML IdP by inspecting the ns.conf file for the following commands:

add authentication samlAction # This means Appliance is configured as a SAML SP

add authentication samlIdPProfile # This means Appliance is configured as a SAML IdP

If either of the commands are present in the ns.conf file and if the Citrix software version is an affected version listed in this advisory, then the appliance must be updated to mitigate the issue.

Organizations leveraging Tenable can leverage the following Tenable Nessus plugin to find see if their appliance software is known to be vulnerable to this critical vulnerability: https://www.tenable.com/plugins/nessus/73204

Solution

All customers using the affected builds should either update to the current 12.1 build (including FIPS and NDcPP variants) or to the current 13.0 build (13.0-88.16). Organizations using an affected build with a SAML SP or IdP configuration are urged to install the current build immediately. As an alternative, organizations may choose to upgrade to the 13.1 version, which is not affected.

Organizations who are running affected builds can set up audit logging to monitor for unauthorized activity on ADC or Gateway devices leveraging the following documentation:

https://docs.citrix.com/en-us/citrix-adc/current-release/system/audit-logging/configuring-audit-logging.html

However, it’s important to note that simply upgrading to the latest version of Citrix ADC or Gateway may not mitigate an exploited appliance as NSA’s advisory states that APT5 has been modifying legitimate binaries after exploitation in order to maintain persistence. Organizations should investigate any appliances that were previously running vulnerable versions of Citrix software and that were configured for SAML authentication for signs of compromise by performing the steps outlined in the “Detection Guidance” section of this advisory

Temporary workarounds and mitigations

Citrix has stated that there are no known workarounds available beyond temporarily disabling SAML authentication on all impacted Citrix appliances.

Detection & Behavioral Check Guidance

Note: This guidance was provided by NSA and has NOT been modified by Kudelski Security

The key executables are the binaries essential for proper running of the Citrix ADC appliance. These files include, but are not limited to: nsaaad, nsconf, nsreadfile, and nsconmsg.

You should compare the hash of those binaries with the known good hashes from the vendor or hashes of the respective binaries from a known good copy downloaded from the vendor. Any deviation requires further investigation.

The following command can be executed from a shell on the Citrix device to facilitate this comparison:

cd /netscaler ; for i in “nsppe nsaaad nsconf nsreadfile nsconmsg”; do md5 ${i} ; done

Additionally, the following command can indicate tampering by one APT5 technique. This is indicated by one line of output, but no output otherwise:

procstat –v $(pgrep –o –i nsppe) | grep “0x10400000 “ | grep “rwx”

In addition to any alterations of legitimate binaries, some of APT5’s activities may be
visible in various system logs. NSA recommends that organizations leverage off-device
logging mechanisms for all system logs, to include dmesg and ns.log, and actively monitor them for the following activity:

Instances of pb_policy appearing in logs without being linked to expected
administrator activity.
- The actors have been seen leveraging tools that run ‘pb_policy’ twice.
  This creates the following logs in ns.log:

<local0.info> [hostname] pb_policy: Changing pitboss policy from X to Y <local0.info> [hostname] pb_policy: Changing pitboss policy from Y to X

Where X and Y are constant values for your system.

Gaps in logs, or mismatches between logs on the device and in your remote
logging solution.
Legitimate user account activity without a corresponding record of a valid SAML token being issued by the identity provider for the environment.
Unauthorized modification of user permissions.
Unauthorized modifications to the crontab file and/or existence of suspicious
file(s) in /var/cron/tabs/ and other locations.
- Files related to this activity have been discovered in /tmp for some, but not all, impacted organizations.
- The command below can assist in finding files that have been associated with this activity. While these files have not been discovered in all environments, their presence may be indicative of actor activity if discovered:
```
find / -type f -name “res*” | grep -E ‘res($|\.[a-z]{3})$’
```

Recovery

In the event that you have results from the above detection methodology and commands, Kudelski Security strongly recommends that you engage your incident response team or IR retainer provider immediately.

Additionally, NSA has provided the following guidance:

Move all Citrix ADC instances behind a VPN or other capability that requires valid
user authentication (ideally multi-factor) prior to being able to access the ADC.
Isolate the Citrix ADC appliances from the environment to ensure any malicious
activity is contained.
Restore the Citrix ADC to a known good state

What the Cyber Fusion Center is doing

The CFC is working with our vulnerability scanning vendor partner to deploy plugins to detect this vulnerability. Once available, organizations with the CFC’s Vulnerability Scanning service will be able to validate if this vulnerability is discovered on systems within vulnerability scan scopes.

The CFC will begin investigating potential avenues for threat hunting – however due to limited visibility provided by Citrix ADC and Gateway appliances, we may only be able to leverage logs from appliances already configured to send log data to CFC monitored SIEM systems. We strongly recommend all clients work with the CFC to ensure their SIEMs are receiving Citrix ADC and Gateway logs.