Request a Demo

Secure your Whales

For our day to day product deployment, we use docker containers. Whenever a new piece of code is being shipped to production, our CI/CD process creates several docker images and pushes them to our private registry – standard deployment process.

In the spirit of “shifting left”, we wanted to scan our Docker images and the 3rd party dockers we use in our architecture as a part of our pipeline. We want to prevent merging to master if they contain any critical vulnerabilities. We decided to try the open source clair-scanner which is based on Clair 'analyze-local-images' by CoreOS for the mission.

In our environment we use Gitlab as our code repository and CI.

Gitlab Ultimate provides an integration with clair-scanner results. When merging branches to master it analyzes the vulnerabilities report and displays the results in the merge request.

Gitlab also recommends running the scanner inside a docker executer that attaches to the hosts docker socket.

Please refer to the following articles for more info about integrating container scanning in Gitlab CI:

CI Container scanning

Merge request container scanning

Both of the above presented an obstacle for us, since we use Gitlab shell executers and don’t own Ultimate license.

In addition, we wanted to be more proactive about the vulnerabilities found.

Here is how we implemented a solution that would not require upgrading our license or changing executer from shell to docker which would create an issue for each vulnerable image with the list of vulnerabilities for our team to analyze.

What needs to be done:

  • Run arminc/clair-db and arminc/clair-local-scan:v2.0.1 containers linked one to another
  • Download the latest script for Clair scanner
  • Run the script to scan the desired images
  • Report an issue in Gitlab in case of vulnerabilities found

 To avoid the overhead of starting clair bundle (scanner and DB) and waiting for it to “warm up” during each CI execution we decided to run the bundle continuously on the executers.

 The issues we needed to tackle:

  1. When using docker run command to start Clair scanner we’ve noticed that it created numerous ‘git-remote-http’ processes without killing them. This ended us up with a bunch of zombie processes on the CI executer, until the container was restarted.
  2. Clair DB image is frequently updated and needs to be pulled and restarted. But there is no reason to recreate it if it wasn’t updated.

 docker

Docker-compose to the rescue!

To overcome both of the above we’ve decided to run the scanner and DB using docker-compose, instead of utilizing maintenance scripts or adding logic to the docker run commands.

To fix the first issue we’ve added ‘init: true’, which forwards signals and reaps processes.

Please refer to this issue to find more info about zombie processes in CoreOS Clair.

For the second issue we run docker-compose pull and then docker-compose up, which will only recreate the containers in case they are newer than the current ones running or will just pull latest versions and run them, in case they are not yet running.

The docker-compose file is quite simple, with the addition of init to clair scanner:

version: '2.2'
services:
  postgres:
    container_name: clair-db
    image: arminc/clair-db:latest
    restart: on-failure

  clair:
    container_name: clair-scanner
    image: arminc/clair-local-scan:v2.0.1
    init: true
    restart: on-failure
    depends_on:
      - postgres
    ports:
      - "6060:6060"

 

TAKE ME TO YOUR SCANNER!

Gitlab CI uses a .yml file to define the stages. Our first step was to add the “container-scanning” as a new stage in the process.

stages:
  - build

  - test
  - container_scanning
  - push_docker_images
  - deploy
  - cleanup

 

Then we created the stage itself:

container_scanning:
  stage: container_scanning
  only:
    - master
  script:
    - cp -r app/devops/clair_scanner/ /opt/
    - docker-compose -f /opt/clair_scanner/docker-compose.yml pull
    - docker-compose -f /opt/clair_scanner/docker-compose.yml up -d
    - curl -s -L -o clair-scanner https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
    - chmod +x clair-scanner
    - for image in $(grep 'image:' docker-compose.yml | awk '{print $2}' | sort -u); do ./scan_docker.sh $image; done

 

Let’s break it down a bit:

What this does is limit this stage to run only on master –

  1. Copies the folder that holds essential files for clair docker-compose (docker-compose.yml and any env files you may want to use with it) to /opt/
  2. Pulls the newest versions of images specified in docker-compose.yml
  3. Executes docker-compose up to boot/update the docker containers for Clair scanner and DB
  4. Downloads the scanner script
  5. Gives the script executable permissions
  6. Searches for images in docker-compose.yml, which defines our applications’ architecture (this is not the same as the clair docker-compose.yml) and passes each image as an argument to our custom script (scan-docker.sh). This script calls clair-scanner executable from the previous stage, parses the output and automatically opens/updates Gitlab tickets based on the results.

The whole process is quite simple and quick overall.

 

The real magic is in our scan_docker.sh script, (I’ll replace most of the code with pseudo code for easier reading but keep the Clair parts intact).

Feel free to contact us for the full code.

The first thing we do is run the scan and save the output. We also keep the JSON file to be later attached to the Gitlab ticket.

IMAGE=$1

vuln_file_name=$(basename $IMAGE | sed 's/:/_/g')

echo "Scanning" $IMAGE
ticket_title_unform=$(./clair-scanner -c http://localhost:6060 --ip $(hostname -i) -r $vuln_file_name.json --threshold="Critical"-l clair.log $IMAGE)

 

We then take the output, parse and strip from formatting (like shell colors) and any unneeded text.

According to the result, we decide what we want to do:

if [ "any vulnerabilities found" ]; then
        if [ "similar ticket already exists" ]; then
            echo "Ticket already exists. Checking if it needs to be updated"
            if [ "similar ticket is not up to date" ]; then
                echo "Updating existing ticket with newer data";
                echo "Uploading json attachment"
                echo "Adding comment with pipeline details to the existing ticket"

            else
                echo "Ticket is already up to date"
                exit 1
            fi
        else
            echo "Will create a new ticket"
            echo "Uploading json attachment"
            echo "Creating GitLab ticket with pipeline details and security tag"
            echo "New ticket has been submitted. Please refer to GitLab"
            exit 1
        fi
else
        echo "No vulnerabilities were found for $IMAGE";
fi

 

 To submit or update an issue we use Gitlab API

Voila! A totally free vulnerability scanner integrated in our pipeline, with reporting capabilities!

 

Don’t miss out on the latest

Get notified on Industry updates.
we promise not to spam

Related Posts

Popular Articles

11.1.2018 | vulnerabilities

| Posted by Roy Horev
The best way to share information about the risks associated with vulnerabilities is via quantifying these risks – i.e. metrics. The question is, ...
Read more
  With nearly 15,000 new vulnerabilities discovered in 2017, and even more expected this year – the competition for ‘worst vulnerability’ is a tough ...
Read more

05.7.2019 | vulnerabilities

| Posted by Tal Morgenstern
There’s a buzz in the vulnerability market surrounding solutions to protect against Zero Day vulnerabilities - vulnerabilities that were previously ...
Read more