Request a Demo

CI/CD: Streamlining Vulnerability Remediation

Continuous integration and continuous delivery and/or deployment (CI/CD) has become a staple within the modern software development landscape, and it is now extending into patch management. The importance of your environment's security cannot be overstated, nor can the difficulty of maintaining that security. The question then becomes: What exactly is CI/CD and how can you relate this to patching rather than traditional code commits?

Continuous Integration represents the constant patching of test environments and validation of those patches. Continuous Delivery represents the automated deployment of validated patches to production environments, As a whole, CI/CD has shown itself to be incredibly useful even outside of software development, and its methodologies can in fact be applied just as well to include system security and patch management. 

In this article, we’ll take a look at practical examples of patching using CI/CD and how it can be applied to your systems.

Read Me: Top Skills Every Security Professional Needs

How CI/CD Keeps Infrastructure Up to Date

A typical environment has a mix of operating systems, primarily Windows and Linux, which both  tend to follow the same general series of steps. Translating this process into CI/CD steps can potentially look like the following:

Continuous Integration

  • Retrieve a list of patches to apply to your servers.
  • Apply those available patches to testing and staging environments.
  • Perform post-patch testing (i.e., network availability, services running, etc.).

Continuous Delivery/Deployment

  • Patch production systems:
    • Deployment strategies could use blue-green or rolling deployments for further control.
  • Perform post-patch testing, which could include watching for an uptick in support tickets.

Now, let's walk through simple examples for both Windows and Linux to see how you can perform some basic scripting to make this work. For these basic examples, we are going to use PowerShell Core since its runtime works on both Windows and Linux. Additionally, these types of scripts can be integrated into many CI/CD systems, where the results then determine the next steps.

Retrieve List of Patches to Apply

Before you apply any patches, you need to determine what patches are available and what software needs to be updated. The below code examples show how you can retrieve patches from both Windows and Linux systems.

# Windows

PS C:\\> $UpdateObjectSession = New-Object -ComObject 'Microsoft.Update.Session'

$UpdateSearcher = $UpdateObjectSession.CreateUpdateSearcher()

$Updates = $UpdateSearcher.Search($null)


PS C:\\> $Updates.Updates | Select-Object Title, RebootRequired, IsDownloaded


Title              RebootRequired   IsDownloaded

-----               --------------             ------------

KB4529935  True                       True

KB4529938  True                       True

KB4529934   True                       True


# Linux - Ubuntu

PS /home/user> $Updates = & apt list --upgradeable | Select -Skip 1 | Foreach-Object {

        $Result = $_ -Split " "


        [PSCustomObject]@{

              "Name"     = ($Result[0] -Split "/")[0]

              "Update"   = $Result[1]

              "Previous" = ($Result[-1] -Split "]")[0]

        }

}

PS /home/user> $Updates

Name                                        Update                                 Previous

-------                                          ---------                                   ----------

libssl1.1                                     1.1.1-1ubuntu2.1~18.04.5    1.1.1-1ubuntu2.1~18.04.4

python3-distupgrade                 1:18.04.36                            1:18.04.34

sosreport                                   3.6-1ubuntu0.18.04.4           3.6-1ubuntu0.18.04.3

ubuntu-release-upgrader-core  1:18.04.36                            1:18.04.34

 

After the list of patches has been retrieved, you can go ahead and apply those patches, selectively, to the test and staging environments.

Apply Patches to Test/Staging Environment

# Windows

PS C:\\> $updateSession  = New-Object -ComObject 'Microsoft.Update.Session'

$updateSearcher = $updateSession.CreateUpdateSearcher()

If ($updates = ($updateSearcher.Search($null))) {

        $downloader =

$updateSession.CreateUpdateDownloader()

        $downloader.Updates = $updates.updates

        $downloadResult = $downloader.Download()

        If ($downloadResult.ResultCode -ne 2) {

              Exit $downloadResult.ResultCode;

        }

        $installer = New-Object -ComObject

'Microsoft.Update.Installer'

        $installer.Updates = $updates

        $installResult = $installer.Install()

        If ($installResult.RebootRequired) {

              Exit 7

        } Else {

              $Updates.Updates | Select-Object Title, RebootRequired, IsDownloaded

              Write-Host "Install Result: $($installResult.ResultCode)"

        }

}


Title              RebootRequired  IsDownloaded

-----               -------------            ------------

KB4529935  True                    True

KB4529938  True                    True

KB4529934  True                    True


Install Result: 0


# Linux - Ubuntu

PS /home/user> $Updates | Select-Object -First 1 | Foreach-Object {

        & apt install --only-upgrade $_.Name -y

}


The following packages will be upgraded:

        libssl1.1

1 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

Need to get 1300 kB of archives.

After this operation, 0 B of additional disk space will be used.

Get:1 <http://archive.ubuntu.com/ubuntu> bionic-updates/main amd64 libssl1.1 amd64 1.1.1-1ubuntu2.1~18.04.5 [1300 kB]

Fetched 1300 kB in 1s (1286 kB/s)

Preconfiguring packages ...

(Reading database ... 46489 files and directories currently installed.)

Preparing to unpack .../libssl1.1_1.1.1-1ubuntu2.1~18.04.5_amd64.deb ...

Unpacking libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5) over (1.1.1-1ubuntu2.1~18.04.4) ...

Setting up libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5) ...

 

Finally, before moving on to production deployment, you need to make sure that your services are still up and running and that the patches have not impacted any functionality critical to operations. 

Thus, post-test testing is required. 

Perform Post-Test Testing

# Windows

PS C:\\> $services = @(

        "Schedule"

        "wuauserv"

        "EventLog"

)


$results = $services | Foreach-Object {

        $service = Get-Service -Name $_


        [PSCustomObject]@{

              "Name" = $service.name

              "Status" = $service.status

        }

}

PS C:\\> $results

Name         Status

----              ------

Schedule    Running

wuauserv    Running

EventLog    Running

# Linux - Ubuntu

# Verify that the cron, rsyslog, and httpd services are running

PS /home/user> $services = @(

        "cron"

        "rsyslog"

        "httpd"

)


$results = $services | Foreach-Object {

        $result = & service $_ status

        Switch ($result) {

              {$_ -Match "is running"} {

                    $status = "Running"

                    Break

              }

              {$_ -Match "is not running"} {

                    $status = "Stopped"

                    Break

              }

              Default {

                    $status = $False

                    Break

              }

        }

        [PSCustomObject]@{

              "Service" = $_

              "Status"  = $status

        }

}

PS /home/user> $results

Service   Status

-------       ------

cron        Running

rsyslog    Stopped

httpd       Stopped

 

These Linux examples assume that you are using an Ubuntu-based environment with the APT command available to you. Any distribution should work though and would require inputting the appropriate commands to return the information needed.

Tracking Changes the Hard Way & How CI/CD Can Help

It is important to know what patches have been deployed to your test and production environments. If you are tracking these changes manually, it can quickly become difficult and prone to error. If you have multiple administrators managing the environment, these issues can be amplified, demonstrating how important tracking your changes can be.

CI/CD, with its reliance on automation, can handle this work much more efficiently–not only in terms of tracking and recording the patches applied to your systems, but also when and how they have affected the environment overall. Additionally, in the deployment phase, it's common to build in approvals in the event that you want to exercise more control and review over the patching process.

With that in mind, how can we build automated patch reports for our Windows and Linux systems? Again, we will be using PowerShell Core for the scripting, as it integrates well into various systems.

Windows Patch Report

PS C:\\> $updateSession = New-Object -ComObject 'Microsoft.Update.Session'

$updateSearcher = $updateSession.CreateUpdateSearcher()

If ($updates = ($updateSearcher.Search('IsInstalled=1'))) {

        $updates.Updates | Select-Object Title, LastDeploymentChangeTime

}

Title               LastDeploymentChangeTime

-----                ------------------------

KB4529935   11/10/2019 12:00:00 AM

KB4529938   11/12/2019 12:00:00 AM

KB4529934   11/14/2019 12:00:00 AM

 

Linux Patch Report

PS /home/user> & grep " upgrade " /var/log/dpkg.log | Foreach-Object {

        $line = $_ -Split " "

        [PSCustomObject]@{

              "DateTime" = "$($line[0]) $($line[1])"

              "Name" = $line[3]

              "Update" = $line[5]

              "Previous" = $line[4]

        }

} | Select-Object -Last 4


DateTime                    Name                                 Update                                  Previous

-----------                      -----------                              ------------                               ----------------

2019-11-23 21:03:57  libssl1.1:amd64                  1.1.1-1ubuntu2.1~18.04.5     1.1.1-1ubuntu2.1~18.04.4

2019-11-23 21:04:02  ubuntu-release-upgrader-core:all     1:18.04.36               1:18.04.34

2019-11-23 21:04:02  python3-distupgrade:all       1:18.04.36                            1:18.04.34

2019-11-23 21:04:06  sosreport:amd64                  3.6-1ubuntu0.18.04.4          3.6-1ubuntu0.18.04.3

 

Tying On-Premises and Cloud Together

More and more common today is the combination of both on-premises and cloud environments. Often termed hybrid clouds, these attempt to offer the best of both worlds. Hybrid clouds allow an organization to retain control of on-premises data and servers while allowing the gradual expansion into cloud offerings.

Of course, this type of arrangement can potentially make patching more difficult. For example, your on-premises environment could be based on VMware, whereas your cloud environment could be based within Azure. Both offer different interfaces and abilities.

Let's take the example of a set of traditional Windows VMware servers combined with Azure Windows virtual machine (VM) instances. How would you report on this combination of environments? Using PowerShell Core as before, you can pull patch reports from both, as shown in the following sections.

Report on Patch Status

# Azure VM Windows Patch Status

$Params = @{

        "Name" = 'AZWindowsVM'

        "ResourceGroupName" = 'azure-vms'

        "ScriptBlock" = {

              $updateSession = New-Object -ComObject 'Microsoft.Update.Session'

              $updateSearcher = $updateSession.CreateUpdateSearcher()


              If ($updates = ($updateSearcher.Search($null))) {

                    $updates.Updates | Select-Object Title, RebootRequired, IsDownloaded

              }

        }

        "Credential" = $AZCredential

}

$AZResults = Invoke-AzVMCommand @Params

# On-Premise VM Windows Patch Status

$Params = @{

        "Name" = 'WindowsVM'

        "ScriptBlock" = {

              $updateSession = New-Object -ComObject 'Microsoft.Update.Session'

              $updateSearcher = $updateSession.CreateUpdateSearcher()

              If ($updates = ($updateSearcher.Search($null))) {

                    $updates.Updates | Select-Object Title, RebootRequired, IsDownloaded

              }

        }

        "Credential" = $Credential

}

$OPResults = Invoke-Command @Params

 

To use PowerShell remoting with Azure VMs you need to use the Azure Cloud Shell, which grants you access to four cmdlets: Enable-AzVMPSRemoting, Disable-AzVMPSRemoting, Invoke-AzVMCommand, and Enter-AzVM. Examples are below.

Enable-AzVMPSRemoting -Name 'AZWindowsVM' -ResourceGroupName 'azure-vms' -Protocol https -OsType Windows

Enable-AzVMPSRemoting -Name 'AZLinuxVM' -ResourceGroupName 'azure-vms' -Protocol ssh -OsType Linux

 

Keeping Containers Up to Date

It's nearly impossible to talk about cloud environments these days without mentioning containers and orchestration platforms such as Kubernetes. How does patching work in an immutable infrastructure environment?

The principle idea behind containers is that they are immutable, such that they are not intended to be changed. Additionally, they are meant to be ephemeral–you can create and destroy them continuously as your demands change.

So what does this mean for patching these containers? Well, you don't really patch the running containers; you patch the image that the container is based on. Here below, we’ll show what that looks like in a Kubernetes environment.

Update & Validate Container Images

# Update all deployment/nginx images to NGINX version 1.17.6 and initiate a rolling update

kubectl set image deployment/nginx nginx=nginx:1.17.6

# Verify that the rollout status

kubectl rollout status deployment/nginx

 

How then does this fit into CI/CD? Just as before, you would build the patching and validation of these containers into your CI pipeline. By applying the patch, spinning up a test environment, and running validations against your containers, this process fits perfectly into your patching CI/CD process.

Docker containers utilize Dockerfiles to define a container’s configuration. By scanning this configuration file for known software versions or operating systems that are vulnerable, you are able to avoid scanning the entire container itself. This can potentially avoid an expensive and lengthy scanning of your containers as well as costly container re-creations.

As Kubernetes itself updates, it's also important to keep the management platform up to date–not only for the security that updates bring, but also updated functionality to make orchestration of your containers that much easier. 

Let’s take a look at a common example of upgrading Kubernetes.

Upgrading Kubernetes

# Verify that v1.16.0 is available and the cluster is upgradeable

kubeadm upgrade plan v1.16.0

# Perform a dry-run of the upgrade

kubeadm upgrade apply v1.16.0 --dry-run

# If everything looks good, apply the upgrade

kubeadm upgrade apply v1.16.0

 

It may be a bit more difficult to see how the updating of Kubernetes itself fits into this process. But just like any CI/CD pipeline, you can create one that works just the same. By spinning up a new Kubernetes version and testing against a test environment, you accomplish the same level of control and set of approvals that can then be automatically deployed into production.

Read more: Bringing a DevOps Mindset to Vulnerability Management

Conclusion

While there are many different types of systems, tools, and platforms that can be used to manage your infrastructure, CI/CD methodologies and practices can be applied to all of them. Taking control of your processes and procedures can help speed up your organization's reaction time when patching goes wrong. It also simply helps to be proactive and avoid the next big misstep.

Don’t miss out on the latest

Get notified on Industry updates.
we promise not to spam

Related Posts

Popular Articles

  With nearly 15,000 new vulnerabilities discovered in 2017, and even more expected this year – the competition for ‘worst vulnerability’ is a tough ...
Read more

11.1.2018 | vulnerabilities

| Posted by Roy Horev
The best way to share information about the risks associated with vulnerabilities is via quantifying these risks – i.e. metrics. The question is, ...
Read more
The number of vulnerabilities uncovered daily has long exceeded what security teams can possibly address. The key to success in vulnerability ...
Read more