Hardening DevSecOps with Domain Data Security and CircleCI

What is DevSecOps?

DevSecOps is an abbreviation for development, security, and operations. It’s an approach to software development that encourages the inclusion of security as a shared responsibility throughout the software development lifecycle. Simply put, DevSecOps means integrating security into the application development process from start to finish.

Having “built-in security” in the DevOps process means implementing security best practices in both code and infrastructure, and automating the security testing process.

What is Webshrinker? 

Webshrinker is an AI-powered domain categorization system owned by DNS security company DNSFilter. Webshrinker is capable of identifying threat domains and labels them according to their threat categories: Phishing, malware, botnet, etc. As of Nov. 2021, Webshrinker has categorized over half a billion domains (563 million to be exact) and continuously scans 3million domains daily.

Webshrinker can be used via the API or by hosting and querying an offline version of its database within your infrastructure.

How a tool like Webshrinker can be involved in the DevSecOps process

The DevSecOps culture promotes security as a shared responsibility between all teams involved in the software development process, including the development team. Developers are encouraged to code with security in mind by working with security teams to have visibility into known vulnerabilities and threats. 

One of the common vulnerabilities in software applications is web forms. Attacks like SQL injection, Cross-site scripting, and sensitive data exposure are amongst the Open Web Application Security Project (OWASP) top 10 web application security risks in 2021 and they all take advantage of the vulnerability of web forms. Web forms allow users to enter values into the application for storage or further processing. Threat actors often take advantage of this entry point to feed malicious data into the application.

Imagine a social networking site where you can supply a link to your personal website. A threat actor can inject a malware domain into their profile so that when other users click on the link, they are taken to a domain that installs malware onto their system. This is where Webshrinker comes in.

With Webshrinker, you can filter the entries into the URL field for submitting a user’s website and block threat domains. But how do we automate this strategy in a standard DevSecOps process? I will be answering that question with a simple experiment in the next section.

Experiment: Running automated tests on a Webshrinker-protected web form in a build pipeline 

To prove the concept of Webshrinker being used in a DevSecOps process, I built a simple form where a user can submit their email address and a URL for their website. To implement the security culture promoted by DevSecOps, I guarded the URL entry field by scanning the URL entry with Webshrinker to block malicious and parked domains.

This form will be passed through a continuous integration pipeline that runs tests to ensure that the form is indeed blocking threat domains submitted by threat actors.

If you wish to follow along with the demonstration, here are a few prerequisites you will need:

• Nodejs installed on your system (version 12 or greater)
• Git installed on your system
• A CircleCI account.
  

If you do not wish to practically follow along, you can continue reading as I explain the process and the results.

Running the webform

The first step is to run the demo web form locally and inspect its behaviour. You can clone the code for the form from this repository by running the following command at any convenient location on your system:


git clone -b base-project --single-branch https://github.com/coderonfleek/webshrinker-devsecops.git


Once you have the code on your system, install the dependencies by running the following command:


cd webshrinker-devsecops
npm install


When the installation is complete, run the application with the following command:


node server


The application boots up, and you can view the webform at the address https://localhost:5000 as shown below being tested with a non-threat domain and a malicious one:

Adding tests

The next step is to write tests that perform the security checks automatically. I will be adding some automated functional tests using Google’s Puppeteer to simulate a real-world scenario of a user filling the form.

‍

‍

This file contains 3 test cases, the first test case checks that non-threat domains like facebook.com are not blocked by the system. This ensures that our security implementation does not overzealously block regular domains.

The second test case checks for malicious entries by using a sample malicious domain. If our form blocks this domain, the test passes, if not, the test fails.

The final test case checks for phishing domain entries into the URL field using a known phishing site. 

Do note that our application only checks for malicious and parked domains, this was intentional for this demonstration. 

As a best practice, it is important to separate the development and testing teams. This will enable the testing team to write exhaustive tests and protect developers from themselves as developers are often tempted to write tests that match their code capabilities.

Adding a CI/CD pipeline configuration with CircleCI

To automate the testing process, a continuous integration (CI) pipeline will be built using CircleCI as the CI server.

‍

‍

‍

‍

Once added, push your code to the GitHub account attached to your CircleCI account and add the repository as a CircleCI project as shown in the screenshots below:

Running the automated tests

Once the project is set up on CircleCI, the tests run immediately. When the build is done, you will see a build status similar to the one below:

This indicates that our build failed which means that one or more tests have failed. 

Analyzing the results

To find why the build failed, we need to dig into the details of the build pipeline that we just ran. Clicking the build link (the one with the red icon beside it) brings up the pipeline run details. Scrolling down to the tests section, the results of the test can be found as displayed below:

Here, we see that 2 tests passed out of 3. Let’s take a closer look at the status message as shown below:

As seen, the non-threat domain and malicious domain tests passed while the phishing test failed. This is not a surprise as the developer did not check for phishing domains, only parked and malicious domains are being blocked by the security gate implemented in the application.

This failed test provides feedback to the developers to ensure that phishing domains are part of the blocklist for this application. This helps enforce the culture of shared responsibility for the security of the system and the feedback loop required in a DevSecOps process.

For this demonstration, the developer can simply fix this issue by adding “deceptive” (Webshrinker’s identifier for domains in the category of “Phishing and Deception”) to the blocklist array in the javascript section of the index.html file like below:

‍

‍

‍

This update can then be pushed to the GitHub repository connected to CircleCI for a fresh build. The new build completes successfully as shown below:

You can find the complete code for the application and tests on the `main` branch of this repository.

Drawing inferences from the experiment results

So far, experimenting with Webshrinker as a security component in a DevSecOps process has been successful; however, there are a few considerations and insights that are worth noting:

• For teams running 100s of builds (and I have seen teams with thousands), making API calls for each scan may not be so effective and can lead to slow builds. A better solution is to use the Webshrinker offline database. This reduces the latency of each domain query and leads to faster builds.
• Another thing to consider is Webshrinker API’s rate-limiting controls. Running multiple concurrent tests can easily hit Webshrinker’s API limits. This is another reason why the offline database might be a better option in this scenario.
• Zero-day threats are also a point of concern as the test cases make use of well-established threat domains. Fortunately, catching zero-day threats is one of the areas where Webshrinker’s AI shines, catching more unique threats than competitors (over 68% of all deceptive sites found by Webshrinker, other site categorization tools don’t catch) and doing it as much as 7 days earlier.
  

Conclusion

This demonstration, though a mere proof of concept, shows that Webshrinker has a role to play in implementing security automation in a DevSecOps process. Threat domains injected into software applications or application infrastructure can be used to gain remote access, create covert channels to command and control servers, install malware, or route users to illegitimate websites that can be used to harvest sensitive data. Ensuring that these types of entries are caught automatically in a DevSecOps process is of immense value to the integrity of the application.

This has been a very exciting experiment for me and I encourage you to please share with us any ideas you have on using Webshrinker in security implementations and test automation.