Best Practices: 5 Risks To Assess for Secure CI Pipeline

an introduction

As the computing world continues to develop new processes for creating software, criminals continue to develop their own technologies that exploit flaws in those processes. DevOps is the latest trend in software development, characterized by high levels of automation. More and more parts of the software development process can happen without human intervention which will speed up the development process. However, this is not without its drawbacks.

Less human involvement means less supervision from start to finish, and it also means more technologies to exploit or abuse. Most of the risks involved in using sensitive information are related to automation, allowing for multiple ways to steal secrets. There are also things like tampering with codes that are cause for concern. To keep your code and secrets safe, you should add the following security practices to your CI pipeline.

Third party workflows

From Codecov’s attack, we can see the impact a single workflow can have on the security of the overall CI pipeline. Supply chain attacks are on the rise, and supply chain risk is an entire topic in itself (learn more about it here). Right now, it’s more important than ever to pay extra attention to the things your code and infrastructure are exposed to.

It is common practice to use third party workflows in the CI pipeline, but you have to understand that you trust that third party with your code and possibly your secrets when you do. If possible, you should take a look at the source code for the images and workflows you are considering using in your CI pipeline. If your workflow is tracked using version control, you can also review changes periodically to make sure nothing suspicious has creeped in.

Due to the time or availability of the source code, this may not always be possible. However, you need to balance this against accepting the risks involved in blindly trusting a third party workflow. At the very least, you should look for workflows that are either published by a trusted authority or that are popular and widely trusted by others. There will always be some risks involved in trusting a third party, which underscores the importance of taking measures that limit the damage that can occur.

Access Control Permission

The next thing you need to consider when securing a CI pipeline is access control. With key management systems, you can use features such as role-based access control to determine who can use secrets. You should always follow the principle of least privilege when deciding who should have access to secrets. Setting the access level of the key management system is very important; However, it is not the only point of risk in the life cycle of the secret.

Consider this scenario: You maintain a popular open source repository on GitHub, and you get pull requests fairly routinely. To help you save time evaluating the submitted code, you can start some automated tests when someone submits a new pull request.

One day, someone sent malicious code in a pull request that was grabbing the environment variables and shipping them to the author’s server. They also added a test to make sure their code runs when the CI pipeline is running. Once the pull request is sent, the pipeline starts and all environment variables in your CI environment are stolen. oops.

The above example is not even the worst case scenario. You can read about similar scenarios like the one in this blog. In this case, the attacker managed to break out of the test environment and steal more valuable information.

The last thing you want is for an untrusted party to be able to run arbitrary code in your build environment. It is not enough to protect your secrets in the key management system. You also need to make sure that you trust the code that will use these secrets. Especially if you’re an open source maintainer, don’t run any workflows on pull requests until you’ve read the code yourself.

Segmentation/signing of premises

Hash designs are more important to the security of your users than to your code, but taking the concept a bit more can benefit you in some ways as well. As an example, let’s examine one of the biggest cyber attacks of 2020: the SolarWinds supply chain attack.

SolarWinds is a leading provider of network monitoring and management tools used by countless organizations. In a long-running campaign, an adversary infected SolarWinds servers with a custom piece of malware called Sunspot. Sunspot was a highly advanced malware that monitored processes running on build servers, and was specifically looking for processes involved in assembling SolarWinds’ Orion product.

When Sunspot sees Orion being compiled on the build server, it will inject additional code, later called “Sunburst”, into the compiled program. Sunburst was a backdoor used by attackers to gain access to every organization running Orion.

So, what is our conclusion from this? Because the building itself got infected, it was signed by SolarWinds and was actually part of the product they launched. In order to detect such an attack in the future, SolarWinds says it is looking at ways to check concurrent architectures against each other. This can be a challenge due to the sensitivity of the hash algorithms, but there is still something to explore to be able to discover build injection.

Protect your building environment

From the SolarWinds attack, we can clearly see that the build environment isn’t just a developer sandbox – it’s a very sensitive asset that requires protection. Sunspot has managed to stay on SolarWinds build servers for a long time, but there were definitely IOCs (indicators of compromise) that would have given up on attackers’ presence.

Without getting too deep into computer and network defense, there are some general best practices for securing build servers. First, make sure that the correct logging is enabled on the build servers and forwarded to SIEM or the log collector. Second, consider deploying an EDR proxy on your build servers to get alerts and additional telemetry when something fishy happens. Third, use network prevention and discovery tools, and treat your build servers with scrutiny.

There are too many potential layers of computer protection to list them all, so we’ll stop there. If you do not have much experience in the security and defense aspect of computer and network, reach out to some experts who can provide additional support or guidance. You can also take a look at a framework like NIST’s Cybersecurity Framework to point you in the right direction.

Secrets Management

If you are a software engineer or an experienced security professional, you may have heard of API keys being leaked from public code repositories. You may have tried leaking your private secrets after you accidentally committed it to an open source project. Depending on the type of secret that was leaked, it could end up being a costly mistake.

The best way to protect your secrets is to practice good secrets management. A good start is to use confidential management tools such as Azure Key Vault or Amazon KWS that provide secure storage and identity-based access (learn more here). Using GitHub’s built-in repository secrets manager also works well depending on your use case, but it’s not as feature-rich as the real key management service.

Another must-have for secret management is a tool that can tell you instantly if you’ve committed a secret to your code base. There are a few different options, but secret disclosure is GitGuardian’s specialty. It has hundreds of built-in secret detectors and is free for open source projects. The right knowledge when you accidentally reveal your secrets is critical in protecting yourself and your code.

Even with the above practices, there is still no guarantee that your secrets are safe. A while ago, Codecov’s Docker image was quietly modified to leak secrets to anyone who uses it in their CI pipeline for testing.

Although it is impossible to prevent something like a Codecov attack on your own, you can still limit its impact on your organization. If possible, use different secrets in production than the ones you use in your CI pipeline. This way if something like a hacked Codecov workflow steals your keys, it won’t affect your production environment.

conclusion

Unfortunately, Internet enemies are always improving their technologies, and the challenge of protecting against new attacks will always be there. The CI pipeline is among the most recently targeted assets, and there are plenty of opportunities due to the lack of human involvement.

The above security practices will greatly improve the security of the CI pipeline, but make sure to always keep abreast of new trends in attacker technologies as well.

.

Leave a Comment