Hello, everybody. Good morning, good afternoon, good evening, depending on where you are
So this talk is on software supply chain sanitization, and there's an alphabet soup of tools and
technologies that we're going to cover. So a little bit about myself. I have been in the industry as the gray hairs show for
over 20 years. I spent 12 of them at Microsoft in consulting services traveling around the
country and internationally. I've been a startup founder in 2018. I found a startup
that was tested to play around at MIT. And then COVID happened and brought a quick end
to any discretionary funding they may have gotten. And so here I am now as a partner
technologist involved in product security architecture at Aviva. Aviva is focused on
industrial IoT. And my hobbies, aside from software development, of course, include filmmaking
I do have a profile on IMDb. I'm also a home brewer and enjoy photography
All right. So the goals of this presentation. We'd like to establish a common vocabulary and
know which tools we're talking about here. We also need standards. For example, we need
to determine how vulnerable a security vulnerability is. What one person may regard as a high
vulnerability may not be a high to somebody else. We'll cover regulatory and industry
requirements and how we got here. Specifically, there's an effective order that has a direct
impact on this. And we'll define patterns and practices for sanitizing your software
supply chain, which go beyond just the tooling. Having tools involved in your pipeline are great
but without guidance and governance to manage that, you're going to get spotty implementations
and assurance. So let's start with the software supply chain itself. What is it
It's composed of components, libraries, tools, and processes used to develop, build, and publish
the software artifact. So it's a short sentence, but it tapers quite a bit. And that software
supply chain is vulnerable. We've had a number of breaches in recent years, most notably from
SolarWinds. The malicious hackers were able to infiltrate the SolarWinds network, check in malicious code directly into SolarWinds repositories that affected the server-side
software that pushed updates off the customer systems and infected customer systems with
malware through that update process. 18,000 customers were affected. It made national news
And it got the attention of our government, which resulted in some changes
We've got a few scary stats. There have been, as of two years ago, 18,000 to 20,000 new
vulnerabilities reported through the Common Vulnerability Enumeration Database. We'll cover that a little bit later. Things-based breaches were quite expensive, at an average of 4.5 million
And it's predicted to get worse. Notably, there are numerous misconfigurations in software
repositories that expose secrets, artifacts, source code. If you look at GitHub, they're
alone. It's got machine keys, certificates, API keys that people have checked in. Hopefully
they've recycled them and retarded them. But not everybody has. One of the common vulnerabilities
that was discussed at Black Hat, for example, with ASP web forms were you've been using machine
keys that people found in sample applications that were on GitHub without changing them
There are even hackers capable of using exposed machine keys. And you can bet they use them for
exports. So, we've got a few things to worry about. Our external threats. We've got our IT
departments that monitor for intrusion detection. We've got phishing attacks coming into our emails
But the ones that I'd like to focus on in this talk are malicious developer tools
open source components, container registries, generally the public repositories that we use
for our open source development. Those repositories can be published to by anyone
There's no betting process. For example, anyone can publish a GitHub action and anyone has
There are GitHub actions that will exploit GitHub itself for crypto mining or exfiltrate
secrets from your repository. But you have to be careful. And it's not restricted to that
We've got Visual Studio Code extensions that are malicious. Microsoft does not vet
what's published there. NPM is no outlier. It's got over 12,000 suspicious packages
And recently, even for packages that we've come to trust, like a lot of people on this call
probably use Mock for unit and integration testing. And Mock recently had a privacy
violation in which they sent developers private email to a third party for soliciting donations
And they're stalkers. And we can say that we'll trust our trusted publishers like Microsoft
like Amazon. But some of these actors will rename the container or the module to be something close
to something we trust. So it's easy to slip up. So how do we prevent ourselves from falling into
that trap? And you're probably thinking, well, how do I put this into action? What concrete
steps can I take to enforce security on my pipelines? So we've got a number of types of
tools. There's a bit of an alphabet soup there. So what you should have is a multi-layered defense
No one single set of tools is going to be your silver bullet. You need to be checking for
secrets and make sure that you're not pushing passwords or certificates into your pipeline
And when you do, it should notify you. And that needs to be rotated out. Your secrets
must be stored in something more secure like Azure key vaults or encrypted pipeline variables
GitHub has secret scanning capabilities. There are secret scanning tools available in ADL
A software composition analysis is going to be a big topic here. This analyzes and identifies
third-party components in your software packages. It identifies the packages that you'll have to use
and ties them into known vulnerabilities with those packages. Static application security testing is something that we've kind of been doing
If you've been using Visual Studio and you've gotten code analysis warnings and violations
that's been a form of static application security testing. It's part of the solution
But it doesn't track the issues and make them available to the rest of the team or
for auditing purposes. Dynamic application security testing, as it states in the name
requires some active components. And that typically involves an API and a automated test that
attempts to reproduce some of the common OS top-10 vulnerabilities. Then there's IaaS and RASP, Interactive Application Security Testing Runtime
Application Security Protection that runs while the application itself is running. That's beyond the scope of this presentation. We certainly don't have the time to cover that
but you should be aware of it. And lastly, there are policies and procedures for governing these
tools. These tools can generate quite a bit of noise and you need some governance to drive
filtering that noise and determining what should have priority. So, at this point, you're probably dealing with this alphabet soup of technologies
Let's break down a little bit further. So, what is software composition analysis used for
So, it scans our redistributable, our MSIs, EXEs, anything that deploys to the cloud
anything that we deploy to a production environment like containers. The risks that it should identify are operational. For example, if you're using a component that's
10 years old and hasn't been updated in 10 years, that's a problem. You should consider replacing that
It certainly should identify security risks, the CVE enumerations, common vulnerabilities that have been identified, and lastly, licenses. So, if you're producing commercial software
so if you're producing commercial software, you should be aware of copyleft licenses like TPL
which states that if you utilize this given component, then the license obligates you to
releasing your software with that same TPL license. It's commonly referred to as copyleft
And that's great for open source projects, but if you're working for a company that wants to make
money, you might want to reconsider using these specific components. And there are some that are
meant exclusively for non-commercial licenses like JRL. So, I've been referring to these CVE
vulnerabilities. So, what are they and how do they get registered? So, there are organizations that
will do cybersecurity research like AquaSec and CrowdStrike. Companies themselves will kind of
eat their own dog food and also self-report. They report to the CVE program, the Common
Vulnerability Enumeration Program, which has an aggregate of all these publicly known vulnerabilities
And typically, this researcher, this cybersecurity researcher will, if they find a zero-day vulnerability, the ethical researcher will report it to the company
when they find it and notify the company that, well, you have 30 days to remediate this or 60
days to remediate this before we go public. And then the report process begins. Unethical
researchers are called hackers, and they'll exploit that zero-day vulnerability. So, the CVE identifier gets issued from an authority. So, we have a well-known
common number to refer to this vulnerability list. And ultimately, it will get published to NIST
Now, once again, what you might consider a high vulnerability item may not be high
for somebody else. So, NIST has established the Common Vulnerability Scoring System, CVSS
which is not necessarily a measure of risk. It's a measure of vulnerability
So, consider this for a moment. Let's say you have a component that you're using
a third-party component that's got a vulnerability that says that if a given method is invoked
with a given set of parameters, that you will experience a denial-of-service attack
You know, the package will fault. But if your application doesn't invoke that method
your application is not at risk. So, our customers have been running their own binary scans and
analysis and have been coming back to organizations with expectations for getting
remediation statements for being reassured that, A, whether they are vulnerable or not
or B, if they are, that a hotfix is forthcoming. So, let's take a look at what one of these vulnerabilities looks like when it's reported
So, here is the NIST vulnerability database. Here is one that I ended up having to investigate
The ZLIB allows memory corruption when deflating, if the input has many different
matches. So, to boil this down, this was a vulnerability that was exposed when calling
the deflate method, specifically deflate2 on ZLIB. ZLIB, by the way, is used by .NET Framework
and .NET. Even if you're not explicitly calling it, it's used by the System.IO compression
library. Oh, I see a question in there. What's alternative for mock? There's XUnits
I think facets have been used in testing. But I wouldn't have abandoned mock. I would have just
given them some time to work out that particular privacy issue. There's been a bit of uproar in
the community about that, and they're probably going to roll that back. So, don't give up on mock
So, with respect to ZLIB, you're using it implicitly if you use System.IO compression
or if you are using GZIP to send HTTP requests. This is a denial... effectively
this is a denial of service vulnerability. There's this vector score which breaks down
the nature of the severity of the vulnerability. This CDFF score is determined using this common
vulnerability scoring system calculator. And this score shows us that, well, you need to be
on network. The attack complexity is pretty low. You just need to call a given method
No significant privileges are required, but the availability for it, the availability of the
component if it is exploited is high. Total loss at event. So, that results in a high vulnerability
score. That's how we get there. If it were low, it would be at a median severity. So
this results in a vector string. And there it is. If you are tasked with investigating one of these vulnerabilities
and coming up with a mitigation statement, this set of hyperlinks is incredibly helpful
But something is conspicuously missing here. And typically, affected components that require the hotfix from a commercial organization that supports that component
like Microsoft, typically has a patch. And a link to that patch is usually found
in the set of missed hyperlinks. There is no Microsoft patch for this. Ultimately, the
zlib was being used through .NET in a safe fashion. It was not exposing this
deflate method by calling these set of parameters that were edge pieces. It was using it in a safe
manner. So, Microsoft had no need to patch it, yet zlib shows up as one of the components that
ends up getting used in a binary analysis because it's packaged with .NET
So, some of the useful links here include things like the issue in GitHub that addressed
the particular vulnerability. So, this is your friend if you need to drill into
the vulnerability investigated and create one of these mitigation statements. Let's turn to the presentation
So, this is telling. There are, at this point, there are hundreds of thousands of CDEs
So, how do you know which ones to pay attention to? Well, the vast majority do not get exploited
They don't get used by a malicious actor. Usually, the complexity for exploiting these is so high
that, well, they can't quite easily do it. But this Venn diagram tells us what we should be
prioritizing on. CBFS scores that are high that overlap with positives that have been used for
an exploit. Right. And so, to close the gap between what is considered a high vulnerability
versus what is a risk to your application, there's this exploit prediction scoring system
which is still evolving. It hasn't been widely adopted yet, but you should be aware of it
that risk or rather severity does not always equate to risk. And we've had a tool at our disposal for free that we can use right now. Nothing else
needs to be installed to find if you're using any NUCA packages that are vulnerable. Excuse me
getting a record. You can use .NET with a package that's vulnerable executed from your
project directory and you'll get that list and along with a helpful URL
So, one of the clauses in the executive order 14028 was to include a software bill of materials
Which is a list of all the third-party open source and commercial components that your
application uses. So, as stated in the executive order, formal record contained the details and
supply chain relationships of various components used in building software. Now, that's rather vague
Right. That's not a JSON schema. That's not an XML schema. That's a rather open-ended definition
We need something a little more concrete if we're going to take action against this. And that's the SBOM format itself. And we've got three definitions. They've been evolving
over the last few years. STDX, SiteLineDX, and Software Identification Tags. We can really
disregard the software tags. They don't contain – they might meet the legal requirements of the
SBOM definition, but they don't have enough. Both SBDX and SiteLineDX include copyright
statements, licenses, hash values to uniquely identify the third-party components. SBDX is a
bit older. It's been around and it's evolved a bit. It tends to be preferred by cybersecurity
professionals, whereas SiteLineDX is more machine-readable and preferred by developers. And here's a small comparison between the two and how two different tools identified – well
two different tools used – one used SBDX, the other one used SiteLineDX, and how it identified
the Microsoft Extension Stipend Key Injection Abstractions package. And as you can see
SiteLineDX is a bit more verbose on the right. The SBDX does include a copyright text
but the tool did not pick it up in this instance. And here's a set of – a minimal set of fields that are expected in an SBOM
irrespective of whether you're using SiteLineDX or SBDX. This – and this is a reference
Simon, I'll send you this PowerPoint, and please feel free to share it out with the audience
There are a number of links in here. So, this is what you can expect to find and be required
to produce. And these are recommended but not required. However, customers are expecting
licenses and copyright statements. So, even if it's not, you know, meeting a legal requirement
even if you can reduce it without that customer, it's going to come back and ask for it. So, there's not just the software bill of materials. There's the bill of materials for
hardware, for formulation BOMs, tools and applications used to produce the software
Like, if you're using – if you're building a UI – a single-page application, you're using Angular
you're very likely using Webpack. That's not something that's shipped, but that's something
that you would expect to see in a formulation BOM. Why they're not calling it an FBOM is – that's
lost opportunity there, as far as I'm concerned. There is an operation bill of materials for
what operating systems were used, your runtime environment, your configurations, and there's others. This is just a short example. There's a comprehensive link at the bottom
But one you should be aware of is this Vulnerability Exploitability Exchange
VEX. So, the FBOM does not include any vulnerabilities. It does not include CVSF scores
VEX fills in that gap and includes the components, its associated CVSF vulnerabilities
more specifically CVE vulnerabilities – I want to make sure I'm using the right acronym – and
mitigations. So, once again, this is another evolving standard. You can expect to find some open-source and free tooling in this space. If you've used GitHub and
have a repo there and have been notified that you've got a third-party component that is due
for an upgrade, you may have used Dependabot. We saw a screenshot from the .NET Scan tool
Microsoft has an open-source FBOM tool that Microsoft is maintaining. Cyclone DX has
their own .NET tool. The screenshots that I showed earlier were from those tools
from FBOM – from this Microsoft FBOM tool and from Cyclone DX
And OWASP has a pretty comprehensive list of open-source tools for – well, in this whole
space – FCA, FAT, and VAS tools. However, if you're trying to assess operational risks
security risks, and license risks, you're not going to find a tool that does all three
You'll end up having to cobble that together. And if you're doing this at scale
you're quite obviously better off at a – with a commercial tool like MEND or Qualys
So, let's say you've got this in place. You've loaned this out to your organization
But you need to give them some guidance. These tools can create quite a bit of noise
So, I'd like to reduce that a bit by focusing only on the components that ship, at least initially
You can use things like package config in your MPM projects that have a package
dev tools section to determine what's internal and what's external. Customers are expecting these mitigation statements
But you're going to get a lot of noise. You're going to get a lot of low and
medium vulnerabilities. Prioritize on the critical highs. Define what your CVSS maximum criticality is for release. Could you – can you accept an
organizational risk to release with a medium CVSS score? Possibly. You should have a process
to block components with licenses that put you in legal judgery, like those copy lab licenses
Replace the components that have high operational risks, like that one back there that might be 10
years old that hasn't gotten any updates. The open source developer has given up on it. Maybe
it's time to locate that out. And developers are often under the gun to deliver on functional
requirements that means, you know, features that we can sell to customers that will make us money
Alleviating this is technical debt. It's overhead. But your organization ought to
allow for some time to cover that overhead. Typically, in companies I've worked with
I've seen teams get, you know, 20 percent of their sprint time allocated to tackling technical debt
and that this falls under. And make an SBOM available to customers. They're going to want
it sooner or later. And you should be familiar with this. So, at this point, I have managed that
mess and this is how I learned to stop worrying and love the SBOM and get it under control
You're not expected to solve everything but solve the highest and critical items. Prioritize
So, moving on to SACs, software application security testing. So, it's helpful in finding
common security and code quality vulnerabilities like writing a plain text password to a log file
or missing an old check somewhere or missing a using statement. It finds code issues early
in the process. You may have heard the term shift left. Some of these tools are starting to run in the
IDE as opposed to like in a server that scans your repo separately asynchronously and produces
a report. That's preferable. Testing it before you even get into repo in the first place is preferable
The advantages of a lot of these tools are that they run passively. They don't break your
application. They don't break your development process. They're easy to launch. They have high
code coverage. But once again, you're likely to see some false positives. It's bad at analyzing
highly specialized libraries. Like if you've got a very detailed business process, it's not going to
catch business logic errors. You still need a code review for that. And it's not going to see
your code in a runtime context. It may catch some cases where you miss like a thread lock
but it's not going to run – it's not going to fire all race conditions
So, you still need a dynamic scan strategy. It looks for common weakness enumerations. These are defined by the MITRE Corporation
that works with the CVE program to identify based on the vulnerabilities that are going to be
reported today and what are the most common. And here's a top 25 set of common application
vulnerabilities, like null pointers. And this does have some overlap with the last top 10
So, we've got a few free open source tools like I mentioned code analysis earlier. GitHub includes
CodeQL. There's ESLint. And it's relatively easy to stand these up in your pipeline. These
screenshots show you how to get CodeQL stood up. And this is just a simple build process to
get it on your build agent to build and then perform the analysis
And we've got a number of commercial tools. And so, once again, if you need to track these
long-term or create work items off of these, then you're better off with a commercial tool
rather than using something that's open source. If you're working on a small team
that might be fine. But if you're in a large organization, you need something more comprehensive
So, let's take a look at GitHub and what CodeQL tells us. So, let's see. How am I doing on time
I'd like to take some notes, I would say. Okay. So, I'll be quick on these because I
still like to cover DASC. So, this is a personal repo I set up. And I've enabled CodeQL on it
And it records a number of security vulnerabilities and quality vulnerabilities, right
And so, I'm going to go ahead and run this. And I'm going to run this. And I'm going to
run this. And I'm going to run this. And I'm going to run this. And I'm going to run this. And I'm going to run this. And I'm going to run this. And I'm going to run this
And I'm going to run this. And I'm going to run this. And I'm going to run this. And I'm going to run this. And I'm going to run this. And I'm going to run this
And I'm going to run this. And I'm going to run this. And I'm going to run this. And interestingly, it also detected an issue with my Azure Evolt
firewall that I haven't protected it well enough. So, it's not just code that it's scanning. It's
also scanning Bicep and RN templates. Now, this is just my open source project. This is sort of a
commercial project that I would block on this. This is something that would be, that should not
make it into production. So, you need to figure out, okay, what are your blockers here
And I'd recommend blocking on high severity security vulnerabilities. And I'd also like to, since we only have a few minutes left, I'm going to skip ahead and give
you an example of a DAST scan. So, dynamic application security testing involves
setting up a process that scans your web API. And these will typically look for the OS top 10
like, let's say, scripting vulnerabilities that will exploit your cookies
SQL injection attacks, second order identifier attacks. So, this tool is a free virtual available
from OS to DAST. If you are not doing any dynamic scanning today, this is better than doing nothing
This is relatively free to set up. Well, it is free to set up. It's relatively easy to use
And this scan was done against the Microsoft ToDo API tutorial in ASP.NET documentation
So, all I did was took the ToDo API, stood it up, and ran that scan against that. And it found a
couple of, yeah, leading priority items, like missing content security policy headers
This one is interesting. It tells you that you're informing the client what type of server you're
running. And that could be used by a malicious actor to exploit the server. They might know that
oh, if you're using, say, ASP.NET Web Forms, that there are a suite of vulnerabilities
for that technology. So, you know, remove that header from your responses
Now, of course, this is just a ToDo application. It's a simple standup
But this shows you what a website might look like prior to hardening
Now, there is a lot of noise here. You can see that this one vulnerability came up in 1,140 tries
So, in this case, removing that header will alleviate all those vulnerabilities
So, once again, prioritize on the high vulnerabilities first. Determine what your
minimum release bar is. Set a bug bar. And set up a death policy test. This can also be run at
the command line. But it does take a little more work to set this up because you need to deploy
the application. You need a host. And so, you need a live site to throw this against
And let's return to the slides and cover what we still have time for
So, these dash tools also do fuzz testing. Some degree of fuzz testing, like sending in
long strings, sending in values that may fool regular expressions. There was a vulnerability that was deployed recently on an OPC IoT server that sent a
string that was badly null terminated and it abandoned the server. The string had a null
terminator and then a few characters after it, and the server didn't know what to do with it
The code wasn't expecting it. Now, dash tools typically have limited dynamic content
And what I mean by that is they're focused on known vulnerabilities, like the OAS Top 10
It's no substitute for a human. So, it's not a substitute for a penetration test
where you've got people using things like Burp Suite and are professional white hat hackers
This is good for day-to-day, but if you've got a public facing API
consider doing a penetration test at least once a year with an external organization
We've got some commercial tools for dash, Rapid7, Detectify, Qualys. Once again, if you're doing nothing today, take a look at OAS ZAttack proxy
That's worth looking into. And these tools typically focus on the OAS Top 10. You can see
what some of the changes look like between 2017 and 2021. There's a couple of new items surfaced, like server-side request flow tree
where the client will try to send a header that should only be attached by a server
behind a firewall, like a load balancer, like an x-forwarded for header that a load balancer
will attach. You could use that to possibly reroute requests if you know something about
the web farm environment that you're targeting to attack. And you'll see a common theme here
One that's worth pointing out is to reduce your attack surface. So if you have retired a route, if you've retired parameters from a request
get rid of them. Give that dash scanner less to look at. And that is one way of securing your
API, reducing simply the number of routes you have. And you're probably wondering, well
excuse me. Still a little bit of a cold. I'm getting over
Pardon me. So if you don't have these tools in your organization yet, start with a pilot
Pick a team, pick a development team that is willing to try something new. Use them to get
some early feedback before rolling it out to the rest of the organization. And when you do
roll out incrementally. Introduce one tool at a time. Give it a quarter. Give it three months
to get some feedback. Back it up with documentation and training. Give teams that are managing
legacy software some time to catch up. Because if you haven't been doing these scans in the first
place, and you've got software that teams have been supporting for several years, no doubt
they're going to get overwhelmed with findings. So don't expect them to solve everything that
it reports on day one. Give them time to catch up and iterate. Both on the tool that you use
and the processes applied to the tool. And as a best practice, sign everything you release
Also use two different certificates while you're signing. One for internal releases
only. That'll tell you if something accidentally slipped out that should not have slipped out
And for external releases, of course. Container signing is an immersion space
All right. Vendors have started to support this, but it's very spotty and uneven
I'd give that a little bit of time to work out. But you should be in a place in about six months
from now to expect to have services in Azure, in the Azure Container Registry
that limits the permitted containers to those that signed. And after you do sign
validate the signatures using something like FinSkin. The Microsoft Security DevOps tool includes a number of subcomponents
and FinSkin can be used to validate that you've signed your binaries and you've signed your
executable releases