Teachable moments in Product Security: an in-depth analysis of the Verkada breach
Verkada’s cloud-based video surveillance application was breached, exposing 150,000 cameras (according to Bloomberg) installed on networks in customer facilities belonging to Tesla, Cloudflare, UK NHS facilities (who may still be recovering from a massive ransomware attack in 2017), banks, universities, hospitals, jails, and more.
The incident was first announced on March 9th, at which point Verkada notified the FBI and engaged Mandiant to conduct an investigation. We’ve been waiting to see if further technical details would come out in media coverage following the incident, but there are still a lot of questions that may only be answered when and if Verkada release details from Mandiant’s report.
We’ve reviewed Verkada’s updates, several articles and blogs covering both the incident and the arrest of the threat actor. We also attended a customer webinar with Verkada’s CEO and CISO. We may not have additional information for a while, so we’ve decided to post our analysis and some security lessons to be learned based on the information that’s currently available.
Who’s Verkada?
Founded in 2016, Verkada is a silicon valley tech startup who provide a cloud-based video surveillance solution to over 5,000 customers (as of 2020, according to their site). Their solution is end-to-end, meaning they provide hardware (cameras, sensors, and access control devices like card readers), cloud storage for recorded video, and a cloud-based video management application which allows customers to manage and monitor their cameras from anywhere on the internet.
It’s similar to Ring, Nest, or Arlo, but with commercial-grade hardware and many more features to make it suitable for larger businesses. If you’re not familiar with any of these - here’s how Verkada works:
If you have a business and need video surveillance, you can pay Verkada (through a dealer partner) an annual fee. In exchange for that annual fee, they’ll give you some cameras to install at your business. All you need to do is connect those cameras to the internet, and they handle the rest. The cameras will record video and send it to their cloud for storage, and you can login to their cloud portal using a browser to do whatever you need to do with your cameras. No recording devices, servers, storage, or software installation - pretty simple, right?
For the right application, a cloud-based video surveillance solution makes a lot of sense because it’s easy, simple, and relatively maintenance-free. If you’re not protecting critical infrastructure like an airport or military base, or a casino, you don’t need a very robust set of features, low latency, or the ability to airgap your system for ‘guaranteed’ protection from internet-based threats.
Unfortunately, this is another incident which proves that internet-based threats are real, and that failure to follow just one security best-practice (least privilege) can completely undermine all of your otherwise-effective security controls.
The threat actor and motive
Tillie Kottmann, a 21-year old from Lucerne, Switzerland (and who prefers the pronouns they/them) took credit for gaining unauthorized access to Verkada’s infrastructure and customer surveillance systems. Tillie was part of a hacktivist group who call themselves APT-69420.
Kottmann has since been arrested by Swiss authorities, charged with conspiracy, wire fraud, and aggravated identity theft by the U.S. Department of Justice, and indicted by a Grand Jury in U.S. federal court. The Verkada incident may have been the straw that broke the camel’s back, as initial charges are for previous hacking activities which date back to 2019.
For several years, Kottmann and other members of APT-69420 have leaked confidential and proprietary source code and information to a repository which has since been seized by the FBI. According to the indictment, leaks included data from over 100 large companies including Intel, Nissan, Lenovo, Nintendo, Motorola, AMD, Qualcomm, Adobe, Johnson Controls, and General Electric.
Motive?
Kottmann said the following to Bloomberg about the incident:
“…exposes just how broadly we’re being surveilled, and how little care is put into at least securing the platforms used to do so, pursuing nothing but profit”
And this to SecurityInfoWatch:
“I also just want to clarify that I am not a white hat; that I hate the corporate infosec industry and I wish more people would actually work on improving the world instead of protecting corporate interests and doing funny bug-bounties for Raytheon or whatever y'all do these days. Hack the planet,”
“I generally dislike any kind of surveillance, but if you want to do it please do not use a centralized cloud platform of some VC-funded startup that has more sales employees than customers and cares about nothing other than profit.”
From that, we gather that the goal was to improve the world by exposing surveillance, and exposing ‘how little care is put into securing the platforms used to do so’.
On exposing broad surveillance, we’re not sure what new information this incident exposed to the world. You would be hard-pressed to find someone who is surprised to know that there’s video surveillance going on in a jail or a bank or a manufacturing facility.
Verkada doesn’t make covert cameras. Also, people who work at companies monitored by these cameras (or any other cameras for that matter) are generally made aware of surveillance by their employment agreements, signage around their facilities, and just by seeing the cameras around their workplace or in public spaces.
Now let’s talk about ‘how little care is put into securing the platforms used to do so’. Unfortunately, that is a true statement more often than not. This is made obvious each time an incident like this occurs, and by the relatively low level of sophistication required to exploit many of the published vulnerabilities in surveillance devices.
It’s not ideal that a situation like this needs to happen to an organization before they begin taking things seriously, but companies don’t have unlimited resources, engineers are expensive, and doing things securely can take a little more time. IT security standards and data protection regulations are pushing progress in the right direction, and the increased frequency of breaches is raising awareness of supply chain risks for end-users like Verkada’s customers.
Being a ‘white hat’ hacker is not about protecting corporate interests. It’s about ‘hacking the planet’ to improve the world while doing no harm, and bonus if a corporation pays you for it. Compromising the safety and security of Verkada’s end-user customers, especially organizations like schools and hospitals where employees work day and night to make the world a better place, is NOT a good way to ‘improve the world’.
The attack
The attack vector which provided the threat actor with initial access still remains unclear.
Based on public statements by Verkada and statements which the threat actor made to several media outlets, our hypothesis is that:
The threat actor found ‘super admin’ credentials which provided privileged access to Verkada’s web application, exposing access to customer video. And, these credentials were sitting in plain text inside a script on a publicly exposed build server which didn’t require authentication?
On the same server, the threat actor also accessed an internal support tool which provided root shell access directly to customer cameras, and authenticated using hardcoded credentials
We’d like to begin our analysis of the attack with a quick quote from the 2020 Verizon Data Breach Investigation Report, since we’re about to describe another breach that supports their findings:
Examining the types of data stolen today, in both small and large organizations, we see that Payment card data is so last year. Today’s criminal (lacking the work ethic of 2013) is primarily concerned with obtaining Credentials
(note that the mention of 2013 refers to a year when installing credit card skimmers to get card numbers was a popular threat action)
Moving on. The threat actor granted interviews to several media outlets including Bloomberg, BleepingComputer, and SecurityInfoWatch, a source dedicated to video surveillance and physical security systems. Here’s what they were told:
Bloomberg:
Kottmann said their group was able to obtain “root” access on the cameras, meaning they could use the cameras to execute their own code. That access could, in some instances, allow them to pivot and obtain access to the broader corporate network of Verkada’s customers, or hijack the cameras and use them as a platform to launch future hacks. Obtaining this degree of access to the camera didn’t require any additional hacking, as it was a built-in feature, Kottmann said.
The hackers’ methods were unsophisticated: they gained access to Verkada through a “Super Admin” account, allowing them to peer into the cameras of all of its customers. Kottmann says they found a user name and password for an administrator account publicly exposed on the internet.
BleepingComputer:
Speaking to BleepingComputer, Kottmann said they found hardcoded credentials for a Verkada super admin account in exposed DevOps infrastructure.
SecurityInfoWatch:
“We found super admin credentials in a python script on a publicly exposed Veracode Jenkins Plugin on the Verkada server, which allowed us to log in to their web app with super admin privileges,” Kottman explains, displaying the actual screenshot of the group’s exploits attached to an email that was sent to SIW. “We did not exploit any flaws or vulnerabilities. The cameras have a built-in maintenance backdoor, which allows anyone with super admin privileges to access a root shell on any camera of any customer at the click of a button.”
Additionally, Verkada’s March 10th update to customers stated that they identified the attack vector, and provided these details:
The attack targeted a Jenkins server used by our support team to perform bulk maintenance operations on customer cameras, such as adjusting camera image settings upon customer request. We believe the attackers gained access to this server on March 7, 2021 and maintained access until approximately noon PST on March 9, 2021. In gaining access to the server, the attackers obtained credentials that allowed them to bypass our authorization system, including two-factor authentication.
So, we know that ‘super admin’ credentials were obtained, possibly from a Python script on a Jenkins server, which was somehow ‘exposed’…
Where did the credentials actually come from?
Publicly exposed…
All three of the interview sources above mention ‘exposed’ or ‘publicly exposed’. Additionally, SecurityInfoWatch was told that no flaws or vulnerabilities were exploited. We assume this means that anyone on the internet could have accessed what the threat actor accessed without authentication or authorization - otherwise that wouldn’t quite mean ‘publicly exposed’.
DevOps infrastructure…and who’s Jenkins?
One article mentions DevOps infrastructure (which describes a Jenkins server), and another specifically mentions a Veracode Jenkins Plugin on a Verkada server. Verkada’s security update also mentions that the attacker gained access to a Jenkins server (notice that they didn’t say hacked, breached, or gained unauthorized access) So, a Jenkins server was definitely involved.
To quickly summarize Jenkins - it’s an open-source platform used by software developers to automate the build process. Once developers finish writing code, Jenkins carries out ‘build steps’ following a pre-defined process to run security scans, tests, change version numbers, and package everything up before it’s ready to distribute.
We’re still working on assumptions, which now have us thinking that Verkada had a Jenkins server publicly exposed to the internet.
Python script for a Veracode Jenkins Plugin…
The SecurityInfoWatch quote seems to provide the most detail around the artifact that was obtained from said Jenkins server, which was claimed to be a python script for a Veracode Jenkins plugin. Let’s break this down:
Python script on a Jenkins server - developers commonly use Python scripts as ‘build steps’ for Jenkins to execute. So, when Jenkins is doing its thing, one of the steps in the process can be to run a Python script.
Veracode Jenkins Plugin - Veracode is an application security platform which helps developers secure their software by scanning code for errors or security flaws. Veracode makes plugins for many software development tools, including Jenkins, which enables Veracode to scan during the software build process.
In order for the plugin to communicate with the Veracode platform, build scripts would need to provide the plugin with Veracode credentials. Developers leave those credentials in their scripts all the time, although Jenkins has a '‘Credentials Binding Plugin’ which can be used to protect credentials in this scenario(which we hope they used considering this Jenkins server was apparently exposed to the internet). But either way - the Veracode credentials weren’t the ones that were stolen. So why would credentials for Verkada’s products end up in there?
Veracode scans using two methods: static analysis and dynamic analysis. During static analysis, it just scans through code to look for problems, without actually executing the code. During dynamic analysis, it examines how the program operates while it’s running. If the program it’s examining has a web interface for a camera or a video management application which requires authentication, it needs credentials for testing.
So, we’re still stacking assumptions on assumptions, but it’s possible that the ‘super admin’ credentials lived in this Python script to allow the Veracode plugin to login to a camera or the web application during dynamic analysis. Just like we mentioned regarding the Veracode platform credentials, there are ways to protect these credentials without completely exposing them in a script. But, even if they were just sitting there in a script, they should have been credentials for a service account with privileges limited to just what Veracode needs to do its job - not a ‘super admin’ account with unlimited privileges across multiple systems.
As an aside for some nerdy comic relief, we found a tweet by Mark Curphey which shared the SecurityInfoWatch article, and it was ironically named ‘when appsec tools go wrong’…
It’s still difficult to think that such a script would just be sitting there on a build server publicly exposed to the internet, but we’re going with it until evidence proves otherwise.
Next question - how super were these ‘super admin’ credentials?
What access did these ‘super admin’ credentials actually provide, and how did the threat actor get shells?
The threat actor provided screenshots on Twitter (which have since been removed) and to several media outlets as evidence that access was obtained to the following:
Verkada’s video management web application, which provides customers with access to live and recorded video, and to cameras for remote configuration (for example, adjusting zoom, focus, audio settings, etc.).
Root shells from Verkada cameras installed on customer networks. IP cameras are not much more than little Linux computers. So, having a root shell essentially means that you are logged into a Linux computer with unrestricted privileges, and this computer is connected to the organization’s network. In one case, a screenshot showed a root shell on a Verkada camera connected to Cloudflare’s corporate network. Verkada claimed that audit logs prove that the threat actor did not pivot from cameras and move laterally within customer networks, but this proves it was possible.
In the statements provided to all of the sources we quoted, the threat actor claimed that these ‘super admin’ credentials were ‘hardcoded’ for maintenance (hardcoded credentials are always a big no-no, and this bad coding practice is listed as CWE-798 on Mitre’s ‘Top 25 Most Dangerous Software Weaknesses’.). But where were they hardcoded? Were they hardcoded in Verkada’s web application, or in camera firmware, or worse, both?
At minimum, these credentials seem to have been used for accessing the web application, according to information provided to SecurityInfoWatch:
…We found super admin credentials in a python script on a publicly exposed Veracode Jenkins Plugin on the Verkada server, which allowed us to log in to their web app…
So, the threat actor used the credentials to enter right through Verkada’s front door - the same portal for their web application which Verkada customers would use to access their surveillance systems. This is where access to recordings or live video streams would have been exposed.
As for shell access, the threat actors statements didn’t seem to confirm exactly where that came from, but Verkada’s March 10th security update points to an internal tool which the threat actor also accessed:
We can also confirm that the attackers gained access to a tool that allowed the execution of shell commands on a subset of customer cameras; however we have no evidence at this time that this access was used maliciously against our customers’ networks. All shell commands issued through our internal tool were logged.
We assume that this tool was accessed or obtained from the Jenkins server where the threat actor found the credentials. It’s possible that this tool required authentication, which was granted using the ‘super admin’ credentials that could be baked into firmware running in all of the Verkada cameras, but it’s also possible that the tool shared a different hardcoded credential with all of the cameras.
We also note Verkada’s statement confirming that commands issued through this internal tool were logged, and wonder whether the threat actor used the tool exclusively, or whether the tool was simply used to spawn a reverse shell directly back to the threat actor (which would have been logged), at which point further commands from the threat actor would not have been logged by Verkada’s tool.
Many people overlook the fact that IP cameras are capable computers that contain vulnerable software components (even in their manufacturer’s latest firmware release). Unless they’re airgapped, they are well-positioned pivot points. It’s critically important that these potential trojan horses are properly hardened, protected, and isolated where possible. They are already authorized devices positioned behind perimeter firewalls, providing beach heads ‘behind enemy lines’ from which a threat actor can attack other systems. This is one of many reasons why network isolation and ‘Zero Trust network architecture’ is so important.
Impact to Verkada customers
Verkada stated that the extent of customer data exposure was relatively limited in this incident, but it’s possible that far more exposure has occurred, and gone undetected, for a long time.
It’s important to note that Verkada’s statements to-date, as well as disclosures made by the aforementioned threat actor to Twitter and other media outlets, only confirm the extent of customer information disclosed as a result of this particular incident.
Remember - the threat actor stated that no vulnerabilities or software flaws were exploited, and that hacking wasn’t necessary in order to obtain the ‘exposed’ credentials used to access customer video and information. So, that begs the question - how long have these credentials been exposed, and during that time, who else found them? Have they been used by earlier threat actors? And if so, what they did access, and was that logged?
Verkada, Mandiant, the FBI, the U.S. Department of Health and Human Services (HHS, who are investigating exposure of patient health information), and affected customers are undoubtedly scouring through logs and any historical data they can get their hands on to confirm whether or not unauthorized access has occurred prior to this incident. Even if you are a Verkada customer and were NOT notified of exposure by Verkada, if your surveillance video is protected or considered sensitive, or if cameras were connected to networks from which they could pivot to access sensitive data, performing an access log audit wouldn’t hurt.
As for this incident’s impact, Verkada confirmed that the threat actor accessed the following:
Video and image data from a limited number of cameras from a subset of client organizations
A list of our client account administrators, including names and email addresses. This list did not include passwords or password hashes.
A list of Verkada sales orders. Sales order information is used by our Command system to maintain the license state of our customers. This information was obtained from our Command system and not from other Verkada business systems.
Additionally, the threat actor disclosed screenshots and video clips that came from cameras in Verkada’s customers’ facilities to media outlets (most of which refrained from publishing other than a couple shots from Tesla facilities and one inside a jail), Twitter (removed), and at least one known repository used for disclosing data leaks (seized by the FBI).
Verkada’s response
Given that it looks like Verkada may not have had a formal Product Security function or team prior to this incident, they seem to have done a lot of things right.
The FBI and Swiss authorities acted relatively quickly as well.
Here’s a quick timeline:
March 7th - Attacker gained access to the Jenkins server (according to Verkada)
March 9th -
The first public indications of the incident, including (now removed) Tweets and contact with media
Verkada posted a public notification to customers stating that they learned of the attack and had begun investigating with their (new) CISO
We didn’t know this until the 10th, but Verkada believe they managed to secure customer systems by the afternoon of the 9th
March 10th - Verkada posts everything they know about the incident, including what they believed to have been accessed based on evidence they had at the time. They also mentioned that they retained Mandiant and a law firm to investigate, and contacted the FBI.
March 12th - Verkada posts a ‘100 day plan’ which pretty much describes the standing up of a proper Product Security program, and committing to work toward a SOC 2 report
March 13th - FBI seizes domains used by the threat actor and ‘APT 69420’
March 15th - Swiss authorities raid the threat actor’s home and arrest the threat actor
March 17th - Verkada CEO holds webinars with customers to address questions or concerns, and commits to hold several more of them on a weekly basis
March 18th - Grand jury indicts the threat actor in U.S. federal court
From an outside viewpoint, it looks like Verkada responded quickly and as transparently as their lawyers would likely allow:
They immediately notified their customers and have been keeping them updated.
They immediately engaged law enforcement, outside legal counsel, and Mandiant - one of the most prominent firms in cybersecurity incident response.
They also immediately hired a CISO, Kyle Randolph, who likely recommended most of these quick responsive actions.
According to LinkedIn, the new CISO’s resume includes security roles at big names like Citrix, Adobe, and Twitter. His LinkedIn also mentions that he’s an advisor and investor at Verkada, so it seems likely that he was conveniently tapped for security leadership as a result of this incident.
As previously mentioned, a ‘100 day plan’ was introduced by Verkada’s CEO and new CISO. This plan essentially introduces a comprehensives security program and includes many of the things we would recommend to a client as they build and mature their security capabilities, including:
Governance to ensure that the board, executives, and staff are able to effect security improvements and measure progress. They also promised to build customer-facing data governance tools to give better visibility to account information and audit logs.
Compliance, specifically working toward SOC 2 Type 1 immediately, followed by SOC 2 Type 2.
We’re shaking our heads wondering how Verkada’s enterprise customers’ own security teams allowed the use of a cloud-based application which stores sensitive data like video recordings from a vendor that didn’t have a SOC 2 or ISO 27001 or similar…
Reviewing access management, specifically ‘identifying new ways to better practice the principle of least privilege’, having just learned the hard way that failing to do so never ends well.
Bug bounty program - incentivizing the global community of white hat hackers to break your stuff is one of the best ways to improve security of a web application. We can only hope that the new CISO ensures this takes place in a safe test environment which is identical to (other than keys and creds), but separate from, the production environment.
Penetration testing
Change management and automated testing because as they say, this does reduce the likelihood of vulnerabilities introduced by human error.
Despite Verkada’s failure to practice least privilege, avoid common software weaknesses like hardcoded credentials, and somehow expose privileged credentials to the internet through misconfiguration or otherwise, they’ve promised to make progress in the right direction. We sincerely hope they follow through.