The 1Password Security team is a crew of wonderful characters responsible for security, privacy, and compliance. We have three very high-level objectives:
- to keep customer data safe
- to keep company/employee data safe
- to keep the product safe
And a lot of (slightly) smaller efforts go into meeting those larger goals. One important onging effort is the tool review, which is an in-depth analysis of a proposed app, tool, or service before it’s used internally. Today, I’ll explore why we chose the tool review method, how we perform our reviews, and share a few things we’ve learned along the way.
Why we chose the tool review method
The in-depth review process isn’t the only way to vet software. Some organizations submit a questionnaire to the developer, others choose their apps based upon reviews and industry recommendations. There’s also the blind faith approach (popular and not recommended).
Back when 1Password was a small startup, we’d get excited to try the latest and greatest apps and services — we still do. But we quickly realized the new technology had access to information that was critical to us and our customers, and we needed to make sure we could trust it.
Around the same time, team members from across the organization began to ask questions about the tools and services they wanted to use.
“I want this third-party screenshot program but I don’t know if I can trust it or what to look for.”
As a team, we knew we had to do something. We began to do general checks into the tools people wanted to use. After some trial and error, those general checks have evolved into a full tool review process.
How we perform tool reviews
When someone is interested in an app, tool, or service for themselves or their team, they file an issue in GitLab, our change control management software. Next, a member of the Security team gathers some information — in particular, we want to learn more about the sensitive data the tool will have access to. Once we have those details, we get to work.
We start on the developer’s website. We look at readily-available documentation, like the privacy policy and terms of service. If we’re lucky, they’re easy to find, but often documents are scattered across the website (or websites, in some cases).
We look for a contact email address — preferably in security or engineering — who can provide documents that often aren’t publicly available, like penetration test results and SOC2 reports.
In addition, we search the National Institute of Standards and Technology’s (NIST) vulnerability database and good ol’ Google for past security breaches and general security practices that have been documented about the vendor.
If it’s a server-based product with a cloud storage component, we also run specific tests like SSL Labs’ server test, to help us make an independent judgment of the developer’s security efforts.
We perform threat modelling on the tool to determine what technical controls are available to protect sensitive data — whatever that data may be. We also try to obtain a test or sandboxed account so we can determine best practices for our IT team.
Once we’re satisfied, the tool is approved for use. If the developer or tool doesn’t pass muster, we deny it, explain why, and try to offer a viable alternative.
What we’ve learned
Our tool review process has certainly matured over the years. Along the way, we’ve learned (and are still learning) what’s important.
Ask the right questions
When we first started to conduct tool reviews, the security analysts who completed the reviews also answered security questionnaires submitted by potential customers vetting 1Password for use within their organizations. This intersection permitted us a discovery we might never have made.
Those security questionnaires didn’t ask the right questions.
Often enough, questions were so generic, there was almost no way the answers could help the organizations better understand our security and privacy practices.
One questionnaire we received listed nearly ten different questions about our loading docks. What kind of security lights are installed in the loading dock area? How many cameras are installed around the loading docks? What kind of locks are used?
I feel like Captain Obvious here, but we make apps. We don’t have a warehouse or storage facility (or an office, for the most part). We don’t have loading docks, or anything even remotely similar to loading docks.
Long story short, it’s important to ask relevant questions of the developer or service. If they create security software, perhaps ask specifics about who has access to production data. If it’s a third-party content writing service, ask if they perform background checks on their freelancers.
Perfection is rare
When we examine penetration tests and SOC reports, we look at the overall picture and don’t expect a squeaky-clean result. We’re interested in the severity of any discovered issues and, maybe even more importantly, how the developer responded to those discoveries.
Did the company insist that the severe issue found during the pen test wasn’t a problem? Sometimes developers will take a defensive stance and deny issues instead of promising to fix them. If you see this happen, I’d suggest moving on to another, comparable tool.
Tool reviews for all
We’ve also learned our review method works outside the world of business. Anyone can search (good ol’ Google again) the name of a website/developer and the word “security” to check for past reports.
Privacy policies, by law, are more accessible, so give the developer’s policy a skim before you use or download their product. If you can’t find their privacy policy, that’s typically a red flag.
Make sure the software is configured correctly. Take the time to go through the security settings and turn off (or on) appropriate options.
If the app stores data, consider the value of the information it will have access to. Do a bit of a risk assessment.
We don’t expect the general public to perform deep analysis on every app they download, but it’s important that everyone knows what to look for, especially they purchase software or services from unfamiliar sources.
Final thoughts
We know we aren’t breaking new ground with our tool reviews. Maybe your company investigates potential apps and services the same way. If that’s the case, wonderful!
What makes our reviews special, in my opinion, is the real reason we do them.
From your Netflix password, to my salary, to plans for the next great feature, 1Password is responsible for the protection of countless secrets, yes. And we have objectives and goals and targets, yes. But for the 1Password Security team, it’s about more than responsibility and objectives — it’s about dedication; it’s about passion.
We believe safety comes before convenience. We maintain that everyone has the right to privacy. We value transparency.
It’s a genuine desire to protect our product, our company, and our customers.
Megan Barker
Security Scribbler
Tweet about this post