Certificates, Provisioning Profiles, and Expiration Dates: The Perfect Storm

But alas, that story has already been told. Now it’s time to share all the technical details about all the forces that aligned against us that had us staring up a giant wall of crashing water like George Clooney and Mark Wahlberg.

Prologue: Not All Certificates Are Created Equal

There’s a lot of information to unpack in this post, but before I get started, I’d like to address an assumption I’m seeing far too many people making: that what happened to us was simply an issue of an expired certificate and that all we needed to do was create a new one, just like you do for SSL certificates.

That’s simply not true.

Developer certificates are much different than SSL certificates and serve a very different purpose. Unlike a simple SSL certificate, our developer certificate is used to sign 1Password and needs to be valid during build time. The expiry time of a certificate or provisioning profile should have no impact on whether or not macOS will allow an app to launch or not. An analogy may be helpful here: if you think of the developer certificate as a carton of eggs, and 1Password as a cake, then it is important not to use expired eggs to make the cake. The fact that the eggs may expire a few days after making the cake should have no effect on the cake itself. After all, the cake is already made and delivered.

Jumping out of the galley and back into our developer world, an expired certificate typically doesn’t affect us until the next time we need to do a release, which would have been this week with our next betas. Certificates control our ability to sign new apps. They don’t affect existing released apps.

For example, we have some users still using 1Password 3 for Mac (hey there, if that’s you, you should really consider upgrading to a 1Password membership as soon as possible!). The first release of 1Password 3 was in 2009, around 8 years ago. Assuming a user is happy with 1Password 3, how long should they expect to be able to continue using the software they paid for? The only acceptable answer to that question is: as long as they feel like it.

Obviously there’s plenty of reasons for why a user would want to upgrade to newer versions, but the fact of the matter is that a user shouldn’t be reliant on us to keep providing updated builds of an unmaintained app just to keep it running. Unlike an SSL certificate, this isn’t something we can simply fix from our end. Fixing the issue we ran into this weekend is a matter of creating a new build of the app and having users update to the new version.

Taking a Tour of the Engine Room

To properly understand what happened, let’s take a step back and look at the different parts of this.

In Mac OS X 10.7 Apple introduced Gatekeeper. Gatekeeper is really quite awesome as it gives users control over what software is allowed to run on their system. The default is to allow software from verified and trusted developers: those apps that have been uploaded to the Mac App Store, or those signed with Developer ID certificates made available to the developer by Apple.

Gatekeeper ensures that apps that have been tampered with will refuse to run, and also provides Apple with a way to revoke certain certificates if a developer has been found to be doing harm (i.e. distributing Developer ID signed malware). These simple steps stop a wide variety of attack vectors and we think the world of Apple for having implemented this.

The next layer is the Provisioning Profile. Provisioning Profiles provide information about what the app can do, as well as who can run it. There are certain services on the Mac that require that the app include a Provisioning Profile. In our case, we needed to start using a Provisioning Profile when we added support for unlocking 1Password using Touch ID.

To be clear, Touch ID itself doesn’t necessitate the profile, but in order to unlock your vault we need to store a secret and we choose to store it the OS X keychain. The specific configuration we’re using for that requires declaring that we want access to a specific keychain access group, which needs to be declared in a provisioning profile. The provisioning profile is included in the app bundle and cannot be updated independently of the app.

Next up… XPC. We use XPC to communicate between the 1Password main app and 1Password mini – the little 1Password that runs in your menu bar – and it’s really quite awesome. 1Password mini acts as the brains of the whole operation, and the larger app is mostly just responsible for displaying information. The reason we love XPC so much is because it’s an inter process communication tool that actually provides us the building blocks we need to perform mutual authentication. What this means is that 1Password mini will refuse to communicate with the main app unless it can prove that it’s signed by us. The inverse is true as well.

Storm Clouds Gather

At around 3pm EST on February 18th we started getting reports of failures in 1Password for Mac. Folks were seeing an error appear that 1Password was unable to connect to 1Password mini.

This initial failure occurred due to the fact that the provisioning profile embedded in 1Password mini had an expiration date. Expiration dates seem to be required, and due to the fact that the expiration date elapsed, Gatekeeper decided that 1Password mini was no longer safe to run. We’ve filed a bug with Apple as we feel that this shouldn’t be the case (rdar://30631939 for those of you reading along inside the Mothership).

Only 1Password mini contains the Provisioning Profile as all Touch ID operations happen within that process. This meant that Gatekeeper was deciding that our main 1Password app could launch. Upon launching, 1Password performs its start up sequence which includes asking the system to launch 1Password mini if it’s not already running. When doing so, the system would log the following to the console:

com.apple.xpc.launchd[1] (2BUA8C4S2C.com.agilebits.onepassword4-helper[11038]): Binary is improperly signed.
com.apple.xpc.launchd[1] (2BUA8C4S2C.com.agilebits.onepassword4-helper[11038]): removing service since it exited with consistent failure reason When validating /Applications/1Password 6.app/Contents/Library/LoginItems/2BUA8C4S2C.com.agilebits.onepassword4-helper.app/Contents/MacOS/2BUA8C4S2C.com.agilebits.onepassword4-helper:
Code has restricted entitlements, but the validation of its code signature failed.
Unsatisfied Entitlements:
com.apple.xpc.launchd[1] (com.apple.ReportCrash[11041]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.ReportCrash

The 1Password main app detected the failure and provided an error panel telling the user that it couldn’t connect to mini.

Due to the expired Provisioning Profile, 1Password mini wouldn’t launch. And without mini running, 1Password itself was unable to startup successfully. Both mini and 1Password itself were signed with the same Developer ID certificate. Gatekeeper allowed 1Password to run, but due to the different rules for apps with provisioning profiles, it would not allow mini to run.

As far as we can tell, the only way to correct this problem is to provide a new build of the app with an updated provisioning profile with a new expiration date. Within a few hours we were able to publish a new version which did exactly this. As of 6.5.4, we had an app that users could download and run again.

The Eye Of The Storm

After this initial bout of terror, death defying feats, and mad scrambles we figured the technical portion of this exercise was finished and had begun transitioning into customer support mode; helping allay the fear, uncertainty, and doubt that this event had caused.

Little did we know at the time, we were only in the eye of the storm – the calm center before things would get rough again.

1Password for Mac includes an updater within the app so that users can easily upgrade to the latest versions as they become available. This updater validates downloads before performing the update to ensure that the updated app is in fact from AgileBits. One of the steps taken during validation is looking at the code signature of the downloaded app and ensuring that it satisfies the following security requirement:

anchor apple generic and identifier com.agilebits.onepassword4 and certificate leaf[subject.CN] = “Developer ID Application: Agilebits Inc.”

This check has worked really well for us. It’s simple and does the trick.

This check is also extremely specific about the common name2 it looks for. When we generated our updated provisioning profile we also needed to generate a new Developer ID certificate. We didn’t realize it at the time, but the common name of newly created certificates now include the team identifier in addition to the company name; “Developer ID Application: AgileBits Inc. (2BUA8C4S2C)” vs. “Developer ID Application: AgileBits Inc.”. Close. Super close. But we weren’t looking for a “close” match.

The result of this new common name was that even though our app would now launch, the automatic updater would never run successfully because as far as it was concerned the update being provided wasn’t valid and therefore needed to be rejected. This is what users who could still run 6.5.3 and tried to update to 6.5.4 saw.

Once we discovered this problem we had no choice but to pull the 6.5.4 update and issue a 6.5.5 update that included a modified security requirement check. Sadly this didn’t address the fact that users running 6.5.3 and earlier are not able to automatically update to 6.5.5.

Moving Forward and Heading Home

This was painful for everyone. We lost sleep over the weekend, but worse than that… our users temporarily lost access to some of their most important information. This is unacceptable to us and we want to make sure this doesn’t happen again.

We’ve reached out to Apple for help and guidance on what we can do to avoid this happening again in the future. Our new provisioning profile doesn’t expire until 2022, but we’ll make sure that this is resolved far before then so that you need not worry about that happening.

If you’re a developer of a Developer ID signed app, we recommend that you check to see if your app includes a provisioning profile. Since that’s mostly handled automatically by Xcode, it’s likely that there are apps out there whose developers aren’t even aware of the inclusion of the provisioning profile. Check the expiration date, and ensure that you release an updated build with an updated provisioning profile well before the expiration date is hit so your users have time to update.

We’ve also filed an enhancement request with Apple asking that developers be notified via email of impending distribution certificate or provisioning profile expirations with explanations of repercussions. This was filed as rdar://30631968.

If you have questions about any of this, please don’t hesitate to ask us in the comments below.

Love, The 1Password Mac Team

P.S. Happy 5th Birthday to Gatekeeper! ? We were one of the first apps to sign with Developer ID certificates, use XPC, and leverage the entitlements required for Touch ID. It’s always exciting being on the cutting edge of technology but we wouldn’t have it any other way. ?

Jeffrey Goldberg

Principal Security Architect