Who do you trust to tell you who to trust?

Who’s in immediate danger?

Some seemingly secure communication over SSL between computer users in Iran and google.com domains is being intercepted and read by a third party, presumably the government of Iran.

Any readers connecting to Gmail (or have other sensitive communication with google.com domains) from networks in Iran, update all your browsers immediately, only use Safari if you follow the instructions below, and pay extremely close attention to warnings from web browsers about certificates, and once you have a trustworthy connection to Google, change your Gmail passwords.

Not a Gmail compromise

It is important to note that although communication between users and Gmail has been compromised, this was not an actual break-in at Gmail. Nor is it a failure of the web browsers people use, nor is it a failure of the cryptography involved. Instead it is a failure of the “trust” infrastructure that we all rely on.

This matters to everyone

The demonstration of how the trust infrastructure can fail matters for everyone, even if they are not using Gmail from Iran. DigiNotar has acknowledged that there may be other, undetected, fraudulent certificates out there as well. Although this issue doesn’t tie in to any of our products at AgileBits, I wanted to help people understand what is going on and why it matters. I’ve actually been surprised this hasn’t been bigger news earlier.

The Trust Infrastructure of Secure (HTTPS) Web Browsing

Browsing the web using SSL/TLS (that is where the URL begins with “https://”) does two things for you. It makes sure that the communication between your web browser and the web server it is talking to is very well encrypted. The other thing that is does is give you assurance that the site you are talking to is the site that it claims to be from its network address.

If someone could trick your browser or your bit of the Internet to go to a their own site when you connect to agilebits.com, then they could create a malicious version of 1Password that you might download. But because we use SSL for crucial parts of our downloads and for our updating processes, you are assured that it really is us you are talking to.

If you visit https://agilebits.com using Safari and click on the little lock icon in the upper right corner, you will see something that looks like the figure on the right. This tells you that the people claiming to be agilebits.com (that’s us!) have proved ourselves sufficiently for PositiveSSL to certify that our website certificate really does belong to us. PositiveSSL in turn has been checked out by User Trust Network (UTN) which has said that PositiveSSL can be trusted to issue certificates like the kind that they issued to us.

Now why should be trust UTN to make such declarations? They are at a root of this chain of trust, so who has checked them out to make sure that they only issue certificates that should be issued? The answer is that the people who built your web browser have made that decision. UTN is one of a couple hundred or so root Certification Authorities (CA) that are trusted by most web browsers and operating systems.

Through the magic of Public Key Cryptography (a topic for another day), these certificates are impossible to counterfeit without knowing a secret key used in the creation of the certificates. When someone at PositiveSSL creates a certificate, they have to enter a password that unlocks a particular key that is used in signing a certificate.

Man in the Middle

If someone, let’s call her Ewa, can get a certificate that says that she is google.com in a way that a user, let’s call him Azada, would trust that certificate then Ewa can intercept communication between Azada and Gmail even though that communication is encrypted. What Ewa does is pretend to be Gmail when talking to Azada, and at the same time she pretends to be Azada when talking to Gmail. When Azada thinks he is sending his username and password to Gmail, Ewa will get that username and password. Ewa will quickly re-encrypt that username and password in her communication with Gmail. So both Azada and Gmail see that they have an encrypted connection, and they think that they are talking directly to each other. But Ewa is our man in the middle. (Please note that “Man in the Middle” attacks are distinct from “Meet in the Middle” attacks even though they share the same abbreviation, MITM.)

What happened to DigiNotar?

DigiNotar is another root certification authority. On July 9 their systems were compromised. Hundreds of certificates were created by those who broke in. DigiNotar detected the break-in and revoked most, but not all, of the certificates generated by the attackers. For reasons that aren’t yet clear, they failed to issue a revocation for the google.com certificate. That’s the certificate that has been used for Man In The Middle attacks in Iran. It is also worth noting that DigiNotar did not notify anyone about the break-in at the time.

Certificate revocations are tricky thing. Browsers don’t always fetch the latest Certificate Revocation Lists (CRLs). Typically a revocation is issued when a site loses its certificate or when a business changes its name so is no longer meets the conditions under which the certificate was issued. (Our name change from Agile Web Solutions to AgileBits has played havoc with certificates for our old domain names for exactly this reason.) On problem with certificate revocations is that they don’t tell us whether the actual CA was compromised, as happened in the DigiNotar case.

No one outside of DigiNotar or the attackers knew that a CA had been compromised until some problems were reported in Iran in the last few days of August. Only since then has DigiNotar and their parent company Vasco acknowledged that there had been a breach and a large number (more than 500) bogus certificates issued. Keep in mind that because DigiNotar is a trusted root CA, they can issue certificates for any domain possible, not just for their customers. In practice DigiNotar is a small provider, but because they are (well actually, had been) trusted by browsers fraudulent certificates could be issued for any site.

What’s happened since

This is an on-going story. But Microsoft, Google, and Mozilla have removed DigiNotar from the list of trusted root certification authorities in updates to Internet Explorer, Chrome, and Firefox respectively.

What this means is that if you visit a site that actually uses a certificate signed by DigiNotar using those browsers your browser will give you a warning. Customers of DigiNotar are scrambling to get their certificates signed by other certification authorities, but this will take some time. After all a CA needs to verify that the person making the request for a certificate really is who they say they are.

This should not be a practical problem for those who don’t browse sites based in The Netherlands. But the Dutch government has many sites whose certificates have been signed by DigiNotar.

Untrusting DigiNotar on the Mac

[Update: September 9: Use Software Update on OS X 10.7.1 or OS X 10.6.8 to get Apple’s Security Update 2011-005. The remainder of this section on untrusting DigiNotar is moot in light of this security update from Apple.]

Apple has not (yet) removed DigiNotar from its list of trusted certification authorities, but you can do this yourself if you wish. Unless you are in Iran there is no real security gain from doing so (unless of course other fraudulent DigiNotar certificates emerge). However there is little harm in doing so unless you visit Dutch government websites.

The certificates are in a system keychain on OS X; so you will need to use Keychain Access.app to make changes.

Keychain Access.app is in the Utilities folder in the system Applications folder. Once you have launched Keychain Access, you need to

1. Select “Certificates” in the lower left panel of Keychain Access
2. Search for “diginotar” using the search field
3. Double click on DigiNotar Root CA in the main panel

After you double click on DigiNotar you will be presented with a window that allows you to change the trust relationship you have with that certificate.

You will need to click on the expansion triangle by “Trust” and there you will be presented with the ability to change “When using this certificate” from “Use System Defaults” to “Never Trust”. You will be prompted for an administrator username and password to make this kind of change. You will also need to log out of OS X and back in again before the change takes effect.

[Update September 8: Because of a bug in Apple’s trust system, some certificates (EV certificates) will be trusted even if the signing certificate is “untrusted”. Therefore it is necessary to actually remove the DigiNotar certificate instead of merely setting trust to “Never”. TUAW has a nice set of instructions for removing the certificate.

Unfortunately, even removing the DigiNotar root CA certificate doesn’t entirely eliminate the problem on the Mac (though it does work better than merely untrusting the certificate).

Again, let me stress that this is of no practical importance unless you have reason to suspect that you are on a network which where entities are trying to use fraudulent certificates to spy on you.

On the other hand, I have “de-trusted” DigiNotar, not so much for any added sense of security but because I feel that DigiNotar failed to live up to the absolutely enormous responsibility it undertook when becoming a root CA. For all we know, they just have suffered from bad luck. Maybe other certification authorities are just as vulnerable. But there is immense power in being a root CA trusted by browsers. DigiNotar’s failure to disclose the breach and their failure to revoke all of the bad certificates means to me that they shouldn’t be trusted with that power.

Lessons

Pay attention to browser warnings

Man in the middle attacks are real, not just theoretical. If people routinely ignore the warnings, that MITM attacks will become more common. In this case, it was a vigilant user who noticed an inconsistency and raised the alarm by asking questions. One concern about browsers no longer trusting DigiNotar certificates is that users of those sites will “learn” to ignore warnings. People need to consider this in the decision to remove DigiNotar from the list of trusted root authorities.

Just as you shouldn’t ignore SSL error messages about untrusted certificates, you shouldn’t panic every time you one. There are lots of ways for things to be misconfigured that lead to these errors. Instead you should try to investigate by other means whether the site really is who they say it is. Report the error to the operators of the site. (Before you all report to us that agile.ws comes up with an error, we know. The certificate was revoked when we changed our business name. Just use our agilebits.com domain.)

The doomsayers may have been right

When the whole trust structure for SSL was devised, there were many people who worried that it gave too much power to certification authorities. In this instance we had one that suffered a security breach, but imagine if there were a corrupt one. With hundreds of trusted certification authorities, each with the power to issue certificates for any domain, the scope for abuse is substantial. I was one of those worriers.

It is easy to say that I don’t like the system, it is much much harder to present an alternative that works better and doesn’t burden users with the task of performing their own audits of certificates or authorities. There are a number of proposals out there, and discussion of them has certainly kicked off again over the past few days. My post here has already been long enough, so I will redirect readers to a post by Mike Caldwell who proposes an idea I haven’t seen before (as well as linking to other proposals).

Browsers need to take revocations more seriously

I mentioned above that revocation lists are a tricky process. A browser isn’t going to fetch these lists every time it connects to a site. And so so certificate revocation doesn’t propagate to users immediately. From what we’ve seen over the past week, Safari on the Mac appears to be the laggard in updating from CRLs. I expect that every browser maker is looking closely at the handing of certificate revocation.

The impending death of DigiNotar shows the system working

DigiNotar’s days as a CA are numbered. Their customers will be extremely angry with them, and it is unclear how they can survive, although their parent company, Vasco, might. This will be a lesson not just to everyone else in the business, but to potential customers. Customers will want to know that a CA’s security is strong enough that they will not have to worry about them becoming untrusted. This lesson may seem to contradict the one above, but we need to look at all angles.

Jeffrey Goldberg

Principal Security Architect