Home > Security > These Are Not The Certs You’re Looking For

These Are Not The Certs You’re Looking For

It began near Africa.

This is not, of course, the *.google.com certificate, seen in Iran, signed by DigiNotar, and announced on August 29th with some fanfare.

This certificate is for Facebook. It was seen in Syria, signed by Verisign (now Symantec), and was routed to me out of the blue by a small group of interesting people — including Stephan Urbach, “The Doctor”, and Meredith Patterson — on August 25th, four days prior.

It wasn’t the first time this year that we’d looked at a mysterious certificate found in the Syrian wild. In May, the Electronic Frontier Foundation reported that it “received several reports from Syrian users who spotted SSL errors when trying to access Facebook over HTTPS.” But this certificate would yield no errors, as it was legitimately signed by Verisign (now Symantec).

But Facebook didn’t use Verisign (now Symantec). From the States, I witnessed Facebook with a DigiCert cert. So too did Meredith in Belgium, with a third strike from Stephan in Belgium: Facebook’s CA was clearly DigiCert.

The SSL Observatory dataset, in all its thirty five gigabytes of glorious detail, agreed completely. With the exception of one Equifax certificate, Facebook was clearly identifiable by DigiCert alone.

And then Meredith found the final nail:

VeriSign intermediary certificate currently used for VeriSign Global certificates. Chained with VeriSign Class 3 Public Primary Certification Authority. End of use in May 2009.

So here we were, looking at a Facebook certificate that couldn’t be found anywhere on the net — by any of us, or by the SSL Observatory — signed by an authority that apparently was never used by the site, found in a country that had recently been found interfering with Facebook communication, using a private key that had been decommissioned for years. Could things get any sketchier?

Well, they could certainly get weirder. That certificate that was killed in 2009, was the certificate killed by us! That’s the MD2-based certificate that Meredith, myself, and the late Len Sassaman warned about in our Black Hat 2009 talk (Slides / Paper)! Somehow, the one cryptographic function that we actually killed before it could do any harm (at the CA, in the browser, even via RFC) was linked to a Facebook certificate issued on July 13th, 2011, two years after its death?

This wasn’t just an apparently false certificate. This was overwhelmingly, almost cartoonishly bogus. Something very strange was happening in Syria.

Facebook was testing some new certificates on one of their load balancers.


I’ve got a few friends at Facebook. So, with the consent of everyone else involved, I pinged them — “Likely Spoofed Facebook Cert in Syria”. Thought it would get their attention.

It did.

We’re safe.

Turns out we just ordered new (smaller) certs from Verisign and deployed them to a few test LBs on Monday night. We’re currently serving a mixture of digicert and verisign certificates.

One of our test LBs with the new cert: a.b.c.d:443

And thus, what could have been a flash of sound and fury, ultimately signifying nothing, was silenced. There was no attack. Facebook had the private key matching the cert — something not even Verisign would have had, in the case of an actual Syrian break.

In traditional security fashion, we dropped our failed investigation and made no public comment about it — perhaps it’d be a funny story for a future conference, but it was certainly nothing to discuss with urgency.

And then, four days later, the DigiNotar break of *.google.com went public. Totally unrelated, and obviously in progress while our false alarm was ringing. The fates are strange sometimes.


The only entity that can truly assert the cryptographic keys of Facebook, is Facebook. We, as outside observers with direct expertise, detected practically every possible sign that this particular Facebook cert was malicious. It wasn’t. It just wasn’t. That such a comically incorrect certificate might actually be real, has genuine design consequences.

It means you have to put an error screen, with bypass, up for the user. No matter what the outside observers say, they may very well be wrong. The user must be given an override. And the data on what users do, when given an SSL error screen, isn’t remotely subtle. Moxie Marlinspike, probably the smartest guy in the world right now on SSL issues, did a study a few years ago on how many Tor users — not even regular users, but Tor users, clearly concerned about their privacy and possessed with some advanced level of expertise — would notice SSL being disabled and refuse to browse their desired content.

Moxie didn’t find a single user who resisted disabled security. His tool, sslsniff, worked against 100% of the sample set.

To be fair, Moxie’s tool didn’t pop an error — it suppressed security entirely. And, who knows, maybe Tor users think they’ve “done enough” for security and just consider their security budget spent after that. Like most things in security, our data is awful.

But, you know, this UI data ain’t exactly encouraging.

Recently, Moxie’s been presenting some interesting technology, in the form of his Convergence platform. There’s a lot to be said about Convergence, which frankly, I don’t have time to write about right now. (Disclosure: I’m going to Burning Man in about twelve hours. Honestly, I was hoping to postpone this entire discussion until N00ter’s first batch of data was ready. Ah well.) But, to be clear, I was fooled, Meredith was fooled, Moxie would have been fooled…I mean, look at this certificate! How could it possibly be right?


It is true that the only entity that can truly assert being Facebook, is Facebook. Does that mean everybody should just host self-signed certificates, where they assert themselves directly? Well, no. It doesn’t scale, and just ends up repeatedly asking the user whether a key (first seen, or changed) should be trusted.

Whether a key should be trusted, is an answer to which users only reply yes. Nice for blaming users, I suppose, but it doesn’t move the ball.

What’s better is a way to go from users having to trust everybody, to users having to trust only a small subset of organizations through which they have a business relationship with. Then, that subset of organization can enter into business relationships with the hordes of entities that need their entities asserted…

…and we just reinvented Certificate Authorities.

There’s a lot to say about CA’s, but one thing my friend at Verisign (there were a number of friendly mails exchanged) did point out is that the Facebook certificate passed a check via OCSP — the Online Certificate Status Protocol. Designed because CRLs just do not scale, OCSP allows a browser to quickly determine whether a given certificate is valid or not.

(Not quickly enough — browsers only enforce OCSP checks under the rules of Extended Validation, and *maybe* not even then. I’ve heard rumors.)

CA’s are special, in that they have an actual business relationship with the named parties. For a brief moment, I was excited — perhaps a way for everybody to know if a particular cert was valid, was to ask the certificate authority that issued it! Maybe even we could do something short of the “Internet Death Penalty” for certificates — browsers could be forced to check OCSP before accepting any certificate from a questioned authority.

Alas, there was a problem — and not just “the only value people are adding is republishing the data from the CA”. No, this concept doesn’t work at all, because OCSP assumes a CA never loses control of its keys. I repeat, the system in place to handle a CA losing its keys, assumes the CA never loses the keys.

Look at this.

$ openssl ocsp -issuer vrsn_issuer_cert.pem -cert Cert.pem -url http://ocsp.verisign.com -resp_text
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
Version: 1 (0x0)
Responder Id: O = VeriSign Trust Network, OU = “VeriSign, Inc.”, OU = VeriSign International Server OCSP Responder – Class 3, OU = Terms of use at http://www.verisign.com/rpa (c)03
Produced At: Aug 30 09:18:44 2011 GMT
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: C0FE0278FC99188891B3F212E9C7E1B21AB7BFC0
Issuer Key Hash: 0DFC1DF0A9E0F01CE7F2B213177E6F8D157CD4F6
Serial Number: 092197E1C0E9CD03DA35243656108681
Cert Status: good
This Update: Aug 30 09:18:44 2011 GMT
Next Update: Sep 6 09:18:44 2011 GMT

Signature Algorithm: sha1WithRSAEncryption
Version: 3 (0x2)
Serial Number:
Signature Algorithm: sha1WithRSAEncryption
Issuer: O=VeriSign Trust Network, OU=VeriSign, Inc., OU=VeriSign International Server CA – Class 3, OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
Not Before: Aug 4 00:00:00 2011 GMT
Not After : Nov 2 23:59:59 2011 GMT
Subject: O=VeriSign Trust Network, OU=VeriSign, Inc., OU=VeriSign International Server OCSP Responder – Class 3, OU=Terms of use at http://www.verisign.com/rpa (c)03
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
X509v3 Certificate Policies:
Policy: 2.16.840.1.113733.
CPS: https://www.verisign.com/CPS
User Notice:
Organization: VeriSign, Inc.
Number: 1
Explicit Text: VeriSign’s CPS incorp. by reference liab. ltd. (c)97 VeriSign

X509v3 Extended Key Usage:
OCSP Signing
X509v3 Key Usage:
Digital Signature
OCSP No Check:

X509v3 Subject Alternative Name:
Signature Algorithm: sha1WithRSAEncryption
Cert.pem: good
This Update: Aug 30 09:18:44 2011 GMT
Next Update: Sep 6 09:18:44 2011 GMT

I’ve spent years saying X.509 is broken, but seriously, this is it for me. I copy this in full because the full reply needs to be parsed, and I don’t see this as dropping 0day because it’s very clearly an intentional part of the design. Here is what OCSP asserts:

Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: C0FE0278FC99188891B3F212E9C7E1B21AB7BFC0
Issuer Key Hash: 0DFC1DF0A9E0F01CE7F2B213177E6F8D157CD4F6
Serial Number: 092197E1C0E9CD03DA35243656108681
Cert Status: good

To translate: “A certificate with the serial number 092197E1C0E9CD03DA35243656108681, using sha1, signed by a name with the hash of C0FE0278FC99188891B3F212E9C7E1B21AB7BFC0, using a key with the hash of 0DFC1DF0A9E0F01CE7F2B213177E6F8D157CD4F6, is good.”

To translate even further, please imagine a scene from one of my favorite shows, the IT crowd.

Roy: Moss…did you…did you sign this certificate?
Moss: I did indeed sign a certificate…with that serial number.
Roy: Yes, but did you sign this particular certificate?
Moss: Yes, I did indeed sign a particular certificate…with that serial number.
Roy: Moss? Did YOU sign, with YOUR key, this particular certificate?
Moss: Roy. I DID sign, with MY key, a certificate….with that serial number.
Roy: Yes, but what if the certificate you signed isn’t the certificate that I have? What if your certificate is for Moe’s Doner Shack, and mine is for *.google.com?
Moss: I would never sign two certificates with the same serial number.
Roy: What if you did?
Moss: I would never sign two certificates with the same serial number.
Roy: What if somebody else did, got your keys perhaps?
Moss: I would never sign two certificates with the same serial number.
Roy: Moss?
Moss: I would never sign two certificates with the same serial number.

Long and short of it is that OCSP doesn’t tell you if a certificate is good, it just tells you if the CA thinks there’s a good certificate out there with that serial number. An attacker who can control the serial number — either by compromising the infrastructure, or just getting the keys — wins.

(This is very similar to the issue Kevin S. McArthur (and Marsh Ray?) and I were musing about on Twitter a few months back during Comodogate, in which it became clear that CRL’s also only secure serial numbers. The difference is that CRL’s are an ancient technology that we knew didn’t work, while OCSP was designed with full awareness of threats and really went out of its way to avoid hashing the parts of the certificate that actually mean something, like Subject Name and whether the certificate is a God Mode Intermediate.)

Now, that’s not to say that another certificate validation endpoint couldn’t be exposed by the CAs, one that actually reported whether a particular certificate was valid. The problem is that CAs can lie — they can claim the existence of a business relationship, when none exists. The CA’s I deal with are actually quite upstanding and hard working — let me call out Comodo and say unequivocally that they took a lot of business risk for doing the right thing, and it couldn’t have been an easy decision to make. The call to push for a browser patch, and not to pretend that browser revocation checking was meaningful, was a pretty big deal. But with over 1500 known entities with the ability to declare certificates for everyone, the trust level is unfortunately degraded.

1500 is too big. We need fewer, and more importantly, we need to reward those that implement better products. We need a situation where there is at least a chain of business (or governmental — we’ll talk about this more next week) relationships between the identity we trust, and the entities that help us recognize that trust — and that those outside that trust (DigiNotar) can be as easily excluded as those inside the trust (Verisign/Symantec) are included.

DNSSEC doesn’t do everything. Specifically, it can’t assert brands, which are different things entirely from domain names. But the degree to which it ties *business relationships* to *key management* is unparalleled — there’s a reason X.509 certificates, for all the gunk they coded into Subject Names, ended up running straight to DNS names as “Common Names” the moment they needed to scale. That’s what every other application does, when it needs to work outside the org! From the registrar that inserts a domain into DNS, to the registry that hosts only the top portion of the namespace, to the one root which is guarded from so much technical and bureaucratic influence, DNSSEC aligns trust with extant relationships.

And, in no uncertain terms, when DNSSEC tells you a certificate is valid, it won’t just be talking about the serial number.

More later. For now, Burning Man. ISPs, a reminder…I’m serious, I genuinely would prefer N00ter not find anything problematic.


Curious how Facebook could have a certificate that chained to a dead cert signed with a broken hashing function? So was I, until I remembered — roots within browsers are not trusted because of their self signature (best I can tell, just a cheap trick to make ASN.1 parsers happy). They are trusted because the code is made to explicitly trust them. The public key in a certificate otherwise self-signed by MD2 is no less capable of signing SHA-1 blobs as well, which is what’s happening in the Facebook cert. If you want to see the MD2 kill actually breaking stuff, see this post by the brilliant Eric Lawrence.

It IS a little weird, but man, that’s ops. Ops is always a little weird.

Categories: Security
  1. August 31, 2011 at 2:39 pm

    I linked to it on twitter, but also worth reading Jeff Ferland’s security stack exchange blog post as well for a summary of the position – http://security.blogoverflow.com/2011/08/31/a-risk-based-look-at-fixing-the-certificate-authority-problem/

  2. August 31, 2011 at 4:48 pm

    that OCSP design flaw is hilarious. grats on finding it.

    thought about Convergence that this article prompted:

    the traditional CA model is — and Moxie explicitly acknowledged this in his BH/DefCon talk — a special case of Convergence with a single notary on localhost which verifies the cert chain back to a root CA.

    the DNSSEC model is a special case of Convergence with a single notary which verifies the cert back to the IANA root key signing key.

    framing the two traditional trust models for the Web in this way lets you think about Convergence as a combination of a) a voting system, b) a set of “non-traditional trust metrics” like the Perspectives algorithm, and c) the UI that presents the Convergence controls and decision to the user, instead of a whole new trust system.

    (for what it’s worth, I think c) is going to be the limiting factor in the whole system.)

  3. September 1, 2011 at 12:37 pm

    Very interesting.

    One query though. Are you sure about this?
    “Facebook had the private key matching the cert — something not even Verisign would have had, in the case of an actual Syrian break.”

    Last time I looked Verisign wouldn’t sign my public key via a CSR, they wanted to generate the public and private key themselves. This means they, and anybody they trust, has your private key.

    • February 26, 2012 at 8:55 pm

      You sure about this?

      • February 27, 2012 at 9:26 pm

        I was sure 5 years ago when I came across this. I was using a company provided login to verisign, and was very surprised when I found this. But now, of course, I can’t find any evidence now, and the basic verisign docs all mention providing a CSR. Sorry for the false alarm.

  4. pulo
    September 1, 2011 at 1:24 pm

    Great article and I learned a lot reading it. One point I’m not sure I get, and I’m just trying to learn here: ” this concept doesn’t work at all, because OCSP assumes a CA never loses control of its keys.” If it becomes known that the CA is compromised, then the CA cert is revoked and the OCSP responder should check the whole chain? Assuming the CA key compromise is detected and the CA cert revoked, this should work? Of course, if the CA keys are compromised and it is not detected, then there is a big problem, and in any case there is certainly a timelag anyway. I suppose a lot depends on the implementation of the OCSP responder and its trust chain – if its keys are signed by the CA then, well. Is this the problem?

  5. Kyle
    September 1, 2011 at 1:39 pm

    Does seem simple enough to hold a database of all the serials ever signed by a given CA, and force a check before signing a new cert.

    • Kyle
      September 1, 2011 at 1:41 pm

      though that doesn’t cover all existing certificates…

  6. Matt
    September 1, 2011 at 3:38 pm

    I think SCVP might be closer to what you were wanting from OCSP.

  7. September 6, 2011 at 9:53 pm

    I can so see that scene from IT crowd.

  8. Lennie
    September 7, 2011 at 2:03 pm

    At first I thought Convergence would just be like Perspectives, but it allows for more tricks and flexibility. It has relaying for privacy and you could use Convergence to check DNSSEC even on ‘Hotel networks’ and other situations where DNSSEC is not supported directly. So it could ‘fix’ (as an indirect fallback) the compatibility problems people might otherwise face.

    I really hope the DigiNotar hack does shake up some people. It really is bad. First thing I would do is add more requirements to the already long list of requirements that the CAB-Forum (?).

    As I understand it the CAB-forum has a lot of rules and requirements. But what is mostly checked at an audit is the procedures and paperwork.

    The implementation setup of DigiNotar was a big mess.

    Regular penetration testing of all related webproperties of every CA and sub-CA and reseller/partner (Instantssl.it) which has automated the request-procedures.

    Including an audit like Fox-It did at DigiNotar (which looked at everything which is not directly exposed to the Internet as well).

    All breaches of security should be reported to atleast the browser vendors. DigiNotar was aware of problems for weeks before anyone from outside of the company noticed.

    I hope N00ter does find them all, if they are doing bad things and no false positives.

    Also I wonder if it is possible to combine it with Convergence to help spot man-in-the-middle-attacks. 😉

  9. Anon
    September 8, 2011 at 8:29 am

    Yeah I noticed this too (because I use Firefox with the certificate patrol plugin). Even if facebook says they are using it in their load balancer it all still seems very dodgy – what in the world is Verisign doing giving them such a cert? If it really is Verisign :).
    BTW one annoyance with Firefox on Windows is it doesn’t make it easy to copy and paste cert fingerprints and other info.

  10. Peter Gutmann
    September 10, 2011 at 8:05 am

    OCSP is actually much, much worse than you describe. The status values are, as you point out, broken. Even if you fix that (as some CAs have proposed, after being surprised to find out how OCSP really worked – yes, some of the folks charged with running OCSP don’t actually know how it really works) it doesn’t help, given OCSP’s broken IDs an attacker can trivially work around this. And if you fix those, given the replay-attack-enabled “high-performance” optimisation an attacker can work around that. And if you fix that, given that half the response is unauthenticated, an attacker can go for that. To paraphrase Lucky Green, OCSP is multiple-redundant broken, by design. If you remove the bits that don’t work (the response status, the cert ID, nonces, and the unauthenticated portions of the response) there is literally nothing left. There’s an empty ASN.1 shell with no actual content. There is not one single bit of OCSP that actually works as it’s supposed to (or at least “as a reasonable person would expect it to”, since technically it does work exactly as the spec says it should).

  11. Jagannath
    October 14, 2011 at 8:01 am

    the certificate is Verisign when using Chrome & Digicert when using Internet Explorer

  12. Abhishek
    January 5, 2012 at 2:21 pm

    Hm. I encountered almost the exact same problem with FB here in India and was redirected to your blog after raising the question on stackexchange.

    The difference being that the certificate looked good in the browser. Downloaded however was a different story; the downloaded certificate showed an intermediary Verisign CPS certificate that expired in 2004!!

    Strangely enough, reconnecting to the internet my browser presented the https in green, but the download still showed an expired cert. Oh well …

  1. September 7, 2011 at 4:19 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: