A Tale of Two Certs

November 14, 2015 at 17:42:32 Rainer Brockerhoff 6 comments

I’m keeping this post updated as details develop…

About ten days ago, something strange happened on my Mac: I was debugging the next version of my RB App Checker Lite app and suddenly I saw the dreaded dialog box: Damaged

Completely abnormal, especially as I was debugging using the Developer ID version (not the Mac App Store version!) from inside Xcode. When I opened Terminal, the same dialog; when I opened Safari, same thing! No new process was allowed to run. Of course I had to reboot to be able to do anything, everything worked fine afterwards, and I couldn’t reproduce the problem, so…

OK, a couple of days ago I concluded all was ready and I uploaded my app for review. A few hours after I announced so on Twitter, the reports began appear: the sky is falling! Major Mac App Store meltdown, everybody was getting the “damaged” dialog, Apple’s certificates were the culprit. I started testing my local apps from the MAS and, sure enough, the MAS leaf cert had expired; no problems, some of them asked anew for the AppleID password, some didn’t. RB App Checker Lite showed the expiration but no other problems, but I pulled it from review just in case.

Two days of confusion and frantic coding later, I had submitted (and pulled!) 4 more builds until I was reasonably sure that everything was working correctly. Thanks to several fellow developers on Twitter, the upcoming version seems to show everything correctly; it turned out that my receipt checks were somewhat obsolete. I usually publish the direct download version only after the MAS version has passed review, but decided to release version 1.1.4, build 351 immediately: you can get it here. It has a long list of improvements and fixes.

Meanwhile, the consensus is that rebooting and re-entering the AppleID and passwords (or even deleting and reinstalling) the affected apps solves 99% of the problems.

There are actually several different unfortunate problems here. First, the “damaged” dialog seems to be caused by some sort of cache or memory corruption in the system processes that coordinate to implement GateKeeper and the app store updates; some reports say killing the “storeagentd” process solves this problem without rebooting. (My system doesn’t seem to run this, FWIW.) What not everyone knows is that this dialog appears before the app it allowed to run; that is, it’s not affected by any checking done inside the app itself!

Second, asking for a new AppleID password. This is caused by the app itself checking the store receipt; something strongly recommended by Apple, since otherwise, it’s easy to copy a downloaded app to another computer and having it run there; I remember some early games not doing this and being widely pirated.

When an app is downloaded from the MAS, a proper receipt for that AppleID and that computer is already inside. A missing or corrupted receipt is the only normal circumstance in which the “damaged” dialog should appear. But if you copy the app to another computer, this will be noticed by the app itself.

Once a MAS app starts up, the first thing it should do is to check the receipt. It’s a complex process and not everybody implements it the same way. At first, checking the receipt’s cert chain would cause the receipt to be rejected in the case of expiration; the app exits with a special numeric code (exit 173) and this code signals the system to put up the dialog asking to confirm the purchaser’s AppleID and password. This, in turn, will cause a new receipt to be downloaded, and the app can now run with no problems. Update: reports indicate that, in at least some cases, the system doesn’t respond properly to exit 173.

A few years ago receipts began to include a new field containing the receipt’s creation date, and developers now had to check the certs against that date (and not against the current date), therefore obviating the need to reenter the password. Unfortunately this was not widely divulged, and Apple’s own sample code hasn’t yet been updated accordingly; I confess to not seeing this myself!

As is usual in disasters, several things have to go wrong at the same time: some bug corrupts a critical system cache, certificates expire normally, some apps incorrectly test for expiration, receipts are corrupted or the AppleID validation servers become slow or unreachable (because of the huge number of simultaneous requests), and… boom.

Many articles, unfortunately, published factual errors or wrong assumptions.Let’s try to counter a few:

Apple “allowed” their Mac App Store certificate to expire. Wrong on several levels. First, there’s not one but 5 (!) certificates involved in any app from the store: Apple’s root certificate: and 4 others: two intermediate and two leaf certificates.
The way these certs work is by so-called certificate chains; every cert vouches for the lower-level ones. At the top is Apple’s Root certificate, which is one of a hundred or so in the System Keychain. There are two different certificate chains in every MAS app; the first is used in the code signature:and the second is used to sign the store receipt:Note the expired certificate there? This is a leaf certificate. These, usually, have a short life — one or two years — and the intermediate certificates usually last a little longer.
So, when a cert expires, is that a serious problem? No – unless it is the root cert, which is why they all expire somewhen in the 2030s — hopefully, by that time, they’ll have figured out something better, Apple will have updated the cert via Software Update, or the horse will have learned to sing.
The root cert can be updated via Software Update because it’s stored in System Keychain — but it’s impractical to push cert updates to each and every signed app, bundle or library; there are many thousands of them! So an expired cert in the code signature doesn’t affect the app at all. What’s important is that the certs were valid when the app was signed. When and if you get a new version of the app, all certs will probably be new ones. So there’s no “allowing” a leaf cert to expire — they do so naturally.
Apple “pushed” a new certificate that expires in 2035. This is probably just looking in the wrong place — not knowing which certificate had expired, someone glanced at the root certificate and noticed the “new” 2035 date. Nothing new to see, of course; that cert was created in 2006! Even more confusingly, someone else deduced from that that Apple let their original root cert expire; also wrong.
The system hasn’t been updated to check SHA2 (256) certificates. Wrong; it’s true that older systems used a version of the OpenSSL library that understood only SHA1 (128) certs, but that actually means 10.5 or so. Newer systems understand SHA2, and in any event, since the MAS went up, Apple has always recommended developers to not use the system’s OpenSSL library (I think it’s not even included anymore), so only very old apps would be affected by that.
Update: Glenn Fleishman has informed me about the SSL situation: there’s the new 1.0.x library branch and the older 0.9.x branch. Both apparently got SHA2 support in 2010, when 1.0.0 and 0.9.8o came out, but some developers seem to have kept older versions, no doubt for valid reasons; space precludes, etc.
Apple is blaming developers. Apparently this can be traced to a single report of misinformation from an anonymous Apple Support person. As I write this, Apple hasn’t yet said anything; I doubt they’ll say anything over the weekend.
This is a serious security/cryptography failure. Nope. This confusion arises from the fact that digital certificates (and libraries like OpenSSL) are used for both secure, encrypted communications and for app/receipt signing. In the latter case, an expired cert doesn’t expose any information or makes the system or apps easier to hack.
Developers are better off not doing any, or little, receipt checking. Not really. True, apps which don’t do full receipt checking might have not been affected in this single instance, but under usual circumstances they’re more vulnerable to hacking or piracy.
Apple’s store/system infrastructure is brittle and can’t be trusted. True, it’s a very complex system that depends on many twisted little interlocking parts to work properly. And, as we’ve seen, this particular instance of failure is as self-amplifying as electrical grid failures _ once it starts, the demands on the working parts grow so huge that those fail, too. In Apple’s defense, it’s very hard to test for or simulate. Let’s hope that all involved have learned something from this incident; I certainly have learned a lot.

Update: forgot to comment on this particular post:

But when I tried to convince my Mac to run this app as an unsigned app, I encountered what is extremely likely to be the store DRM: I initially got the “your app was bought on another machine” message, so I tried deleting the receipt, but then I got the dreaded “app damaged” message, at which point I removed the signature.

…the only way I can see is to create a new root CA which I install on the machine as a trusted root, and redo the signing chain, and even that might not work if the DRM is somehow tied to the signature chain.

While I can understand the frustration implicit in not being able to run purchased apps “forever”, I think this is a fundamentally wrong approach. Let’s educate developers to check receipts properly, as I mentioned above. Figuring out a way to run store apps (or even developer-ID purchased apps) without “DRM” means that anyone else can use the same method to install pirated copies; we wouldn’t be able to trust users anymore.

Much worse, re-signing someone else’s app and expecting it to run is an even greater violation of trust. The days when you could hack someone’s app with ResEdit and having fun making it look different, or do unexpected things, are long gone. I implement very strict checks that my complete app bundle has not been altered in any way and that it’s running with my original signature, otherwise any user could freely alter files, hack the code, change graphic resources or even — and such cases have happened! — repost the app somewhere else as being their own. No, flawed as the current approach may be in implementation, I see no better alternative.

Update: reports are in from some helpful fellow developers, confirming my suspicions of cache corruption — RB App Checker Lite says the app bundle and receipt contents are OK, yet the apps will not run. I use the same APIs (hopefully) that the system processes use — but those APIs can take a long time to run, so the results are cached somewhere.

Update: Apple sent email to all developers:

In anticipation of the expiration of the old Mac App Store certificate, we issued a new certificate in September.

As I said — no “let[ting] certificates expire”. They all do.

We are addressing this caching issue in an upcoming OS X update.

Confirms this is a caching issue, as I suspected.

…some apps are running receipt validation code using very old versions of OpenSSL that don’t support SHA-2. We addressed this by replacing the new SHA-2 certificate with a new SHA-1 certificate last Thursday night.

I’m a little surprised that the number of apps using“very old versions” justifies going back to SHA1; but, OK.

Please ensure your code adheres to the Receipt Validation Programming Guide and check that all receipt validation issues are resolved.

Good, but:

the link goes to the page detailing the online receipt validation. Very few apps use that IMHO — you have to be online every time the app runs, you have to have a reasonably fast connection, and app launch will be significantly slower. Linking to this page (Validating Receipts Locally) would’ve been better;
it would have been more helpful to call out specifically the certificate expiration check and update the sample code to properly use the receipt creation date.

Update: I’ve now submitted rdar:///23611335 — a bug report to call attention to this documentation problem.

Update: Fixed the “Validating Receipts Locally” link, which was also pointing to the wrong page. Sorry. Also, here’s one way to do the correct date checking (copied from Matt Steven’s code):

X509_STORE *store; // set up the store X509_VERIFY_PARAM *param = X509_VERIFY_PARAM_new(); X509_VERIFY_PARAM_set_time(param, time_from_receipt); // option 1: verify using a specific time X509_STORE_set1_param(store, param); X509_VERIFY_PARAM_free(param); // call PKCS7_verify() using configured store

Apple, Development, Software App Store, Mac, RB Utilities

#1 | Written by Rainer Brockerhoff on November 16, 2015.

Right, as I said, the receipt creation time field (since 2013) is to be used for that, but the developer has to pass it explicitly to the receipt certificate check API — and not all do. I didn’t do it myself, or know about it, until this weekend. AFAIK this is solely up to developers.

Also notice that we’re NOT talking about code signing, but about receipt signing; using the signature creation time has always worked for code signing on OS X, there’s plenty of apps running merrily for years with expired code signing certificates.
#2 | Written by Chris on November 16, 2015.

Is it really the case that apps from MAS cannot be “run forever”?

This is something that Microsoft code signing certificates solved on Windows by including a cryptographically signed timestamp. If that is present and in the certificate’s validity period, then the whole signature is trusted even after the certificate is expired. Googling for “authenticode” should find a better description of this.

I thought Apple code signing did something like this too, possibly this is what the receipt creation time field is.
#3 | Written by Rainer Brockerhoff on November 15, 2015.

it is hard to know what is code signing and what is copy protection…

Indeed. The copy protection is in the receipt only; the integrity protection is in the code signature only. For instance, my free DeveloperID apps can be freely copied with no problem, but if you remove the signature or re-sign the app, it will stop working.

Making a new receipt with the proper contents, signed by yourself, would ordinarily work for your purposes, but the standard checking algorithm demands Apple’s Root cert at the top of the chain. Hm… that needs more testing, but it gives me some ideas.
#4 | Written by Pierre Lebeaupin on November 15, 2015.

I don’t fault you or any developer for taking steps to ensure your software has the appropriate level of copy protection. My investigations are to figure out what is possible and not possible for me to do given my prerogatives as a user, and yes, it involves a bit of manipulation because otherwise it is hard to know what is code signing and what is copy protection; for instance if I had been able to put aside that cert without affecting the copy protection, I would not have asked developers of apps I bought on the Mac App Store to provide me with direct versions of their apps.

Furthermore, some level of integrity checking is a necessary part of a software copy protection scheme, no doubt about it. But this integrity check is not the same as code signing: the two ought to be separate, because they do not work towards the same goal. It is Apple I fault here for relying and making developers rely on X.509 certificates (that necessarily expire) as part of the copy protection integrity check: even if they had always given correct advice as to how to validate receipts, Murphy’s law was bound to make some developers make the mistake of validating the receipts against the current date.
#5 | Written by Rainer Brockerhoff on November 15, 2015.

Thanks for the excellent response. Your concerns are well-founded. I put “DRM” in quotes because I associate it more with music and movie files, where one tries to protect a passive blob of bits against misappropriation. This is also, in a sense, the case with the receipt files, which are also passive blobs of bits that must be checked by external software. Code signatures can also be checked by external software — that’s what my own app does, after all — but I prefer to say it’s DRM as “Developers’s Rights Management”, because the real check is done by the signed software itself.

So, I protect my software by checking that all pieces protected by the code signature are in place, are valid, and that my certificates are also valid and consistent and were valid at the time of signature. I could check if the certificates are still valid at the time of execution, but that would amount to time-limited software, so I never do that — except for public betas, where I do check against the execution date, period. Every developer producing Apple-signed software is aware that the developer certificates are short-lived, so nobody minds if a codesigning cert expired.

Now, some items inside the app bundle can be removed without invalidating the signature — that mostly means .lproj folders, which contain localised files. But the MAS receipt itself is one of these items, and it’s checked only by the software itself, as far as I know. The receipt, too, has certs embedded, as I said, and the check suggested in Apple’s docs does fail if the certs expire, which I didn’t know. Apple’s sample code doesn’t say so, anyway, and I’ve looked at some other people’s code and apparently they don’t say so, either. But logically one should do this check against the time the receipt was signed, just as we do with the code signatures. In Sept.2013 fields were added to the receipt that contain this time, so from then onwards it became possible to do the proper check — but I, and apparently other developers, hadn’t noticed that.

Anyway, my point is that DRM in this sense is now fully in the hands of every individual developer doing the right thing with the informations available at the time. Should I, perhaps, be more lenient with the execution environment after a certain date (say, 20 years after build date)? Such things are easy to do but hard to do properly. Food for thought.
#6 | Written by Pierre Lebeaupin on November 14, 2015.

Indeed, the “2035” thing seemed odd to me at first. Except for the fact it is tied to a certificate expiring and that Apple appears to have taken some action to help resolve the situation (after potentially redownloading the app from the store), there is not much we really know about the process behind the meltdown.

Anyway, my process here is to put myself in a position of a software archivist who inherits my computer and has to figure out how to run the software on it without any cooperation from any online service (assuming they are all long dead). Conceptually, there are two “security” services the Mac App Store provides: DRM, to protect the developer against unlicensed use of the app or the app being pilfered, modified, and passed off as being the modifier’s creation; and code signing, to protect the user against an attacker tampering the app between the moment the app was signed by someone the user (supposedly) trusts and the moment he runs it.

Code signing, by its nature, relies on digital certificates, and these certificates expire, for what I hear are good security reasons. The archivist does not particularly care about code signing: even if the app was tampered with by an attacker, the archivist has a pristine copy of the data, and the machine is off the network and nothing will ever exit it. Since code signing is put for the user’s benefit he should have as a last resort the ability to disengage it, otherwise this is not done for the user’s benefit and is not just code signing, is it? The archivist, however, will try to respect DRM if at all possible, because he knows the DRM protections are in place for the benefit of the software developer, not his; so the archivist will try to put himself in a position where the DRM is able to validate the app: same machine (MAC address in particular), same OS, etc. The archivist has heard of many kinds of more or less restrictive DRM and copy protection schemes; including ones that require phoning home each launch (games are particularly fond of those).

But the archivist, until this day, had never heard of DRM schemes where, everything else being otherwise equal, an already installed app will run or not today depending on the date the machine is set to. In other words, self-destructing app installs. He is in disbelief, how come, these certificates may have expired, but I assume they are part of the code signing infrastructure, one that benefits me, I can always bypass these no longer valid certificates, right?

But it appears the reports were right. To a first analysis at least, there does not appear to be a way to avoid what is now invalid while still allowing the DRM to perform its job, as it seems the certificate contributes to the DRM one way or another (it is unclear whether the certificate in question contributes to code signing). And so the archivist reports his finding and what he tried to show he did not just wave a dead chicken at the issue, because he honestly does not tamper with commercial software for fun.

Solipsism Gradient

A Tale of Two Certs

Leave a Comment

Tag Cloud

Archives

Categories

Meta

Links