Solipsism Gradient

Rainer Brockerhoff’s blog

Browsing Posts tagged App Store

A Tale of Two Certs


I’m keeping this post updated as details develop…

About ten days ago, something strange happened on my Mac: I was debugging the next version of my RB App Checker Lite app and suddenly I saw the dreaded dialog box:Damaged

Completely abnormal, especially as I was debugging using the Developer ID version (not the Mac App Store version!) from inside Xcode. When I opened Terminal, the same dialog; when I opened Safari, same thing! No new process was allowed to run. Of course I had to reboot to be able to do anything, everything worked fine afterwards, and I couldn’t reproduce the problem, so…

OK, a couple of days ago I concluded all was ready and I uploaded my app for review. A few hours after I announced so on Twitter, the reports began appear: the sky is falling! Major Mac App Store meltdown, everybody was getting the “damaged” dialog, Apple’s certificates were the culprit. I started testing my local apps from the MAS and, sure enough, the MAS leaf cert had expired; no problems, some of them asked anew for the AppleID password, some didn’t. RB App Checker Lite showed the expiration but no other problems, but I pulled it from review just in case.

Two days of confusion and frantic coding later, I had submitted (and pulled!) 4 more builds until I was reasonably sure that everything was working correctly. Thanks to several fellow developers on Twitter, the upcoming version seems to show everything correctly; it turned out that my receipt checks were somewhat obsolete. I usually publish the direct download version only after the MAS version has passed review, but decided to release version 1.1.4, build 351 immediately: you can get it here. It has a long list of improvements and fixes.

Meanwhile, the consensus is that rebooting and re-entering the AppleID and passwords (or even deleting and reinstalling) the affected apps solves 99% of the problems.

There are actually several different unfortunate problems here. First, the “damaged” dialog seems to be caused by some sort of cache or memory corruption in the system processes that coordinate to implement GateKeeper and the app store updates; some reports say killing the “storeagentd” process solves this problem without rebooting. (My system doesn’t seem to run this, FWIW.) What not everyone knows is that this dialog appears before the app it allowed to run; that is, it’s not affected by any checking done inside the app itself!

Second, asking for a new AppleID password. This is caused by the app itself checking the store receipt; something strongly recommended by Apple, since otherwise, it’s easy to copy a downloaded app to another computer and having it run there; I remember some early games not doing this and being widely pirated.

When an app is downloaded from the MAS, a proper receipt for that AppleID and that computer is already inside. A missing or corrupted receipt is the only normal circumstance in which the “damaged” dialog should appear. But if you copy the app to another computer, this will be noticed by the app itself.

Once a MAS app starts up, the first thing it should do is to check the receipt. It’s a complex process and not everybody implements it the same way. At first, checking the receipt’s cert chain would cause the receipt to be rejected in the case of expiration; the app exits with a special numeric code (exit 173) and this code signals the system to put up the dialog asking to confirm the purchaser’s AppleID and password. This, in turn, will cause a new receipt to be downloaded, and the app can now run with no problems. Update: reports indicate that, in at least some cases, the system doesn’t respond properly to exit 173.

A few years ago receipts began to include a new field containing the receipt’s creation date, and developers now had to check the certs against that date (and not against the current date), therefore obviating the need to reenter the password. Unfortunately this was not widely divulged, and Apple’s own sample code hasn’t yet been updated accordingly; I confess to not seeing this myself!

As is usual in disasters, several things have to go wrong at the same time: some bug corrupts a critical system cache, certificates expire normally, some apps incorrectly test for expiration, receipts are corrupted or the AppleID validation servers become slow or unreachable (because of the huge number of simultaneous requests), and… boom.

Many articles, unfortunately, published factual errors or wrong assumptions.Let’s try to counter a few:

  • Apple “allowed” their Mac App Store certificate to expire. Wrong on several levels. First, there’s not one but 5 (!) certificates involved in any app from the store: Apple’s root certificate: and 4 others: two intermediate and two leaf certificates.
    The way these certs work is by so-called certificate chains; every cert vouches for the lower-level ones. At the top is Apple’s Root certificate, which is one of a hundred or so in the System Keychain. There are two different certificate chains in every MAS app; the first is used in the code signature:and the second is used to sign the store receipt:Note the expired certificate there? This is a leaf certificate. These, usually, have a short life — one or two years — and the intermediate certificates usually last a little longer.
    So, when a cert expires, is that a serious problem? No – unless it is the root cert, which is why they all expire somewhen in the 2030s — hopefully, by that time, they’ll have figured out something better, Apple will have updated the cert via Software Update, or the horse will have learned to sing.
    The root cert can be updated via Software Update because it’s stored in System Keychain — but it’s impractical to push cert updates to each and every signed app, bundle or library; there are many thousands of them! So an expired cert in the code signature doesn’t affect the app at all. What’s important is that the certs were valid when the app was signed. When and if you get a new version of the app, all certs will probably be new ones. So there’s no “allowing” a leaf cert to expire — they do so naturally.
  • Apple “pushed” a new certificate that expires in 2035. This is probably just looking in the wrong place — not knowing which certificate had expired, someone glanced at the root certificate and noticed the “new” 2035 date. Nothing new to see, of course; that cert was created in 2006! Even more confusingly, someone else deduced from that that Apple let their original root cert expire; also wrong.
  • The system hasn’t been updated to check SHA2 (256) certificates. Wrong; it’s true that older systems used a version of the OpenSSL library that understood only SHA1 (128) certs, but that actually means 10.5 or so. Newer systems understand SHA2, and in any event, since the MAS went up, Apple has always recommended developers to not use the system’s OpenSSL library (I think it’s not even included anymore), so only very old apps would be affected by that.
    Update: Glenn Fleishman has informed me about the SSL situation: there’s the new 1.0.x library branch and the older 0.9.x branch. Both apparently got SHA2 support in 2010, when 1.0.0 and 0.9.8o came out, but some developers seem to have kept older versions, no doubt for valid reasons; space precludes, etc.
  • Apple is blaming developers. Apparently this can be traced to a single report of misinformation from an anonymous Apple Support person. As I write this, Apple hasn’t yet said anything; I doubt they’ll say anything over the weekend.
  • This is a serious security/cryptography failure.  Nope. This confusion arises from the fact that digital certificates (and libraries like OpenSSL)  are used for both secure, encrypted communications and for app/receipt signing. In the latter case, an expired cert doesn’t expose any information or makes the system or apps easier to hack.
  • Developers are better off not doing any, or little, receipt checking. Not really. True, apps which don’t do full receipt checking might have not been affected in this single instance, but under usual circumstances they’re more vulnerable to hacking or piracy.
  • Apple’s store/system infrastructure is brittle and can’t be trusted. True, it’s a very complex system that depends on many twisted little interlocking parts to work properly. And, as we’ve seen, this particular instance of failure is as self-amplifying as electrical grid failures _ once it starts, the demands on the working parts grow so huge that those fail, too. In Apple’s defense, it’s very hard to test for or simulate. Let’s hope that all involved have learned something from this incident; I certainly have learned a lot.

Update: forgot to comment on this particular post:

But when I tried to convince my Mac to run this app as an unsigned app, I encountered what is extremely likely to be the store DRM: I initially got the “your app was bought on another machine” message, so I tried deleting the receipt, but then I got the dreaded “app damaged” message, at which point I removed the signature.

…the only way I can see is to create a new root CA which I install on the machine as a trusted root, and redo the signing chain, and even that might not work if the DRM is somehow tied to the signature chain.

While I can understand the frustration implicit in not being able to run purchased apps “forever”, I think this is a fundamentally wrong approach. Let’s educate developers to check receipts properly, as I mentioned above. Figuring out a way to run store apps (or even developer-ID purchased apps) without “DRM” means that anyone else can use the same method to install pirated copies; we wouldn’t be able to trust users anymore.

Much worse, re-signing someone else’s app and expecting it to run is an even greater violation of trust. The days when you could hack someone’s app with ResEdit and having fun making it look different, or do unexpected things, are long gone. I implement very strict checks that my complete app bundle has not been altered in any way and that it’s running with my original signature, otherwise any user could freely alter files, hack the code, change graphic resources or even — and such cases have happened! — repost the app somewhere else as being their own. No, flawed as the current approach may be in implementation, I see no better alternative.

Update: reports are in from some helpful fellow developers, confirming my suspicions of cache corruption — RB App Checker Lite says the app bundle and receipt contents are OK, yet the apps will not run. I use the same APIs (hopefully) that the system processes use — but those APIs can take a long time to run, so the results are cached somewhere.

Update: Apple sent email to all developers:

In anticipation of the expiration of the old Mac App Store certificate, we issued a new certificate in September.

As I said — no “let[ting] certificates expire”. They all do.

We are addressing this caching issue in an upcoming OS X update.

Confirms this is a caching issue, as I suspected.

…some apps are running receipt validation code using very old versions of OpenSSL that don’t support SHA-2. We addressed this by replacing the new SHA-2 certificate with a new SHA-1 certificate last Thursday night.

I’m a little surprised that the number of apps using“very old versions” justifies going back to SHA1; but, OK.

Please ensure your code adheres to the Receipt Validation Programming Guide and check that all receipt validation issues are resolved.

Good, but:

  • the link goes to the page detailing the online receipt validation. Very few apps use that IMHO — you have to be online every time the app runs, you have to have a reasonably fast connection, and app launch will be significantly slower. Linking to this page (Validating Receipts Locally) would’ve been better;
  • it would have been more helpful to call out specifically the certificate expiration check and update the sample code to properly use the receipt creation date.

Update: I’ve now submitted rdar:///23611335 — a bug report to call attention to this documentation problem.

Update: Fixed the “Validating Receipts Locally” link, which was also pointing to the wrong page. Sorry. Also, here’s one way to do the correct date checking (copied from Matt Steven’s code):

X509_STORE *store;
// set up the store
X509_VERIFY_PARAM *param = X509_VERIFY_PARAM_new();
X509_VERIFY_PARAM_set_time(param, time_from_receipt); // option 1: verify using a specific time
X509_STORE_set1_param(store, param);
// call PKCS7_verify() using configured store

Apple’s (pre-)announcement of the Apple Watch left the tech world in the usual disarray. Is it an expensive knock-off of Android watches (people tell me there is such a thing!)? Is it an attack on the high-end Swiss watch market? Is it an attack on the low-end Japanese watch market? Is it an even more transparent lock-in attempt on soi-disant “Apple fanbois”? I’d answer “no” to all those questions, but right now I’m more interested in the hardware and software technology of the watch.

Notice that the above link doesn’t mention iOS anywhere, but this other link has the magic word: WatchKit. Quote: “WatchKit Apps. Soon your favorite apps will feature controls and interactions unique to Apple Watch, enabling you to enjoy them in dynamic new ways.

Speculations about WatchKit since then usually have mentioned one or two assumptions:

  1. WatchKit will be written in/accessible only from Swift;
  2. WatchKit apps will run under iOS on the Apple Watch.

The first is, of course, wishful thinking from developers investing in the new Swift language. The second is, in my opinion, completely unwarranted and I’ll try to explain why.

This post is the most plausible so far: “WatchKit apps will ship as embedded binaries in iPhone apps, using the same basic principals [sic] as iOS 8 extensions. There will be some mechanism for the watch paired to an iPhone to detect and automatically install these ‘apps’ based on what is available on the paired iPhone. Delete the container app from the iPhone, it disappears from the watch. Xcode will have a template to add a WatchKit app to an iPhone app project.

Let’s back off WatchKit for a second and look at what we’ve seen of the hardware. The entire main board is shrunk down to a single unit: the S1. If you stop the middle introduction film at 4:46, you’ll see that it’s really a collection of chips and SMT components on an encapsulated multilayer board — not really a “single chip” as the narration says, but many large CPU “chips” nowadays are like that, too. Other than the S1, there’s of course the “Taptic Engine” assembly which does the wrist tapping, the crown sensor assembly, antennas and display, and the most important part: the battery.

Battery life is the make-or-break feature of the Apple Watch. iFixit’s disassembly of the Moto 360 watch shows why: there’s a square peg battery inside a round casing, rated at 320 mAh. Even though Motorola apparently build their own batteries, they don’t have enough volume to do a round one. Apple doesn’t have a volume problem and their casing is square, so they’re free to use all remaining volume for a longer-lasting battery.

The 320 mAh rating and the typical battery life of 12 hours of the Moto 360 means that the watch consumes, on the average, just under 27 mA. But they run Android on the watch, using an off-the-shelf TI ARM processor with attached RAM, flash memory, and so forth, so that figure is not surprising. In other words, it’s a stripped-down cellphone/MP3 player.

Suppose that Apple did its usual optimization of battery size, usage, etc., in a stripped-down iPod nano. It’s half the size of the nano, which has a 30-hour life, so we can assume half the battery, meaning 15 hours. OK, that would be marginally acceptable, perhaps.

But remember, the Apple Watch needs an iPhone nearby. In fact, many of the published functions, such as Siri, cellphone call response, GPS and so forth certainly use the iPhone’s hardware and software for that. Remember that one of the culprits of excessive battery usage is generic apps and processes running on the device. Remember that Apple, since the first iPod in 2001, has been very aggressive in optimizing their embedded systems. Remember that the first iPods and iPhones didn’t have any generic apps running on them, either. Remember that Apple already has technologies like Clang, OpenCL and Metal…

All that said, why run iOS and generic applications on the Watch at all? So here’s what I think likely about the real implementation.

  • Watch OS (or whatever it’s called — did they explicitly call it anything?) will not be a stripped-down iOS; maybe even not a Darwin derivative. It will be a highly optimized embedded system that has a few apps running in as few processes as possible. It will be very robust because it will be able to do only a fixed set of functions.
  • In other words, it will run only those things that may run while the paired iPhone is not available; we don’t know yet, but that might be just the timekeeping and pulse measuring apps. If the iPhone is there, the Watch will also work as a specialized I/O and display device for the apps installed there.
  • WatchKit will run on the paired iPhone inside a special server process; a matching iOS app will show installed Watch apps — probably those apps will be from the normal App Store, since they usually will have an iOS counterpart.
  • So, an installed Watch app will have at least some sort of preference app or pane on the iPhone; no use typing in passwords and such on the Watch, right? The part written in/for WatchKit will contain a server plugin that does the heavy lifting, data collating and communicating with the outside world, but it will also contain the application logic itself, commanding the Watch to do or display certain things.
  • I don’t mean to imply that the Watch will run a full WebKit client and the iPhone a web server, that might be overkill. Perhaps a useful subset of that, perhaps some variation of Display Postscript, some interpreted command language, or just a sequence of drawing orders? The important part is that there’ll be a single process on the Watch for doing the UI, and all the application-specific parts can be offloaded to the iPhone.

One consequence is that you can forget the idea of “jailbreaking” the Watch to connect to a non-iPhone, of course. Another one is that battery life might be at least a day, maybe even two or more. Nothing on Apple’s site so far contradicts any of my reasoning.

So, will WatchKit be accessible from Swift apps? Certainly. Will it itself be written in Swift? I doubt it for now. Maybe in iOS 9 some of the frameworks in iOS (and OS X) will have been rewritten, assuming that by then the Swift optimizer will be good enough. But that won’t be the case in a few months.

Possible but unlikely: WatchKit may have an API to download actual application code to the S1, which may (or may not) have an ARM-like architecture. Only in such a case — and since there will be no Cocoa/iOS frameworks on the Watch — I would expect the downloaded code to be in Swift (without optionals!), for extra safety; can’t have the Watch crashing and rebooting, right?

Update: Marcel Weiher kindly reminded me of CarPlay, which apparently works like that; nobody would say that cars are running iOS. On the other hand, in that case, the device is connected over USB (that is, reasonable bandwidth) and the car doesn’t have any battery life problems.

Comments welcome.

Please stand by…

No comments

…while Apple catches up with sent-in FAXes. Yes, you read that right.

In the meantime, RB App Checker Lite is momentarily off the Mac App Store. (It’s still normally available from the product page, of course.)

With very few interruptions, I’ve been a registered Apple Developer since mid-1984, when I bought the original loose-leaf edition of “Inside Macintosh”. From a reasonably early date onwards, I always paid by credit card. At some point – I think it was the late 1990’s – most developers were switched to paying over the iTunes store – quite practical in the US and Europe, since that was already tied to the Apple ID.

However, Brazil was very late in getting the iTunes store and from then on developers here had to send in an annual FAX stating their identification, credit card number, and so forth. A few years ago, the the iOS app store, the iTunes store, and the Mac app store (in that order, I think) became available and every time we heaved a collective sigh of relief, thinking that easy payment would be again possible.

Unfortunately, it’s now late 2012. All the stores are available but we still have to send in a FAX to Cupertino to renew our developer programs! No shipping Mac even has a built-in FAX-Modem anymore, which was what I used in the early years. Twice, I’ve asked developer relations at WWDC to send in the FAX for me – which they did, to their credit; and for the last few years I’ve asked their counterparts at Apple Brazil to do so, as they’re the only place I know of which still has a FAX machine stashed away somewhere. Still, the renewal was effected in a few days.

When I renewed my iOS membership a few months ago it took an unprecedented 7 days to be effected after sending the FAX, so when my Mac membership renewal came up, I thought it prudent to send in the FAX 20 days (!) before to ensure a timely response.

Unfortunately, the 20 days have come and gone, my renewal application still hasn’t been processed, and I’ve been cut off from the developer sites. Moreover, my app is no longer visible in the Mac App Store. I’ve already complained twice to Developer Relations and re-sent the FAX (both by machine and by email) 3 times; at first they admitted that it should have taken at least two weeks (!) to process, but haven’t replied since.

In the meantime, the new version of RB App Checker Lite had been in the review queue for 15 days and was finally rejected this week; first because of the pending renewal, second because of a temporary entitlement that had been accepted in the first version. I’m working on that. More soon (hopefully).

Update: just got email from Apple, stating that they put in a temporary extension while my renewal is being processed. The app isn’t back on the store yet, but at least it gives me breathing time to submit a new version; perhaps as soon as tomorrow.

Update #2: the app is back on the store. Stay tuned…

Update #3: my renewal has been processed, and we’re back to normal. Now working on the app again.

Recently someone figured out an attack against in-app purchasing on iOS. Only a few days later Apple, with commendable speed, put up a page detailing how to counter this crack by implementing better receipt checking.

Now there’s news that a similar attack also works on OS X. For this, users have to install two bogus certificates, point their DNS at the cracker’s server, and run an auxiliary application while making the in-app purchase; this builds an apparently valid receipt inside the application bundle. (Of course this means that the user is trusting those certificates, that server and that application to be otherwise innocuous – not a good policy! And it asks you for your admin password while you’re connected to that server, too…)

So how to implement better receipt checking on the Mac? The details are different, in that OS X in-app receipts are stored inside the application bundle, inside the application’s original receipt from the Mac App Store. Furthermore this receipt has a known format and is signed by 3 certificates:

  • “Mac App Store Receipt Signing” (SHA1: 4A 7B 3A 17 00 A4 DA 4A D4 EA 43 3A 83 61 43 2E CF 1C A1 AF)
  • “Apple Worldwide Developer Relations Certificate” (SHA1: 09 50 B6 CD 3D 2F 37 EA 24 6A 1A AA 20 DF AA DB D6 FE 1F 75)
  • and the “Apple Root CA” (SHA1: 61 1E 5B 66 2C 59 3A 08 FF 58 D1 4A E2 24 52 D1 98 DF 6C 60)

the latter can be found, of course, inside every Mac user’s System Keychain. All this was easily obtained with my RB App Checker Lite utility (ahem). 🙂

Apple’s currently recommended receipt verification code, however, does not contain any recommendation to check the certificates used to sign the receipt; it does check if the receipt is for that particular application (otherwise people would just copy a receipt from another app) and if the receipt was generated on that particular computer (otherwise one could just copy the app from a friend’s computer).

No doubt Apple will now recommend to all OS X developers that their receipt verification codes also check the certificates – and in fact, that’s what my apps are already doing. The certificates are, after all, available from the same parsing process recommended in the above link. At the very least, I recommend obtaining the SHA1 fingerprints of all 3 certificates (openssl has a SHA1() function for that) and checking them against the list above. And once that’s done, obtaining the app’s own signing certificates, and checking them, is also advisable, even if the app is signed with a Developer ID.

I’m not giving specific code examples here, to avoid people copy&pasting it into their apps and offering a clear path to hacking the binary. The usual precautions to make binary hacking more difficult (though it’ll never be impossible) apply, of course.

Photos licensed by Creative Commons license. Unless otherwise noted, content © 2002-2017 by Rainer Brockerhoff. Iravan child theme by Rainer Brockerhoff, based on Arjuna-X, a WordPress Theme by SRS Solutions. jQuery UI based on Aristo.