Solipsism Gradient

Rainer Brockerhoff’s blog

Browsing Posts tagged RB Utilities

A Tale of Two Certs


I’m keeping this post updated as details develop…

About ten days ago, something strange happened on my Mac: I was debugging the next version of my RB App Checker Lite app and suddenly I saw the dreaded dialog box:Damaged

Completely abnormal, especially as I was debugging using the Developer ID version (not the Mac App Store version!) from inside Xcode. When I opened Terminal, the same dialog; when I opened Safari, same thing! No new process was allowed to run. Of course I had to reboot to be able to do anything, everything worked fine afterwards, and I couldn’t reproduce the problem, so…

OK, a couple of days ago I concluded all was ready and I uploaded my app for review. A few hours after I announced so on Twitter, the reports began appear: the sky is falling! Major Mac App Store meltdown, everybody was getting the “damaged” dialog, Apple’s certificates were the culprit. I started testing my local apps from the MAS and, sure enough, the MAS leaf cert had expired; no problems, some of them asked anew for the AppleID password, some didn’t. RB App Checker Lite showed the expiration but no other problems, but I pulled it from review just in case.

Two days of confusion and frantic coding later, I had submitted (and pulled!) 4 more builds until I was reasonably sure that everything was working correctly. Thanks to several fellow developers on Twitter, the upcoming version seems to show everything correctly; it turned out that my receipt checks were somewhat obsolete. I usually publish the direct download version only after the MAS version has passed review, but decided to release version 1.1.4, build 351 immediately: you can get it here. It has a long list of improvements and fixes.

Meanwhile, the consensus is that rebooting and re-entering the AppleID and passwords (or even deleting and reinstalling) the affected apps solves 99% of the problems.

There are actually several different unfortunate problems here. First, the “damaged” dialog seems to be caused by some sort of cache or memory corruption in the system processes that coordinate to implement GateKeeper and the app store updates; some reports say killing the “storeagentd” process solves this problem without rebooting. (My system doesn’t seem to run this, FWIW.) What not everyone knows is that this dialog appears before the app it allowed to run; that is, it’s not affected by any checking done inside the app itself!

Second, asking for a new AppleID password. This is caused by the app itself checking the store receipt; something strongly recommended by Apple, since otherwise, it’s easy to copy a downloaded app to another computer and having it run there; I remember some early games not doing this and being widely pirated.

When an app is downloaded from the MAS, a proper receipt for that AppleID and that computer is already inside. A missing or corrupted receipt is the only normal circumstance in which the “damaged” dialog should appear. But if you copy the app to another computer, this will be noticed by the app itself.

Once a MAS app starts up, the first thing it should do is to check the receipt. It’s a complex process and not everybody implements it the same way. At first, checking the receipt’s cert chain would cause the receipt to be rejected in the case of expiration; the app exits with a special numeric code (exit 173) and this code signals the system to put up the dialog asking to confirm the purchaser’s AppleID and password. This, in turn, will cause a new receipt to be downloaded, and the app can now run with no problems. Update: reports indicate that, in at least some cases, the system doesn’t respond properly to exit 173.

A few years ago receipts began to include a new field containing the receipt’s creation date, and developers now had to check the certs against that date (and not against the current date), therefore obviating the need to reenter the password. Unfortunately this was not widely divulged, and Apple’s own sample code hasn’t yet been updated accordingly; I confess to not seeing this myself!

As is usual in disasters, several things have to go wrong at the same time: some bug corrupts a critical system cache, certificates expire normally, some apps incorrectly test for expiration, receipts are corrupted or the AppleID validation servers become slow or unreachable (because of the huge number of simultaneous requests), and… boom.

Many articles, unfortunately, published factual errors or wrong assumptions.Let’s try to counter a few:

  • Apple “allowed” their Mac App Store certificate to expire. Wrong on several levels. First, there’s not one but 5 (!) certificates involved in any app from the store: Apple’s root certificate: and 4 others: two intermediate and two leaf certificates.
    The way these certs work is by so-called certificate chains; every cert vouches for the lower-level ones. At the top is Apple’s Root certificate, which is one of a hundred or so in the System Keychain. There are two different certificate chains in every MAS app; the first is used in the code signature:and the second is used to sign the store receipt:Note the expired certificate there? This is a leaf certificate. These, usually, have a short life — one or two years — and the intermediate certificates usually last a little longer.
    So, when a cert expires, is that a serious problem? No – unless it is the root cert, which is why they all expire somewhen in the 2030s — hopefully, by that time, they’ll have figured out something better, Apple will have updated the cert via Software Update, or the horse will have learned to sing.
    The root cert can be updated via Software Update because it’s stored in System Keychain — but it’s impractical to push cert updates to each and every signed app, bundle or library; there are many thousands of them! So an expired cert in the code signature doesn’t affect the app at all. What’s important is that the certs were valid when the app was signed. When and if you get a new version of the app, all certs will probably be new ones. So there’s no “allowing” a leaf cert to expire — they do so naturally.
  • Apple “pushed” a new certificate that expires in 2035. This is probably just looking in the wrong place — not knowing which certificate had expired, someone glanced at the root certificate and noticed the “new” 2035 date. Nothing new to see, of course; that cert was created in 2006! Even more confusingly, someone else deduced from that that Apple let their original root cert expire; also wrong.
  • The system hasn’t been updated to check SHA2 (256) certificates. Wrong; it’s true that older systems used a version of the OpenSSL library that understood only SHA1 (128) certs, but that actually means 10.5 or so. Newer systems understand SHA2, and in any event, since the MAS went up, Apple has always recommended developers to not use the system’s OpenSSL library (I think it’s not even included anymore), so only very old apps would be affected by that.
    Update: Glenn Fleishman has informed me about the SSL situation: there’s the new 1.0.x library branch and the older 0.9.x branch. Both apparently got SHA2 support in 2010, when 1.0.0 and 0.9.8o came out, but some developers seem to have kept older versions, no doubt for valid reasons; space precludes, etc.
  • Apple is blaming developers. Apparently this can be traced to a single report of misinformation from an anonymous Apple Support person. As I write this, Apple hasn’t yet said anything; I doubt they’ll say anything over the weekend.
  • This is a serious security/cryptography failure.  Nope. This confusion arises from the fact that digital certificates (and libraries like OpenSSL)  are used for both secure, encrypted communications and for app/receipt signing. In the latter case, an expired cert doesn’t expose any information or makes the system or apps easier to hack.
  • Developers are better off not doing any, or little, receipt checking. Not really. True, apps which don’t do full receipt checking might have not been affected in this single instance, but under usual circumstances they’re more vulnerable to hacking or piracy.
  • Apple’s store/system infrastructure is brittle and can’t be trusted. True, it’s a very complex system that depends on many twisted little interlocking parts to work properly. And, as we’ve seen, this particular instance of failure is as self-amplifying as electrical grid failures _ once it starts, the demands on the working parts grow so huge that those fail, too. In Apple’s defense, it’s very hard to test for or simulate. Let’s hope that all involved have learned something from this incident; I certainly have learned a lot.

Update: forgot to comment on this particular post:

But when I tried to convince my Mac to run this app as an unsigned app, I encountered what is extremely likely to be the store DRM: I initially got the “your app was bought on another machine” message, so I tried deleting the receipt, but then I got the dreaded “app damaged” message, at which point I removed the signature.

…the only way I can see is to create a new root CA which I install on the machine as a trusted root, and redo the signing chain, and even that might not work if the DRM is somehow tied to the signature chain.

While I can understand the frustration implicit in not being able to run purchased apps “forever”, I think this is a fundamentally wrong approach. Let’s educate developers to check receipts properly, as I mentioned above. Figuring out a way to run store apps (or even developer-ID purchased apps) without “DRM” means that anyone else can use the same method to install pirated copies; we wouldn’t be able to trust users anymore.

Much worse, re-signing someone else’s app and expecting it to run is an even greater violation of trust. The days when you could hack someone’s app with ResEdit and having fun making it look different, or do unexpected things, are long gone. I implement very strict checks that my complete app bundle has not been altered in any way and that it’s running with my original signature, otherwise any user could freely alter files, hack the code, change graphic resources or even — and such cases have happened! — repost the app somewhere else as being their own. No, flawed as the current approach may be in implementation, I see no better alternative.

Update: reports are in from some helpful fellow developers, confirming my suspicions of cache corruption — RB App Checker Lite says the app bundle and receipt contents are OK, yet the apps will not run. I use the same APIs (hopefully) that the system processes use — but those APIs can take a long time to run, so the results are cached somewhere.

Update: Apple sent email to all developers:

In anticipation of the expiration of the old Mac App Store certificate, we issued a new certificate in September.

As I said — no “let[ting] certificates expire”. They all do.

We are addressing this caching issue in an upcoming OS X update.

Confirms this is a caching issue, as I suspected.

…some apps are running receipt validation code using very old versions of OpenSSL that don’t support SHA-2. We addressed this by replacing the new SHA-2 certificate with a new SHA-1 certificate last Thursday night.

I’m a little surprised that the number of apps using“very old versions” justifies going back to SHA1; but, OK.

Please ensure your code adheres to the Receipt Validation Programming Guide and check that all receipt validation issues are resolved.

Good, but:

  • the link goes to the page detailing the online receipt validation. Very few apps use that IMHO — you have to be online every time the app runs, you have to have a reasonably fast connection, and app launch will be significantly slower. Linking to this page (Validating Receipts Locally) would’ve been better;
  • it would have been more helpful to call out specifically the certificate expiration check and update the sample code to properly use the receipt creation date.

Update: I’ve now submitted rdar:///23611335 — a bug report to call attention to this documentation problem.

Update: Fixed the “Validating Receipts Locally” link, which was also pointing to the wrong page. Sorry. Also, here’s one way to do the correct date checking (copied from Matt Steven’s code):

X509_STORE *store;
// set up the store
X509_VERIFY_PARAM *param = X509_VERIFY_PARAM_new();
X509_VERIFY_PARAM_set_time(param, time_from_receipt); // option 1: verify using a specific time
X509_STORE_set1_param(store, param);
// call PKCS7_verify() using configured store

As you may have seen elsewhere, Apple has just approved publication of RB App Checker Lite 1.1.3 (build 320) on the Mac App Store, and I’ve simultaneously published the Developer ID version (build 321) and updated the product page.

While the main focus of this version is to do some additional checks, fix bugs and display some information that users have requested, the XcodeGhost story broke just after I uploaded the first binary for review, and I had to remove it in order to see what RB App Checker Lite could do to detect this breach of security.

Assuming you’ve read the excellent summaries linked to in the previous paragraph, you may be interested in what I found inside the “ghosted” version of Xcode. Briefly, the hack alters the script Xcode uses to link an app’s binary; it also inserts look-alike versions of the CoreServices framework into the iOS, iPhone Simulator and OS X SDKs inside Xcode.

All this of course breaks Xcode’s code signature and, under normal circumstances, running such a hacked version would — after the customary delay for checking, 2 to 5 minutes — be detected by GateKeeper and it would advise the developer that “‘’ will damage your computer. You should move it to the Trash.” And the previous version of RB App Checker Lite would advise that “…requirements and resources didn’t pass static validation” and point at the changed file.

You’d think that that would take care of the matter, but it turns out that the affected developers turned GateKeeper off entirely, no doubt to get rid of the several minutes delay. After that, versions of their apps uploaded to the App Store would have been linked with a static library containing categories on Cocoa classes such as UIApplication, UIWindow and so forth; this static library having been hidden inside the added frameworks.

Needless to say, the new version of RB App Checker Lite also detects the added frameworks and warns: “3 frameworks are suspect: they use system names but are NOT signed by Apple!”.

This is both good and bad news. The good news is that this specific version of XcodeGhost — or any similar hack that hides code inside bogus frameworks looking like Apple’s frameworks — can be detected. The bad news is that this specific tactic depends on passing a casual visual inspection of the SDKs inside Xcode; in other words, the names and file paths used look reasonable and mostly duplicate Apple’s names and conventions.

This works because Xcode is a huge application; it contains nearly 5 thousand auxiliary executables. The latest Xcode beta has several SDKs for each of the 7 platforms it supports, and each SDK has an included instance of all system frameworks, both public and private, for that particular combination. Unfortunately, not all these frameworks are currently signed by Apple — only 2/3 of them are, and not in a consistent manner. (In all fairness, the percentage has been creeping up a little with each release.)

Therefore, unless you check the entire app contents with GateKeeper, RB App Checker Lite (or even the codesign command-line utility), it will be humanly impossible to pick out visually — by inspection in the Finder — if anything has been changed inside Xcode. So keep GateKeeper turned on! One suggestion Apple should implement is running GateKeeper tests for Apple-signed software even if GateKeeper has been deliberately disabled.

So, what to do about “infected” apps? Unfortunately the news is not good there. (By the way, I’m surprised that no infected apps were — as yet — found on the Mac App Store.) As I said, infected apps contain linked-in categories on Cocoa classes, using plausible English method names. Writing such categories is perfectly legal and even plausible — I’ve done so myself. Having code inside these categories do things that are allowed by the app’s entitlements, such as sending/receiving data over the net, is also perfectly legal and plausible. There seem to be some utilities out already that purport detecting such code, but I suppose they’d turn up a lot of false positives unless they check for these specific combinations of symbols — not very future-proof.

By the same token, Apple can’t really do these tests comprehensively when an app is uploaded to the store. They can and do check for private or “suspect” APIs being called, but as far as I can see the present XcodeGhost doesn’t use anything like that.

Coming back to RB App Checker Lite: it currently does NOT look inside executable code at all. Should it do so? I’m reluctant to implement that; it’s not clear what exactly to look for, regarding hacks like XcodeGhost, and it would mean that checking Xcode and similar huge apps would take tens of minutes or even more. I’m open to suggestions, however… comment here or email me!

Updates update

No comments

Good news: we’re back. Here’s my updated world map:

Bad news: my plans to work underway were largely derailed, and I had to wait an additional couple of weeks for my new iMac to arrive. Still, everything is finally installed and working now, and an update to RB App Checker Lite should be out in a day or two.

I fixed a crash that happened on some (apparently very few) systems and the new version is 1.1.1 — build 288 for the Mac App Store version, and build 289 for the direct download version. As a bonus, the app should also start up a little faster.

In fact, the MAS version has been out since the day before yesterday, but we’re on the road (currently in Germany) and Internet connections have been a little irregular.

On the downside, in the next 4 weeks my connections will be even more uncertain — we’ll be in Sikkim, Bhutan, Nepal, Italy and in the Eastern Aegean, so further updates or postings will probably have to wait until I’m back home. On the upside, I’m taking my laptop and will try to work in my copious free time while underway… 🙂


No comments

The recent update to WordPress 4.0 seemed to have gone smoothly — perhaps too smoothly.

Turns out it has some fancy new redirection facilities that interfered with some of the pages outside this blog — specifically, some of the product pages, but only if the final /index.html was missing — it then would search the blog for the oldest post mentioning the product. Go figure.

Anyway, I managed to mostly fix it; the sole remaining exception seems to be the Klicko page. Please use this link to get there, in the unlikely case you’re still using that.

Another recent addition is the Crayon plugin to properly format source code on these pages. I’ve checked it out and it seems to have a bazillion options. I’ll have to play around with it and find the one that breaks the fewest older pages — this may take a week or so.

Speaking of a week or so, we’re currently on the road for yet another long trip; this time to Germany, Northern India and Sikkim, Bhutan, Nepal, Italy, Greece and Montenegro. I’ll probably have Internet for another 8 or 9 days, but by then whatever’s not fixed will have to wait for our return around the end of October.

Meanwhile, a small update(1.1.1) to RB App Checker Lite is waiting for review in the Mac App Store. I’ll post the direct download version ASAP; the only change is a fix for a crash that a couple of users complained about. Meanwhile, RB App Quarantine is about to be updated with helpful suggestions from users; I’ll try to do that while underway.

The direct download version of RB App Checker Lite 1.1 (281) is out, and RB App Quarantine also has been updated to build# 281 (its version is still 1.1 since nothing has changed for the user). Check out the release notes!

Most important: this version of RB App Checker Lite considers the new signing rules in TN2206 for OS X 10.9.5 and 10.10. In particular, for applications, code signatures are now checked recursively, both version 1 and version 2 resource rules are shown if present, and the spctl utility is called to check the Gatekeeper assessment.

However, this signals a new development: direct download and Mac App Store versions of RB Utilities will now, unfortunately, have different functionality. In particular, the MAS version of RB App Checker Lite (which is currently in review) will not be allowed to call  spctl, as this utility requires special entitlements to work; and the MAS version of RB App Quarantine (also in review), for the same reason, will not clear the quarantine flag. So for now you may want to download directly from the product pages.

Apple’s rules for calling attention to such differences are a little tricky to navigate but hopefully the MAS versions will be approved and in a future release I’ll develop a way to bring this functionality back through an optional download. Research is underway!

RB App Quarantine 1.1 (273) is out. It’s the second app in the RB Utilities software suite — RB App Checker Lite was the first one.

As the name implies, it’s a utility that checks or changes the “quarantine” attribute of other applications. This attribute is set whenever an application is directly or indirectly downloaded by the user from anywhere except the Mac App Store. (Applications produced from installer packages, disk images or compressed files inherit the attribute automatically.)

When a quarantined application is first opened or executed, OS X’s Gatekeeper function will check the application’s code signature and several other details and either reject it or throw up the well-known dialog, confirming that you want to execute a downloaded application. If you agree, the quarantine is cleared and Gatekeeper will not check the application again.

Using RB App Quarantine to clear some just-downloaded application’s quarantine attribute is not really recommended: you’ll be bypassing Gatekeeper and — unless you’re a developer yourself and/or have already used RB App Checker Lite to check that application’s bona fides — may be opening your system to a potentially untrusted application.

If you are a developer yourself, using RB App Quarantine to set quarantine on your own application will allow you to check its Gatekeeper status without using Terminal commands or (perish forbid) uploading it to some server and downloading it again.

It took me just 5 days to write this little application since all the UI and other logic common to all RB Utilities is contained in a prepackaged framework and I just had to write the app-specific file/folder handling. Setting up a new project with everything in place and drawing the new icon took a single day. Unfortunately clearing the quarantine attribute takes a special sandbox entitlement which would certainly be frowned upon by the Mac App Store reviewers, so I didn’t even try submitting it.

In other news, I submitted a new version of RB App Checker Lite to the Mac App Store and, if everything goes well, it should be out soon. This new version fixes some bugs and — most requested by users — shows some details pertaining to the latest version of Apple’s Technote 2206, namely showing version 1 and 2 resource rules and showing the Gatekeeper (spctl) assessment results. Stay tuned…

Yay! Another update!

No comments

RB App Checker Lite 1.0.3 is now live both for direct download and from the Mac App Store. Here are the release highlights:

  • Now opens and checks .ipa bundles.
  • Opening packages (like .xcarchives) that contain applications now works properly, showing the package icon and path.
  • More and better help and credits text, now with active popups.
  • Library licenses have been included in the credits.
  • New interrupt/redo scan button.
  • The list of known entitlements has been updated.
  • Better explanations for most code signing errors.
  • Complains about missing receipt for App Store apps.
  • Pop-up file lists are now slightly better-looking.
  • Unsigned frameworks aren’t incorrectly flagged with a signing error anymore.
  • Fixed: the app froze after clicking the full-screen button in QuickLook preview.
  • Fixed: issues with mailto: links in the About window.
  • Fixed: the File->Select… menu is now enabled; it doesn’t work for frameworks, though; the workaround is to drag one onto the app icon or window.

And a most important note that I wasn’t allowed to mention in the app itself because of Mac App Store restrictions: the new version should be fully compatible with Mac OS X 10.9 (Mavericks).

On the travel front, we had a few wonderful days in the old villages in the interior of Portugal, and now an excellent week in Ireland. The weather, which was abnormally splendid, has today returned to its more normal Irish standard of wet and windy. More anon.

Photos licensed by Creative Commons license. Unless otherwise noted, content © 2002-2017 by Rainer Brockerhoff. Iravan child theme by Rainer Brockerhoff, based on Arjuna-X, a WordPress Theme by SRS Solutions. jQuery UI based on Aristo.