Solipsism Gradient

The subject of private APIs has been raised again with the recent availability of the iPad. (Unless noted, the following discussion applies to both the Mac and the iPad/iPhone/iPod touch platforms.)

Brief recap: API means “application programming interface”; in other words, it’s the way that an application programmer calls some system routine. APIs are grouped in libraries or frameworks – the distinction between these is largely unimportant here, and we’ll use the term “framework” from now on. Frameworks can be (and usually are) layered, either by explicitly being contained within each other, or by calling APIs from other frameworks.

If you browse into the /System/Library folder on a Mac, you’ll find both “Framework” and “Private Framework” folders. The “Framework” folder contains nearly a hundred subfolders with the .framework extension. Each one will have either a binary file containing the actual executables or a folder of contained frameworks, and a “Headers” folder with one or more .h files. These header files, in turn, list what is known as the “public APIs” for that framework. Browsing will reveal such well-known frameworks as Carbon and Cocoa, as well as more obscure ones like GLUT, LDAP and so forth.

Now look into “Private Frameworks”; there, too, is a bunch of .framework folders, but none of those have a “Headers” folder. This means that there is no public API for these frameworks at all; normal application programs aren’t supposed to use these frameworks. The names are often intriguing: AppleFSCompression, BezelServices, CoreMedia, DiskManagement, SystemMigration, iTunesAccess and so forth. Many of these are actually shared between several applications; for instance, the ProKit framework is shared by Apple’s “pro” applications (Final Cut Pro, and so forth) and this makes these applications smaller and easier to fix.

But not only the private frameworks have private API. The public frameworks too have APIs that aren’t included in the header files; or, rather, the header files for those APIs are available only internally at Apple. In other words, private APIs (some of which are also known as SPIs: “system programming interfaces”) are everywhere, but common mortals should not use them. Indeed, on Apple’s mailing lists, discussing private APIs and SPIs is a definite no-no. Like APIs, the formats and contents of some system files are undocumented, and fall under the same restrictions.

At this point, it’s useful to recall the basic motivations for having an API (be it private or public) at all. First, an API prevents the programmer from having to reinvent the wheel by doing several, often complex, lower-level steps to get the same results; second, isolating these steps into a single place means fixing a bug there will fix the bug for all programs or frameworks that call the API. In fact, having such APIs calling each other is the magic that enables Apple to keep older software working while changing parts of the infrastructure, as happens with every major OS release.

Still, it’s fun to find out about private APIs and test them, but inadvisable to use them for actual shipping applications, as they may vanish as suddenly as they appear, or change radically from one system version to the next.

The official process of API “promotion” is explained by Bertrand Serlet in this snippet from a recent WWDC presentation: an API begins as private, is reviewed internally, then is exposed to developers in some beta version of the system, and after some time is “committed” as public API. And (Serlet doesn’t say), several versions later an API may be “deprecated” – meaning, it will still work but isn’t recommended for new applications – and, some more versions later, it may be removed entirely from the system. The latter process happens to private APIs as well, and no doubt even more frequently, since Apple can safely assume that no shipping application other than their own uses these APIs directly. And this of course is why Apple’s own applications are always updated along with every system revision.

Now, some people see an insidious conspiracy here: Apple’s programmers have access to all these powerful resources that are denied to third-party programmers. In the past, Microsoft was ordered by a court to disclose APIs used by their Internet Explorer browser; these APIs were supposed to give them an unfair advantage over browsers from other companies. See, for instance, Alex Payne:

Apple should remove the concept of private APIs from its developer offerings. Give developers the same tools that Apple?s own programmers get to use. If an API is still too unsafe or experimental for developers to make use of, don?t ship it, or gate it to development versions. Don?t restrict third parties from taking full advantage of the device and its software.

I find this comment puzzling. There’s no way to “remove the concept” of private APIs from the system, nor are they “shipped” – in fact, they’re private precisely because they’re not shipped (though I’d say documented, instead of shipped). But they have to exist, otherwise there’d be no way to make a layered system.

By the way, even the first Mac 128 in 1984 had private APIs! There were no real layers to that first System 1.0, but some functions in the ROM were not supposed to be called by the application programmer. Needless to say, some enterprising programmer soon found out their machine addresses and thus could call them directly; also, needless to say, these hacks broke immediately when a new model came out.

Marco Arment says:

iBooks? use of tons of private APIs is frustrating on a few levels, the biggest that it makes all third-party reading-related apps second-class citizens.

I won?t be able to offer some features that iBooks has (such as a true brightness control), but my customers will expect them, making my app inferior to Apple?s in key areas.

This app?s undocumented API use wouldn?t pass the App Store submission process, yet developers need to compete with it for App Store attention. One of the great potential failures of an app-review system is inconsistent or unfair enforcement of the rules.

While it is of course possible for Apple to abuse the system in this way, it’s not reasonable to expect Apple’s applications not to use any new APIs while they are being tested; after all, this is the best (and perhaps only) way for them to be tested!

Marco also points at proof that iBooks uses private frameworks (and the Kindle app, of course, doesn’t). Some of those, no doubt, are justified code sharing between Apple apps – WebKit, iTunes stuff, etc. Others, like WebKit, Accessibility and JavaScriptCore may be just still in that “testing” stage; by keeping them private, Apple is free to change all those frameworks (along with all of their apps that use them) at every minor system revision. I don’t doubt that WebKit, for instance, will be moved to the public framework stage in a not-so-distant revision.

In conclusion, I don’t think that worrying overmuch over private APIs will be productive. Better to observe Apple’s applications closely, and then file (early, and often) detailed requests for specific APIs that it would be productive to have available – Apple does listen to such requests, although they may end up doing things a little differently.

No, I haven’t ordered one myself; they’ll take some months to appear hereabouts, and will be expensive. Maybe at WWDC I’ll buy one, though I may not be able to attend this year.

Of course most of the reviewers already have theirs, and the divide between the geek and non-geek users is being endlessly rehashed. David Pogue at the NYT went so far as to do a double review. One predictable vote for the geek side came from Cory Doctorow:

Why I won’t buy an iPad (and think you shouldn’t, either)

…Relying on incumbents to produce your revolutions is not a good strategy. They’re apt to take all the stuff that makes their products great and try to use technology to charge you extra for it, or prohibit it altogether.

…But with the iPad, it seems like Apple’s model customer is that same stupid stereotype of a technophobic, timid, scatterbrained mother as appears in a billion renditions of “that’s too complicated for my mom”…

…As an adult, I want to be able to choose whose stuff I buy and whom I trust to evaluate that stuff. I don’t want my universe of apps constrained to the stuff that the Cupertino Politburo decides to allow for its platform.

Now, Cory is one of my favorite writers, but I think he went a little overboard there. Once you accept that Apple’s model customer indeed is the “technophobic, timid, scatterbrained mother”, all his other arguments become much weaker, since they apply only to old-fashioned tinkerers who are concerned about absolute freedom for their “universe of apps”, or even want to just run Linux on the thing.

Both Joel Johnson and John Gruber agree with me that the iPad is essentially a consumer device, not a device for tinkerers and makers.

Now, I consider myself a serious tinkerer. I learned programming in the days when that meant plugboards and punch cards, and programming books started out with chapters of flip-flop and half-adder schematics. I’ve owned a 1961 VW of which I knew every single wire and screw, learned how to do radio and TV maintenance, designed hardware and OS for special-purpose computers, and so forth.

However, even as a tinkerer, I don’t really want to open up my computer and tinker with it just for the fun of it – or install Linux on it, or whatever – as long as it works like I want it to. Now, I bought my first Mac in 1984, and have owned at least two dozen different models since then; I’ve opened up (and in many cases, expanded or modified) nearly all of them. The exceptions are my current iMac and MacBook Air; they just work fine, and I won’t risk marring the finish. If I want to tinker, I’ll get an old junk PC somewhere. But, nowadays, I find writing applications is more satisfying.

But I digress. What I do disagree with is some of this “freedom to tinker” entitlement philosophy. Quoting Cory again:

Having gotten into business with the two industries that most believe that you shouldn’t be able to modify your hardware, load your own software on it, write software for it, override instructions given to it by the mothership (the entertainment industry and the phone companies), Apple has defined its business around these principles.

Exactly; that’s Apple’s privilege, and as an Apple stockholder I agree with their position. Cory (or anyone) should be perfectly free to modify an iPad, write their own software for it and so forth – with one caveat: they can’t force Apple to help them with that, and once they do it, it’s not an Apple device anymore for all practical purposes: no warranty, at the very least, no support, and so forth. I suppose that they want all of that, too; quite unrealistic.

Update: yes, there’s the AppStore. I dislike the current terms, at least from the developer’s side, and they’re one of the factors that keep me from developing for the iPhone/iPad right now. I do think (make that hope) that the terms will – very gradually – become less restrictive. All this “politburo” stuff is, regrettably, fueled by today’s insanely litigious society and Apple’s reaction to avoid liability at all costs.

Just published a droplet application that fixes the spelling error in nibs produced by my previous (beta) version of the RBSplitView Interface Builder plugin. Details on the RBSplitView page.

(This was a good question asked of me at formspring.)

So far, I haven’t done anything for the iPhone, for three reasons.

First, I don’t own a cellphone, and I find it quite difficult to write an app that I won’t use myself. I do have an iPod Touch, and I carry it with me on trips, but it seems I use it strictly as an alarm clock and as a normal iPod – the only apps I have installed are PCalc Lite and the Flycast internet radio. I look at other apps now and then, without finding any that give me that “wow I need that” thrill. (For the record, I’ve also owned the first Newton, the first Palm Pilot, and the Sony Magic Cap – all with their development SDKs – and I couldn’t think of a single useful app for them either; I suppose it’s a mental deficiency.) Also, recall that where I live there’s very little public WiFi, and 3G charges are ruinous… any app I’d use would need to work offline 99% of the time, which narrows things down considerably.

Second, the current AppStore model is extremely different from what I’m comfortable with – ideally, I want to distribute and support it myself, and be able to post updates immediately, without waiting for approval from anybody else.

Third, my main interest is in writing utilities – applications that extend or change the way “the system” or other applications do things. There’s no space whatsoever for that on the iPhone, and it doesn’t look like there’ll be space for it on the iPad for now.

That said, I do plan to attend WWDC 2010 and it’s possible that until June I think of some mainstream application that I’d like use myself. The larger screen could make all the difference. Send in suggestions!

So, to start things going again… I was looking at my RSS feeds for the first time in almost 5 months, and read Dan Wood of Karelia fame explaining about Converting Rich Text to TEXT/styl resources for an SLA on a Disk Image.

My own workflow for building a release disk image uses a small tool I’ve written for that. You can download it here. Here’s the help text it prints out if you run it without arguments:

   Add one license at a time to a (unflattened) disk image.
   Usage: AddLicense /path/to/TheUnflattened.dmg Language /path/to/TheLicense.rtf
   Languages supported: da nl ko ja fr it fi pt sv en es de nb
   You can also use long equivalents like English, French etc.
   The first language added will be the default language (usually English).

Here's an actual usage example from a build script:

   hdiutil unflatten "$SOURCE_ROOT/My.dmg"
   "$BUILT_PRODUCTS_DIR/AddLicense" "$SOURCE_ROOT/My.dmg" English "$SOURCE_ROOT/EnglishLicense.rtf"
   "$BUILT_PRODUCTS_DIR/AddLicense" "$SOURCE_ROOT/My.dmg" French "$SOURCE_ROOT/FrenchLicense.rtf"
   hdiutil flatten "$SOURCE_ROOT/My.dmg"

and I use the “flatten” and “unflatten” arguments to hdiutil to massage the disk image.

The trick (as Dan points out in his post update) is that you can use ‘RTF ‘ resources instead of TEXT/styl. I received this interesting tidbit through oral tradition; I’m not a 100% certain, but I think it was through some code that Marko Karppinen showed me a few years ago. Pass it on.

If there’s interest, I may clean up and publish the source sometime, though it uses all sort of gronky old APIs (Resource Manager etc.).

Update: the link above now downloads the complete Xcode project. I also included it on my source code page.

I just published the Xcode project for the AddLicense tool I mentioned previously. Enjoy.

Posted by 0xced:

Rainer Brockerhoff wrote:

If there’s interest, I may clean up and publish the source sometime, though it uses all sort of gronky old APIs (Resource Manager etc.).

Oh yes, there is interest!

On the stopover at Hongkong; amazing place. I bought a new camera today (the Panasonic Lumix DMC-FX65) and it seems reasonably improved from my previous FX35. Tomorrow we go off again southwards, first to Manila (weather permitting).

In other news, RBSplitView 1.2 is up. I’m still not satisfied with some aspects, but it should now work OK with Snow Leopard, 64-bit, garbage collection, etc.