Solipsism Gradient

Rainer Brockerhoff’s blog

Browsing Posts in Software

Over two weeks ago, Apple at WWDC announced something entirely unexpected: thousands of new APIs and a brand-new programming language, Swift. No hardware, of course; it’s a developers conference, remember?

Reactions varied all over the spectrum. Non-developers (especially “industry analysts”) mostly had no idea what it meant: they said Apple had announced “nothing”. Almost all developers, however, were ecstatic — “the most significant event Apple ever staged“. Regarding Swift, this initial enthusiasm diverged as soon as people read the (relatively sparse) documentation and actually began to play around with the language — a very early beta version was available for download soon after the announcement. Hilarity, chaos and pandemonium ensued; tension, apprehension and dissension had begun.

As usual, almost everybody tried to project their grievances, expectations and experiences onto the new language. The open-source advocates griped that no source was available. The cross-platform advocates complained that there was no version running/compiling for Android (as if Apple would have any interest in promoting that!). The Objective-C programmers unsuccessfully tried to translate their code into Swift and complained that there was only limited dynamic dispatching and introspection. The C programmers complained that there were no preprocessor macros and that Swift seemed to be “Objective-C without the C”. The Haskell/Erlang/Scala programmers complained that many functional programming facilities were missing, and that the language was “too mutable”. The Java programmers complained that the language was “too C++-like”, but resented the lack of exceptions. The C++ programmers also resented the lack of exceptions and wanted std::somethings. The Type Theorists complained that generics were “not generic enough”. JavaScript programmers… well, you get the idea. Almost everybody complained about Array mutability semantics, about missing semicolons and the parsing of whitespace, and (of course) said that the syntax “looked weird”. Serious fights erupted on Twitter, disagreeing on whether Swift was a “modern” language and what Apple’s intentions were.

And, as always happens, many people said, in effect, “OMG Apple you’re soo stooopid WTF fix this now!”. This is the usual symptom of looking at the surface and not understanding what might be happening underneath.

Voluminous disclaimer and sidenote with historical digressions:

Many of the complaints in the paragraphs above are condensations of what I understood people to be saying and none are meant to be actual live quotes — which is why I didn’t link to any specific instance. I’m not interested in discussing most of these personally right now, thank you.

I’ve been programming since 1969,  in C since 1984, in Objective-C since 2000. I wrote only one application in C++ back in the Classic days — it was pretty much mandatory in the CodeWarrior/PowerPlant days. I did my CS degree in the early 1970’s, when “modern” language still meant ALGOL 68 – see the mind-boogling official reference (large PDF).

When BYTE Magazine‘s special Smalltalk issue came out in 1981, I was very interested, but couldn’t come to grips with the weird syntax. I bought Adele Goldberg‘s classic books about Smalltalk — the blue book (large PDF), the orange book (large PDF) and the green book (large PDF) — and periodically tried to understand them; very difficult without access to a working compiler! In the late 80’s I put these aside (and, unfortunately, lost them in a move). After Apple acquired NeXT in 1996, I became aware of Objective-C’s roots in Smalltalk, but didn’t give it much thought.

Around 2000, restarting my work as an indie developer, I started programming in Objective-C and Cocoa. As an experienced C programmer I had little difficulty with Objective-C, and quickly got used to the nested [[ ]]s. I never wrote a full Carbon app as such. I also never managed to acquire a working Smalltalk compiler, even after a few became available on the Mac. However, a couple of years ago, I found the Smalltalk books in PDF format (as linked above) and was astounded: the formerly opaque things about methods, messages, dynamic dispatching, objects and so forth — suddenly all was clear and obvious! That’s the advantage of using-while-learning, at least for me.

Unlike many colleagues I never hesitated to go beyond Cocoa, always using CoreFoundation, BSD/Darwin and a variety of interfaces according to necessity, and once manual memory management became ingrained, tossing objects and buffers back and forth between the various APIs. Except for short utilities for my own use, I haven’t adopted ARC yet — I found too many edge cases for my established programming habits.

 So, back to Swift. It really appears to be a very pragmatic language. If you look at the generated library header (in Xcode, command-doubleclick on any Swift type to see it), nearly all operators and types are defined there, in often surprising detail. In other words, few language features are hard-wired into the parser/compiler – the Swift library/runtime and the pre-LLVM optimizer are, instead, responsible for the language and its implementation details, and therefore more easily twiddled if necessary.

This is, of course, very convenient for Apple: a small team could tinker around with all aspects of Swift while leveraging most of the existing LLVM infrastructure and keeping up with the latest changes in iOS and OS X. Indeed, in retrospect, it appears that Swift was even driving many of those changes!

Let’s look at a brief timeline to explain what I mean:

  • 2000-2002: Chris Lattner‘s masters thesis on LLVM;
  • 2005: Lattner hired by Apple; Apple uses LLVM for the OpenGL shading language in Mac OS X 10.5;
  • 2006-2008: Apple introduces experimental llvm-gcc in Xcode 3.1; “blocks” and GCD appear;
  • 2009: Apple introduces Clang as an alternative for gcc; OpenCL and Clang static analyzer appear;
  • 2010: Lattner begins working on Swift; Clang fully supports C++ and llvm-gcc is the default compiler;
  • 2011: gcc/gdb are discarded, Clang/lldb are defaults, ARC introduced in Xcode 4.2;
  • 2012-2013: iOS/OS X are fully built with the new infrastructure, Objective-C literals in Xcode 4.4;
  • 2013: Lattner becomes head of the developer tools department;
  • 2014: Swift comes out in Xcode 6.0.

The LLVM team (Lattner, Evan Cheng who is also at Apple, and Vikram Adve of UIUC) also received the 2012 ACM Software System Award, and of course, LLVM, Clang, LLDB are open-source projects being driven forward by many people who also deserve lots of credit.

Nevertheless, it’s tempting to see all this as Chris Lattner’s plan for world domination… just picture him stroking a white cat and going “mwahaha!” 🙂 [Update: Thanks to @darth for the illustration!]

But really, all this points to progress in Apple’s platforms being driven by a consistent plan to modernize and implement new technologies everywhere; even hardware was affected, as the Apple A6 CPU (and no doubt its successors) were designed in parallel with the corresponding LLVM code generator. Similarly, from 2009 forward, software advances like ARC, blocks, GCD, runtime modernizations etc. are now seen as preparing the ground for Swift at all levels.

Sidenote:

A few years ago I posted about Apple’s hardware options being enabled by LLVM, and with the A6 that has indeed begun to happen. Apple’s in position now to design their own CPU and just have to write a new optimizer backend for it — and switch architectures in new hardware without users, or even developers, noticing any significant change.

When I began studying programming languages and compilers, UNCOL was the holy grail of programming:  a universal intermediate language to adapt any high-level language/compiler to any machine architecture. LLVM is the first implementation of that.

What does all this mean for Swift? Contrary to what you may hear from some quarters, it’s not an amateurish, ham-fisted attempt at locking developers in to Apple’s “walled garden”. As Apple has said publicly, it’s a systems programming language that ties in to key Apple technologies. I don’t doubt that it’s already being deployed internally and we can expect to see key low-level frameworks — Security, dyld, IOKit are candidates which come to mind — rewritten in Swift as soon as feasible. In the long run, the kernel itself, Core Foundation and others may follow suit; picture “SwiftKit” unifying much of AppKit and UIKit. Making Swift available to developers at this beta stage is good policy but probably not Apple’s primary focus.

But, you may ask, why not use C++ or the hybrid Objective-C++? Why not use a “modern” cross-platform language? What was wrong with Objective-C anyway?

Well, there’s a reason so many low-level frameworks are written in C++ or pure C: runtime speed. Objective-C’s dynamic dispatching has vastly improved over the years but is still a bottleneck, and in 95% of cases is not really necessary — we rarely use id, and strong typing is encouraged everywhere. As for pure C code, when you look at it, there’s always tons of crufty #defines, tricks to avoid C’s legacy problems, spinlocks and stack arrays and overflow checks and… so it’s no wonder Apple decided to start anew with a new language that avoids all of those problems and still interoperates with Cocoa etc. — all while the infrastructure’s being changed underneath.

So, why not C++? Lattner is a C++ wizard, right? All of Clang/LLVM is coded in C++. So is WebKit, Apple’s other major open-source success. I can’t see that changing, and their effort to fully support all of C++’s experimental future features argues that it won’t change. But C++ doesn’t look like a good match for internal Apple technologies like GCD and ARC, and the C++ Standards Committee is certainly not interested in adopting those. On the other hand, judiciously adopting certain things like generics, operator overloading and optimized dispatching is certainly a good thing. And last but not least, Apple now owns/controls the entire toolchain and the systems programming language!

More later; I’ve started to write an entire application in Swift and after that may feel qualified to comment on language details. For now, I’m quite happy with the prospects.

Quay 1.1.5 (357) is now out. This is compatible with the upcoming OS X 10.9.x (Mavericks). As before, be sure to update Quay to this new version before installing 10.9.

This will probably be the last version of Quay that comes out. Now that I’ve figured out how to get this type of application working under 10.9 it’s time to begin working on a new one…

Yay! Another update!

No comments

RB App Checker Lite 1.0.3 is now live both for direct download and from the Mac App Store. Here are the release highlights:

  • Now opens and checks .ipa bundles.
  • Opening packages (like .xcarchives) that contain applications now works properly, showing the package icon and path.
  • More and better help and credits text, now with active popups.
  • Library licenses have been included in the credits.
  • New interrupt/redo scan button.
  • The list of known entitlements has been updated.
  • Better explanations for most code signing errors.
  • Complains about missing receipt for App Store apps.
  • Pop-up file lists are now slightly better-looking.
  • Unsigned frameworks aren’t incorrectly flagged with a signing error anymore.
  • Fixed: the app froze after clicking the full-screen button in QuickLook preview.
  • Fixed: issues with mailto: links in the About window.
  • Fixed: the File->Select… menu is now enabled; it doesn’t work for frameworks, though; the workaround is to drag one onto the app icon or window.

And a most important note that I wasn’t allowed to mention in the app itself because of Mac App Store restrictions: the new version should be fully compatible with Mac OS X 10.9 (Mavericks).

On the travel front, we had a few wonderful days in the old villages in the interior of Portugal, and now an excellent week in Ireland. The weather, which was abnormally splendid, has today returned to its more normal Irish standard of wet and windy. More anon.

Yay! An update!

No comments

So, a long-delayed update.

A few days ago, and exactly 365 days after the last version (1.0.2, I dimly recall), I submitted RB App Checker Lite 1.0.3 to the Mac App Store. It should be out before the end of the week, I hope; watch this space for news. As soon as it’s on the store the developer ID-signed version will also be available for direct download.

There are a few new features and some bug fixes – the exact list will be out with the update. There are, also, many new features not directly visible by the user. In particular, RB App Checker (non-Lite) is now being built on almost exactly the same codebase. This will be a paid application, probably around US$16, with a new UI and explanations for the non-technical user – but the very detailed geeky stuff will still be visible with a click, don’t worry. It will also be able to scan the user’s Application folder.

Both versions of the App Checker build on a generic application framework that will make it easy for me publish more file-twiddling utilities. Two of them – one to count and scan folder contents, one to generate various types of aliases and file links – are already in alpha and should be available before the end of the year.

In parallel, I hope to, very soon, restart work on my next-generation System Preferences panel – the one that will obsolete and subsume my previous apps like Quay and (perhaps) Klicko. If all works out as I hope, this panel will be able to leverage the RB Utilities  to get extra funcionality not allowed by the Mac App Store, and centralize preferences and auto-updating for my non-App Store utilities.

In a day we’ll leave for a short vacation in Ireland, followed by a visit to lovely Köln and the Objective-Cologne conference, where I’ll present a short talk on (ahem) “Coding Secrets of the Ancients”. Well, the subject is a little misleading – there will be a section about history and reminiscences about early computing, but there’ll be a very practical and up-to-date section about protecting applications in the Mac App Store; tricky stuff like receipt and certificate checking. More details should be up soon at the ObjCgn website.

We’ll also seize the opportunity to visit friends, relatives and developers in Germany, and should be back early in October. I’ll post updates here whenever possible.

Back, Busy

No comments

Time flies. Our 4-week trip to the US Southwest’s parks has been over for almost 2 weeks already. The trip itself was marvelous; one of the best in recent years, even though we had to skip a few parks – it was just too tiring. At least we now have a shorter list of places to revisit in the future. We then rested for a week in Denver and nearby Boulder, which are wonderful places, too. All in all we drove about 5000km (3000 miles). Here’s my updated map of visited US states:

In other news, I’ve been busy fighting off jet lag, catching up, and taking care of various private issues. If all goes well, coding (and therefore, progress on my applications) should resume this weekend. Don’t go away…

RB App Checker Lite 1.0.2 is out. Sorry for the delay; to compensate, there are some neat new features and a lot of infrastructure work has been done to prepare for the upcoming non-lite version.

Please stand by…

No comments

…while Apple catches up with sent-in FAXes. Yes, you read that right.

In the meantime, RB App Checker Lite is momentarily off the Mac App Store. (It’s still normally available from the product page, of course.)

With very few interruptions, I’ve been a registered Apple Developer since mid-1984, when I bought the original loose-leaf edition of “Inside Macintosh”. From a reasonably early date onwards, I always paid by credit card. At some point – I think it was the late 1990’s – most developers were switched to paying over the iTunes store – quite practical in the US and Europe, since that was already tied to the Apple ID.

However, Brazil was very late in getting the iTunes store and from then on developers here had to send in an annual FAX stating their identification, credit card number, and so forth. A few years ago, the the iOS app store, the iTunes store, and the Mac app store (in that order, I think) became available and every time we heaved a collective sigh of relief, thinking that easy payment would be again possible.

Unfortunately, it’s now late 2012. All the stores are available but we still have to send in a FAX to Cupertino to renew our developer programs! No shipping Mac even has a built-in FAX-Modem anymore, which was what I used in the early years. Twice, I’ve asked developer relations at WWDC to send in the FAX for me – which they did, to their credit; and for the last few years I’ve asked their counterparts at Apple Brazil to do so, as they’re the only place I know of which still has a FAX machine stashed away somewhere. Still, the renewal was effected in a few days.

When I renewed my iOS membership a few months ago it took an unprecedented 7 days to be effected after sending the FAX, so when my Mac membership renewal came up, I thought it prudent to send in the FAX 20 days (!) before to ensure a timely response.

Unfortunately, the 20 days have come and gone, my renewal application still hasn’t been processed, and I’ve been cut off from the developer sites. Moreover, my app is no longer visible in the Mac App Store. I’ve already complained twice to Developer Relations and re-sent the FAX (both by machine and by email) 3 times; at first they admitted that it should have taken at least two weeks (!) to process, but haven’t replied since.

In the meantime, the new version of RB App Checker Lite had been in the review queue for 15 days and was finally rejected this week; first because of the pending renewal, second because of a temporary entitlement that had been accepted in the first version. I’m working on that. More soon (hopefully).

Update: just got email from Apple, stating that they put in a temporary extension while my renewal is being processed. The app isn’t back on the store yet, but at least it gives me breathing time to submit a new version; perhaps as soon as tomorrow.

Update #2: the app is back on the store. Stay tuned…

Update #3: my renewal has been processed, and we’re back to normal. Now working on the app again.

Recently someone figured out an attack against in-app purchasing on iOS. Only a few days later Apple, with commendable speed, put up a page detailing how to counter this crack by implementing better receipt checking.

Now there’s news that a similar attack also works on OS X. For this, users have to install two bogus certificates, point their DNS at the cracker’s server, and run an auxiliary application while making the in-app purchase; this builds an apparently valid receipt inside the application bundle. (Of course this means that the user is trusting those certificates, that server and that application to be otherwise innocuous – not a good policy! And it asks you for your admin password while you’re connected to that server, too…)

So how to implement better receipt checking on the Mac? The details are different, in that OS X in-app receipts are stored inside the application bundle, inside the application’s original receipt from the Mac App Store. Furthermore this receipt has a known format and is signed by 3 certificates:

  • “Mac App Store Receipt Signing” (SHA1: 4A 7B 3A 17 00 A4 DA 4A D4 EA 43 3A 83 61 43 2E CF 1C A1 AF)
  • “Apple Worldwide Developer Relations Certificate” (SHA1: 09 50 B6 CD 3D 2F 37 EA 24 6A 1A AA 20 DF AA DB D6 FE 1F 75)
  • and the “Apple Root CA” (SHA1: 61 1E 5B 66 2C 59 3A 08 FF 58 D1 4A E2 24 52 D1 98 DF 6C 60)

the latter can be found, of course, inside every Mac user’s System Keychain. All this was easily obtained with my RB App Checker Lite utility (ahem). 🙂

Apple’s currently recommended receipt verification code, however, does not contain any recommendation to check the certificates used to sign the receipt; it does check if the receipt is for that particular application (otherwise people would just copy a receipt from another app) and if the receipt was generated on that particular computer (otherwise one could just copy the app from a friend’s computer).

No doubt Apple will now recommend to all OS X developers that their receipt verification codes also check the certificates – and in fact, that’s what my apps are already doing. The certificates are, after all, available from the same parsing process recommended in the above link. At the very least, I recommend obtaining the SHA1 fingerprints of all 3 certificates (openssl has a SHA1() function for that) and checking them against the list above. And once that’s done, obtaining the app’s own signing certificates, and checking them, is also advisable, even if the app is signed with a Developer ID.

I’m not giving specific code examples here, to avoid people copy&pasting it into their apps and offering a clear path to hacking the binary. The usual precautions to make binary hacking more difficult (though it’ll never be impossible) apply, of course.

Photos licensed by Creative Commons license. Unless otherwise noted, content © 2002-2022 by Rainer Brockerhoff. Iravan child theme by Rainer Brockerhoff, based on Arjuna-X, a WordPress Theme by SRS Solutions. jQuery UI based on Aristo.