Solipsism Gradient

I was going to post some follow-ups to my MP3Concept “trojan” comments tody but it turns out John “Daring Fireball” Gruber already did it very nicely for me. (The only fault I can find in his article is that he didn’t link to me … oh well, can’t have everything.

Meanwhile, it’s interesting to read Symantec’s security response on the subject. This may be the closest a company that sells antivirus software may come to saying “don’t worry about this”.

I’ve just posted a new installment of my Interesting Times column, after quite a hiatus. This time, I’m commenting on the recent MP3Concept Trojan flap. Here are some highlights:

…Let’s see what we have so far:

– You can name an application anything and have it show any icon you wish. Always could.

– You can code an application to do anything, even harmful or deceitful things.

– The Finder will always show if it’s an application – but it can’t protect you from misleading icons.

– If you download such an application over a browser, file types, resource forks, or folder structures won’t be properly preserved unless it’s in an encoded form such as .bin, .zip, .sit or .dmg.

– If you get such an application as an e-mail attachment, most attachment formats will preserve resource fork and file type, but most e-mail clients will present an alert when you double-click on an application that came in as an attachment.

– Nevertheless, if there’s a deceitful document icon and an enticing name the user may double-click on the application even if there’s no extension at all.

…

What can the user do, then? Standard cautions apply. Don’t download applications from unknown sources. Check data files in the Finder to see if the “kind” field says “Application”. Don’t trust file icons on downloaded files. Don’t double-click on files you’ve downloaded; either drag the file onto the application you want to use it with or use the “Open With” contextual menu (or my Zingg! contextual menu to open it in a specific application. Don’t trust antivirus programs either, as they will either give you too many false positives or a false feeling of security. Disable automatic unstuffing, unzipping or running of downloaded items.

…What could Apple do? Options are limited here. Recall that this is not a Finder or Launch Services bug, so there’s no obvious fix.

…Summing up, a “social engineering” trojan application is not a new concept, but now that attention has been called to it, we may begin to see malicious implementations of it, more’s the pity.

Read the whole article if you want more details…

As most readers will have noticed, the Mac press for the last few days was full of scary and conflicting news about a new “MP3Concept” virus or trojan.

It started on April 8 when Intego, a company that sells several security and antivirus programs, announced “Protection Against the First Mac OS X Trojan Horse”:

This Trojan horse, MP3Concept (MP3Virus.Gen), exploits a weakness in Mac OS X where applications can appear to be other types of files.

The Trojan horse’s code is encapsulated in the ID3 tag of an MP3 (digital music) file. This code is in reality a hidden application that can run on any Macintosh computer running Mac OS X.

Mac OS X displays the icon of the MP3 file, with an .mp3 extension, rather than showing the file as an application, leading users to believe that they can double-click the file to listen to it. But double clicking the file launches the hidden code, which can damage or delete files on computers running Mac OS X, then iTunes to play the music contained in the file, to make users think that it is really an MP3 file.

Apparently they’re referring to this March 20 post by Bo Lindbergh, where he presents his proof of concept virus.mp3 application:

Download and unstuff to get a Carbon/CFM application that runs on both Mac OS 9 and Mac OS X. The PowerPC code fragment is stored as a general encapsulated object inside the ID3 information. It tries to locate iTunes and tell it to open the file as audio, displays an alert, and then quits. (If you’re paranoid, create a throwaway account to test it with.)

…this was in response to a thread where people were speculating on the feasibility of writing a “social engineering” virus as recently seen on other platforms.

Intego’s press release misrepresented some aspects of the issue; in particular there’s no specific “weakness in Mac OS X where applications can appear to be other types of files”: it has always been possible on Mac OS (since System 1.0 in 1984!) to write an application, put a document icon on it, and name it in a way that it appears to be a document. (More about that later on.) However, many people understood that to mean that this exploits a bug in Mac OS X. Even such a usually well-informed site as MacFixit fell into this trap, reporting that:

…the OS X Finder can be fooled to represent an application as a file. This occurs because the Finder depends on two different sources of information for how to display an object: it uses both file type/creator codes, for compatibility with OS 9; and also can use .extensions, introduced in OS X.

In this instance, the file type is set to “application”, and the .extension is set to “file”.

This “exploit” is only one step further from simply renaming an application file with a different extension, “.mp3” for instance. The only difference is that Mac OS X can be fooled into actually launching an application based on the false file type/creator codes and .extensions – iTunes for a file with a .mp3 extension for instance.

This is quite wrong. The Finder is not fooled at all and correctly identifies the file as an application; this can be seen in both column and list view, and in the “Get Info” window. Only in icon view (does anyone use this anymore?) can the user be fooled.

In reality, the ‘APPL’ file type takes precedence over the extension, so double-clicking it launches it as an application. The MP3 icon is not generated by the Finder in response to the .mp3 extension; rather, it’s contained inside the file itself as a normal application icon – which happens to be identical to the iTunes document icon. After the application launches, it in turn launches iTunes and passes itself to iTunes. This makes use of the fact that an MP3 file consists of a sequence of data blocks, each one identified by a tag. For this particular type of application, the executable code can start anywhere inside the file’s data fork which can contain unused space both before and after the code; so from the standpoint of iTunes reading the file’s data fork, the code is hidden inside an unused part of the identification (ID3) tag’s data block.

It’s easy to check this out. Save some file (from inside TextEdit, for instance). Open it in my XRay utility, change its name to something.mp3, set its file type to ‘APPL’ and clear the creator code. After saving those changes, it will appear in the Finder as a generic application, not as an MP3 file. Double-clicking it will give the usual icon zoom-out effect but nothing will happen – and iTunes won’t be notified.

Yes, XRay erroneously will say that this file, despite being an application, will be opened by iTunes; it won’t. That would be true only if it were a Classic application, in which case it wouldn’t be run at all. I’ll have this fixed in the next version…

The day after, Intego had a new press release out, where they attempted to explain some details:

First of all, Mac OS X runs two types of applications: Cocoa and Carbon. Cocoa applications are native OS X applications, and have an .app extension. Cocoa applications are, in fact, folders containing all the bits and pieces of a program code, resources, graphics, etc. The .app extension tells Mac OS X that an application is going to run natively.

Carbon applications are different. Most Carbon programs can run under either Mac OS 9 or Mac OS X, and, for this reason, have no .app extension. The Mac OS knows they are executable programs because of two resources, carb and cfrg. The carb resource indicates that it is a Carbon application and the cfrg resource indicates the location of executable code in a file’s data fork.

This Trojan horse is, in reality, an MP3 file, to which the two resources mentioned above (carb and cfrg) have been added. In addition, the ID3 tag of the MP3 file contains the actual code of the Trojan horse and the cfrg resource contains a pointer to that location in the file’s data fork.

Not at all. Firstly, Carbon applications, too, can be bundled and have the .app extension; just look at any recent Adobe or Macromedia program for an example. Single-file applications are recognized first by the ‘APPL’ file type and then further defined by several resources in the resource fork.

Single-file applications can be in several formats. Legacy 68K Classic applications (like ResEdit) have ‘CODE’ resources where the executable code is; the data fork can be empty or can contain anything at all. In the old days, it would have been easy to write an application that had a deceptive name and icon, and valid data for some document format in its data fork. As soon as Apple Events came out, convincing another application to open the data fork would have been easy too – except that most applications in those days would also check the file type and reject any ‘APPL’.

When the PowerPC came out, ‘CODE’ resources were deprecated and code was moved into the data fork; it used a new format called PEF (Preferred Executable Format). PowerPC code could come in several “code fragments”, so they were indexed from a ‘cfrg’ resource – that’s why these applications are also called CFM (Code Fragment Manager) apps. This resource is a list of code fragments and for each one it contains (among other data) the length of each fragment and its displacement within the data fork. Of course, normally there’s only one fragment which starts at zero. So, the programmer is perfectly free to put other stuff into the data fork as long as it doesn’t interfere with the parts that contain executable code. With a tagged data format such as MP3, TIFF, GIF, QuickTime (and so on…) that’s easy to do; just include the code into a tag data block that’s unused, or invalid, or reserved for comments. I used this technique in a dictionary application I wrote some years ago; the database is embedded into an unused part of the data fork.

When Mac OS X came out, the old Classic programming interfaces were redone into what became known as Carbon. To distinguish Carbon applications that could run natively on Mac OS X from the Classic apps, the ‘carb’ resource was defined. When it is present, the application is known to use the Carbon interface.

The native Mac OS X executable format, for a variety of reasons, is not PEF but Mach-O. Although most Mach-O applications are of the bundled variety, you can also write single-file Mach-O apps (AppleScript Studio can generate such applications). These are very similar to the CFM apps described above, except that there is no ‘carb’ resource and the code is indexed by a ‘CFRg’ resource instead. By the way, several people are proposing fiddling with the “executable” permission bits on downloaded files. That won’t work; Mach-O applications need them to be set, and they’re just ignored on CFM applications.

Coming back to the Intego press release, I think they have it the wrong way around. The sample application is a normal CFM app with several resources (not just ‘cfrg’ and ‘carb’). The writer added MP3 data blocks to its data fork and shifted the displacement in the ‘cfrg’ resource accordingly.

Now let’s look at the bundled application format. It’s actually a folder with the .app extension (which is always hidden by the Finder) and a certain folder structure. The actual executable can be PEF or Mach-O – it can even be a dual-fork file – but it’s hidden a few levels down in the folder, and other subfolders can contain anything whatsoever. So, it would be easy to write a Cocoa application called whatever.mp3.app (the .app would be hidden), define the application’s icon to be an MP3 document icon, and the application could then pass an actual MP3 file to iTunes when it executes.

All those formats – except for the bundled format – depend on the resource forks and file types to work. So usually, unless you download a deceitful application in .bin, .sit or .dmg format, the resource fork goes away and you’re left with a simple data fork.

So. Let’s see what we have so far:

• You can name an application anything and have it show any icon you wish. Always could.
• You can code an application to do anything, even harmful or deceitful things.
• The Finder will always show if it’s an application – but it can’t protect you from misleading icons.
• If you download such an application over a browser, file types, resource forks, or folder structures won’t be properly preserved unless it’s in an encoded form such as .bin, .zip, .sit or .dmg.
• If you get such an application as an e-mail attachment, most attachment formats will preserve resource fork and file type, but most e-mail clients will present an alert when you double-click on an application that came in as an attachment.
• Nevertheless, if there’s a deceitful document icon and an enticing name the user may double-click on the application even if there’s no extension at all.

In sum, any application can be a Trojan, but the user has to be careless to be affected. Now let’s look at what can be done. Apple has published a cautious statement saying that they’re aware of the issue and “investigating”. Intego, Symantec and other antivirus makers have announced that their latest upgrades detect that particular application. I suppose they detect non-zero offsets in the ‘cfrg’ resource and emit a warning; however, this will of course catch many legitimate applications (such as the dictionary I mentioned earlier). But as we’ve seen, even a Cocoa application can be deceitful.

What can the user do, then? Standard cautions apply. Don’t download applications from unknown sources. Check data files in the Finder to see if the “kind” field says “Application”. Don’t trust file icons on downloaded files. Don’t double-click on files you’ve downloaded; either drag the file onto the application you want to use it with or use the “Open With” contextual menu (or my Zingg! contextual menu) to open it in a specific application. Don’t trust antivirus programs either, as they will either give you too many false positives or a false feeling of security. Disable automatic unstuffing, unzipping or running of downloaded items.

What could Apple do? Options are limited here. Recall that this is not a Finder or Launch Services bug, so there’s no obvious fix. I see the following possibilities:

1) Dave Schroeder says on MacNN:

The only way for Apple to “fix” this would be to universally visually identify executable applications in some fashion.

I’ve seen this suggestion somewhere else too, but couldn’t recall the URL. Mac OS X could overlay a badge over any application icon, like it already does with aliases and locked files. The downside is that this would be visible only in the Finder and not be easily visible in all but the icon view… and there adding an “(application)” text tag might be more appropriate. And overlaying a huge red “caution” badge would probably be met with howls of protest from icon designers .

2) Apple (or someone) could patch LaunchServices to throw up a warning when a previously unregistered application is executed. This could be general, suppressed for applications in the standard /Applications folders, or even be shown only for certain folders… perhaps the user could mark certain folders (such as the default download folder) as not allowing applications to be run. This would probably catch most sneaky applications, at the cost of some extra work for the user. A ready-made alternative would be to extend the “Limitations” tab in the System Preferences to administrators.

3) Apple (or someone) could patch LaunchServices to check the application’s extension and see if another app is already claiming to handle this extension. This is already done for Classic applications, mainly for the special case of Classic .smi applications, which were self-mounting disk images but now are handled as files by the disk mounting utility. They would have to ignore a hidden .app extension if present and do some other extra checking. This might slow application launching down a little and would not, of course, catch Trojans that have a document icon and no extension.

On the other hand, most proposals I’ve so far read on the net are impractical, such as:

• Turn off executable permission flags by default (with umask or some other way): wouldn’t affect CFM Trojans.
• Deprecate CFM applications completely: will kill Adobe and Macromedia (and hundreds of others) and won’t affect Mach-O Trojans.
• Disallow extensions in application names: will kill compatibility with many applications who put version numbers in the name, and won’t affect extensionless Trojans.
• Remove the resource fork of suspicious files: will work against single-file Trojans but not against bundled ones.
• Wait for antivirus software to catch up: unrealistic, as they usually scan for certain binary patterns, and there’s no general pattern to Trojans. They can catch specific programs after the fact, but not pre-emptively.

Summing up, a “social engineering” trojan application is not a new concept, but now that attention has been called to it, we may begin to see malicious implementations of it, more’s the pity.

Still more comments on the Ronco Spray-On Usability debate, itself (just to remind people) sparked by this ESR article.

Matt Gemmell, the Irate Scotsman, weighs in with a long and reasoned article:

UI must be designed from the start. Interface considerations infest your design choices, like it or not. Functionality can rarely be completely divorced from the controls which will trigger and modify it. Output must often lead to feedback for the user. The core modes of the application entirely determine the user’s type of experience. Like it or not, we have to realise that as much as it might offend our software engineering sensibilities, for the user the UI is the software.

…Hopefully it’s clear that interface and interaction design is a fine art, and is incredibly difficult to do well. It requires a huge amount of knowledge of your software’s platform, purpose, target market, and indeed of human psychology both in general and in the specific field of human-computer interaction. Certainly, it’s not for the faint of heart, and nor is it to be rushed or attempted without due experience and consideration.

Likewise, Rui Carmo at The Tao of Mac also writes at length:

…usability is a lot of hard work spent on ironing out even the smallest wrinkle. It is not something you “spray on” after the fact. It is either part of the design from the outset, or it isn’t. No amount of paint will disguise hard, unforgiving edges.

Usability has no tolerance for hard edges – the only kind of edge users notice is the bleeding edge, and guess what, they don’t like it.

Both are extremely recommended reading for interface designers.

John “Daring Fireball” Gruber has posted an update on his previous Ronco Spray-On Usability article, responding to e-mails and comments:

…Perhaps the biggest misconception is that I’m somehow “rooting against” desktop Linux. I really don’t see how anything I’ve written implies that, unless you subscribe to the “if you’re not with us, you’re against us” school of thought.

Regardless, it’s not the case. I’m not rooting against desktop Linux, nor have I ever claimed it can’t succeed. What I am saying is:

1) It hasn’t succeeded yet.

2) It’s unlikely to succeed without direction and substantial commercial support.

Where by “succeed” I mean “provide a terrific user experience”.

…

The key is that there’s never going to be a good desktop user interface for Linux that pleases the Linux nerds who don’t care about usability. If the reason you use Linux is that you value tweakability over usability, or if you get off on the fact that a normal person couldn’t sit down in front of your computer and figure out how to use it, you’re probably not going to like a system that doesn’t even have a replaceable “window manager”. Trying to create a cohesive GUI system that appeals to these guys is like trying to write music that appeals to the tone deaf.

An aggravating agglomerated aggregate of glitches glommed onto me yesterday. After nearly 10 years without losing any significant data, an errant shell script (in an Open Source project I had just downloaded from the net) removed a goodly part of my Home folder. I noticed things crashing and disappearing, finally deduced what must have been happening, and killed the miscreant task.

Of course, Murphy decreed that no recent backup of these data existed… the external FireWire drive I had been using for backups turned out to be unreliable over a mont ago and I’m still searching for a replacement.

The BSD/Unix underpinnings of Mac OS X imply that many actions that involve command-line stuff fail when confronted with file and folder names that contain characters classic Mac users are prone to use: hyphens, spaces and accented characters all make naïve shell commands go astray. (This rarely if ever affects BSD tools called directly from normal applications, as there’s no parsing of the actual file names involved, and by Mac OS X all filenames use UTF-8 encoding).

In the days of the public betas and Mac OS X 10.0.x, I recall running into several such glitches, mostly while running Apple’s now-obsolete Project Builder programming environment. Project Builder (and its successor Xcode) invoke the gcc compiler and associated tools through complex shell scripts, and I still run into snags now and then when accented characters are involved. Still, Xcode has been remarkably tolerant of embedded spaces and for the past year I’ve consistently used spaces in my current folder names without any trouble; mainly for having the projects I’m working on list first in my somewhat bloated Home folder.

The leading space, unfortunately, was the cause of the aforementioned shell script interpret a “remove all nested folders from /Users/rainer/ Test Project/subfolder/” command as “remove all nested folders from /Users/rainer/, from Test, and from Project/subfolder” (paraphrased). There are two ways of writing such things correctly: one is to precede embedded spaces with a backslash, another is to enclose the whole file name in quotes. Because of the somewhat convoluted nature of the shell script, which passed a list of search results to the actual remove command, the author did not consider that the list might not be properly formatted…

Besides my current project folders my Desktop and Documents folders were removed. On the desktop I kept mostly recently-downloaded items, which can always be downloaded again as soon as I need them. The Documents folder was more painful to lose, as it contained my address book, e-mail folders and several other files that contained ongoing corresponce.

So I spent most of yesterday and today trying to restore my working environment. I tried some disk rescue utilities without success. I wrote a disk image of the volume I keep my Home folder on to an improvised external drive, only to find later that this doesn’t copy unallocated blocks and is therefore unsuitable for data rescue.

Fortunately, my Home volume was nearly full, meaning my working files had a good chance of having been allocated near the end of the volume – and indeed some judicious albeit tedious scanning allowed me to recover most of the source files I had been working on for the past month. Whew.

Unfortunately, the same trick won’t work for e-mail; there’s just too much of it and the fragments can’t be fitted together as easily; and a CDR backup I did last month proved unreadable, too. So, if you e-mailed me anything important since Christmas, please send it again; sorry.

Quick update.

A few hours after posting the lamentation below I found out how avoid the text cursor. Both selecting and editing have to be disabled. Turning editing on (as I do, briefly, when inserting new stuff) turns selecting on again.

Turning all that off messed up some other functionality but I’m happy to report it was all easy to fix, and I now seem to be where I should have been a month ago.

Ah yes, and the contextual menu itself turned out to be a piece of cake… most of the code from Zingg! and Nudge could be reused.

So I hope to have something publishable Real Soon Now™…

This will take some explaining, and may not make much sense unless you’re a Cocoa developer. So feel free to skip this.

In early February – about a month and a half ago – someone asked an obscure question on the developer lists and, by chance, I had a code sample that did almost exactly what that guy wanted. So I sent it to him. So far so good… warm fuzzy feelings all around…

…it turned out that this gave me an idea for a new contextual menu for the Finder. So I thought, the new CM will be just that code sample with some padding, so let me start first on the application to configure stuff for the CM. Shouldn’t take more than a week or two.

As usual, I started out by designing the application icon first, which sidetracked me into a long investigation about the habits and aspect of a certain mammal. That done, I decided the user interface should consist only of a list of items. Fixed-size items each consisting of a few icons and a name. Of course, the first choice was using a NSTableView.

It turns out that the icons were too dissimilarly sized, so a table with one row per item showed too few items, and the UI looked ugly. So my next idea was to use a NSMatrix of cells, each cell with the subitems artfully arranged. This looked better, but…

I soon noticed that a NSMatrix operates with rows and columns of cells (duh!). I wanted a view that adapted itself to the window size the user wanted, but having to scroll both horizontally and vertically was ugly as soon as there were more than a dozen items. So I messed around for a couple of days with adapting the number of columns to the window width, and couldn’t get it to work reliably.

So I decided that I wanted something like iPhoto’s browser window, where there’s only a vertical scrollbar, and the view adapts instantly to width variations. So I started to write a custom view that would be inserted into a NSScrollView and do automatic layout of as many items fit into the current width.

After a week of this it was almost working… however, I was getting some strange behavior in certain conditions; when the window was resized very rapidly, or when the scroller had to appear during insertions, or, and so forth. It appeared that I wasn’t managing the communications between the NSScrollView and my own view correctly under all circumstances.

So I was trying to trace when NSTextViews communicate with their NSScrollViews, when it hit me that I could simply use a NSTextView and insert the item as specialized NSTextAttachmentCells. Bliss! This would give me all the required scrolling and resizing behavior automatically, and I would just have to write the specialized cell code and disable some of the NSTextView’s functions… it wouldn’t do to allow the user to insert actual text there, for instance.

There I went off into another two-week-long detour of writing and debugging the cell code. My cells consist of several subentities which are actually NSViews themselves – images and text – and they in turn have to be editable, selectable, and so forth. I’ll leave the saga of how I finally solved this for another post – it involves adding a subview to the NSTextView, something not normally done.

So after that I had to decide which text behaviors I wanted to leave in the NSTextView. No editing, OK. Selecting multiple items with the text cursor? Hmm… it looked interesting. The user could select several items and drag them somewhere! Perhaps for backing up the items into a Finder window – or to make a list of his items into a text windows. I happily set to work implementing routines for copying and dragging the current selection in various formats, which in turned allowed me to exercise my code under unforeseen conditions and decide how the items would be stored and saved.

Then when it was nearly finished I started to run into snags again. Suddenly I wanted to select two non-adjacent items, which NSTextView doesn’t allow. I changed my system-wide selection color (for a non-related reason) and suddenly selections in my NSTextView began to look ugly – worse, the way selections were made called the user’s attention to the fact that this was a NSTextView with much reduced functionality. This is never a good idea; it leads the user to expect certain things to work, and when they won’t work, it’s a negative experience.

So today I disabled all text selections wholesale, as well as all related functionality. This enabled me to throw out several chunks of code and someday might allow me to implement discontinuous item selections, too. So the NSTextView really just handles reflowing of my item list, and all the remaining functionality is shuffled off into my specialized NSTextAttachmentCell class.

So far so good; I expected to finally freeze my UI and get to work on the shareware related aspects – registration, trial expiration, and so forth. However, I still have to extirpate the final clue that I’m using a text view, namely the I-beam (text insertion) cursor. So far I’ve hammered away at this for the whole day without finding a solution that works all of the time.

As usual. 😥

Stay tuned…