Interesting how some design choices you make early on cause a “snowball effect” – many product details end up depending on these choices. Sometimes you have to go back and change the initial direction of the snowball, so to speak, because some of those end effects turns out to be unacceptable.

When I started rewriting Quay for the first 1.1 public beta, I decided to change the way the background application (QuayMenu) was run. In 1.0, a Quay icon in the Dock was a file hidden away inside the Quay database package. When the user clicked on the icon, QuayMenu started up (if it wasn’t running already), and presented the menu. This simple scheme interfered not at all with the Dock itself but depended on several tricks to find out, approximately, the on-screen location of the clicked icon – something that a normal application doesn’t need to know. (It also didn’t work at all if you accessed the Dock over keyboard navigation.)

For 1.1, I decided to convert QuayMenu into a LaunchAgent: a per-user background application that runs constantly and is restarted by the system if it fails. I also worked out a way to have QuayMenu monitor click and keyboard events for the Dock, and use the Accessibility APIs to work out exact locations and details of a clicked Dock icon. So when the user clicks on a Dock icon, QuayMenu checks out if it ought to handle that click and pass it on if not.

This design decision had immediate consequences. It’s much easier to control (and update) a LaunchAgent if it’s in a fixed, known place. I decided to store the main Quay application inside the Quay database itself (which is in ~/Library/Application Support), both to prevent the user from moving it and to have a single known location for updating. The idea of having everything inside ~/Library stems from complaints I had about XRay storing stuff inside /Library, years before. So there was an a priori concern to make Quay a simple, no-hassle, per-user application; nothing stored outside the user’s home folder, no administrator password needed, no security concerns.

However, this immediately conflicted with the need of using the Accessibility API. Basically, if you use this API to ask other applications about their user interface, you have two options. You can wimp out and ask the user to turn on “Enable access for assistive devices” in the Universal Access preference panel, or you can run a “trusted” accessibility client. The first option means that you depend on the user having this turned on all the time; if it gets turned off, things stop working and you have to show a dialog asking for it to be turned on again; clumsy. Also, some people (including myself) don’t like turning this on because it changes the way tabbing between text fields behaves – it then tabs over buttons too.

The second option – running as a trusted client – was the better one, then. However, it too comes with a trade-off. A trusted accessibility client runs “setgid 90”, meaning the executable is forced to run from the special “accessibility” group. This is a tamer version of the tricky, all-powerful and potentially unsafe “setuid root” executable; its only advantage over a plain vanilla executable is that it can see (and affect) other applications’ user interface elements. However, there’s one common aspect; to turn the setgid bit on, you need to ask for an administrator password. As a side-effect, copying the executable to another place turns the bit off again, and it’s ignored altogether if run from an external volume or a disk image.

All this meant that I would have to write an installer to move the applications into their place, ask for an administrator password and turn on the setgid bit. Apple’s guidelines say that in such a case you should do a standard Apple Installer package and have all this accomplished by scripts inside the package. I decided against that for several reasons.

One, it’s harder to check an installer package against unauthorized modifications, while an installer application can use code signing to prevent those. Personally, after some bad experiences with badly-written install scripts, I distrust installer packages a bit more than separate installers – although I know people who distrust installers even more. The third option, of writing a self-checking, self-repairing application that could be drop-installed anywhere – as Quay 1.0x was – I reluctantly discarded. Some people prefer to have multiple copies of applications available, some use weird separate application folders (as was usual in the Classic days), and if you back up your applications with Time Machine, you’d have copies right there which shouldn’t be accessed except when restoring from the backup volume.

So the decision of writing a suitably self-explanatory and “just works”-type installer appears to be the right one. It’s actually working pretty well in the current (1.1b2) beta… except that I’d forgotten one important thing. FileVault (on the list of system features I don’t use myself) is based on a special “sparse” disk image that is mounted in place of your home folder. Meaning that, if you use FileVault, you can’t have setuid or setgid executables inside your home folder – and certainly not inside ~/Library.

OK, so I’m dutifully rewriting my installer to move everything into /Library which has no such limitations. An added advantage is that now there’s a single copy of Quay installed for all users; but should the serial number, then, also be valid for all users? That means the added hassle of separating the preferences into yet another file, installed into /Library/Preferences; one more thing to uninstall. An added disadvantage is that, now, the uninstaller – which I’m building into the Quay application itself – will need to ask for an administrator password, too. Or maybe I should have a setuid root uninstaller tool inside the application; I don’t like that either, but having such a tool would also mean an easy way of adding extra functionality to the Quay popups.

Well, all these are trade-offs, and I hope that when 1.1 (final) comes out, that the net effect will be positive. If all goes well, 1.1b3 should be out sometime next week, and you’ll have the chance to comment. Stay tuned.