Archive for the 'Software distribution' Category

How to make a custom bundle act as a file

Sunday, October 14th, 2007

Recently, Gus Mueller had a blog post about a new plug-in for his image editor Acorn. One thing I noticed is that the plug-in is delivered as a folder:

The plug-in is a bundle, but since it shows up in Finder as a folder, users can navigate into it and poke around. It's unlikely that anything really serious will come of it, but you never know what the average user will wreck when he or she goes exploring in the internals of something.

Fortunately, there are two solutions.

The ad-hoc solution: Turn on the bundle bit

The Mac OS X Xcode Tools include a program called SetFile that, among other things, lets you set the bundle bit on an item:

% SetFile -a B ImageIO\ Export.acplugin

Note that case matters here: If we use an uppercase B, the bit is set; if we use a lowercase b, the bit is cleared.

Those of you who came from the Classic Mac OS may remember that the bundle bit originally indicated that a file contained a 'BNDL' resource (which simplified scanning a folder full of applications to see which ones had some icons to display). I doubt Mac OS X uses it for that purpose anymore.

Its new purpose is for directories: Finder will check for the bundle bit, and if it finds it, it will list the directory as a package.

Packages are directories that Finder treats as files (such as .app bundles). That's exactly what we want. Here's how the plug-in appears after running the above command:

(Note that to get the change to show in Finder, you have to either quit the Finder—such as by logging out—or use AppleScript. I use a command-line tool I wrote that calls -[NSWorkspace noteFileSystemChanged:] with the path to the changed item.)

The permanent solution: Define all items of this type as packages

In the current version of Acorn, it's up to Jens Ayton and other plug-in authors to set the bundle bit on their plug-ins. I think a lot of host apps have this problem.

However, a host app can take care of this for all its plug-ins. All it has to do is have a declaration of its plug-in type (in Acorn's case, that's .acplugin) in its Info.plist, with LSTypeIsPackage set to true in the declaration.

Then, plug-in authors don't need to do anything, because all plug-ins of that type will be shown in the Finder as packages, simply because the declaration in the host app's Info.plist says so. No more setting the bundle bit; in fact, that bit is then ignored.

Posted in Programming, Software distribution | 3 Comments »

Appcasts on Google

Thursday, September 27th, 2007

If you publish an appcast, you should also publish a robots.txt file that tells search engines such as Google not index the appcast file. I came to this realization after searching for “appcast editor”: the first four hits, and several of the others, are all just appcasts.

There is no point to having these indexed by Google, so save the bandwidth and help people's search results—tell the search engines not to index them.

And I never did find an appcast editor. I guess I get to write it by hand in vim. (There's one in MacCode, but it's not working yet.)

UPDATE: I knew there was an appcast editor out there! It's Feeder, by Reinvented Software. Thanks to Andy Kim for pointing it out in the comments.

Posted in Software distribution, Toolchain | 4 Comments »

Mac app checklist in TaskPaper format

Wednesday, August 8th, 2007

TaskPaper is a to-do list app from Hog Bay Software. It's currently free during its public beta. Its main stand-apart feature is its plain-text document format.

I've just created a TaskPaper version of my Mac app checklist, to make it easier for you to follow it for each application you make. You can download it in plain format or stationery format.

UPDATE 18:06 PDT: I just added the Info.plist key names for the check-your-Info.plist item, so if you're missing those, you should redownload the file(s).

UPDATE 19:48 PDT: I also just added the sub-items of the ReadMe item. Again, if you downloaded it earlier, you should get a fresh copy.

UPDATE 21:54 PDT: Added advice to choose Prevent Editing from TextEdit's Format menu.

Posted in Programming, Software distribution | No Comments »

WWDC 2007 session videos are out

Monday, July 30th, 2007

If you attended WWDC, you can head over to ADC on iTunes and see what you missed.

Posted in Apple, Carbon, Cocoa, Documentation, GCC, Interface Builder, Mac OS X, Mach, PackageMaker, Programming, Python, Quartz, QuickTime, Safari/WebKit, Software distribution, Toolchain, Xcode | No Comments »

Disk images suck: An examination of why, and of the alternatives

Sunday, May 13th, 2007

Here is a rundown of every file format you can use to distribute your software, along with their advantages and disadvantages. They are listed in ascending order of effectiveness.

In case you're wondering, 7-zip and StuffIt X are not listed here because nobody has decompressors for them. I wrote this list for the context of software distribution; the days of saying “You will need StuffIt Expander to open this file” are long over.

Disk image

Advantage:

Sandboxes the application: If it won't work from the disk image, it probably sucks

Disadvantage:

Confuses the average user; they typically run the application from the disk image, then encounter problems when they try to delete the image file or (much later) run a Sparkle update

Back in March, an Adium user had a problem trying to perform the Sparkle update. It gave her an error:

“Update Error! Adium does not have permission to write to the application's directory! Are you running off a disk image? If not, ask your system administrator for help.”

This wouldn't be so bad, except that she didn't know what a disk image was, so she assumed that she was not running Adium from the disk image (I imagine she thought something like “surely I'd know what that was if I was using one”). She looked up the troubleshooting instructions in the Adium help (good), then critically misunderstood them (bad) with the result that she moved her Adium 2.0 folder out of the Application Support folder. (Don't ask me to explain it. I don't know, either.)

All this ultimately resulted from the fact that average users, including her, don't recognize a disk image when they see it. They don't expect a file to act as a drive.

The same non-obvious nature results in other problems: specifically, the error message “The operation cannot be completed because ‘SurfWriter-1.0.dmg’ is in use”. This occurs when the user tries to delete the disk image file without unmounting it first. Users who encounter this message end up contacting us (usually asking “how do i uninstall it”). This is only natural, because who would expect that a file is mounted, or that a file can be in use in another file?

It was then that I realized that disk images are not as great for the average user as I had previously believed.

You may be thinking “well, just put a background image with an arrow to an Applications folder”. We did that. It didn't help this user.

The stories I heard from other developers suggest that people in general zero in on the application they want to use, without paying attention to anything else (e.g., arrows, help text, or symlinks). Besides, the questions I hear from the users via the feedback list suggest that users see the disk image window and think the application is already installed—they aren't looking for or expecting installation instructions, so your big arrow means nothing to them.

I suspect that installers are partly to blame for that: I don't know about Windows, but on Mac OS, installers would always open the folder containing the freshly-installed application, so that you could use it right away. I suspect that users mistake the disk image window for a freshly-installed-folder window.

Anyway, one solution to the tunnel-vision problem would be a “DRAG THIS OVER HERE” message in a big font—but that's no good, because you can't have localized background images. You'd have to pick one language, and hope that all your users know it, but anybody who doesn't know the language you chose wouldn't benefit from your nice obnoxiously big text. (On the other hand, this provides a second reason for unilingual builds: One disk image per language, with both the background and the application localized in that language only. Hmmmm.)

Another solution is a runtime check whether the app is running from the disk image. You'll have to be careful with wording here, though—you need to be 100% sure that you're running from a disk image, and write the alert text accordingly. There can be no vacillation like in the Adium alert box (which vacillates because it's caused by a simple permission failure, not an actual search for the disk image nature); it must be a statement, not a question. And of course, you must tell the user the remedy (copy the app, then run the copy, then eject the disk image file and never mount it again).

One last point: bzip2 compression (-format UDBZ) requires Tiger. Your disk image won't mount on any earlier version of the operating system; your users will think it's corrupt, since that's what the error message suggests, and they'll contact you with that assumption. This probably doesn't matter anymore, since the Omni Group's software-update statistics say that 98.4% of users are on Tiger (as of 2007-05-12), but if you still support users of earlier OS versions, you need to use zlib compression (-format UDZO) instead.

Tarball

Advantages:

Smaller than disk images; unpacks into a folder or bare application

Disadvantage:

Safari mishandles bzip2'd tarballs pretty spectacularly; one must gzip one's tarballs to avoid that problem, but then one loses out on yummy bzip compression

Safari, for some reason, saves a bzip2'd tarball as “foo.tbz.tar”, even though it is still a tbz. Unpacking it results in a tar archive named “foo.tbz”, followed by the folder that the tar archive exists to hold. So the actual unpacking behavior works as expected, but it badly screws up the filenames.

This all works correctly with tgz, but of course you can't use gzip and have bzip2 compression.

Zip archive

Advantages:

Unpacks into a folder or bare application; can be created from the Finder

Disadvantage:

Bigger than anything else

Zip archives are the easiest to create, because you can simply right-click on the contents and choose “Create Archive of SurfWriter.app”. Unfortunately, the compression ratio is just not there; I consider zip files obsolete for this reason. You should be optimizing for your users' download time, not your own compression time.

Internet-enabled disk image

Advantage:

Unpacks into a folder or bare application for the average user (Safari with “Open Safe Files” turned on); perfectly normal for everybody else

Disadvantages:

Won't work the same way twice (when the user goes to unpack it the second time, it behaves as a perfectly normal disk image, with the attendant confusing UI); hard to create; unpacks slowly

This is the best of both worlds. The average user uses Safari and has “Open Safe Files” turned on; in this case, Safari will unpack the disk image just as if it were a tarball or zip archive. The sort of person who turns “Open Safe Files” off, or uses a different browser, is also the sort of person who can handle a normal disk image, and will indeed be handling it because that's how the disk image behaves in those cases. The extremely-rare exceptions can be handled by the aforementioned runtime check.

They're the hardest to create because you need to use a Terminal command (hdiutil internet-enable SurfWriter-1.0.dmg) to set the internet-enabled bit on the image. Dear Lazyweb: Please make a contextual menu item that generates an internet-enabled UDBZ disk image directly from a folder in one step, the same way I can make a zip archive in one step. (Michael Tsai, in a comment, says that his $20 DropDMG utility can do this, with Automator's help.)

Another disadvantage is that a disk image, internet-enabled or otherwise, takes much longer to process than a zip archive. I think this is because of the verification step, but the user won't care. I think people will put up with it for most archives, but if your archive is huge (let's say over 50 MiB), you may want to switch to a zip or tarball to save time. Of course, you'll be sacrificing part of your bandwidth bill for that. (Thanks to Sven-S. Porst for bringing this up in another comment.)

Also, the format notes (UDBZ vs UDZO) for disk image above apply to internet-enabled disk images as well. Just in case you were wondering.

I bring this up because, having in mind my objections above to disk images, I released EasyMD5 as a zip archive. I did it this way because EasyMD5 is targeted at Adium users (specifically, those who have problems downloading Adium); experience has proven that I can't assume that an Adium user will know how to deal with disk images. Everybody knows how to handle zip archives (hello, Chris!), so I made it a zip archive.

I hadn't yet done the study of file sizes that I did and published yesterday. Now that I have, later today, I'll replace the zip archive with an internet-enabled disk image.

UPDATE 11:37: Added mentions of Paul Kim's proposal of a runtime check for the disk image nature, and clarified the Lazyweb request.

UPDATE 20:47: Updated to include comments from Sven-S. Porst and Michael Tsai.

Posted in Mac OS X, Software distribution | 13 Comments »

Audio version of ‘Compression and archive formats’

Monday, January 16th, 2006

I made my earlier blog post into a podcast. It's just under ten minutes long, and in chaptered AAC format. Have a listen.

I used ChapterToolMe to do the chapters. It's nice.

Posted in Creations, Software distribution, This blog | No Comments »

How to distribute your software

Monday, January 16th, 2006

Recently, I went to Apple’s games site and downloaded a bunch of puzzle/arcade games. I was dismayed to find that many of these games were distributed in formats other than UDIF (the .dmg format). So clearly this matter bears some explanation.

This is the second part of a two-part post. Part 1 came previously; it discussed the difference between compression, archive, and disk-image formats and described the different specific formats in each category. Part 2 covers why you should use a plain dmg (no StuffIt or *zip sprinkles) to distribute your software. This was originally one long post, but I Kill Billed it.

Part 2: How to distribute your software

The sad current state of affairs is that much software is not distributed in a plain dmg. Here’s a brief hall of shame, in increasing order of silliness:

I’m not singling any of these products out (especially for .dmg.gz, .zip, and .sit, as these are absurdly common), nor am I critiquing the quality of the software; these are just examples of how not to distribute your application.

Regarding the last one: StuffIt format? WHY?!?!?! Mac OS X doesn’t even come with StuffIt Expander anymore!

And it’s not even StuffIt X (that would be .sitx); .sit is the older StuffIt 5 format, that doesn’t properly support application packages! Worse: because StuffIt 5 format does not support storage of UNIX permissions, StuffIt Expander relies on a preference setting to know whether to set a file as executable or not. So if you UnStuff an application from a StuffIt 5 archive, it might not even run. Doesn’t that make a great first user impression? And who’s going to suspect that their unarchiver is the cause of the problem?

In fact, it isn’t the cause of the problem at all. If you are still distributing your software in StuffIt format, you are to blame.

Panther added support in the Finder for zipping and unzipping files. Tiger added support in the Finder for gunzipping, bunzip2ing, and untarring files. All of these features are the wrong way to distribute an application.

First off, tarring a dmg is pointless. tar files exist to put multiple files into one file. A dmg is already a single file. There is no need to tar it.

Second, if you created your dmg correctly, it is already compressed (probably with zlib). ‘Correctly’ means making a read-only compressed image, either from a folder or from another (read/write) image. There is no need to compress it further. Be very surprised if the additional savings exceed 20 K. (the .dmg.tar.gz listed above is particularly egregious — hello, Mr. Redundancy Man!)

Third, if somebody on Jaguar (or earlier) downloads your zip file, and you created it using Finder, it will not unpack correctly in StuffIt Expander. Specifically, it will create ugly ‘__MACOSX’ folders. Your application might work (unless you actually did intend to use the resource fork for something, in which case it’s gone then), but littering a user’s system is not a way to build goodwill.

Fourth, the purpose of Finder’s zip support is to allow you to send files to Windows users. Distributing Mac software does not count.

There is only one valid use for zip: getting around stupid web servers that either don’t allow dmgs (*shakes fist at GeoCities*) or that don’t have a proper media type (application/octet-stream) set for it. If you have access to add that media type, do so, and distribute a plain dmg; otherwise, politely badger your admin. And until such time as they fix it, you should include a note on your downloads page acknowledging that having to zip your dmgs is lame.

Lest this article sound completely negative, let me explain the upsides of using UDIF.

The major advantage is that it sandboxes your application. An application shouldn’t install anything outside of its bundle (are you listening, Adobe?). If your application is written correctly, it is self-contained (everything it needs is in the .app bundle — this is what bundles are for) and does not require write access to the volume it is on. If your application can be run without problems from the dmg, then you are doing all right.

The second advantage is that a dmg does not require proprietary software (i.e. StuffIt). This seems to be a bigger advantage than it should be — StuffIt does not come with the OS anymore, and requiring people to download other software in order to use your software at all is not a good way to make them fans of it. Disk Copy/Disk Utility has always come with Mac OS X.

(Before you bring up Growl: Growl isn’t required by most of the applications that use it; it’s just value-added. Those applications that do require Growl are aimed specifically at existing Growl users.)

Third: with a dmg, you can easily embed a license agreement. There is no need for an installer to display it; Disk Copy/Disk Utility will do it for you.

That gives me an idea. Here are a couple of points about installers.

Unless you are installing a framework (that really needs to be installed into a Library directory) or a kernel extension, do not create an installer or an Installer package. There is no point.

The user should be able to try your application without installing it (remember what I said about it being usable from the dmg?). An installer is a barrier to this. You should make it as easy as possible for people to try your app.

Posted in Creations, Software distribution, Toolchain | 5 Comments »

Compression and archive formats: A description

Sunday, January 15th, 2006

Recently, I went to Apple's games site and downloaded a bunch of puzzle/arcade games. I was dismayed to find that many of these games were distributed in formats other than UDIF (the .dmg format). So clearly this matter bears some explanation.

This is going a two-part post. Part 1 follows. Part 2 covers why you should use a plain dmg (no StuffIt or *zip sprinkles) to distribute your software. This was originally one long post, but I Kill Billed it.

UPDATE 2006-01-15: Changed the initials to capital letters (sentence case), so it reads better; changed to smart quotes. Also added mention of Compact Pro and a link to ESR's article on The Art of UNIX Programming.

UPDATE 2006-02-17: Linkified Compact Pro.

Part 1: A brief history of compression and archiving

First, a couple definitions:

A compression format contains compressed data: data that has been fit into less space than it would without compression. gzip format is a good example of this; a gzip file is nothing but compressed data. It doesn’t even contain a filename — that’s worked out from the name of the gzip file. (for example, if you download Foo.tar.gz, and rename it Bar.tar.gz, and gunzip it, you will get a file named Bar.tar.)

An archive format aggregates one or more files into a single file. USTAR format (the format used by tar and pax; see below) is a good example of this. It isn’t compressed (unless you compress it with gzip, bzip2, or something else), but it is an archive format.

On Mac OS (that is, before Mac OS X), almost all software was distributed in StuffIt format. StuffIt was software developed by Aladdin Systems (now Allume Systems). (Before StuffIt caught on, the Mac market was majorly ruled by Compact Pro. But StuffIt, combined with apathy from the author, defeated it.)

The StuffIt format was both an archive format and a compression format: a StuffIt file was an archive of files whose contents were compressed. StuffIt achieved its greatest popularity with the advent of the internet — Mac OS applications could not be transmitted over networks, because most of their essential parts were in the resource fork, which was omitted when a file was sent to another operating system or copied to another file system. StuffIt, being data-fork only, kept everything intact. (MacBinary was another archive format, developed separately, that served that purpose, but did not offer compression.)

In the interest of being thorough, I should mention that MacBinary and BinHex were also used on top of StuffIt files (.sit.bin and .sit.hqx, respectively). MacBinarying a StuffIt file accomplished exactly nothing: you were wrapping what was already wrapped. BinHex was useful if the file was believed likely to undergo newline conversion (conversion between CR, which indicated a newline on the Apple II and Mac OS; LF, which indicated a newline on UNIX; and CRLF, which indicated a newline on DOS and Windows), but I think such situations were really the minority.

Eventually, Allume completely replaced the old StuffIt formats (there are at least four versions) with StuffIt X. StuffIt X supported much more detail in how the compression was specified, as well as Unicode filenames, 64-bit dates, packages (folders that behave as files, e.g. .app bundles) and stronger encryption. But it never achieved popular usage, for two reasons.

The first reason was simply that StuffIt 6-8 sucked. The larger reason is that OS X has its own facilities for archiving, compressing, and expanding files.

Mac OS had for many years (I used it on a Mac Plus and an SE/30, among other models) an application called Disk Copy. You’d open Disk Copy, then insert a floppy disk (80mm [aka 3.5"] disks were the prevailing medium of the day). Disk Copy would read from the disk, then prompt you to insert a blank disk or cancel. You could do this as many times as you wanted; thus, Disk Copy was a disk-duplication (i.e. copying) program.

In Disk Copy version 4.2 (I think), Apple added the ability to create disk images. A disk image is literally a snapshot of the contents of a disk. Disk Copy was able to both create and apply these disk images.

The idea was that you could make an image of a disk, make a hundred copies, sell them all, then come back with another box of disks and make a hundred more copies. Resumable duplication, in effect, without having to read the disk in each time. It was also possible to distribute disk images (by compressing them and/or sending them over the internet), so that the image could be applied to floppy disks by separate individuals.

Disk Copy also had the ability to ‘mount’ these images. This was the equivalent of putting the disk in, as it would show up on the desktop, but you could have incinerated the original real disk and it would still work — everything was read from the disk image. Once the internet got popular, it was possible to send a disk image to somebody else, and they could make their own floppy disks from the disk image.

Around the time of System 7.5, Apple began distributing SMIs: Self-Mounting Images. Regular disk images were Disk Copy documents, so when you opened one, Disk Copy would launch to take care of it. An SMI had the disk-mounting code in it, so you didn’t need Disk Copy anymore. I think Disk Copy was made a custom-install option at this point.

Disk Copy 6 introduced a new format called NDIF (New Disk Image Format), which came in several variants: a read/write format, a read-only uncompressed format, and two read-only compressed formats (one using an unknown codec, and the other using a proprietary Apple codec named KenCode).

Independently of all of this, UNIX in all its flavours (and later GNU operating systems as well) had a program called tar. tar is a tape archive program, designed for recording and playing back backup tapes.

At some point, tar gained the ability to make tape-archive files (now colloquially known as tarballs). This is the -f part of tar -cf. Tarballs, in this way, are analogous to disk images. The tar format was eventually standardised as USTAR.

The UNIX philosophy of software design is ‘do one thing well’. So the tar format (and its companion application, in all its various separate implementations) never gained any sort of compression. Instead, various compression programs are used in series upon the tar data. The original way to do this was tar -cf - | compress > foo.tar.Z.

Eventually, gzip replaced compress, and not long after, GNU tar introduced a -z option that made tar do the pipeline into gzip for you. Thus were compression and archiving merged in the UNIX world.

With Mac OS X, Disk Copy came back, and SMIs went away, and a new disk image format was introduced: UDIF. Like StuffIt X, UDIF was created to handle all the new and improved metadata supported by HFS+ — most conspicuously, Unicode filenames — as well as strong (AES-128) encryption. The old disk images had a filename extension of .img; this, unfortunately, was quite a common extension (used by at least one other imager and at least one picture format), so UDIF was given the filename extension of .dmg.

UDIF is a very flexible format. It comes in many variants. As of Mac OS X 10.4.3, these include:

  • Read-only, uncompressed
  • Read/write, uncompressed
  • Read-only, ADC compression (another Apple codec)
  • Read-only, zlib compression
  • Read-only, bzip2 compression
  • Entire device (?)
  • UDIF stub (?)
  • DVD/CD master
  • Sparse

(These were also taken from the manpage for hdiutil.)

Numerous other formats have existed for many years — specifically zip (on Windows) and gzip and bzip2 (on UNIX and Linux) — and StuffIt Expander has supported all of them since 3.0 (bzip2 since maybe 5).

But in OS X, StuffIt Expander (which came with the OS until Tiger) showed its age. Versions 6-8 were slow and ugly. This got better with version 9, but by that time, StuffIt Expander lost to the alternative that also came with the OS: Disk Copy (which merged with Disk First Aid and Drive Setup to form Disk Utility in Jaguar).

For several years, most software for Mac OS X was distributed in the form of a zlib-compressed dmg.

But now things have deteriorated.

Posted in Creations, Mac OS X, Software distribution | 4 Comments »