New tool: The Symbolicator

2009-04-19 15:45:32 UTC

The most significant behind-the-scenes change in Growl 1.1.5 is that we now build using the DWARF-with-dSYM debug symbol format.

Now, we distribute a fully-stripped executable for every release—even betas—and we keep the dSYM bundles on hand separately. The idea here is that we can use the dSYM bundles to obtain symbolic information (function name, filename, and line number) for the bare addresses in users’ crash logs.

Unfortunately, there are no good tools for symbolicating Mac OS X crash logs. If this were an iPhone app, we could just drag the crash logs into the Xcode Organizer window, but Xcode doesn’t do this for Mac crash logs.

I tried all the other tools as well, and every one of them had one of two problems:

  1. Didn’t work at all
  2. Needed the dSYM bundle and main bundle to be next to each other on disk

#2 is not a deal-breaker, but it is a hassle.

So I wrote my own symbolication tool.

The Symbolicator is a Python script that:

  1. Reads in a crash log.
  2. Finds the necessary dSYM bundles using Spotlight. (This means that you can put the dSYM bundles anywhere you want, and as long as Spotlight can find them, the Symbolicator will be able to use them.)
  3. Uses dwarfdump to extract the relevant symbol information.
  4. Replaces the address offsets with the symbol information (just like CrashReporter does when it has debug symbols to work with).
  5. Writes the symbolicated log to stdout.

Use it like this:

% symbolicator < unsymbolicated.crash > symbolicated.crash

Or use ThisService to make a service out of it.

If you want to see it in action right now, download Growl 1.1.5b1 and the corresponding dSYM bundles from the Growl beta page. Make Growl crash (killall -TRAP works well), then unpack the bundles and use the Symbolicator on the crash log. (If you unpack the bundles first, CrashReporter will symbolicate the log before you even get to it. Handy, unless you’re trying to test the Symbolicator. ☺)

To make this work on your own app, follow the instructions in the aforementioned ADC article, then make sure you archive the dSYM bundles for every release, including betas. On Growl, I added code to our Release Makefile for this.

Note: If your app is closed-source, you should not put your dSYM bundles on your website, since the debug symbols are arguably trade secrets (information about your source code). Keep them locally, perhaps on a flash-memory drive. Disclaimer: IANAL.

8 Responses to “New tool: The Symbolicator”

  1. Mattias Arrelid Says:

    This is wonderful news. I’ve been thinking of writing something similar myself, but that seems to be unnecessary now. Will give it a try and report any strange behaviour (if any).

    Again – thanks!

  2. Sean Says:

    “… there are no good tools for symbolicating Mac OS X crash logs …” what about dSymbolizer?
    http://www.softedge.se/Softedge/dSymbolizer.html

  3. Peter Hosey Says:

    Sean: Didn’t work when I tried it. It uses gdb, so I believe it needs both the main and dSYM bundles, and in the same folder. The Symbolicator can work with the dSYM bundle alone.

  4. voipuser Says:

    I mentioned this in IRC but am not sure who replied to me there (you or someone else), so let me post it here. What I would most like to see as a future Growl enhancement is, if any program has gone into full screen mode (e.g. you maximize a YouTube video or some other type of video) then the Growl notifications would be held in a buffer and not displayed until that program goes out of full screen mode – UNLESS you have set certain notifications to be of Emergency or High priority – they would get displayed anyway. Someone said it could not be done, so my alternate suggestion was to have an icon in the top menu bar that you could click to put Growl notifications “on hold”, and it would change color (turn bright red or something) to remind you, and buffer any incoming notifications until you turn it back on (then display the buffered notifications, except perhaps those you’ve indicated have “Very Low” priority – those would just be thrown away).

    So basically what I’m asking for is an easy (one click) way to enable/disable this mode, without having to go into System Preferences, but still save any “normal” notifications so you can read them when your leave that mode. If you’re trying to watch a video, most Growl notifications are just annoying but there are a few I’d still like to see, and those could be specified using priorities.

  5. Peter Hosey Says:

    voipuser: I don’t implement stuff based on off-topic comments on my personal blog. Try the Growl contact page—then I’ll consider it.

  6. Tony Arnold Says:

    I’ve just tried to use this under Snow Leopard, but it throws the following error:

    Traceback (most recent call last):
    File “/usr/local/bin/symbolicator”, line 8, in
    load_entry_point(‘Symbolicator==1.0.1’, ‘console_scripts’, ‘symbolicator’)()
    File “build/bdist.macosx-10.6-universal/egg/symbolicator.py”, line 203, in main
    File “build/bdist.macosx-10.6-universal/egg/symbolicator.py”, line 15, in architecture_for_code_type
    KeyError: ‘X86-64’

    Any ideas?

  7. David Reitter Says:

    Seems like this hasn’t been updated to work with Lion or 64 bit builds. It’s a useful tool, though… Here’s the simple change that is needed to make it work. Haven’t thoroughly tested the results…

    *** symbolicator.py 2009-04-25 02:02:44.000000000 -0400
    — symbolicator-lion.py 2011-10-04 18:04:02.000000000 -0400
    ***************
    *** 9,21 ****
    def architecture_for_code_type(code_type):
    arch_code_type_name = code_type.split()[0]
    code_types_to_architectures = {
    ‘X86’: ‘i386’,
    ‘PPC’: ‘ppc’,
    }
    return code_types_to_architectures[arch_code_type_name]

    recognized_versions = [
    ! 6,
    ]

    def reformat_UUID(UUID):
    — 9,22 —-
    def architecture_for_code_type(code_type):
    arch_code_type_name = code_type.split()[0]
    code_types_to_architectures = {
    + ‘X86-64’: ‘i386’,
    ‘X86’: ‘i386’,
    ‘PPC’: ‘ppc’,
    }
    return code_types_to_architectures[arch_code_type_name]

    recognized_versions = [
    ! 6,9
    ]

    def reformat_UUID(UUID):

  8. Peter Hosey Says:

    David Reitter: The development source in the Mercurial repository is compatible with 64-bit crashes. I have a patch waiting for me that adds support for crashes from Lion, though I haven’t tested it yet. (If the contents of the patch I’m referring to are any indication, your change does not add Lion support.)

Leave a Reply

Do not delete the second sentence.