Benchmark Ruby-macho vs otool?

[xu-cheng]

First of all, good work here. 🙇

Since now this is included in Homebrew, I'm curious to know the performance between Ruby implementation vs old otool. So how about adding benchmark along side test suite?

[woodruffw]

Thank you!

Benchmarking is a great idea. Do you have any instrumentation in mind?

[xu-cheng]

You may find https://github.com/evanphx/benchmark-ips useful.

[woodruffw]

This looks great. I'm expanding the test suite on rarer/unusual load commands today, but I'll see about adding this tomorrow.

[UniqMartin]

Certainly not a full-featured benchmark, but here are some numbers that give an indication of the status quo on crunching through a large number of files and identifying/dumping Mach-O files among them. (Spoiler: There's still room for improvement). Numbers obviously depend on a lot of factors, including what formulae are installed and disk speed. I'm showing numbers for three consecutive runs, so caching effects were taken into account to some degree.

Identifying Mach-O files:

$ unset HOMEBREW_RUBY_MACHO
$ brew time macho-find "$(brew --cellar)" > /dev/null
brew macho-find:   2.150s user,   1.670s system,   3.841s total ( 99.5% cpu)
brew macho-find:   2.240s user,   1.720s system,   3.974s total ( 99.7% cpu)
brew macho-find:   2.260s user,   1.770s system,   4.054s total ( 99.4% cpu)

$ export HOMEBREW_RUBY_MACHO=1
$ brew time macho-find "$(brew --cellar)" > /dev/null
brew macho-find:   6.790s user,   7.360s system,  30.213s total ( 46.8% cpu)
brew macho-find:   6.400s user,   7.000s system,  28.919s total ( 46.3% cpu)
brew macho-find:   6.560s user,   7.160s system,  28.965s total ( 47.4% cpu)

Dumping Mach-O files:

$ unset HOMEBREW_RUBY_MACHO
$ brew time macho-dump "$(brew --cellar)" > /dev/null
brew macho-dump:   9.470s user,  15.820s system,  40.027s total ( 63.2% cpu)
brew macho-dump:   8.120s user,  12.300s system,  21.513s total ( 94.9% cpu)
brew macho-dump:   8.390s user,  12.910s system,  22.518s total ( 94.6% cpu)

$ export HOMEBREW_RUBY_MACHO=1
$ brew time macho-dump "$(brew --cellar)" > /dev/null
brew macho-dump:   7.690s user,  13.940s system,  47.522s total ( 45.5% cpu)
brew macho-dump:   7.110s user,   7.980s system,  36.315s total ( 41.6% cpu)
brew macho-dump:   7.070s user,   7.870s system,  35.938s total ( 41.6% cpu)

You'll need brew tap UniqMartin/tools and Homebrew/legacy-homebrew#48817 for a roughly similar setup.

[woodruffw]

If I had to take an educated guess, I'd bet that a significant part of the overhead is in the exception handling used to choose which class to load a Mach-O with.

The simple solution is to change MachO.open to open the file, do a magic test, and then pass the read file to FatFile/MachOFile via new_from_bin (and set the filename correctly). I'll add this tonight or tomorrow.

[UniqMartin]

If I had to take an educated guess, I'd bet that a significant part of the overhead is in the exception handling used to choose which class to load a Mach-O with.

Good guess, but probably an insignificant overhead in the grand scheme of things. I haven't rigorously tested it, but my guess would be the IO overhead from reading every file completely into memory, while it would be sufficient to read only a few bytes to decide whether it's a Mach-O file (or not) and at most a few KB to read and parse all load commands for all architectures in case it actually is a (fat) Mach-O file. This would also explain why the CPU utilization is relatively low.

The simple solution is to change MachO.open to open the file, do a magic test, and then pass the read file to FatFile/MachOFile via new_from_bin (and set the filename correctly). I'll add this tonight or tomorrow.

This is a good and worthwhile optimization, but I think addressing the above will have a higher impact.

[woodruffw]

I haven't rigorously tested it, but my guess would be the IO overhead from reading every file completely into memory

This makes a lot of sense.

Since Homebrew interacts with ruby-macho solely through MachO.open, I can optimize this as well within the same PR.

[UniqMartin]

Since Homebrew interacts with ruby-macho solely through MachO.open, I can optimize this as well within the same PR.

You could focus on optimizing this part while I try to clean up my endianess-WIP branch and start moving chunks of it here (via PRs). How does that sound?

[woodruffw]

You could focus on optimizing this part while I try to clean up my endianess-WIP branch and start moving chunks of it here (via PRs). How does that sound?

That sounds fantastic. I've opened a preliminary PR as #22 (not much in it yet)