Apologia

This isn’t topical. This is a story from the past that I thought about this week and want to write down before I forget more details.

What

Compiling and linking objects are things that you do a lot. Compiler flags are all very useful (particularly when trying to make your builds reproducible or cacheable). Linker flags can be too! What’s fun is that ld has an entire scripting language you can use to change its behavior.

I once wrote some bash to generate a linkerscript in order to find memory leaks to fix. I think that sentence is fun and want to write down details about all of it.

Why

I was porting a large and old C++ codebase from Windows to Linux. We repeatedly found places where the Microsoft C++ compiler (MSVC) and its implementation of the C++ standard library were dramatically more forgiving in how they would interpret various C++ footguns. We spent a lot of time finding and fixing various types of memory bugs.

The tool that was most useful for squashing these was the address sanitizer. ASAN is wonderful! You pass some flags around in your build and you end up with a program that will yell at you when it detects all kinds of memory problems. There are limitations to what sorts of builds and platforms it supports, but that’s the general idea. This ended up being most useful to build against our tests because for various reasons it wasn’t practical to build or use the actual final executables with it.

This was a large codebase which did not have the best hygiene. We compiled the same header files in multiple shared objects. Each of these declared their symbols as things that are going to be imported or exported using a macro. With MSVC, this would be __declspec(export/import) to export or import respectively (see their docs for more). There is no “import” concept for symbols on Linux, you just say “this symbol is public” or “this symbol is hidden.” See GNU’s docs for some examples (and a macro that looks a lot like what was used).

ASAN would let us know when we were running our executables that we had multiple definitions of symbols. If you have multiple libraries using the same symbol for different things, this can be really bad! In our case it was less bad since it was the same symbol and the same definition that we were exporting. Even if the dynamic linker didn’t cache its choice of which definition it had used, we would only ever have the same definition again. It still was really polluting our logs, and it was a sign that we were doing something kinda goofy.

How

There’s a world in which the solution is a massive rearchitecting and rearranging of things so that this could never happen organizationally. Go find all of the things that we’re redefining, put them in their own appropriate separate units, and then link against those. That would probably take a really long time! I did not have a really long time to spend on this problem.

What I did was use an export map as described in this amazing paper by Ulrich Drepper. His blog is a source of other wonderful technical documents.

I wrote bash which looked up what symbols were being exported, then wrote the export map. I hooked this into the build system so that it would run before trying to link the test executables.

I forget what I did to look up which symbols “should” be exported. I think what I did was make a combination of a regex and objdump in some order to get the symbols that were supposed to be exported from a given library. I’d save those to disk, then write an export map to only export those symbols, which I’d then give to the linker.

It was fun! and I wish I remembered more details.