Matt Jacobson
January 2022

A few evenings ago, I decided to fire up an old build of Mac OS X in QEMU, to bask in its relative simplicity. I chose to install Puma, along with its developer tools. It was wonderful.

As I was sitting here admiring at the beautifully concise output of top, I had the idea to list all of the Mach services available on the system. On a modern macOS, you can kinda sorta do this with launchctl, although there are all sorts of boring complicating factors that make the answer more complicated than a single list. In any case, Puma predates the existence of launchd by a few releases. Back then, the Mach bootstrap server was mach_init, the one inherited from NeXTSTEP.

I was disappointed to find that there wasn't a built-in shell utility to do the equivalent operation on Puma. Luckily, though, Puma also predates Apple's more recent tendencies to obscure and complexify the inner workings of the OS, so the information I was after was (I foolishly thought) a mere API call away:

<servers/bootstrap.h>:

kern_return_t bootstrap_info
(
        mach_port_t bootstrap_port,
        name_array_t *service_names,
        mach_msg_type_number_t *service_namesCnt,
        name_array_t *server_names,
        mach_msg_type_number_t *server_namesCnt,
        bool_array_t *service_active,
        mach_msg_type_number_t *service_activeCnt
);

So I fired up Project Builder and cranked out a simple program to fetch and display the information. bootstrap_port is a Mach send right for the port owned by mach_init, which you can get via a Mach trap, and the rest of the parameters describe the returned information. Simple, simple.

Or not. Here's what I actually got when I ran my program:

% ./build/lsbootstrap
45 services

* FontObjectsServer
* 
[43 more blank lines like that]

That is, the name for every service showed as an empty string, except the first, which seemed like a plausible service name.

Now, I've been burned by Mach APIs enough times before that I had already reflexively added error checking to each one. All of them were returning success.

So I then proceeded to spend the better part of half an hour convinced that I was getting confused by pointer typedefs. name_array_t is a typedef for name_t *, where name_t is a typedef for char [128]. I eventually assured myself that I was right all along: service_names is a pointer to an array of array 128 of char.

Therefore, it was time for some poking in the debugger. I stopped the program just after checking the return value of bootstrap_info() and saw:

(gdb) p service_names
$1 = 0x46000
(gdb) p server_names
$2 = 0x47000
(gdb) p service_actives
$3 = 0x48000

These looked reasonable. Since the values are returned as out-of-line memory descriptors in the reply Mach message, it's expected that they would be page-aligned. And since the arrays are entered into our (otherwise rather sparse) address space consecutively, it's unsurprising (and somewhat reassuring) that they should have nice sequential addresses like that.

But, indeed, apart from that mysterious first entry, the name buffers were blank:

(gdb) p (char *)0x46000
$4 = 0x46000 "FontObjectsServer"
(gdb) p (char *)0x46000 + 128
$5 = 0x46080 ""
(gdb) p (char *)0x47000
$6 = 0x47000 ""

The service_actives array did contain nonzero values -- in fact, it contained the exact right number of them:

(gdb) p service_actives[0]
$7 = 1
(gdb) p service_actives[1]
$8 = 1
(gdb) p service_actives_count
$9 = 54
(gdb) x/60w service_actives
0x48000:        0x00000001      0x00000001      0x00000001      0x00000001
0x48010:        0x00000001      0x00000001      0x00000001      0x00000001
0x48020:        0x00000001      0x00000001      0x00000001      0x00000001
0x48030:        0x00000001      0x00000001      0x00000001      0x00000001
0x48040:        0x00000001      0x00000001      0x00000001      0x00000001
0x48050:        0x00000001      0x00000001      0x00000001      0x00000001
0x48060:        0x00000001      0x00000001      0x00000001      0x00000001
0x48070:        0x00000001      0x00000001      0x00000001      0x00000001
0x48080:        0x00000001      0x00000001      0x00000001      0x00000001
0x48090:        0x00000001      0x00000001      0x00000001      0x00000001
0x480a0:        0x00000001      0x00000001      0x00000001      0x00000001
0x480b0:        0x00000001      0x00000001      0x00000001      0x00000001
0x480c0:        0x00000001      0x00000001      0x00000001      0x00000001
0x480d0:        0x00000001      0x00000001      0x00000000      0x00000000
0x480e0:        0x00000000      0x00000000      0x00000000      0x00000000

That, along with the fact that server_names_count == service_names_count == service_actives_count, suggested that I wasn't just seeing false patterns in bits of garbage memory. Something more interesting was happening.

I still figured I was just screwing something up in a subtle way. Mach's APIs are not always the most intuitive, and in any case I was hungry. I took a break for some dinner.

When I returned to my code with a full stomach, I noticed something interesting. Since service_names was at 0x46000 and server_names at 0x47000, that left 4096 bytes—a single page—of room. 4096 / 128 == 32, but the responses were very clearly telling me that there were more than 32 services.

So, even putting aside the question of why the page I had wasn't full, there didn't even appear to be enough space to contain all the strings required. In fact, my code was blowing right through the end of the buffer trying to read the 50 or so service names. It was only by luck that it wasn't crashing.

By this point, I'd decided I needed to see what was going on inside mach_init, to see what it thought it was sending me. Ordinarily I'd start by attaching a debugger and poking around. Unfortunately, suspending the bootstrap server with a debugger is a pretty terrible idea. If for some reason your debugger (or terminal emulator, or window server, or something) needs to contact the bootstrap server, you're pretty much screwed.

Luckily, though, mach_init is open-source as part of the system_cmds project. So, after jumping through a few hoops (for instance, curl from 2001 is incompatible with the modern-day TLS used on opensource.apple.com), I successfully built my own mach_init, installed it, and rebooted.^[1]

Next step was to add some logging. I started by confirming the basics. Yes, the server was indeed receiving my RPC call. There were, in fact, 50-some services. The number varied a bit based on what was running at the time, but the server always reported the same number I was getting in my program. The RPC handler was, indeed, going down the codepath to fill the name buffers with values. Everything so far seemed all right.

Just before replying, the server set the counts of all three arrays to the same value:

    *service_names_cnt = *server_names_cnt =
            *service_actives_cnt = cnt;

    return BOOTSTRAP_SUCCESS;
}

which also agreed with what I was getting back on the other side.

A few logging iterations later (including a few instances of bricking the emulator by causing mach_init to crash), I had the offhand idea to log out the pointers to the buffers allocated by the servers, the ones that it would send back in its response:

service_names = 0x6a000
server_names = 0x6c000
service_actives = 0x6e000

Again, page-aligned, nearby addresses, as I'd expect. But this time, the separation between the values is 0x2000—two pages. I double-checked my program. It was still receiving single-page buffers.

MIG

Like most Mach messaging of its time, mach_init uses MIG, the Mach Interface Generator, to handle the details of receiving requests and sending back replies. MIG takes as input an interface specification file and emits very machine-generated-quality C code. The generated code takes a datagram received on a Mach port, decodes it, and passes the decoded arguments to a human-written handler function. The result of the handler function is packaged up into a new reply datagram to be sent back over the wire.

It would be a stretch to say that MIG is easy to use, but it's certainly a nicer abstraction to program to than the (rather arcane) Mach IPC kernel APIs. Unfortunately, I felt pretty confident from my printf debugging that the mach_init handler routine was behaving correctly, so the obvious next step was to debug the MIG-generated code.

Like most machine-generated source, MIG-generated code is pretty awful to read. The best strategy in my opinion is simply to look for the call out to the handler routine—whose name you know—and to work your way out from there.

Luckily, in this case, I was able to spot something interesting pretty quickly. Here's what the core of the glue code looked like:

RetCode = x_bootstrap_info(In0P->Head.msgh_request_port, (name_array_t *)&(OutP->service_names.address), &OutP->service_namesCnt, (name_array_t *)&(OutP->server_names.address), &OutP->server_namesCnt, (bool_array_t *)&(OutP->service_active.address), &OutP->service_activeCnt);
if (RetCode != KERN_SUCCESS) {
        MIG_RETURN_ERROR(OutP, RetCode);
}
OutP->service_names.size = OutP->service_namesCnt;

OutP->server_names.size = OutP->server_namesCnt;

OutP->service_active.size = OutP->service_activeCnt * 4;

x_bootstrap_info is the human-written server handler routine. (Common practice is to add a prefix, like x_, to the server routines to avoid potential symbol collisions with the unprefixed client routine, and MIG has a facility to do this automatically.)

OutP->service_namesCnt, OutP->server_namesCnt, and OutP->service_activeCnt are fields filled in by x_bootstrap_info, denoting the number of items in their respective arrays. I already confirmed above that each of these was being filled with cnt.

What stuck out to me was the * 4. Knowing that service_active is an array of boolean_t—and, knowing further that (for reasons unknown to me) Mach's boolean_t type is 4 bytes in size—I deduced that OutP->service_active.size was being filled with the size in bytes of the returned service_active array.

All good. Why, though, was the same sort of logic not applying to service_names and server_names? Just like each element of service_active is four bytes, each element of service_names is 128 bytes.

If my hunch was right, it would explain everything I'd seen. Here's how. Suppose there are 50 services on the system. x_bootstrap_info allocates an array of 50 * 128 == 6400 bytes, which gets rounded up to two pages. It fills the first name at offset zero, the second at offset 128, and so on.

When x_bootstrap_info returns to the MIG generated code, though, this bug leads MIG to think that a 50-byte array (not a 50-element) array is being returned. Since that's smaller than MSG_OOL_SIZE_SMALL (4096, or a single page), the kernel is free to copy the bytes directly instead of using a shared mapping. 50 bytes encompasses the entirety of the first string—"FontObjectsServer"—but none of the rest.

Testing a theory

Testing this theory is easy. Since mig generates C code that is then fed like any other into the compiler, it's not too hard to modify the generated code and see what happens. And indeed, after adding * 128 to the service_names and server_names lines above, rebuilding mach_init, and rebooting, my program sees a nice full list of services—which, for reference, are:

43 services

* FontObjectsServer
* DiskArbitration
* System Configuration Server
* IPConfig Server
* lookup daemon
* com.apple.coreservicesd
* Apple CFNotificationCenter Server
* SecurityServer
* Processes-0.131073
* AppleEvents-253
* WindowServer
* Apple Pasteboard Server
* CFPBS
* AppleEvents-System
* RecentItemServer
* Processes-0.262145
* AppleEvents-260
* CFPasteboardClient-4978bc0f2
* CoreDrag-14851
* DockClient-40001-0
* O3Master
* CoreDrag-7723
* UNCUserNotification
* ScreenSaverDaemon
* Processes-0.393217
* AppleEvents-261
* SchedulerUpdateNotificationPort
* Processes-0.524289
* AppleEvents-262
* CoreDrag-18435
* DocklingServer
* NSApplication-MainThread-52854729318#
* DockClient-80001-0
* CoreDrag-19203
* CFPasteboardClient-5bc4b8e9e
* DockServer
* CoreDrag-18947
* Processes-0.917505
* AppleEvents-265
* CFPasteboardClient-b279d2c2
* NSApplication-MainThread-1155201142#
* DockClient-E0001-0
* CoreDrag-25607

While it was satisfying to see my program work, I was still confused as to why the original MIG-generated code wasn't working. By this time, developing inside the emulator was getting a little limiting, so I extracted the pertinent .defs source to my Catalina machine and ran mig on it there, intending to do some deeper analysis of what it was doing.

Lo and behold, with no intervention:

OutP->service_names.size = OutP->service_namesCnt * 128;

OutP->server_names.size = OutP->server_namesCnt * 128;

Catalina's mig was generating source with the change I made!

mig itself is also open-source, and while I might argue that its human-authored source is even less legible than the code it generates, I was eventually able, with the help of lldb, to track down the fix to this change:

diff --git a/migcom.tproj/server.c b/migcom.tproj/server.c
index 2017352..c6fd36d 100644
--- a/migcom.tproj/server.c
+++ b/migcom.tproj/server.c
@@ -1727,7 +1727,7 @@ WriteKPD_ool(file, arg)
        else
            fprintf(file, "OutP->%s%s", count->argMsgField, subindex);

-        if (howbig > 8)
+        if (count->argMultiplier > 1 || howbig > 8)
             fprintf(file, " * %d;\n", 
                count->argMultiplier * howbig / 8);
         else

That change landed in bootstrap_cmds-35, which shipped with Jaguar. Oops! (If I were still at Apple, I'd do some archaeology in Radar and in the internal source control to see what prompted someone to find this bug. Maybe someone I know will do that for me. 🙂)