Debugging the Mac OS X kernel with VMware and GDB
Edit 13 July 2013: I’ve made a couple of updates to this post to clarify a couple of things and resolve issues people have had.
fG! did a great write up here on how to set up two-machine debugging with VMware on Leopard a couple of years ago, but as a few things have changed since then and I will probably refer to this topic in future posts I thought it was worth revisiting.
Debugging kernel extensions can be a bit of a pain. printf()-debugging is the worst, and being in kernel-land, it might not be immediately obvious how to go about debugging your (or other people’s) code. Apple has long provided methods for kernel debugging via the Kernel Debugger Protocol (KDP), along with ddb, the in-kernel serial debugger. KDP is implemented in-kernel by an instance of IOKernelDebugger, and allows you to connect to the debug stub from an instance of gdb (Apple’s outdated fork only AFAIK) running on another machine connected via FireWire or Ethernet. ddb can be used to debug the running kernel from the target machine itself, but is pretty low-level and arcane. Apple suggests in the Kernel Programming Guide that you are better off using gdb for most tasks, so that’s what we’ll do.
Enter VMware
We don’t really want to use two physical machines for debugging, because who the hell uses physical boxes these days when VMs will do the job? With the release of Mac OS X 10.7 (Lion), Apple changed the EULA to allow running virtualised instances of Lion on top of an instance running on bare metal. Prior to this, only the “server” version of Mac OS X was allowed to be virtualised, and VMware ‘prevented’ the client version from being installed through some hardcoded logic in vmware-vmx (which some sneaky hackers patched). VMware Fusion 4 introduced the ability to install Mac OS X 10.7 into a VM without any dodgy hacks, just by choosing the Install Mac OS X Lion.app bundle as the installation disc.
So, the first step of the process is: install yourself a Mac OS X VM as per the VMware documentation.
Edit 13 July 2013: Once you’re done it’s probably a good idea to take a snapshot of your VM in case there are problems installing the debug kernel. Generally it’s not a problem, but it’s annoying to roll back and much easier to use a VMware snapshot.
Install the debug kernel
Once we’ve got our VM installed, we need to install the Kernel Debug Kit. This contains a version of the XNU kernel built with the DEBUG flag set, which includes the debug stubs for KDP and ddb, and a second DEBUG version with a full symbol table to load in GDB so we can use breakpoints on symbol names and not go insane. The debug kits used to live here, but it seems Apple decided they only want ADC members to be able to access them, so now they’re here (requires ADC login). Download the appropriate version for the target kernel you’re debugging in the VM (not necessarily the same as the kernel version on your host debugger machine). In this case I’m using Kernel Debug Kit 10.7.3 build 11D50. Copy this image up to the target VM, and install the debug kernel as per the instructions in the readme file:
macvm$ sudo -s
macvm# cd /
macvm# ditto /Volumes/KernelDebugKit/DEBUG_Kernel/System.kext /System/Library/Extensions/System.kext
macvm# cp -r /Volumes/KernelDebugKit/DEBUG_Kernel/mach_kernel* /
macvm# chown -R root:wheel /System/Library/Extensions/System.kext /mach_kernel*
macvm# chmod -R g-w /System/Library/Extensions/System.kext /mach_kernel*
macvm# touch /System/Library/Extensions
macvm# shutdown -r now
Hopefully your VM has successfully booted with the debug kernel and no magic blue smoke has been let out.
Edit 13 July 2013: If your VM has panicked at boot time make sure you’ve allocated at least 4GB of RAM to the VM or it will not boot on newer OS X versions.
Next we need to set the kernel boot arguments to tell it to wait for a debugger connection at boot time. There are other options but, as fG! said previously, there isn’t an obvious way to generate an NMI within VMware (I haven’t really looked further into this - if there is I’d like to hear about it). In VMware Fusion 4, the proper NVRAM support means we can specify normal boot-args in NVRAM rather than the old com.apple.Boot.plist, by using the nvram utility on the target VM like this:
macvm# nvram boot-args="-v debug=0x1"
Now we’ll do a bit of config on the debug host, then reboot the VM.
Debug host config
Traditionally, two-machine debugging would either use FireWire or Ethernet. We can simulate Ethernet with the VMware network bridging.
Edit 13 July 2013: With newer versions of OS X (I’m not sure exactly when they introduced this but it definitely works on 10.8.4) you don’t actually need to do this static ARP trick any more. When the VM boots it will stop at “Waiting for remote debugger connection” after telling you its MAC and IP address. You should be able to skip the static ARP and just kdp-reattach (as below) directly to the IP address displayed here.
Grab the MAC address and IP address of your VM:
macvm$ ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=2b<RXCSUM,TXCSUM,VLAN_HWTAGGING,TSO4>
ether 00:0c:29:d6:df:02
inet6 fe80::20c:29ff:fed6:df02%en0 prefixlen 64 scopeid 0x4
inet 10.0.0.15 netmask 0xffffff00 broadcast 10.0.0.255
media: autoselect (1000baseT <full-duplex>)
status: active
And back on your debug host, add a static ARP entry for the VM:
debughost# arp -s 10.0.0.15 0:c:29:d6:df:2
debughost# arp 10.0.0.15
macvm (10.0.0.15) at 0:c:29:d6:df:2 on en0 permanent [ethernet]
I also have an /etc/hosts entry for the VM, hence the hostname macvm.
Now we should be able to reboot the VM and it will pause waiting for the debugger connection at the start of the boot process. It used to actually say Waiting for debugger connection… or something similar in previous kernel versions, but it seems to pause after [PCI configuration begin] on 10.7.
Fire up GDB
Now it’s time to actually start GDB and connect to the KDP debug stub. Assuming you’ve just mounted the Kernel Debug Kit dmg file, the following paths should be correct. On the debug host machine:
$ gdb /Volumes/KernelDebugKit/DEBUG_Kernel/mach_kernel
GNU gdb 6.3.50-20050815 (Apple version gdb-1708) (Thu Nov 3 21:59:02 UTC 2011)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...This is contrary to the instructions in the readme file for the Kernel Debug Kit, which tells you to target /Volumes/KernelDebugKit/mach_kernel with gdb. I haven’t been able to get this kernel to work correctly - symbols are not looked up properly and lots of addresses seem to be wrong, resulting in the kgmacros stuff not working, and breakpoints being set at the wrong addresses. If you load the kernel in the DEBUG_Kernel directory it works OK.
Next, source the kgmacros file - this contains a bunch of GDB macros that make dealing with kernel introspection and debugging much easier (particularly when you want to start looking at stuff like the virtual memory subsystem, and other fun stuff):
gdb$ source /Volumes/KernelDebugKit/kgmacros
Loading Kernel GDB Macros package. Type "help kgm" for more info.Note: if you’re attaching to a kernel running on a different arch (ie. you created a 32-bit VM on a 64-bit machine), you’ll need to use the --arch flag:
The –arch=i386 option allows you to use a system running the 64-bit kernel to connect to a target running the 32-bit kernel. The –arch=x86_64 option allows you to go the other direction.
Now we attach to the debug target machine:
gdb$ kdp-reattach 10.0.0.15
Connected.Edit 13 July 2013: If you’re using a recent OS X you can kdp-reattach to the IP address that was printed when the debug kernel paused waiting for the debugger.
You can also attach using target remote-kdp and attach 10.0.0.15. Allow the kernel to continue execution:
gdb$ cAt this point the disk icon in VMware should be going blue with activity, and the VM should continue booting as normal.
Breaking into the debugger
Unfortunately, we can’t use the normal method of hitting ^C in the debugger to pause execution, so we have to rely on software breakpoints. The method fG! initially suggested was to break on tcp_connect() or something similar, so you can drop into the debugger by attempting to telnet somewhere. This proves to be a bit cumbersome in Lion with all the fancy (scary) network autodetect stuff - connections going out from agents all over the place means constantly dropping into the debugger.
The method that I have primarily used is to set a breakpoint on the kext_alloc() function. This is called once during the initialisation of a kernel extension, so it can be a reasonably useful point at which to break if you want to debug the initialisation of the kext, and a good on-demand breakpoint for general kernel memory inspection.
Edit 13 July 2013: @chicagoben pointed me at a simple method of replicating the behaviour of an NMI and dropping into the debugger using the technique in this handy kernel module.
Breaking on kext_alloc():
Breakpoint 1, kext_alloc (_addr=0xffffff804650b5f0, size=0x3000, fixed=0x0) at kext_alloc.c:107
107 in kext_alloc.cAnd getting a stack trace:
gdb$ bt
#0 kext_alloc (_addr=0xffffff804650b5f0, size=0x3000, fixed=0x0) at kext_alloc.c:107
#1 0xffffff80008f4166 in kern_allocate (size=0x3000, flags=0xffffff804650b664, user_data=0xffffff80096f9880) at OSKext.cpp:408
#2 0xffffff8000922874 in allocate_kext (context=0xffffff800af06420, callback_data=0xffffff80096f9880, vmaddr_out=0xffffff804650b710, vmsize_out=0xffffff804650b708, linked_object_alloc_out=0xffffff804650b6f8) at kxld.c:468
#3 0xffffff8000921e69 in kxld_link_file (context=0xffffff800af06420, file=0xffffff8036641000 "????\a", size=0x2600, name=0xffffff8007e14a90 "ax.ho.kext.DebugTest", callback_data=0xffffff80096f9880, dependencies=0xffffff80091e4a60, ndependencies=0x6, linked_object_out=0xffffff804650b8f8, kmod_info_kern=0xffffff80096f98c8) at kxld.c:273
#4 0xffffff80008f0b55 in OSKext::loadExecutable (this=0xffffff80096f9880) at OSKext.cpp:4751
#5 0xffffff80008f3cc4 in OSKext::load (this=0xffffff80096f9880, startOpt=0x0, startMatchingOpt=0x0, personalityNames=0x0) at OSKext.cpp:4420
#6 0xffffff80008f741b in OSKext::loadKextWithIdentifier (kextIdentifier=0xffffff8007e1adf0, allowDeferFlag=0x0, delayAutounloadFlag=0x0, startOpt=0x0, startMatchingOpt=0x0, personalityNames=0x0) at OSKext.cpp:4184
#7 0xffffff80008f8c91 in OSKext::loadFromMkext (clientLogFilter=0x0, mkextBuffer=0xffffff8046362400 "MKXTMOSX", mkextBufferLength=0x2da8, logInfoOut=0xffffff804650bc30, logInfoLengthOut=0xffffff804650bc2c) at OSKext.cpp:3271
#8 0xffffff8000909f32 in kext_request (hostPriv=0xffffff8000c8bec0, clientLogSpec=0x0, requestIn=0xffffff80075c9d30, requestLengthIn=0x2da8, responseOut=0xffffff800a976918, responseLengthOut=0xffffff800a976940, logDataOut=0xffffff800a976928, logDataLengthOut=0xffffff800a976944, op_result=0xffffff800a976948) at OSKextLib.cpp:281
#9 0xffffff800028d9ab in _Xkext_request (InHeadP=0xffffff800abbec38, OutHeadP=0xffffff800a9768f4) at host_priv_server.c:5961
#10 0xffffff80002443d2 in ipc_kobject_server (request=0xffffff800abbebc0) at ipc_kobject.c:339
#11 0xffffff8000221570 in ipc_kmsg_send (kmsg=0xffffff800abbebc0, option=0x0, send_timeout=0x0) at ipc_kmsg.c:1376
#12 0xffffff8000237393 in mach_msg_overwrite_trap (args=0xffffff80067c65a4) at mach_msg.c:487
#13 0xffffff80002375b4 in mach_msg_trap (args=0xffffff80067c65a4) at mach_msg.c:554
#14 0xffffff8000354a01 in mach_call_munger64 (state=0xffffff80067c65a0) at bsd_i386.c:534If you’re debugging a kernel extension that you are writing yourself (or have the code for) a better method of dropping into the debugger is to put an int 3 (software breakpoint) in your code at the point you want to break, like this:
kern_return_t DebugTest_start(kmod_info_t * ki, void *d)
{
printf("hurr\n");
asm("int $3");
derp();
return KERN_SUCCESS;
}Now when we load this kext we get dropped into the debugger:
Program received signal SIGTRAP, Trace/breakpoint trap.
0xffffff7f80b2af12 in ?? ()The call stack at this point looks somewhat similar to before, passing through the OSKext class:
gdb$ bt
#0 0xffffff7f80b27f12 in ?? ()
#1 0xffffff80008eebb4 in OSKext::start (this=0xffffff8007d37400, startDependenciesFlag=0x1) at OSKext.cpp:5456
#2 0xffffff80008f3e97 in OSKext::load (this=0xffffff8007d37400, startOpt=0x0, startMatchingOpt=0x0, personalityNames=0x0) at OSKext.cpp:4475
#3 0xffffff80008f741b in OSKext::loadKextWithIdentifier (kextIdentifier=0xffffff80068955b0, allowDeferFlag=0x0, delayAutounloadFlag=0x0, startOpt=0x0, startMatchingOpt=0x0, personalityNames=0x0) at OSKext.cpp:4184
#4 0xffffff80008f8c91 in OSKext::loadFromMkext (clientLogFilter=0x0, mkextBuffer=0xffffff804623e400 "MKXTMOSX", mkextBufferLength=0x2da8, logInfoOut=0xffffff8045c23c30, logInfoLengthOut=0xffffff8045c23c2c) at OSKext.cpp:3271
<snip>And we can disassemble the code at and after the breakpoint:
gdb$ x/11i 0xffffff7f80b2df12 - 1
0xffffff7f80b2df11: int3
0xffffff7f80b2df12: xor cl,cl
0xffffff7f80b2df14: mov al,cl
0xffffff7f80b2df16: call 0xffffff7f80b2df70
0xffffff7f80b2df1b: mov DWORD PTR [rbp-0x18],0x0
0xffffff7f80b2df22: mov eax,DWORD PTR [rbp-0x18]
0xffffff7f80b2df25: mov DWORD PTR [rbp-0x14],eax
0xffffff7f80b2df28: mov eax,DWORD PTR [rbp-0x14]
0xffffff7f80b2df2b: add rsp,0x20
0xffffff7f80b2df2f: pop rbp
0xffffff7f80b2df30: retThis corresponds to the following code from the binary (extracted using otool -tv):
0000000000000f11 int $0x3
0000000000000f12 xorb %cl,%cl
0000000000000f14 movb %cl,%al
0000000000000f16 callq 0x00000f70
0000000000000f1b movl $0x00000000,0xe8(%rbp)
0000000000000f22 movl 0xe8(%rbp),%eax
0000000000000f25 movl %eax,0xec(%rbp)
0000000000000f28 movl 0xec(%rbp),%eax
0000000000000f2b addq $0x20,%rsp
0000000000000f2f popq %rbp
0000000000000f30 retPoking around in kernel memory
Let’s check out a few neat things in memory. The start of the Mach-O header for the kernel image in memory:
gdb$ x/x 0xffffff8000200000
0xffffff8000200000: 0xfeedfacfThis is the “magic number” indicating a 64-bit Mach-O executable. The 32-bit version is 0xfeedface.
The “system verification code”:
gdb$ x/s 0xffffff8000002000
0xffffff8000002000: "Catfish "On previous PowerPC versions of the OS this was located at 0x5000 and said "Hagfish ". Here is the corresponding assembly source from osfmk/x86_64/lowmem_vectors.s in the kernel source tree:
/*
* on x86_64 the low mem vectors live here and get mapped to 0xffffff8000200000 at
* system startup time
*/
.text
.align 12
.globl EXT(lowGlo)
EXT(lowGlo):
.ascii "Catfish " /* +0x000 System verification code */Interestingly, that comment appears to be incorrect - 0xffffff8000200000 is where the kernel image itself starts and the stuff in lowmem_vectors.s starts at 0xffffff8000002000 as we’ve seen.
If you’re interested in kernel internals (which you probably are if you’re reading this) then you might want to have a look at the kgmacros help at this point:
gdb$ help kgm
| These are the kernel gdb macros. These gdb macros are intended to be
| used when debugging a remote kernel via the kdp protocol. Typically, you
| would connect to your remote target like so:
| (gdb) target remote-kdp
| (gdb) attach <name-of-remote-host>
<snip>There’s heaps of cool and useful stuff there to look at.
Listing the process tree by walking the list from allproc down:
gdb$ showproctree
PID PROCESS POINTER]
=== ======= =======
0 kernel_task [ 0xffffff80073e8820 ]
|--1 launchd [ 0xffffff80073e8820 ]
| |--163 xpchelper [ 0xffffff800912a9f0 ]
| |--158 launchd [ 0xffffff8007c65e40 ]
| | |--162 distnoted [ 0xffffff80081f8010 ]
| | |--161 mdworker [ 0xffffff80073e83d0 ]
| |--157 mdworker [ 0xffffff80082c6010 ]
| |--139 com.apple.dock.e [ 0xffffff800912ae40 ]
| |--138 filecoordination [ 0xffffff800912b290 ]
| |--111 xpchelper [ 0xffffff8007c66f80 ]
| |--106 launchdadd [ 0xffffff80081fa290 ]
| |--104 launchd [ 0xffffff80082c86e0 ]
<snip>Print the struct proc (kernel version, not the userland one) for the kernel task:
gdb$ print *(struct proc *)0xffffff80073e8820
$4 = {
p_list = {
le_next = 0xffffff8000cb4c20,
le_prev = 0xffffff80073e76e0
},
p_pid = 0x1,
task = 0xffffff80067c25a0,
p_pptr = 0xffffff8000cb4c20,
p_ppid = 0x0,
p_pgrpid = 0x1,
p_uid = 0x0,
p_gid = 0x0,
<snip>Have a poke around and see what you can find.
Source-level debugging
Now that we’ve explored kernel memory a bit, it’s probably worth noting that you can use the kernel source for source-level debugging within GDB, or possibly even in Xcode (anybody done this?). Some of the documentation seems to be a bit out of date on this - e.g. the Kernel Programming Guide references a .gdbinit file defined in the osfmk directory (the Mach part of the kernel) which no longer exists, and previous documentation mentions creation of a /SourceCache/xnu/... directory for source-level debugging, but this trick doesn’t seem to work any more. It seems that these days the kernel debug symbol information relates only to filename and line number, not full file path, like this:
Breakpoint 1, kext_alloc (_addr=0xffffff80463735f0, size=0x3000, fixed=0x0) at kext_alloc.c:107
107 kext_alloc.c: No such file or directory.We can still load source code on a per-directory basis if we know where the file in question is located. In this instance it’s osfmk/kern/kext_alloc.c within the kernel source tree, we’ll do this:
gdb$ dir /path/to/xnu-1699.24.23/osfmk/kern/And magic:
gdb$ l
102 }
103
104 kern_return_t
105 kext_alloc(vm_offset_t *_addr, vm_size_t size, boolean_t fixed)
106 {
107 kern_return_t rval = 0;
108 mach_vm_offset_t addr = (fixed) ? *_addr : kext_alloc_base;
109 int flags = (fixed) ? VM_FLAGS_FIXED : VM_FLAGS_ANYWHERE;
110
111 /* Allocate the kext virtual memory */Go grab yourself a copy of the source for your kernel version at opensource.apple.com and give it a try.
So, yeah…
Have fun.
