I got launchd and recoveryd to start on an emulated iPhone running iOS 12 beta 4’s kernel using a modified QEMU. Here’s what I learned, and how you can try this yourself.

Introduction

This is Part 2 of a series on the iOS boot process. Part 1 is here. Sign up with your email to be the first to read new posts.

skip to: tutorial, writeup

First, let me repeat: this is completely useless unless you’re really interested in iOS internals. If you want to run iOS, you should ask @CorelliumHQ instead, or just buy an iPhone.

I’ve been interested in how iOS starts, so I’ve been trying to boot the iOS kernel in QEMU.

I was inspired by @cmwdotme’s Corellium, a service which can boot any iOS in a virtual machine. Since I don’t have 9 years to build a perfect simulation of an iPhone, I decided to go for a less lofty goal: getting enough of iOS emulated until launchd, the first program to run when iOS boots, is able to start.

Since last week’s post, I got the iOS 12 beta 4 kernel to fully boot in QEMU, and even got it to run launchd and start recoveryd from the restore ramdisk. Here’s the output from the virtual serial port:

iBoot version: 

corecrypto_kext_start called
FIPSPOST_KEXT [64144875] fipspost_post:156: PASSED: (4 ms) - fipspost_post_integrity
FIPSPOST_KEXT [64366750] fipspost_post:162: PASSED: (1 ms) - fipspost_post_hmac
FIPSPOST_KEXT [64504187] fipspost_post:163: PASSED: (0 ms) - fipspost_post_aes_ecb
FIPSPOST_KEXT [64659750] fipspost_post:164: PASSED: (0 ms) - fipspost_post_aes_cbc
FIPSPOST_KEXT [72129500] fipspost_post:165: PASSED: (117 ms) - fipspost_post_rsa_sig
FIPSPOST_KEXT [76481625] fipspost_post:166: PASSED: (67 ms) - fipspost_post_ecdsa
FIPSPOST_KEXT [77264187] fipspost_post:167: PASSED: (11 ms) - fipspost_post_ecdh
FIPSPOST_KEXT [77397875] fipspost_post:168: PASSED: (0 ms) - fipspost_post_drbg_ctr
FIPSPOST_KEXT [77595812] fipspost_post:169: PASSED: (1 ms) - fipspost_post_aes_ccm
FIPSPOST_KEXT [77765500] fipspost_post:171: PASSED: (1 ms) - fipspost_post_aes_gcm
FIPSPOST_KEXT [77941875] fipspost_post:172: PASSED: (1 ms) - fipspost_post_aes_xts
FIPSPOST_KEXT [78176875] fipspost_post:173: PASSED: (1 ms) - fipspost_post_tdes_cbc
FIPSPOST_KEXT [78338625] fipspost_post:174: PASSED: (1 ms) - fipspost_post_drbg_hmac
FIPSPOST_KEXT [78460125] fipspost_post:197: all tests PASSED (233 ms)
AUC[<ptr>]::init(<ptr>)
AUC[<ptr>]::probe(<ptr>, <ptr>)
Darwin Image4 Validation Extension Version 1.0.0: Mon Jul  9 21:36:59 PDT 2018; root:AppleImage4-1.200.16~357/AppleImage4/RELEASE_ARM64
AppleCredentialManager: init: called, instance = <ptr>.
ACMRM: init: called, ACMDRM_ENABLED=YES, ACMDRM_STATE_PUBLISHING_ENABLED=YES, ACMDRM_KEYBAG_OBSERVING_ENABLED=YES.
ACMRM: _loadRestrictedModeForceEnable: restricted mode force-enabled = 0 .
ACMRM-A: init: called, .
ACMRM-A: _loadAnalyticsCollectionPeriod: analytics collection period = 86400 .
ACMRM: _getDefaultStandardModeTimeout: default standard mode timeout = 604800 .
ACMRM: _loadStandardModeTimeout: standard mode timeout = 604800 .
ACMRM-A: notifyStandardModeTimeoutChanged: called, value = 604800 (modified = YES).
ACMRM: _loadGracePeriodTimeout: device lock timeout = 3600 .
ACMRM-A: notifyGracePeriodTimeoutChanged: called, value = 3600 (modified = YES).
AppleCredentialManager: init: returning, result = true, instance = <ptr>.
AUC[<ptr>]::start(<ptr>)
AppleKeyStore starting (BUILT: Jul  9 2018 21:51:06)
AppleSEPKeyStore::start: _sep_enabled = 1
AppleCredentialManager: start: called, instance = <ptr>.
AppleCredentialManager: start: initializing power management, instance = <ptr>.
AppleCredentialManager: start: started, instance = <ptr>.
AppleCredentialManager: start: returning, result = true, instance = <ptr>.
AppleARMPE::getGMTTimeOfDay can not provide time of day: RTC did not show up
: apfs_module_start:1277: load: com.apple.filesystems.apfs, v748.200.53, 748.200.53.0.1, 2018/07/09
com.apple.AppleFSCompressionTypeZlib kmod start
IOSurfaceRoot::installMemoryRegions()
IOSurface disallowing global lookups
apfs_sysctl_register:818: done registering sysctls.
com.apple.AppleFSCompressionTypeZlib load succeeded
L2TP domain init
L2TP domain init complete
PPTP domain init
BSD root: md0, major 2, minor 0
apfs_vfsop_mountroot:1468: apfs: mountroot called!
apfs_vfsop_mount:1231: unable to root from devvp <ptr> (root_device): 2
apfs_vfsop_mountroot:1472: apfs: mountroot failed, error: 2
hfs: mounted PeaceSeed16A5327f.arm64UpdateRamDisk on device b(2, 0)
: : Darwin Bootstrapper Version 6.0.0: Mon Jul  9 00:39:56 PDT 2018; root:libxpc_executables-1336.200.86~25/launchd/RELEASE_ARM64
boot-args = debug=0x8 kextlog=0xfff cpus=1 rd=md0
Thu Jan  1 00:00:05 1970 localhost com.apple.xpc.launchd[1] <Notice>: Restore environment starting.

If you would like to examine iOS’s boot process yourself, here’s how you can try it out.

Building QEMU

The emulation uses a patched copy of QEMU, which must be compiled from source.

Install dependencies

To compile QEMU, you first need to install some libraries.

macOS:

According to the QEMU wiki and the Homebrew recipe, you need to install Xcode and Homebrew, then run

brew install pkg-config libtool jpeg glib pixman

to install the required libraries to compile QEMU.

Ubuntu 18.04:

According to the QEMU wiki, run

sudo apt install libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev libsdl1.2-dev

to install the required libraries to compile QEMU.

Windows:

QEMU can be built on Windows, but their instructions doesn’t seem to work for this modified QEMU. Please build on macOS or Linux instead. You can set up a virtual machine running Ubuntu 18.04 with Virtualbox or VMWare Player.

Download and build source

Open a terminal, and run

git clone https://github.com/zhuowei/qemu.git
cd qemu
git submodule init
git submodule update
mkdir build-aarch64
cd build-aarch64
../configure --target-list=aarch64-softmmu
make -j4

Preparing iOS files for QEMU

Once QEMU is compiled, you need to obtain the required iOS kernelcache, device tree, and ramdisk.

If you don’t want to extract these files yourself, I packaged all the files you need from iOS 12 beta 4. You can download this archive if you sign up for my mailing list.

If you want to extract your own files directly from an iOS update, here’s how:

1. Download the required files:

git clone https://github.com/zhuowei/XNUQEMUScripts.git
cd XNUQEMUScripts

2. Extract the kernel using Joker:

./joker.universal -dec ~/path/to/iphonex12b4/kernelcache.release.iphone10b
mv /tmp/kernel kcache_out.bin

replace joker.universal with joker.ELF64 if you are using Linux.

3. extract the ramdisk:

dd if=~/path/to/iphonex12b4/048-22007-059.dmg bs=27 skip=1 of=ramdisk.dmg

4. Modify the devicetree.dtb file:

python3 modifydevicetree.py ~/Path/To/iphonex12b4/Firmware/all_flash/DeviceTree.d22ap.im4p devicetree.dtb

Installing a debugger

You will also need lldb or gdb for arm64 installed.

macOS

The version of lldb included in Xcode 9.3 should work. (Later versions should also work.) You don’t need to install anything in this step.

Ubuntu 18.04

I can’t find an LLDB compatible with ARM64: neither the LLDB from the Ubuntu repository nor the version from LLVM’s own repos support ARM64. (Someone please build one!)

Instead, you can use GDB on Linux.

Two versions of GDB can be used: the version from devkitA64, or the Linaro GDB (recommended).

Enter your xnuqemu directory (from the downloaded package or from the clone of the XNUQEMUScripts repo)

Run

./linux_installgdb.sh

to download the Linaro GDB.

Running QEMU

Place your qemu directory into the same directory as the scripts, kernel, devicetree, and ramdisk.

You should have these files:

~/xnuqemu_dist$ ls
README.md                       lldbit.sh
devicetree.dtb                  lldbscript.lldb
devicetreefromim4p.py           modifydevicetree.py
fixbootdelay_lldbscript_doc.txt qemu
gdbit.sh                        ramdisk.dmg
gdbscript.gdb                   readdevicetree.py
kcache_out.bin                  runqemu.sh
linux_installgdb.sh             windows_installgdb.sh

./runqemu.sh to start QEMU.

$ ./runqemu.sh 
QEMU 2.12.90 monitor - type 'help' for more information
(qemu) xnu

in a different terminal, ./lldbit.sh to start lldb, or if you’re using Linux, ./gdbit.sh to start gdb.

$ ./lldbit.sh 
(lldb) target create "kcache_out.bin"
Current executable set to 'kcache_out.bin' (arm64).
(lldb) process connect --plugin gdb-remote connect://127.0.0.1:1234
(lldb) command source -s 0 'lldbscript.lldb'
Executing commands in 'lldbscript.lldb'.
(lldb) b *0xFFFFFFF007433BE8
Breakpoint 1: address = 0xfffffff007433be8
(lldb) breakpoint command add
Enter your debugger command(s).  Type 'DONE' to end.
(lldb) b *0xFFFFFFF005FA5D84
Breakpoint 2: address = 0xfffffff005fa5d84
(lldb) breakpoint command add
Enter your debugger command(s).  Type 'DONE' to end.
(lldb) b *0xfffffff00743e434
Breakpoint 3: address = 0xfffffff00743e434
(lldb) breakpoint command add
Enter your debugger command(s).  Type 'DONE' to end.
(lldb) b *0xfffffff00743e834
Breakpoint 4: address = 0xfffffff00743e834
(lldb)

Type c into lldb or gdb to start execution.

In the terminal running QEMU, you should see boot messages. Congratulations, you’ve just ran a tiny bit of iOS with a virtual iPhone! Or as UnthreadedJB would say, “#we r of #fakr!”

What works

  • Booting XNU all the way to running userspace programs
  • Console output from virtual serial port

What doesn’t work

  • Wi-Fi
  • Bluetooth
  • USB
  • Screen
  • Internal storage
  • Everything except the serial port
  • You tell me

Seriously, though, this only runs a tiny bit of iOS, and is nowhere close to iOS emulation. To borrow a simile from the creator of Corellium, if Corellium is a DeLorean time machine, then this is half a wheel at most.

This experiment only finished the easy part of booting iOS, as it doesn’t emulate an iPhone at all, relying on only the parts common to all ARM devices. No drivers are loaded whatsoever, so there’s no emulation of the screen, the USB, the internal storage… You name it: it doesn’t work.

For full iOS emulation, the next step would be reverse engineering the iPhone’s SoC to find out how its peripherals work. Unfortunately, that’s a 9-year project, as shown by the development history of Corellium. I can’t do that on my own - that’s why I wrote this tutorial!

It’s my hope that this work inspires others to look into proper iOS emulation - from what I’ve seen, it’ll be a great learning experience.

How I did this

Last week, I started modifying QEMU to load an iOS kernel and device tree: the previous writeup is here. Here’s how I got from crashing when loading kernel modules to fully booting the kernel.

Tweaking CPU emulation, part 3: Interrupting cow

When we left off, the kernel crashed with a data abort when it tries to bzero a write only region of memory. Why?

To confirm that it’s indeed writing to read-only memory, I implemented a command to dump out the kernel memory mappings, and enabled QEMU’s verbose MMU logging to detect changes to the memory map.

I tracked down the crashing code to OSKext::updateLoadedKextSummaries. After every kext load, this code resets the kext summaries region to writable with vm_map_protect, writes information for the new kext, then sets the region back to read-only. The logs show that the call to protect the region modifies the memory mappings, but the call to reset it to read-write doesn’t do anything. Why isn’t it setting the page to writable?

According to comments in vm_map_protect, it turns out that readonly->readwrite calls actually don’t change the protection immediately, but only sets it on-demand when a program tries - and fails - to write to the page. This is to implement copy on write.

So, it seems the data abort exception is supposed to happen, but the panic is not.

In the data abort exception, the page should be set to writable in arm_fast_fault. The code in open-source XNU can only return KERN_FAILURE or KERN_SUCCESS, but with a breakpoint, I saw it was returning KERN_PROTECTION_FAILURE.

I checked the disassembly: yes, there’s extra code (0xFFFFFFF0071F953C in iOS 12 beta 4) returning KERN_PROTECTION_FAILURE if the page address doesn’t match one of the new KTRR registers added on the A11 processor .

I had been ignoring all writes to KTRR registers, so this code can’t read the value from the register (which the kernel stored at startup), and believes that all addresses are invalid. Thus, instead of setting the page to writable, the kernel panics instead.

I fixed this by adding these registers to QEMU’s virtual CPU, allowing the kernel to read and write them.

After this change, a few more kexts started up, but the kernel then hangs… like it’s waiting for something.

Connecting the timer interrupt

My hunch for why the kernel hangs: one of the kexts tries to sleep for some time during initialization, but never wakes up because there are no timer interrupts, as shown by QEMU not logging any exceptions when it hangs.

On ARM, there are two ways for hardware to signal the CPU: IRQ, shared by many devices, or FIQ, dedicated to just one device.

QEMU’s virt machine hooks up the processor’s timer to IRQ, like most real ARM platforms. FIQ is usually reserved for debuggers.

Apple, however, hooks up the timer directly to the FIQ. With virt’s timer hooked up to the wrong signal, the kernel would wait forever for an interrupt that would never come.

All I had to do to get the timer working was to hook it up to FIQ. This gets me… a nice panic in the Image4 parser.

Getting the Image4 parser module working

panic(cpu 0 caller 0xfffffff006c1edb8): "could not instantiate ppl environment: 0x60"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleImage4/AppleImage4-1.200.12/include/abort.h:24

What does this mean? What’s error 0x60?

I found the panic string, and looked for where the error message is generated.

It turns out that the Image4 parser queries the device tree for various nodes in “/chosen” or “/default”; if the value doesn’t exist, it returns error 0x60. If the value is the wrong size, it returns 0x54.

iOS’s device tree is missing two properties: chip-epoch and security-domain, which causes the module to panic with the 0x60 error.

Oddly, the device tree doesn’t reserve extra space for these properties. I had to delete two existing properties to make space for them.

With the modified device tree, the Image4 module initializes, but now I have a panic from a data abort in rorgn_lockdown.

Failed attempt to get device drivers to not crash

Of course the KTRR driver crashes when it tries to access the memory controller: there isn’t one! QEMU’s virt machine doesn’t have anything mapped at that address.

Since I don’t have an emulation of the memory controller, I just added a block of empty memory to avoid the crash.

This strategy didn’t work for the next crash, though, from the AppleInterruptController driver. That driver reads and validates values from the device, so just placing a blank block of memory causes the driver to panic.

Something more drastic is needed if I don’t want to spend 9 years reverse engineering each driver.

Driverless like we’re Waymo

To boot XNU, I don’t really need all those drivers, do I? Who needs interrupts or the screen or power management or storage, anyways? All XNU needs to boot into userspace is a serial port and a timer.

I disabled every other driver in the kernel. Drivers are loaded if their IONameMatch property corresponds to a device’s “compatible”, “name”, or “device_type” fields. To disable all the drivers, I erased every “compatible” property in the device tree, along with a few “name” and “device_type” properties.

Now, with no drivers, XNU seems to hang, but after I patiently waited for a minute…

Waiting on <dict ID="0"><key>IOProviderClass</key><string ID="1">IOMedia</string><key>Content</key><string ID="2">Apple_HFS</string></dict>
Still waiting for root device

It’s trying to mount the root filesystem!

Loading a RAMDisk

If it’s looking for a root filesystm, let’s give it one. I don’t have any drivers for storage, but I can mount an iOS Recovery RAMDisk, which requires no drivers. All I had to do was:

  • Load the ramdisk at the end of the kernel, just before the device tree blob
  • put its address and size in the device tree so XNU can find it
  • set boot argument to rd=md0 to boot from ramdisk
hfs: mounted PeaceSeed16A5327f.arm64UpdateRamDisk on device b(2, 0)

The kernel mounts the root filesystem! … but then hangs again.

Using LLDB to patch out hanging functions

By putting breakpoints all over bsd_init, I found that the kernel was hanging in IOBSDSecureRoot, when it tries to call the platform function. The platform function looks for a device, but since I removed all the device drivers, it waits forever, in vain.

To fix this, I just skipped the problematic call. I used an LLDB breakpoint to jump over the call and simulate a true return instead.

And, after three weeks, the virtual serial port finally printed out:

: : Darwin Bootstrapper Version 6.0.0: Mon Jul  9 00:39:56 PDT 2018; root:libxpc_executables-1336.200.86~25/launchd/RELEASE_ARM64

“Houston, the kernel has booted.”

What I learned

  • quirks of iOS memory management
  • how iOS handles timer interrupts
  • how iOS loads ramdisks
  • building QEMU on different platforms
  • modifying QEMU to add new CPU configuration registers
  • differences between GDB and LLDB’s command syntax
  • how to get people to subscribe to my mailing list. (muhahaha, one last signup link.)

Thanks

Thanks to everyone who shared or commented on my last article. To those who tried building and running it - sorry about taking so long to write up instructions!

Thanks to @matteyeux, @h3adsh0tzz, @_th0ex, and @enzolovesbacon for testing the build instructions.

Thanks to @winocm, whose darwin-on-arm project originally inspired me to learn about the XNU kernel.