recoveryd to start on an emulated iPhone running iOS 12 beta 4’s kernel using a modified QEMU. Here’s what I learned, and how you can try this yourself.
First, let me repeat: this is completely useless unless you’re really interested in iOS internals. If you want to run iOS, you should ask @CorelliumHQ instead, or just buy an iPhone.
I’ve been interested in how iOS starts, so I’ve been trying to boot the iOS kernel in QEMU.
I was inspired by @cmwdotme’s Corellium, a service which can boot any iOS in a virtual machine. Since I don’t have 9 years to build a perfect simulation of an iPhone, I decided to go for a less lofty goal: getting enough of iOS emulated until
launchd, the first program to run when iOS boots, is able to start.
Since last week’s post, I got the iOS 12 beta 4 kernel to fully boot in QEMU, and even got it to run
launchd and start
recoveryd from the restore ramdisk. Here’s the output from the virtual serial port:
iBoot version: corecrypto_kext_start called FIPSPOST_KEXT  fipspost_post:156: PASSED: (4 ms) - fipspost_post_integrity FIPSPOST_KEXT  fipspost_post:162: PASSED: (1 ms) - fipspost_post_hmac FIPSPOST_KEXT  fipspost_post:163: PASSED: (0 ms) - fipspost_post_aes_ecb FIPSPOST_KEXT  fipspost_post:164: PASSED: (0 ms) - fipspost_post_aes_cbc FIPSPOST_KEXT  fipspost_post:165: PASSED: (117 ms) - fipspost_post_rsa_sig FIPSPOST_KEXT  fipspost_post:166: PASSED: (67 ms) - fipspost_post_ecdsa FIPSPOST_KEXT  fipspost_post:167: PASSED: (11 ms) - fipspost_post_ecdh FIPSPOST_KEXT  fipspost_post:168: PASSED: (0 ms) - fipspost_post_drbg_ctr FIPSPOST_KEXT  fipspost_post:169: PASSED: (1 ms) - fipspost_post_aes_ccm FIPSPOST_KEXT  fipspost_post:171: PASSED: (1 ms) - fipspost_post_aes_gcm FIPSPOST_KEXT  fipspost_post:172: PASSED: (1 ms) - fipspost_post_aes_xts FIPSPOST_KEXT  fipspost_post:173: PASSED: (1 ms) - fipspost_post_tdes_cbc FIPSPOST_KEXT  fipspost_post:174: PASSED: (1 ms) - fipspost_post_drbg_hmac FIPSPOST_KEXT  fipspost_post:197: all tests PASSED (233 ms) AUC[<ptr>]::init(<ptr>) AUC[<ptr>]::probe(<ptr>, <ptr>) Darwin Image4 Validation Extension Version 1.0.0: Mon Jul 9 21:36:59 PDT 2018; root:AppleImage4-1.200.16~357/AppleImage4/RELEASE_ARM64 AppleCredentialManager: init: called, instance = <ptr>. ACMRM: init: called, ACMDRM_ENABLED=YES, ACMDRM_STATE_PUBLISHING_ENABLED=YES, ACMDRM_KEYBAG_OBSERVING_ENABLED=YES. ACMRM: _loadRestrictedModeForceEnable: restricted mode force-enabled = 0 . ACMRM-A: init: called, . ACMRM-A: _loadAnalyticsCollectionPeriod: analytics collection period = 86400 . ACMRM: _getDefaultStandardModeTimeout: default standard mode timeout = 604800 . ACMRM: _loadStandardModeTimeout: standard mode timeout = 604800 . ACMRM-A: notifyStandardModeTimeoutChanged: called, value = 604800 (modified = YES). ACMRM: _loadGracePeriodTimeout: device lock timeout = 3600 . ACMRM-A: notifyGracePeriodTimeoutChanged: called, value = 3600 (modified = YES). AppleCredentialManager: init: returning, result = true, instance = <ptr>. AUC[<ptr>]::start(<ptr>) AppleKeyStore starting (BUILT: Jul 9 2018 21:51:06) AppleSEPKeyStore::start: _sep_enabled = 1 AppleCredentialManager: start: called, instance = <ptr>. AppleCredentialManager: start: initializing power management, instance = <ptr>. AppleCredentialManager: start: started, instance = <ptr>. AppleCredentialManager: start: returning, result = true, instance = <ptr>. AppleARMPE::getGMTTimeOfDay can not provide time of day: RTC did not show up : apfs_module_start:1277: load: com.apple.filesystems.apfs, v748.200.53, 718.104.22.168.1, 2018/07/09 com.apple.AppleFSCompressionTypeZlib kmod start IOSurfaceRoot::installMemoryRegions() IOSurface disallowing global lookups apfs_sysctl_register:818: done registering sysctls. com.apple.AppleFSCompressionTypeZlib load succeeded L2TP domain init L2TP domain init complete PPTP domain init BSD root: md0, major 2, minor 0 apfs_vfsop_mountroot:1468: apfs: mountroot called! apfs_vfsop_mount:1231: unable to root from devvp <ptr> (root_device): 2 apfs_vfsop_mountroot:1472: apfs: mountroot failed, error: 2 hfs: mounted PeaceSeed16A5327f.arm64UpdateRamDisk on device b(2, 0) : : Darwin Bootstrapper Version 6.0.0: Mon Jul 9 00:39:56 PDT 2018; root:libxpc_executables-1336.200.86~25/launchd/RELEASE_ARM64 boot-args = debug=0x8 kextlog=0xfff cpus=1 rd=md0 Thu Jan 1 00:00:05 1970 localhost com.apple.xpc.launchd <Notice>: Restore environment starting.
If you would like to examine iOS’s boot process yourself, here’s how you can try it out.
The emulation uses a patched copy of QEMU, which must be compiled from source.
To compile QEMU, you first need to install some libraries.
brew install pkg-config libtool jpeg glib pixman
to install the required libraries to compile QEMU.
According to the QEMU wiki, run
sudo apt install libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev libsdl1.2-dev
to install the required libraries to compile QEMU.
QEMU can be built on Windows, but their instructions doesn’t seem to work for this modified QEMU. Please build on macOS or Linux instead. You can set up a virtual machine running Ubuntu 18.04 with Virtualbox or VMWare Player.
Download and build source
Open a terminal, and run
git clone https://github.com/zhuowei/qemu.git cd qemu git submodule init git submodule update mkdir build-aarch64 cd build-aarch64 ../configure --target-list=aarch64-softmmu make -j4
Preparing iOS files for QEMU
Once QEMU is compiled, you need to obtain the required iOS kernelcache, device tree, and ramdisk.
If you don’t want to extract these files yourself, I packaged all the files you need from iOS 12 beta 4. You can download this archive if you sign up for my mailing list.
If you want to extract your own files directly from an iOS update, here’s how:
1. Download the required files:
- Download my XNUQEMUScripts repository:
git clone https://github.com/zhuowei/XNUQEMUScripts.git cd XNUQEMUScripts
Download the iOS 12 beta 4 for iPhone X.
To decompress the kernel, download newosxbook’s Joker tool.
2. Extract the kernel using Joker:
./joker.universal -dec ~/path/to/iphonex12b4/kernelcache.release.iphone10b mv /tmp/kernel kcache_out.bin
joker.ELF64 if you are using Linux.
3. extract the ramdisk:
dd if=~/path/to/iphonex12b4/048-22007-059.dmg bs=27 skip=1 of=ramdisk.dmg
4. Modify the devicetree.dtb file:
python3 modifydevicetree.py ~/Path/To/iphonex12b4/Firmware/all_flash/DeviceTree.d22ap.im4p devicetree.dtb
Installing a debugger
You will also need lldb or gdb for arm64 installed.
The version of lldb included in Xcode 9.3 should work. (Later versions should also work.) You don’t need to install anything in this step.
I can’t find an LLDB compatible with ARM64: neither the LLDB from the Ubuntu repository nor the version from LLVM’s own repos support ARM64. (Someone please build one!)
Instead, you can use GDB on Linux.
Enter your xnuqemu directory (from the downloaded package or from the clone of the XNUQEMUScripts repo)
to download the Linaro GDB.
qemu directory into the same directory as the scripts, kernel, devicetree, and ramdisk.
You should have these files:
~/xnuqemu_dist$ ls README.md lldbit.sh devicetree.dtb lldbscript.lldb devicetreefromim4p.py modifydevicetree.py fixbootdelay_lldbscript_doc.txt qemu gdbit.sh ramdisk.dmg gdbscript.gdb readdevicetree.py kcache_out.bin runqemu.sh linux_installgdb.sh windows_installgdb.sh
./runqemu.sh to start QEMU.
$ ./runqemu.sh QEMU 2.12.90 monitor - type 'help' for more information (qemu) xnu
in a different terminal,
./lldbit.sh to start lldb, or if you’re using Linux,
./gdbit.sh to start gdb.
$ ./lldbit.sh (lldb) target create "kcache_out.bin" Current executable set to 'kcache_out.bin' (arm64). (lldb) process connect --plugin gdb-remote connect://127.0.0.1:1234 (lldb) command source -s 0 'lldbscript.lldb' Executing commands in 'lldbscript.lldb'. (lldb) b *0xFFFFFFF007433BE8 Breakpoint 1: address = 0xfffffff007433be8 (lldb) breakpoint command add Enter your debugger command(s). Type 'DONE' to end. (lldb) b *0xFFFFFFF005FA5D84 Breakpoint 2: address = 0xfffffff005fa5d84 (lldb) breakpoint command add Enter your debugger command(s). Type 'DONE' to end. (lldb) b *0xfffffff00743e434 Breakpoint 3: address = 0xfffffff00743e434 (lldb) breakpoint command add Enter your debugger command(s). Type 'DONE' to end. (lldb) b *0xfffffff00743e834 Breakpoint 4: address = 0xfffffff00743e834 (lldb)
c into lldb or gdb to start execution.
- Booting XNU all the way to running userspace programs
- Console output from virtual serial port
What doesn’t work
- Internal storage
- Everything except the serial port
- You tell me
Seriously, though, this only runs a tiny bit of iOS, and is nowhere close to iOS emulation. To borrow a simile from the creator of Corellium, if Corellium is a DeLorean time machine, then this is half a wheel at most.
This experiment only finished the easy part of booting iOS, as it doesn’t emulate an iPhone at all, relying on only the parts common to all ARM devices. No drivers are loaded whatsoever, so there’s no emulation of the screen, the USB, the internal storage… You name it: it doesn’t work.
For full iOS emulation, the next step would be reverse engineering the iPhone’s SoC to find out how its peripherals work. Unfortunately, that’s a 9-year project, as shown by the development history of Corellium. I can’t do that on my own - that’s why I wrote this tutorial!
It’s my hope that this work inspires others to look into proper iOS emulation - from what I’ve seen, it’ll be a great learning experience.
How I did this
Last week, I started modifying QEMU to load an iOS kernel and device tree: the previous writeup is here. Here’s how I got from crashing when loading kernel modules to fully booting the kernel.
Tweaking CPU emulation, part 3: Interrupting cow
When we left off, the kernel crashed with a data abort when it tries to
bzero a write only region of memory. Why?
I tracked down the crashing code to
OSKext::updateLoadedKextSummaries. After every kext load, this code resets the kext summaries region to writable with
vm_map_protect, writes information for the new kext, then sets the region back to read-only. The logs show that the call to protect the region modifies the memory mappings, but the call to reset it to read-write doesn’t do anything. Why isn’t it setting the page to writable?
According to comments in
vm_map_protect, it turns out that readonly->readwrite calls actually don’t change the protection immediately, but only sets it on-demand when a program tries - and fails - to write to the page. This is to implement copy on write.
So, it seems the data abort exception is supposed to happen, but the panic is not.
In the data abort exception, the page should be set to writable in
arm_fast_fault. The code in open-source XNU can only return KERN_FAILURE or KERN_SUCCESS, but with a breakpoint, I saw it was returning KERN_PROTECTION_FAILURE.
I checked the disassembly: yes, there’s extra code (
0xFFFFFFF0071F953C in iOS 12 beta 4) returning KERN_PROTECTION_FAILURE if the page address doesn’t match one of the new KTRR registers added on the A11 processor .
I had been ignoring all writes to KTRR registers, so this code can’t read the value from the register (which the kernel stored at startup), and believes that all addresses are invalid. Thus, instead of setting the page to writable, the kernel panics instead.
I fixed this by adding these registers to QEMU’s virtual CPU, allowing the kernel to read and write them.
After this change, a few more kexts started up, but the kernel then hangs… like it’s waiting for something.
Connecting the timer interrupt
My hunch for why the kernel hangs: one of the kexts tries to sleep for some time during initialization, but never wakes up because there are no timer interrupts, as shown by QEMU not logging any exceptions when it hangs.
On ARM, there are two ways for hardware to signal the CPU: IRQ, shared by many devices, or FIQ, dedicated to just one device.
virt machine hooks up the processor’s timer to IRQ, like most real ARM platforms. FIQ is usually reserved for debuggers.
Apple, however, hooks up the timer directly to the FIQ. With
virt’s timer hooked up to the wrong signal, the kernel would wait forever for an interrupt that would never come.
Getting the Image4 parser module working
panic(cpu 0 caller 0xfffffff006c1edb8): "could not instantiate ppl environment: 0x60"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleImage4/AppleImage4-1.200.12/include/abort.h:24
What does this mean? What’s error 0x60?
I found the panic string, and looked for where the error message is generated.
It turns out that the Image4 parser queries the device tree for various nodes in “/chosen” or “/default”; if the value doesn’t exist, it returns error 0x60. If the value is the wrong size, it returns 0x54.
iOS’s device tree is missing two properties:
security-domain, which causes the module to panic with the 0x60 error.
Oddly, the device tree doesn’t reserve extra space for these properties. I had to delete two existing properties to make space for them.
With the modified device tree, the Image4 module initializes, but now I have a panic from a data abort in rorgn_lockdown.
Failed attempt to get device drivers to not crash
Of course the KTRR driver crashes when it tries to access the memory controller: there isn’t one! QEMU’s
virt machine doesn’t have anything mapped at that address.
Since I don’t have an emulation of the memory controller, I just added a block of empty memory to avoid the crash.
This strategy didn’t work for the next crash, though, from the AppleInterruptController driver. That driver reads and validates values from the device, so just placing a blank block of memory causes the driver to panic.
Something more drastic is needed if I don’t want to spend 9 years reverse engineering each driver.
Driverless like we’re Waymo
To boot XNU, I don’t really need all those drivers, do I? Who needs interrupts or the screen or power management or storage, anyways? All XNU needs to boot into userspace is a serial port and a timer.
I disabled every other driver in the kernel. Drivers are loaded if their
IONameMatch property corresponds to a device’s “compatible”, “name”, or “device_type” fields. To disable all the drivers, I erased every “compatible” property in the device tree, along with a few “name” and “device_type” properties.
Now, with no drivers, XNU seems to hang, but after I patiently waited for a minute…
Waiting on <dict ID="0"><key>IOProviderClass</key><string ID="1">IOMedia</string><key>Content</key><string ID="2">Apple_HFS</string></dict> Still waiting for root device
It’s trying to mount the root filesystem!
Loading a RAMDisk
If it’s looking for a root filesystm, let’s give it one. I don’t have any drivers for storage, but I can mount an iOS Recovery RAMDisk, which requires no drivers. All I had to do was:
- Load the ramdisk at the end of the kernel, just before the device tree blob
- put its address and size in the device tree so XNU can find it
- set boot argument to
rd=md0to boot from ramdisk
hfs: mounted PeaceSeed16A5327f.arm64UpdateRamDisk on device b(2, 0)
The kernel mounts the root filesystem! … but then hangs again.
Using LLDB to patch out hanging functions
By putting breakpoints all over
bsd_init, I found that the kernel was hanging in
IOBSDSecureRoot, when it tries to call the platform function. The platform function looks for a device, but since I removed all the device drivers, it waits forever, in vain.
To fix this, I just skipped the problematic call. I used an LLDB breakpoint to jump over the call and simulate a
true return instead.
And, after three weeks, the virtual serial port finally printed out:
: : Darwin Bootstrapper Version 6.0.0: Mon Jul 9 00:39:56 PDT 2018; root:libxpc_executables-1336.200.86~25/launchd/RELEASE_ARM64
“Houston, the kernel has booted.”
What I learned
- quirks of iOS memory management
- how iOS handles timer interrupts
- how iOS loads ramdisks
- building QEMU on different platforms
- modifying QEMU to add new CPU configuration registers
- differences between GDB and LLDB’s command syntax
- how to get people to subscribe to my mailing list. (muhahaha, one last signup link.)
Thanks to everyone who shared or commented on my last article. To those who tried building and running it - sorry about taking so long to write up instructions!