I made a Swift Playground for iOS 12 that opens the Memoji editor on iPad. Along the way, I learned to swizzle Obj-C methods in Swift and to use a library with no documentation.

Download

screenshot of the Memoji editor in Swift Playgrounds

Screenshot of a Memoji on an iPad Pro 10.5, by Filippo Claudi. (thanks!)

You need an iPad running iOS 12 beta 6 and Swift Playgrounds. The original iPad Air doesn’t work - its GPU can’t handle the shaders. The iPad Pro 10.5 is known to work.

To run the Playground:

  • Sign up for my mailing list to download the Playground:
  • Unzip the downloaded Playground, and AirDrop it to your iPad.
  • turn your iPad to landscape.
  • open the playground, and press Run My Code.
  • A carousel of Animoji should appear. The leftmost should be a + sign.
  • tap on the + to open the Memoji creator.

Introduction

When Apple introduced Animoji last year, they were only available on the iPhone X. In iOS 12, Apple added the code for Animoji and Memoji to iPads as well, but made it hidden and not launchable.

Recently, @stroughtonsmith managed to unlock Animoji on an iPad with iOS 12, and found that the Animoji rendered fine, although there’s no face tracking as the iPad doesn’t have the TrueDepth Camera.

This means I can launch the most fun part of Animoji: the Memoji editor, which doesn’t need face tracking. Thus, I made a Swift Playground that opens the Animoji carousel on iOS 12 and lets you create Memoji on iPad.

How I did this

This was supposed to be a simple project, but it ended up taking two months because of a detour. In the end, it took four steps:

  • Launching Animoji in the Xcode Simulator
  • Porting to real device
  • Porting to Swift
  • Porting to Playgrounds

Launching in the Simulator

The Xcode simulator for iOS 12 already contains a working copy of the Animoji frameworks. Thanks to the research by @simonbs, I knew that Animoji are implemented in /System/Library/PrivateFrameworks/AvatarKit.framework. On iOS 12, a new framework was also added: /System/Library/PrivateFrameworks/AvatarUI.framework, which, from its name and resources, seems to contain the new Memoji editor.

I copied the AvatarUI framework from the Simulator, and used Class-dump to dump its Objective-C headers. This gave me 249 classes. I have to find which of the 249 classes lets me start the library.

When I’m examining the structure of a program to see how to invoke some function, I usually identify some code that’s obviously involved, and then go “upwards” to see what uses that code.

In this case, I know that, when a user uses the Memoji editor for the first time, an onboarding view controller shows up. I found AVTSplashScreenViewController, which is responsible for that screen. Then, I looked for the code that launches this view controller.

To find code referring to AVTSplashScreenViewController, I loaded the simulator’s framework in IDA Free, looked for the class in the objc_classlist section, and used the “xrefs” function to find references to the class. This finds all the code that calls class methods on AVTSplashScreenViewController - such as any constructors.

Using this technique, I found that this view controller is created by -[AVTAvatarEditorViewController loadSplashScreen]. That class name sounds promising, but its constructor needs an avatar store, which I don’t know how to construct.

Let’s go up one more level: who creates the AVTAvatarEditorViewController? It’s -[AVTAvatarActionsModel didTapCreateNew:].

Who creates the AVTAvatarActionsModel? -[AVTCarouselController presentActionsForAvatar:].

The AVTCarouselController seems like a good candidate for an entry point to the whole library: judging by its name, it would present Animoji in a scrollable list. The header corroborates that guess:

@interface AVTCarouselController : NSObject
// irrelevant properties/methods removed
@property(strong, nonatomic) UIView *avtViewContainer;
@property(strong, nonatomic) AVTMultiAvatarController *multiAvatarController;
@property(weak, nonatomic) id <AVTPresenterDelegate> presenterDelegate;
+ (AVTCarouselController*)displayingCarouselForRecordDataSource:(id)arg1;
- (UIView*)view;
@end

The constructor only takes one argument; based on the method name, it’s probably an AVTAvatarRecordDataSource. The data source just needs a domain identifier:

@interface AVTAvatarRecordDataSource : NSObject
// irrelevant properties/methods removed
+ (AVTAvatarRecordDataSource*)defaultUIDataSourceWithDomainIdentifier:(NSString*)arg1;
@end

I knew I found the library’s entry point because everything’s well encapsulated and easy to use: I just need to:

  • provide a string domain identifier to make a data source
  • pass the data source into the AVTCarouselController
  • and add its view to the view controller.

This is where I ran into my first issue with reference counting in this project. When I added the view from AVTCarouselController, nothing showed up. Inspect View Hierarchy in Xcode showed an empty UICollectionView, with a nil data source. That’s odd: it wasn’t nil when I added the view.

I finally remembered that a UICollectionView holds its data source in a weak pointer, and doesn’t keep the data source alive. I had allocated the AVTCarouselController as a temporary variable, so of course it was deallocated when my method returned. Storing a reference to the carousel in my view controller fixed the issue.

Unfortunately, the Simulator doesn’t support Metal, so Animoji/Memoji’s shaders couldn’t run, leaving me with pink-haired avatars:

glitched rendering of the Memoji editor from Simulator

Yeah, I’m not cool enough to have pink hair. I would need to run on a real device to render the Animoji and Memoji properly.

Porting to a real device

I wasted two months trying to port AvatarKit’s code to a real device, only to find that the code was already there.

I started by checking if the iPad Air’s firmware contains the two needed frameworks: in the iOS 12 beta, there’s no /System/Library/PrivateFrameworks/AvatarKit.framework. I thought this meant the library isn’t on the iPad, so I spent two months trying to extract the framework’s executable code from the iPhone X firmware.

Then @stroughtonsmith showed that the code for AvatarKit and AvatarUI are on the iPad all along - it’s just the data files that are missing. Those can be easily copied unmodified from the iPhone X. So all that work to extract the code was wasted. Lesson learned: before trying to port something, check if that thing already exists on the target platform…

I did have to override two parts of the Animoji code. The first change allows the Animoji code to find its resources. The Animoji libraries, AvatarKit.framework and AvatarUI.framework, uses the -[NSBundle bundleForClass:]method to find their own resources. On the iPad, as mentioned before, these resources don’t exist, so this method returns the main bundle instead.

Stroughtonsmith worked around this by simply copying all the resources from AvatarKit.framework into the launcher app’s main bundle. However, I need to launch the Memoji editor, which requires resources from AvatarUI.framework as well. I can’t just copy both their resources to the same main bundle, since one framework would overwrite the other’s files.

Instead, I swizzled the -[NSBundle bundleForClass:] method, following the steps I learned when I fixed tabs in VS Code, to return the copies of the frameworks I extracted from the iPhone X and bundled with my application. This allowed the Animoji carousel to show up like the Simulator, but the + sign to open the Memoji was nowhere to be seen. Instead, this error shows up in the Xcode console:

[General] Error creating backend content Error Domain=NSCocoaErrorDomain Code=513 “You don’t have permission to save the file “Avatar” in the folder “Library”.” UserInfo={NSFilePath=/var/mobile/Library/Avatar, NSUnderlyingError=0x2802a7780 {Error Domain=NSPOSIXErrorDomain Code=1 “Operation not permitted”}}

The second change switched this path to point to a writable location inside my app’s Documents folder. To do so, I had to find where this path is stored. I did this in the craziest way possible: I put a breakpoint on open to print a backtrace on every file access done by the app. Using this exhaustive log, I found that the error was raised by -[AVTArchiverBasedStoreBackend initWithStoreLocation:domainIdentifier:environment].

Using my previous reverse engineering strategy of moving upwards, I found that this method gets the store location from +[AVTArchiverBasedStoreBackend storeLocationForDomainIdentifier:environment:], which gets it from +[AVTUIEnvironment storeLocation]. This method contains the hardcoded path.

Once again, I swizzled the method to return a different location, and the Memoji editor opened! (The code can be found here.)

Unfortunately, the graphics are still glitched. In fact, the shaders crash the Metal compiler on the iPad Air. I concluded that my iPad is too old to run the needed shaders, and that I need help testing this code.

glitched Memoji Editor on iPad Air

Before I can ask others for help, I needed a simple way for others to run this code. The simplest way to run sideloaded code on an iPad is, of course, through Swift Playgrounds.

Porting to Swift

To run this application on Swift Playgrounds, I first had to port my launcher to Swift. I chose to develop it as a normal app first, since Swift Playgrounds offers no way to debug crashes. Even @IMcD23’s advice to look at the logs for ExecutionExtension didn’t help - my crash didn’t produce any output.

I decided to just port my working Objective-C program line by line to Swift. The swizzling was exactly the same as the Objective-C version, which was neat - I was expecting trouble from the function pointer casts, but it mostly just worked. Only one quirk: I had to cast the function to a @convention(c) function pointer and store this in a variable before the unsafeBitCast. Just using unsafeBitCast on the function directly gave me a “can’t cast between different sizes” error.

Calling the Objective-C functions in AvatarUI, on the other hand, gave me quite a bit of trouble. In Objective-C, I simply added headers for AvatarUI’s classes, and then called them normally. In Swift, there’s no way to declare a class without implementing that class. In Swift Playgrounds, I can’t make Objective-C bridging headers, so I had to call all the methods by reflection.

Reflection in Objective-C - and thus Swift - is done through the performSelector: method. However, this method returns Unmanaged as it doesn’t know the return type. To get the returned object, I need to know the retained status of the return value. I could find absolutely nothing documenting this. Apple’s own documentation on the performSelector page boiled down to “use takeUnretainedValue() unless you’re supposed to use takeRetainedValue()”. Wow, that’s helpful. I searched online for guides, but after 7 years of ARC, there’s no-one left who remembers how to manually manage memory.

Finally, I resorted to guess and check. I tried takeRetained(). App crashes. I tried takeUnretained(). App didn’t crash. I immediately saved all the references in my ViewController properties, to avoid my objects disappearing under me if I guessed wrong after all. (Once bitten with the disappearing UICollectionView dataSource, twice shy.) For setting properties, I even decided to just use Key-Value Coding to avoid this craziness again.

Conclusion: You might joke about Swift creating a generation of programmers who can’t read old code, but ARC already did that 7 years ago.

Now that my app works, I copied it into a Swift Playground, and it just worked!

The source for the app can be found here, and the Playground’s source can be found here.

screenshot of the Memoji splash screen in Swift Playgrounds

What I learned

  • Before spending 2 months porting code, check if the code’s already present on the device.
  • How to use a library with no documentation and just a pile of dumped headers.
  • I can understand a system by working from the middle up.
  • The super-verbose method and class names of Objective-C really does help me understand code.
  • How to swizzle Objective-C methods using pure Swift
  • How to call private Objective-C methods using pure Swift
  • It’s hard to choose between takeRetained and takeUnretained when converting unmanaged references, since there’s not enough documentation on dealing with reference counting. Someone should write a tutorial.
  • Swift Playgrounds are impossible to debug, so develop code as a Swift app instead, and deploy as a Playground. (Seems backwards, I know.)
  • I need a new iPad with a better GPU.

Thanks

Thanks to Filippo Claudi for the screenshot of the Playground running on an iPad Pro.