KISS

Keep It Simple Stupid

Replacing iOS application container in terminal

| comments

Xcode has the “Devices and Simulators” window (Cmd+Shift+2) where you can see the applications installed on your iOS device and more importantly can download and upload a data container for an application that you’re developing. That container contains all the data generated by that application, so that for example you can get it into a certain state where you see a bug, save it and then restore it back to device as many times as necessary. The only official way to do that is via this Xcode’s window, however I find it clumsy because you need to open the window, select the application, click the gear icon, select “Download container…” in the popup menu, select where to save it, wait and after a while a new Finder window will be helpfully open, but stealing the focus of course. This whole process is much easier with the iOS Simulator, but sometimes you just need to use the app on a real device.

As alternative, there is this great program called ios-deploy, which can almost replace a container on the device. There are a few questions on StackOverflow, like this one, about programmatically uploading/downloading an app container, however none of the answers provide a complete guide. Here I describe my recent experience with replacing an application container in the command-line using ios-deploy.

Downloading a container

The ios-deploy’s usage mentions these options:

1
2
  -o, --upload <file>          upload file
  -w, --download[=<path>]      download app tree or the specified file/directory

I’ve started with downloading an entire app container because I used a similar command before (I will use org.example.app as the application’s bundle identifier in the examples below):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ ios-deploy -v -1 org.example.app --download --to=$HOME/Desktop/mycontainer
[....] Waiting for iOS device to be connected
<…skip…>
[....] Using abcdeffedcba (J72bAP, iPad (2018), iphoneos, arm64) a.k.a. 'iPad'.
/Documents/
/Documents/data.sqlite
/Documents/default.realm.management/
/Documents/default.realm.management/lock.fifo
AFCFileRefOpen("/Documents/default.realm.management/lock.fifo") failed: 10
/Documents/default.realm.management/access_control.write.mx.fifo
AFCFileRefOpen("/Documents/default.realm.management/access_control.write.mx.fifo") failed: 10
/Documents/default.realm.management/access_control.new_commit.cv
AFCFileRefOpen("/Documents/default.realm.management/access_control.new_commit.cv") failed: 10
/Documents/default.realm.management/access_control.write.mx
/Documents/default.realm.management/access_control.control.mx
/Documents/default.realm.management/access_control.pick_writer.cv
AFCFileRefOpen("/Documents/default.realm.management/access_control.pick_writer.cv") failed: 10
/Documents/default.realm.management/access_control.control.mx.fifo
AFCFileRefOpen("/Documents/default.realm.management/access_control.control.mx.fifo") failed: 10
/Documents/default.realm
/Documents/users.db
<…skip…>

There are errors in the output complaining about a few files, but the downloading completed fine. The application uses Realm and those failed files are actually FIFOs, not regular files.

I’ve downloaded the same container using Xcode to compare them. There is a marvelous trick to compare directories: git diff --no-index mycontainer.xcappdata/AppData ~/Desktop/mycontainer! In this case, I was surprised to see no output — meaning the files in the two directories are the same. Apparently Xcode also fails to download those FIFOs, but doesn’t complain about it.

I also used DiffMerge to compare the two directories and there was one difference: Xcode’s container had an empty /SystemData/ directory whereas the other didn’t. It’s probably a virtual directory mapped to somewhere else and ios-deploy doesn’t have access to it. In my tests it was always empty, so no big deal.

Another way to confirm this is to compare the tree hierarchies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ diff -u <( tree ~/Desktop/org.example.app.xcappdata/AppData ) <( tree ~/Desktop/mycontainer )
--- /dev/fd/11  2021-02-06 20:52:48.000000000 +0200
+++ /dev/fd/12  2021-02-06 20:52:48.000000000 +0200
@@ -1,4 +1,4 @@
-/Users/user/Desktop/org.example.app.xcappdata/AppData
+/Users/user/Desktop/mycontainer
 ├── Documents/
 │   ├── data.sqlite
 │   ├── file
@@ -205,11 +205,10 @@
 │   ├── users.db
 │   ├── users.db-shm
 │   └── users.db-wal
-├── SystemData/
 └── tmp/
     ├── WebKit/
     │   └── MediaCache/
     └── com.apple.dyld/
         └── App-AB89C30AA120EB55C290F6B650815125ABECA9DB65A7E45369C68B9AF3405E06.closure

-97 directories, 115 files
+96 directories, 115 files

Uploading a container

Moving to the --upload option, the usage says it can only upload a file, in contrast to --download’s path parameter. Looking in the source code, I’ve found that it does support uploading directories as well. Let’s try uploading another container:

1
2
3
4
$ ios-deploy -v -1 org.example.app --upload=$HOME/Desktop/org.example.app_moredata.xcappdata/AppData/ --to=/
[....] Waiting for iOS device to be connected
<…skip…>
[....] Using abcdeffedcba (J72bAP, iPad (2018), iphoneos, arm64) a.k.a. 'iPad'.

Seems like nothing happened because the verbose mode didn’t print any uploaded files. Turns out it did upload the data, however ios-deploy simply doesn’t print the paths, so I patched it locally and tried again (I’m using AppCode to build the project, that’s why the path looks like that):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ ~/Library/Caches/JetBrains/AppCode2020.3/DerivedData/ios-deploy-abcdef/Build/Products/Debug/ios-deploy -v -1 org.example.app --upload=$HOME/Desktop/org.example.app_moredata.xcappdata/AppData/ --to=/
[....] Waiting for iOS device to be connected
<…skip…>
[....] Using abcdeffedcba (J72bAP, iPad (2018), iphoneos, arm64, 13.7, 17H35) a.k.a. 'iPad'.
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library to /Library
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library/Saved Application State to /Library/Saved Application State
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library/Saved Application State/org.example.app.savedState to /Library/Saved Application State/org.example.app.savedState
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library/Saved Application State/org.example.app.savedState/KnownSceneSessions to /Library/Saved Application State/org.example.app.savedState/KnownSceneSessions
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library/Saved Application State/org.example.app.savedState/KnownSceneSessions/data.data to /Library/Saved Application State/org.example.app.savedState/KnownSceneSessions/data.data
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library/WebKit to /Library/WebKit
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library/WebKit/WebsiteData to /Library/WebKit/WebsiteData
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library/WebKit/WebsiteData/IndexedDB to /Library/WebKit/WebsiteData/IndexedDB
Copying /Users/user/Desktop/org.example.app_moredata.xcappdata/AppData/Library/WebKit/WebsiteData/IndexedDB/v1 to /Library/WebKit/WebsiteData/IndexedDB/v1
<…skip…>

To verify that, I downloaded the container using Xcode and compared with the original. git diff showed only a bunch of added files in the newly downloaded container; those files were part of the previous app container. So far so good, ios-deploy did upload all the files to the device, but didn’t delete any of the old files. Unfortunately there is no sync option, so the solution is to remove all the existing data on the device before uploading a new container.

Clearing a container

Luckily this is already implemented with the --rmtree parameter:

1
2
3
4
5
6
7
8
$ ios-deploy -v -1 org.example.app --rmtree /
[....] Waiting for iOS device to be connected
<…skip…>
[....] Using abcdeffedcba (J72bAP, iPad (2018), iphoneos, arm64) a.k.a. 'iPad'.
Deleting /Documents/users.db-shm
Deleting /Documents/users.db-wal
Deleting /Documents/default.realm.management/lock.fifo
2021-02-06 13:00:42.000 ios-deploy[1234:5678] [ !! ] Error 0xa: unknown. AFCRemovePath(conn, name)

Boom! It failed to delete this FIFO and stopped. It didn’t take long for me to find the place in the sources this situation is handled and create a small patch to continue removing files after errors (it’s available in the commit 528dfd29ba07ee15c47e03a842d9de2e38fc4959):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ ~/Library/Caches/JetBrains/AppCode2020.3/DerivedData/ios-deploy-abcdef/Build/Products/Debug/ios-deploy -v -1 org.example.app --rmtree /
[....] Waiting for iOS device to be connected
<…skip…>
[....] Using abcdeffedcba (J72bAP, iPad (2018), iphoneos, arm64, 13.7, 17H35) a.k.a. 'iPad'.
Deleting /Documents/data.sqlite
Deleting /Documents/default.realm.management/lock.fifo
2021-02-06 14:14:38.193 ios-deploy[2345:6789] [ !! ] Error 0xa: unknown. AFCRemovePath(conn, name)
Deleting /Documents/default.realm.management/access_control.write.mx.fifo
2021-02-06 14:14:38.208 ios-deploy[2345:6789] [ !! ] Error 0xa: unknown. AFCRemovePath(conn, name)
Deleting /Documents/default.realm.management/access_control.new_commit.cv
2021-02-06 14:14:38.211 ios-deploy[2345:6789] [ !! ] Error 0xa: unknown. AFCRemovePath(conn, name)
Deleting /Documents/default.realm.management/access_control.write.mx
Deleting /Documents/default.realm.management/access_control.control.mx
Deleting /Documents/default.realm.management/access_control.pick_writer.cv
2021-02-06 14:14:38.219 ios-deploy[2345:6789] [ !! ] Error 0xa: unknown. AFCRemovePath(conn, name)
Deleting /Documents/default.realm.management/access_control.control.mx.fifo
2021-02-06 14:14:38.222 ios-deploy[2345:6789] [ !! ] Error 0xa: unknown. AFCRemovePath(conn, name)
Deleting /Documents/default.realm.management
2021-02-06 14:14:38.223 ios-deploy[2345:6789] [ !! ] Error 0x1: unknown. AFCRemovePath(conn, name)
Deleting /Documents/users.db
Deleting /Documents/default.realm
<…skip…>

Good, this worked well enough. Listing the available files shows only these FIFOs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ ios-deploy -v -1 org.example.app --list
[....] Waiting for iOS device to be connected
<…skip…>
[....] Using abcdeffedcba (J72bAP, iPad (2018), iphoneos, arm64) a.k.a. 'iPad'.
//
/Documents/
/Documents/default.realm.management/
/Documents/default.realm.management/lock.fifo
/Documents/default.realm.management/access_control.write.mx.fifo
/Documents/default.realm.management/access_control.new_commit.cv
/Documents/default.realm.management/access_control.pick_writer.cv
/Documents/default.realm.management/access_control.control.mx.fifo
/Documents/default.realm.note
/Library/
/Library/Caches/
/Library/Preferences/

I uploaded an app container again, downloaded it, and this time git diff finally shows nothing!

Conclusion

Replacing an app container on an iOS device in the CLI is possible! It involves removing all the application data on the device (--rmtree) and then uploading your application container (--upload). Moreover, it feels like ios-deploy downloads and uploads containers somewhat faster than Xcode. I’d like to have something similar to my rmtree patch upstream, but I need to clean it up before submitting a PR.

Pro tip: If you brew install noti, you can use noti to get notified in the OSX notification center when a download/upload is done by prepending noti to a command: noti ios-deploy ….

Note: apparently iOS (tested on 13.7) shows the updated app size (after uploading with ios-deploy) in General > iPad Storage > AppName only after launching the app.

I’d love to know if these steps work for you and whether I missed anything, so please feel free to leave a comment.

ps. Just as an idea, it would be awesome to be able to use rsync to sync containers like this: rsync -a mycontainer.xcappdata/AppData/ org.example.app@ipad:/. That’s crazy, right? Apple just doesn’t trust its users with the power of the CLI in most cases forcing them to use GUIs.

Comments