Damien Deville
Member of Technical Staff at OpenAI. Previously at Dropbox, Realmac Software. UCL Computer Science alumnus.
https://ddeville.me
2023-01-02T21:51:40.171407+00:00
Damien Deville
damien@ddeville.me
https://ddeville.me
Copyright © 2007-2023 Damien Deville
/favicon.ico
/favicon-touch.png
/2020/05/home-office
Setting up my home office
2020-05-28T06:00:00Z
2020-05-28T06:00:00Z
<p>Between Covid-19 and moving into a new house, it was finally time to setup a proper home office (I used to work at the kitchen table whenever I worked from home in the past which was not the most productive environment).</p>
<p><img src="/static/media/images/posts/2020/05/home-office.jpeg" alt="My home office" /></p>
<p>I’m pretty happy with the result. The desk and chair are from <a href="https://www.upliftdesk.com">UpliftDesk</a>. There’s plenty of light coming from the window during the day and the little plants definitely warm up the place.</p>
<p>I’m still not a huge fan of WFH and, for one, cannot wait to get back to the office, but for the time being I’m actually enjoying the time I get to spend here.</p>
/2020/03/ddeville-tls
This site is now secure
2020-03-08T21:00:00Z
2020-03-10T20:00:00Z
<p>I’m a little bit late to the party but this site is now served over TLS and is thus marked as "secure" in browsers.</p>
<p>This blog was originally using Jekyll and was hosted on GitHub Pages, which didn’t use to have support for HTTPS when using a custom domain (which I did).
They seem to have added support for it recently but I procrastinated and never actually switched.</p>
<p>As I was looking at updating some of the content on the about page, I decided that it was time to start supporting HTTPS and since checking a single checkbox on GitHub didn’t seem quite fun enough, I actually rebuilt the static content generation and decided to host the site on my own box (well, a shared box on Digital Ocean really).
Sure, I could have kept Jekyll but I didn’t love the fact that it took something like 10 seconds to generate a few dozens HTML pages when my simple Rust program takes a fraction of a second.</p>
<p><a href="https://www.digitalocean.com">Digital Ocean</a> and <a href="https://letsencrypt.org">Let’s Encrypt</a> really make it trivial to serve a simple static website over TLS. Only a couple of minutes to create a new instance, set up Nginx and kick off <code>certbot</code> to get an SSL certificate from Let’s Encrypt.</p>
/2016/01/xnu-make
xnu-make: a simple project to build and install the XNU kernel
2016-01-30T21:00:00Z
2016-01-30T21:00:00Z
<p>As you probably know, the Mac OS X kernel, XNU, is open source and building it from source is fairly straightforward (thanks to <a href="http://shantonu.blogspot.com">yearly instructions</a> by Shantonu Sen).</p>
<p>However, building the kernel requires one to install a couple of dependencies that are not available on a Mac OS X installation by default (such as <code>ctfconvert</code>, <code>ctfdump</code> and <code>ctfmerge</code> that are part of the Dtrace project).</p>
<p>Since these dependencies are installed in the local Xcode Developer directory (or in <code>/usr/local</code>, as long as it’s in your <code>PATH</code>), one needs to install them on each new machine that one wants to build XNU on. Similarly, building <code>libsyscall</code> requires one to modify the local Mac OS X SDK in Xcode which might not always be desirable.</p>
<p>Finally, installing XNU and the respective <code>libsystem_kernel.dylib</code> user-space dynamic library requires a bunch of copying and manual terminal commands to be executed which is not ideal when one wants to quickly deploy a new version of the kernel to a virtual machine for example.</p>
<p>For this reason, I’ve written <code>xnu-make</code> that should make the process of building, installing and deploying XNU to a remote machine a bit more self contained and straightforward.</p>
<p>Currently <code>xnu-make</code> is composed of a <code>Makefile</code> and two simple scripts: <code>install.sh</code> and <code>deploy.sh</code>. It packages XNU and dependencies as submodules and the <code>Makefile</code> takes care of building the kernel, <code>libsyscall</code> and dependencies without touching your Xcode installation or current Mac OS X SDK (it actually makes a copy of the SDK, updates it while building and symlinks it so that Xcode can find it should you need to). Then, the scripts take care of installing the kernel, <code>libsyscall</code>, clearing the various kext and <code>dyld</code> caches and offering you to reboot. <code>install.sh</code> will take care of installing on your local machine (which you should probably never want, unless you’re building from a VM) and <code>deploy.sh</code> will copy the build output and install them on a remote host (such as a VM or a physical remote machine).</p>
<p>You can find <code>xnu-make</code> on <a href="https://github.com/ddeville/xnu-make">GitHub</a>. I’m hoping to improve it over time (I haven’t really tested the user space components installation much for example) but I think it’s a good starting point and it’s, at least, making my life a little bit easier.</p>
/2015/08/using-the-vmware-fusion-gdb-stub-for-kernel-debugging-with-lldb
Using the VMware Fusion GDB stub for kernel debugging with LLDB
2015-08-18T16:00:00Z
2015-08-18T16:00:00Z
<p>In a previous <a href="/2015/08/kernel-debugging-with-lldb-and-vmware-fusion">post</a> I discussed kernel debugging with VMware Fusion and LLDB. In that approach we were connecting LLDB to the kernel via the Kernel Debugging Protocol (KDP). That method works thanks to a stub implemented in the (target) kernel itself. One drawback we discussed was not being able to halt the kernel execution from the debugger and instead requiring a slightly cumbersome keyboard shortcut to generate a NMI on the target VM.</p>
<p>After publishing the article I received some very great feedback including a <a href="https://twitter.com/rgov/status/633000853415202816">tweet</a> from <a href="https://twitter.com/rgov">Ryan Govostes</a>:</p>
<blockquote>
<p>VMware Fusion has a GDB stub built-in, which lldb can talk to if you load a target definitions file.</p>
</blockquote>
<p>To be fair I didn’t have a clear idea of what this exactly meant when I first read it but since it sounded pretty interesting I started doing some research.</p>
<p>I found a great <a href="http://ho.ax/posts/2012/02/vmware-hardware-debugging">post</a> by <a href="https://twitter.com/snare">snare</a> that explains how to use GDB to connect to the remote debug stub in VMware Fusion and debug the target kernel from the host machine.</p>
<p>I will briefly discuss the approach here and then show how we can instead use LLDB to connect to the remote.</p>
<h1><a href="#gdb-stub-in-vmware-fusion" aria-hidden="true" class="anchor" id="gdb-stub-in-vmware-fusion"></a>GDB stub in VMware Fusion</h1>
<p>It turns out that VMware Fusion implements the GDB stub. I don’t think it is a documented feature (all mentions I’ve found about it were from users in the VMware forums) but it can be enabled by setting a preference. Each VM file contains a <code>.vmx</code> config file in the <code>.vmwarevm</code> package that can be edited (make sure that the VM is not running while you edit it).</p>
<p>Open it in a text editor and add the following line:</p>
<pre><code class="language-bash"># If you are debugging a 32-bit machine use `guest32`
debugStub.listen.guest64 = "TRUE"
</code></pre>
<p>With this in place and after rebooting, the VM will listen to connections on the <code>8864</code> port (<code>8832</code> if you’re using <code>guest32</code>) on <code>localhost</code>.</p>
<p>If you wanted to connect from another machine you could use a different option instead and would need to connect to the IP used by the VM:</p>
<pre><code class="language-bash"># If you are debugging a 32-bit machine use `guest32`
debugStub.listen.guest64.remote = "TRUE"
</code></pre>
<p>For our use case we will simply connect to <code>localhost</code> so no need for the <code>remote</code> part.</p>
<h2><a href="#gdb-debugging-stub" aria-hidden="true" class="anchor" id="gdb-debugging-stub"></a>GDB debugging stub</h2>
<p>Before explaining how to connect from GDB let’s quickly discuss what is the GDB stub.</p>
<p>In order to setup a communication between two hosts, we need (among other things) a transmission protocol and an application protocol that both client and server can understand. Then obviously both server and client need to have code that is able to send, receive and interpret packets that come through.</p>
<p>This whole system is implemented as <em>GDB Remote</em> and consists of mainly four parts:</p>
<ul>
<li>TCP as the transmission protocol (KDP on the other hand uses UDP).</li>
<li>The Remote Serial Protocol as the application protocol. It is a <a href="https://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Protocol.html">well-documented</a> protocol and one rarely needs to know the details of it.</li>
<li>The client side of the connection is GDB and, as expected, knows how to connect to the remote and understands the Remote Serial Protocol to send and receive packets.</li>
<li>The server side of the connection is the tricky part since it’s the guest system and rarely has any knowledge of GDB and how to act as a remote out of the box. In order for the debugged program to allow connecting to GDB, one would use either one of these two solutions:</li>
</ul>
<ol>
<li>Using <code>gdbserver</code>, which is a control program for Unix-like systems that allows you to connect your program with a remote GDB. It can be a good option if you have no or little control over the target environment. The <a href="https://sourceware.org/gdb/current/onlinedocs/gdb/Server.html">docs</a> explain <code>gdbserver</code> in much more details.</li>
<li>Implementing the GDB debugging stub on the target. By doing so a program can itself implement the target side of the communication protocol. The official <a href="https://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Stub.html">docs</a> have a lot more info if you’re interested in the particular implementation.</li>
</ol>
<p>In the case of VMware Fusion, a full GDB remote stub is implemented by the virtual machine and can be enabled by setting the option described above, allowing a remote GDB session to connect to the VM.</p>
<h1><a href="#gdb-remote" aria-hidden="true" class="anchor" id="gdb-remote"></a>GDB Remote</h1>
<p>With the <code>debugStub.listen.guest64</code> option set and the VM rebooted, we can start a GDB session on the host machine and attempt to connect to the VM.</p>
<pre><code class="language-bash">(gdb) file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Reading symbols from /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development...Reading symbols from /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development.dSYM/Contents/Resources/DWARF/kernel.development...
done.
(gdb) target remote localhost:8864
Remote debugging using localhost:8864
0xffffff800f9f1e52 in ?? ()
</code></pre>
<p>And at this point we are connected to the remote through the debug stub and we can do anything in the debugger (forget about the missing symbols here, I haven’t looked too much into it). After continuing, one can stop the kernel execution by doing <code>^c</code> in the debugger as usual.</p>
<p>However, I had to install GDB on my host just to try this out (GDB stopped shipping with OS X since Mavericks) and I’d really like to use LLDB wherever I can since it’s what I’m most familiar with nowadays.</p>
<h1><a href="#connecting-lldb-to-a-gdb-remote-stub" aria-hidden="true" class="anchor" id="connecting-lldb-to-a-gdb-remote-stub"></a>Connecting LLDB to a GDB remote stub</h1>
<p>LLDB actually has support for connecting to a GDB remote out of the box with the <code>gdb-remote</code> command. To quote the LLDB <a href="http://lldb.llvm.org/remote.html">docs</a>:</p>
<blockquote>
<p>To enable remote debugging, LLDB employs a client-server architecture. The client part runs on the local system and the remote system runs the server. The client and server communicate using the gdb-remote protocol, usually transported over TCP/IP.</p>
</blockquote>
<p>In particular, the LLDB-specific extensions are discussed in a fantastic <a href="http://www.opensource.apple.com/source/lldb/lldb-310.2.36/docs/lldb-gdb-remote.txt">document</a> in the LLDB repo.</p>
<blockquote>
<p>LLDB has added new GDB server packets to better support multi-threaded and remote debugging. Why? Normally you need to start the correct GDB and the correct GDB server when debugging. If you have mismatch, then things go wrong very quickly. LLDB makes extensive use of the GDB remote protocol and we wanted to make sure that the experience was a bit more dynamic where we can discover information about a remote target with having to know anything up front. [...] Again with GDB, both sides pre-agree on how the registers will look (how many, their register number,name and offsets). We prefer to be able to dynamically determine what kind of architecture, OS and vendor we are debugging, as well as how things are laid out when it comes to the thread register contexts. Below are the details on the new packets we have added above and beyond the standard GDB remote protocol packets.</p>
</blockquote>
<p>So we should be able to just connect to the remote system from LLDB? Let’s find out.</p>
<pre><code class="language-bash">(lldb) file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Current executable set to '/Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development' (x86_64).
(lldb) gdb-remote 8864
Kernel UUID: C75BDFDD-9F27-3694-BB80-73CF991C13D8
Load Address: 0xffffff800f800000
Kernel slid 0xf600000 in memory.
Loaded kernel file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Loading 87 kext modules ....................................................................................... done.
Target arch: x86_64
Connected to live debugserver or arm core. Will associate on-core threads to registers reported by server.
Process 1 stopped
* thread #3: tid = 0x0066, name = '0xffffff801c91d9c0', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffffffffffffff
</code></pre>
<p>Cool! So we were able to connect to the GDB stuff on the VM system. Let’s try and get a backtrace and see how things look.</p>
<pre><code class="language-bash">(lldb) thread backtrace
* thread #3: tid = 0x0066, name = '0xffffff801c91d9c0', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffffffffffffff
</code></pre>
<p>Hmm, that’s not a lot of information. Also, the only frame being at address <code>0xffffffffffffffff</code> doesn’t sound right either.</p>
<h2><a href="#lldb-target-definition" aria-hidden="true" class="anchor" id="lldb-target-definition"></a>LLDB target definition</h2>
<p>Remember that Ryan’s <a href="https://twitter.com/rgov/status/633000853415202816">tweet</a> mentionned a target definitions file? I did some more research and found that other <a href="https://twitter.com/shantonusen/status/477796461313159169">tweet</a> from <a href="https://twitter.com/shantonusen">Shantonu Sen</a> that pointed me to the right approach.</p>
<p>We can download the <a href="http://llvm.org/svn/llvm-project/lldb/trunk/examples/python/x86_64_target_definition.py"><code>x86_64_target_definition.py</code></a> file and use it as our <code>plugin.process.gdb-remote.target-definition-file</code> in LLDB’s settings.</p>
<pre><code class="language-bash"># You can alternatively add this to the `.lldbinit` so that it's loaded whenever lldb starts
(lldb) settings set plugin.process.gdb-remote.target-definition-file /path/to/x86_64_target_definition.py
</code></pre>
<p>The file has a great comment explaining what the target definition does and why it is necessary.</p>
<blockquote>
<p>This file can be used with the following setting: <code>plugin.process.gdb-remote.target-definition-file</code></p>
<p>This setting should be used when you are trying to connect to a remote GDB server that doesn't support any of the register discovery packets that LLDB normally uses.</p>
<p>Why is this necessary? LLDB doesn't require a new build of LLDB that targets each new architecture you will debug with. Instead, all architectures are supported and LLDB relies on extra GDB server packets to discover the target we are connecting to so that is can show the right registers for each target. This allows the GDB server to change and add new registers without requiring a new LLDB build just so we can see new registers.</p>
<p>This file implements the x86_64 registers for the darwin version of GDB and allows you to connect to servers that use this register set.</p>
</blockquote>
<p>Let’s try to use <code>gdb-remote</code> after setting the target definition file.</p>
<pre><code class="language-bash">(lldb) settings set plugin.process.gdb-remote.target-definition-file /path/to/x86_64_target_definition.py
(lldb) file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Current executable set to '/Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development' (x86_64).
(lldb) gdb-remote 8864
Kernel UUID: C75BDFDD-9F27-3694-BB80-73CF991C13D8
Load Address: 0xffffff800f800000
Kernel slid 0xf600000 in memory.
Loaded kernel file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Loading 87 kext modules ....................................................................................... done.
Target arch: x86_64
Connected to live debugserver or arm core. Will associate on-core threads to registers reported by server.
Process 1 stopped
* thread #3: tid = 0x0066, 0xffffff800f9f1e52 kernel.development`machine_idle + 370 at pmCPU.c:174, name = '0xffffff801c91d9c0', queue = 'cpu-0', stop reason = signal SIGTRAP
frame #0: 0xffffff800f9f1e52 kernel.development`machine_idle + 370 at pmCPU.c:174
</code></pre>
<p>It already looks better. Let’s now try to get a backtrace:</p>
<pre><code class="language-bash">(lldb) thread backtrace
* thread #3: tid = 0x0066, 0xffffff800f9f1e52 kernel.development`machine_idle + 370 at pmCPU.c:174, name = '0xffffff801c91d9c0', queue = 'cpu-0', stop reason = signal SIGTRAP
* frame #0: 0xffffff800f9f1e52 kernel.development`machine_idle + 370 at pmCPU.c:174
frame #1: 0xffffff800f8fddb3 kernel.development`processor_idle(thread=0x0000000000000000, processor=0xffffff80100ef658) + 179 at sched_prim.c:4605
frame #2: 0xffffff800f8fe300 kernel.development`idle_thread + 32 at sched_prim.c:4729
frame #3: 0xffffff800f9ea347 kernel.development`call_continuation + 23
</code></pre>
<p>Perfect! We have a complete symbolicated trace and the addresses now look correct.</p>
<h2><a href="#in-practice" aria-hidden="true" class="anchor" id="in-practice"></a>In practice</h2>
<p>To make sure that things are working as expected, let’s set a breakpoint on <code>forkproc</code> (this <a href="http://www.opensource.apple.com/source/xnu/xnu-2782.10.72/bsd/kern/kern_fork.c">function</a> is used to create a new process structure given a parent process and is called from the <code>fork</code> syscall) and make sure that our breakpoint is hit and that we can inspect the frame arguments.</p>
<pre><code class="language-bash">(lldb) breakpoint set --name forkproc
Breakpoint 1: where = kernel.development`forkproc + 20 at cpu_data.h:330, address = 0xffffff8006da6414
(lldb) continue
Process 1 resuming
Process 1 stopped
* thread #6: tid = 0x0f4c, 0xffffff8006da6414 kernel.development`forkproc(parent_proc=0xffffff8013f37b00) + 20 at cpu_data.h:330, name = '0xffffff8013e4f9c0', queue = 'cpu-1', stop reason = breakpoint 1.1
frame #0: 0xffffff8006da6414 kernel.development`forkproc(parent_proc=0xffffff8013f37b00) + 20 at cpu_data.h:330
(lldb) thread backtrace
* thread #6: tid = 0x0f4c, 0xffffff8006da6414 kernel.development`forkproc(parent_proc=0xffffff8013f37b00) + 20 at cpu_data.h:330, name = '0xffffff8013e4f9c0', queue = 'cpu-1', stop reason = breakpoint 1.1
* frame #0: 0xffffff8006da6414 kernel.development`forkproc(parent_proc=0xffffff8013f37b00) + 20 at cpu_data.h:330
frame #1: 0xffffff8006da6d69 kernel.development`cloneproc(parent_task=0xffffff80135c7718, parent_coalition=0xffffff80135c4400, parent_proc=0xffffff8013f37b00, inherit_memory=0, memstat_internal=0) + 41 at kern_fork.c:977
frame #2: 0xffffff8006da6038 kernel.development`fork1(parent_proc=0xffffff8013f37b00, child_threadp=0xffffff8014613ac0, kind=<unavailable>, coalition=<unavailable>) + 328 at kern_fork.c:554
frame #3: 0xffffff8006d9b441 kernel.development`posix_spawn(ap=0xffffff8013f37b00, uap=<unavailable>, retval=0xffffff80135d0040) + 1937 at kern_exec.c:2078
frame #4: 0xffffff8006e2c0c1 kernel.development`unix_syscall64(state=0xffffff80135db540) + 753 at systemcalls.c:368
frame #5: 0xffffff8006a0e656 kernel.development`hndl_unix_scall64 + 22
(lldb) p *(struct proc *)$rdi
(struct proc) $1 = {
p_list = {
le_next = 0xffffff80177e6cf0
le_prev = 0xffffff801610d840
}
p_pid = 275
task = 0xffffff801776cd08
...
</code></pre>
<p>Everything is working as expected, our breakpoint is hit, we can get a complete backtrace and print the first argument (a reference to the parent process structure that we want to fork from, I’ve cut the output, the <code>proc</code> struct is huge).</p>
<h1><a href="#conclusion" aria-hidden="true" class="anchor" id="conclusion"></a>Conclusion</h1>
<p>We showed an alternative approach to do remote debugging with VMware Fusion and LLDB. This method has some advantages over KDP since it lets us interrupt the execution of the program from the debugger at any time and doesn’t require us to use a NMI from the target VM to give control to the debugger on the host.</p>
<p>I’ve read that this method is also faster but I haven’t noticed a major difference in my testing so far. I’m sure heavy use of both methods will provide much more insights in that regard.</p>
<p>Thanks to Ryan Govostes for the idea, snare for the great post, Shantonu Sen for the target definition solution and VMware for making an awesome product.</p>
/2015/08/kernel-debugging-with-lldb-and-vmware-fusion
Kernel debugging with LLDB and VMware Fusion
2015-08-15T14:00:00Z
2015-08-15T14:00:00Z
<p>Being able to use LLDB to debug anything on my Mac has been the basis of my job for the last few years. Regardless of the particular use case – to debug my own program in Xcode or attach to another process – being able to set up breakpoints and inspect a program call stack frames and memory at runtime is just invaluable.</p>
<p>However, there’s one case where one cannot easily attach a debugger at any time: the kernel. Even if you could attach to the kernel on your local machine and halt its execution while you’re having a look around, the debugger wouldn’t be able to do much since it relies on the kernel to do, well anything. An alternative approach would be to use <code>printf</code> or <code>IOLog</code> and call it a day but that’s just not how I roll. If I need to debug something, I want a debugger.</p>
<p>In this article, we will discuss how one can set up a <a href="http://www.vmware.com/products/fusion">VMware Fusion</a> virtual machine (VM) and use LLDB to do remote kernel debugging. Note that this approach could be used to debug any process remotely – which can be very useful for system critical processes such as <code>launchd</code> – but in this article we will mostly focus on the XNU kernel.</p>
<h1><a href="#setup" aria-hidden="true" class="anchor" id="setup"></a>Setup</h1>
<p>Setting up your environment for kernel debugging isn’t too hard but it can seem daunting at first since not much documentation is available.</p>
<p>In the old days, one would have needed to use two physical machines connected with an ethernet or firewire cable. However, nowadays one can easily install OS X on a VMware Fusion virtual machine. Since using multiple OS X systems on a single machine is pretty cool, we will setup the VM and discuss how to connect the debugger from the host.</p>
<p>I will assume that you are using VMware Fusion 7 and OS X 10.10.5. All this should very likely work with previous and later versions of VMware Fusion (or other virtualization products such as <a href="http://www.parallels.com/products/desktop/">Parallels</a> or <a href="https://www.virtualbox.org/">VirtualBox</a>, though I’m not sure since I’ve never used them) and OS X. It’s important to note however that El Capitan introduces System Integrity Protection that makes some breaking changes with respect to the <code>boot-args</code> arguments. I will update this article when El Capitan ships and the updated approach is final.</p>
<h2><a href="#on-the-virtual-machine" aria-hidden="true" class="anchor" id="on-the-virtual-machine"></a>On the virtual machine</h2>
<h3><a href="#install-os-x-1010" aria-hidden="true" class="anchor" id="install-os-x-1010"></a>Install OS X 10.10</h3>
<p>You will need to install the same version of OS X on the host system and the virtual machine. This is not technically required but it will make things easier. I’m currently using 10.10.5 but any version of Yosemite should work in the same way.</p>
<h3><a href="#install-the-kernel-debug-kit" aria-hidden="true" class="anchor" id="install-the-kernel-debug-kit"></a>Install the Kernel Debug Kit</h3>
<p>You should install the Kernel Debug Kit (KDK) from Apple Developer Downloads <a href="https://developer.apple.com/downloads">page</a>. Make sure to pick the KDK that matches the version of OS X that you’re installed. The kit is a simple package install that you should run. It will install some components under <code>/Library/Developer/KDKs</code>.</p>
<p>If you take a look through this directory after installing, you should see something like this:</p>
<pre><code>/Library/Developer/KDKs
KDK_10.10.5_14F27.kdk
ReadMe.html
System
Library
Extensions
AppleUSBAudio.kext
AppleUSBAudio.kext.dSYM
...
Kernels
kernel.debug
kernel.debug.dSYM
...
</code></pre>
<p>You now have a copy of the kernel itself and each kernel extension that Apple ships with the system, including the symbols for each of these executables. There is also a development and debug versions of the kernel that we will use in the VM.</p>
<p>The <code>ReadMe.html</code> file also has a ton of information about the KDK so make sure you at least skim through.</p>
<p>You will need the KDK to be installed on both the host and the VM:</p>
<ul>
<li>It is needed on the VM to install the development version of the kernel</li>
<li>It is needed on the host in order for the debugger to use as the target or simply resolve the symbols</li>
</ul>
<h3><a href="#install-the-development-kernel" aria-hidden="true" class="anchor" id="install-the-development-kernel"></a>Install the development kernel</h3>
<p>On OS X 10.10, the kernel executable is located under <code>/Systems/Library/Kernels</code>.</p>
<p>The KDK comes with a development and debug version of the kernel. To quote the KDK <code>ReadMe.html</code> docs:</p>
<blockquote>
<p>The OS X Yosemite Kernel Debug Kit includes the DEVELOPMENT and DEBUG kernel builds. These both have additional assertions and error checking compared to the RELEASE kernel. The DEVELOPMENT kernel can be used for every-day use and has minimal performance overhead, while the DEBUG kernel has much more error checking.</p>
</blockquote>
<p>It’s not technically required to install and boot into the development kernel to do kernel debugging (since the KDK has the symbols for the shipping kernel too and, as far as I can tell, the release kernel ships with the <code>kdp_*</code> symbols required for remote debugging) but we could definitely use the additional assertions and error checking so we will use it.</p>
<p>In order to install a distinct build of the kernel, there are two steps to take. We will discuss the <code>boot-args</code> changes in a following section so for now, we only need to copy the kernel in the right location.</p>
<p>To do so, you will need to copy the <code>kernel.development</code> executable located under the KDK install directory to <code>/Systems/Library/Kernels</code>:</p>
<pre><code class="language-bash"># `KDK_10.10.5_14F27.kdk` is your current KDK install
damien$ sudo cp /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development /Systems/Library/Kernels
</code></pre>
<p>With this in place, your system can boot into the development kernel, assuming it’s told how to do so.</p>
<h3><a href="#update-the-nvram-boot-args" aria-hidden="true" class="anchor" id="update-the-nvram-boot-args"></a>Update the NVRAM <code>boot-args</code></h3>
<p>The non-volatile random-access memory (NVRAM) is memory that retains its information when power is turned off. In particular, it can contain boot arguments that the system checks when booting. On OS X, its variables can be queried and manipulated by using the <code>nvram</code> command line utility. Note that this is drastically changing with OS X El Capitan but as far as Yosemite is concerned, <code>nvram</code> is the right tool to change boot arguments.</p>
<p>In order to set up our VM for debugging we will need to update the boot arguments by running the following command:</p>
<pre><code class="language-bash">damien$ sudo nvram boot-args="debug=0x141 kext-dev-mode=1 kcsuffix=development pmuflags=1 -v"
</code></pre>
<p>where each boot argument does a very specific thing:</p>
<ul>
<li><code>debug=0x141</code> enables the debug mode and sets a few debug options such as ensuring that the system on the VM waits for a remote debugger connection when booting. The <a href="https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221-BABDGEGF">Kernel Programming Guide</a> has more info about the various options. Here we are using <code>(DB_HALT | DB_ARP | DB_LOG_PI_SCRN)</code>.</li>
<li><code>kext-dev-mode=1</code> allows us to load unsigned kexts. This is not required if you’re only tweaking with the kernel itself but if you’re developing a kernel extension and don’t have a kext-enabled Developer ID certificate to sign it with, this will allow you to load your kext while debugging.</li>
<li><code>kcsuffix=development</code> allows us to boot with the development kernel that we previously copied on the system (change to <code>kcsuffix=debug</code> to boot the debug one).</li>
<li><code>pmuflags=1</code> disables the watchdog timer. The <a href="https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221-SW2">Kernel Programming Guide</a> has much more info about this option.</li>
<li><code>-v</code> enables the kernel verbose mode that will be useful when debugging.</li>
</ul>
<h3><a href="#retrieve-the-vm-network-configuration-info-optional" aria-hidden="true" class="anchor" id="retrieve-the-vm-network-configuration-info-optional"></a>Retrieve the VM network configuration info (Optional)</h3>
<p>In order to connect the debugger to the VM, we will need some info about its network configuration. We will need to change a few things on the host system but on the VM we only need to retrieve the network interface parameters and note them somewhere.</p>
<pre><code class="language-bash">damien$ ifconfig
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
ether 00:0d:29:56:8a:70
inet6 fe80::20c:29ff:fe56:8a70%en0 prefixlen 64 scopeid 0x4
inet 192.168.156.140 netmask 0xffffff00 broadcast 192.168.156.255
nd6 options=1<PERFORMNUD>
media: autoselect (1000baseT <full-duplex>)
status: active
...
</code></pre>
<p>Make sure to write down the MAC address (<code>ether</code>, here <code>00:0d:29:56:8a:70</code>) and IP address (<code>inet</code>, here <code>192.168.156.140</code>) somewhere.</p>
<h3><a href="#invalidate-the-kext-cache" aria-hidden="true" class="anchor" id="invalidate-the-kext-cache"></a>Invalidate the kext cache</h3>
<p>We need to invalidate the kext cache to use the new kernel for debugging. This can easily be done by running the following command:</p>
<pre><code class="language-bash"># / means the root of the current volume, you can specify another volume if needed
damien$ sudo kextcache -invalidate /
</code></pre>
<h3><a href="#reboot" aria-hidden="true" class="anchor" id="reboot"></a>Reboot</h3>
<p>Finally, we need to reboot the VM for all these changes to get picked up. If your system is then stuck on some console output, feel free to shut it down for now. We’ll look into this once we have the host setup for debugging.</p>
<pre><code class="language-bash">damien$ sudo reboot
</code></pre>
<h2><a href="#on-the-host-machine-system" aria-hidden="true" class="anchor" id="on-the-host-machine-system"></a>On the host machine system</h2>
<p>With the VM setup, we now need to set things up for debugging on the host system.</p>
<h3><a href="#install-xcode" aria-hidden="true" class="anchor" id="install-xcode"></a>Install Xcode</h3>
<p>In order to debug you will need Xcode to be installed on your system. You can download it from the Apple Developer <a href="https://developer.apple.com/xcode/downloads">page</a>. Make sure that you launch it at least once to accept the license and install the additional tools.</p>
<h3><a href="#install-the-kernel-debug-kit-1" aria-hidden="true" class="anchor" id="install-the-kernel-debug-kit-1"></a>Install the Kernel Debug Kit</h3>
<p>As we previously said, the Kernel Debug Kit will also have to be installed on the host system for the debugger to use as the target and correctly symbolicate. The installation instructions are identical as the ones for the VM above.</p>
<h3><a href="#update-the-address-translation-table-optional" aria-hidden="true" class="anchor" id="update-the-address-translation-table-optional"></a>Update the address translation table (Optional)</h3>
<p>Optionally, we need to add an entry to the Internet-to-Ethernet address translation table for the VM network interface. We need this for the debugger to pick up the right interface when connecting to the VM IP address. The address translation table is basically a lookup table for IP addresses to MAC addresses.</p>
<p>With the IP address and MAC address for the VM that we wrote down earlier, we can run the following command:</p>
<pre><code class="language-bash"># The `-S` option will add a new entry to the table, making sure to remove any existent entry for this IP first
sudo arp -S 192.168.156.140 00:0d:29:56:8a:70
</code></pre>
<p>Note that we can skip this by using the <code>DB_ARP</code> debug <code>boot-args</code> option as previously explained. This option is defined in the <a href="https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221-BABDGEGF">Kernel Programming Guide</a> as:</p>
<blockquote>
<p>Allow debugger to ARP and route (allows debugging across routers and removes the need for a permanent ARP entry, but is a potential security hole)—not available in all kernels.</p>
</blockquote>
<p>This essentially means that the debugger itself can take care of this for us. If you’re using <code>0x145</code> for the debug <code>boot-args</code> option as shown above, you won’t need to do update the table entry. If you’re worried about security, feel free to use <code>0x105</code> as the option and add the entry to the table manually.</p>
<p>That’s it! Everything is setup, let’s try things in practice.</p>
<h1><a href="#in-practice" aria-hidden="true" class="anchor" id="in-practice"></a>In practice</h1>
<p>In order to test our setup, let’s say that we’re debugging the kernel and want to check what is calling the <code>hfs_vnop_setxattr</code> <a href="http://www.opensource.apple.com/source/xnu/xnu-2782.10.72/bsd/hfs/hfs_xattr.c">function</a> and what the arguments are (this function writes the extended attribute data for a given vnode to the HFS+ store). We will want the debugger to break whenever this function is hit.</p>
<p>First, we’ll have to launch LLDB on the host machine and set the target to the (local) development kernel binary located in the KDK:</p>
<pre><code class="language-bash">damien$ lldb
(lldb) target create /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Current executable set to '/Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development' (x86_64).
</code></pre>
<p>Next, we need to launch the VM and wait for the boot screen to appear. After a few seconds, you should see the following message printed to the screen:</p>
<pre><code class="language-bash">Darwin Bootstrapper Version 2.0.2: Sun Jul 5 21:53:13 PDT 2015; root:libxpc_executables-559.40.1~1/launchd/RELEASE_X86_64
boot-args = debug=0x001 kext-dev-mode=1 kcsuffix=development -v
IOKernelDebugger: registering debugger
ethernet MAC address: 00:0d:29:56:8a:70
ip address: 192.168.156.140
Waiting for remote debugger connection.
</code></pre>
<p>We can confirm that the MAC and IP addresses of the VM are the one that we noted before.</p>
<p>At this point, the host VM system is basically waiting for a debugger connection. Back to the LLDB session on the host machine, we can connect to the VM kernel by running the following command:</p>
<pre><code class="language-bash">(lldb) kdp-remote 192.168.156.140
</code></pre>
<p>The VM will now print <code>Connected to remote debugger</code> and some info will also be printed to the LLDB console output.</p>
<pre><code class="language-bash">(lldb) kdp-remote 192.168.156.140
Version: Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/DEVELOPMENT_X86_64; UUID=C75BDFDD-9F27-3694-BB80-73CF991C13D8; stext=0xffffff8011200000
Kernel UUID: C75BDFDD-9F27-3694-BB80-73CF991C13D8
Load Address: 0xffffff8011200000
Kernel slid 0x11000000 in memory.
Loaded kernel file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Loading 50 kext modules .................................................. done.
Target arch: x86_64
Instantiating threads completely from saved state in memory.
Process 1 stopped
* thread #2: tid = 0x0123, 0xffffff80112bb4c8 kernel.development`kdp_register_send_receive(send=<unavailable>, receive=<unavailable>) + 392 at kdp_udp.c:463, name = '0xffffff801eae94d0', queue = '0x0', stop reason = signal SIGSTOP
frame #0: 0xffffff80112bb4c8 kernel.development`kdp_register_send_receive(send=<unavailable>, receive=<unavailable>) + 392 at kdp_udp.c:463
</code></pre>
<p>At this point, we are stopped in the debugger and the VM kernel is waiting for us to continue. We will first set up our breakpoint:</p>
<pre><code class="language-bash">(lldb) breakpoint set --name hfs_vnop_setxattr
Breakpoint 1: where = kernel.development`hfs_vnop_setxattr + 34 at hfs_xattr.c:711, address = 0xffffff800b548da2
</code></pre>
<p>We can now <code>continue</code> for the kernel to complete booting:</p>
<pre><code class="language-bash">(lldb) continue
Process 1 resuming
</code></pre>
<p>A bunch of text will be output on both the debugger and the system console and eventually the VM system will boot. Shortly after booting (or likely during the boot process), our breakpoint should be hit.</p>
<pre><code class="language-bash">Process 1 stopped
* thread #15: tid = 0x0195, 0xffffff8013b48da2 kernel.development`hfs_vnop_setxattr(ap=0xffffff80e230b290) + 34 at hfs_xattr.c:711, name = '0xffffff802177d1b0', queue = '0x0', stop reason = breakpoint 1.1
frame #0: 0xffffff8013b48da2 kernel.development`hfs_vnop_setxattr(ap=0xffffff80e230b290) + 34 at hfs_xattr.c:711
</code></pre>
<p>Back in the LLDB debugger we can print information about the current stack trace and arguments:</p>
<pre><code class="language-bash">(lldb) thread backtrace
* thread #15: tid = 0x0195, 0xffffff8013b48da2 kernel.development`hfs_vnop_setxattr(ap=0xffffff80e230b290) + 34 at hfs_xattr.c:711, name = '0xffffff802177d1b0', queue = '0x0', stop reason = breakpoint 1.1
* frame #0: 0xffffff8013b48da2 kernel.development`hfs_vnop_setxattr(ap=0xffffff80e230b290) + 34 at hfs_xattr.c:711
frame #1: 0xffffff801394df64 kernel.development`VNOP_SETXATTR(vp=0xffffff8021dcc000, name=<unavailable>, uio=<unavailable>, options=<unavailable>, ctx=<unavailable>) + 84 at kpi_vfs.c:5162
frame #2: 0xffffff801394165d kernel.development`vn_setxattr(vp=0xffffff8021dcc000, name=0xffffff7f94247d3d, uio=0xffffff80e230b780, options=8, context=0xffffff8020ff9d90) + 589 at vfs_xattr.c:216
frame #3: 0xffffff8013d21a3d kernel.development`mac_vnop_setxattr(vp=0xffffff8021dcc000, name=<unavailable>, buf=0xffffff8022813008, len=29) + 189 at mac_vfs_subr.c:173
frame #4: 0xffffff7f94246777 Quarantine`quarantine_set_ea + 125
frame #5: 0xffffff7f94245f37 Quarantine`hook_vnode_notify_open + 430
frame #6: 0xffffff8013d19645 kernel.development`mac_vnode_notify_open(ctx=<unavailable>, vp=0xffffff8021dcc000, acc_flags=35) + 181 at mac_vfs.c:405
frame #7: 0xffffff801393ea6a kernel.development`vn_open_auth [inlined] vn_open_auth_finish(vp=<unavailable>, fmode=35, ctx=<unavailable>) + 14 at vfs_vnops.c:193
frame #8: 0xffffff801393ea5c kernel.development`vn_open_auth(ndp=<unavailable>, fmodep=0xffffff80e230b9fc, vap=<unavailable>) + 2220 at vfs_vnops.c:614
frame #9: 0xffffff801392a095 kernel.development`open1(ctx=<unavailable>, ndp=0xffffff80e230bbf0, uflags=<unavailable>, vap=0xffffff80e230bd88, fp_zalloc=<unavailable>, cra=<unavailable>, retval=<unavailable>) + 549 at vfs_syscalls.c:3344
frame #10: 0xffffff801392ac70 kernel.development`open [inlined] open1at(fp_zalloc=<unavailable>, cra=<unavailable>) + 32 at vfs_syscalls.c:3541
frame #11: 0xffffff801392ac50 kernel.development`open [inlined] openat_internal(ctx=0xffffff8020ff9d90, path=140423473571856, flags=546, mode=<unavailable>, fd=-2, segflg=UIO_USERSPACE, retval=0xffffff8020ff9ca0) + 281 at vfs_syscalls.c:3674
frame #12: 0xffffff801392ab37 kernel.development`open [inlined] open_nocancel + 18 at vfs_syscalls.c:3689
frame #13: 0xffffff801392ab25 kernel.development`open(p=<unavailable>, uap=<unavailable>, retval=0xffffff8020ff9ca0) + 85 at vfs_syscalls.c:3682
frame #14: 0xffffff8013c2c0c1 kernel.development`unix_syscall64(state=0xffffff8021009ea0) + 753 at systemcalls.c:368
frame #15: 0xffffff801380e656 kernel.development`hndl_unix_scall64 + 22
(lldb) p *(struct vnop_setxattr_args *)$rdi
(struct vnop_setxattr_args) $71 = {
a_desc = 0xffffff8013e61430
a_vp = 0xffffff8021dcc000
a_name = 0xffffff7f94247d3d "com.apple.quarantine"
a_uio = 0xffffff80e230b780
a_options = 8
a_context = 0xffffff8020ff9d90
}
</code></pre>
<p>Almost magical!</p>
<p>It’s important to note that once the kernel has launched and the debugger continued, the kernel cannot be halted again from the debugger. In fact, if you try you will get an error message:</p>
<pre><code class="language-bash">(lldb) process interrupt
error: Failed to halt process: KDP cannot interrupt a running kernel
</code></pre>
<p>For this reason, you should make sure that all your breakpoints are registered in the debugger before running <code>continue</code> for the kernel to complete its boot.</p>
<h2><a href="#a-note-about-non-maskable-interrupts-nmi" aria-hidden="true" class="anchor" id="a-note-about-non-maskable-interrupts-nmi"></a>A note about Non-Maskable Interrupts (NMI)</h2>
<p>In the section about <code>boot-args</code>, we discussed using <code>(DB_HALT | DB_ARP | DB_LOG_PI_SCRN)</code> as the flag for the <code>debug</code> option. This flag contains <code>DB_HALT</code> that gets the kernel to wait for a debugger to attach when it boots.</p>
<p>As you can imagine, it is not always necessary to attach a debugger at system boot and often one only wants to attach at panic time (or even at execution time). Luckily, there’s an option to drop into the debugger on an NMI: <code>DB_NMI</code>.</p>
<p>In order to do this, we will have to change the flags for the <code>debug</code> <code>boot-args</code> option to <code>0x144</code> which is <code>(DB_NMI | DB_ARP | DB_LOG_PI_SCRN)</code>.</p>
<p>With this in place, the VM kernel will boot normally without waiting for a debugger to attach. Then, at any time during execution you can generate a NMI which gives an opportunity for the remote debugger to connect.</p>
<p>The way to generate a NMI from a virtual machine is not obvious. Older versions of OS X would use the power button only, as documented in the <a href="https://developer.apple.com/library/mac/qa/qa1264/_index.html">Technical QA1264</a>. However, as discussed in the WWDC 2013 session <a href="https://developer.apple.com/videos/wwdc/2013/">707</a> this was changed to <em>Left Command + Right Command + Power</em> on OS X 10.9. I don’t think it is possible to simulate the power button being pressed on VMware Fusion so, at first, I thought it was actually impossible to generate a NMI from the virtual machine.</p>
<p>However, after reading through the <a href="https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/build/build.html#//apple_ref/doc/uid/TP30000905-CH221-BABDGEGF">Kernel Programming Guide</a> in more details I noticed that the <code>DB_NMI</code> option was documented as:</p>
<blockquote>
<p>Drop into debugger on NMI (Command–Power, Command-Option-Control-Shift-Escape, or interrupt switch).</p>
</blockquote>
<p>And indeed, using <em>Command-Option-Control-Shift-Escape</em> works beautifully in the VM!</p>
<p>If you hit this key combo in your VM at any time (including at panic time), execution will stop. Back to the host system, you can fire up LLDB and connect to the VM.</p>
<pre><code class="language-bash">(lldb) target create /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Current executable set to '/Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development' (x86_64).
(lldb) kdp-remote 192.168.156.140
Version: Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/DEVELOPMENT_X86_64; UUID=C75BDFDD-9F27-3694-BB80-73CF991C13D8; stext=0xffffff8008800000
Kernel UUID: C75BDFDD-9F27-3694-BB80-73CF991C13D8
Load Address: 0xffffff8008800000
Kernel slid 0x8600000 in memory.
Loaded kernel file /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel.development
Loading 93 kext modules ............................................................................................ done.
Target arch: x86_64
Instantiating threads completely from saved state in memory.
Process 1 stopped
* thread #2: tid = 0x00b8, 0xffffff80089f4e77 kernel.development`Debugger(message=<unavailable>) + 759 at model_dep.c:1013, name = '0xffffff8015e169a0', queue = '0x0', stop reason = signal SIGSTOP
frame #0: 0xffffff80089f4e77 kernel.development`Debugger(message=<unavailable>) + 759 at model_dep.c:1013
</code></pre>
<p>And that’s it, as you can see that a <code>SIGSTOP</code> was sent in the VM kernel and back in the host debugger that’s where we’ve attached. From here we can do anything we would do in a debugger: inspect memory, print a backtrace, set a few breakpoints, etc… Once you’re done, you simply have to <code>continue</code> for the kernel to resume executing.</p>
<h2><a href="#a-note-about-kernel-extensions-and-kernel-patches" aria-hidden="true" class="anchor" id="a-note-about-kernel-extensions-and-kernel-patches"></a>A note about kernel extensions and kernel patches</h2>
<p>In this example, we’ve only set up breakpoints and inspected memory of the release kernel. If you plan to run a modified build of the kernel or more importantly if you are building a kernel extension, you will have to copy the <code>.dSYM</code> symbol file built along your kext to the host machine. It can be located anywhere that is indexed by Spotlight. When encountering an unknown symbol (for example a function in your kext), LLDB will look for a <code>.dSYM</code> file that matches this symbol’s Mach-O binary UUID. If the <code>.dSYM</code> file is on your host machine and was indexed by Spotlight then LLDB will symbolicate things nicely.</p>
<h1><a href="#conclusion" aria-hidden="true" class="anchor" id="conclusion"></a>Conclusion</h1>
<p>In this article, we presented how to set up an environment for kernel debugging. While it might sound daunting at first, we saw that it is actually pretty trivial to set up and once everything is in place, debugging the kernel is not more difficult than debugging a simple user-space application.</p>
/2015/02/interprocess-communication-on-ios-with-mach-messages
Interprocess communication on iOS with Mach messages
2015-02-15T22:00:00Z
2015-02-15T22:00:00Z
<p>In my last <a href="/2015/02/interprocess-communication-on-ios-with-berkeley-sockets">article</a> I mentioned that <code>CFMessagePortCreateLocal()</code> was not usable on iOS, quoting the <a href="https://developer.apple.com/library/ios/documentation/CoreFoundation/Reference/CFMessagePortRef/index.html#//apple_ref/c/func/CFMessagePortCreateLocal">documentation</a> as evidence. However, I should have known better since I had recently experimented (and succeeded, more about this soon) with registering a Mach port name with the Bootstrap Server on iOS by using XPC. And, as <a href="/2015/02/interprocess-communication-on-ios-with-berkeley-sockets/#comment-1854411477">pointed out</a> by <a href="https://twitter.com/AriX">Ari Weinstein</a>, it looks like one can indeed use <code>CFMessagePortCreateLocal()</code> on iOS!</p>
<p>There are a few things at play here so let’s first review the API documentation.</p>
<h1><a href="#documentation-deep-dive" aria-hidden="true" class="anchor" id="documentation-deep-dive"></a>Documentation deep dive</h1>
<p>The documentation for <code>CFMessagePort</code> on the iOS Developer Library states the following:</p>
<blockquote>
<p><strong>Special Considerations</strong></p>
</blockquote>
<blockquote>
<p>This method is not available on iOS 7 and later—it will return NULL and log a sandbox violation in syslog. See Concurrency Programming Guide for possible replacement technologies.</p>
</blockquote>
<p>This sounds like a pretty clear statement and it sure feels like one should stay away for this API on iOS even though its header is fully available. So why do I believe this is actually a red-herring?</p>
<p>Well, while this consideration makes sense on iOS 7, things have changed quite a bit since the introduction of iOS 8. Starting with iOS 8, an iOS application looks a lot more like an OS X sandboxed application than it used to. The main reason for this, as previously discussed, are application groups.</p>
<p>There is not a lot of documentation regarding sandboxing rules for iOS but luckily the Mac Developer Library has a bunch. Since the technology behind sandboxing on both platforms is pretty identical we can get a good understanding about how things work on iOS by reading the OS X one.</p>
<p>To step back a little, let’s discuss what being sandboxed means for an application. In short, it means no access to the file system outside of the sandbox, no sharing of ports with other processes, no ingoing or outgoing connection, etc… Obviously, since applications wouldn’t do much if all these rules were enforced, Apple provide some <a href="https://developer.apple.com/library/ios/documentation/Miscellaneous/Reference/EntitlementKeyReference/Chapters/EnablingAppSandbox.html">entitlements</a> that can restore targeted capabilities to the sandboxed target.</p>
<p>Among these, the <code>com.apple.security.application-groups</code> entitlement was introduced with 10.8 (well technically with 10.7.4 but things didn’t work too well back then) and iOS 8. As previously <a href="/2015/02/interprocess-communication-on-ios-with-berkeley-sockets">discussed</a>, application groups give access to a shared container directory on disk and shared user defaults for applications in the same group.</p>
<p>But the extra capabilities provided by application groups are not limited to the file system. In particular, the section about the <a href="https://developer.apple.com/library/mac/documentation/Security/Conceptual/AppSandboxDesignGuide/AppSandboxInDepth/AppSandboxInDepth.html#//apple_ref/doc/uid/TP40011183-CH3-SW21">Application Group Container Directory</a> in the Mac Sandbox Design Guide also has some very interesting information about IPC:</p>
<blockquote>
<p><strong>Note:</strong> Applications that are members of an application group also gain the ability to share Mach and POSIX semaphores and to use certain other IPC mechanisms in conjunction with other group members. See IPC and POSIX Semaphores and Shared Memory for more details.</p>
</blockquote>
<p>Looking at the <a href="https://developer.apple.com/library/mac/documentation/Security/Conceptual/AppSandboxDesignGuide/AppSandboxInDepth/AppSandboxInDepth.html#//apple_ref/doc/uid/TP40011183-CH3-SW24">IPC and Posix Semaphores and Shared Memory</a> section we learn that:</p>
<blockquote>
<p>Normally, sandboxed apps cannot use Mach IPC, POSIX semaphores and shared memory, or UNIX domain sockets (usefully). However, by specifying an entitlement that requests membership in an application group, an app can use these technologies to communicate with other members of that application group.</p>
</blockquote>
<p>We’ve already covered the <a href="/2015/02/interprocess-communication-on-ios-with-berkeley-sockets">UNIX domain sockets</a> but the Mach IPC statement is definitely intriguing and luckily there’s some good news a few paragraphs below:</p>
<blockquote>
<p>Any semaphore or Mach port that you wish to access within a sandboxed app must be named according to a special convention:</p>
</blockquote>
<blockquote>
<ul>
<li>Mach port names must begin with the application group identifier, followed by a period (.), followed by a name of your choosing.</li>
</ul>
</blockquote>
<p>Boom!</p>
<p>So it sure looks like multiple applications in the same application group can send Mach messages through a Mach port communication channel, assuming the port name is carefully chosen to start with the application group identifier.</p>
<p>Remember that these docs are for OS X but I don’t see any reason why it wouldn’t work on iOS, despite what the <code>CFMessagePort</code> docs have to say.</p>
<p>OK, that was quite a lot of theory. Let’s take a look at the code and see if these assumptions actually hold in practice.</p>
<h1><a href="#in-practice" aria-hidden="true" class="anchor" id="in-practice"></a>In practice</h1>
<p>Let’s try to use <code>CFMessagePortCreateLocal()</code> to create a port with a dynamically created name (i.e. not prefixed by the application group identifier) and confirmed that it fails.</p>
<pre><code class="language-objc">CFDataRef message_callback(CFMessagePortRef local, SInt32 msgid, CFDataRef data, void *info) {
return NULL;
}
void create_port(void) {
CFStringRef port_name = (__bridge CFStringRef)[[NSUUID UUID] UUIDString];
CFMessagePortRef port = CFMessagePortCreateLocal(kCFAllocatorDefault, port_name, &message_callback, NULL, NULL);
}
</code></pre>
<p>Note that you will have to run this on a device since the iOS Simulator doesn’t seem to respect any sandboxing rule.</p>
<p>As expected, the call to <code>CFMessagePortCreateLocal()</code> returns <code>NULL</code> and the following error is printed to the console.</p>
<pre><code class="language-objc">*** CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0x450f, name = 'E030B15B-3EC6-4A21-A415-668AD0CB1B52'
See /usr/include/servers/bootstrap_defs.h for the error codes.
(lldb) po port
<nil>
</code></pre>
<p>Looking at the <code>bootstrap_defs.h</code> header (which is really just a symbolic link to <code>/usr/include/bootstrap.h</code>) as suggested by the error message, we can see that the 1100 error code is <code>BOOTSTRAP_NOT_PRIVILEGED</code>.</p>
<p>So that’s confirmed, we definitely cannot register random dynamic port name in our sandboxed iOS application.</p>
<p>But if we change the port name to be prefixed with the application group identifier (such as <code>com.ddeville.app.group.port</code>) a port is returned and no error is printed to the console, as previously inferred from the docs.</p>
<p>We did also confirm that a local port can indeed be created on iOS when the port name is prefixed by the application group identifier. However, what does make that prefixed name so special and why does <code>CFMessagePortCreateLocal()</code> fail to create a port for names that do not match this requirement?</p>
<p>In order to figure this out we first need to discuss Mach bootstrap registration.</p>
<h1><a href="#mach-bootstrap-registration" aria-hidden="true" class="anchor" id="mach-bootstrap-registration"></a>Mach bootstrap registration</h1>
<p>Mach ports are used everywhere on OS X. Due to the fact that Apple <a href="https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/Architecture/Architecture.html#//apple_ref/doc/uid/TP30000905-CH1g-TPXREF102">treats</a> the Mach layer as the very bottom layer in the kernel – with the BSD layer and other subsequent components being implemented on top of it – many operations on OS X eventually translate to Mach ports being involved down the stack.</p>
<h2><a href="#mach-ports-101" aria-hidden="true" class="anchor" id="mach-ports-101"></a>Mach ports 101</h2>
<p>In my opinion, the <a href="https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/KernelProgramming/Mach/Mach.html#//apple_ref/doc/uid/TP30000905-CH209-TPXREF104">Kernel Programming Guide</a> has the clearest documentation about Mach ports.</p>
<blockquote>
<p>A <strong>port</strong> is an endpoint of a unidirectional communication channel between a client who requests a service and a server who provides the service. If a reply is to be provided to such a service request, a second port must be used. This is comparable to a (unidirectional) pipe in UNIX parlance.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>Tasks have permissions to access ports in certain ways (send, receive, send-once); these are called <strong>port rights</strong>. A port can be accessed only via a right. Ports are often used to grant clients access to objects within Mach. Having the right to send to the object’s IPC port denotes the right to manipulate the object in prescribed ways. As such, port right ownership is the fundamental security mechanism within Mach. Having a right to an object is to have a capability to access or manipulate that object.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>Traditionally in Mach, the communication channel denoted by a port was always a queue of <strong>messages</strong>.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>Ports and port rights do not have systemwide names that allow arbitrary ports or rights to be manipulated directly. Ports can be manipulated by a task only if the task has a port right in its port namespace. A port right is specified by a <strong>port name</strong>, an integer index into a 32-bit port namespace. Each task has associated with it a single port namespace.</p>
</blockquote>
<p>and finally</p>
<blockquote>
<p>Tasks acquire port rights when another task explicitly inserts them into its namespace, when they receive rights in messages, by creating objects that return a right to the object, and via Mach calls for certain special ports (mach_thread_self, mach_task_self, and mach_reply_port.)</p>
</blockquote>
<p>(Note: in Mach, a task is basically what you would call a process in the BSD world).</p>
<p>Phew! That’s a lot of information. To summarize, a port is a communication channel that lets multiple tasks send messages to each other, assuming that they have the appropriate right to do so and that the port right, specified by the port name, is available in the task namespace.</p>
<p>In user space, this basically translates to <code>mach_port_t</code>, <code>mach_port_name_t</code> and calls to <code>mach_msg()</code>. Since using Mach ports directly can be pretty hard, Core Foundation (<code>CFMachPort</code> and <code>CFMessagePort</code>) and Cocoa (<code>NSMachPort</code> and <code>NSPortMessage</code>) wrappers were created.</p>
<h2><a href="#bootstrap-registration" aria-hidden="true" class="anchor" id="bootstrap-registration"></a>Bootstrap registration</h2>
<p>When it is created, a Mach task is given a set of special ports including one called the <strong>bootstrap port</strong> whose purpose is to send messages to the <strong>bootstrap server</strong>.</p>
<p>Let’s discuss what is the bootstrap server (for more information about this section – and really about the whole article – make sure to read chapter 9 of Amit Singh’s <a href="http://www.amazon.com/Mac-OS-Internals-Systems-Approach/dp/0321278542">Mac OS X Internals: A Systems Approach</a>).</p>
<h3><a href="#the-bootstrap-server" aria-hidden="true" class="anchor" id="the-bootstrap-server"></a>The Bootstrap Server</h3>
<p>In short, the bootstrap server allows tasks to publish ports that other tasks on the same machine can send messages to. It achieves this by managing a list of name-port bindings. The bootstrap server’s functionality is provided by the bootstrap task, whose program encapsulation nowadays is the <code>launchd</code> program.</p>
<p>The reason why a bootstrap server is necessary is because Mach port namespaces are local to tasks. The bootstrap server allows service names and associated ports to be registered and looked up, across tasks.</p>
<h3><a href="#registration" aria-hidden="true" class="anchor" id="registration"></a>Registration</h3>
<p>In the pre-<code>launchd</code> days (before Mac OS X 10.4 Tiger), one would register a port name by means of the <code>bootstrap_register()</code> function:</p>
<pre><code class="language-objc">kern_return_t
bootstrap_register(mach_port_t bootstrap_port,
name_t service_name,
mach_port_t service_port);
</code></pre>
<p>The server side of the connection would thus register a name for the port it will read from. With this call, the bootstrap server would provide send rights for the bound port to the client.</p>
<p>On the client side, the <code>bootstrap_look_up()</code> function can be used to retrieve send rights for the service port of the service specified by the service name. Obviously, the service must have been previously registered under this name by the server.</p>
<pre><code class="language-objc">kern_return_t
bootstrap_look_up(mach_port_t bootstrap_port,
name_service_t service_name,
mach_port_t *service_port);
</code></pre>
<p>The <a href="http://www.opensource.apple.com/source/mDNSResponder/mDNSResponder-176.2/mDNSMacOSX/helper-main.c"><code>register_service()</code></a> function in the helper application source for <code>mDNSResponder</code> (Rest In Peace) provides a nice demonstration of this technique.</p>
<p>However, the <code>bootstrap_register()</code> function was deprecated with Mac OS X 10.5 Leopard and Apple now recommends to use <code>launchd</code> instead. I won’t go into the details of this decision here (there was a great <a href="http://lists.apple.com/archives/darwin-dev/2009/Nov/threads.html#00060">discussion</a> about it on the darwin-dev mailing list a while ago) but Apple was essentially trying to encourage a launch-on-demand pattern with <code>launchd</code> and this API just didn’t fit with it.</p>
<p>Since using a <code>launchd</code> service or submitting a job via the <code>ServiceManagement</code> is not always appropriate (or possible), there are Cocoa and Core Foundation APIs that take care of registering the name with the bootstrap server by means of an SPI: <code>bootstrap_register2()</code>. These are <code>NSMachBootstrapServer</code> and <code>CFMessagePort</code>.</p>
<p>Since Core Foundation is open source, one can check the implementation of <a href="http://opensource.apple.com/source/CF/CF-855.17/CFMessagePort.c"><code>CFMessagePortCreateLocal()</code></a> and double check that the port name is indeed being registered. It’s also easy to disassemble <code>-[NSMachBootstrapServer registerPort:name:]</code> and realize that it’s essentially wrapping <code>bootstrap_register2()</code>. Remember that <code>NSMachBootstrapServer</code> is only available on OS X so it’s not actually useful to this discussion but it’s still worth keeping in mind.</p>
<p>The circle is now complete.</p>
<h1><a href="#cfmessageportcreatelocal-and-the-magic-name-prefix" aria-hidden="true" class="anchor" id="cfmessageportcreatelocal-and-the-magic-name-prefix"></a><code>CFMessagePortCreateLocal</code> and the magic name prefix</h1>
<p>Now that we understand the process of registering the port name with the bootstrap server we can look into why using the application group identifier as a prefix for the port name magically works.</p>
<p>By calling into <code>CFMessagePortCreateLocal()</code> with a random name that doesn’t meet the sandbox criteria and setting a symbolic breakpoint on the function we can step through the instructions and find out where it fails.</p>
<p>We can quickly individuate the point of failure (remember that <a href="http://opensource.apple.com/source/CF/CF-855.17/CFMessagePort.c"><code>CFMessagePort.c</code></a> is open source):</p>
<pre><code class="language-objc">kern_return_t ret = bootstrap_register2(bs, (char *)utfname, mp, perPID ? BOOTSTRAP_PER_PID_SERVICE : 0);
if (ret != KERN_SUCCESS) {
CFLog(kCFLogLevelDebug, CFSTR("*** CFMessagePort: bootstrap_register(): failed %d (0x%x) '%s', port = 0x%x, name = '%s'\nSee /usr/include/servers/bootstrap_defs.h for the error codes."), ret, ret, bootstrap_strerror(ret), mp, utfname);
CFMachPortInvalidate(native);
CFRelease(native);
CFAllocatorDeallocate(kCFAllocatorSystemDefault, utfname);
CFRelease(memory);
return NULL;
}
</code></pre>
<p>It looks like <code>bootstrap_register2()</code> itself is failing leading <code>CFMessagePortCreateLocal()</code> to print the error and return <code>NULL</code>.</p>
<p><code>bootstrap_register2()</code> probably ends up being implemented somewhere between <code>launchd</code> and the kernel so we can take a look at the <code>launchd</code> source to try and figure out why it would fail. <code>launchd</code> was not open sourced as part of 10.10 but the <a href="http://opensource.apple.com/source/launchd/launchd-842.92.1/">10.9.5 source</a> will do (remember, the source between iOS and OS X will likely be extremely similar if not identical and application groups were introduced on OS X 10.8).</p>
<p>I was not entirely sure where to look for but <code>job_mig_register2</code> in <a href="http://opensource.apple.com/source/launchd/launchd-842.92.1/src/core.c"><code>core.c</code></a> looks like a good candidate.</p>
<p>These few lines in particular are definitely interesting:</p>
<pre><code class="language-objc">bool per_pid_service = flags & BOOTSTRAP_PER_PID_SERVICE;
#if HAVE_SANDBOX
if (unlikely(sandbox_check(ldc->pid, "mach-register", per_pid_service ? SANDBOX_FILTER_LOCAL_NAME : SANDBOX_FILTER_GLOBAL_NAME, servicename) > 0)) {
return BOOTSTRAP_NOT_PRIVILEGED;
}
#endif
</code></pre>
<p>Once again, I had no idea where that <code>sandbox_check()</code> function was implemented so I poked around the included headers to see if anything jump to my eyes. <code>sandbox.h</code> definitely seemed promising but the version in <code>/usr/include/sandbox.h</code> doesn’t declare the function. After some more poking around <code>/usr</code> and disassembling a few libraries I found the implementation in <code>/usr/lib/system/libsystem_sandbox.dylib</code>!</p>
<p><code>sandbox_check()</code> is pretty lame and is basically a proxy into <code>sandbox_check_common()</code>. The latter does the actual work of checking whether the process requesting the <code>mach-register</code> action can use the provided service name. We could spend another article going through the disassembly of the function so let’s just assume that it does a few checks based on the entitlements of the process and returns whether the service name is allowed or not. In our case, it’s obvious that the function checks whether the service name is prefixed with the application group identifier retrieved from the process entitlements and denies it if it doesn’t.</p>
<p>It’s worth noting that it would take Apple a simple change to the sandbox check to disallow that API on iOS. However, given how many APIs rely on the current <code>mach-register</code> behavior (XPC being one of them) I don’t see this happen any time soon.</p>
<h1><a href="#using-mach-messages-on-ios" aria-hidden="true" class="anchor" id="using-mach-messages-on-ios"></a>Using Mach messages on iOS</h1>
<p>Now that we’ve uncovered the conditions under which the API worked, let’s see how one would use it to do IPC on iOS.</p>
<h2><a href="#creating-the-ports" aria-hidden="true" class="anchor" id="creating-the-ports"></a>Creating the ports</h2>
<p>Similarly to Berkeley sockets, we will have a process acting as the server and another one acting as the client. The server will be in charge of registering the port name by creating a local port while the client will simply connect to it by creating a remote port for the same port name. Ordering is important since the remote port creation will fail if the server hasn’t had a chance to register the name yet.</p>
<p>As shown above, the server would create the port by doing:</p>
<pre><code class="language-objc">CFDataRef callback(CFMessagePortRef local, SInt32 msgid, CFDataRef data, void *info)
{
return NULL;
}
void create_port(void)
{
CFStringRef port_name = CFSTR("com.ddeville.myapp.group.port");
CFMessagePortRef port = CFMessagePortCreateLocal(kCFAllocatorDefault, port_name, &callback, NULL, NULL);
CFMessagePortSetDispatchQueue(port, dispatch_get_main_queue());
}
</code></pre>
<p>We schedule the message callbacks to happen on the main queue so that we don’t need to setup a runloop source for the callbacks and manually having to run the runloop while waiting for a reply to a message.</p>
<p>Similarly, on the client side we create the remote message port:</p>
<pre><code class="language-objc">void create_port(void)
{
CFStringRef port_name = CFSTR("com.ddeville.myapp.group.port");
CFMessagePortRef port = CFMessagePortCreateRemote(kCFAllocatorDefault, port_name));
}
</code></pre>
<p>Since the port creation will fail if the server hasn’t registered the local port yet, an appropriate solution would be to retry every few seconds until it succeeds.</p>
<h2><a href="#sending-messages" aria-hidden="true" class="anchor" id="sending-messages"></a>Sending messages</h2>
<p>It is important to note that the connection is somewhat unidirectional. While the client can send messages to the server, the server can only reply to the messages synchronously when they are received (you have probably noted that the client doesn’t have a way to set up a message callback).</p>
<p>In the simple case (that is where no reply is expected), the client can send a message by doing:</p>
<pre><code class="language-objc">void send_message(void)
{
SInt32 messageIdentifier = 1;
CFDataRef messageData = (__bridge CFDataRef)[@"Hello server!" dataUsingEncoding:NSUTF8StringEncoding];
CFMessagePortSendRequest(port, messageIdentifier, messageData, 1000, 0, NULL, NULL);
}
</code></pre>
<p>As you can see, any data can be sent in the message so <a href="https://github.com/ddeville/LLBSDMessaging"><code>LLBSDMessaging</code></a> could be re-implemented on top of Mach messages. The message identifier integer is also a nice API to distinguish between message types.</p>
<p>Upon sending, on the server side, the callback function will be invoked and the message identifier and data passed through. Nice!</p>
<h2><a href="#replying-to-a-message" aria-hidden="true" class="anchor" id="replying-to-a-message"></a>Replying to a message</h2>
<p>As previously noted, the server can optionally reply to the message by returning some data synchronously in the callback function. For it to work client side, we need to slightly change the way we send the message.</p>
<pre><code class="language-objc">// On the client
void send_message(void)
{
SInt32 messageIdentifier = 1;
CFDataRef messageData = (__bridge CFDataRef)[@"Hello server!" dataUsingEncoding:NSUTF8StringEncoding];
CFDataRef response = NULL;
SInt32 status = CFMessagePortSendRequest(port, messageIdentifier, messageData, 1000, 1000, kCFRunLoopDefaultMode, &response);
}
// On the server
CFDataRef callback(CFMessagePortRef local, SInt32 msgid, CFDataRef data, void *info)
{
return (__bridge CFDataRef)[@"Hello client!" dataUsingEncoding:NSUTF8StringEncoding];
}
</code></pre>
<p>Upon return, if no error has happened (you can check the returned status integer) the response reference will point to the data that was sent back by the server.</p>
<p>It’s important to note that <code>CFMessagePortSendRequest()</code> will run the runloop in the specified mode (here <code>kCFRunLoopDefaultMode</code>) thus blocking until the response comes through. We can assume that IPC is pretty fast but the server might still be taking some time to reply. This is where the timeout becomes important: using an appropriate timeout will prevent a thread from being blocked for too long. It’s also probably not a great idea to block the main thread but should you use a background thread remember that it will need to have a serviced runloop ( threads created by a dispatch queue do not have one for example). Another option could be to provide a custom mode on the main thread but be extremely cautious if you need to do this.</p>
<h2><a href="#bidirectional-communication" aria-hidden="true" class="anchor" id="bidirectional-communication"></a>Bidirectional communication</h2>
<p>As mentioned above, while the server can reply to messages sent by the client, it cannot initiate a new message.</p>
<p>A way to workaround this issue would be to create another pair or ports where the current client act as the registrar. Upon the initial connection from the server, the client would register an additional local port with a new name and send the name to the server. Upon receiving, it would create a remote port matching that name.</p>
<p>This solution is slightly more complicated than the bidirectional-by-nature one provided by Berkeley sockets but it should work as expected. Also, most server-client architectures don’t actually require the server to ever initiate a request since it almost always acts as a response provider.</p>
<h1><a href="#conclusion" aria-hidden="true" class="anchor" id="conclusion"></a>Conclusion</h1>
<p>We were able to register a Mach port with the bootstrap server by creating a service name prefixed by the application group identifier and using <code>CFMessagePortCreateLocal()</code> to take care of the registration. With this set up, it was trivial to send messages between applications.</p>
<p>It wouldn’t be too hard to rewrite <a href="https://github.com/ddeville/LLBSDMessaging/blob/master/LLBSDMessaging/LLBSDConnection.m"><code>LLBSDConnection</code></a> on top of Mach ports rather than Berkeley sockets. It would also likely be slightly faster. But I guess this post was already long enough so we’ll leave this for another time :-)</p>
/2015/02/interprocess-communication-on-ios-with-berkeley-sockets
Interprocess communication on iOS with Berkeley sockets
2015-02-08T18:06:04Z
2015-02-08T18:06:04Z
<p>With iOS 8, Apple introduced App Extensions. App Extensions are self-contained apps that developers can ship along with their main application. They can be launched on demand by the system to perform things such as sharing, editing a photo, displaying a widget in the Notification Center, presenting a custom keyboard or even provide content to an app running on the Apple Watch.</p>
<p>In order to allow an extension and its hosting application to share data and resources, application groups were also introduced (It’s worth noting that they were originally introduced on OS X in 10.7.4 to support sharing data between an application and a Login Item application shipped within its bundle).</p>
<p>An application group is an identifier shared between various applications in the same group (it can be enabled through an entitlement). Applications in the same group can share a group container directory that is located outside of their own sandbox but accessible by all applications in the group. Think of it as another sandbox shared between applications. An application and its extensions can share this group container but so can multiple applications released by the same developer (assuming they have all specified the same identifier for the <code>com.apple.security.application-groups</code> entitlement).</p>
<p>While it might not seem much at first, this means that applications belonging to the same group can now:</p>
<ul>
<li><strong>Share a directory on the file system</strong>. Multiple applications can read and write files in this directory (but also making sure that access is coordinated, more about this later) and could for example share a Core Data store or a sqlite database.</li>
<li><strong>Share preferences</strong>. When creating an instance of <code>NSUserDefaults</code> with a suite name equal to the application group identifier, <code>cfprefsd</code> will store the preferences plist in the group container directory and all applications in the group will be able to share its content (A couple of years ago I <a href="/2012/12/shared-preferences-between-sandboxed-applications/">built</a> shared user defaults on the Mac before the application group suite was made available, it was fun).</li>
</ul>
<p>While this is pretty cool and is a big step forward (previously, multiple apps by the same developer could only share keychain entries) there are times where you wish you could just send regular messages between applications without having to store them somewhere on the file system. Also, while <code>NSUserDefaults</code>, Core Data or sqlite support concurrent access neither actually notify other processes when one makes some changes to the shared data. What this means is that you could be inserting an object in one process but the other process wouldn’t be aware of the change until it attempts fetching the data again.</p>
<p>If we had an IPC mechanism to communicate between applications in the same group we could make sure that other processes are notified of changes in real-time without the need for polling.</p>
<p>This article will describe a complete and general solution to this problem on iOS. However, before diving into it I’d like to briefly discuss the IPC situation on OS X. If you can’t wait and just want to see the code, you can find the project on <a href="https://github.com/ddeville/LLBSDMessaging">GitHub</a>.</p>
<h1><a href="#current-state-of-ipc-on-os-x" aria-hidden="true" class="anchor" id="current-state-of-ipc-on-os-x"></a>Current state of IPC on OS X</h1>
<p>Mike Ash did an excellent job discussing the state of IPC on the Mac a few years ago so I will not repeat everything but will rather recommend that you read his <a href="https://www.mikeash.com/pyblog/friday-qa-2009-01-16.html">post</a> (it’s from 2009 but apart from XPC not yet being a thing it’s still a very good overview).</p>
<p>As you probably already know, both OS X and iOS are built on top of Darwin. Darwin itself is built around XNU, a hybrid kernel that combines the Mach 3 microkernel and various elements of BSD, itself a Unix derivative. The dual nature of XNU means that many features of both Mach and Unix are available on Darwin, including several IPC mechanisms.</p>
<h2><a href="#mach-ports" aria-hidden="true" class="anchor" id="mach-ports"></a>Mach ports</h2>
<p>Mach ports are the fundamental IPC mechanism on Mach. Using them directly is hard but there are Core Foundation (<code>CFMachPort</code>) and Foundation (<code>NSMachPort</code>) wrappers available that make things slightly easier. You rarely use a Mach port directly but assuming you have a sending and receiving ports available you should be able to construct a <code>NSPortMessage</code> and send it over.</p>
<p>While creating a local port is just a matter of initializing a new <code>NSMachPort</code> instance, retrieving the remote one requires the Mach bootstrap server. On OS X this usually means having the server side of the connection registering the port for the name with the shared <code>NSMachBootstrapServer</code> such as:</p>
<pre><code class="language-objc">NSMachPort *port = [NSMachPort port];
NSString *name = @"com.ddeville.myapp.myport";
[[NSMachBootstrapServer sharedInstance] registerPort:port name:name];
</code></pre>
<p>On the client side of the connection, one could retrieve the remote port by doing:</p>
<pre><code class="language-objc">NSString *name = @"com.ddeville.myapp.myport";
NSMachPort *port = [[NSMachBootstrapServer sharedInstance] portForName:name];
</code></pre>
<p>Alternatively, one could use <code>CFMessagePortCreateLocal</code> that registers the service with the bootstrap server under the cover and <code>CFMessagePortCreateRemote</code> that uses the bootstrap server to look up a registered service by name.</p>
<p>Note that if the application is sandboxed, the system will not let you register service names that aren’t starting with the application group identifier. When creating a Login Item application, LaunchServices implicitly registers a mach service for the login item whose name is the name as the login item’s bundle identifier. See the <a href="https://developer.apple.com/library/mac/documentation/Security/Conceptual/AppSandboxDesignGuide/AppSandboxInDepth/AppSandboxInDepth.html">App Sandbox Design Guide</a> and the <a href="https://developer.apple.com/library/mac/samplecode/AppSandboxLoginItemXPCDemo">iDecide sample project</a> for more info.</p>
<p>Since working directly with Mach messages can be cumbersome, Distributed Objects, Distributed Notifications and Apple Events were built on top of them and simplify things a lot. I won’t discuss these here so go read Mike Ash’s <a href="https://www.mikeash.com/pyblog/friday-qa-2009-01-16.html">post</a> and the <a href="https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/DistrObjects/Concepts/architecture.html">Distributed Objects Architecture</a> if you want more info.</p>
<p>It’s also worth noting that with OS X 10.7 Lion, Apple shipped a revolutionary API built on top of Mach messages and libdispatch: XPC. As per its man page, XPC is “a structured, asynchronous interprocess communication library”. With 10.8, a new <code>NSXPCConnection</code> class was also released. <code>NSXPCConnection</code> brings back the Distributed Objects idea with an XPC flavor by letting you send messages to a remote proxy object through an XPC connection. Since the introduction of XPC, using Mach ports or one of its derivative is essentially unnecessary. However, it’s important to keep in mind that service name bootstrapping is still required with XPC. In particular, the docs for <code>xpc_connection_create_mach_service</code> clearly state that “the service name must exist in a Mach bootstrap that is accessible to the process and be advertised in a launchd.plist”.</p>
<h2><a href="#posix-file-descriptors" aria-hidden="true" class="anchor" id="posix-file-descriptors"></a>POSIX file descriptors</h2>
<p>By being a BSD derivative Darwin has all the Unix IPC goodies, in particular Berkeley sockets a.k.a. Unix Domain Sockets. Since these will be the core part of the article I’ll save their discussion for a later section.</p>
<h1><a href="#current-state-of-ipc-on-ios" aria-hidden="true" class="anchor" id="current-state-of-ipc-on-ios"></a>Current state of IPC on iOS</h1>
<p>As you’re probably aware, a big part of App Extensions on iOS is built around XPC. An XPC connection is set up between the hosting application and the extension in order to communicate. Most of the methods in <code>NSExtensionRequestHandling</code> and <code>NSExtensionContext</code> end up sending some message to the host application through an XPC connection.</p>
<p>However, XPC is also private on iOS and third-party developers cannot use it directly (yet). Similarly, Distributed Objects, Distributed Notifications and Apple Events are not available on the platform.</p>
<p>Regarding using Mach ports directly, while the Mach port API is indeed available on iOS, <code>NSMachBootstrapServer</code> isn’t and creating a remote port with <code>CFMessagePortCreateLocal</code> will return <code>NULL</code> and log a sandbox violation to the console.</p>
<p>[<strong>Update:</strong> This was proven incorrect. See this follow-up <a href="/2015/02/interprocess-communication-on-ios-with-mach-messages">post</a> for more info.]</p>
<h2><a href="#notifyh" aria-hidden="true" class="anchor" id="notifyh"></a><code><notify.h></code></h2>
<p>OS X has had support for the <code><notify.h></code> Core OS notification mechanism since 10.3. Similarly, iOS has had support for these mechanism since its inception. <code><notify.h></code> allows processes to exchange stateless notification events. As explained in the <code><notify.h></code> header:</p>
<blockquote>
<p>These routines allow processes to exchange stateless notification events. Processes post notifications to a single system-wide notification server, which then distributes notifications to client processes that have registered to receive those notifications, including processes run by other users.</p>
</blockquote>
<blockquote>
<p>Notifications are associated with names in a namespace shared by all clients of the system. Clients may post notifications for names, and may monitor names for posted notifications. Clients may request notification delivery by a number of different methods.</p>
</blockquote>
<p>In a nutshell, processes can post and receive system-wide notifications with the <code>notifyd</code> daemon acting as the server by receiving and broadcasting notifications. Observers can be registered on a dispatch queue, a signal, a mach port or a file descriptor!</p>
<p>A simple example would be registering for lock notifications on iOS:</p>
<pre><code class="language-objc">#import <notify.h>
const char *notification_name = "com.apple.springboard.lockcomplete";
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
int registration_token;
notify_register_dispatch(notification_name, &registration_token, queue, ^ (int token) {
// do something now that the device is locked
});
</code></pre>
<p>Pretty straightforward.</p>
<h2><a href="#cfnotificationcentergetdarwinnotifycenter" aria-hidden="true" class="anchor" id="cfnotificationcentergetdarwinnotifycenter"></a><code>CFNotificationCenterGetDarwinNotifyCenter</code></h2>
<p>Core Foundation offers a notification center based on <code><notify.h></code>: <code>CFNotificationCenterGetDarwinNotifyCenter</code>. By retrieving this notification center, one can post and observe system-level notifications that originate in <code><notify.h></code>.</p>
<pre><code class="language-objc">static void notificationCallback(CFNotificationCenterRef center, void *observer, CFStringRef name, const void *object, CFDictionaryRef userInfo)
{
// do something now that the device is locked
}
CFStringRef notificationName = CFSTR("com.apple.springboard.lockcomplete");
CFNotificationCenterRef notificationCenter = CFNotificationCenterGetDarwinNotifyCenter();
CFNotificationCenterAddObserver(notificationCenter, NULL, notificationCallback, notificationName, NULL, CFNotificationSuspensionBehaviorDeliverImmediately);
</code></pre>
<p>As you can see, this is actually more code since it requires adding a callback function. So, unless you need the behavior provided by <code>CFNotificationSuspensionBehavior</code>, I’d say stick with <code><notify.h></code> if you want to post or receive these notifications.</p>
<p>It’s also important noting that the <code>userInfo</code> parameter of the notification is not supported for the Darwin notify center. What this means is that even if you specify a <code>userInfo</code> dictionary when sending the notification from one process, it will not be available on the receiving side (Notification Center simply ignores it).</p>
<p>This drastically reduces the appeal of this method and essentially restricts its usage to simple stateless system-wide notifications – as it was designed for.</p>
<p>One use case I can think of is to reduce polling. You could for example post a notification whenever one process changes a user default so that other process can fetch the new value without having to constantly check whether the value has changed. However, this method is racy by nature since it requires the receiver to fetch the value after the notification was received and its value could have changed or even be reverted. Ideally the new value would be sent with the notification but as we’ve seen the API prevents this from happening.</p>
<h2><a href="#nsfilepresenter" aria-hidden="true" class="anchor" id="nsfilepresenter"></a><code>NSFilePresenter</code></h2>
<p>I previously discussed <code>NSFilePresenter</code> and <code>NSFileCoordinator</code> extensively in this <a href="http://localhost:4000/2012/12/shared-preferences-between-sandboxed-applications">article</a>. In a few words, <code>NSFileCoordinator</code> class coordinates the reading and writing of files and directories among multiple processes and objects in the same process. Similarly, by adopting the <code>NSFilePresenter</code> protocol, an object can be notified whenever the file is changed on disk by another process.</p>
<p>I have used this method to implement messaging between process on OS X and <a href="http://www.atomicbird.com/blog/sharing-with-app-extensions">people</a> have been attempting the same on iOS.</p>
<p>Unfortunately, with <a href="https://developer.apple.com/library/ios/technotes/tn2408/_index.html">TN2408</a> Apple made it clear that using file coordination to coordinate reads and writes between an app extension and its containing app is a <strong>very bad</strong> idea:</p>
<blockquote>
<p>Important: When you create a shared container for use by an app extension and its containing app in iOS 8, you are obliged to write to that container in a coordinated manner to avoid data corruption. However, you must not use file coordination APIs directly for this. If you use file coordination APIs directly to access a shared container from an extension in iOS 8.0, there are certain circumstances under which the file coordination machinery deadlocks.</p>
</blockquote>
<p>I suspect file coordination is not handling the case where the app extension is killed by the system while the containing app is blocked waiting to read or write. This will likely be fixed soon but for now this means using this API is not an option.</p>
<h2><a href="#unix-domain-sockets" aria-hidden="true" class="anchor" id="unix-domain-sockets"></a>Unix Domain Sockets</h2>
<p>As previously stated, Darwin has its foundations in BSD. BSD being a Unix derivative, Darwin inherited many Unix goodies, one of them being Berkeley sockets, also known as Unix domain sockets or BSD sockets.</p>
<p>You might be familiar with the <a href="http://en.wikipedia.org/wiki/Everything_is_a_file">“Everything in Unix is a file”</a> adage. Well, turns out it’s quite true, in Unix pretty much everything is a file descriptor!</p>
<p>As you probably remember, we previously said that an application group provided a shared container directory to multiple applications in the same group. What if we could use a file to send messages between processes without having to deal with file coordination? Well, turns out a socket is a file and sockets are a pretty good channel for sending and receiving messages. As a side note, Apple wrote a nice section in favor of Unix Domain Sockets vs Mach Messages in <a href="https://developer.apple.com/library/mac/technotes/tn2083/_index.html#//apple_ref/doc/uid/DTS10003794-CH1-SECTION31">TN2083</a>.</p>
<p>In the next section we will discuss how to build a general solution for interprocess communication on iOS based on Berkeley sockets.</p>
<h1><a href="#ipc-on-ios-based-on-berkeley-sockets" aria-hidden="true" class="anchor" id="ipc-on-ios-based-on-berkeley-sockets"></a>IPC on iOS based on Berkeley sockets</h1>
<p>In order to build a general solution for IPC, we need to narrow down some requirements. I have used XPC as a model so the feature set I’m aiming at will be a subset of the one offered by XPC. Without further ado, here’s the list of requirements for our solution:</p>
<ol>
<li>A communication channel</li>
<li>A client-server connection architecture</li>
<li>Non-blocking communication (i.e. Asynchronous I/O)</li>
<li>Message framing for data that is sent through the channel</li>
<li>Support for complex data to be sent through the channel</li>
<li>Secure encoding and decoding of data on each end of the channel</li>
<li>Error and invalidation handling</li>
</ol>
<p>Let’s now discuss each requirement individually and let’s try to figure out a way to solve each of them.</p>
<h2><a href="#a-communication-channel" aria-hidden="true" class="anchor" id="a-communication-channel"></a>A communication channel</h2>
<p>By creating a stream socket of Unix local type at a given path in the application group container directory we can support multiple processes connecting and sending data to each other. By specifying a unique name for the socket we can even support multiple simultaneous connections. It is however important for the client and server to agree on the name so that they can find each other.</p>
<p>Creating such a socket is as simple as:</p>
<pre><code class="language-objc">dispatch_fd_t fd = socket(AF_UNIX, SOCK_STREAM, 0);
</code></pre>
<p><code>AF_UNIX</code> creates a Unix domain socket (vs an Internet socket for example) and <code>SOCK_STREAM</code> makes the socket a stream socket (other options could be datagram or raw).</p>
<p>Note that creating a socket doesn’t even require a path to be specified. The path is usually specified later so we’ll discuss it in the next section. For now, we can just remember that calling <code>socket</code> returns a file descriptor.</p>
<h2><a href="#a-client-server-connection-architecture" aria-hidden="true" class="anchor" id="a-client-server-connection-architecture"></a>A client-server connection architecture</h2>
<p>For our hosts to communicate with each other we need a particular architecture where we can make sure that multiple hosts can connect to each other. In order to achieve this, we use a client-server architecture where one host acts as the server by creating the socket on disk and waiting for connections and the other host(s) act as clients by connecting to the server.</p>
<p>In socket parlance, the server <strong>binds</strong> to the socket and <strong>listens</strong> for connections while the client <strong>connects</strong>. For the client to fully connect, the server has to <strong>accept</strong> the connection. Let’s see how that translates to code.</p>
<p>We can construct the socket path by appending a unique identifier to the path of the application group container directory that all applications in the group have read/write access to. As long as both server and clients agree on the unique identifier, we now have a channel through which they can communicate.</p>
<p>First, on the server side we would start the connection by doing (error handling removed for brevity):</p>
<pre><code class="language-objc">const char *socket_path = ...
dispatch_fd_t fd = socket(AF_UNIX, SOCK_STREAM, 0);
struct sockaddr_un addr;
memset(&addr, 0, sizeof(addr));
addr.sun_family = AF_UNIX;
unlink(socket_path);
strncpy(addr.sun_path, socket_path, sizeof(addr.sun_path) - 1);
bind(fd, (struct sockaddr *)&addr, sizeof(addr));
listen(fd, kLLBSDServerConnectionsBacklog);
</code></pre>
<p>Similarly, on the client side we would connect to the server with:</p>
<pre><code class="language-objc">const char *socket_path = ...
dispatch_fd_t fd = socket(AF_UNIX, SOCK_STREAM, 0);
struct sockaddr_un addr;
memset(&addr, 0, sizeof(addr));
addr.sun_family = AF_UNIX;
strncpy(addr.sun_path, socket_path, sizeof(addr.sun_path) - 1);
connect(fd, (struct sockaddr *)&addr, sizeof(addr));
</code></pre>
<p>A couple notes. On the server, we call <code>unlink()</code> before binding to the socket. This is to make sure that a previous socket that was not correctly cleaned up is correctly removed before creating a new one. Also, note that <code>connect()</code> will fail if the server is not listening for connection. Usually, one should make sure that the server is running before attempting to connect a client (ideally, the server runs all the time, like a daemon process for example). Given that applications are short-lived on iOS, a typical scenario would involve attempting to connect the client multiple times until it succeeds.</p>
<p>Note that whereas the client has already called <code>connect()</code> and returned, the server hasn’t yet accepted the connection. Since this could potentially involve a blocking call we will discuss this in the next section.</p>
<p>Since accepting a new client is really a decision that the server application should make, the connection also has a <code>-server:shouldAcceptNewConnection:;</code> delegate method that the application can implement in order to decide whether a client should connect. It is also the perfect opportunity for the server to keep track of which client have connected.</p>
<p>Since there could potentially be multiple client connected to the server, sending a message from the server comes in two flavors:</p>
<ul>
<li>Broadcasting a message. This will send the message to every connected client.</li>
<li>Sending a message to a particular client, assuming that the application knows its identity.</li>
</ul>
<h2><a href="#non-blocking-communication-asynchronous-io" aria-hidden="true" class="anchor" id="non-blocking-communication-asynchronous-io"></a>Non-blocking communication (Asynchronous I/O)</h2>
<p>In order to accept the client connection, the server needs to call <code>accept()</code>. By default, if no pending connections are present on the queue, <code>accept()</code> blocks the caller until a connection is present. Since one of our requirement is non-blocking communication this is clearly not acceptable. Luckily, we can use the <code>O_NONBLOCK</code> property and a dispatch source to solve the problem. Since we can retrieve the pending connection by issuing a read we can set up a dispatch source for <code>DISPATCH_SOURCE_TYPE_READ</code>:</p>
<pre><code class="language-objc">dispatch_source_t listeningSource = dispatch_source_create(DISPATCH_SOURCE_TYPE_READ, fd, 0, NULL);
dispatch_source_set_event_handler(listeningSource, ^ {
struct sockaddr client_addr;
socklen_t client_addrlen = sizeof(client_addr);
dispatch_fd_t client_fd = accept(self.fd, &client_addr, &client_addrlen);
});
dispatch_resume(listeningSource);
</code></pre>
<p>Now that we have both server and client correctly connected, we need a way to send and receive messages. Like with any other socket, this can be achieved by mean of <code>read()</code> and <code>write()</code>. Remember that these two calls are blocking by default so not acceptable for our solutions. Luckily both also support a non-blocking variant and dispatch IO provides a very nice API to deal with such reads and writes asynchronously.</p>
<p>We can first create a dispatch IO channel with the following:</p>
<pre><code class="language-objc">dispatch_io_t channel = dispatch_io_create(DISPATCH_IO_STREAM, fd, NULL, ^ (int error) {});
dispatch_io_set_low_water(channel, 1);
dispatch_io_set_high_water(channel, SIZE_MAX);
</code></pre>
<p>To read on the channel asynchronously:</p>
<pre><code class="language-objc">dispatch_io_read(channel, 0, SIZE_MAX, NULL, ^ (bool done, dispatch_data_t data, int error) {
if (error) {
return;
}
// read data
if (done) {
// cleanup
}
});
</code></pre>
<p>And to write on the channel asynchronously:</p>
<pre><code class="language-objc">dispatch_data_t message_data = ...
dispatch_io_write(self.channel, 0, message_data, NULL, ^ (bool done, dispatch_data_t data, int write_error) {
// check for errors
});
</code></pre>
<h2><a href="#message-framing-for-data-that-is-sent-through-the-channel" aria-hidden="true" class="anchor" id="message-framing-for-data-that-is-sent-through-the-channel"></a>Message framing for data that is sent through the channel</h2>
<p>As seen in the previous section, we can now read and write data asynchronously on the connection. However, this data is raw bytes and we have no idea where it starts and ends (remember, we are using a stream socket so the data arrives in order but it also arrives broken into pieces).</p>
<p>We need some mechanism in which we could wrap our raw data and that would make it easy to know where it starts and ends. This is known as message framing.</p>
<p>We could implement a new format that has some sort of delimiters for the start and end of a message. It would also probably have some kind of a header that lets one specify the content length of the message as a whole, the encoding to expect, etc… Finally, our framing should support binary data and not simply text since we want to be able to send anything through our connection.</p>
<p>As you can imagine, this is a somewhat solved problem. Many message framing “formats” have been created along the years and HTTP is definitely the most well-known and maybe most widely used. Also, HTTP support binary data (one misconception about HTTP is that it only supports a text body. This is actually not true: the HTTP headers have to be text but the body itself can be any binary data). Since CFNetwork has a very support for HTTP we’ll just use it to frame our messages.</p>
<p>Given a data object to send through the connection, we can create an HTTP message for it.</p>
<pre><code class="language-objc">NSData *contentData = ...
CFHTTPMessageRef response = CFHTTPMessageCreateResponse(kCFAllocatorDefault, 200, NULL, kCFHTTPVersion1_1);
CFHTTPMessageSetHeaderFieldValue(response, (__bridge CFStringRef)@"Content-Length", (__bridge CFStringRef)[NSString stringWithFormat:@"%ld", (unsigned long)[contentData length]]);
CFHTTPMessageSetBody(response, (__bridge CFDataRef)contentData);
NSData *messageData = CFBridgingRelease(CFHTTPMessageCopySerializedMessage(response));
CFRelease(response);
</code></pre>
<p>As you can see, this is actually pretty simple. Given a raw <code>NSData</code> we can create an HTTP message by setting the raw data as the body and specifying a <code>Content-Length</code> header of the size of the data. We can then serialize the HTTP message as an <code>NSData</code> instance to send through the connection.</p>
<p>On the other end, we can do a similar thing, by creating a new HTTP message and appending bytes as they come in through the connection. Once the number of bytes is equal to the one we’re expecting from the <code>Content-Length</code> header, we can get our original data by looking at the message body.</p>
<pre><code class="language-objc">CFHTTPMessageRef framedMessage CFHTTPMessageCreateEmpty(kCFAllocatorDefault, false);
while (... get more bytes ...) {
NSData *data = ...
CFHTTPMessageAppendBytes(framedMessage, data.bytes, (CFIndex)data.length);
if (!CFHTTPMessageIsHeaderComplete(framedMessage)) {
continue;
}
NSInteger contentLength = [CFBridgingRelease(CFHTTPMessageCopyHeaderFieldValue(framedMessage, CFSTR("Content-Length"))) integerValue];
NSInteger bodyLength = (NSInteger)[CFBridgingRelease(CFHTTPMessageCopyBody(framedMessage)) length];
if (contentLength != bodyLength) {
continue;
}
NSData *rawData = CFBridgingRelease(CFHTTPMessageCopyBody(message));
}
</code></pre>
<p>This is indeed very similar. There’s a one little gotcha to be aware of: since we are receiving the HTTP message in chunks, the header might not actually be complete after receiving the first few bytes. Luckily CFNetwork has a handy <code>CFHTTPMessageIsHeaderComplete</code> function that we can use to check for it. Once the header is complete, for each chunk of data that we receive we can check whether the body length matches the expected <code>Content-Length</code> and just retrieve the body data once it’s been fully received.</p>
<h2><a href="#support-for-complex-data-to-be-sent-through-the-channel" aria-hidden="true" class="anchor" id="support-for-complex-data-to-be-sent-through-the-channel"></a>Support for complex data to be sent through the channel</h2>
<p>As discussed previously, one of the major drawbacks of the current solutions is the lack of support for complex data to be sent through the channel. <code><notify.h></code> for example only lets us send a message name.</p>
<p>Since our solution supports sending an <code>NSData</code> instance through the channel we can use an encoding and decoding mechanism to transform <code>data <-> objects</code> on each side of the connection.</p>
<p>Once again, luckily Foundation has very good support for this through <code>NSKeyedArchiver</code> and <code>NSKeyedUnarchiver</code>. By encoding the object graph on one end and decoding on the other end we can give the appearance that real objects are being sent through the connection. As far as the library user is concerned, a message containing objective-c objects is being sent and objective-c objects are similarly being received on the other side. Magical!</p>
<p>One requirement for this to work is that the object has to conform to the <code>NSCoding</code> protocol. Most of the Foundation classes already do and it’s pretty easy to implement for one’s custom classes. Obviously, when using custom classes one must make sure that such class is available on both end of the connection.</p>
<h2><a href="#secure-encoding-and-decoding-of-data-on-each-end-of-the-channel" aria-hidden="true" class="anchor" id="secure-encoding-and-decoding-of-data-on-each-end-of-the-channel"></a>Secure encoding and decoding of data on each end of the channel</h2>
<p>One concern with encoding and decoding random objects between processes is security. Not only do we have to make sure that the class is available on the client and the server for it to be decoded but we also have to make sure that the right class is being decoded and not one pretending to be.</p>
<p>Along <code>NSXPCConnection</code> in OS X 10.8, Apple introduced <code>NSSecureCoding</code>. This protocol extends <code>NSCoding</code> and by adopting <code>NSSecureCoding</code> an object indicates that it handles encoding and decoding instances of itself in a manner that is robust against object substitution attacks.</p>
<p>As per the <code>NSSecureCoding</code> header:</p>
<blockquote>
<p>NSSecureCoding guarantees only that an archive contains the classes it claims. It makes no guarantees about the suitability for consumption by the receiver of the decoded content of the archive. Archived objects which may trigger code evaluation should be validated independently by the consumer of the objects to verify that no malicious code is executed (i.e. by checking key paths, selectors etc. specified in the archive).</p>
</blockquote>
<p>By requiring the messaged objects to conform to <code>NSSecureCoding</code> and by providing a whitelist of classes to the connection, we can make sure that our messages are encoded and decoded in a secure fashion.</p>
<pre><code class="language-objc">id content = ...
NSMutableData *contentData = [NSMutableData data];
NSKeyedArchiver *archiver = [[NSKeyedArchiver alloc] initForWritingWithMutableData:contentData];
archiver.requiresSecureCoding = YES;
@try {
[archiver encodeObject:content forKey:NSKeyedArchiveRootObjectKey];
}
@catch (NSException *exception) {
if ([exception.name isEqualToString:NSInvalidUnarchiveOperationException]) {
return;
}
@throw exception;
}
@finally {
[archiver finishEncoding];
}
</code></pre>
<pre><code class="language-objc">NSSet *allowedClasses = ...
NSData *contentData = ...
NSKeyedUnarchiver *unarchiver = [[NSKeyedUnarchiver alloc] initForReadingWithData:contentData];
unarchiver.requiresSecureCoding = YES;
NSSet *classes = [NSSet setWithObjects:[NSDictionary class], [NSString class], [NSNumber class], [LLBSDProcessInfo class], nil];
classes = [classes setByAddingObjectsFromSet:allowedClasses];
@try {
content = [unarchiver decodeObjectOfClasses:classes forKey:NSKeyedArchiveRootObjectKey];
}
@catch (NSException *exception) {
if ([exception.name isEqualToString:NSInvalidUnarchiveOperationException]) {
return;
}
@throw exception;
}
@finally {
[unarchiver finishDecoding];
}
</code></pre>
<h2><a href="#error-and-invalidation-handling" aria-hidden="true" class="anchor" id="error-and-invalidation-handling"></a>Error and invalidation handling</h2>
<p>In an environment involving multiple processes that could start and die at any time, we cannot assume that errors and connection invalidation will never happen.</p>
<p>Luckily, as stated in <a href="https://developer.apple.com/library/mac/technotes/tn2083/_index.html#//apple_ref/doc/uid/DTS10003794-CH1-SUBSECTION32">TN2083</a>, since Berkeley sockets are a connection-oriented API the server automatically learns about the death of a client and it’s easy for the server to asynchronously notify the client.</p>
<p>We can thus easily implement an invalidation handler where the server or client can be notified whenever the connection becomes invalid.</p>
<p>Similarly, sending a message through the connection can take a completion handler that can pass a possible error that occurred while sending the message.</p>
<h1><a href="#conclusion" aria-hidden="true" class="anchor" id="conclusion"></a>Conclusion</h1>
<p>We were able to build a full-fledged IPC mechanism to communicate between applications and extensions within an application group on iOS. Our solution is based on powerful yet simple technologies such as Berkeley sockets, HTTP message framing and libdispatch.</p>
<p>You can find <strong>LLBSDMessaging</strong> on <a href="https://github.com/ddeville/LLBSDMessaging">GitHub</a>. It is built as a framework so that you can easily integrate it in an existing project.</p>
/2014/07/announcing-spillo
Announcing Spillo
2014-07-15T09:59:21Z
2014-07-15T09:59:21Z
<p>I’ve started using <a href="http://pinboard.in">Pinboard</a> a couple of years ago and have soon become a real fan of the website. It does everything I need: storing and organizing links I want to remember in the future, without all the social madness that services seem to be crazy about these days. I also admire Maciej’s <a href="https://blog.pinboard.in/2014/07/pinboard_turns_five/">business model</a> and Pinboard is one the few services that I’m confident with using for the years to come.</p>
<p>While the website is very minimal it lets you find your stuff in a very efficient manner. The website’s design has practically not changed since its inception and it has actually been for the best. Pinboard does one thing and it does it well!</p>
<p>However, having been in love with the Mac for a long time I was starting to miss having a strong Mac client to create and manage my bookmarks, in an efficient manner that only a native application can provide.</p>
<p>So a couple of months ago, I started to work on a side project that I’m proud to announce today: Spillo for Mac!</p>
<p><img src="/static/media/images/posts/2014/07/spillo.png" alt="Spillo" /></p>
<p>In a nutshell, Spillo is a powerful, beautiful and amazingly fast Pinboard client for OS X. Spillo lets you browse and organize your bookmarks in a stunning modern interface. Spillo also makes creating a bookmark from anywhere on your Mac as convenient as possible.</p>
<p>Spillo is available today for $9.99 on the <a href="https://itunes.apple.com/us/app/spillo/id873245660">Mac App Store</a> or directly from our <a href="http://bananafishsoftware.com/products/spillo/">website</a>.</p>
<p>If you would like to give Spillo a try, a 14-day demo is available on the <a href="http://bananafishsoftware.com/products/spillo/">Bananafish Software</a> website.</p>
<p>I hope you will enjoy using Spillo as much as I do!</p>
/2014/04/dynamic-linking
Dynamic linking on iOS
2014-04-02T10:00:00Z
2014-04-02T10:00:00Z
<p><em>This is a repost of an <a href="http://realmacsoftware.com/blog/dynamic-linking">article</a> I published on the Realmac Software blog</em>.</p>
<p>iOS is often criticized for not supporting dynamic libraries. While you might agree or disagree, it is interesting to think about why it is so and how this rule is eventually enforced. In this post we will see what these libraries are, how they come in practice, how they would work if they were fully supported on iOS and what makes it impossible to ship one in your iOS application.</p>
<h1><a href="#library-linking" aria-hidden="true" class="anchor" id="library-linking"></a>Library linking</h1>
<p>Applications are very rarely built as one big monolithic executable but are rather constructed by assembling various blocks usually called libraries. From a practical perspective, a library can be thought as an agglomerate of executable code along with some public headers and resources conveniently packaged to be linked into and consumed by an application.</p>
<p>While this general definition fits for most library types, there is one area where they diverge: linking. Based on this distinction, there are two families of libraries: static and dynamic libraries. I will quickly summarise the differences between the two, however, if you want to know more about the topic, I recommend reading the <a href="https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/OverviewOfDynamicLibraries.html">Dynamic Library Programming Topics</a> guide on Apple’s website.</p>
<h2><a href="#static-library" aria-hidden="true" class="anchor" id="static-library"></a>Static library</h2>
<p>A static library can be thought of as an archive of object files. When linking such a library into an application, the <em>static linker</em> will collect the object files from the library and package them along with the application object code into one single executable file. This means that the size of the application executable file will grow with the number of libraries added. Also, when the application launches, the application code (which includes the library code) will be loaded all at once into the program’s address space.</p>
<h2><a href="#dynamic-library" aria-hidden="true" class="anchor" id="dynamic-library"></a>Dynamic library</h2>
<p>Dynamic libraries allow an application to load code into its address space when it’s actually needed. This can be either at launch time or at runtime. Dynamic libraries are not part of the application executable file.</p>
<p>When an application is launched, the application’s code is first loaded into the address space of the process. The <em>dynamic loader</em> – usually <em>dyld</em> on Apple’s platforms – takes control of the process and loads the dependent libraries. This involves resolving their location on the file system based on their install name, and then resolving the undefined external symbols needed by the application. The <em>dynamic loader</em> will also load additional libraries at runtime, as they are requested.</p>
<h2><a href="#framework" aria-hidden="true" class="anchor" id="framework"></a>Framework</h2>
<p>In the Apple world, a Framework is a bundle or package containing a dynamic library, header files and resources. Frameworks are a very neat way to group related resources together, provide an executable along with the public headers in one package that is easy to install.</p>
<p>It is important to note that while a framework would need to contain a dynamic library, one could quite easily create a static framework for iOS. I won’t go into details about this here but I highly recommend reading “<a href="http://landonf.bikemonkey.org/code/iphone/iPhone_Framework_Support.20081202.html">iPhone Framework Support - Shipping Libraries</a>” and “<a href="http://landonf.bikemonkey.org/code/ios/Radar_15800975_iOS_Frameworks.20140112.html">iOS Static Libraries Are, Like, Really Bad, And Stuff</a>” by Landon Fuller.</p>
<h1><a href="#dynamic-libraries-on-ios" aria-hidden="true" class="anchor" id="dynamic-libraries-on-ios"></a>Dynamic libraries on iOS</h1>
<p>So, is it true that dynamic libraries cannot be used on iOS? Well, it is actually a bit of a misconception. Every Apple framework that you link into your application contains a dynamic shared library. Can you imagine what the size of the executable would be if you had to statically link <em>UIKit</em> and other frameworks into every single application?
As a matter of fact, dynamic libraries are wildly used on iOS. When your code in <code>applicationDidFinishLaunching:</code> starts executing, <em>dyld</em> has already loaded over 150 libraries!</p>
<p>It would be neat if we could find out which libraries are being loaded while the application is running. Luckily <em>dyld</em> offers some hooks for an application to be notified when an image is added or removed. Let’s create a <code>LLImageLogger</code> class and set some callback functions when the class loads. Let’s go through the implementation of this class. You can find the code on <a href="https://github.com/ddeville/ImageLogger">GitHub</a> if you want to run it; it comes with a couple of iOS and OS X sample applications.</p>
<h2><a href="#logging-loaded-dynamic-libraries" aria-hidden="true" class="anchor" id="logging-loaded-dynamic-libraries"></a>Logging loaded dynamic libraries</h2>
<p><code>mach-o/dyld.h</code> declares two very useful functions: <code>_dyld_register_func_for_add_image</code> and <code>_dyld_register_func_for_remove_image</code>. These functions are documented as follows:</p>
<blockquote>
<p>The following functions allow you to install callbacks which will be called by dyld whenever an image is loaded or unloaded. During a call to <code>_dyld_register_func_for_add_image()</code> the callback func is called for every existing image. Later, it is called as each new image is loaded and bound (but initializers not yet run). The callback registered with <code>_dyld_register_func_for_remove_image()</code> is called after any terminators in an image are run and before the image is un-memory-mapped.</p>
</blockquote>
<p>We can easily add a couple of callbacks when our class loads.</p>
<pre><code class="language-objc">#import <mach-o/dyld.h>
@implementation LLImageLogger
+ (void)load
{
_dyld_register_func_for_add_image(&image_added);
_dyld_register_func_for_remove_image(&image_removed);
}
@end
</code></pre>
<p>We now need to implement both functions. Note that the signature of the callback function is as follows:</p>
<pre><code class="language-c">void callback_function(const struct mach_header *mh, intptr_t vmaddr_slide);
</code></pre>
<p>Ideally, we will want to log some info about the loaded image to the console. The most familiar way will be to try and mimic the format of a crash report. A crash report usually has a list of images that were loaded when the program crashed. The information for each image contains the executable path, the base address, the executable text section size (or the end address) and the image UUID. This information is very useful when it comes to symbolicate a crash report.</p>
<pre><code class="language-objc">0x2fd23000 - 0x2ff0dfff Foundation armv7s <b75ca4f9d9b739ef9b16e482db277849> /System/Library/Frameworks/Foundation.framework/Foundation
0x31c2c000 - 0x3239ffff UIKit armv7s <f725ad0982673286911bff834295ec99> /System/Library/Frameworks/UIKit.framework/UIKit
</code></pre>
<p>Since the callback functions are passed a pointer to the Mach-O header as first argument it should be fairly easy to retrieve this info. Let’s implement both callback functions and simply call into a common function with an extra argument determining whether the image was added or removed.</p>
<pre><code class="language-objc">#import <mach-o/loader.h>
static void image_added(const struct mach_header *mh, intptr_t slide)
{
_print_image(mh, true);
}
static void image_removed(const struct mach_header *mh, intptr_t slide)
{
_print_image(mh, false);
}
</code></pre>
<p>We will now look at implementing the <code>_print_image</code> function. Most of the info about the Mach-O header can be retrieved by means of a function defined in <code>dlfcn.h</code>: <code>dladdr</code>. By passing the pointer to the Mach-O header and a reference to a <code>Dl_info</code> struct we can retrieve some key information about the image. The <code>Dl_info</code> struct contains the following members:</p>
<pre><code class="language-objc">typedef struct dl_info {
const char *dli_fname; /* Pathname of shared object */
void *dli_fbase; /* Base address of shared object */
const char *dli_sname; /* Name of nearest symbol */
void *dli_saddr; /* Address of nearest symbol */
} Dl_info;
</code></pre>
<p>Keeping this in mind, we can now take a look at the implementation of the <code>_print_image</code> function:</p>
<pre><code class="language-objc">#import <dlfcn.h>
static void _print_image(const struct mach_header *mh, bool added)
{
Dl_info image_info;
int result = dladdr(mh, &image_info);
if (result == 0) {
printf("Could not print info for mach_header: %p\n\n", mh);
return;
}
const char *image_name = image_info.dli_fname;
const intptr_t image_base_address = (intptr_t)image_info.dli_fbase;
const uint64_t image_text_size = _image_text_segment_size(mh);
char image_uuid[37];
const uuid_t *image_uuid_bytes = _image_retrieve_uuid(mh);
uuid_unparse(*image_uuid_bytes, image_uuid);
const char *log = added ? "Added" : "Removed";
printf("%s: 0x%02lx (0x%02llx) %s <%s>\n\n", log, image_base_address, image_text_size, image_name, image_uuid);
}
</code></pre>
<p>As you can see, it’s nothing too fancy. We start by retrieving the <code>Dl_info</code> struct for the Mach-O header and then populate the required information. While the base address and image path are directly available in the struct, we will have to manually retrieve the image text segment size and the image UUID from the binary. This is exactly what the <code>_image_retrieve_uuid</code> and <code>_image_text_segment_size</code> do.</p>
<p>For these two functions, we will have to walk the Mach-O file load commands. I recommend reading the <a href="https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html">OS X ABI Mach-O File Format Reference</a> from Apple for a good overview of the Mach-O file format. In a nutshell, a Mach-O file is composed of a header, a series of load commands and data composed of multiple segments. Information about the segments (such as their offset and size) is available in the segment load commands.</p>
<p><img src="/static/media/images/posts/2014/04/mach_o_format.gif" alt="Mach-O file format" /></p>
<p>We will start with creating a visitor function that we’ll be able to reuse across both functions.</p>
<pre><code class="language-objc">static uint32_t _image_header_size(const struct mach_header *mh)
{
bool is_header_64_bit = (mh->magic == MH_MAGIC_64 || mh->magic == MH_CIGAM_64);
return (is_header_64_bit ? sizeof(struct mach_header_64) : sizeof(struct mach_header));
}
static void _image_visit_load_commands(const struct mach_header *mh, void (^visitor)(struct load_command *lc, bool *stop))
{
assert(visitor != NULL);
uintptr_t lc_cursor = (uintptr_t)mh + _image_header_size(mh);
for (uint32_t idx = 0; idx < mh->ncmds; idx++) {
struct load_command *lc = (struct load_command *)lc_cursor;
bool stop = false;
visitor(lc, &stop);
if (stop) {
return;
}
lc_cursor += lc->cmdsize;
}
}
</code></pre>
<p>This functions takes a pointer to a Mach-O header and a visitor block. It then invokes the block for each load command it finds. Note the helper function to retrieve the size of the Mach-O header which we’ll need in order to find the address of the first load command. This is due to the fact that there are two different structs for the Mach-O header: <code>mach_header</code> and <code>mach_header_64</code>, based on whether the architecture is 64-bit or not. Luckily the first field in the header struct is a magic number giving information about the architecture.</p>
<p>With this helper function written we should be able to implement the <code>_image_retrieve_uuid</code> and <code>_image_text_segment_size</code> functions.</p>
<pre><code class="language-objc">static const uuid_t *_image_retrieve_uuid(const struct mach_header *mh)
{
__block const struct uuid_command *uuid_cmd = NULL;
_image_visit_load_commands(mh, ^ (struct load_command *lc, bool *stop) {
if (lc->cmdsize == 0) {
return;
}
if (lc->cmd == LC_UUID) {
uuid_cmd = (const struct uuid_command *)lc;
*stop = true;
}
});
if (uuid_cmd == NULL) {
return NULL;
}
return &uuid_cmd->uuid;
}
</code></pre>
<p>This function is very simple too. It looks for the <code>LC_UUID</code> command and retrieves the <code>uuid_t</code> once it finds it. The <code>_print_image</code> will then take care of transforming the <code>uuid_t</code> into a string by mean of <code>uuid_unparse</code>.</p>
<p>Finally, here is the implementation of the <code>_image_text_segment_size</code> function:</p>
<pre><code class="language-objc">static uint64_t _image_text_segment_size(const struct mach_header *mh)
{
static const char *text_segment_name = "__TEXT";
__block uint64_t text_size = 0;
_image_visit_load_commands(mh, ^ (struct load_command *lc, bool *stop) {
if (lc->cmdsize == 0) {
return;
}
if (lc->cmd == LC_SEGMENT) {
struct segment_command *seg_cmd = (struct segment_command *)lc;
if (strcmp(seg_cmd->segname, text_segment_name) == 0) {
text_size = seg_cmd->vmsize;
*stop = true;
return;
}
}
if (lc->cmd == LC_SEGMENT_64) {
struct segment_command_64 *seg_cmd = (struct segment_command_64 *)lc;
if (strcmp(seg_cmd->segname, text_segment_name) == 0) {
text_size = seg_cmd->vmsize;
*stop = true;
return;
}
}
});
return text_size;
}
</code></pre>
<p>Nothing too fancy here either. The visitor block simply looks for segment commands (<code>LC_SEGMENT</code> on 32-bit and <code>LC_SEGMENT_64</code> on 64-bit) and checks whether the current load segment is the <code>__TEXT</code> segment. If it is, it then retrieves the <code>vmsize</code> and returns it as the text size.</p>
<p>By running the application on the iOS Simulator, the following is logged:</p>
<pre><code class="language-objc">Added: 0x10000b000 (0x2a8000) /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator7.1.sdk/System/Library/Frameworks/Foundation.framework/Foundation <C299A741-488A-3656-A410-A7BE59926B13>
…
Added: 0x110527000 (0x385000) /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator7.1.sdk/System/Library/Frameworks/AudioToolbox.framework/AudioToolbox <57B61C9C-8767-3B3A-BBB5-8768A682383A>
</code></pre>
<p>There are 147 images loaded when launching a very simple iOS application!</p>
<p>We thus showed that dynamic libraries were indeed loaded in our iOS application. So what about the <em>dynamic libraries are not supported on iOS</em> argument? Well, let’s try and build one and see what happens!</p>
<h2><a href="#building-a-dynamic-library-on-ios" aria-hidden="true" class="anchor" id="building-a-dynamic-library-on-ios"></a>Building a dynamic library on iOS</h2>
<p>In the following section, we will attempt to build three products that are common on the Mac but not supported on iOS:</p>
<ul>
<li>A simple dynamic library linked by the application</li>
<li>A framework (a legit one, containing a dynamic shared library)</li>
<li>A plugin (i.e. a bundle containing an executable, not packaged with the application but loaded at runtime)</li>
</ul>
<p><img src="/static/media/images/posts/2014/04/dynamic-ios-app.png" alt="The Dynamic iOS app" /></p>
<p>As usual, you can find the sample project on <a href="https://github.com/ddeville/Dynamic-iOS">GitHub</a>.</p>
<h3><a href="#dynamic-library-on-ios" aria-hidden="true" class="anchor" id="dynamic-library-on-ios"></a>Dynamic library on iOS</h3>
<p>Let’s start by creating a dynamic library target for our iOS application. Oh wait…</p>
<p><img src="/static/media/images/posts/2014/04/create-library-choices-ios.png" alt="Library choices on iOS" /></p>
<p>Xcode (logically) doesn’t provide a configuration template for a dynamic library on iOS. Luckily we can select a library from the OS X section and simply change the deployment target and architectures to build in the Build Settings.</p>
<p><img src="/static/media/images/posts/2014/04/create-library-choices-osx.png" alt="Library choices on OS X" /></p>
<p>If you do this and create your library however, you will get an error when trying to build</p>
<blockquote>
<p>Check dependencies:</p>
<p>Target specifies product type 'com.apple.product-type.library.dynamic', but there's no such product type for the 'iphoneos' platform</p>
</blockquote>
<p>In order to fix this you will have to update the Xcode Specifications for the PackageTypes and ProductTypes for both the iPhone Simulator and the iPhone OS. Instructions on how to do this can be found <a href="http://mysteri0uss.diandian.com/post/2013-06-06/40050450784">here</a>.</p>
<p>With this in mind, we can create a simple class, add a <code>sayHello</code> method which implementation simply creates and shows an alert view and add the class to the library target. We can then add the library to the <em>Link Binary with Library</em> build phase for the application. Finally we have to add a Copy build phase and copy the dynamic library to the application bundle. See the sample project for more info on how to set this up.</p>
<p>We can then import the library header (we marked it as public and made sure that it was in the Header Search Paths), instantiate an object and call the <code>sayHello</code> method. If you run the application in the iOS simulator you should see the alert view popping up. Wait, did we just load and run some code from a dynamic library and iOS? Looks like it… (Don’t attempt to run this on a device just yet, we’ll discuss it later).</p>
<h3><a href="#framework-on-ios" aria-hidden="true" class="anchor" id="framework-on-ios"></a>Framework on iOS</h3>
<p>Now that we‘ve seen that we can load and use a dynamic library, it would be awesome if we could just build a framework around it so that we can pack additional resource easily and don’t have to worry about header search paths.</p>
<p>Similarly to the dynamic library case we can create a framework by using the Mac template and change the deployment target and build architectures. We will also create a plist that will be copied as a resource in the framework bundle. We will load the alert view message from the plist, mostly to demonstrate that resource loading works fine from the framework bundle.</p>
<p>Making sure that the header has been marked as public, we can then simply import the framework in the application:</p>
<pre><code class="language-objc">#import "Framework/Framework.h"
</code></pre>
<p>Finally, we can instantiate the class and call the method. We should see the alert which message has been loaded from a resource in the framework bundle. It works!</p>
<h3><a href="#plugin-on-ios" aria-hidden="true" class="anchor" id="plugin-on-ios"></a>Plugin on iOS</h3>
<p>This last one is the most interesting. Applications on the Mac can be extended in a very neat way by supporting plugins. In simple words, a plugin, just like a framework, is a bundle containing an executable and some resources. The plugin declares its principal class in its <code>Info.plist</code>. Usually the principal class is a subclass of an abstract class defined by the application or conforms to a protocol made available by the application.</p>
<p>In order to load the plugin at runtime, the host application will instantiate an <code>NSBundle</code> instance for the plugin location, preflight the bundle to check that it can indeed be loaded and eventually load it. Loading a bundle will dynamically loads the bundle’s executable code into the running program. Once the bundle has been loaded the host application can retrieve the principal class and instantiate it.</p>
<p>So let’s try and build this for iOS. We can create a new target by using the Bundle template. After changing the deployment target and build architecture we create a new class and set it as the <code>NSPrincipalClass</code> in the <code>Info.plist</code>. Similarly to the framework we will create a plist and load the alert message from it at runtime. We also need to make sure that we specify a custom extension for the bundle – here we will use <code>llplugin</code>.</p>
<p>We can then build the plugin and keep the product somewhere safe. We will then copy it in the Documents directory of the iOS application (you can find the application under <code>~/Library/Application Support/iPhone Simulator/7.1-64/Applications</code>. You might have to change the SDK version based on your platform)</p>
<p>In the host application, we will need to add an imported UTI for the plugin so that we can recognize its file type.</p>
<p><img src="/static/media/images/posts/2014/04/plugin-imported-UTI.png" alt="Plugin imported UTI" /></p>
<p>With the plugin type declared we can now iterate through the contents of the Documents directory and look for plugins, checking conformance to the type for every found item by using <code>UTTypeConformsTo</code>. For each found plugin we can then load the bundle, retrieve the principal class and instantiate it.</p>
<pre><code class="language-objc">- (void)_loadPluginAtLocation:(NSURL *)pluginLocation
{
NSBundle *plugin = [[NSBundle alloc] initWithURL:pluginLocation];
NSError *preflightError = nil;
BOOL preflight = [plugin preflightAndReturnError:&preflightError];
if (!preflight) {
return;
}
BOOL loaded = [plugin load];
if (!loaded) {
return;
}
Class pluginClass = [plugin principalClass];
if (pluginClass == nil) {
return;
}
id pluginInstance = [[pluginClass alloc] init];
if (![pluginInstance respondsToSelector:@selector(sayHello)]) {
return;
}
[pluginInstance sayHello];
}
</code></pre>
<p>If we copy our built plugin in the Documents directory and run the application we should see an alert saying hello. By using the <code>LLImageLogger</code> we can also note that the <code>LLPlugin</code> image is only loaded by the time <code>-[NSBundle load]</code> is called.</p>
<p>We were able to link and load a dynamic library and a framework in an iOS application running in the simulator. We were also able to dynamically load a plugin at runtime and run some of its code, also in the simulator. As you might have guessed there is a catch. So far we’ve only been running our application in the simulator. What happens if we run it on a device?</p>
<h1><a href="#what-about-running-on-the-device" aria-hidden="true" class="anchor" id="what-about-running-on-the-device"></a>What about running on the device?</h1>
<p>If you attempt to build for running on an actual device Xcode will likely complain that the libraries need to be codesigned. Even if you codesign them you will run into problems when launching your application (this might not happened while you’re debugging the application due to the fact that applications behave slightly differently while being debugged on a device on which Developer Mode has been enabled).</p>
<h2><a href="#linked-dynamic-library" aria-hidden="true" class="anchor" id="linked-dynamic-library"></a>Linked dynamic library</h2>
<p>When attempting to run the application containing the dynamic library on the device Xcode will prompt you that the dynamic library has to be code signed. We can simply sign it using the same code sign identity as the application’s. When running it on the device however you will get the following message during <em>dyld</em> bootstrap and the program will be halted.</p>
<blockquote>
<p>dyld: Library not loaded: @executable_path/Library.dylib
Referenced from: /var/mobile/Applications/DC816A37-F0D4-4F72-9EC8-A642A03C0ABC/Dynamic.app/Dynamic
Reason: no suitable image found.
Did find: /var/mobile/Applications/DC816A37-F0D4-4F72-9EC8-A642A03C0ABC/Dynamic.app/Library.dylib: code signature invalid for '/var/mobile/Applications/DC816A37-F0D4-4F72-9EC8-A642A03C0ABC/Dynamic.app/Library.dylib'</p>
</blockquote>
<p>So it seems like <em>dyld</em> is denying the load of the library based on the invalidity of its signature.</p>
<p>This happens during the program load, precisely during the <em>dyld</em> bootstrapping. But what about loading a plugin bundle at runtime?</p>
<h2><a href="#plugin-loaded-at-runtime" aria-hidden="true" class="anchor" id="plugin-loaded-at-runtime"></a>Plugin loaded at runtime</h2>
<p>Let’s see what happens when loading our plugin. When attempting to load the plugin at runtime the application will likely crash with the following backtrace:</p>
<pre><code class="language-objc">Exception Type: EXC_CRASH (SIGKILL - CODESIGNING)
Thread 0 Crashed:
0 dyld 0x2be50c40 ImageLoaderMachO::crashIfInvalidCodeSignature + 72
1 dyld 0x2be5557a ImageLoaderMachOCompressed::instantiateFromFile + 286
2 dyld 0x2be50b44 ImageLoaderMachO::instantiateFromFile + 204
3 dyld 0x2be48036 dyld::loadPhase6 + 390
4 dyld 0x2be4b9b0 dyld::loadPhase5stat + 296
5 dyld 0x2be4b7c6 dyld::loadPhase5 + 390
6 dyld 0x2be4b61c dyld::loadPhase4 + 128
7 dyld 0x2be4b53c dyld::loadPhase3 + 1000
8 dyld 0x2be4afd0 dyld::loadPhase1 + 108
9 dyld 0x2be47e0a dyld::loadPhase0 + 162
10 dyld 0x2be47bb4 dyld::load + 208
11 dyld 0x2be4d1b2 dlopen + 790
12 libdyld.dylib 0x3a09a78a dlopen + 46
13 CoreFoundation 0x2f392754 _CFBundleDlfcnLoadBundle + 120
14 CoreFoundation 0x2f3925a4 _CFBundleLoadExecutableAndReturnError + 328
15 Foundation 0x2fd7f674 -[NSBundle loadAndReturnError:] + 532
16 Foundation 0x2fd8f51e -[NSBundle load] + 18
17 Dynamic 0x000f64be -[LLViewController _loadPluginAtLocation:]
</code></pre>
<p>The application is killed while <em>dyld</em> is attempting to load the bundle. We can only see the userland side of things here but the function at the top frame of the stack should give us a good idea: <code>ImageLoaderMachO::crashIfInvalidCodeSignature</code>. It is worth noting that the plugin we copied into the Documents folder was not code signed. Before attempting to codesign it, let’s quickly analyze what leads to the program being killed when loading the plugin.</p>
<h2><a href="#in-user-space" aria-hidden="true" class="anchor" id="in-user-space"></a>In User Space</h2>
<p>Luckily, <a href="http://opensource.apple.com/source/dyld/dyld-239.3"><em>dyld</em></a> is open-source so we can have a quick look at the implementation of the <code>ImageLoaderMachO::crashIfInvalidCodeSignature</code> function to try and figure out what happened. The file in question is <a href="http://opensource.apple.com/source/dyld/dyld-239.3/src/ImageLoaderMachO.cpp">ImageLoaderMachO.cpp</a> and its implementation is pretty simple:</p>
<pre><code class="language-objc">int ImageLoaderMachO::crashIfInvalidCodeSignature()
{
// Now that segments are mapped in, try reading from first executable segment
// If code signing is enabled the kernel will validate the code signature
// when paging in, and kill the process if invalid
for (unsigned int i = 0; i < fSegmentsCount; ++i) {
if ((segFileOffset(i) == 0) && (segFileSize(i) != 0)) {
// return read value to ensure compiler does not optimize away load
int* p = (int*)segActualLoadAddress(i);
return *p;
}
}
return 0;
}
</code></pre>
<p>This is a pretty naïve (but effective!) way to check the signature and crash if it’s invalid or inexistent: attempt to read from the first executable segment in the executable letting the kernel kill the process should it discover that the code signature cannot be validated when paging in.</p>
<h2><a href="#in-the-kernel" aria-hidden="true" class="anchor" id="in-the-kernel"></a>In the kernel</h2>
<p>The <a href="http://opensource.apple.com/source/xnu/xnu-2422.1.72">kernel</a> itself is also open-source so we can take a look at it and try and figure out where and how the code signature is verified. I should point out here that reading kernel code is slightly out of my confort zone and I found some great help in Charlie Miller’s SyScan’11 talk <a href="http://reverse.put.as/wp-content/uploads/2011/06/syscan11_breaking_ios_code_signing.pdf">Don’t Hassle The Hoff: Breaking iOS Code Signing</a> and book <a href="http://www.amazon.com/iOS-Hackers-Handbook-Charlie-Miller/dp/1118204123">iOS Hacker’s Handbook</a>.</p>
<p>While this is a fascinating subject it could get quite lengthy so I won’t go into details here.</p>
<p>In a nutshell, when codesigned, a Mach-O executable will contain a <code>LC_CODE_SIGNATURE</code> load command referencing a code signature segment in the binary. We can verify it with <em>otool</em> on a signed binary:</p>
<pre><code class="language-bash">> otool -l Plugin.llplugin/Plugin
…
Load command 17
cmd LC_CODE_SIGNATURE
cmdsize 16
dataoff 9968
datasize 9616
…
</code></pre>
<p>In the kernel, the Mach-O file is loaded and parsed in the <code>parse_machfile</code> function and the signature is loaded in <code>load_code_signature</code>, both in <a href="http://opensource.apple.com/source/xnu/xnu-2422.1.72/bsd/kern/mach_loader.c">mach_loader.c</a>. Eventually the signature is checked and its validity is stored inside the <code>csflags</code> member of the kernel’s <a href="http://opensource.apple.com/source/xnu/xnu-2422.1.72/bsd/sys/proc_internal.h"><code>proc</code></a> struct for the process.</p>
<p>Later on, whenever a page fault occurs the <code>vm_fault</code> function in <a href="http://opensource.apple.com/source/xnu/xnu-2422.1.72/osfmk/vm/vm_fault.c">vm_fault.c</a> is called. During the page fault the signature is validated if necessary. The signature will need to be validated if the page is mapped in user space, if the page belongs to a code-signed object, if the page will be writable or simply if it has not previously been validated. Validation happens in the <code>vm_page_validate_cs</code> function inside <a href="http://opensource.apple.com/source/xnu/xnu-2422.1.72/osfmk/vm/vm_fault.c">vm_fault.c</a> (the validation process and how it is enforced continually and not only at load time is interesting, see Charlie Miller’s book for more details).</p>
<p>If for some reason the page cannot be validated, the kernel checks whether the <code>CS_KILL</code> flag has been set and kills the process if necessary. There is a major distinction between iOS and OS X regarding this flag. All iOS processes have this flag set whereas on OS X, although code signing is checked it is not set and thus not enforced.</p>
<p>In our case we can safely assume that the (missing) code signature couldn’t be verified leading to the kernel killing the process.</p>
<h2><a href="#situation-when-code-signing-the-plugin" aria-hidden="true" class="anchor" id="situation-when-code-signing-the-plugin"></a>Situation when code signing the plugin</h2>
<p>It was pretty obvious to see that <em>dyld</em> refused to load the plugin executable due to its lack of code signature. We can change this.</p>
<pre><code class="language-bash">codesign --sign "iPhone Developer" --force --verbose=4 Plugin.llplugin
</code></pre>
<p>When code signing the plugin bundle by using the <code>codesign</code> Terminal utility <em>dyld</em> correctly loads the file at runtime and the plugin code executes correctly, even on the device.</p>
<p><img src="/static/media/images/posts/2014/04/plugin-loaded-alert.png" alt="Plugin Loaded!" /></p>
<p>We thus have no problem dynamically loading an external plugin bundle containing executable code at runtime, even on a device. There is a big caveat to this: we were able to sign the plugin with our Developer certificate given that our device is present in the provisioning profile. For this scenario to happen with a shipped application, the plugin would need to be signed by Apple just like any application submitted to the App Store.</p>
<p>While I don’t see Apple supporting (signed and approved) plugins on iOS anytime soon, I think the dynamic library restrictions could potentially be loosened. By requiring any included executable to be signed with the same certificate as the main application – just like for the Mac App Store – Apple could review them as part of the App Store approval process. Dynamic libraries (in particular when bundled in a framework) would make library distribution so much cleaner and would obsolete the multitude of hacks we have created in order to simply share code.</p>
<h1><a href="#conclusion" aria-hidden="true" class="anchor" id="conclusion"></a>Conclusion</h1>
<ul>
<li>Dynamic libraries signed by Apple can be (and are!) loaded by an iOS application</li>
<li>A simple iOS application loads over 150 dynamic libraries on launch</li>
<li>Xcode doesn’t support creating dynamic libraries, frameworks or plugins on iOS but it is very easy to work around this</li>
<li>If code signing were not a thing we could load dynamic libraries, frameworks and load plugins at runtime on iOS just like we do on the Mac</li>
<li>In practice, the kernel will kill any application that attempts to load a dynamic library that is not signed or which signature cannot be validated</li>
<li>A shipping dynamic library would need to be signed by Apple with the same certificate as the one used for App Store applications</li>
<li>Last but not least, the App Store policies don’t allow dynamic libraries and even if it were technically possible it wouldn’t pass App Store validation</li>
</ul>
<p>You can find the source code for the <a href="https://github.com/ddeville/ImageLogger">Image Logger</a> and <a href="https://github.com/ddeville/Dynamic-iOS">Dynamic iOS</a> projects on GitHub.</p>
<p>If you have any question, you can find me on <a href="https://twitter.com/damiendeville">Twitter</a>.</p>
/2014/02/improved-mobile-experience
Improved mobile experience
2014-02-27T15:49:51Z
2014-02-27T15:49:51Z
<p>Since I redesigned this site three years ago I’ve wanted to update it to be more friendly when browsed on mobile devices. I’ve finally found some time to do this last night and I’m pleased to say that articles are now much more readable from your phone!</p>
<p><img src="/static/media/images/posts/2014/02/ddeville_mobile.png" alt="alt:ddeville.me on the iPhone" /></p>
<p>I haven’t updated the stylesheet for tablets since I believe that a desktop-like format is usually a good fit for this kind of device form factor.</p>
<p>Happy reading! ;)</p>
/2013/11/a-python-script-to-disassemble-a-block-in-lldb
A Python script to disassemble a block in LLDB
2013-11-09T13:10:05Z
2013-11-09T13:10:05Z
<p><em>This is a repost of an <a href="http://realmacsoftware.com/blog/a-python-script-to-disassemble-a-block-in-lldb">article</a> I published on the Realmac Software blog</em>.</p>
<p>In a previous <a href="/2013/02/block-debugging/">article</a> I discussed how to debug Objective-C blocks with LLDB and in particular how to disassemble the block’s <code>invoke</code> function.</p>
<p>While not particularly difficult per se, the process was slightly tedious, involving quite a few steps. It would be nice if we could automate these steps. Luckily, LLDB has a very powerful script bridging interface where the entire LLDB API is available as Python functions.</p>
<p>In this article I’ll go through the creation of a Python script that we will be able to invoke from the LLDB debugger.</p>
<p>Before we dive into it, I suggest that you first have a read through the previous <a href="/2013/02/block-debugging/">article</a>. The test program code is shown below.</p>
<pre><code class="language-objc">// clang -framework Foundation -fobjc-arc -o block block.m
#import <Foundation/Foundation.h>
@interface HelperClass : NSObject
- (void)doThingWithBlock:(BOOL (^)(NSString *arg1, NSInteger arg2))block;
@end
@implementation HelperClass
- (void)doThingWithBlock:(BOOL (^)(NSString *arg1, NSInteger arg2))block
{
block(@"Oh Hai", 22);
}
@end
int main(int argc, char **argv)
{
@autoreleasepool {
HelperClass *object = [HelperClass new];
NSInteger capturedInteger = 2;
[object doThingWithBlock:^ BOOL (NSString *arg1, NSInteger arg2) {
NSInteger someInteger = arg2 + capturedInteger;
printf("%p %li\n", arg1, someInteger);
return YES;
}];
return 0;
}
}
</code></pre>
<p>If you launch the debugger, attach it to the program, set a breakpoint at the start of the <code>doThingWithBlock:</code> method and run you should be able to print the block argument by typing <code>po block</code> in the LLDB prompt once the breakpoint has been hit.</p>
<p><img src="/static/media/images/posts/2013/11/print_block.png" alt="alt:Print block" /></p>
<p>As you can see, the description only gives us the class and address. We could now manually read the memory at this address and figure out the address of the <code>invoke</code> function based on its position in the block structure in order to disassemble it. Similarly, we could have a look at the block descriptor struct and determine whether the block has a signature so that we can feed it to <code>NSMethodSignature</code> to get a print of the arguments and return value.</p>
<p>This is exactly what our script will do!</p>
<p>In order to use the script in an embedded python interpreter using LLDB we can import it by running the command <code>command script import /path/to/block.py</code>.
When doing such, the module initializer is being run. We thus need to implement this initializer and register the command with LLDB</p>
<pre><code class="language-python">def __lldb_init_module (debugger, dict):
debugger.HandleCommand('command script add -f block.block_disass_command block_disass')
print 'The "block_disass" command has been installed'
</code></pre>
<p>Now that our command has been added, we need to implement the <code>block_disass</code> function</p>
<pre><code class="language-python">def block_disass_command(debugger, command, result, dict):
</code></pre>
<p>We use the <code>shlex</code> module to split the command string using shell-like syntax. We then use the <code>optparse</code> module to parse the command arguments and options.</p>
<p>Once we have retrieved all the arguments (and done some validation) we will need to retrieve the current target, process, thread and frame. In a command, the <code>lldb.*</code> convenience variables are not to be used and their values are undefined. We thus need to access these objects as following</p>
<pre><code class="language-python">target = debugger.GetSelectedTarget()
process = target.GetProcess()
thread = process.GetSelectedThread()
frame = thread.GetSelectedFrame()
</code></pre>
<p>With the current frame and the variable name, we can proceed to retrieve the actual variable. We do this by using the <code>FindVariable</code> function on <code>SBFrame</code>. This function returns an <code>SBValue</code>. After checking that the value is valid, we can get the address by invoking the <code>GetValueAsSigned</code> function on the value. If the value is not valid, we check whether the argument was originally an address by using the <code>int</code> function to convert the string into an integer.</p>
<pre><code class="language-python">variable = frame.FindVariable(variable_arg)
if variable.IsValid():
address = variable.GetValueAsSigned()
else:
try:
address = int(variable_arg, 0)
except:
print "The argument is not a valid address or variable in the frame"
return
</code></pre>
<p>As this point, we can assume that we have a valid address for the block and we can proceed with finding its <code>invoke</code> function and disassemble it.</p>
<p>It is worth remembering the block structure (that you can find in the <a href="http://llvm.org/svn/llvm-project/compiler-rt/trunk/BlocksRuntime/Block_private.h">Block_private.h</a> header on the LLVM website).</p>
<pre><code class="language-objc">struct Block_literal_1 {
void *isa;
int flags;
int reserved;
void (*invoke)(void *, ...);
struct Block_descriptor_1 {
unsigned long int reserved;
unsigned long int size;
void (*copy_helper)(void *dst, void *src);
void (*dispose_helper)(void *src);
const char *signature;
} *descriptor;
};
</code></pre>
<p>The <code>disass_block_invoke_function</code> function first finds the address of the <code>invoke</code> function by adding 16 (8 bytes for <code>isa</code> pointer and 4 bytes for each integer) to the original address and then reads the pointer from the memory at this location by using the <code>ReadPointerFromMemory</code> on <code>SBProcess</code>. Assuming no error happened during the reading, we now have the <code>invoke</code> function pointer and we can construct an LLDB command to disassemble instructions starting from this address. Once constructed, we can tell the debugger (<code>SBDebugger</code>) to handle the command for us by invoking <code>HandleCommand</code>. And that’s it, the results of the disassembly should now be printed to the console.</p>
<pre><code class="language-python">def disass_block_invoke_function(debugger, process, block_address, instruction_count):
# The `invoke` function is 16 bytes in the struct
invoke_function_address = block_address + 16
invoke_function_error = lldb.SBError()
invoke_function_pointer = process.ReadPointerFromMemory(invoke_function_address, invoke_function_error)
if not invoke_function_error.Success():
print "Could not retrieve the block invoke function pointer"
return
disass_cmd = "disassemble --start-address " + str(invoke_function_pointer) + " -c " + str(instruction_count)
debugger.HandleCommand(disass_cmd)
</code></pre>
<p><img src="/static/media/images/posts/2013/11/disass_block.png" alt="Disass block" /></p>
<p>Next, we will be retrieving the block signature and print it by mean of <code>NSMethodSignature</code>. Given that a block might not have a signature and, if it does, its position in the <code>descriptor</code> struct will depend on the presence of a <code>copy</code> and <code>dispose</code> function pointers we will want to first inspect the <code>flags</code>.</p>
<p>The <code>flags</code> integer is located 8 bytes in the block struck so we can find its address and read it from memory by using the <code>ReadUnsignedFromMemory</code> function on <code>SBProcess</code>. Since <code>flags</code> is an integer, we specify 4 as the number of bytes to read.</p>
<pre><code class="language-python">flags_address = block_address + 8 # The `flags` integer is 8 bytes in the struct
flags_error = lldb.SBError()
flags = process.ReadUnsignedFromMemory(flags_address, 4, flags_error)
</code></pre>
<p>We can then inspect these flags and find out whether the block has a signature, a <code>copy_helper</code> and a <code>dispose_helper</code> function pointers (see the <a href="http://opensource.apple.com/source/libclosure/libclosure-38/Block_private.h">Block_private.h</a> header for an explanation of these flags).</p>
<pre><code class="language-python">block_has_signature = ((flags & (1 << 30)) != 0)
block_has_copy_dispose_helpers = ((flags & (1 << 25)) != 0)
</code></pre>
<p>Keeping this in mind, we can get the address of the <code>descriptor</code> struct pointer and read it from memory. Eventually, we can get to the <code>signature</code> address by adding 16 (8 bytes for each unsigned long integer) + another 16 (8 bytes for each function pointer) if the block has copy and dispose helper function pointers.</p>
<p>Since the signature is typed as <code>const char *</code> we can read it as a C string. Thankfully there is a <code>ReadCStringFromMemory</code> function on <code>SBProcess</code> that we can use to retrieve it.</p>
<pre><code class="language-python">block_descriptor_address = block_address + 24
block_descriptor_error = lldb.SBError()
block_descriptor = process.ReadPointerFromMemory(block_descriptor_address, block_descriptor_error)
if not block_descriptor_error.Success():
print "Could not read the block descriptor struct"
return
signature_address = block_descriptor + 16
if block_has_copy_dispose_helpers:
signature_address += 16
signature_pointer_error = lldb.SBError()
signature_pointer = process.ReadPointerFromMemory(signature_address, signature_pointer_error)
signature_error = lldb.SBError()
signature = process.ReadCStringFromMemory(signature_pointer, 256, signature_error)
</code></pre>
<p>With the signature in hand, we can now create a command to create an <code>NSMethodSignature</code> and print it to the console. Similarly to the <code>disassembly</code>, we’ll ask the debugger to handle the command for us.</p>
<pre><code class="language-python">method_signature_cmd = 'po [NSMethodSignature signatureWithObjCTypes:"' + escaped_signature + '"]'
debugger.HandleCommand(method_signature_cmd)
</code></pre>
<p>And that’s it!</p>
<p><img src="/static/media/images/posts/2013/11/print_block_signature.png" alt="Print block signature" /></p>
<p>The easier way to use the script is to add the following line to <code>~/.lldbinit</code></p>
<pre><code class="language-python">command script import /path/to/the/script/block.py
</code></pre>
<p>With this in place, you should be able to simply call <code>block_disass</code> in the debugger.</p>
<p>The script is on <a href="https://github.com/ddeville/block-lldb-script">GitHub</a>. Have a look at it at the README in particular for a list of the supported arguments.</p>
<p>Being able to write Python scripts with the <code>lldb</code> module is extremely powerful and opens the door to dozens of applications. You can read more about the Python <a href="http://lldb.llvm.org/python-reference.html">reference</a> and <a href="http://lldb.llvm.org/python_reference/">API</a> on the LLVM website.</p>
/2013/10/nsarraycontroller-and-kvo-dependent-keys
NSArrayController and KVO dependent keys
2013-10-27T09:00:00Z
2013-10-27T09:00:00Z
<p>In the <a href="/2013/10/a-note-about-kvo-dependent-keys/">previous post</a> I went through dependent keys with KVO. It is a very powerful concept that will help your code look much cleaner and declarative (even more if used in addition to Cocoa bindings).</p>
<p>In this post I’ll discuss a scenario that any developer that ever worked with Cocoa on the Mac will be very accustomed to: <code>NSArrayController</code> and KVO dependent keys.</p>
<p><code>NSArrayController</code>, a subclass of <code>NSObjectController</code> (itself a subclass of <code>NSController</code>) is a controller class that manages a collection of objects. From the docs:</p>
<blockquote>
<p><code>NSArrayController</code> is a bindings compatible class that manages a collection of objects. Typically the collection is an array, however, if the controller manages a relationship of a managed object the collection may be a set. <code>NSArrayController</code> provides selection management and sorting capabilities.</p>
</blockquote>
<p><code>NSArrayController</code> is extremely powerful, particularly when used with Core Data and bindings.</p>
<p>There are times when you want to use an <code>NSArrayController</code> to manage objects inside a window or view controller and need to expose these objects (so that a client can observe or bind a view to them for example) without exposing the array controller itself. Basically you want to expose an <code>NSArray</code> property represented internally by an array controller’s arranged objects.</p>
<p>Let’s take an example. We have a <code>Shop</code> object with a very simple interface:</p>
<pre><code class="language-objc">@interface Shop : NSObject
@property (readonly, strong, nonatomic) NSArray *products;
- (void)addProduct:(NSString *)product;
- (void)removeProduct:(NSString *)product;
@end
</code></pre>
<p>Internally, we will use an <code>NSArrayController</code> to manage these products.</p>
<pre><code class="language-objc">@interface Shop (/* Private */)
@property (strong, nonatomic) NSArrayController *productsController;
@end
</code></pre>
<p>The <code>products</code> getter simply return the array controller’s arranged objects and the implementation of the <code>addProduct:</code> and <code>removeProduct:</code> is also fairly simple.</p>
<pre><code class="language-objc">@implementation Shop
- (id)init
{
self = [super init];
if (self == nil) {
return nil;
}
NSMutableArray *products = [NSMutableArray arrayWithObjects:@"Cheetah", @"Puma", @"Jaguar", @"Panther", @"Tiger", @"Leopard", @"Snow Leopard", @"Lion", @"Mountain Lion", nil];
_productsController = [[NSArrayController alloc] initWithContent:products];
return self;
}
- (NSArray *)products
{
return [[self productsController] arrangedObjects];
}
- (void)addProduct:(NSString *)product
{
[[self productsController] addObject:product];
}
- (void)removeProduct:(NSString *)product
{
[[self productsController] removeObject:product];
}
@end
</code></pre>
<p>Since we want a client to be able to observe or bind to the <code>products</code> property, we need to make sure that observers are correctly notified when its content is updated.
Since <code>products</code> simply returns the array controller’s arranged objects it is thus entirely dependent on the <code>arrangedObjects</code> key and we should be able to implement <code>+keyPathsForValuesAffectingValueForKey:</code> accordingly.</p>
<pre><code class="language-objc">+ (NSSet *)keyPathsForValuesAffectingValueForKey:(NSString *)key
{
NSMutableSet *keyPaths = [NSMutableSet setWithSet:[super keyPathsForValuesAffectingValueForKey:key]];
if ([key isEqualToString:@"products"]) {
[keyPaths addObject:@"productsController.arrangedObjects"];
}
return keyPaths;
}
</code></pre>
<p>With this in place, however, if you observe the <code>products</code> key path and try to add a product, you might notice that the observer doesn’t seem to ever be notified.
After spending some time trying to figure out what is going on with your code you might come to the realization that there might be something wrong with <code>NSArrayController</code>. And you might indeed be right.</p>
<p>As a good debugger, before jumping to conclusions we will want to analyze the code flow in the working case and try to infer what is preventing it to work in our very special non-working case. So first thing first, let’s discuss what how dependent keys with KVO actually work.</p>
<p>From a comment on <code>+keyPathsForValuesAffectingValueForKey:</code> in the <code>NSKeyValueObserving</code> header:</p>
<blockquote>
<p>Return a set of key paths for properties whose values affect the value of the keyed property.</p>
<p>When an observer for the key is registered with an instance of the receiving class, KVO itself automatically observes all of the key paths for the same instance, and sends change notifications for the key to the observer when the value for any of those key paths changes.</p>
</blockquote>
<p>So essentially, KVO will inspect the observe the key paths you return from the <code>+keyPathsForValuesAffectingValueForKey:</code> method on the observed object and send change notifications when the value of any of these key path on the object changes.</p>
<p>The mechanism by which KVO detects changes is by inspection of the change dictionary (the one you get in <code>-observeValueForKeyPath:ofObject:change:context:</code>). Based on the observation options, the change dictionary will contain values for the following keys:</p>
<pre><code class="language-objc">NSKeyValueChangeKindKey
NSKeyValueChangeNewKey
NSKeyValueChangeOldKey
NSKeyValueChangeIndexesKey
NSKeyValueChangeNotificationIsPriorKey
</code></pre>
<p>In order to detect value changes, KVO very likely observes the object’s key paths with the old and new options so that it can figure out whether a value was updated. Obviously, this requires a well-formed change dictionary.</p>
<p><strong>Breaking news: <code>NSArrayController</code> doesn’t correctly populate the change dictionary.</strong></p>
<p>Actually, this is far from being a breaking news. It’s a well-known bug that has existed since 10.3 and that is very unlikely to be fixed in the foreseeable future.</p>
<p>If you manually observe an array controller’s arranged objects key path with the old and new observation options, the change dictionary you will get when an object is inserted will look as following:</p>
<pre><code class="language-objc">{
kind = 1;
new = "<null>";
old = "<null>";
}
</code></pre>
<p>It thus explains why KVO wasn’t able to send change notifications to observers of a dependent key paths when the array controller’s arranged objects were updated: it was unable to figure out whether the value had changed, being <code>null</code> before and after.</p>
<p>There is, as always, a workaround. Even though the change dictionary is not correctly populated, observers are still notified with the content changes. One can thus manually track the content, fetch the new content when notified of the change and manually do the comparison if necessary.</p>
<p>In order to fix our example, we have two options:</p>
<ul>
<li>
<p>Manually call <code>willChangeValueForKey:</code> and <code>didChangeValueForKey:</code> for the <code>products</code> key path in the <code>addProduct:</code> and <code>removeProduct:</code> methods. While this will certainly work for our simple case, it won’t scale very well since one will need to remember to call these methods every time the array controller’s content is modified. There are also cases (such as when working with Core Data) where contents might be modified under your feet without you having the opportunity to manually send these change messages.</p>
</li>
<li>
<p>Observe the array controller’s arranged objects and modify the value for the <code>products</code> key path in a KVO-compliant way to ensure that observers are correctly notified. A nice way to do this would be to make the <code>products</code> property <code>readwrite</code> and set a copy of the content every time it is updated.</p>
</li>
</ul>
<pre><code class="language-objc">static NSString * ShopProductsControllerArrangedObjectsContext = @"ShopProductsControllerArrangedObjectsContext";
[_productsController addObserver:self forKeyPath:@"arrangedObjects" options:(NSKeyValueObservingOptionInitial | NSKeyValueObservingOptionOld | NSKeyValueObservingOptionNew) context:&ShopProductsControllerArrangedObjectsContext];
- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
{
if (context == &ShopProductsControllerArrangedObjectsContext) {
NSArray *products = [[self productsController] arrangedObjects];
[self setProducts:[NSArray arrayWithArray:products]];
}
else {
[super observeValueForKeyPath:keyPath ofObject:object change:change context:context];
}
}
</code></pre>
<p>Note that it’s extremely important to take a copy of the arranged objects array. In the case of an ordered collection, the array controller’s arranged objects will usually be an instance of <code>_NSControllerArrayProxy</code> which, even if a subclass of <code>NSArray</code>, will be mutated under your feet. In fact, by the time you receive the change notification the reference you have stored would have already been mutated, hiding the change.</p>
<p>By manually setting the <code>products</code> property with the new value KVO will notify any observers of the changes and correctly populate the change dictionary. If you decide to bind some view in the user interface to the <code>products</code> key path it will now work nicely, as expected.</p>
<p>Even though I am well aware of this bug I seem to constantly fall for it. By writing it down here’s hoping I won’t make that mistake again.</p>
<p>As usual I’ve uploaded the sample code as a <a href="https://gist.github.com/ddeville/7173457">gist</a> so that’s it’s easier to follow.</p>
/2013/10/a-note-about-kvo-dependent-keys
A note about KVO dependent keys
2013-10-26T10:21:38Z
2013-10-26T10:21:38Z
<p>In my <a href="/2013/10/key-value-observing-for-the-mainstream">previous post</a> I mentioned <code>+keyPathsForValuesAffectingValueForKey:</code> and how you should always add the result from calling this method on <code>super</code> to the set you return.</p>
<p>I also noted that using <code>+keyPathsForValuesAffecting<Key></code> could be a bad idea since it adds complexity when using dependencies across a class hierarchy.</p>
<p>Let’s see what I meant by that with an example (the example is available as a <a href="https://gist.github.com/ddeville/7169443">gist</a>, probably easier to read)</p>
<pre><code class="language-objc">@interface Animal : NSObject
- (NSString *)identity;
@property (copy, nonatomic) NSString *name;
@end
@implementation Animal
- (NSString *)identity
{
return [NSString stringWithFormat:@"My name is %@.", [self name]];
}
@end
@interface Monkey : Animal
@property (assign, nonatomic) NSUInteger age;
@end
@implementation Monkey
- (NSString *)identity
{
return [[super identity] stringByAppendingFormat:@" I am %lu.", [self age]];
}
@end
</code></pre>
<p>This is a simple example: an <code>Animal</code> class declaring an <code>identity</code> method and a <code>name</code> property. The identity will change based on the name. And another <code>Monkey</code> class, a subclass of <code>Animal</code>, adding an <code>age</code> property and overriding <code>identity</code> to include the <code>age</code>.</p>
<p>Now, given that the value returned by <code>identity</code> will differ based on the values of the <code>name</code> and <code>age</code> properties we will want to let KVO know about these dependencies so that an observer is dully notified when such value changes.</p>
<h2><a href="#returning-key-paths-which-values-affect-the-value-of-a-key" aria-hidden="true" class="anchor" id="returning-key-paths-which-values-affect-the-value-of-a-key"></a>Returning key paths which values affect the value of a key</h2>
<p>The <code>NSKeyValueObservingCustomization</code> informal protocol declares a method that lets us do just this: <code>+keyPathsForValuesAffectingValueForKey:</code>.</p>
<p>Alternatively, you can implement a <em>shortcut</em> method <code>+keyPathsForValuesAffecting<Key></code> where <code><Key></code> is the key being affected. I usually don’t recommend using this method and I will explain why.</p>
<p>But first, let’s see how one would implement <code>+keyPathsForValuesAffectingValueForKey:</code> for our example.</p>
<h3><a href="#using-keypathsforvaluesaffectingvalueforkey" aria-hidden="true" class="anchor" id="using-keypathsforvaluesaffectingvalueforkey"></a>Using <code>+keyPathsForValuesAffectingValueForKey:</code></h3>
<p>The docs clearly specify how <code>+keyPathsForValuesAffectingValueForKey:</code> works:</p>
<blockquote>
<p>The default implementation of this method searches the receiving class for a method whose name matches the pattern <code>+keyPathsForValuesAffecting<Key></code>, and returns the result of invoking that method if it is found. Any such method must return an NSSet. If no such method is found, an NSSet that is computed from information provided by previous invocations of the now-deprecated <code>setKeys:triggerChangeNotificationsForDependentKey:</code> method is returned, for backward binary compatibility.</p>
</blockquote>
<p>A key point is that the default implementation (leaving the backward compatibility note aside) will return an <code>NSSet</code>. In other words, it is always safe to call <code>super</code>, even if the superclass is not directly implementing this method.</p>
<p>Keeping this in mind, a natural implementation in both <code>Animal</code> and <code>Monkey</code> will be as following:</p>
<pre><code class="language-objc">// In Animal
+ (NSSet *)keyPathsForValuesAffectingValueForKey:(NSString *)key
{
NSMutableSet *keyPaths = [NSMutableSet setWithSet:[super keyPathsForValuesAffectingValueForKey:key]];
if ([key isEqualToString:@"identity"]) {
[keyPaths addObject:@"name"];
}
return keyPaths;
}
// In Monkey
+ (NSSet *)keyPathsForValuesAffectingValueForKey:(NSString *)key
{
NSMutableSet *keyPaths = [NSMutableSet setWithSet:[super keyPathsForValuesAffectingValueForKey:key]];
if ([key isEqualToString:@"identity"]) {
[keyPaths addObject:@"age"];
}
return keyPaths;
}
</code></pre>
<p>As explained in the <a href="/2013/10/key-value-observing-for-the-mainstream">previous post</a>, it is very important to aggregate the key paths in <code>Monkey</code> with the ones returned by the superclass <code>Animal</code>. Since the subclass is overriding this method, if it lacked doing so the <code>identity</code> key wouldn’t be seen as affected by the <code>name</code> key path by KVO.</p>
<h3><a href="#using-keypathsforvaluesaffectingkey" aria-hidden="true" class="anchor" id="using-keypathsforvaluesaffectingkey"></a>Using <code>+keyPathsForValuesAffecting<Key></code></h3>
<p>People hating the fact that KVO funnels most of its work through a couple of methods might be lured into using <code>+keyPathsForValuesAffecting<Key></code>. These same people might also end up implementing these methods as following:</p>
<pre><code class="language-objc">// In Animal
+ (NSSet *)keyPathsForValuesAffectingIdentity
{
return [NSSet setWithObject:@"name"];
}
// In Monkey
+ (NSSet *)keyPathsForValuesAffectingIdentity
{
return [NSSet setWithObject:@"age"];
}
</code></pre>
<p>You probably can spot the problem here in 2 seconds: the <code>Monkey</code> implementation overrides the <code>Animal</code> one, discarding anything it returns.</p>
<p>We need to, somehow, retrieve the key paths from the <code>Animal</code> implementation and aggregate to the key paths we return in <code>Monkey</code>.</p>
<p>But here comes the problem: which method should we call on <code>super</code>? We need to assume that the implementation of this method is not exposed by the superclass. Even if it was declared, can we safely assume that it’s correctly implemented by the superclass? Remember that <code>+keyPathsForValuesAffecting<Key></code> will not end up calling <code>+keyPathsForValuesAffectingValueForKey:</code> on your behalf but is rather the other way around.</p>
<pre><code class="language-objc">// In Monkey
+ (NSSet *)keyPathsForValuesAffectingIdentity
{
/*
This will not compile if `Animal` doesn't declare that it implements
`+keyPathsForValuesAffectingIdentity`.
If it did declare it but didn't implement, KVO would not end up
calling `+keyPathsForValuesAffectingValueForKey:` for us and an
unrecognized selector exception would be thrown at runtime.
*/
NSMutableSet *keyPaths = [NSMutableSet setWithSet:[super keyPathsForValuesAffectingIdentity]];
[keyPaths addObject:@"age"];
return keyPaths;
}
</code></pre>
<p>I cannot think of a nice way to fix this problem other than always declaring the
<code>+keyPathsForValuesAffecting<Key></code> methods that you implement so that subclasses can safely call <code>super</code>. This is not very nice.</p>
<p>A workaround would be to call <code>+keyPathsForValuesAffectingValueForKey:@"<Key>"</code> on <code>super</code>.</p>
<pre><code class="language-objc">// In Monkey
+ (NSSet *)keyPathsForValuesAffectingIdentity
{
NSMutableSet *keyPaths = [NSMutableSet setWithSet:[super keyPathsForValuesAffectingValueForKey:@"identity"]];
[keyPaths addObject:@"age"];
return keyPaths;
}
</code></pre>
<p>However, if you ended up doing this, I’d argue that you could probably forgo using <code>+keyPathsForValuesAffecting<Key></code> altogether and simply implement <code>+keyPathsForValuesAffectingValueForKey:</code> in the subclass.</p>
<h2><a href="#when-should-i-use-keypathsforvaluesaffectingkey" aria-hidden="true" class="anchor" id="when-should-i-use-keypathsforvaluesaffectingkey"></a>When should I use <code>+keyPathsForValuesAffecting<Key></code>?</h2>
<p>As we’ve seen, it’s never really appropriate to <code>+keyPathsForValuesAffecting<Key></code>. There is a very obvious case for it though. Can you think about it? Let me try and lead you to it.</p>
<p>Can you think of a case where you wouldn’t want to override a method? Maybe because you wouldn’t know how to call this method on <code>super</code>? Maybe because you might end up stomping its implementation? Yes, categories!</p>
<h3><a href="#dependent-keys-in-a-category" aria-hidden="true" class="anchor" id="dependent-keys-in-a-category"></a>Dependent keys in a category</h3>
<p>Once again, the docs for <code>+keyPathsForValuesAffectingValueForKey:</code> give you quite a hint:</p>
<blockquote>
<p><strong>Note</strong>
You must not override this method when you add a computed property to an existing class using a category, overriding methods in categories is unsupported. In that case, implement a matching <code>+keyPathsForValuesAffecting<Key></code> to take advantage of this mechanism.</p>
</blockquote>
<p>Say we created a category on <code>Animal</code> to add a new method <code>funkyName</code>. This method will simply prepend <code>"Funky "</code> to the animal name. Obviously, the value returned by this method will depend on the value returned by the <code>name</code> method.</p>
<p>In order for observers of <code>funkyName</code> to be notified when the name changes, we will want to mark the <code>name</code> key path as affecting the <code>funkyName</code> key. Being in a category, we wouldn’t think of “overriding” the <code>+keyPathsForValuesAffectingValueForKey:</code> method. It is indeed the perfect time to use <code>+keyPathsForValuesAffecting<Key></code>!</p>
<pre><code class="language-objc">@interface Animal (Funky)
@property (readonly, copy, nonatomic) NSString *funkyName;
@end
@implementation Animal (Funky)
- (NSString *)funkyName
{
return [@"Funky " stringByAppendingString:[self name]];
}
+ (NSSet *)keyPathsForValuesAffectingFunkyName
{
return [NSSet setWithObjects:@"name", nil];
}
@end
</code></pre>
<p>The implementation of <code>+keyPathsForValuesAffectingFunkyName</code> here doesn’t need to try and aggregate the result with the one from the class itself or its superclass. In fact, it’s nicely isolated in the category, along with the dependent property.</p>
<p>I hope this helped you understand how the key dependency KVO mechanism works. As I said in the previous post, KVO has very simple rules and is very powerful, as long as you follow the rules.</p>
<hr />
<p><em>I have to credit my friend and former colleague <a href="http://github.com/keithduncan">Keith Duncan</a>. He's the KVO guru. I have merely been trying to absorb as much knowledge from him as I could while we were working together. True story.</em></p>
/2013/10/key-value-observing-for-the-mainstream
Key-Value Observing for the mainstream
2013-10-22T08:57:59Z
2013-10-22T08:57:59Z
<p>I really enjoy reading the <a href="http://nshipster.com">NSHipster</a> blog. The article about <a href="http://nshipster.com/key-value-observing/">Key-Value Observing</a> (KVO) from October 7th was however slightly misleading and should be taken with a grain of salt.</p>
<p>I wasn’t expecting much after the introduction (“Key-Value Observing has the <em>worst</em> API in all of Cocoa”) but unfortunately it went from bad to worse.</p>
<p>I don’t like to point fingers without explaining what I think is wrong and what I would have done instead so I will try to expose my thoughts in this post.</p>
<h2><a href="#nskeyvalueobservingoptioninitial-is-indeed-compelling" aria-hidden="true" class="anchor" id="nskeyvalueobservingoptioninitial-is-indeed-compelling"></a>NSKeyValueObservingOptionInitial is indeed compelling</h2>
<p>When discussing the method for adding an observer, Mattt explains:</p>
<blockquote>
<p>Yuck. What makes this API so unsightly is the fact that those last two parameters are almost always 0 and NULL, respectively.</p>
</blockquote>
<p>I will discuss the context in the next section but for now let’s see what are the benefits of the <code>NSKeyValueObservingOptions</code>, in particular <code>NSKeyValueObservingOptionInitial</code>.</p>
<p>How many times have you written code similar to the following:</p>
<pre><code class="language-objc">- (void)loadView
{
[super loadView];
[_object addObserver:self forKeyPath:@"someKey" options:0 context:NULL];
[self _updateSomething];
}
- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
{
[self _updateSomething];
}
</code></pre>
<p>What if I told you that you can get rid of that first <code>_updateSomething</code> call? That’s right, by using <code>NSKeyValueObservingOptionInitial</code> your observer will be notified (and the <code>_updateSomething</code> called) even before the registration completes. Your update code is now concentrated in the observation handling and you don’t need to duplicate code or even have to worry about making sure that method was called when setting up the object.</p>
<p>It might sound like very little but trust me it’s actually quite nice in practice.</p>
<h2><a href="#always-always-always-use-the-context" aria-hidden="true" class="anchor" id="always-always-always-use-the-context"></a>Always, always, always use the context</h2>
<p>Granted, Mattt does talk about using the context but unfortunately, the way it’s discussed implies that it’s an edge case you shouldn’t really worry or care about.</p>
<blockquote>
<p>As for context, this parameter is a value that can be used later to differentiate between observations of different objects with the same key path. It's a niche use case, ...</p>
</blockquote>
<p>The truth is, in practice it is <em>very</em> important to <em>always</em> use the context to differentiate between observed objects and not rely on the key path. We are working with an object-oriented language and we should always accommodate for the case where a class might be subclassed or a super class expecting a behavior that your own class will prevent.</p>
<p>Let’s take an example. Assume you have the following class hierarchy (see this <a href="https://gist.github.com/ddeville/7097600">gist</a> for the complete example):</p>
<pre><code class="language-objc">#import <Foundation/Foundation.h>
@interface Stalked : NSObject
@property (assign, nonatomic) NSInteger tag;
@end
@implementation Stalked
@end
@interface Parent : NSObject
{
@protected
Stalked *_stalked;
}
@property (strong, nonatomic) Stalked *stalked;
@end
@implementation Parent
- (id)init
{
self = [super init];
if (self == nil) {
return nil;
}
_stalked = [[Stalked alloc] init];
[_stalked addObserver:self forKeyPath:@"tag" options:0 context:NULL];
return self;
}
- (void)dealloc
{
[_stalked removeObserver:self forKeyPath:@"tag" context:&_ParentStalkedTagContext];
}
@end
@interface Child : Parent
@end
@implementation Child
- (id)init
{
self = [super init];
if (self == nil) {
return nil;
}
[_stalked addObserver:self forKeyPath:@"tag" options:0 context:NULL];
return self;
}
- (void)dealloc
{
[_stalked removeObserver:self forKeyPath:@"tag" context:&_ChildStalkedTagContext];
}
@end
int main(int argc, const char **argv)
{
Child *child = [[Child alloc] init];
[[child stalked] setTag:2];
return 0;
}
</code></pre>
<p>As you can see, we have a simple class hierarchy: <code>Child</code> subclassing <code>Parent</code> itself subclassing <code>NSObject</code>. <code>Parent</code> has a <code>stalked</code> property. Both the <code>Parent</code> and <code>Child</code> are observing the <code>tag</code> key path on the <code>Stalked</code> object. Simple and pretty common scenario. You can notice that neither the child nor the parent are specifying a context when registering as an observer.</p>
<p>Now if we had to implement the observation as described in the article, what do you think would happen?</p>
<pre><code class="language-objc">// In Parent
- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
{
if ([keyPath isEqualToString:@"tag"]) {
NSLog(@"Parent stuff");
}
else {
[super observeValueForKeyPath:keyPath ofObject:object change:change context:context];
}
}
// In Child
- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
{
if ([keyPath isEqualToString:@"tag"]) {
NSLog(@"Child stuff");
}
else {
[super observeValueForKeyPath:keyPath ofObject:object change:change context:context];
}
}
</code></pre>
<p>That’s right, since the <code>Child</code> overrides this method it would be called twice: once for the child and once for the parent. However, since it’s only filtering based on the key path this will print <em>Child stuff</em> twice and the parent will never have a chance to see the changes. This is bad.</p>
<p>What you should do instead is use the context as a way to filter between observers.</p>
<pre><code class="language-objc">// In Parent
static NSString * _ParentStalkedTagContext = @"_ParentStalkedTagContext";
[_stalked addObserver:self forKeyPath:@"tag" options:0 context:&_ParentStalkedTagContext];
- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
{
if (context == &_ParentStalkedTagContext) {
NSLog(@"Parent stuff");
}
else {
[super observeValueForKeyPath:keyPath ofObject:object change:change context:context];
}
}
// In child
static NSString * _ChildStalkedTagContext = @"_ChildStalkedTagContext";
[_stalked addObserver:self forKeyPath:@"tag" options:0 context:&_ChildStalkedTagContext];
- (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context
{
if (context == &_ChildStalkedTagContext) {
NSLog(@"Child stuff");
}
else {
[super observeValueForKeyPath:keyPath ofObject:object change:change context:context];
}
}
</code></pre>
<p>By doing such, the child observation method will still be called twice but this time the second call will be filtered out and dispatched to the parent instead.</p>
<p>I cannot believe anyone slightly experienced with KVO wouldn’t see the benefit of doing this.</p>
<h2><a href="#unsubscribing-is-no-madness" aria-hidden="true" class="anchor" id="unsubscribing-is-no-madness"></a>Unsubscribing is no madness!</h2>
<p>In KVO, subscribing and unsubscribing follows a very simple rule: you are responsible for unsubscribing to things you subscribed to.</p>
<p>There is no need to check whether an object is observing another object to decide whether it should unsubscribe. An object should be able to track whether it subscribed to some changes on the key path of another object in the first place and unsubscribe from it in due time.</p>
<p>Does this remind you of something? To me it sounds exactly like reference counting in memory management. Would you ever write the following code?</p>
<pre><code class="language-objc">while ([object retainCount] > 0) {
[object release];
}
</code></pre>
<p>I don’t think so.</p>
<p>Or, assuming every object had a table of objects that called <code>retain</code> on it, would you write the following code?</p>
<pre><code class="language-objc">if ([[object retainers] containsObject:self]) {
[object release];
}
</code></pre>
<p>Sounds unlikely.</p>
<p>Or even worse, would you ever consider handling a signal for a memory error due to calling <code>free</code> on a freed piece of memory because you couldn’t remember whether you previously retained this object and decided to release it anyway.</p>
<p>No. Because this is madness.</p>
<p>So why would you try and catch an exception when attempting to remove an observer for something you weren’t observing in the first place?</p>
<p>Remember the simple rule: remove an observer only if you were observing it in the first place.</p>
<h2><a href="#always-call-super-in-keypathsforvaluesaffectingvalueforkey" aria-hidden="true" class="anchor" id="always-call-super-in-keypathsforvaluesaffectingvalueforkey"></a>Always call super in <code>keyPathsForValuesAffectingValueForKey:</code></h2>
<p><code>keyPathsForValuesAffectingValueForKey:</code> is an amazing feature of KVO that allows one to set a dependency tree for values affecting a key path.</p>
<p>Imaging having a <code>title</code> property in a parent view controller depending on the title in each child view controller, itself depending on the title in its own children. Using this method you can have the parent view controller’s title depend on the whole tree of titles without having to write any change notification code or having to update state for each object on the tree: how declarative!</p>
<p>Mattt’s example is as following:</p>
<pre><code class="language-objc">+ (NSSet *)keyPathsForValuesAffectingAddress
{
return [NSSet setWithObjects:@"streetAddress", @"locality", @"region", @"postalCode", nil];
}
</code></pre>
<p>I have a problem with this: it doesn’t take into consideration what <code>super</code> might have to say about this key path (assuming the superclass actually itself responds to <code>address</code>).</p>
<p>The right way, in my opinion, to write this is as following:</p>
<pre><code class="language-objc">+ (NSSet *)keyPathsForValuesAffectingValueForKey:(NSString *)key
{
NSMutableSet *keyPaths = [NSMutableSet setWithSet:[super keyPathsForValuesAffectingValueForKey:key]];
if ([key isEqualToString:@"address") {
[keyPaths addObjectsFromArray:@[@"streetAddress", @"locality", @"region", @"postalCode"]];
}
return keyPaths;
}
</code></pre>
<p>If the superclass returns some key paths affecting this key, they will be taken into considerations.</p>
<h2><a href="#conclusion" aria-hidden="true" class="anchor" id="conclusion"></a>Conclusion</h2>
<p>Sorry about the rant. I’d like to reiterate that I have a great esteem of the NSHipster blog. I read amazing articles and learned a ton of things from it.
I guess that’s why it feels weird to read misleading information once in a while ;)</p>
<p>Keep on the good work Mattt.</p>
/2013/09/why-you-should-use-wstrict-selector-match
Why you should use -Wstrict-selector-match
2013-09-28T13:05:25Z
2013-09-28T13:05:25Z
<p>This morning I’ve read an interesting <a href="http://www.cocoanetics.com/2013/09/toplayoutguide-97-pixels/">post</a> on Oliver Drobnik’s blog and since answering properly on Twitter is kind of hard I decided to write a post about my understanding of the issue.</p>
<p>In a nutshell, his post shows how invoking the <code>length</code> method on an object conforming to the <code>UILayoutSupport</code> protocol might not return what one would expect if the reference is typed as <code>id</code>.
The reason for this is that the compiler doesn’t have enough information about the type of the object this method will be invoked onto and it needs to assume things by picking <em>a</em> method signature to set up the calling environment. In this case, it will likely pick the <code>length</code> method on <code>NSString</code> which happens to return an <code>NSUInteger</code>. However, in the <code>UILayoutSupport</code> case, the <code>length</code> method returns a <code>CGFloat</code>.</p>
<p>Now, you might think this is not a big deal but it actually is and let’s try and figure why 2 very distinct numbers are returned when using the correctly typed reference or not.</p>
<p>Since I have a much better understanding of the x86-64 assembly, ABI and calling conventions than I have of ARM or i386, I will try and illustrate the issue in a little Cocoa application. Luckily AppKit has a similar <code>CGFloat</code> returning <code>length</code> function in <code>NSStatusItem</code> that we can use to reproduce a similar issue (<code>NSString</code> is itself available in Foundation in both iOS and OS X).</p>
<p>Let’s assume we have the following program:</p>
<pre><code class="language-objc">@implementation AppDelegate
- (void)applicationDidFinishLaunching:(NSNotification *)notification
{
NSStatusItem *statusItem = [[NSStatusItem alloc] init];
[statusItem setLength:60.0];
CGFloat correctLength = [self _checkCorrectLength:statusItem];
CGFloat wrongLength = [self _checkWrongLength:statusItem];
NSParameterAssert(correctLength == wrongLength);
}
- (CGFloat)_checkCorrectLength:(NSStatusItem *)statusItem
{
return [statusItem length];
}
- (CGFloat)_checkWrongLength:(id)statusItem
{
return [statusItem length];
}
@end
</code></pre>
<p>This is a simple enough program and if we run it we will fail on the assertion that 60 is not equal to 16. No matter what value we set for the length, the incorrect value will always be 16.</p>
<p>Similarly as what Oliver describes in his post, we have here two different cases. In the <code>_checkCorrectLength</code>, the first argument is correctly typed as a reference to an <code>NSStatusItem</code> object on which the <code>length</code> method is later invoked on. In the the <code>_checkWrongLength</code>, the argument is typed as <code>id</code> and when the <code>length</code> method is invoked, we let the compiler figure things out on its own.</p>
<p>Before diving into the assembly generated by the compiler for our own code, let’s disassemble the <code>length</code> function in <code>NSStatusItem</code>.</p>
<pre><code class="language-nasm">AppKit`-[NSStatusItem length]:
0x7fff906bd264: pushq %rbp
0x7fff906bd265: movq %rsp, %rbp
0x7fff906bd268: movq -442811119(%rip), %rax ; NSStatusItem._fLength
0x7fff906bd26f: movsd (%rdi,%rax), %xmm0
0x7fff906bd274: popq %rbp
0x7fff906bd275: ret
</code></pre>
<p>Before explaining what is going on, it is worth having a quick look at the <code>NSStatusItem</code> instance variables structure.</p>
<pre><code class="language-objc">@interface NSStatusItem : NSObject
{
@private
NSStatusBar* _fStatusBar;
CGFloat _fLength;
NSWindow* _fWindow;
NSView* _fView;
int _fPriority;
etc...
}
</code></pre>
<p>Coming back to the disassembly of <code>-[NSStatusItem length]</code>, we can see that it is pretty simple. The offset of the <code>_fLength</code> iVar in the class struct is moved into the <code>%rax</code> register. At this stage it is worth noting that <code>%rdi</code> still contains the receiver hence the <code>NSStatusItem</code> object reference. Further on, the value at the address <code>object pointer + iVar offset</code> is dereferenced into the <code>%xmm0</code> register, which is a register used for floating point calculations. The stack is cleaned up and the function then returns.</p>
<p>At this point, it is very important to note that, as per the <a href="http://www.x86-64.org/documentation/abi.pdf">x86-64 ABI</a>, the <code>%xmm0</code> register is documented as <em>used to pass and return floating point arguments</em>. In the usual case where an integer argument is returned the <code>%rax</code> is used to return values from a function.</p>
<p>If you are used into looking at disassembly you might have figured out what went wrong by now but for sake of completeness let’s have a look at the disassembly of the <code>_checkCorrectLength:</code> and <code>_checkWrongLength</code> methods.</p>
<pre><code class="language-nasm">Reg`-[AppDelegate _checkCorrectLength:] at AppDelegate.m:26:
0x100001420: pushq %rbp
0x100001421: movq %rsp, %rbp
0x100001424: subq $32, %rsp
0x100001428: leaq 5281(%rip), %rax
0x10000142f: movq %rdi, -8(%rbp)
0x100001433: movq %rsi, -16(%rbp)
0x100001437: movq %rdx, -24(%rbp)
0x10000143b: movq -24(%rbp), %rdx
0x10000143f: movq %rdx, %rdi
0x100001442: movq %rax, %rsi
0x100001445: callq *5253(%rip)
0x10000144b: addq $32, %rsp
0x10000144f: popq %rbp
0x100001450: ret
</code></pre>
<pre><code class="language-nasm">Reg`-[AppDelegate _checkWrongLength:] at AppDelegate.m:31:
0x100001460: pushq %rbp
0x100001461: movq %rsp, %rbp
0x100001464: subq $48, %rsp
0x100001468: movq %rdi, -8(%rbp)
0x10000146c: movq %rsi, -16(%rbp)
0x100001470: movq %rdx, -24(%rbp)
0x100001474: movq 5205(%rip), %rsi
0x10000147b: leaq 5198(%rip), %rdi
0x100001482: movq %rdi, -32(%rbp)
0x100001486: movq %rdx, %rdi
0x100001489: movq -32(%rbp), %rdx
0x10000148d: movq %rsi, -40(%rbp)
0x100001491: movq %rdx, %rsi
0x100001494: movq -40(%rbp), %rax
0x100001498: callq *%rax
0x10000149a: movd %rax, %xmm0
0x10000149f: movaps 218(%rip), %xmm1
0x1000014a6: punpckldq %xmm1, %xmm0
0x1000014aa: movapd 222(%rip), %xmm1
0x1000014b2: subpd %xmm1, %xmm0
0x1000014b6: haddpd %xmm0, %xmm0
0x1000014ba: addq $48, %rsp
0x1000014be: popq %rbp
0x1000014bf: ret
</code></pre>
<p>Keep in mind that both methods are returning a <code>CGFloat</code> so the the <code>%xmm0</code> register will be used to hold the return (floating point) argument. Also, in both method implementations, the <code>callq</code> instruction is a call to <code>objc_msgSend</code> with the object reference and the <code>length</code> selector.</p>
<p>In the correct case, we can see that upon return from the <code>objc_msgSend</code> function call for <code>length</code>, the stack is quickly cleaned up and the method implementation returns. Since the <code>%xmm0</code> register hasn’t been touched, upon return of the method implementation it will still contain the value that was stored in the <code>length</code> method implementation.</p>
<p>In the incorrect case, we can clearly see that the compiler has emitted code as if the <code>length</code> method implementation returned an integer, by retrieving the return value in the <code>%rax</code> register. Since the <code>_checkWrongLength</code> is supposed to return a floating point, the value is then moved to the <code>%xmm0</code> register and after a few instructions I honestly don’t have the knowledge to understand and a bit of stack cleanup, the function returns. So basically, the function moved the value stored by the <code>length</code> implementation into <code>%rax</code> into <code>%xmm0</code> and returned.</p>
<p>If you followed up to here you have the answer: the value returned by the <code>length</code> method on an <code>NSStatusItem</code> object reference typed to <code>id</code> is the offset of the <code>_fLength</code> instance variable in the <code>NSStatusItem</code> class structure.</p>
<p>In your case Oliver, I also suspect 97 to be the offset of some instance variable into some layout guide object class structure. However, it's also important to note that this would be completely coincidental. <code>%rax</code> (or its counterpart in ARM) is not preserved across function calls and for the case of a function returning a floating point argument (and thus writing its return value in <code>xmm0</code> or <code>xmm1</code>, or their ARM counterpart) <code>%rax</code> could contain any garbage.</p>
<p>There is a very easy way to avoid such issues by trusting the compiler and enabling the <code>-Wstrict-selector-match</code> warning. The compiler will warn you that it cannot make a smart decision about it and will likely result in some unexpected behavior.</p>
/2013/09/an-actually-asynchronous-and-cancelable-url-connection
An actually asynchronous URL connection
2013-09-07T19:53:41Z
2013-09-07T19:53:41Z
<p>Last week I read a <a href="https://twitter.com/landonfuller/status/375403178206171137">tweet</a> from <a href="https://twitter.com/landonfuller">Landon Fuller</a> that showed the disassembly of <code>+[NSURLConnection sendAsynchronousRequest:queue:completionHandler:]</code> mainly as a simple call to <code>+[NSURLConnection sendSynchronousRequest:returningResponse:error:]</code> dispatched on a global queue.</p>
<p>I have never been a big fan of this asynchronous request API on <code>NSURLConnection</code> since it lacks a major feature in my opinion: <em>cancellation</em>.</p>
<p>Since writing a simple URL connection as an <code>NSOperation</code> subclass is not too hard, let’s go and build it!</p>
<p>First thing first, we will want our method look as close as possible to the one previously mentioned. We will thus provide a new method in a category on <code>NSURLConnection</code></p>
<pre><code class="language-objc">+ (id)sendActuallyAsynchronousRequest:(NSURLRequest *)request queue:(NSOperationQueue *)queue completionHandler:(void (^)(NSURLResponse *response, NSData *data, NSError *connectionError))completionHandler;
</code></pre>
<p>This method returns an opaque object that can be used in order to cancel the request by mean of another method</p>
<pre><code class="language-objc">+ (void)cancelActuallyAsynchronousRequest:(id)connection;
</code></pre>
<p>The implementation is fairly simple: we create a URL connection operation with the request that we dispatch to a global queue. We then have a dependent operation that retrieves the connection’s results and invoke the completion handler. This operation is dispatched to the provided queue or the main queue if none is provided.
Finally, the connection operation is returned.</p>
<pre><code class="language-objc">+ (id)sendActuallyAsynchronousRequest:(NSURLRequest *)request queue:(NSOperationQueue *)queue completionHandler:(void (^)(NSURLResponse *response, NSData *data, NSError *connectionError))completionHandler
{
queue = queue ? : [NSOperationQueue mainQueue];
completionHandler = completionHandler ? : ^ (NSURLResponse *response, NSData *data, NSError *connectionError) {};
_NSURLActuallyAsynchronousURLConnectionOperation *connectionOperation = [[_NSURLActuallyAsynchronousURLConnectionOperation alloc] initWithRequest:request];
[[self _actuallyAsynchronousURLConnectionOperationQueue] addOperation:connectionOperation];
NSOperation *completionOperation = [NSBlockOperation blockOperationWithBlock:^ {
NSURLResponse *response = [connectionOperation response];
NSData *data = [NSData dataWithData:[connectionOperation data]];
NSError *connectionError = [connectionOperation error];
completionHandler(response, data, connectionError);
}];
[completionOperation addDependency:connectionOperation];
[queue addOperation:completionOperation];
return connectionOperation;
}
</code></pre>
<p>Cancellation is also very easy since the opaque object we returned is the operation itself, we simply need to call <code>cancel</code> on it.</p>
<pre><code class="language-objc">+ (void)cancelActuallyAsynchronousRequest:(id)asynchronousRequest
{
NSParameterAssert([asynchronousRequest isKindOfClass:[_NSURLActuallyAsynchronousURLConnectionOperation class]]);
_NSURLActuallyAsynchronousURLConnectionOperation *operation = (_NSURLActuallyAsynchronousURLConnectionOperation *)asynchronousRequest;
[operation cancel];
}
</code></pre>
<p>The implementation of the <code>_NSURLActuallyAsynchronousURLConnectionOperation</code> operation is also fairly basic. This operation is concurrent so that we can benefit from the asynchronous nature of <code>NSURLConnection</code>.</p>
<p>We create a URL connection in the initializer and provide an operation queue as the connection delegate queue so that we don’t need to provide a runloop (usually by using the main runloop or by creating a connection thread running the runloop and that is shared by all the connection operations).</p>
<p>The implementation of <code>start</code> simply starts the connection. Similarly, the implementation of <code>cancel</code> cancels the connection and finish the operation.</p>
<p>We create a mutable data to which we append data each time it is received. This is not an ideal situation when dealing with large response bodies but it is the API provided by <code>NSURLConnection</code> and for the sake of this example we will try to mimic it as much as possible.</p>
<p>Any failure on the connection will be reported as an <code>NSError</code> to the delegate, in which case we store the error and finish the operation. Similarly, when the connection finishes loading we can nicely finish the operation.</p>
<p>Have a look at the complete implementation of the operation on <a href="https://github.com/ddeville/asynchronous-connection">GitHub</a>.</p>
<p>And that’s it! As you can see, providing a simple asynchronous and cancelable asynchronous request built as an operation on top of <code>NSURLConnection</code> is not so hard.</p>
/2013/08/a-class-dump-cocoa-framework
A class-dump Cocoa framework
2013-08-25T11:00:00Z
2013-08-25T11:00:00Z
<p>When building a small OS X app some time ago (more about this soon), I needed a way to access and parse the ObjC segment of a given executable. This can easily be achieved with the command line utility <em>otool</em> but the output is not very readable (or parsable).</p>
<h2><a href="#class-dump" aria-hidden="true" class="anchor" id="class-dump"></a>Class-dump</h2>
<p><a href="https://github.com/nygard">Steve Nygard</a> wrote an amazing tool called <em>class-dump</em> for generating declarations of classes, categories and protocols by examining the Objective-C runtime information stored in a Mach-O file. It also has the option to write these as headers on disk.</p>
<h3><a href="#as-a-command-line-utility" aria-hidden="true" class="anchor" id="as-a-command-line-utility"></a>As a command-line utility</h3>
<p>Since <em>class-dump</em> is a command line utility my first attempt was to use an <strong>NSTask</strong> in order to fork/exec the <em>class-dump</em> process, read the task’s standard out and get the result back into my application’s process.
In order to nicely encapsulate the task I wrapped it in a concurrent <strong>NSOperation</strong> subclass, launching the task from the <code>start</code> method and waiting for the task to exit to finish the operation. I’ve been using <strong>dispatch_io</strong> to read the standard output and standard error out of the <em>class-dump</em> process and it has proven to work very nicely.</p>
<p>While this approach worked very well it had one particular caveat: error handling. <em>Class-dump</em> being a command line utility, its main medium to report error is by printing to <code>stderr</code>. While this works well when running the program in Terminal, one expects a nicer interface than a few logs from a Cocoa application. Also, the program exits with a termination status different from 0 most of the times it encounters an error but there seems to be times where an error is logged even though the termination status is 0. A noticeable example is a Mach-O executable that doesn’t contain any Objective-C runtime information.
In summary, I couldn’t rely 100% on the termination status being different from 0 when an error occurred and relying on <code>stderr</code> receiving data to determine a failure was just too fragile.</p>
<h3><a href="#run-in-process" aria-hidden="true" class="anchor" id="run-in-process"></a>Run in process</h3>
<p>Luckily, <em>class-dump</em> is open-source and instead of trying to patch the command line utility to behave the way I needed I decided to use the <em>class-dump</em> Objective-C interface directly in my Cocoa application, in process.</p>
<p>If you ever took a look inside the Ember application you might have noticed that I really like frameworks. This said, I couldn’t just drag a few classes from the <em>class-dump</em> project into my own project but I instead <a href="https://github.com/ddeville/class-dump">forked</a> the project on GitHub and added a new target for a Framework: <strong>ClassDump.framework</strong>. Its public interface only consists of a single <strong>NSOperation</strong> subclass, the whole complexity being nicely abstracted away. I’ve also added a few categories that help providing better error reporting.</p>
<p>Now, <em>class-dump</em> is a very well written and tested program but it is still dealing with untrusted input data. By having <em>class-dump</em> run in process, if something goes wrong the whole application goes down. This is bad.</p>
<h2><a href="#class-dump-as-a-service" aria-hidden="true" class="anchor" id="class-dump-as-a-service"></a>Class-dump as a service</h2>
<p>Luckily, we have tools to work around this problem. We could step back and decide to launch the <em>class-dump</em> process as an <strong>NSTask</strong> but we would lose the nicer error handling API that we adopted. Introduce Services! By having our code running in an XPC service we can make sure that the service only goes down if something should go wrong, but still be able to directly interface with our Objective-C interface by mean of an XPC connection. This seems like the best of both worlds.</p>
<p>We could have the XPC service as a target of the main application’s project or the <em>class-dump</em> project. However, since I consider it an implementation detail I don’t like the idea of exposing its internals to the rest of the system.</p>
<h2><a href="#operation-framework-service" aria-hidden="true" class="anchor" id="operation-framework-service"></a>Operation-Framework-Service</h2>
<p>Let me introduce my favorite pattern: Operation-Framework-Service! A framework encapsulating an XPC service and vending a simple <strong>NSOperation</strong> as interface. Boom!</p>
<p><img src="/static/media/images/posts/2013/08/operation-framework-service.png" alt="Operation-Framework-Service" title="Operation-Framework-Service" /></p>
<p>As far as the client application is concerned a simple framework import is needed. The API is also extremely simple since it only consists of an <strong>NSOperation</strong> subclass which has a well-known and simple, yet extremely powerful, API.
The operation implementation takes care of setting up an XPC connection with the XPC service. The service itself runs the actual internal operation. As far as the client is concerned however, the whole architecture is abstracted away it just feels as if the operation were running in process.
In order to make full use of the asynchronous behavior of the XPC setup we want our in-process operation to be concurrent. It will simply message the service through the XPC connection on <code>start</code> and wait for the message’s response handler to finish.</p>
<h2><a href="#fitting-in-the-sandbox" aria-hidden="true" class="anchor" id="fitting-in-the-sandbox"></a>Fitting in the Sandbox</h2>
<p>Now, when designing a framework, one needs to plan for every use case. In particular, one needs to take into account the environment the host application will be running in. We want our framework to work nicely in the most restrictive environment while still supporting looser ones. Translating this into modern security tools for Cocoa application this means that our framework will need to play nicely with a sandboxed application, assuming no particular entitlements have been specified.</p>
<p>The XPC service will thus have to be sandboxed (and can specify its own entitlements even though one can argue that it shouldn’t require more than what the host application does, which could well be none). The client-side part of the framework will be running in the host application’s process and will thus inherit its sandbox restrictions.</p>
<p>Keeping this in mind, we should design our framework to work with a sandboxed application with <em>no</em> entitlements.</p>
<p>Luckily, the framework will only need two URLs, one of the executable and one of the directory in which the headers will be written into, and won’t require any network connectivity or fancier capabilities that would require a special entitlement. We can safely assume that the URLs have been acquired by the application in a sandbox-safe manner and we have write access to the export directory.</p>
<h3><a href="#sharing-url-between-sandboxed-applications" aria-hidden="true" class="anchor" id="sharing-url-between-sandboxed-applications"></a>Sharing URL between sandboxed applications</h3>
<p>With this in mind, our problem is only reduced to providing the XPC service with valid URL that it can resolve. <strong>NSURL</strong> has a very nice feature to send URL containing security information between processes called bookmark data. By making sure that the client-side part of the framework create bookmark data before messaging the XPC service we can be sure that the service will be able to resolve this bookmark to a valid URL it will have right to read and/or write.</p>
<h2><a href="#conclusion" aria-hidden="true" class="anchor" id="conclusion"></a>Conclusion</h2>
<p>By slightly extending <em>class-dump</em> and running it in process as an operation we were able to provide a nicer error API. In order to provide fault tolerance we moved the execution of <em>class-dump</em> to an XPC service. Finally, we abstract this service away in a Cocoa framework vending a simple <strong>NSOperation</strong>. Both the service and the framework play nicely in a sandboxed environment.</p>
<p>You can check out the fork on <a href="https://github.com/ddeville/class-dump">GitHub</a>. I have also created a simple OS X sample application that demonstrates how to integrate the framework. The project is also on <a href="https://github.com/ddeville/class-dump-Example">GitHub</a>.</p>
/2013/08/a-case-against-logging
A case against logging
2013-08-24T12:28:43Z
2013-08-24T12:28:43Z
<p><em>This is a repost of an <a href="http://realmacsoftware.com/blog/block-debugging">article</a> I published on the Realmac Software blog</em>.</p>
<p>I don’t use logging. From the simple <code>printf</code> or <code>NSLog</code> to more complex logging frameworks such as <a href="https://github.com/robbiehanson/CocoaLumberjack">CocoaLumberjack</a>. In this post I will try to explain the reasoning behind it, from a development, testing and release perspective, and propose alternatives that are, in my opinion, much more powerful.</p>
<h2><a href="#during-development" aria-hidden="true" class="anchor" id="during-development"></a>During development</h2>
<p>A common use case for logging is to follow the flow while developing. So one would insert some <code>NSLog</code> statements throughout the code, mainly to figure out if some part of the code has been executed.</p>
<p>I have two problems with this approach.</p>
<p>First, it requires to build your project and run again. If you are working on a very small project it might be acceptable but it can become a large hit to your productivity if your project takes even only one minute to build. Also, some code path might only occur after the application has been running for some time and you probably don’t want to wait for it occur after each launch.</p>
<p>Second, once the message has been logged, that’s it, all the information you might have been after has been written to the console and there’s no more you can go after. You might be wondering about the state of other parts of the application but there is just no way you can find out about them since your logging statements were added at compile time and logging more things would require you to build your application again.</p>
<p>I have a very simple solution for both of these problems: use the debugger and use breakpoints.</p>
<p>A breakpoint can be added at any time during the execution of your program and does not require recompiling. The huge benefit of breakpoints is that when you broke in the debugger, it can give everything you might want to know about the current state of your program. It actually has a perfect, complete snapshot of your program at this very moment in time. Just be curious and dive into it. You might want to ask for the current value of some variables, ask for a backtrace of a current thread, step into some function, disassemble some parts of the framework, and much much more.</p>
<p>One of the critics I hear more often against breakpoints is: “but I don’t want to break!”. Well, no one said that you had to pause execution when a breakpoint is reached. In Xcode you can define the type of action your breakpoint should take and whether it should continue execution after evaluating. So if you’re into logging, you could have the action being logging a message and continue after evaluating. This gives you even more power than simple logging since you can decide to log a hit count for example, log only if a condition has been met, you name it. And remember, you don’t need to build the application again, you can add a breakpoint right in the middle of the execution of your program.</p>
<p><img src="/static/media/images/posts/2013/08/breakpoint_logging.png" alt="Setting a breakpoint that logs and continue after evaluating" /></p>
<p>A situation where you could run into limitations with breakpoints is when you are debugging multiple programs simultaneously. Let’s take an example: you have an application that uses a helper application or an XPC service launched by the main application. Both the main and helper applications are in the same project and even share some code. When you run the main application, the helper application will be launched. Now see, only the main application will be running in the debugger and any breakpoint in portions of the code run by the helper application will not be hit.
However, there is a solution for this. You can simply attach the debugger to your helper application, directly from within Xcode.</p>
<p><img src="/static/media/images/posts/2013/08/attach_process_1.png" alt="Attaching to a process that is currently running" /></p>
<p>Once attached, your helper application will be running in the debugger too and your breakpoints will be hit, just like they do in the main application. If you need to break early in the lifetime of the helper application or just cannot wait for it to launch to attach to it, you can also specify a process ID or name and it will be attached as soon as it launches. You can set it up directly from Xcode, it only uses the LLDB <code>process attach --waitfor</code> though.</p>
<p><img src="/static/media/images/posts/2013/08/attach_process_2.png" alt="Attaching to a process by name" /></p>
<h2><a href="#during-testing" aria-hidden="true" class="anchor" id="during-testing"></a>During testing</h2>
<p>Another area where logging is particularly unhelpful is during testing. During testing, you want your bugs to be as obvious as possible so that you can catch them early and not ship them in your product.</p>
<p>I have a solution for this: assertions! Wherever I expect a variable or a method return to have a particular value I tend to assert it with <code>NSParameterAssert</code> (or <code>NSCParameterAssert</code> when inside a function). It is important to know that <code>NSParameterAssert</code> throws a <code>NSInternalInconsistencyException</code> Objective-C exception when the assert fails. Thus, make sure that Objective-C exceptions are enabled for each configuration you need them in your build settings. We have them enabled in every configuration, including the App Store release configuration, for reasons that will become clearer in the next paragraph.</p>
<p>Second, by default a Cocoa application doesn’t crash in case of an uncaught exception. While this might sound like a good thing, it really isn’t since it simply logs the exception and backtrace to the console and silently fails. You can set NSApplicationCrashOnExceptions to <code>YES</code> in the user defaults and have your application to cleanly crash when an uncaught exception happens. Again, we have this turned on in every configuration, including the App Store release one.</p>
<p>With all this set up, your application should nicely crash and a crash report will be generated every time your tester encounters some situation where the application is not behaving as it should. Trust me, this is invaluable. A good crash log is far more useful than a thousand lines of console logs since it gives you a perfect snapshot of what went wrong at the exact moment when your application crashed.</p>
<p>Relying on logging during testing means a tester should really always keep the Console open in order to catch issues that might just silently fail and log. I might sound like a broken record but a clean crash is a very clear hint that something went wrong and it just saves both you and your tester’s time, by not having to peak through hundred of lines of log to find oddities.</p>
<h2><a href="#once-the-product-has-been-released" aria-hidden="true" class="anchor" id="once-the-product-has-been-released"></a>Once the product has been released</h2>
<p>This is, in my opinion, the most interesting part. To start with, I’d like to discuss something that might be a bit controversial: you should never assume your customers to be technical. Asking a user to copy and paste logs from the Console, launch your application with a special keyboard shortcut to enable logging mode, modifying some plist somewhere ~/Library, etc… should never be an option. In Cocoa, we have much better tools to report problems to users and we should really use them rather than relying on technologies such as logging that should never be applicable outside of a command line environment.</p>
<p>This said, what you should do when you encounter problems in your released application? First of all, use <code>NSError</code>. Rather than log an error, create an instance of <code>NSError</code> and return it to the caller, often by mean of a error by-ref parameter in the method. By doing this, the caller is in charge of inspecting the error if the method didn’t return the expected result and can either <a href="http://realmacsoftware.com/blog/cocoa-error-handling-and-recovery">attempt some recovery option</a> or have it bubble up to the interface where it can be presented to the user. Error presentation can be anything from an alert to an inline text in the current view. The key concept is that the error is presented exactly where the problem happened, making it obvious to the user that something went wrong and explaining the problem.</p>
<p>As in every piece of software, even after thorough testing, some unexpected behavior might still occur. As I stated above, in such cases a clean crash is far more appropriate than a silent failure generating a log while potentially leaving the application in a state that might damage user data.</p>
<p>Therefore, you should avoid silent failures at all cost. By using assertions in place of logging, by enabling exceptions in release configurations and by crashing on exceptions you can ensure that your application will cleanly crash in case something unexpected happens. And you will receive a crash report. Obviously, all this goes in hand with thorough testing. After all you want to minimize the number of crashes for your users.</p>
/2013/08/debugging-icloud-URL-sharing
Debugging iCloud URL sharing
2013-08-18T16:08:28Z
2013-08-18T16:08:28Z
<p>While working with iCloud recently I ran into a situation where I wanted to share a link to an item stored inside the application’s sandbox via iCloud.</p>
<p>Luckily, since OS X 10.7 <code>NSFileManager</code> has a nice API to create a public URL to an item inside the application’s ubiquity container suitable for sharing:</p>
<pre><code class="language-objc">- (NSURL *)URLForPublishingUbiquitousItemAtURL:(NSURL *)url expirationDate:(NSDate **)outDate error:(NSError **)error;
</code></pre>
<p>The key point here is that the item has to be in the ubiquity container in order for this method to succeed. In our case, the item would be in the application sandbox. Luckily again, OS X 10.7 introduced another <code>NSFileManager</code> API that makes it easy to move an item to the ubiquity container:</p>
<pre><code class="language-objc">- (BOOL)setUbiquitous:(BOOL)flag itemAtURL:(NSURL *)url destinationURL:(NSURL *)destinationURL error:(NSError **)error;
</code></pre>
<p>Both methods are documented as synchronous which means that they will 1) block until completed 2) make sure that the result and error are available upon return. This is not an API I particularly like myself but it has the advantage of making things easy to follow, assuming that the calls are moved off the main thread to keep the UI reactive.</p>
<p>Keeping this in mind, a naive implementation could be an <code>NSOperation</code> subclass which <code>main</code> method would look as following:</p>
<pre><code class="language-objc">- (void)main
{
NSURL *mediaLocationInSandbox = ...
NSString *filename = [[[NSUUID UUID] UUIDString] stringByAppendingPathExtension:@"png"];
NSURL *mediaLocationInUbiquityContainer = [[[NSFileManager defaultManager] URLForUbiquityContainerIdentifier:nil] URLByAppendingPathComponent:filename];
NSError *setUbiquitousError = nil;
BOOL setUbiquitous = [[NSFileManager defaultManager] setUbiquitous:YES itemAtURL:mediaLocationInSandbox destinationURL:mediaLocationInUbiquityContainer error:&setUbiquitousError];
NSError *mediaLocationInCloudError = nil;
NSDate *expirationDate = nil;
NSURL *mediaLocationInCloud = [[NSFileManager defaultManager] URLForPublishingUbiquitousItemAtURL:mediaLocationInUbiquityContainer expirationDate:&expirationDate error:&mediaLocationInCloudError];
NSParameterAssert(mediaLocationInCloud != nil);
}
</code></pre>
<p>However, such an implementation would fail with <code>mediaLocationInCloudError</code> being set as</p>
<pre><code class="language-objc">Error Domain=NSCocoaErrorDomain Code=256 "The file “20ECBE26-41A5-4D79-81E3-63F5C13D1A02.png” couldn’t be opened." UserInfo=0x60000007ecc0 {NSURL=/Users/damien/Library/Mobile Documents/P97H7FTHWN~com~realmacsoftware~CloudSharing/20ECBE26-41A5-4D79-81E3-63F5C13D1A02.png, NSUnderlyingError=0x610000059170 "The operation couldn’t be completed. (LibrarianErrorDomain error 4 - The connection to the server was interrupted.)"}
</code></pre>
<p>This doesn’t look good at all. The error code 256 in the <code>NSCocoaErrorDomain</code> is <code>NSFileReadUnknownError</code> which is far from helpful in this case and the underlying error implies that things have gotten even worse underneath.</p>
<p>My first idea was to use file coordination with the <code>NSFileCoordinator</code> API to make sure that every read and write were coordinated with other readers and writers. However it is worth noting two points:</p>
<ul>
<li><code>NSFileManager</code> seems to provide a <code>NSFileCoordinator</code> if it can detect that the calls are made outside of a coordinated environment. This is not technically documented but if a custom <code>NSFilePresenter</code> is not needed file coordination should be taken care of by the file manager.</li>
<li>Using coordination doesn’t improve things and the API fails in the same way with the same error.</li>
</ul>
<p>It is however important to note that if waiting an arbitrary amount of time (sometimes between 2 to 6 seconds) the second API call succeeds. Second important point is that the second API call can arbitrarily succeed no matter the ubiquitous state of the item (i.e. whether it has been upload or not, easy to check by checking that value of the <code>NSURLUbiquitousItemIsUploadedKey</code> property on <code>NSURL</code>).</p>
<p>This smells of race condition so let’s dive into the debugger and find out!</p>
<hr />
<p><em>tl;dr -[NSFileManager setUbiquitous:itemAtURL:destinationURL:error:] is racy (or should I say “synchronous but not really”) and you should not assume the entirety of the system to be fully aware upon return that the item at the destination URL is actually a ubiquitous item. Things are even worse since making such assumption leads a critical system daemon (<code>librariand</code>) to crash 100% of the time.</em></p>
<p>Our first step will be to set a breakpoint in <code>-[NSFileManager URLForPublishingUbiquitousItemAtURL:expirationDate:error:]</code> and set where this leads us to.</p>
<h3><a href="#-nsfilemanager-urlforpublishingubiquitousitematurlexpirationdateerror" aria-hidden="true" class="anchor" id="-nsfilemanager-urlforpublishingubiquitousitematurlexpirationdateerror"></a>-[NSFileManager URLForPublishingUbiquitousItemAtURL:expirationDate:error:]</h3>
<p>After stepping over a few instructions we soon realize that there is no much going on here. A semaphore is created before calling the <code>LBGetURLForPublishedItem</code> function. This function takes a URL, a queue and a block which implies that it performs work on a background queue and invokes the block on completion on the given queue. Since a synchronous behavior is required here <code>wait</code> is called on the semaphore after calling this function and the semaphore will likely be signaled in the completion. This is a usual pattern used to make an asynchronous API behave like a synchronous one. The rest of the method deals with unpacking results from the called function and return the results to the called.</p>
<h3><a href="#lbgeturlforpublisheditem" aria-hidden="true" class="anchor" id="lbgeturlforpublisheditem"></a>LBGetURLForPublishedItem</h3>
<p>As said before, nothing exciting so let’s set a breakpoint in <code>LBGetURLForPublishedItem</code>. This functions lives in the Librarian.framework (a system private framework) and by stepping a few instructions we can notice that it quickly goes setting up an xpc connection with a daemon: <code>librariand</code>, a system daemon that lives under <code>/usr/libexec/librariand</code>.</p>
<p>Once the xpc connection with <code>com.apple.librariand</code> has been set up a message dictionary is created and two objects are inserted:</p>
<ul>
<li>An integer of value 11 under the key <code>LIBRARIAND_KEY_REQUEST</code> (<em>"Request Type"</em>)</li>
<li>A string representing the path to the item in the ubiquity container that we want to publish under the key <code>LIBRARIAND_KEY_PATH</code> (<em>"Path"</em>)</li>
</ul>
<p>Finally, the message is sent through the xpc connection by calling <code>xpc_connection_send_message_with_reply</code>. The reply parameter is a block of type <code>xpc_handler_t</code>. By using some debugging tips from a previous <a href="/2013/02/block-debugging/">post</a> we can read the memory at the block address, find the invoke function pointer in the block, disassemble it and set a breakpoint at the first instruction.</p>
<h3><a href="#__lbgeturlforpublisheditem_block_invoke" aria-hidden="true" class="anchor" id="__lbgeturlforpublisheditem_block_invoke"></a>__LBGetURLForPublishedItem_block_invoke</h3>
<p>Being typed as <code>xpc_handler_t</code> this function will take an <code>xpc_object_t</code> as second argument (remember that the first is the block itself). This object is the message reply and will usually be either a dictionary (<code>xpc_dictionary_t</code>) or an error.</p>
<p>Let’s go through the working case and the non-working one.</p>
<p>In the working case the object we get back is a dictionary as following:</p>
<pre><code class="language-objc"><OS_xpc_dictionary: dictionary[0x600000194290]: { refcnt = 1, xrefcnt = 1, count = 3, dest port = 0x0 } <dictionary: 0x600000194290> { count = 3, contents =
"URL" => <string: 0x61000007acc0> { length = 86, contents = "https://www.icloud.com/download/documents/?p=01&t=BAJKBzUxo_qdYOZe8TEBzLJ38Ttb64dorQcE" }
"Date" => <date: 0x610000058780> Sat Aug 31 12:13:16 2013 BST (approx)
"Success" => <bool: 0x7fff7dfc5920>: true
}>
</code></pre>
<p>This looks like what we were expecting.</p>
<p>In the non-working case we get an xpc object that we cannot trivially print in the debugger but that looks a lot like an error. We can confirm this by printing its type:</p>
<pre><code class="language-objc">(lldb) p (struct xpc_type_s *)xpc_get_type($rsi)
(struct xpc_type_s *) $0 = 0x00007fff72212e28
(lldb) p &_xpc_type_error
(void **) $1 = 0x00007fff72212e28
</code></pre>
<p>So yeah, we are getting an error. In oder to figure out the error description (<code>XPC_ERROR_KEY_DESCRIPTION</code>) we could use an applier block to iterate through its keys and values but we can safely assume that it’s either <code>XPC_ERROR_CONNECTION_INTERRUPTED</code> or <code>XPC_ERROR_CONNECTION_INVALID</code>. We can probably trust the connection to be valid so it is likely to be an interruption.</p>
<h3><a href="#_lbhandleserverreply" aria-hidden="true" class="anchor" id="_lbhandleserverreply"></a>_LBHandleServerReply</h3>
<p>The reply block calls immediately into another function <code>_LBHandleServerReply</code>, passing the xpc object as first argument and a reply block as second.</p>
<p>This function simply checks for the <code>LIBRARIAND_KEY_SUCCESS</code> key in the dictionary and calls into the reply block with it if successful.
If not, it unpacks the XPC error into a CF error via the function <code>_LBCreateCFErrorFromXPCError</code>:</p>
<pre><code class="language-objc">Error Domain=LibrarianErrorDomain Code=4 "The operation couldn’t be completed. (LibrarianErrorDomain error 4 - The connection to the server was interrupted.)" UserInfo=0x61000007b0c0 {NSDescription=The connection to the server was interrupted.}
</code></pre>
<p>This is exactly the underlying error we are getting from <code>NSFileManager</code>.</p>
<p>By quickly inspecting the reply block invoke function’s body we see that it checks for an eventual error, then retrieve the URL (<code>LIBRARIAND_KEY_URL</code>) and expiration date (<code>LIBRARIAND_KEY_DATE</code>) and calls back into <code>NSFileManager</code> that will return to the caller.</p>
<p>This is all there is client side. In order to find out why the connection is interrupted we will have to attach to the <code>librariand</code> process in the debugger and see what’s going on over there.</p>
<h3><a href="#librariand" aria-hidden="true" class="anchor" id="librariand"></a>librariand</h3>
<p>Having attached to <code>librariand</code>, we can simply run the client side code and see if anything obvious could explain the connection interruption.</p>
<p>And yes, there is an obvious explanation to the interruption: <code>librariand</code> crashes!</p>
<p>We get an <code>EXC_BAD_ACCESS</code> at address <code>0x0</code> inside <code>CFStringGetCString</code>. If this wasn’t clear enough, the crashing instruction shows that we are trying to dereference <code>NULL</code></p>
<pre><code class="language-nasm">movq (%r13), %rax
</code></pre>
<p>It is worth remembering the declaration of <code>CFStringGetCString</code>:</p>
<pre><code class="language-objc">Boolean CFStringGetCString(CFStringRef string, char *buffer, CFIndex bufferSize, CFStringEncoding encoding);
</code></pre>
<p>Now, remember that CoreFoundation is open-source and we can have a peak at the implementation of <code>CFStringGetCString</code> in <a href="http://www.opensource.apple.com/source/CF/CF-744.19/CFString.c">CFString.c</a>.</p>
<p>After a check for a buffer size of 0 the function asserts that the first argument is indeed a string. This assertion macro expands into the following function, implemented in <a href="http://www.opensource.apple.com/source/CF/CF-744.19/CFRuntime.c">CFRuntime.c</a></p>
<pre><code class="language-objc">__CFGenericValidateType(cf, __kCFStringTypeID)
</code></pre>
<p>The first step of this function is to validate that the type is not <code>NULL</code>.
However, by inspecting the value of <code>%rdi</code> (the first argument) in the prolog of the offending function call we can clearly see that the <code>CFStringRef</code> is <code>NULL</code>.</p>
<p>Even though this explains the crash we still need to figure out how this functions ended up being called with a <code>NULL</code> string. We can have a look at the stack trace at the moment of the crash in order to know where it has occurred.</p>
<p><img src="/static/media/images/posts/2013/08/librariand_crash_stack_trace.png" alt="Librariand stack trace at the moment of the crash" /></p>
<p>We are clearly missing function names here but thankfully LLDB is kind enough to let us set up symbolic breakpoints for these unnamed functions.</p>
<h3><a href="#___lldb_unnamed_function50librariand" aria-hidden="true" class="anchor" id="___lldb_unnamed_function50librariand"></a>___lldb_unnamed_function50$$librariand</h3>
<p>Stepping over <code>___lldb_unnamed_function50$$librariand</code>, we can’t note anything very exciting, a few registers values are moved around up to where it gets to that <code>CFStringGetCString</code>. Given the function declaration the string is expected to be in <code>%rdi</code>. Following <code>%rdi</code> back to the start of the function we can see that its value originates in <code>%rsi</code> in the function prolog, that is the second argument of the <code>___lldb_unnamed_function50$$librariand</code> function.</p>
<p>Let’s go to the stack frame above, into <code>___lldb_unnamed_function34$$librariand</code>.</p>
<h3><a href="#___lldb_unnamed_function34librariand" aria-hidden="true" class="anchor" id="___lldb_unnamed_function34librariand"></a>___lldb_unnamed_function34$$librariand</h3>
<p>As per the x86-64 calling convention, observing the registers in the prolog we can see that the first argument is an xpc connection and the second one an xpc dictionary.</p>
<p>By inspecting the xpc connection we can find out the name of the receiver on the other side <code>com.apple.librariand.peer.0x7fe84be0bad0</code>, likely to be our own process. The xpc dictionary will be the request that originated client side.</p>
<p>By stepping over a few instructions we can find the call to <code>___lldb_unnamed_function50$$librariand</code>. Tracing <code>%rsi</code> up through the instructions we can find where it value originated. In this case it is the return value of another function call to <code>UBItemCreatePublicURLForPath</code>.</p>
<p>When <code>UBItemCreatePublicURLForPath</code> returns nil, the domain, code and localized description of an error (that has probably been returned by reference) are retrieved and <code>___lldb_unnamed_function50$$librariand</code> ends up being called with <code>"handle_publish_item_request"</code> passed as first argument, the error domain as second, the error code as third and the error localized description as fourth.</p>
<p>The issue here is that the error reference is not populated by <code>UBItemCreatePublicURLForPath</code> when returning nil and we end up passing a <code>nil</code> error domain to <code>___lldb_unnamed_function50$$librariand</code> that is not handled and lead to the failed assertion in <code>CFStringGetCString</code> that we have previously observed.</p>
<p>We need to find out the reason why <code>UBItemCreatePublicURLForPath</code> does not populate the error reference when failing to creating a public URL in our particular use case.</p>
<h3><a href="#ubitemcreatepublicurlforpath" aria-hidden="true" class="anchor" id="ubitemcreatepublicurlforpath"></a>UBItemCreatePublicURLForPath</h3>
<p><code>UBItemCreatePublicURLForPath</code>, in the Ubiquity.framework, only takes one argument: a path as an <code>NSString</code>.</p>
<p>It very simply calls into <code>IPCCopyItemPublicURL</code>. This function itself returns a dictionary, containing the <code>url</code> and <code>expiration</code> if it succeeds or the <code>error</code> if it fails.
<code>UBItemCreatePublicURLForPath</code> simply unpacks these objects are return them.</p>
<p>In our case, <code>IPCCopyItemPublicURL</code> fails to return a dictionary altogether (not event one containing an error) leading <code>UBItemCreatePublicURLForPath</code> itself to never populates its error by reference.</p>
<p>We need to figure out why <code>IPCCopyItemPublicURL</code> returns <code>nil</code>.</p>
<h3><a href="#ipccopyitempublicurl" aria-hidden="true" class="anchor" id="ipccopyitempublicurl"></a>IPCCopyItemPublicURL</h3>
<p><code>IPCCopyItemPublicURL</code> takes a single argument: a path as an <code>NSString</code>. It then jumps straight to <code>IPCSendCFMessageDictionarySimple</code>.</p>
<p><code>IPCSendCFMessageDictionarySimple</code> calls into <code>IPCSendCFMessage</code>, takes its return value, makes sure it’s a dictionary and returns.</p>
<h3><a href="#ipcsendcfmessage" aria-hidden="true" class="anchor" id="ipcsendcfmessage"></a>IPCSendCFMessage</h3>
<p>This function first retrieves an <code>SRConnection</code> and calls <code>SRSendCFMessage</code> with the path string as parameter. <code>SRSendCFMessage</code> itself jumps straight to <code>SRSendCFMessageTimed</code> where the CF object is packed into an xpc object via the <code>_CFXPCCreateXPCMessageWithCFObject</code> function.</p>
<p>Eventually, <code>_SRSendMessage</code> is called with the xpc object. In it, an xpc connection is created and a semaphore is waited and signaled around the sending of the message to simulate synchronicity.</p>
<pre><code class="language-objc"><OS_xpc_connection: connection[0x7fbd43804b70]: { refcnt = 2, xrefcnt = e, name = com.apple.ubd, type = named, state = checked in, error = 0x0 mach = true, privileged = false, bssend = 0x5903, recv = 0x5703, send = 0x5a03, pid = 193, euid = 501, egid = 20, asid = 100004 } <connection: 0x7fbd43804b70> { name = com.apple.ubd, listener = false, pid = 193, euid = 501, egid = 20, asid = 100004 }>
</code></pre>
<p>From the name of the connection, we can find out the process on the other side of the connection: in this case it is <code>ubd</code>, a daemon that lives under <code>/System/Library/PrivateFrameworks/Ubiquity.framework/Versions/A/Support/ubd</code>.</p>
<p>The message dictionary sent through the connection is as following:</p>
<pre><code class="language-objc"><OS_xpc_dictionary: dictionary[0x7fbd43902bc0]: { refcnt = 1, xrefcnt = 2, count = 2, dest port = 0x0 } <dictionary: 0x7fbd43902bc0> { count = 2, contents =
"ECF19A18-7AA6-4141-B4DC-A2E5123B2B5C" => <data: 0x7fbd43901210>: { length = 4096 bytes, contents = 0x62706c697374313513940000000000008012000000007f10... }
"type" => <uint64: 0x7fbd41d09800>: 1073741866
}>
</code></pre>
<p>The <code>contents</code> is the path string and the <code>type</code> will be of interest further on.</p>
<p>Next, we can set a breakpoint in the connection reply handler block and observe the reply message content.</p>
<p>The reply dictionary in the working case looks like following:</p>
<pre><code class="language-objc"><OS_xpc_dictionary: dictionary[0x7fd900610910]: { refcnt = 1, xrefcnt = 1, count = 2, dest port = 0x0 } <dictionary: 0x7fd900610910> { count = 2, contents =
"contents" => <dictionary: 0x7fd900513070> { count = 1, contents =
"ECF19A18-7AA6-4141-B4DC-A2E5123B2B5C" => <data: 0x7fd900516980>: { length = 4096 bytes, contents = 0x62706c69737431351388000000000000801200000000d273... }
}
"result" => <int64: 0x7fd900513520>: 0
}>
</code></pre>
<p>whereas in the case that leads to a crash of <code>librariand</code>:</p>
<pre><code class="language-objc"><OS_xpc_dictionary: dictionary[0x7fd9005168d0]: { refcnt = 1, xrefcnt = 1, count = 1, dest port = 0x0 } <dictionary: 0x7fd9005168d0> { count = 1, contents =
"result" => <int64: 0x7fd90040c050>: 0
}>
</code></pre>
<p>Clearly, the reply returns the same results but the <code>contents</code> is missing in the second case.</p>
<p>Following, the semaphore is signaled at the end of the reply handler block, giving execution back to <code>_SRSendMessage</code>.</p>
<p>The rest of the instructions dispose of a few objects and the function eventually returns. We are then back in <code>SRSendCFMessage</code> and eventually in <code>IPCSendCFMessage</code> without any <code>nil</code> check on the way.</p>
<p>Whatever happens on <code>ubd</code> side, it is not expected to not populate the dictionary with an object for the <code>contents</code> key.</p>
<p>We will have to attach the debugger to <code>ubd</code> and find out what is going wrong over there.</p>
<h3><a href="#ubd" aria-hidden="true" class="anchor" id="ubd"></a>ubd</h3>
<p><code>ubd</code> itself doesn’t crash so it’s not as easy to find an instruction to break on as it was with <code>librariand</code>.</p>
<p>We will then have to apply the good old “guess and hope for some luck” approach. We do know that we are messaging <code>ubd</code> through an xpc connection so whatever it does, it will have to use some xpc functions to message us back. A good start would be to set a breakpoint on <code>xpc_create_dictionary</code> and look at the name of the functions in the backtrace when the breakpoint is hit.</p>
<p>Quite luckily, one function that could be a good candidate stands out: <code>_SRTransportHandleRequest</code>. We can then set a breakpoint on it and inspect the request coming through. Not surprisingly there are a lot! Thus, we need a way to skim off the requests we are not interested in. Remembering the dictionary that was sent through the connection from <code>librariand</code> we can update the breakpoint with a condition that the type in the request dictionary be <code>1073741866</code>.</p>
<h3><a href="#srtransporthandlerequest" aria-hidden="true" class="anchor" id="srtransporthandlerequest"></a>SRTransportHandleRequest</h3>
<p>With <code>ubd</code> attached and our conditional breakpoint set up we will only break in the debugger when the appropriate message from <code>librariand</code> is received.</p>
<p>Upon receiving the request from <code>librariand</code>, the xpc message is unpacked and a call to yet another unnamed function is made: <code>___lldb_unnamed_function149$$ubd</code>. This function is called with an <code>SRConnection</code> and the path string. This function returns a dictionary on the stack.</p>
<p>In order to better understand the return value of such function we will take 3 cases:</p>
<ul>
<li>An URL is successfully created for the ubiquitous item and replied back to <code>librariand</code>.</li>
<li>A URL cannot be created and an error is sent back to <code>librariand</code>. We will simulate this by turning off any network connectivity making sure a URL cannot be created for a legit reason.</li>
<li>A URL cannot be created and no error is sent back to <code>librariand</code>. This is the race we’ve been observing.</li>
</ul>
<p>In order to test the first case we will simply stick a <code>sleep(10)</code> between the calls to <code>-[NSFileManager setUbiquitous:itemAtURL:destinationURL:error:]</code> and <code>-[NSFileManager URLForPublishingUbiquitousItemAtURL:expirationDate:error:]</code>.</p>
<p>In the first case, the returned dictionary is as expected:</p>
<pre><code class="language-objc">{
expiration = 399743378;
url = "https://www.icloud.com/download/documents/?p=01&t=BAJ_K9mgDi2YrwWRjFAB5dQwLqaZAKSuPwvE";
}
</code></pre>
<p>Following, an xpc object is created from the dictionary and added to the reply dictionary as <code>contents</code>. Finally, the message is sent back through the xpc connection and everyone’s happy.</p>
<p>In the second case, the returned dictionary is also as expected:</p>
<pre><code class="language-objc">{
error = {
code = 2;
domain = kCFErrorDomainCFNetwork;
kCFGetAddrInfoFailureKey = 8;
};
}
</code></pre>
<p>Just as the first case, the dictionary containing the error is packed up as <code>contents</code> in the reply dictionary and sent to <code>librariand</code> through the xpc connection.
Note that the useful network error ends up being wrapped by <code>Foundation</code> into a less than useful one. This is quite unfortunate.</p>
<pre><code class="language-objc">Error Domain=NSCocoaErrorDomain Code=256 "The file “4AF039A7-FD55-496C-BD80-04C865361AC1.png” couldn’t be opened." UserInfo=0x60000007ac40 {NSURL=/Users/damien/Library/Mobile Documents/P97H7FTHWN~com~realmacsoftware~CloudSharing/4AF039A7-FD55-496C-BD80-04C865361AC1.png, NSUnderlyingError=0x618000245be0 "The operation couldn’t be completed. (kCFErrorDomainCFNetwork error 2 - The operation couldn’t be completed. (kCFErrorDomainCFNetwork error 2.))"}
</code></pre>
<p>In the third case however, no dictionary is returned at all. This leads to the reply dictionary not containing any <code>contents</code> and <code>librariand</code> receiving the unexpected response we observed earlier.</p>
<p>Since we would expect <code>___lldb_unnamed_function149$$ubd</code> to return a dictionary containing an error in any failing case, let’s set a breakpoint on it and try to follow back to the source of the problem.</p>
<h3><a href="#___lldb_unnamed_function149ubd" aria-hidden="true" class="anchor" id="___lldb_unnamed_function149ubd"></a>___lldb_unnamed_function149$$ubd</h3>
<p><code>___lldb_unnamed_function149$$ubd</code> also takes an <code>SRConnection</code> and a path string and quickly calls into <code>___lldb_unnamed_function1010$$ubd</code> which itself returns the <code>url</code>, <code>expiration</code> and <code>error</code>.</p>
<p>In the first case described before, a <code>url</code> and <code>expiration</code> are returned, in the second case an <code>error</code> is returned whereas the third case does not return a <code>url</code> but also lacks an <code>error</code>.</p>
<h3><a href="#___lldb_unnamed_function1010ubd" aria-hidden="true" class="anchor" id="___lldb_unnamed_function1010ubd"></a>___lldb_unnamed_function1010$$ubd</h3>
<p>This function seems to do much of the actual work in publishing the item URL. A bunch of properties are first retrieved in order to create an HTTP request such as the server URL, an AuthToken. Assuming all goes smoothly it will end up creating a POST HTTP request, creating a read stream and schedule the request on a runloop. The response is then read, parsed, checked for errors and the content is returned to the caller.</p>
<p>The creation of the HTTP request is conditioned by the return value of yet another function: <code>___lldb_unnamed_function186$$ubd</code>. This function is expected to retrieve a dictionary of properties for an item at a given ubiquity path.</p>
<pre><code class="language-objc">{
Checksum = 011c490058dcc81d6fef74f5dc1f3a4dcbeb73f39e;
IsPackageRoot = 0;
ItemName = "30610207-C807-4BB9-86F7-E0D048EF033D.png";
ItemSize = 992318;
ItemStatus = 1051648;
LastEditor = "MacBook Pro Damien";
ModTime = 1376642747;
ModeBits = 33188;
TotalBytesToDownload = 992318;
}
</code></pre>
<p>Based on the state of the network connection, this dictionary might contain some additional upload/download error.</p>
<p>In the broken third case described above, no dictionary is returned at all. We can quickly examine <code>___lldb_unnamed_function186$$ubd</code> and find out why such is happening.</p>
<h3><a href="#___lldb_unnamed_function186ubd" aria-hidden="true" class="anchor" id="___lldb_unnamed_function186ubd"></a>___lldb_unnamed_function186$$ubd</h3>
<p>This function seems to perform a few sanity checks on the given path. The different branching in the code path between the working and broken case seems to occur based on the return value of yet another function: <code>___lldb_unnamed_function371$$ubd</code>.</p>
<p>I can’t say for sure what is going on in this function but it seems that it attempts to retrieve an item from a sqlite database. Two SQL statements stand out in particular:</p>
<pre><code class="language-sql">select item_id from shadow_table where parent_id = ? and (local_filename = ? or filename = ?) and ((state & ?) == 0);
select item_id from item_table where parent_id = ? and (local_filename = ? or filename = ?) and ((state & ?) == 0);
</code></pre>
<p>Again, I’m not sure what precisely these statements are attempting to retrieve nor the identity of the database they are querying. However it is obvious that, in the broken case, an ubiquity item cannot be found for the given path.</p>
<p>I think we can conclude there is a race condition between <code>-[NSFileManager setUbiquitous:itemAtURL:destinationURL:error:]</code> and <code>-[NSFileManager URLForPublishingUbiquitousItemAtURL:expirationDate:error:]</code>. The first method does make the item ubiquitous returning YES synchronously when succeeding. However upon return we cannot assume that the item at this path has been fully registered by <code>ubd</code>.</p>
<p>Again, these are guesses but I believe that the item will be fully registered <code>ubd</code> only after attempting (not succeeding) to upload it. We have no idea about the process <code>ubd</code> follows when attempting to upload items nor the delay between the registration of an item as ubiquitous and the attempt to upload it to the iCloud server. With that in mind, the best advice would be to track the upload status of the ubiquity item by the mean of an <code>NSMetadataQuery</code> and retrieve a URL for publishing only after the item has been fully uploaded to the iCloud server (by tracking the <code>NSMetadataUbiquitousItemIsUploadedKey</code> and <code>NSMetadataUbiquitousItemIsUploadingKey</code> attributes).</p>
<hr />
<p>So, to summarize, we set a breakpoint in <code>-[NSFileManager URLForPublishingUbiquitousItemAtURL:expirationDate:error:]</code> and step over instructions through the Librarian and Ubiquity frameworks. This lead us to attaching to the <code>librariand</code> process that we found was crashing because of a non-populated error for an API that failed to return a result. Attempting to trace the issue this lead us to attaching to the <code>ubd</code> process and figuring out that a registered item couldn’t be retrieved by <code>ubd</code> for the ubiquitous path.</p>
<p>Here’s the list of breakpoints that were added in order to debug this.</p>
<p><img src="/static/media/images/posts/2013/08/NSFileManager_ubd_librarian_breakpoints.png" alt="Breakpoints" /></p>
<p>This was a fun debugging journey, just a bit longer than I expected!</p>
<p>I have obviously filed a radar about this: <a href="rdar://problem/14772373">rdar://problem/14772373</a>.</p>
/2013/08/the-podcasts-i-listen-to
The podcasts I listen to
2013-08-11T12:29:00Z
2013-08-11T12:29:00Z
<p>I like listening to podcasts. I find it relaxing and it’s often a nicer way to learn about stuff than by reading an article. I also like the fact that it gives a more human touch to a bunch of people I follow on the Internet.</p>
<p>Following is the list of podcasts I currently have in my podcast app (I use <a href="http://vemedio.com/products/instacast-mac">Instacast</a>, an app that I couldn’t recommend enough).</p>
<h2><a href="#the-talk-show-with-john-gruber" aria-hidden="true" class="anchor" id="the-talk-show-with-john-gruber"></a><a href="http://www.muleradio.net/thetalkshow/">The Talk Show with John Gruber</a></h2>
<p>There’s no need to introduce <a href="http://daringfireball.net">Mr. John Gruber</a>. His podcast is a very good way to stay up to date with everything surrounding the Apple world. Gruber always has very interesting guests and while the podcast can sometimes be a tad too long it’s always great to listen.</p>
<h2><a href="#accidental-tech-podcast" aria-hidden="true" class="anchor" id="accidental-tech-podcast"></a><a href="http://atp.fm">Accidental Tech Podcast</a></h2>
<p>I won’t lie, the only reason I listen to this podcast is <a href="http://hypercritical.co">John Siracusa</a>. I miss the podcast he used to host on 5by5 <a href="http://5by5.tv/hypercritical">Hypercritical</a>. John is a great technical <a href="http://arstechnica.com/author/john-siracusa/">writer</a> and is very opinionated about everything OS X related. A great podcast.</p>
<h2><a href="#the-prompt" aria-hidden="true" class="anchor" id="the-prompt"></a><a href="http://5by5.tv/prompt">The Prompt</a></h2>
<p>A fairly recent addition to the 5by5 family of podcasts, hosted by <a href="http://ticci.org">Federico Viticci</a>, <a href="http://mykehurley.net">Myke Hurley</a> and <a href="http://512pixels.net">Stephen Hackett</a>. I particularly enjoy listening to Federico since I happen to share most of his cultural traits, having grown up in the same environment. It’s nice to listen about technology by someone that is not from the Anglo-Saxon world.</p>
<h2><a href="#debug" aria-hidden="true" class="anchor" id="debug"></a><a href="https://itunes.apple.com/us/podcast/debug/id578812394">Debug</a></h2>
<p>Hosted by <a href="http://kickingbear.com">Guy English</a> and <a href="http://www.imore.com/users/rene%20ritchie">Rene Ritchie</a>, Debug always has amazing guests. Depending on the guest, some episode can get a bit technical which I particularly enjoy.</p>
<h2><a href="#bitsplitting" aria-hidden="true" class="anchor" id="bitsplitting"></a><a href="http://bitsplitting.org/podcast/">Bitsplitting</a></h2>
<p>Hosted by <a href="http://bitsplitting.org">Daniel Jalkut</a> from <a href="http://www.red-sweater.com">Red Sweater</a>, Bitsplitting features well-known people from the tech industry. The podcast usually focuses on more personal traits that have affected the life of the interviewee. A top quality podcast.</p>
<h2><a href="#core-intuition" aria-hidden="true" class="anchor" id="core-intuition"></a><a href="http://www.coreint.org">Core Intuition</a></h2>
<p>Hosted by <a href="http://www.manton.org">Manton Reece</a> and <a href="http://bitsplitting.org">Daniel Jalkut</a>, Core Intuition is a fun podcast discussing (Apple) developer related stuff.</p>
<h2><a href="#edge-cases" aria-hidden="true" class="anchor" id="edge-cases"></a><a href="http://edgecasesshow.com">Edge Cases</a></h2>
<p>Hosted by <a href="http://subjectiveobserver.wordpress.com">Andrew Pontious</a> and <a href="http://rentzsch.com">Wolf Rentzsch</a>, Edge Cases is a podcast about Apple-related software development. One of the few (actually) technical podcast about software development, it’s always a great listen!</p>
<h2><a href="#postmodem" aria-hidden="true" class="anchor" id="postmodem"></a><a href="http://postmodemcast.com">Postmodem</a></h2>
<p>Hosted by <a href="https://twitter.com/cbarrett">Colin Barrett</a>, <a href="https://twitter.com/importantshock">Patrick Thomson</a>, and <a href="https://twitter.com/pbowden">Phillip Bowden</a>. I couldn’t really find words to describe Postmodem so just give it a listen (P.S. it’s amazing).</p>
<h2><a href="#real-talk-podcast" aria-hidden="true" class="anchor" id="real-talk-podcast"></a><a href="http://realtalk.io">Real Talk Podcast</a></h2>
<p>Hosted by <a href="http://jamesgolick.com">James Golick</a> and <a href="http://timetobleed.com">Joe Damato</a>, it’s describe as “The only technical podcast, highly technical and unedited”. A great show if you like debugging and low-level stuff.</p>
<h2><a href="#developing-perspective" aria-hidden="true" class="anchor" id="developing-perspective"></a><a href="http://developingperspective.com">Developing Perspective</a></h2>
<p>Hosted by <a href="http://david-smith.org">David Smith</a>, Developing is a very short podcast (under 15 minutes) discussing iOS development. Highly recommended.</p>
<h2><a href="#new-yorker-fiction" aria-hidden="true" class="anchor" id="new-yorker-fiction"></a><a href="http://www.newyorker.com/online/blogs/books/podcast">New Yorker: Fiction</a></h2>
<p>The podcasting facet of the Books blog from the New Yorker, Fiction discusses a short story, preceded by an actual reading. Top quality, as you would expect from the New Yorker.</p>
<h2><a href="#just-the-tip" aria-hidden="true" class="anchor" id="just-the-tip"></a><a href="http://justthetip.fm">Just the Tip</a></h2>
<p>Hosted by <a href="https://twitter.com/AmyJane">Amy Jane Gruber</a> and <a href="http://onefoottsunami.com">Paul Kafasis</a>, Just the Tip is a fun and relaxing podcast. While fairly recent, I am already looking forward to it every week!</p>
<h2><a href="#alza-il-volume" aria-hidden="true" class="anchor" id="alza-il-volume"></a><a href="http://www.alzailvolume.rai.it">Alza il Volume</a></h2>
<p>(In Italian) Alza il Volume is a very good podcast discussing everything music related. I’ve discovered plenty of new bands and genre through it. If you speak Italian, give it a listen!</p>