LC-Asia: An Android upstreaming update

Many people have talked about the Android kernel code and its relation to the mainline. One of the people who has done the most to help bring Android and the mainline closer together is John Stultz; at the 2013 Linaro Connect Asia event, he talked about the status of the Android code. The picture that emerged shows that a lot of progress has been made, but there is still a lot of work yet to be done.

What's out there

John started by reviewing the existing Android kernel patches by category, starting with the core code: the binder interprocess communication mechanism, the ashmem shared memory mechanism, the Android logger, and monotonic event timestamps. The timestamp patch is needed to get timestamps from the monotonic clock for input events; otherwise it is hard to be sure of the timing between events, which makes gesture recognition hard. The problem is that these events cannot be added without breaking the kernel's ABI, so they cannot be just merged without further consideration.

There is a set of changes that John categorized as performance and power-consumption improvements. At the top of the list is the infamous "wakelock" mechanism, used by Android to know when the system as a whole can be suspended to save power. There is a special alarm device that can generate alarms that will wake the system from a suspended state. The Android low-memory killer gets rid of tasks when memory gets tight; it is designed to activate more quickly than the kernel's out-of-memory killer, which will not act until a memory shortage is seriously affecting system performance. Also in this category is the interactive CPU frequency governor, which immediately ramps the CPU up to its maximum speed in response to touch events; its purpose is to help the system provide the fastest response possible to user actions.

The "debugging features" category includes a USB gadget driver that supports communication with the adb debugging tools; it is also used to support file transfer using the media transfer protocol (MTP). The FIQ debugger is a low-level kernel debugger with some unique features — communication through the device's headphone jack being one of them. The RAM console will save kernel messages for later recovery in case of a crash. There is the "key-reset" driver, a kind of "control-alt-delete for phones." The patches to the ARM architecture's "embedded trace macrocell" and "embedded trace buffer" drivers offer improved logging of messages from peripheral processors. Then there is the "goldfish" emulator, derived from QEMU, which allows Android to be run in an emulated mode on a desktop system.

The list of networking features starts with the "paranoid networking framework," the mechanism that controls which applications have access to the network; it restricts that access to members of a specific group. There is a set of netfilter changes mostly aimed at providing better accounting for which applications are using data. There are some Bluetooth improvements and the Broadcom "bcmhd" WiFi driver.

In the graphics category is the ION memory allocator, which handles DMA buffer management. The "sync" driver provides a sort of mutex allowing applications to wait for a vertical refresh cycle. There is also a miscellaneous category that includes the battery meta-driver, which provides wakelock support and thermal management. That category contains various touch screen drivers, the "switch" class for dealing with physical switches, and the timed GPIO facility as well. Finally, the list of deprecated features includes the PMEM memory allocator, the early suspend mechanism, the "apanic" driver, and the yaffs2 filesystem, which has been replaced by ext4.

Upstreaming status

Having passed over the long list of Android patches, John moved on to discuss where each stands with regard to upstreaming. The good news is that some of these features are already upstream. Wakelocks are, arguably, the most important of those; Rafael Wysocki's opportunistic suspend work, combined with a user-space emulation library, has made it possible for Android to move over to a mainline-based solution. John's monotonic event timestamp patches are also in the mainline, controlled by a special ioctl() command to avoid breaking the ABI; Android is using this mechanism as of the 4.2 ("Jelly Bean") release. The RAM console functionality is available via the pstore mechanism. The switch class is now supported via the kernel's "extconn" driver, but Android is not yet using this functionality.

A number of the Android patches are currently in the staging tree. These include the binder, ashmem, the logger, the low-memory killer, the alarm device, the gadget device, and the timed GPIO feature. The sync driver was also just pulled into the staging tree for merging in the 3.10 development cycle. With all of the staging code, John said, Android "just works" on a mainline kernel.

That does not mean that the job is done, though; quite a few Android patches are still in need of more work to get upstream. One such patch is the FIQ debugger; work is being done to integrate it with the kdb debugger, but, among other problems, the developers are having a hard time getting review attention for their patches. The key-reset driver was partially merged for the 3.9 kernel, but there are a number of details to be dealt with still. The plan for the low-memory killer is to integrate it with the mempressure control group patch and use the low-memory notification interface that is part of that mechanism; the developers hope to merge that code sometime soon. Ashmem is to be reimplemented via one of the volatile ranges patch sets, but there is still no agreement on the right direction for this feature. Much of the goldfish code has been merged for the 3.9 release.

The ION memory allocator has not yet been submitted for consideration at all. Much of this code duplicates what has been done with the CMA allocator and the DMA buffer sharing mechanism; integrating everything could be a challenge. There should be pieces that can be carved out and submitted, John said, even if the whole thing requires more work.

The interactive CPU frequency driver has been rejected by the scheduler developers in its current form. Supporting this feature properly could require some significant reworking of the scheduler code.

The netfilter changes have been submitted for inclusion, but there is some cleanup required before they can be merged. The paranoid networking code, instead, is not appropriate for upstream and will not be submitted. The right solution here would appear to be for Android to use the network namespaces feature, but that would require some big changes on the Android side, so it is not clear when it might happen.

The alarm device code needs to be integrated with the kernel's timerfd subsystem. Much of that integration has been done, but it requires an Android interface change, which is slowing things down. The embedded trace driver changes have been submitted, but the developer who did that work has moved on, so the code is now unmaintained. It is also undocumented and nobody else fully understands it at this point. There is a desire to replace the Android gadget driver with the CCG ("configurable composite gadget") code that is currently in the staging tree, but CCG does not yet do everything that Android needs, and it appears to be unmaintained as well. There was talk in the session of Linaro possibly taking over the development of that driver in the future.

Finally, it would be good to get the binder and logger patches out of the staging tree. That, however, is "complicated stuff" and may take a while. There is hope that the upcoming patches to support D-Bus-like communication mechanisms in the kernel will be useful to provide binder-like functionality as well.

There are a few issues needing longer-term thought. The integration of the sync driver and the DMA buffer sharing mechanism is being thought through now; there are a lot of details to be worked out. The upstreaming of ION could bring its own challenges. Much of that code has superficial similarities to the GEM and TTM memory managers that already exist in the kernel. Figuring out how to merge the interactive CPU frequency driver is going to be hard, even before one gets into details like how it plays with the ongoing big.LITTLE initiative. Some fundamental scheduler changes will be needed, but it's not clear who is going to do this work. The fact that Google continues to evolve its CPU frequency driver is not helping in this regard. There will, in other words, be plenty to keep developers busy for some time.

Concluding remarks

In total, John said, there are 361 Android patches for the kernel, with the gadget driver being the largest single chunk. Some of these patches are quite old; one of the patches actually predates Android itself. Google is not standing still; there is new code joining that which has been around for a while. Current areas of intensive development include ION, the sync driver, the CPU frequency driver, the battery driver, and the netfilter code. While some of the code is going into the mainline, the new code adds to the pile of out-of-tree patches shipped by the Android project.

Why should we worry about this, John asked, when it really is just another one of many forks of the kernel? Forking is how development gets done; see, for example, the development of the realtime patches or how many filesystems are written. But, he said, forks of entire communities, where code does not get merged back, are more problematic. In this case, we are seeing a lot of ARM systems-on-chip being developed with Android in mind from the beginning, leading to an increase in the use of out-of-tree drivers and kernels. Getting the Android base into the mainline makes it easier for developers to work with, and makes it easier to integrate Android-related code developed by others. John would like Android developers to see the mainline kernel, rather than the Android world, as their community.

Things are getting better; Zach Pfeffer pointed out that the work being done to bring Android functionality into the mainline kernel is, indeed, being used by the Android team. The relationship between that team and the kernel development community is getting better in general. It is a good time for people who are interested to join the effort and help get things done.

[Your editor would like to thank Linaro for travel assistance to attend this event.]

---

(Log in to post comments)