Securing Firefox

Recently, we decided to try two new code analysis techniques for the Mozilla code base, the memory error detector “Address Sanitizer (ASan)” andĀ  a static analysis tool, the “Clang Static Analyzer.” While both tools are very different (especially how they work), they also have something in common: Both use the LLVM framework (and the C/C++ frontend Clang). These tools are only two examples of the wide range of possible applications of LLVM in security and we encourage everyone interested in these topics to have a look at it.

Address Sanitizer (ASan)

ASan is a memory error detector, designed to discover invalid memory reads/writes (both on the stack and the heap), e.g. stack/heap overflows, out-of-bounds access or invalid/use-after free problems. These goals are very similar to those of Valgrind’s MemCheck. However, because ASan utilizesĀ compile-time code instrumentation rather than a runtime emulation approach, programs can run much faster under ASan compared to Valgrind. Furthermore, Valgrind’s Memcheck is not designed to detect stack-based memory problems (there is however an experimental tool called “sgcheck” for Valgrind that is supposed to detect these).

During code compilation, ASan adds code to all memory accesses which checks if the memory being read or written is valid. In order to be able to tell if memory is valid, it stores the state in so called “shadow memory” in a very efficient way. By hooking the malloc and free functions at runtime, ASan can then keep track of allocated memory. Additionally, the instrumentation adds “red zones” around stack variables, in order to detect stack-based overflows. More technical details about how ASan works can be found in their wiki.

Since the Chrome Security Team has found quite a few security issues so far by using ASan, it seemed likely that it would help us finding issues within Firefox as well. Especially for the purpose of fuzzing and running automated tests we assumed it could be beneficial to use a very performant memory checking tool. Additionally, we figured that there might be stack-based memory problems that Valgrind previously missed.

After working with our build system, sorting out some issues with compiler and linker flags not being propagated correctly, problems with shared objects that led us to patching Clang, we produced working Firefox builds with ASan. It should be noted that much of this complexity is caused by our build system rather than ASan itself, so it’s possible that other projects are a lot easier to adapt. To date, we have identified 4 bugs through our use of ASan. In addition, another bug was reported by Atte Kettunen of OUSPG. It should be noted that these bugs are only the result of experimental testing, as we have not integrated ASan into our testing infrastructure yet.

Due to several bugs being identified so far (despite the fact that we’re using a lot of automated testing, even with Valgrind), we have reason to believe that ASan could be a valuable addition to the Firefox testing process. We also think that the huge performance gain for builds using ASan, combined with the guarantee that certain classes of bugs that Valgrind finds are also detected by ASan as well, will be helpful in certain situations: Imagine a special nightly/debug branch of Firefox for developers, that detects memory errors while running but is still fast enough for regular browsing. Another major field is fuzz testing, where speed is certainly an important factor.

You can search for all ASan bugs in Bugzilla using this search query. (Note that some bugs are currently not visible due to them being security-related since they involve memory corruption.)

We’d like to encourage other security researchers to try Firefox built with ASan and report bugs. For that purpose, we now have a manual for building Firefox with ASan. However, as following that manual requires quite some time and effort , we also decided provided some unofficial builds of Firefox with Address Sanitizer so you can get an initial impression without too much effort. Please note that these builds are experimental and you will not get the normal Firefox updates. Don’t use them in a production environment. Also note that the executable files are larger than normal due to being not stripped (so you can get a symbolized trace for an ASan crash). Especially with the optimized builds, you will notice that browsing is still possible at a decent speed (even faster than regular debug builds sometimes ! ).