In this development log, we continue our previous work on reproducible Cwtch bindings, uncovering the final few sources of variation between our Repliqate scripts and our docker/drone builds, leading to fully reproducible builds for Cwtch Android bindings!
Changes Necessary for Reproducible Android Bindings
After a thorough investigation of the build artifacts produced by Repliqate and Drone we uncovered three additional sources of variation:
- Insufficient path stripping introduced by Android NDK tools - it turns out that Android builds using NDK versions below 22 are not reproducible as they produce randomized artifacts (through unstripped temporary directory paths appearing in compiled binares). NDK 22 changed the binutils and default linker to versions that correctly strip such paths from build artifacts. As such it was necessary for us to update the NDK version we used. We chose the technically outdated NDK 22 rather than the more modern NDK 25 to minimize Android OS compatibility changes during this switch. However, per our long term support plan, we will be moving towards adopting the latest NDK in the future.
- Paths in DWARF entries - while we have been unable to track down exactly where these are being introduced, we did track the final difference in the produced bindings to DWARF debug lines embedded in compiled ELF binaries. These entries encoded the actual location of the NDK on the disk of the build machine, instead of the symbolic link that we believed should have been followed. By physically placing the NDK at same location in repliqate as in our Docker container we were able to get these entries to be consistent - however there is still work to do to understand exactly why they are being introduced at all.
readelf --debug-dump=line) DWARF debug section of Drone-produced Android bindings v.s. Repliqate-produced. The difference in paths is highlighted.- Go Compiler Acquisition - our Docker container was compiling the Go compiler from source, while Repliqate was downloading a pre-compiled version. During debugging we changed the Dockerfile to also download the pre-compiled version in order to eliminate the difference as a potential reproducibility issue. Our tests indicated that there was a difference between artifacts produced by the precompiled compiler v.s. one built from source - this is likely explained by introduced environmental differences caused by the compilation of the compiler itself e.g. the contents/versions of modules in the Go package cache which we have seen as having an impact on other produced binaries.
Repliqate Scripts
With those issues now fixed, Cwtch Android bindings are officially reproducible! The first version that officially met this requirement was 1.10.5, and you can find the Repliqate script under cwtch-bindings-v1.10.5/libcwtch.v1.10.5-android.script in the Cwtch Repliqate scripts repository.
This is another big milestone towards our ultimate goal of full reproducibility for Cwtch releases.
Help us go further!
We couldn't do what we do without all the wonderful community support we get, from one-off donations to recurring support via Patreon.
If you want to see us move faster on some of these goals and are in a position to, please donate. If you happen to be at a company that wants to do more for the community and this aligns, please consider donating or sponsoring a developer.
Donations of $5 or more can opt to receive stickers as a thank-you gift!
For more information about donating to Open Privacy and claiming a thank you gift please visit the Open Privacy Donate page.
