Skip to main content

Making Cwtch Bindings Reproducible

· 8 min read
Sarah Jamie Lewis

From the start of the Cwtch project, the source code for all components making up Cwtch has been freely available for anyone to inspect, use, and modify.

But open source code is only one defense against malicious actors who might seek to undermine your privacy and security. This is why, as part of our ongoing Cwtch Stable work, we are working towards making all parts of the Cwtch chain reproducible and verifiable.

The whole point of reproducible builds is that you no longer have to trust binaries provided by the Cwtch Team because you can independently verify that the binaries we release are built from the Cwtch source code.

In this devlog we will talk about how Cwtch bindings are currently built, the changes we have made to Cwtch bindings to make future distributions verifiable, and the next steps we will be taking to make all Cwtch builds reproducible. This will be useful to anyone who is looking to reproduce Cwtch bindings specifically, and to anyone who wants to start implementing reproducible builds in their own project.

How Cwtch Bindings are Built​

Since we launched Cwtch Beta we have used Docker containers as part of our continuous build process.

When a new change is merged into the repository it kicks off the Cwtch bindings build pipeline which result in the new source tree being downloaded, inspected, compiled, tested, and eventually packaged for different platforms.

The Cwtch Bindings build pipeline results in four compiled libraries:

These compiled libraries eventually make their way into Cwtch-based applications, like the Cwtch UI.

Making libCwtch Reproducible​

Docker containers alone aren't enough to guarantee reproducibility. On inspection of several builds of the same source tree, we noticed a few elements that were distinct to each build:

  • Go Build ID: By default, Go includes a build ID as part of compiled binaries. When using CGO this build ID is non-deterministic and differs for every build. We made the decision to override this build ID for all outputs, setting it to the version of the code being built.
  • Build Paths and Go Environment Variables: By default, Go includes full filesystem paths, and many Go-specific environment variables in the compiled binary – ostensibly to aid with debugging. These can be removed using the trimPath option, which we now specify for all bindings builds.

Linux Specific Considerations​

After the general fixes for Go builds are applied, the main variable input that impacts reproducibility is the version of libc that the bindings are compiled against.

Our Drone/Docker build environments are based on Debian Bullseye which provides libc6-dev version 2.31. Other development setups will likely link libc-dev 2.34+.

libc6-dev 2.34 is notable because it removed dependencies on libpthread and libdl – neither are used in libCwtch, but they are currently referenced – which increases the number of sections (and thus the virtual addresses of those sections) defined in the produced ELF file.

This means that in order to reproduce libCwtch Linux bindings it is necessary to have a development environment that will link libc 2.31. We have provided a small, standalone environment which can be used for this purpose (see the section on Next Steps for more information).

Windows Specific Considerations​

The headers of PE files technically contain a timestamp field. In recent years an effort has been made to use this field for other purposes, but by default go build will still include the timestamp of the file when producing a DLL file (at least when using CGO).

Fortunately this field can be zeroed out through passing -Xlinker –no-insert-timestamp into the mingw32-gcc process.

With that, and the universal Go fixes outlined above, Windows bindings are now reproducible using the same standalone Linux environment.

Android Specific Considerations​

With the above universal Go fixes, Android build artifacts become almost repeatable. And on certain setups they appear to be reproducible. However,achieving full reproducibility for Android builds requires a number of specific environment dependencies, and considerations:

  • Cwtch makes use of GoMobile for compiling Android libraries. We pin to a specific version 43a0384520996c8376bfb8637390f12b44773e65 in our Docker containers. Unlike go build, the trimpPath parameter passed to GoMobile does not strip all development environment paths. This means that the build environment needs consistent directory structures. We have noticed inconsistencies in the detail stripped between setups e.g. cwtch.aar files build by our Docker and Repliqate builds still contain randomized /tmp/go-build* references that developer builds do not. We are still in the process of tracking down how these inconsistencies are introduced.
  • We still use sdk-tools instead of the new commandline-tools. The latest version of sdk-tools is 4333796 and available from: https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip. As part of our plans for Cwtch Stable we will be updating this dependency.
  • Cwtch Android builds currently use OpenJDK 8, unchanged from the earliest prototypes when Android development required Java 8. There is no nice way of obtaining this JDK version anymore, our Docker Containers are based on the now deprecated openjdk:8 image. As with sdk-tooks, as part of our plans for Cwtch Stable we will be updating this dependency.

All of the above mean that we cannot consider Android builds to be reproducible yet, but we believe this is an achievable goal within the next couple of release cycles.

OSX Specific Considerations​

Perhaps surprisingly, OSX builds present the biggest reproducibility challenge. Unlike Linux, Windows, and Android builds we do not have a Dockerized build environment for OSX builds - relying instead on a dedicated machine to perform the builds.

As with Linux above, the general fixes for setting Go build id and trimming paths are enough to ensure repeatability on the same machine.

In order to fully guarantee reproducibility, OSX libraries need to be built on the same version of OSX with the same version of Xcode. For reference our current build system uses: Big Sur 11.6.1 with Xcode version 13.2.1.

In an ideal world we would be able to cross-compile OSX libraries on Linux the same way we do for Windows and Android. While there are no technical limits, compiling for OSX is dependent on a proprietary SDK. There is no way to trustfully obtain this SDK from anyone except Apple, and the license appears to strictly prohibit transferring the SDK to non-Apple hardware.

Because of these limitations we cannot yet offer a way to automatically verify OSX builds, in the same way that we can for Linux, Windows, and Android. We will continue to look for ways to bring OSX builds to the same level as the rest of our Windows and Linux distributions.

Introducing Repliqate!​

With all the above changes, Cwtch Bindings for Linux and Windows are fully reproducible!

That alone is great, but we also want to make it easier for you to check the reproducibility of our builds yourself! As we noted in the introduction, the whole point of reproducible builds is that you no longer have to trust binaries provided by the Cwtch Team.

To make this process accessible we are releasing a new tool called repliqate.

Repliqate makes it easy to construct isolated build environments, powered by Qemu and a standard Debian Cloud Image distribution.

Repliqate runs build-scripts to perform actions like downloading the specific versions of Go used in Cwtch official builds, grabbing a copy of the source code for Cwtch bindings, compiling the latest tagged version, and checking the hash against the same version that is available from builds.openprivacy.ca.

We now provide Repliqate build-scripts for reproducible both Linux libCwtch.so builds, Windows libCwtch.dll builds!

We also have a partially repeatable Android cwtch.aar build script that reproduces the official build environment, which we will be using to complete Android reproducible builds as detailed in the last section.

You can (and I want to highly encourage you to) perform all these steps yourself (either via Repliqate, or a setup with the same specifications) and report back. We want to know if there are any other barriers to reproducing Cwtch bindings, and anything that we can do to make the process easier.

Next Steps​

Reproducible bindings are a big achievement, but there is obviously much more to do. In the coming weeks we are committed to undertaking the same process with our Cwtch UI builds to determine what needs to be done to make this as reproducible as bindings.

As we go through this process we also expect to add additional functionality to Repliqate. If you have any feedback or would like to contribute to Repliqate development then please get in touch!

Help us go further!​

We couldn't do what we do without all the wonderful community support we get, from one-off donations to recurring support via Patreon.

If you want to see us move faster on some of these goals and are in a position to, please donate. If you happen to be at a company that wants to do more for the community and this aligns, please consider donating or sponsoring a developer.

Donations of $5 or more can opt to receive stickers as a thank-you gift!

For more information about donating to Open Privacy and claiming a thank you gift please visit the Open Privacy Donate page.

A Photo of Cwtch Stickers