Reproducible Canister Builds

Typically, you will version your code in git or some other version control system, and your versioned code may be available in a public repository, e.g., GitHub. In this case, you should note the particular commit that you used when producing the code to be deployed to the Internet Computer, and communicate it to the users who are trying to verify your canister. Alternatively, you can also provide a package (e.g., a zip file or a tarball) containing the source code you used to build your canister.

Before building your code, you should document the build environment you are using in detail. In particular, for the languages supported by the Internet Computer SDK:

You should communicate all of these to your user in the instructions. Ideally, do so by providing an executable recipe to recreate the build environment, using tools such as Docker or Nix. We recommend using Docker, as this allows you to also pinpoint the operating system used for building the software.

Docker containers are a popular solution for providing build environments. You can use a Dockerfile such as the following to provide the user with a particular version of the operating system, as well as dfx, Node.js and the Rust toolchain.

There are a couple of things worth noting about this Dockerfile:

To use this Dockerfile, get Docker up and running, place the Dockerfile in the project directory of your canister, and create the Docker container by running:

This creates a Docker container image called mycanister, with Node.js, Rust and dfx installed in it, and your canister source code copied to /canister (recall that you should invoke docker build from the canister project directory). You can then enter an interactive shell inside of your container by running:

From here, you can experiment with the steps needed to build your canister. Once you are confident that the steps are deterministic, you can also put them in the Dockerfile, to allow the user to automatically reproduce your build when creating the canister. You can see an example in the Dockerfile of the Internet Identity canister. Next, we will investigate what is necessary to make the build deterministic.

For the build process to be deterministic:

If reproducibility is vital for your code, you should test your builds to increase your confidence in their reproducibility. Such testing is non-trivial: we have seen real-world examples where non-determinism in a canister build took a month to show up! Fortunately, the Debian Reproducible Builds project created a tool called reprotest, which can help you automate reproducibility tests. It tests your build by running it in two different environments that differ in characteristics such as paths, time, file order, and others, and comparing the results. To use it, you can put the following Dockerfile in the root directory of your canister project.

Next, create a canister_ids.json file containing the IDs of your canisters on the Internet Computer, and put it in your project directory. Now, from the root directory of your canister project, you can test the reproducibility of your dfx builds as follows:

The first command builds the Docker canister using the above Dockerfile. The second one opens an interactive shell (hence the -it flags) in the canister. We run this in privileged mode (the --privileged flag), as reprotest uses kernel modules for some build environment variations. You can also run it in non-privileged mode by excluding some of the variations; see the reprotest manual. The --rm flag will destroy the canister after you close its shell. Finally, once inside of the canister, we launch reprotest in verbose mode (the -vv flags). You need to give it the build command you want to run as the first argument. Here, we assume that it’s dfx build --network ic - adjust it if you’re using a different build process. It will then run the build in two different environments. Finally, you need to tell reprotest which paths to compare at the end of the two builds. Here, we compare the Wasm code for all canisters, which is found in the .dfx/ic directory.

If the comparison doesn’t find any differences, you will see an output similar to this one:

Congratulations - this is a good indicator that your build is not affected by your environment! Note that reprotest can’t check that your dependencies are pinned properly - use guidelines from the previous section for that. Moreover, we recommend you to run the container reprotest builds under several host operating systems and compare the results. If the comparison does find differences between the Wasm code produced in two builds, it will output a diff. You will then likely want to use the --store-dir flag of reprotest to store the outputs and the diff somewhere where you can analyze them. If you are struggling to achieve reproducibility, consider also using DetTrace, which is a container abstraction that tries to make arbitrary builds deterministic.

Finally, even after you achieve reproducibility for your builds, there are still other things to consider for the long term.

Reproducibility can be more demanding if you expect your canister canister code to stay around for years, and stay reproducible. The biggest challenges are to ensure that your:

Distributions and package archives may drop old versions of packages, including both your toolchain and their dependencies. Web sites may go offline and URLs might stop working. Thus, it’s prudent to back up all of your toolchain and dependencies. You should consider getting involved in projects such as Software Heritage, which do this on a large scale. At some later point in time, you might have to adjust your build process (e.g., by changing URLs) to ensure that your canister still builds. Even if the build changes, if it still yields the same result, your users can be confident that your canister is running the correct code. The trust argument is easier if your dependencies come from a trustworthy source, such as the Software Heritage project.