At the Tor Project, we’re always on the lookout for opportunities to contribute
back to the communities around the platforms and tools we depend on to keep the
lights on. Puppet and Debian are two such projects, so we’re happy to
announce that the upcoming Debian stable release, codename bookworm, will
deliver an up-to-date suite of Puppet software thanks to the efforts of the Tor
Project!

A year ago, TPA (AKA Tor Project sysadmin Team) started planning an
upgrade of our fleet of nearly 100 Debian machines to the latest stable
release, bullseye. One item of concern was that not only were the Puppet
packages in Debian bullseye already nearly end-of-life (version 5.5), but the
PuppetDB package was also now missing entirely from the distribution. At
this point it seemed the only feasible option would be to migrate our entire
Puppet infrastructure to the vendor-supplied packages.

Advantages of distribution packages

So why not switch over to the upstream Puppetlabs packages and call it a
day? Essentially, because deploying software directly from vendors is not a
decision we take lightly, and because Puppet is such a core component of our
infrastructure, this called for careful consideration.

There are several reasons to prefer software shipped through distribution
packages.

Improved security and reliability

Reports of increasing numbers of software supply-chain attacks in
recent years have shed light on the difficult problem of shipping software
securely. Many popular solutions, like npm or PyPI suffer from vulnerabilities
which are not easily fixed, such as typosquatting and developper
sabotage, at least not without breaking what makes them popular.

Puppetlabs doesn’t rely on third-party package managers, and instead maintains
its own APT repository to ship packages to end-users. Although this is an
improvement, configuring an additional APT repository on a Debian-based system
effectively grants whoever controls the repository the ability to deploy
software and execute code with maximum privileges. This is a significant
potential attack surface.

We understand the challenge of running an APT repository, and we trust in
the ability of the Debian project to maintain their own repositories reliably
and securely.

Even assuming the delivery channel is secure, concerns still remain about the
process used to build the binary packages. Not only does Puppetlabs employ a
somewhat exotic in-house build system, EZBake, but we have no means to
evaluate the integrity of the build process itself.

On the other hand, Debian package are built from source (as opposed to
redistributing upstream binaries) on purpose-built infrastructure, and logs
generated during this process are made available for examination.

The Debian project, in collaboration with reproducible-builds.org even
provides automated reproducibility tests for packages and encourages maintainers
to fix issues hampering the ability for anyone to compile a given package and
obtain a byte-for-byte identical binary. Both the puppet-agent and
puppetserver package we uploaded to Debian are currently reproducible.

User-centric policies

Another concern is that the Puppetlabs packages ship with analytics,
enabled by default. Although an opt-out configuration parameter exists,
features like this are necessarily in conflict with users’ right to privacy.

This is one area where Debian also shines: the project’s social contract
guarantees the interests of users are always placed first and foremost. This
guides Debian the established practice of adressing privacy leaks in
package resources and runtime code.

As such, Debian’s puppetserver package doesn’t implement telemetry, and the
update check that “phones home” at regular intervals is disabled by default.

Greater system integration

The Puppetlabs packages deploys its runtime libraries and code under the
/opt/puppetlabs directory. In addition to the Puppet applications, this
includes copies of the Ruby, JRuby and Clojure interpreters, as well as dozens
of third-party Clojure and Java libraries.

This duplication of code is a burden that is ultimately carried by system
administrators because they’re the ones responsible for ensuring their
infrastructure is running code that is up to date and free from security
issues. Debian has a solid track record on this front, and we’re confident that
security issues in, for example, the Ruby intrepreter, are adressed swiftly.

In addition, security updates to the Debian stable distribution are crafted in
a way that minimizes as much as possible the upgrade footprint. Puppetlabs
offers no such guarantees for their packages, so it’s not unlikely that at some
point, addressing a security issue might require a disruptive upgrade.

A sustained effort

Work on packaging new PuppetDB and Puppet Server Clojure dependencies in
Debian had already progressed, thanks to funding from the Wikimedia
Foundation. So it felt completing the work was within reach.

Thus, over the past 6 months, in addition to our regular TPA duties we have
been working to get Puppet in Debian ready for bookworm, the upcoming Debian
12 stable release. We started with the Puppet agent package, before moving on
to PuppetDB and finally Puppet Server, upgrading or adding dozens of dependency
packages along the way including many Clojure and Ruby libraries, JRuby and
other Puppet components such as Facter, Hiera and Trocla.

In collaboration with the Debian Puppet Team, we worked to triage and fix
bugs reported in the Debian bug tracker, as well as implementing new or
improved build and integration tests for Debian CI. We also collaborated
with Puppetlabs to merge several patches upstream, including one patch to
improve the reproducibility of Puppet agent builds.

Looking ahead

With this work now behind us, we can look forward to bringing our Puppet
infrastructure into a more modern age. Dealing with legacy tools and
technical debt is a constant challenge at the Tor Project, as it is no doubt in
many other organizations, so we’re very happy to share news of this
accomplishment. Also planned is migrating our various tor and onion deployments
to the community tor Puppet module.

We also know that many privacy-minded individuals and outfits (Tails, for
example) also rely on Puppet being packaged in Debian, so we’re thrilled
that this work will directly benefit them.

What about Puppet over Tor?

For those of you who were hoping to learn about using Puppet over Tor onion
services, you’ll enjoy this tutorial from the good folks at the
Immerda.ch tech collective. This isn’t something we implement as the list
of project-administered machines is public, but should be useful for anyone
wishing to add a privacy layer to their Puppet server-node communications,
especially if those are transported over the Internet.