See also: Cookbook · Reference Manual · Workflows · Roadmap
- Introduction
- Installation & Prerequisites
- Quick Start
- Configuration
- Building Packages
- Managing Environments
- Cleaning Up
- Publishing, Trust and Release Tasks
Bits is a build orchestration and dependency management tool for complex software stacks. It is derived from aliBuild, the build system developed for the ALICE experiment software at CERN, and is designed for communities that need to build and maintain large collections of interdependent packages with reproducibility, parallelism, and minimal overhead.
Acknowledgement. Bits is a fork of aliBuild, originally created by the ALICE collaboration at CERN. The recipe format, dependency-resolution model, content-addressable build hashing, remote binary store, and Docker build support all originate from aliBuild. Bits extends aliBuild with the repository provider mechanism, package families, shared packages, extended parallel builds and other features described in this document.
Bits is not a traditional package manager like apt or conda. Instead it automates fetching sources, resolving dependencies, building, and installing software in a controlled, reproducible environment. Each package is described by a recipe — a plain-text file with a YAML metadata header and a Bash build script — stored in a version-controlled recipe repository.
Key capabilities at a glance:
- Automatic topological dependency resolution and ordering
- Content-addressable incremental builds — only rebuilds what changed
- Parallel package builds and multi-core compilation
- Remote binary stores (HTTP, S3, rsync) to share pre-built artifacts, plus reuse of packages already deployed on CVMFS (
--reuse-from) - Docker-based builds for cross-compilation or reproducible CI environments
- Git and Sapling SCM support
- Dynamic recipe repositories loaded at dependency-resolution time
- Deterministic package tarballs, signed manifests and passkey-approved certification for trusted binary reuse
- SBOM export (CycloneDX / SPDX) and recipe-wide source checksums
The same bits build command that a developer runs interactively also drives the CI pipeline that publishes packages to CVMFS for the entire community — there is no separate local and CI toolchain. The full development-to-deployment workflow is described in WORKFLOWS.md.
| Requirement | Notes |
|---|---|
| Linux or macOS | x86-64 or ARM64 |
| Python 3.8+ | Required |
| Git | Required; Sapling (sl) is optional |
modulecmd |
Required for bits enter / load / unload |
| GNU tar (macOS) | Recommended on macOS (brew install gnu-tar) — without it bits warns and tarballs may not be byte-reproducible |
Install Environment Modules for your platform:
# macOS
brew install modules
# Debian / Ubuntu
apt-get install environment-modules
# RHEL / CentOS / AlmaLinux
yum install environment-modulesbits builds almost everything it needs from source, but it still relies on a host
toolchain to bootstrap (a working C/C++/Fortran compiler, the autotools/CMake build
tools, git, patch, make, and the usual archive utilities). Install these once
before your first build:
# Debian / Ubuntu
sudo apt install \
build-essential gfortran git patch make cmake autoconf automake libtool m4 \
pkg-config curl wget tar gzip bzip2 xz-utils unzip \
python3 python3-pip python3-venv environment-modules
# RHEL / AlmaLinux / Rocky (enable CRB/EPEL for some packages)
sudo dnf groupinstall "Development Tools"
sudo dnf install \
gcc-gfortran git patch make cmake autoconf automake libtool m4 \
pkgconfig curl wget tar gzip bzip2 xz unzip \
python3 python3-pip environment-modules
# macOS (Xcode command-line tools provide the compiler / git / make)
xcode-select --install
brew install cmake autoconf automake libtool pkg-config gnu-tar wget modulesPer-recipe system packages. A few recipes deliberately use a library or tool from
the system instead of building it (these are declared as system_requirement recipes,
e.g. readline, elfutils/libdw, perf). When such a package is missing, bits stops
early and lists the missing requirements rather than failing mid-build;
bits doctor <package> then prints each recipe's install hint. You don't need to
install them all up front — build what you need and follow the hint, or check ahead of
time with:
bits doctor # no package: checks this machine (Python modules, git/compiler,
# docker or rootless podman, disk space, store access)
bits doctor <package> # reports any missing system requirements for that packageCommon ones on Debian / Ubuntu (RHEL/AlmaLinux equivalents in parentheses):
sudo apt install libreadline-dev # readline (readline-devel)
sudo apt install libdw-dev libelf-dev # elfutils / libdw (elfutils-devel) e.g. heaptrack
sudo apt install linux-perf # perf (perf) e.g. adaptystmacOS: the Homebrew system layer. On macOS, stable low-level libraries are taken from
Homebrew instead of being built (recipes declare homebrew_formula:). Every macOS build
records the formulae declared by the recipes it sees (config dir and providers) in sw/<arch>/Brewfile. On the first macOS build
that file does not exist yet, so bits writes it and stops with instructions; install the
formulae and re-run:
bits build ROOT # first run: writes sw/<arch>/Brewfile and stops
brew bundle --file sw/<arch>/Brewfile # install the listed formulae
bits build ROOT # now builds
bits build --brew ROOT # alternative: install missing formulae on demand
bits brew # write the Brewfile without resolving a build
bits build --dry-run ROOT # also writes the Brewfile, nothing builtSee REFERENCE.md — macOS Homebrew system layer.
git clone https://github.com/bitsorg/bits.git
cd bits
export PATH=$PWD:$PATH
python3 -m venv .venv && source .venv/bin/activate # activate it in every new shell; recent distributions refuse a system-wide pip install
pip install -e .bits always builds from a community recipe repository: a *.bits repository with the
community's defaults (defaults-release.sh), CVMFS layout and recipes. Clone it first and
run bits inside it — bits uses the current directory as its recipe directory. In a
directory without recipes a build stops.
git clone https://github.com/bitsorg/stacks.bits && cd stacks.bits
# or resolve it in the bits-providers registry: bits init stacks.bits && cd stacks.bitsCommunity repositories: stacks.bits (LCG-based stacks), alice.bits, atlas.bits,
cms.bits, key4hep.bits, lhcb.bits, ship.bits. They pull shared recipe pools
(lcg.bits, common.bits, alidist.bits) on demand; bits init <name>.bits clones a
community repository from the bits-providers
registry.
bits doctor # check that this machine is set up to run bits
bits doctor --defaults gcc15 ROOT # check ROOT's system requirements
bits build --dry-run --defaults gcc15 ROOT # optional: what would be reused and what built
bits build --defaults gcc15 ROOT # resolves and builds ROOT and all dependencies
bits enter ROOT/latest # open a sub-shell with the environment loaded
root -b
exit # return to your normal shellFor another community the steps are the same, e.g.
git clone https://github.com/bitsorg/lhcb.bits && cd lhcb.bits && bits build DaVinci.
Record per-directory options (work directory, stores) once with bits use build …
inside the repository (see Configuration).
The aliBuild wrapper keeps ALICE's classic workflow, working from an alidist checkout:
aliBuild init # check out alidist
aliBuild build O2Record per-directory settings once with bits use, so you do not repeat them on every command. It saves command-line options in a profile and inserts them into later commands. A leading section name says which command they belong to (build, deps, publish, …); without one they go to [common], which is applied to every architecture-aware command, so keep it to options all of them accept (--architecture).
bits use --architecture x86_64-el9-gcc14-opt # [common]: every arch-aware command
bits use build --work-dir /path/to/sw --parallel 4 \
--remote-store https://s3.cern.ch/swift/v1/mybucket # [build]: bits build only
bits use # show the active profile and where it lives
bits use --clear build # clear one section (no SECTION: clear all)Each bits use SECTION … replaces that whole section, so give all of a section's options in one call. Saved arguments are inserted before your own, so an explicit flag on the command line still wins. The flags of any command that reads the profile (build, deps, doctor, status, clean, prune, publish, certify, sign, compliance, import, q, enter, load, …) can be saved this way, e.g. bits use build --docker --store-integrity (SHA-256 verification of every recalled tarball). For a work directory shared by every command, including bits enter and bits q, set $BITS_WORK_DIR instead.
bits init with configuration options and no PACKAGE writes the same profile for the settings it knows: --architecture to [common]; --remote-store, --write-store, --defaults, -c/--config-dir, -w/--work-dir and --reference-sources to [build] (replacing those sections, as bits use does). Add --dry-run to see what it would save:
bits init --work-dir /path/to/sw --remote-store https://s3.cern.ch/swift/v1/mybucketThe profile is ./.bitsuse in the current directory, or a record under ~/.bits/use/ when the directory is not writeable or not owned by you. A local .bitsuse is only honoured when it is owned by you; otherwise the ~/.bits/use/ record is used. bits use --help lists these forms.
bits.rcis retired. Earlier versions readbits.rc/.bitsrc/~/.bitsrc; those files are no longer read. Move per-directory settings into abits useprofile and global ones into the environment variables below.
Global settings come from environment variables:
| Variable | Related flag | Description |
|---|---|---|
$BITS_COMMUNITY |
— | Community name (uppercase), e.g. LHCB ($BITS_ORGANISATION, its former name, still works). Used only when -c/--config-dir names a directory that does not exist: bits then clones that community's recipe repository from the registry and uses it. The aliBuild wrapper sets ALICE. |
$BITS_WORK_DIR |
-w / --work-dir |
Output directory for built packages (default: sw). |
$BITS_REPO_DIR |
-c / --config-dir |
Root directory for recipe repositories. |
$BITS_PROVIDERS |
— | URL of the bits-providers registry, optionally @tag (default https://github.com/bitsorg/bits-providers; off under the aliBuild wrapper). |
$BITS_PATH |
--search-path |
Recipe search path. |
$BITS_S3_STORE |
--remote-store (store ops) |
Default S3 store for bits store (gc/stats/upload), certify, publish, compliance. |
$BITS_COMMUNITY is set uppercase (ALICE, LHCB, …). Bits lowercases it internally when resolving the community recipe repository from bits-providers (e.g. LHCB → lhcb.bits.sh → https://github.com/bitsorg/lhcb.bits). Normally you do not need it: check out the community repository and run bits inside it.
Settings follow the precedence CLI flag > bits use profile > environment variable > built-in default; the exception is $BITS_PATH, which, when set, overrides --search-path. For the full list of environment variables, see REFERENCE.md §20 and REFERENCE.md — bits init.
bits build [options] PACKAGE [PACKAGE ...]Bits resolves the full transitive dependency graph of each requested package, computes a content-addressable hash for every node, downloads any pre-built artifacts that already exist in a remote store, and builds the rest in topological order.
- Recipe discovery — Bits locates
<package>.shin the recipe directory, then in eachBITS_PATHentry (--search-path; a relative NAME means<config-dir>/NAME.bits, an absolute path is used as is). Repository-provider packages (see §13) are cloned first to extend the search path before the main resolution pass. - Dependency resolution —
requires,build_requires, andruntime_requiresfields are read recursively, forming a DAG. Cycles are reported as errors. - Hash computation — A hash is computed for each package from its recipe text, source commit, dependency hashes, and environment. Packages with a matching hash in a store are downloaded instead of rebuilt.
- Source fetching — Source repositories are cloned into a local mirror and then checked out into a build area. Up to 8 repositories are fetched in parallel.
- Build execution — Each package's Bash script runs in an isolated environment with sanitised locale and only its declared dependencies visible.
- Post-build — A modulefile and a versioned tarball are written; the tarball may be uploaded to a write store.
| Option | Description |
|---|---|
--defaults PROFILE |
Defaults profile(s) to load, combined with ::; release is always the base (--defaults gcc15 means release::gcc15). Default: release. |
--set NAME[=VALUE] |
Set a build-wide flavour variable (alias of --flavour); gates conditional (?NAME) requires/sources/patches and overrides a defaults variables: entry. |
-j N, --jobs N |
Parallel compilation jobs per package. Default: CPU count. |
--parallel [N] |
Number of packages to build simultaneously. Bare --parallel uses 4; omit it for serial (the default). With N>1 each build's $JOBS is divided across the builders (-j ÷ N) so the concurrent jobs together stay within one machine's worth of cores. (--builders is a kept alias.) |
--build-nice |
Stagger concurrent builders across OS priority levels so CPU contention degrades gracefully — one build runs at full speed, the others are backed off, and the freed top slot is taken over as builds finish. Native builds use nice; --docker builds use docker run --cpu-shares. Opt-in; only affects --parallel > 1. Memory is still capped separately. |
--build-nice-step N |
Priority spread between concurrent build slots for --build-nice (slot k → nice min(k×N, 19)). N=1 is a gentle ladder; larger separates slots more. Default: 5. |
--prefetch-workers N |
Background threads that fetch remote tarballs and sources ahead of the build loop. Default: auto (min(builders, 4)); 0 disables. |
-u, --fetch-repos |
Update all source mirrors before building (the default), so a branch builds its current commit. |
--no-fetch-repos |
Build what the mirrors already have (offline). |
-w DIR, --work-dir DIR |
Work/output directory. Default: sw. |
--remote-store URL |
Binary store to pull pre-built tarballs from. Append ::rw to also upload to it. |
--write-store URL |
Binary store to push newly-built tarballs to. Given alone (no --remote-store), it is also the store reused from, except where a default remote store applies (slc7/8/9, ubuntu x86-64, slc9_aarch64); use --remote-store URL::rw there. |
--force-rebuild PKG |
Rebuild these packages from scratch even if they were built before (repeatable or comma-separated). |
--prefer-system |
Always use a compatible system package instead of building it (--always-prefer-system is the older spelling). --no-system [PKGS] does the opposite. |
--brew |
macOS only: brew install missing Homebrew formulae on demand (see Installation). |
--docker |
Build inside a Docker container. |
-d, --debug |
Verbose debug output. |
-n, --dry-run |
Print a per-package reuse plan without building (see below). |
--no-auto-cleanup |
Keep build directories after a successful build (useful for debugging). |
For the full option list, parallel build tuning (--parallel, --oversubscribe, --prefetch-workers, --parallel-sources) and Docker/cross-compilation options, see REFERENCE.md — bits build and REFERENCE.md §22. The --makeflow / --pipeline build modes have been removed; use --parallel.
bits build --parallel ROOT # 4 packages at a time
bits build --parallel 8 -j 32 ROOT # 8 builders sharing 32 compile jobsWithout --parallel packages are built one after another, each with the full -j. With N builders each package gets a share of -j; the final top-level package, which builds alone, gets the full -j again. --builders N still works as an alias.
bits build --dry-run ROOTComputes every package's hash exactly as the build would and prints, in build order, where each one would come from — installed, local tarball, from remote store, from reuse overlay — or that it would be built, followed by a summary. Nothing is downloaded or installed; http(s):// and b3:// stores are only listed, and other stores (s3://, rsync) are not checked. bits status --check-store uses the same store listing. On macOS a dry run also writes sw/<arch>/Brewfile.
# Developer: reuse what CI built, never upload
bits build --remote-store https://s3.cern.ch/swift/v1/mybucket ROOT
# CI: reuse from and upload to the same bucket
bits build --remote-store b3://mybucket::rw ROOTRemote reuse is fail-closed: a tarball from a remote store is only reused when a verified signed manifest vouches for it (--no-require-signed-reuse turns this off, e.g. to bootstrap a new store). Local and CVMFS artifacts are unaffected. To upload a single package you already built, use bits store upload PKG. See REFERENCE.md §21.
bits deps ROOT --outgraph root.pdf # Graphviz graph
bits deps ROOT --outmake root.mk # Makefile rules (pkg: dep1 dep2), no Graphviz needed
bits deps ROOT --outmake root.mk --runtime-only # skip build-only dependenciesBits uses the standard Environment Modules system (modulecmd) to manage runtime environments. A module corresponds to one built package version. The bits shell script discovers modulecmd automatically — on $PATH, next to envml, or via Homebrew's modules formula. If it cannot be found, it prints the appropriate install command.
bits enter ROOT/latest
# A new sub-shell opens with ROOT and all its dependencies in PATH etc.
exit # return to your normal shellbits enter prefixes the shell prompt with the loaded modules, e.g. [ROOT/latest] (sh/bash/ksh/zsh, unless --shellrc is given), so it is always clear when inside a bits environment. Nesting bits enter inside another bits environment is blocked.
| Option | Description |
|---|---|
--shellrc |
Source your shell startup file (.bashrc, .zshrc, etc.) in the new shell. |
--dev |
Source each package's etc/profile.d/init.sh directly instead of using modulecmd. |
--view |
Collapse PATH, LD_LIBRARY_PATH, PYTHONPATH, … onto one merged view of the loaded closure, keeping the environment short on big stacks (also accepted by bits load). Entering a package whose recipe sets view: true turns it on automatically. |
-q |
Silence the module-load messages. |
Add the shell helper to your .bashrc or .zshrc once so that bits load and bits unload modify the current shell's environment without requiring an explicit eval:
BITS_WORK_DIR=/path/to/sw
eval "$(bits shell-helper)"Then in any shell session:
bits load ROOT/latest # adds ROOT to the current environment
bits unload ROOT # removes it (version can be omitted)
bits list # show currently loaded modules
bits q [REGEXP] # list available modules, optionally filteredWithout shell-helper, use eval manually:
eval "$(bits load ROOT/latest)"
eval "$(bits unload ROOT)"bits setenv ROOT/latest -c root -b
# Everything after -c is executed as-is; the exit code is preserved.bits setenv loads the modules into the current process environment and then execs the command — no new shell is spawned.
Bits provides two distinct cleaning subcommands for different scenarios.
bits clean [options]| Option | Description |
|---|---|
-w DIR |
Work directory to clean. Default: $BITS_WORK_DIR (or $ALICE_WORK_DIR), else sw. |
-C DIR, --chdir DIR |
Change to DIR first (or $BITS_CHDIR). |
-a ARCH |
Restrict to this architecture. |
--aggressive-cleanup |
Also remove SOURCES/ (source checkouts and download cache) and the tarballs under TARS/<arch>/store and TARS/share/store. Git mirrors (MIRROR/) are kept. |
-n, --dry-run |
Show what would be removed without deleting. |
The default (non-aggressive) clean removes the TMP/ and INSTALLROOT/ staging areas, BUILD/ directories no *-latest* symlink points to, and installed versions (for the selected architecture and share/) no latest* symlink points to. Aggressive cleanup additionally removes SOURCES/ and the tarballs under TARS/<arch>/store and TARS/share/store; tarball symlinks and MIRROR/ are kept. Use bits clean after temporary or experimental builds to reclaim disk space without affecting the persistent package cache.
Renamed to
bits prune.bits cleanupstill works as a deprecated alias that warns and forwards; usebits prunein new scripts.
bits prune manages a long-lived, shared workDir by evicting packages that have not been used recently or when disk space falls below a threshold. It is intended for persistent CI build caches where packages accumulate over time.
bits prune [options]| Option | Default | Description |
|---|---|---|
-w DIR, --work-dir DIR |
$BITS_WORK_DIR (or $ALICE_WORK_DIR), else sw |
workDir to manage. |
-a ARCH, --architecture ARCH |
auto-detected | Architecture to evict packages for. |
--max-age DAYS |
7.0 |
Evict packages whose sentinel has not been touched in more than DAYS days. Set to 0 to disable age-based eviction. |
--min-free GIB |
(none) | Evict the least-recently-used packages until at least GiB GiB of free disk space is available. |
--disk-pressure-only |
— | Run only the disk-pressure eviction pass; skip age-based eviction. |
--retain |
— | Manifest-rooted sweep over all architectures: keep the packages of the newest --keep-builds N (default 2) build manifests and anything certified but not yet on CVMFS; evict what is safely upstream (stored, in the verified signed manifest and published to CVMFS). Signed manifests come from --remote-store and/or --trust-manifest; an architecture whose manifest cannot be verified is skipped. |
--grace-days DAYS |
1.0 |
With --retain: never evict anything modified more recently than DAYS days ago. |
-n, --dry-run |
— | Show which packages would be evicted without removing anything. |
How it works. Every time a package is built or confirmed already installed, bits touches a sentinel file at $WORK_DIR/.packages/<arch>/<package>/<version>. The prune command reads these sentinels, sorts packages by last-touched time (oldest first), and evicts those that are too old or that need to be removed to recover disk space.
Typical usage patterns:
# Pre-build: free space if below 50 GiB, evicting LRU packages first
bits prune --min-free 50 --disk-pressure-only || true
# Nightly cron: evict packages not used in 7 days
bits prune --max-age 7
# See what would be removed without touching anything
bits prune --max-age 3 --min-free 100 --dry-runFor the full option list see REFERENCE.md — bits prune. To clean the shared S3 store rather than a local workDir, use bits store gc (formerly bits gc) — see Confusing command pairs.
Short recipes for the commands that act beyond your local build. Most are run by CI or by a group's release manager; each links to the full reference.
bits publish PACKAGE relocates a built package to its final CVMFS path and hands it to the cvmfs-prepub service; it is CVMFS-only:
bits publish ROOT --cvmfs-target /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.32.02/x86_64-el9 \
--prepub-url https://prepub.example.org:8080
bits publish --release-view LCG_110 --cvmfs-target /cvmfs/sft.cern.ch/lcg # merged view under Views/ (--view is deprecated)- With no PACKAGE,
bits publishuploads every package of the latest build manifest to the S3 store (--manifest FILEpicks another), together with the release's NOTICE, GPL source offer and SBOMs. To upload one package to the S3 store usebits store upload PKG. - When a
cvmfs_packages_templateis set, the pipeline'sbits cvmfs publishsends each package's identity path and hash, so prepub skips a duplicate that is already queued or published. With--replace-on-conflicta package another build published (a different hash) is replaced, if prepub allows it (replace_on_conflict); bits checks before uploading. The ingest path also refuses up front a tarball larger than prepub's advertisedmax_tar_size. - A group that sets
cvmfs_packages_templatepublishes each package once per build architecture; releases are then views over those packages (cvmfs_releases_template, and withcvmfs_views_templatean LCG-style merged view with a self-locatingsetup.sh). Path templates may use{arch}and the nightly{day}token. - The producer-side commands are now grouped:
bits cvmfs stage/bits cvmfs publish(formerlybits cvmfs-stage/bits cvmfs-publish) andbits store stats(formerlybits store-stats); the old names still work and warn.
See REFERENCE.md — bits store / bits cvmfs and REFERENCE.md §26.
bits cvmfs-path resolves a package's publish path from the group's templates (in defaults-release.sh under system:) without building:
bits cvmfs-path --package ROOT --version 6.32.02 --admin --set release=LCG_110
bits cvmfs-path --package ROOT --version 6.32.02 --login jdoe # personal (user-prefix) pathPass the same --set / --day values as the build so {release} and {day} resolve identically; --kind selects the packages, releases, modules or shared template. See bits cvmfs-path --help.
bits certify # latest build: upload what is missing, approve with a passkey, open the MRbits certify is what a user runs: it uploads anything still missing from the store, asks bits-console for a passkey approval (scan the QR code, compare the code, approve on your phone) and then opens the certification merge request in the manifests repository. bits sign — the command formerly called certify — is run by the manifests-repo CI to merge the approved build manifests into the signed common manifest. Group, manifests repo and console URL default from the defaults' system: block. See REFERENCE.md — Publishing and certifying a build and Signing — bits sign.
bits sbom sw/MANIFESTS/bits-manifest-latest.json # sbom.cdx.json + sbom.spdx.json
bits sbom --format spdx -o - sw/MANIFESTS/bits-manifest-latest.jsonWrites a deterministic CycloneDX 1.6 and/or SPDX 2.3 document from a build manifest (credentials are stripped from source URLs); manifests older than schema v4 give components only, without dependency edges. A whole-build upload (bits publish with no package, or bits certify) stores both next to the release NOTICE. See REFERENCE.md — bits sbom.
cd myproject.bits
bits checksums # report: hash every tarball and patch, resolve git tags
bits checksums --write # record new entries in checksums/ (never overwrites)
bits checksums --defaults all --recipes ../lcg.bits # also the sources its defaults-*.sh profiles overrideGit tags are pinned to their commit; branches are reported as moving and never pinned. The exit status is 1 on any mismatch or failure. See REFERENCE.md — bits checksums.
bits overlay lcg writes an lcgcmake-style LCG release over a built closure: LCG_<num><postfix>/LCG_externals_<platform>.txt (plus an LCG_generators_<platform>.txt placeholder) under --out. With --build-view DIR it also builds a merged symlink-farm view in DIR, with a setup.sh that puts the view on PATH, LD_LIBRARY_PATH and friends:
bits overlay lcg -a x86_64-el9-gcc15 --platform x86_64-el9-gcc15-opt \
--version-number 110 --out /path/to/releases --build-view /path/to/viewbits lcg-view is the deprecated former name. See bits overlay lcg --help.
- Cookbook — practical recipes for common tasks
- Reference Manual — command-line flags, recipe format, environment variables, Docker, stores, CVMFS pipeline, developer guide
- Workflows — development-to-deployment walkthrough
- Roadmap — planned features and priorities