devenv is switching to Tvix
Domen Kožar / Cachix
NixCon 2024
Crossing the Chasm
Nix going mainstream
Positive feedback
Negative feedback
Absolutely no one:
Nix users:
Developer Experience
Cognitive Overhead
Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
- Antoine de Saint-Exupéry
Perfection is achieved when there is nothing left to take away.
- Antoine de Saint-Exupéry
Perfection is achieved when nothing is left to take away.
- Antoine de Saint-Exupéry
Perfection is achieved when nothing is left to take away.
YAGNI - Developers
Developer Environments
- Develop natively
- Deploy containers
- 100.000+ packages
- Write scripts
- Automate using tasks
- 50+ supported languages
- Define processes
- 30+ services
- Run tests
- Enforce git hooks
Nix is the JavaScript of DevOps
Workshop at 15h
✅ CLI interface
$ devenv init
$ devenv shell
$ devenv up
$ devenv test
$ devenv search ncdu
$ devenv update
$ devenv build
$ devenv container build
- Add new options
- Remove options
- Deprecate options
- Rename options
- Extend options
✅ Nix interface
Nix API in 2022: CLI
- ✘highly restrictive interface
- ✘maintained C++ Nix patches
Precise Nix eval caching in devenv
❯ devenv shell
• Building shell
✔ Building shell in 0.1s.
• Entering shell
Running tasks devenv:enterShell
Succeeded devenv:pre-commit:install 17ms
Succeeded devenv:enterShell 8ms
2 Succeeded 27.38ms
rustc 1.76.0 (07dca489a 2024-02-04) (built from a source tarball)
❯ devenv --refresh-eval-cache shell
• Building shell ...
✔ Building shell in 5.5s.
• Entering shell
Running tasks devenv:enterShell
Succeeded devenv:pre-commit:install 10ms
Succeeded devenv:enterShell 5ms
2 Succeeded 17.12ms
rustc 1.76.0 (07dca489a 2024-02-04) (built from a source tarball)
Using the cache
Without the cache
nix --no-eval-cache build '.#git' --log-format internal-json -vvvv|&grep "evaluating file"
...
@nix {"action":"msg","level":4,"msg":"evaluating file '/nix/store/nwd941897rv3nnpj665lnvbnz45spf49-source/pkgs/development/python-modules/pygments/default.nix'"}
@nix {"action":"msg","level":4,"msg":"evaluating file '/nix/store/nwd941897rv3nnpj665lnvbnz45spf49-source/pkgs/development/python-modules/lxml/default.nix'"}
@nix {"action":"msg","level":4,"msg":"evaluating file '/nix/store/nwd941897rv3nnpj665lnvbnz45spf49-source/pkgs/development/python-modules/cython/default.nix'"}
@nix {"action":"msg","level":4,"msg":"evaluating file '/nix/store/nwd941897rv3nnpj665lnvbnz45spf49-source/pkgs/build-support/fetchgitlab/default.nix'"}
...How it works today
- Parse Nix logs to detect exactly which paths were accessed.
- Store the CLI as the key in sqlite.
- Look up the cache on the next invocation comparing mtime and hash.
loc devenv-eval-cache/src
--------------------------------------------------------------------------------
Language Files Lines Blank Comment Code
--------------------------------------------------------------------------------
Rust 7 1625 182 96 1347
--------------------------------------------------------------------------------
Total 7 1625 182 96 1347
--------------------------------------------------------------------------------
Nix CLI
No guarante if the API will change.
Rust via Nix C FFI or tvix.dev?
Rewrites are scary, unless done in a modular way.
Tvix: modular Nix (2021)
Tvix evaluator
bytecode (with cache interface)
Nix
derivations
compilation + optimizations
execution
Tvix eval caching
/// Represents all possible filesystem interactions that exist in the Nix
/// language, and that need to be executed somehow.
pub trait EvalIO {
fn path_exists(&self, path: &Path) -> io::Result<bool>;
fn open(&self, path: &Path) -> io::Result<Box<dyn io::Read>>;
fn file_type(&self, path: &Path) -> io::Result<FileType>;
fn read_dir(&self, path: &Path) -> io::Result<Vec<(bytes::Bytes, FileType)>>;
fn import_path(&self, path: &Path) -> io::Result<PathBuf>;
fn store_dir(&self) -> Option<String>;
}Fast, Declarative, Reproducible and Composable Developer Environments using Nix
FAAAAAAAAST
1. Check if any of the files needed for this evaluation changed ✅ ~100ms
2. Evaluate devenv.nix: ~600 Nix files.
3. Use bytecode caching underneath to save 99% of compilation
Roadmap towards Fast
1. devenv.nix is rarely changed
2. If it does change, it's only one or a few files
3. devenv update invalidates the cache
Tvix, when?
Tvix evaluator in devenv
>>> builtins.fetchTree "github:NixOS/nixpkgs/ae2e6b3958682513d28f7d633734571fb18285dd"
{
lastModified = 1686503798;
lastModifiedDate = "20230611171638";
narHash = "sha256-rA9RqKP9OlBrgGCPvfd5HVAXDOy8k2SmPtB/ijShNXc=";
outPath = "/nix/store/l5m6qlvfs9sdw14ja3qbzpglcjlb6j1x-source";
rev = "ae2e6b3958682513d28f7d633734571fb18285dd";
shortRev = "ae2e6b3";
}- Use gitoxide (pure Rust git)
- Expose caching layer as a trait.
- Get rid of GitHub api calls.
builtins.fetchTree
Nix debugger
Tvix evaluator in devenv
$ devenv --nix-debugger
13 | buildInputs = [ pkgs.gcc pkgs.make ];
14 |
15 | installPhase = ''
16 | ${x 1}
^^^^^^
17 | mkdir -p $out/bin
18 | cp my-binary $out/bin/
19 | '';
Evaluation error in package.nix:16:
value is a set while a function was expected
Bindings
x: {}
>>>Regression tests against nixpkgs,
debugging tooling.
Tvix evaluator in devenv
nix-daemon: nix-compat or FFI
Tvix evaluator in devenv
Reverse engineering nix-daemon protocol
It's fun!
It's fun!
ThaigerSprint.org
12th - 19th February 2025, Chiang Mai
OceanSprint.org
17th - 21th March 2025, Lanzarote
Thank you.
domen@cachix.org / @domenkozar
