devenv is switching to Tvix

Domen Kožar / Cachix

NixCon 2024

Crossing the Chasm

Nix going mainstream

Positive feedback

Negative feedback

Absolutely no one:

Nix users:

Developer Experience

Cognitive Overhead

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

- Antoine de Saint-Exupéry

Perfection is achieved when there is nothing left to take away.

- Antoine de Saint-Exupéry

Perfection is achieved when nothing is left to take away.

- Antoine de Saint-Exupéry

Perfection is achieved when nothing is left to take away.

 

YAGNI - Developers

 

 

Developer Environments

Nix is the JavaScript of DevOps

Workshop at 15h

✅ CLI interface

$ devenv init

$ devenv shell

$ devenv up

$ devenv test

$ devenv search ncdu

$ devenv update

$ devenv build

$ devenv container build

  • Add new options
  • Remove options
  • Deprecate options
  • Rename options
  • Extend options

✅ Nix interface

Nix API in 2022: CLI

  • ✘highly restrictive interface
  • ✘maintained C++ Nix patches

Precise Nix eval caching in devenv

❯ devenv shell
• Building shell
✔ Building shell in 0.1s.
• Entering shell
Running tasks     devenv:enterShell
Succeeded         devenv:pre-commit:install 17ms
Succeeded         devenv:enterShell         8ms
2 Succeeded                                 27.38ms

rustc 1.76.0 (07dca489a 2024-02-04) (built from a source tarball)

❯ devenv --refresh-eval-cache shell
• Building shell ...
✔ Building shell in 5.5s.
• Entering shell
Running tasks     devenv:enterShell
Succeeded         devenv:pre-commit:install 10ms
Succeeded         devenv:enterShell         5ms
2 Succeeded                                 17.12ms

rustc 1.76.0 (07dca489a 2024-02-04) (built from a source tarball)

Using the cache

Without the cache

nix --no-eval-cache build '.#git' --log-format internal-json -vvvv|&grep "evaluating file"
...
@nix {"action":"msg","level":4,"msg":"evaluating file '/nix/store/nwd941897rv3nnpj665lnvbnz45spf49-source/pkgs/development/python-modules/pygments/default.nix'"}
@nix {"action":"msg","level":4,"msg":"evaluating file '/nix/store/nwd941897rv3nnpj665lnvbnz45spf49-source/pkgs/development/python-modules/lxml/default.nix'"}
@nix {"action":"msg","level":4,"msg":"evaluating file '/nix/store/nwd941897rv3nnpj665lnvbnz45spf49-source/pkgs/development/python-modules/cython/default.nix'"}
@nix {"action":"msg","level":4,"msg":"evaluating file '/nix/store/nwd941897rv3nnpj665lnvbnz45spf49-source/pkgs/build-support/fetchgitlab/default.nix'"}
...

How it works today

  1. Parse Nix logs to detect exactly which paths were accessed.
  2. Store the CLI as the key in sqlite.
  3. Look up the cache on the next invocation comparing mtime and hash.
loc devenv-eval-cache/src
--------------------------------------------------------------------------------
 Language             Files        Lines        Blank      Comment         Code
--------------------------------------------------------------------------------
 Rust                     7         1625          182           96         1347
--------------------------------------------------------------------------------
 Total                    7         1625          182           96         1347
--------------------------------------------------------------------------------

Nix CLI

No guarante if the API will change.

Rust via Nix C FFI or tvix.dev?

Rewrites are scary, unless done in a modular way.

Tvix: modular Nix (2021)

Tvix evaluator

bytecode (with cache interface)

Nix

derivations

compilation + optimizations

execution

Tvix eval caching

/// Represents all possible filesystem interactions that exist in the Nix
/// language, and that need to be executed somehow.
pub trait EvalIO {
  fn path_exists(&self, path: &Path) -> io::Result<bool>;

  fn open(&self, path: &Path) -> io::Result<Box<dyn io::Read>>;

  fn file_type(&self, path: &Path) -> io::Result<FileType>;

  fn read_dir(&self, path: &Path) -> io::Result<Vec<(bytes::Bytes, FileType)>>;

  fn import_path(&self, path: &Path) -> io::Result<PathBuf>;

  fn store_dir(&self) -> Option<String>;
}

Fast, Declarative, Reproducible and Composable Developer Environments using Nix

FAAAAAAAAST

1. Check if any of the files needed for this evaluation changed ✅ ~100ms

2. Evaluate devenv.nix: ~600 Nix files.

3. Use bytecode caching underneath to save 99% of compilation

Roadmap towards Fast

1. devenv.nix is rarely changed

2. If it does change, it's only one or a few files

3. devenv update invalidates the cache

Tvix, when?

Tvix evaluator in devenv

>>> builtins.fetchTree "github:NixOS/nixpkgs/ae2e6b3958682513d28f7d633734571fb18285dd"
{
  lastModified = 1686503798;
  lastModifiedDate = "20230611171638";
  narHash = "sha256-rA9RqKP9OlBrgGCPvfd5HVAXDOy8k2SmPtB/ijShNXc=";
  outPath = "/nix/store/l5m6qlvfs9sdw14ja3qbzpglcjlb6j1x-source";
  rev = "ae2e6b3958682513d28f7d633734571fb18285dd";
  shortRev = "ae2e6b3";
}
  • Use gitoxide (pure Rust git)
  • Expose caching layer as a trait.
  • Get rid of GitHub api calls.

builtins.fetchTree

Nix debugger

Tvix evaluator in devenv

$ devenv --nix-debugger
 13 |   buildInputs = [ pkgs.gcc pkgs.make ];
 14 |
 15 |   installPhase = ''
 16 |     ${x 1}
          ^^^^^^
 17 |     mkdir -p $out/bin
 18 |     cp my-binary $out/bin/
 19 |   '';
Evaluation error in package.nix:16: 
  
  value is a set while a function was expected
 
  Bindings
    x: {}
  
>>>

Regression tests against nixpkgs,

debugging tooling.

Tvix evaluator in devenv

nix-daemon: nix-compat or FFI

Tvix evaluator in devenv

Reverse engineering nix-daemon protocol

It's fun!

It's fun!

ThaigerSprint.org

12th - 19th February 2025, Chiang Mai

OceanSprint.org

17th - 21th March 2025, Lanzarote

Thank you. 

domen@cachix.org / @domenkozar

https://devenv.sh

https://tvix.dev

devenv is switching to tvix

By ielectric

Made with Slides.com

devenv is switching to tvix

  • 106

More from ielectric