Guix Alchemy

Customisation with package transformations

Mastering Guix Packaging

Rather than trying to learn it all pick a path
The main scenarios are
- Using an older package (downgrading)
- Rebuilding an existing package
- Modifying an existing package
- Creating a new package (with and without source)
- Contributing a package to the distribution

Package variant: the local build is different from the Guix archives

Learning Path

Profiles

Package

Commands

Shell

Command

Basic Build

Command

Manifests

Transform

Command

Manifests

Transforms in Manifests

Guix Build Overview

Guix build is used to build a package (or a derivation)
Receives:
- package definition
- Guix build options
It outputs:
- build logs
- build derivation
- package
Recommendation: use with guix shell for a clean build environment

$ guix build hello --no-substitutes --no-grafts
The following derivation will be built:
     /gnu/store/xn12lv3ckihkai4044p28h5im78lvad5-hello-2.12.1.drv
    
    [... lots of output as it builds the package ...]
    
    successfully built /gnu/store/xn12lv3ckihkai4044p28h5im78lvad5-hello-2.12.1.drv
	/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1

Option	Description
--dry-run	Give output but don't take the action
--no-substitutes	Don't use binary substitutes
--no-grafts	Do not put security grafts onto packages
--verbosity=<lvl>	0=no output; 1=quiet; 2=show urls; 3=build log
--cores=n	Use N CPU cores for each build job
--source	Build a source package
--max-jobs=N	Allow the Guix Daemon to run at most N build jobs. Each build can be limited to use --cores=N
--check

Common Build Options

1

Create build environment with guix shell

3

Create a test environment with guix shell and test the new build

2

Build the current package to test dependencies

Package build steps

$ guix shell --container --nesting --development hello \
  --network --no-grafts nss-certs
The following derivation will be built:
  /gnu/store/jaz9fabc7l9iwkxf648iq0ajafzgs049-profile.drv
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 23 packages...

Create a build environment

- nesting - run further package commands in the shell environment

- add nss-certs so that commands can use SSL

1 Rebuild the Package

$ guix shell --container --nesting --development hello \
  --network --no-grafts nss-certs
The following derivation will be built:
  /gnu/store/jaz9fabc7l9iwkxf648iq0ajafzgs049-profile.drv
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 23 packages...

[env]$ guix build --no-substitutes --no-grafts --source hello
The following derivation will be built:
  /gnu/store/2p9bkv91gr64nm17qq0byavwg5q6w2lg-hello-2.12.1.tar.gz.drv
building /gnu/store/2p9bkv91gr64nm17qq0byavwg5q6w2lg-hello-2.12.1.tar.gz.drv...

Starting download of /gnu/store/3dq55rw99wdc4g4wblz7xikc8a2jy7a3-hello-2.12.1.tar.gz
From https://ftpmirror.gnu.org/gnu/hello/hello-2.12.1.tar.gz...
following redirection to `https://mirrors.tripadvisor.com/gnu/hello/hello-2.12.1.tar.gz'...
downloading from https://ftpmirror.gnu.org/gnu/hello/hello-2.12.1.tar.gz ...
 hello-2.12.1.tar.gz  1009KiB                                                               1.6MiB/s 00:01 [##################] 100.0%
successfully built /gnu/store/2p9bkv91gr64nm17qq0byavwg5q6w2lg-hello-2.12.1.tar.gz.drv
/gnu/store/3dq55rw99wdc4g4wblz7xikc8a2jy7a3-hello-2.12.1.tar.gz

download the source and put it into the store.

2 Rebuild the Package












[env]$ guix build --no-substitutes --no-grafts --source hello
The following derivation will be built:
  /gnu/store/2p9bkv91gr64nm17qq0byavwg5q6w2lg-hello-2.12.1.tar.gz.drv
building /gnu/store/2p9bkv91gr64nm17qq0byavwg5q6w2lg-hello-2.12.1.tar.gz.drv...

Starting download of /gnu/store/3dq55rw99wdc4g4wblz7xikc8a2jy7a3-hello-2.12.1.tar.gz
From https://ftpmirror.gnu.org/gnu/hello/hello-2.12.1.tar.gz...
following redirection to `https://mirrors.tripadvisor.com/gnu/hello/hello-2.12.1.tar.gz'...
downloading from https://ftpmirror.gnu.org/gnu/hello/hello-2.12.1.tar.gz ...
 hello-2.12.1.tar.gz  1009KiB                                                               1.6MiB/s 00:01 [##################] 100.0%
successfully built /gnu/store/2p9bkv91gr64nm17qq0byavwg5q6w2lg-hello-2.12.1.tar.gz.drv
/gnu/store/3dq55rw99wdc4g4wblz7xikc8a2jy7a3-hello-2.12.1.tar.gz

[env]$ guix build --no-substitutes --no-grafts --verbosity=1 hello
The following derivation will be built:
  /gnu/store/3zh2qpi897s2x229s93iakji86b08a20-hello-2.12.1.drv

building /gnu/store/3zh2qpi897s2x229s93iakji86b08a20-hello-2.12.1.drv...
/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1

2

$ guix shell --container --nesting --no-grafts
guix shell: warning: no packages specified; creating an empty environment
The following derivation will be built:
  /gnu/store/jv8hws211gw8vbv9rcjnigl5xik1k8vq-profile.drv

building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...

Test the Package

- no network as we're installing locally

- no packages as we're just going to test it

- add testing packages through a manifest (for example coreutils)

Install the package into a test shell - use it!

🏆 TIP: by avoiding installing it into a standard profile we can delete it:

$ guix gc --delete /gnu/store/<blah>

OR use --check

3

$ guix shell --container --nesting --no-grafts
guix shell: warning: no packages specified; creating an empty environment
The following derivation will be built:
  /gnu/store/jv8hws211gw8vbv9rcjnigl5xik1k8vq-profile.drv

building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...

[env]$ guix shell --install  /gnu/store/3zh2qpi897s2x229s93iakji86b08a20-hello-2.12.1.drv
The following package will be installed:
   hello 2.12.1

The following derivation will be built:
  /gnu/store/kkf3dwfjy1mkll3m4wbcxx76v2p2yjyc-profile.drv

building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
hint: Consider setting the necessary environment variables by running:

     GUIX_PROFILE="/home/steve/.guix-profile"
     . "$GUIX_PROFILE/etc/profile"

Alternately, see `guix package --search-paths -p "/home/steve/.guix-profile"'.

$ /home/steve/.guix-profile/bin/hello
Hello,world!

Test the Package

Success!

3 VIDEO

Guix Build

2:40 minutes

1

2

3 Package Transformations

Modify an existing package
Create a package variant by alterating
Elements we can change
- Source
  - latest available
  - specific version
  - git repository
- Inputs
- Build process
- Source alterations - patches
- Tuning for specific hardware

The package transformation can take place:
- using the cli tools
- using a package definition
- using a manifest

CLI Package Transformations

$ guix build calcurse --with-source=https://calcurse.org/files/calcurse-4.8.1.tar.gz \
      --no-substitutes --no-grafts

package to build

source URL

$ guix build calcurse --with-source=calcurse@4.8.1.01=https://calcurse.org/files/calcurse-4.8.1.tar.gz \
      --no-substitutes --no-grafts

normal build options

version to use

Option	Description
with-source	Build the package using a specified source. URL or file. Can be either source for the build, or an input.
with-input	Use a specified package as an input to the build.

with-git-url	The same as with-source but use a specified git URL. Builds from HEAD, unless --with-commit is used.
with-commit	Use the a set commit
with-branch	Build a particular branch, default is master

with-latest	Build a package using the latest upstream release of the source that the importer can detect.
with-version	Use a specific release of the upstream source release.

with-configure-flags	Customise by adding configure flags
with-patch	Customise the build's source with a patch

Transformation Options

1

Create build environment with guix shell

3

Create a test environment with guix shell and test the new build

2

Build the current package to test dependencies

Package build steps

1

Create build environment with guix shell

3

Build the package using a transformation

2

Build the current package to test dependencies

4

Install the new package into a new test environment

Package build steps

$ guix shell --development fzf --container --nesting \
    --network --no-grafts nss-certs
The following derivation will be built:
  /gnu/store/fgc04pns05wp0yxwmm7xnvkpcybs3k1c-profile.drv

2.0 MB will be downloaded
 go-github-com-mattn-go-isatty-0.0.11  4KiB          201KiB/s 00:00 ▕██████████████████▏ 100.0%
 go-golang-org-x-crypto-0.4.0  1.4MiB                817KiB/s 00:02 ▕██████████████████▏ 100.0%
 go-golang-org-x-sys-0.8.0-0.ca59eda  569KiB          90KiB/s 00:06 ▕██████████████████▏ 100.0%
 go-golang-org-x-term-0.3.0  13KiB                    74KiB/s 00:00 ▕██████████████████▏ 100.0%
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 37 packages...

Git Package Transformation

- development: all the pkgs needed to build

- nesting: so we can run commands

- nss-certs pkg added for downloads

[env]$ guix build fzf --with-git-url=fzf=https://github.com/junegunn/fzf \
  --with-commit=fzf=0.44.1 --no-substitutes --no-grafts
updating checkout of 'https://github.com/junegunn/fzf'...
retrieved commit d7d2ac39513f2068f70fbe717d212c9bce8750ed
The following derivation will be built:
  /gnu/store/ix845m79aw8l08h7d7737x2gl0pn25mn-fzf-0.44.1.drv
building /gnu/store/ix845m79aw8l08h7d7737x2gl0pn25mn-fzf-0.44.1.drv...

[... lots of build output ...]

successfully built /gnu/store/ix845m79aw8l08h7d7737x2gl0pn25mn-fzf-0.44.1.drv
/gnu/store/ihlh8nip211l2lm6p3h0502jwbpzrps1-fzf-0.44.1

Package to build

Package to build from git

- the package itself

- input to package

Commit to use

- commit hash

- tag

Git repository URL

VIDEO

Guix Transformation

~ minutes
TBD

Transformations in Manifests

A manifest contains a collection of packages
Often used to declaratively install packages into a profile
Actually manifests are code
- GNU Guile
- Uses lots Guix provided functions
Guile is in the Scheme family
- A lot of parentheses
- Be a rebel and space them vertically if it helps!
For the most part consider it a DSL





(specifications->manifest (list "calcurse"))

Package Transformations

(specifications->manifest 
	(list "calcurse" "cbonsai@1.3.1" "git:send-email")
)

Lists hold multiple things

Different package specs using their names

$ guix shell --container --nesting --manifest=basic-manifest.scm
The following derivation will be built:
  /gnu/store/4hr96x21bs1w23gfp6ynlghkfiav1zh6-profile.drv

applying 4 grafts for calcurse-4.5.1 ...
applying 3 grafts for cbonsai-1.3.1 ...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
generating GLib schema cache...
building directory of Info manuals...
building XDG desktop file cache...
building XDG MIME database...
building profile with 4 packages...

[env]$ guix package --list-installed --profile=$GUIX_ENVIRONMENT
guix      31e736...  out         /gnu/store/caqz00i1b1qwhlsb7v5j6lgr5l9x78n8-profile
git       2.41.0     send-email  /gnu/store/gdh0p...p14n6fsj1gz04mx-git-2.41.0-send-email
cbonsai   1.3.1      out         /gnu/store/zbjphbd9v7rbadv2y8jb7k9qqvzf5s4q-cbonsai-1.3.1
calcurse  4.5.1      out         /gnu/store/1bgjd0l2dlpjj29x8faik724xz2jzivn-calcurse-4.5.1

A function that's called with the provided list

The Lisperati will hunt you down for this!!! It is evil OMG!!!!!

Create a manifest called basic-manifest.scm:

works with guix shell and guix package

Reading from inside to the outside often helps
Lines 1-3: Guile code can contain modules
- The profile and transformations provide the Guix functions
- We need (gnu packages calcurse) so we can call the calcurse package
Lines 10: calcurse-transform the calcurse variable
- Next it's formed into manifest by package->manifest
Line 7-8: calcurse-transform
- calcurse-src is evaluated
- calcurse-src is transformed with options->transformation
Line 5: calcurse-src is a Guile pair consisting of the 'with-source' transformation and a location for the code

Basic manifest transformation

(use-modules (guix profile)
             (guix transformations)
             (gnu packages calcurse))
             
(define calcurse-src '(with-source . "https://calcurse.org/files/calcurse-4.8.1.tar.gz"))

(define calcurse-transform
	(options->transformation (list calcurse-src)))
    
(package->manifest (list (calcurse-transform calcurse)))

1

Create build environment with guix shell

3

Build the package using the transformation

2

Build the current package to test dependencies

4

Install the new package into a new test environment

Package build steps

1

Create build environment with guix shell

3

Create a manifest

Install the new package into a new test environment

2

Build the current package to test dependencies

4

Build the package using the transformation

Package build steps

5 Git transformation

Conceptually the same as previously:
- we create a transformation and apply it to the package
- we convert from the transformed package to a manifest
A bit more condensed as we're now familiar with the format
Uses a list of git transformations

(use-modules (guix profiles)
             (guix transformations)
             (gnu packages terminals))
             
(define fzf-transform (options->transformation
							(list '(with-git-url . "fzf=https://github.com/junegunn/fzf")
                                  '(with-commit . "fzf=2024010"))))

(packages->manifest (list (fzf-transform fzf)))

Advanced: multiple packages

Multiple packages and multiple transformations ...

4 and 8: two transforms
14-15: two transformations, converted into a list
13: a separate list of packages
12: add the transforms to the first list creating a big list

(use-modules (guix profiles)
             (guix transformations))
 
(define fzf-transform (options->transformation
                        (list '(with-git-url . "fzf=https://github.com/junegunn/fzf")
                              '(with-commit . "fzf=2024010"))))
 
 (define calcurse-transform (options->transformation
       '((with-source . "calcurse@20231207=https://calcurse.org/files/calcurse-4.8.1.tar.gz"))))
 
 (packages->manifest
  (append
    (specifications->packages (list "hello" "cbonsai" "gnupg"))
    (list (fzf-transform (specification->package "fzf"))
          (calcurse-transform (specification->package "calcurse")))))

The equivalent of: guix shell --development mutt vim git

8-9: normal specification of packages
6-7: converts to a development manifest
5: create a list
4: turn into a manifest

Advanced: multiple manifests

(use-modules (guix profiles)
             (guix transformations))
 
(concatenate-manifests
  (list
    (package->development-manifest
      (specification->package "mutt"))
    (packages->manifest
      (specifications->packages (list "vim" "git")))))

Development manifest to install all required libraries ...

Advanced: package name

What we did so far was create the same package but with some different inputs
This time we're doing a package inheritance
- inheritance uses Scheme records - bottom line we can change some fields
- We run the transform which outputs a package
- We then use package/inherit to change some fields: technically this is a new package

(use-modules (guix profiles)
             (guix transformations)
             (guix packages)
             (gnu packages calcurse))

(define calcurse-transform
    ((options->transformation
        (list '(with-source . "calcurse=https://calcurse.org/files/calcurse-4.5.0.tar.gz")))
      calcurse))

(define calcurse-local (package/inherit
                         calcurse-transform
                         (name "calcurse-local")
                         (version "4.5.0")))

(packages->manifest (list calcurse-local))

Give the package variant a different name ...

What we've seen

Package transformations give us sophisticated capabilities to build package variants

1 Transforms

Manifests are a declarative way to manage an environment

2 Manifests

Transforms and manifests together give us a lot of flexibility and repeatability

3 Alchemy

Package Transformation Alchemy

Package transformations let us create locally build package variants
Implications
- They use the package definition (recipe) that's in the archive
- Only used to alter packages, not to create new ones
- The source location needs to be processable by the package definition
- The source can't have changed in some way the package definition isn't expecting
Uncomfortable tracking which packages have been altered

Taking it further:

Gentoo Use flags are a more flexible system
Flags have to be exposed by the packager
Examples: https://www.gentoo.org/support/use-flags/
GSoC 2023: https://blog.lispy.tech/2023/05/parameterized-packages.html

Steps from here

Profiles

Package

Commands

Shell

Command

Basic Build

Command

Manifests

Transform

Command

Manifests

Transforms in Manifests

Package Definitions

Creating new packages

Thank You!

Questions?

1

Create build environment with guix shell

3

Build the package using a transformation

2

Build the current package to test dependencies

4

Install the new package into a new test environment

Package build steps

Reading from inside to the outside often helps
Lines 1-3: Guile code can contain modules
- The profile and transformations provide the Guix functions
- We need (gnu packages calcurse) so we can call the calcurse package
Lines 10: calcurse-transform the calcurse variable
- Next it's formed into manifest by package->manifest
Line 7-8: calcurse-transform
- calcurse-src is evaluated
- calcurse-src is transformed with options->transformation
Line 5: calcurse-src is a Guile pair consisting of the 'with-source' transformation and a location for the code

Basic manifest transformation

(use-modules (guix profile)
             (guix transformations)
             (gnu packages calcurse))
             
(define calcurse-src '(with-source . "https://calcurse.org/files/calcurse-4.8.1.tar.gz"))

(define calcurse-transform
	(options->transformation (list calcurse-src)))
    
(package->manifest (list (calcurse-transform calcurse)))

Palette

By futurile

Guix Alchemy

Mastering Guix Packaging

Learning Path

Guix Build Overview

Common Build Options

1

3

2

Package build steps

Create a build environment

1

Rebuild the Package

2

Rebuild the Package

2

Test the Package

3

Test the Package

3

VIDEO

Guix Build

1

2

3

Package Transformations

CLI Package Transformations

Transformation Options

1

3

2

Package build steps

1

3

2

4

Package build steps

Git Package Transformation

VIDEO

Guix Transformation

Transformations in Manifests

Package Transformations

Basic manifest transformation

1

3

2

4

Package build steps

1

3

2

4

Package build steps

5

Git transformation

Advanced: multiple packages

Advanced: multiple manifests

Advanced: package name

What we've seen

1

Transforms

2

Manifests

3

Alchemy

Package Transformation Alchemy

Steps from here

Thank You!

Questions?

1

3

2

4

Package build steps

Basic manifest transformation

Palette

More from futurile