Thursday, August 28, 2014

Nix pill 12: the inputs design pattern

Welcome to the 12th Nix pill. In the previous 11th pill we stopped packaging and cleaned up the system with the garbage collector.

We restart our packaging, but we will improve a different aspect. We only packaged an hello world program so far, what if we want to create a repository of multiple packages?

Repositories in Nix


Nix is a tool for build and deployment, it does not enforce any particular repository format. A repository of packages is the main usage for Nix, but not the only possibility. See it more like a consequence due to the need of organizing packages.
Nix is a language, and it is powerful enough to let you choose the format of your own repository. In this sense, it is not declarative, but functional.
There is no preset directory structure or preset packaging policy. It's all about you and Nix.

The nixpkgs repository has a certain structure, which evolved and evolves with the time. Like other languages, Nix has its own history and therefore I'd like to say that it also has its own design patterns. Especially when packaging, you often do the same task again and again except for different software. It's inevitable to identify patterns during this process. Some of these patterns get reused if the community thinks it's a good way to package the software.

Some of the patterns I'm going to show do not apply only to Nix, but to other systems of course.

The single repository pattern


Before introducing the "inputs" pattern, we can start talking about another pattern first which I'd like to call "single repository" pattern.
Systems like Debian scatter packages in several small repositories. Personally, this makes it hard to track interdependent changes and to contribute to new packages.
Systems like Gentoo instead, put package descriptions all in a single repository.

The nix reference for packages is nixpkgs, a single repository of all descriptions of all packages. I find this approach very natural and attractive for new contributions.

From now on, we will adopt this technique. The natural implementation in Nix is to create a top-level Nix expression, and one expression for each package. The top-level expression imports and combines all expressions in a giant attribute set with name -> package pairs.

But isn't that heavy? It isn't, because Nix is a lazy language, it evaluates only what's needed! And that's why nixpkgs is able to maintain such a big software repository in a giant attribute set.

Packaging graphviz


We have packaged GNU hello world, I guess you would like to package something else for creating at least a repository of two projects :-) . I chose graphviz, which uses the standard autotools build system, requires no patching and dependencies are optional.

Download graphviz from here. The graphviz.nix expression is straightforward:
let
  pkgs = import  {};
  mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
}
Build with nix-build graphviz.nix and you will get runnable binaries under result/bin. Notice how we did reuse the same autotools.nix of hello.nix. Let's create a simple png:
$ echo 'graph test { a -- b }'|result/bin/dot -Tpng -o test.png
Format: "png" not recognized. Use one of: canon cmap [...]
Oh of course... graphviz can't know about png. It built only the output formats it supports natively, without using any extra library.

I remind you, in autotools.nix there's a buildInputs variablewhich gets concatenated to baseInputs.  That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions.

This 2.38 version of graphviz has several plugins to output png. For simplicity, we will use libgd.

Digression about gcc and ld wrappers


The gd, jpeg, fontconfig and bzip2 libraries (dependencies of gd) don't use pkg-config to specify which flags to pass to the compiler. Since there's no global location for libraries, we need to tell gcc and ld where to find includes and libs.

The nixpkgs provides gcc and binutils, and we are using them for our packaging. Not only, it also provides wrappers for them which allow passing extra arguments to gcc and ld, bypassing the project build systems:
  • NIX_CFLAGS_COMPILE: extra flags to gcc at compile time
  • NIX_LDFLAGS: extra flags to ld
What can we do about it? We can employ the same trick we did for PATH: automatically filling the variables from buildInputs. This is the relevant snippet of setup.sh:
for p in $baseInputs $buildInputs; do
  if [ -d $p/bin ]; then
    export PATH="$p/bin${PATH:+:}$PATH"
  fi
  if [ -d $p/include ]; then
    export NIX_CFLAGS_COMPILE="-I $p/include${NIX_CFLAGS_COMPILE:+ }$NIX_CFLAGS_COMPILE"
  fi
  if [ -d $p/lib ]; then
    export NIX_LDFLAGS="-rpath $p/lib -L $p/lib${NIX_LDFLAGS:+ }$NIX_LDFLAGS"
  fi
done
Now by adding derivations to buildInputs, will add the lib, include and bin paths automatically in setup.sh.

The -rpath flag in ld is needed because at runtime, the executable must use exactly that version of the library.
If unneeded paths are specified, the fixup phase will shrink the rpath for us!

Completing graphviz with gd


Finish the expression for graphviz with gd support (note the use of the with expression in buildInputs to avoid repeating pkgs):
let
  pkgs = import  {};
  mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
  buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}
Now you can create the png! Ignore any error from fontconfig, especially if you are in a chroot.

The repository expression


Now that we have two packages, what's a good way to put them together in a single repository? We do something like nixpkgs does. With nixpkgs, we import it and then we peek derivations by accessing the giant attribute set.
For us nixers, this a good technique, because it abstracts from the file names. We don't refer to a package by REPO/some/sub/dir/package.nix but by importedRepo.package (or pkgs.package in our examples).

Create a default.nix in the current directory:
{
  hello = import ./hello.nix; 
  graphviz = import ./graphviz.nix;
}
Ready to use! Try it with nix-repl:
$ nix-repl
nix-repl> :l default.nix
Added 2 variables.
nix-repl> hello
«derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv»
nix-repl> graphviz
«derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv»
With nix-build:
$ nix-build default.nix -A hello
[...]
$ result/bin/hello
Hello, world!
The -A argument is used to access an attribute of the set from the given .nix expression.
Important: why did we choose the default.nix? Because when a directory (by default the current directory) has a default.nix, that default.nix will be used (see import here). In fact you can run nix-build -A hello without specifying default.nix. 
For pythoners, it is similar to __init__.py.

With nix-env, to install the package in your user environment:
$ nix-env -f . -iA graphviz
[...]
$ dot -V
The -f option is used to specify the expression to use, in this case the current directory, therefore ./default.nix.
The -i stands for installation.
The -A is the same as above for nix-build.

We reproduced the very basic behavior of nixpkgs.

The inputs pattern


After a long preparation, we finally arrived. I know you have a big doubt in this moment. It's about the hello.nix and graphviz.nix. They are very, very dependent on nixpkgs:
  • First big problem: they import nixpkgs directly. In autotools.nix instead we pass nixpkgs as an argument. That's a much better approach.
  • Second problem: what if we want a variant of graphviz without libgd support?
  • Third problem: what if we want to test graphviz with a particular libgd version?
The current answer to the above questions is: change the expression to match your needs (or change the callee to match your needs).
With the inputs pattern, we choose to give another answer: let the user change the inputs of the expression (or change the caller to pass different inputs).

By inputs of an expression, we refer to the set of derivations needed to build that expression. In this case:
  • mkDerivation from autotools. Recall that mkDerivation has an implicit dependency on the toolchain.
  • libgd and its dependencies.
The src is also an input but it's pointless to change the source from the caller. For version bumps, in nixpkgs we prefer to write another expression (e.g. because patches are needed or different inputs are needed).

Goal: make package expressions independent of the repository.

How do we achieve that? The answer is simple: use functions to declare inputs for a derivation. Doing it for graphviz.nix, will make the derivation independent of the repository and customizable:
{ mkDerivation, gdSupport ? true, gd, fontconfig, libjpeg, bzip2 }:

mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
  buildInputs = if gdSupport then [ gd fontconfig libjpeg bzip2 ] else [];
}
I recall that "{...}: ..." is the syntax for defining functions accepting an attribute set as argument.

We made gd and its dependencies optional. If gdSupport is true (by default), we will fill buildInputs and thus graphviz will be built with gd support, otherwise it won't.
Now back to default.nix:
let
  pkgs = import  {};
  mkDerivation = import ./autotools.nix pkgs;
in with pkgs; {
  hello = import ./hello.nix { inherit mkDerivation; }; 
  graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
  graphvizCore = import ./graphviz.nix {
    inherit mkDerivation gd fontconfig libjpeg bzip2;
    gdSupport = false;
  };
}
So we factorized the import of nixpkgs and mkDerivation, and also added a variant of graphviz with gd support disabled. The result is that both hello.nix (exercise for the reader) and graphviz.nix are independent of the repository and customizable by passing specific inputs.
If you wanted to build graphviz with a specific version of gd, it would suffice to pass gd = ...;.
If you wanted to change the toolchain, you may pass a different mkDerivation function.

Clearing up the syntax:
  • In the end we return an attribute set from default.nix. With "let" we define some local variables.
  • We bring pkgs into the scope when defining the packages set, which is very convenient instead of typing everytime "pkgs".
  • We import hello.nix and graphviz.nix, which will return a function, and call it with a set of inputs to get back the derivation.
  • The "inherit x" syntax is equivalent to "x = x". So "inherit gd" here, combined to the above "with pkgs;" is equivalent to "x = pkgs.gd".
You can find the whole repository at the pill 12 gist.

Conclusion


The "inputs" pattern allows our expressions to be easily customizable through a set of arguments. These arguments could be flags, derivations, or whatelse. Our package expressions are functions, don't think there's any magic in there.

It also makes the expressions independent of the repository. Given that all the needed information is passed through arguments, it is possible to use that expression in any other context.

Next pill


...we will talk about the "callPackage" design pattern. It is tedious to specify the names of the inputs twice, once in the top-level default.nix, and once in the package expression. With callPackage, we will implicitly pass the necessary inputs from the top-level expression.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Thursday, August 21, 2014

Nix pill 11: the garbage collector

Welcome to the 11th Nix pill. In the previous 10th pill we managed to obtain a self-contained environment for developing a project. The concept is that whereas nix-build is able to build a derivation in isolation, nix-shell is able to drop us in a shell with (almost) the same environment used by nix-build. This allows us to debug, modify and manually build software.

Today we stop packaging and look at a mandatory nix component, the garbage collector. When using nix tools, often derivations are built. This include both .drv files and out paths. These artifacts go in the nix store, and we never cared about deleting them until now.

How does it work


Other package managers, like dpkg, have somehow a way to remove unused software. However, nix is much more precise compared to other systems.
I bet with dpkg, rpm or anything else, you end up with having some unnecessary package installed or dangling files. With nix this does not happen.

How do we determine whether a store path is still needed? The same way programming languages with a garbage collector decide whether an object is still alive.

Programming languages with a garbage collector have an important concept in order to keep track of live objects: GC roots. A GC root is an object that is always alive (unless explicitly removed as GC root). All objects recursively referred to by a GC root are live.

Therefore, the garbage collection process starts from GC roots, and recursively mark referenced objects as live. All other objects can be collected and deleted.

In Nix there's this same concept. Instead of being objects, of course, GC roots are store paths. The implementation is very simple and transparent to the user. GC roots are stored under /nix/var/nix/gcroots. If there's a symlink to a store path, then that store path is a GC root.
Nix allows this directory to have subdirectories: it will simply recurse directories in search of symlinks to store paths.

So we have a list of GC roots. At this point, deleting dead store paths is as easy as you can imagine. We have the list of all live store paths, hence the rest of the store paths are dead.
In particular, Nix first moves dead store paths to /nix/store/trash which is an atomic operation. Afterwards, the trash is emptied.

Playing with the GC


Before playing with the GC, first run the nix garbage collector once, so that we have a cleaned up playground for our experiments:
$ nix-collect-garbage
finding garbage collector roots...
[...]
deleting unused links...
note: currently hard linking saves -0.00 MiB
1169 store paths deleted, 228.43 MiB freed
Perfect, if you run it again it won't find anything new to delete, as expected.
What's left in the nix store is everything being referenced from the GC roots.

Let's install for a moment bsd-games:
$ nix-env -iA nixpkgs.bsdgames
$ readlink -f `which fortune`
/nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17/bin/fortune
$ nix-store -q --roots `which fortune`
/nix/var/nix/profiles/default-9-link
$ nix-env --list-generations
[...]
   9   2014-08-20 12:44:14   (current)
The nix-store command can be used to query the GC roots that refer to a given derivation. In this case, our current user environment does refer to bsd-games.

Now remove it, collect garbage and note that bsd-games is still in the nix store:
$ nix-env -e bsd-games
uninstalling `bsd-games-2.17'
$ nix-collect-garbage
[...]
$ ls /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17
bin  share
That's because the old generation is still in the nix store because it's a GC root. As we'll see below, all profiles and their generations are GC roots.

Removing a GC root is simple. Let's try deleting the generation that refers to bsd-games, collect garbage, and note that now bsd-games is no longer in the nix store:
$ rm /nix/var/nix/profiles/default-9-link
$ nix-env --list-generations
[...]
   8   2014-07-28 10:23:24   
  10   2014-08-20 12:47:16   (current)
$ nix-collect-garbage
[...]
$ ls /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17
ls: cannot access /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17: No such file or directory
Note: nix-env --list-generations does not rely on any particular metadata. It is able to list generations based solely on the file names under the profiles directory.

However we removed the link from /nix/var/nix/profiles, not from /nix/var/nix/gcroots. Turns out, that /nix/var/nix/gcroots/profiles is a symlink to /nix/var/nix/profiles. That is very handy. It means any profile and its generations are GC roots.

It's as simple as that, anything under /nix/var/nix/gcroots is a GC root. And anything not being garbage collected is because it's referred from one of the GC roots.

Indirect roots


I remind you that building the GNU hello world package with nix-build produces a result symlink in the current directory. Despite the collected garbage done above, the hello program is still working: therefore it has not been garbage collected. Clearly, since there's no other derivation that depends upon the GNU hello world package, it must be a GC root.

In fact, nix-build automatically adds the result symlink as a GC root. Yes, not the built derivation, but the symlink. These GC roots are added under /nix/var/nix/gcroots/auto .
$ ls -l /nix/var/nix/gcroots/auto/
total 8
drwxr-xr-x 2 nix nix 4096 Aug 20 10:24 ./
drwxr-xr-x 3 nix nix 4096 Jul 24 10:38 ../
lrwxrwxrwx 1 nix nix   16 Jul 31 10:51 xlgz5x2ppa0m72z5qfc78b8wlciwvgiz -> /home/nix/result/
Don't care about the name of the symlink. What's important is that a symlink exists that point to /home/nix/result. This is called an indirect GC root. That is, the GC root is effectively specified outside of /nix/var/nix/gcroots. Whatever result points to, it will not be garbage collected.

How do we remove the derivation then? There are two possibilities:
  • Remove the indirect GC root from /nix/var/nix/gcroots/auto.
  • Remove the result symlink.
In the first case, the derivation will be deleted from the nix store, and result becomes a dangling symlink. In the second case, the derivation is removed as well as the indirect root in /nix/var/nix/gcroots/auto .
Running nix-collect-garbage after deleting the GC root or the indirect GC root, will remove the derivation from the store.

Cleanup everything


What's the main source of software duplication in the nix store? Clearly, GC roots due to nix-build and profile generations. Doing a nix-build results in a GC root for a build that somehow will refer to a specific version of glibc, and other libraries. After an upgrade, if that build is not deleted by the user, it will not be garbage collected. Thus the old dependencies referred to by the build will not be deleted either.

Same goes for profiles. Manipulating the nix-env profile will create further generations. Old generations refer to old software, thus increasing duplication in the nix store after an upgrade.

What are the basic steps for upgrading and removing everything old, including old generations? In other words, do an upgrade similar to other systems, where they forget everything about the older state:
$ nix-channel --update
$ nix-env -u
$ rm /nix/var/nix/gcroots/auto/*
$ nix-collect-garbage -d
First, we download a new version of the nixpkgs channel, which holds the description of all the software. Then we upgrade our installed packages with nix-env -u. That will bring us into a fresh new generation with all updated software.

Then we remove all the indirect roots generated by nix-build: beware, this will result in dangling symlinks. You may be smarter and also remove the target of those symlinks.

Finally, the -d option of nix-collect-garbage is used to delete old generations of all profiles, then collect garbage. After this, you lose the ability to rollback to any previous generation. So make sure the new generation is working well before running the command.

Conclusion


Garbage collection in Nix is a powerful mechanism to cleanup your system. The nix-store commands allows us to know why a certain derivation is in the nix store.

Cleaning up everything down to the oldest bit of software after an upgrade seems a bit contrived, but that's the price of having multiple generations, multiple profiles, multiple versions of software, thus rollbacks etc.. The price of having many possibilities.

Next pill


...we will package another project and introduce what I call the "inputs" design pattern. We only played with a single derivation until now, however we'd like to start organizing a small repository of software. The "inputs" pattern is widely used in nixpkgs; it allows us to decouple derivations from the repository itself and increase customization opportunities.

Pill 12 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Tuesday, August 19, 2014

Nix pill 10: developing with nix-shell

Welcome to the 10th Nix pill. In the previous 9th pill we have seen one of the powerful features of nix, automatic discovery of runtime dependencies and finalized the GNU hello world package.

In return from vacation, we want to hack a little the GNU hello world program. The nix-build tool creates an isolated environment for building the derivation, we want to do the same in order to modify some source files of the project.

What's nix-shell


The nix-shell tool drops us in a shell by setting up the necessary environment variables to hack a derivation. It does not build the derivation, it only serves as a preparation so that we can run the build steps manually.
I remind you, in a nix environment you don't have access to libraries and programs unless you install them with nix-env. However installing libraries with nix-env is not good practice. We prefer to have isolated environments for development.
$ nix-shell hello.nix
[nix-shell]$ make
bash: make: command not found
[nix-shell]$ echo $baseInputs
/nix/store/jff4a6zqi0yrladx3kwy4v6844s3swpc-gnutar-1.27.1 [...]
First thing to notice, we call nix-shell on a nix expression which returns a derivation. We then enter a new bash shell, but it's really useless. We expected to have the GNU hello world build inputs available in PATH, including GNU make, but it's not the case.
But, we have the environment variables that we set in the derivation, like $baseInputs, $buildInputs, $src and so on.

That means we can source our builder.sh, and it will build the derivation. You may get an error in the installation phase, because the user may not have the permission to write to /nix/store:
[nix-shell]$ source builder.sh
...
It didn't install, but it built. Things to notice:
  • We sourced builder.sh, therefore it ran all the steps including setting up the PATH for us.
  • The working directory is no more a temp directory created by nix-build, but the current directory. Therefore, hello-2.9 has been unpacked there.
We're able to cd into hello-2.9 and type make, because now it's available.

In other words, nix-shell drops us in a shell with the same (or almost) environment used to run the builder!

A builder for nix-shell


The previous steps are a bit annoying of course, but we can improve our builder to be more nix-shell friendly.

First of all, we were able to source builder.sh because it was in our current directory, but that's not nice. We want the builder.sh that is stored in the nix store, the one that would be used by nix-build. To do so, the right way is to pass the usual environment variable through the derivation.
Note: $builder is already defined, but it's the bash executable, not our builder.sh. Our builder.sh is an argument to bash.

Second, we don't want to run the whole builder, we only want it to setup the necessary environment for manually building the project. So we'll write two files, one for setting up the environment, and the real builder.sh that runs with nix-build.

Additionally, we'll wrap the phases in functions, it may be useful, and move the set -e to the builder instead of the setup. The set -e is annoying in nix-shell.

The codebase is becoming a little long. You can find all the files in this nixpill10 gist.
Noteworthy is the setup = ./setup.sh; attribute in the derivation, which adds setup.sh to the nix store and as usual, adds a $setup environment variable in the builder.
Thanks to that, we can split builder.sh into setup.sh and builder.sh. What builder.sh does is sourcing $setup and calling the genericBuild function. Everything else is just some bash changes.

Now back to nix-shell:
$ nix-shell hello.nix
[nix-shell]$ source $setup
[nix-shell]$
Now you can run, for example, unpackPhase which unpacks $src and enters the directory. And you can run commands like ./configure, make etc. manually, or run phases with their respective functions.

It's all that straight, nix-shell builds the .drv file and its input dependencies, then drops into a shell by setting up the environment variables necessary to build the .drv, in particular those passed to the derivation function.

Conclusion


With nix-shell we're table to drop into an isolated environment for developing a project, with the necessary dependencies just like nix-build does, except we can build and debug the project manually, step by step like you would do in any other operating system.
Note that we did never install gcc, make, etc. system-wide. These tools and libraries are available per-build.


Next pill


...we will clean up the nix store. We wrote and built derivations, added stuff to nix store, but until now we never worried about cleaning up the used space in the store. It's time to collect some garbage.

Pill 11 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Friday, August 08, 2014

Nix pill 9: automatic runtime dependencies

Welcome to the 9th Nix pill. In the previous 8th pill we wrote a generic builder for autotools projects. We feed build dependencies, a source tarball, and we get a Nix derivation as a result.

Today we stop by the GNU hello world program to analyze build and runtime dependencies, and enhance the builder in order to avoid unnecessary runtime dependencies.

Build dependencies


Let's start analyzing build dependencies for our GNU hello world package:
$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
$ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
/nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.9.tar.gz
/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv
/nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-gnused-4.2.2.drv
/nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-gnutar-1.27.1.drv
/nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-builder.sh
/nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-gnugrep-2.14.drv
/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv
/nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-gnumake-3.82.drv
/nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-gawk-4.1.0.drv
/nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-binutils-2.23.1.drv
/nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-gcc-wrapper-4.8.3.drv
/nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-gzip-1.6.drv
It has exactly the derivations referenced in the derivation function, nothing more, nothing less. Some of them might not be used at all, however given that our generic mkDerivation function always pulls such dependencies (think of it like build-essential of Debian), for every package you build from now on, you will have these packages in the nix store.

Why are we looking at .drv files? Because the hello.drv file is the representation of the build action to perform in order to build the hello out path, and as such it also contains the input derivations needed to be built before building hello.

Digression about NAR files


NAR is the Nix ARchive. First question: why not tar? Why another archiver? Because commonly used archivers are not deterministic. They add padding, they do not sort files, they add timestamps, etc.. Hence NAR, a very simple deterministic archive format being used by Nix for deployment.
NARs are also used extensively within Nix itself as we'll see below.

For the rationale and implementation details you can find more in the Dolstra's PhD Thesis.

To create NAR archives, it's possible to use nix-store --dump and nix-store --restore. Those two commands work regardless of /nix/store.

Runtime dependencies


Something is different for runtime dependencies however. Build dependencies are automatically recognized by Nix once they are used in any derivation call, but we never specify what are the runtime dependencies for a derivation.

There's really black magic involved. It's something that at first glance makes you think "no, this can't work in the long term", but at the same it works so well that a whole operating system is built on top of this magic.

In other words, Nix automatically computes all the runtime dependencies of a derivation, and it's possible thanks to the hash of the store paths.

Steps:
  1. Dump the derivation as NAR, a serialization of the derivation output. Works fine whether it's a single file or a directory.
  2. For each build dependency .drv and its relative out path, search the contents of the NAR for this out path.
  3. If found, then it's a runtime dependency.

You get really all the runtime dependencies, and that's why Nix deployments are so easy.
$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
$ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
$ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
Ok glibc and gcc. Well, gcc really should not be a runtime dependency!
$ strings result/bin/hello|grep gcc
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3/lib64
Oh Nix added gcc because its out path is mentioned in the "hello" binary. Why is that? That's the ld rpath. It's the list of directories where libraries can be found at runtime. In other distributions, this is usually not abused. But in Nix, we have to refer to particular versions of libraries, thus the rpath has an important role.

The build process adds that gcc lib path thinking it may be useful at runtime, but really it's not. How do we get rid of it? Nix authors have written another magical tool called patchelf, which is able to reduce the rpath to the paths that are really used by the binary.

Not only, even after reducing the rpath the hello binary would still depend upon gcc. Because of debugging information. For that, the well known strip can be used.

Another phase in the builder


We will add a new phase to our autotools builder. The builder has these phases already:
  1. First the environment is set up
  2. Unpack phase: we unpack the sources in the current directory (remember, Nix changes dir to a temporary directory first)
  3. Change source root to the directory that has been unpacked
  4. Configure phase: ./configure
  5. Build phase: make
  6. Install phase: make install
We add a new phase after the installation phase, which we call fixup phase. At the end of the builder.sh follows:
find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null
That is, for each file we run patchelf --shrink-rpath and strip. Note that we used two new commands here, find and patchelf. These two deserve a place in baseInputs of autotools.nix as findutils and patchelf.

Rebuild hello.nix and...:
$ nix-build hello.nix
[...]
$ nix-store -q --references result
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
/nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello
...only glibc is the runtime dependency. Exactly what we wanted.

The package is self-contained, copy its closure on another machine and you will be able to run it. I remind you the very few components under the /nix/store necessary to run nix when we installed it. The hello binary will use that exact version of glibc library and interpreter, not the system one:
$ ldd result/bin/hello
 linux-vdso.so.1 (0x00007fff11294000)
 libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f7ab7362000)
 /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f7ab770f000)
Of course, the executable runs fine as long as everything is under the /nix/store path.

Conclusion


Short post compared to previous ones as I'm still on vacation, but I hope you enjoyed it. Nix provides tools with cool features. In particular, Nix is able to compute all runtime dependencies automatically for us. Not only shared libraries, but also referenced executables, scripts, Python libraries etc..

This makes packages self-contained, because we're sure (apart data and configuration) that copying the runtime closure on another machine is sufficient to run the program. That's why Nix has one-click install, or reliable deployment in the cloud. All with one tool.


Next pill


...we will introduce nix-shell. With nix-build we build derivations always from scratch: the source gets unpacked, configured, built and installed. But this may take a long time, think of WebKit. What if we want to apply some small changes and compile incrementally instead, yet keeping a self-contained environment similar to nix-build?

Pill 10 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Friday, August 01, 2014

Nix pill 8: generic builders

Welcome to the 8th Nix pill. In the previous 7th pill we successfully built a derivation. We wrote a builder script that compiled a C file and installed the binary under the nix store.

In this post, we will generalize the builder script, write a Nix expression for GNU hello world and create a wrapper around the derivation built-in function.

Packaging GNU hello world


In the previous pill we packaged a simple .c file, which was being compiled with a raw gcc call. That's not a good example of project. Many use autotools, and since we're going to generalize our builder, better do it with the most used build system.

GNU hello world, despite its name, is a simple yet complete project using autotools. Fetch the latest tarball here: http://ftp.gnu.org/gnu/hello/hello-2.9.tar.gz .

Let's create a builder script for GNU hello world:
hello_builder.sh 
export PATH="$gnutar/bin:$gcc/bin:$gnumake/bin:$coreutils/bin:$gawk/bin:$gzip/bin:$gnugrep/bin:$gnused/bin:$binutils/bin"
tar -xzf $src
cd hello-2.9
./configure --prefix=$out
make
make install
And the derivation hello.nix:
with (import <nixpkgs> {});
derivation {
  name = "hello";
  builder = "${bash}/bin/bash";
  args = [ ./hello_builder.sh ];
  inherit gnutar gzip gnumake gcc binutils coreutils gawk;
  src = ./hello-2.9.tar.gz;
  system = builtins.currentSystem;
}
Now build it with nix-build hello.nix and you can launch result/bin/hello. Nothing easier, but do we have to create a builder.sh for each package? Do we always have to pass the dependencies to the derivation function?

Please note the --prefix=$out we were talking about in the previous pill.


A generic builder


Let's a create a generic builder.sh for autotools projects:
set -e
unset PATH
for p in $buildInputs; do
  export PATH=$p/bin${PATH:+:}$PATH
done

tar -xf $src

for d in *; do
  if [ -d "$d" ]; then
    cd "$d"
    break
  fi
done

./configure --prefix=$out
make
make install
What do we do here?
  1. Exit the build on any error with set -e.
  2. First unset PATH, because it's initially set to a non-existant path.
  3. We'll see this below in detail, however for each path in $buildInputs, we append bin to PATH.
  4. Unpack the source.
  5. Find a directory where the source has been unpacked and cd into it.
  6. Once we're set up, compile and install.
As you can see, there's no reference to "hello" in the builder anymore. It still does several assumptions, but it's certainly more generic.
Now let's rewrite hello.nix:
with (import <nixpkgs> {});
derivation {
  name = "hello";
  builder = "${bash}/bin/bash";
  args = [ ./builder.sh ];
  buildInputs = [ gnutar gzip gnumake gcc binutils coreutils gawk gnused gnugrep ];
  src = ./hello-2.9.tar.gz;
  system = builtins.currentSystem;
}
All clear, except that buildInputs. However it's easier than any black magic you are thinking in this moment.

Nix is able to convert a list to a string. It first converts the elements to strings, and then concatenates them separated by a space:
nix-repl> builtins.toString 123
"123"
nix-repl> builtins.toString [ 123 456 ]
"123 456"
Recall that derivations can be converted to a string, hence:
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> builtins.toString gnugrep
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14"
nix-repl> builtins.toString [ gnugrep gnused ]
"/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14 /nix/store/krgdc4sknzpw8iyk9p20lhqfd52kjmg0-gnused-4.2.2"
Simple! The buildInputs variable is a string with out paths separated by space, perfect for bash usage in a for loop.

A more convenient derivation function


We managed to write a builder that can be used for multiple autotools projects. But in the hello.nix expression we are specifying tools that are common to more projects; we don't want to pass them everytime.
A natural approach would be to create a function that accepts an attribute set, similar to the one used by the derivation function, and merge it with another attribute set containing values common to many projects.

Create autotools.nix:
pkgs: attrs:
  with pkgs;
  let defaultAttrs = {
    builder = "${bash}/bin/bash";
    args = [ ./builder.sh ];
    baseInputs = [ gnutar gzip gnumake gcc binutils coreutils gawk gnused gnugrep ];
    buildInputs = [];
    system = builtins.currentSystem;
  };
  in
  derivation (defaultAttrs // attrs)

Ok now we have to remember a little about Nix functions. The whole nix expression of this autotools.nix file will evaluate to a function. This function accepts a parameter pkgs, then returns a function which accepts a parameter attrs.
The body of the function is simple, yet at first sight it might be hard to grasp:
  1. First drop in the scope the magic pkgs attribute set.
  2. Within a let expression we define an helper variable, defaultAttrs, which serves as a set of common attributes used in derivations.
  3. Finally we create the derivation with that strange expression, (defaultAttrs // attrs).
The // operator is an operator between two sets. The result is the union of the two sets. In case of conflicts between attribute names, the value on the right set is preferred.
So we use defaultAttrs as base set, and add (or override) the attributes from attrs.

A couple of examples ought to be enough to clear out the behavior of the operator:
nix-repl> { a = "b"; } // { c = "d"; }
{ a = "b"; c = "d"; }
nix-repl> { a = "b"; } // { a = "c"; }
{ a = "c"; }

Complete the new builder.sh by adding $baseInputs in the for loop together with $buildInputs. As you noticed, we passed that new variable in the derivation. Instead of merging buildInputs with the base ones, we prefer to preserve buildInputs as seen by the caller, so we keep them separated. Just a matter of choice.

Then we rewrite hello.nix as follows:
let
  pkgs = import <nixpkgs> {};
  mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
  name = "hello";
  src = ./hello-2.9.tar.gz;
}
Finally! We got a very simple description of a package! A couple of remarks that you may find useful to keep understanding the nix language:
  • We assigned to pkgs the import that we did in the previous expressions in the "with", don't be afraid. It's that straightforward.
  • The mkDerivation variable is a nice example of partial application, look at it as (import ./autotools.nix) pkgs. First we import the expression, then we apply the pkgs parameter. That will give us a function that accepts the attribute set attrs.
  • We create the derivation specifying only name and src. If the project eventually needed other dependencies to be in PATH, then we would simply add those to buildInputs (not specified in hello.nix because empty).
Note we didn't use any other library. Special C flags may be needed to find include files of other libraries at compile time, and ld flags at link time.

Conclusion


Nix gives us the bare metal tools for creating derivations, setting up a build environment and storing the result in the nix store.

Out of this we managed to create a generic builder for autotools projects, and a function mkDerivation that composes by default the common components used in autotools projects instead of repeating them in all the packages we would write.

We are feeling the way a Nix system grows up: it's about creating and composing derivations with the Nix language.

Analogy: in C you create objects in the heap, and then you compose them inside new objects. Pointers are used to refer to other objects.
In Nix you create derivations stored in the nix store, and then you compose them by creating new derivations. Store paths are used to refer to other derivations.

Next pill


...we will talk a little about runtime dependencies. Is the GNU hello world package self-contained? What are its runtime dependencies? We only specified build dependencies by means of using other derivations in the "hello" derivation.

I'm going on vacation right now, so I may not be able to keep up with new pills for a few weeks. Have fun with Nix in the while :-)

Pill 9 available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Thursday, July 31, 2014

Nix pill 7: a working derivation

Welcome to the seventh Nix pill. In the previous sixth pill we introduced the notion of derivation in the Nix language. How to define a raw derivation and how to (try) to build it.

In this post, we will continue along the path, by creating a derivation that successfully builds something.
Then we try to package a real program: we compile a simple C file and create a derivation out of it, given a blessed toolchain.

I remind you how to enter the Nix environment: source ~/.nix-profile/etc/profile.d/nix.sh

Using a script as builder


What's the easiest way to run a sequence of commands for building something? A bash script. We write a custom bash script, and we want it to be our builder.
Given a builder.sh, we want the derivation to run bash builder.sh.

We don't use hash bangs in builder.sh, because at the time we are writing builder.sh we do not know the path to bash in the nix store. Yes, even bash is in the nix store, everything is in there.

We don't even use /usr/bin/env, because then we lose the cool stateless property of Nix. Not to say PATH gets cleared when building therefore it wouldn't work anyway.

In summary: we want the builder to be bash, and pass it an argument, builder.sh . Turns out the derivation function accepts an optional args attribute that is exactly used to pass arguments to the builder executable.

First of all, let's write our builder.sh in the current directory:
declare -xp
echo foo > $out
Ok, let's get the hang of it. I remind you from the previous pill, Nix creates the out path (not physically, you know the path) of the derivation. In the .drv there's a list of environment variables passed to the builder. One of them is $out.
What we have to do is to create something in $out, be it a file or a directory. In this case we are creating a file.

In addition, we also debug the environment variables during the build process. We cannot use env, because env is part of coreutils and we don't have a dependency to it. Not yet. It's plain bash, only bash.

Like for coreutils in the previous pill, we get a blessed bash for free from our magic nixpkgs stuff:
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> "${bash}"
"/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45"

Great, with the usual trick we can then refer to bin/bash and create our derivation:
nix-repl> d = derivation { name = "foo"; builder = "${bash}/bin/bash"; args = [ ./builder.sh ]; system = builtins.currentSystem; }
nix-repl> :b d
these derivations will be built:
  /nix/store/ybnysdh5k6cjznhg4afjgbhr6czbwb4s-foo.drv
building path(s) `/nix/store/72v14vk4li47n8sx3z2ibd802ihpqyvx-foo'
these derivations will be built:
  /nix/store/ibwr68l3rjlx02kgz61dkkkrlpgljfxd-simple.drv
[...]
this derivation produced the following outputs:
  out -> /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo

What? We did it! The contents of /nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo is really foo. We built our first derivation.

Note: we used ./builder.sh, not "./builder.sh". This way it gets parsed as path and Nix does wonders with it as we'll see later. Try using the string version, it will say it cannot find ./builder.sh , because that would be relative to the temporary build directory.

The builder environment


Let's inspect those debugged environment variables during the build process.
  • $HOME is not your home, and /homeless-shelter doesn't exist at all. We force packages to not depend upon $HOME during the build process.
  • $PATH plays the same game of $HOME
  • $NIX_BUILD_CORES and $NIX_STORE are nix configurations
  • $PWD and $TMP clearly shows nix created a temporary build directory.
  • Then builder, name, out and system are variables set due to the .drv contents.
And that's how we used the $out variable in our derivation, put stuff inside it. It's like Nix reserved a slot in the nix store for us, and we must fill it.

In terms of autotools, that will be the --prefix path. Yes, not the make DESTDIR, but the --prefix. That's a big difference between Nix and other package managers. That's the essence of stateless packaging. You don't install the package in a global common path under /, you install it in a local isolated path under your nix store slot.

The .drv contents


We added something else this time to the derivation. The args attribute. Let's see how this changed the .drv compared to the previous pill:
$ pp-aterm -i /nix/store/g6jj1mjzq68i66rbqyb3gpx3k0x606af-foo.drv
Derive(
  [("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo", "", "")]
, [("/nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv", ["out"])]
, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-builder.sh"]
, "x86_64-linux"
, "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45/bin/bash"
, ["/nix/store/5d1i99yd1fy4wkyx85iz5bvh78j2j96r-builder.sh"]
, [ ("builder", "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45/bin/bash")
  , ("name", "foo")
  , ("out", "/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo")
  , ("system", "x86_64-linux")
  ]
)

Perfect, much like the usual .drv, except there's a list of arguments in there passed to the builder (bash), with the builder.sh... builder.sh... what? It's not pointing to my home's builder.sh .
Nix automatically copies files or directories needed for the build in the nix store, to ensure, for example, that they do not get changed during the build process. Also to ensure the deployment to be stateless and independent of the building machine.

Not only builder.sh is in the arguments passed to the builder, it's also in the input derivations.

Being builder.sh a plain file, it has no .drv associated with it. The store path will be computed based on the hash of its contents, and the name itself. We will talk about store paths at some point, in a dedicated pill.

Packaging a simple C executable


Start off writing a simple.c file:
void main () {
  puts ("Simple!");
}
And its simple_builder.sh:
export PATH="$coreutils/bin:$gcc/bin"
mkdir $out
gcc -o $out/simple $src
Don't spend time understanding where those variables come from. Let's write the derivation and build it:
nix-repl> :l <nixpkgs>
nix-repl> simple = derivation { name = "simple"; builder = "${bash}/bin/bash"; args = [ ./simple_builder.sh ]; gcc = gcc; coreutils = coreutils; src = ./simple.c; system = builtins.currentSystem; }
nix-repl> :b simple
this derivation produced the following outputs:
  out -> /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple

Perfect, now you can run /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple/simple in your shell.

Explanation

We added two new attributes to the derivation call, gcc and coreutils. Please, don't get an headache by reading "gcc = gcc". On the left, it's the attribute name of the set. On the right, there's an expression, it's the gcc derivation. Same goes for coreutils.

We also added the src attribute, nothing magic it's just a name with the ./simple.c path. Like for simple_builder.sh, simple.c will be added to the store.

The trick: every attribute in the set will be converted to a string and passed as environment variable to the builder. Now it's all clear. $coreutils and $gcc are then the out paths of the derivations, and of course appending "/bin" will point to their binaries.

Same goes for the src variable, $src is the path to simple.c in the nix store. As an exercise, pretty print the .drv file. You'll see in the input derivations simple_builder.sh and simple.c files, then bash, gcc and coreutils .drv files. Plus the new environment variables described above.

In simple_builder.sh we set the PATH for gcc and coreutils binaries, so that gcc can find the necessary binaries like "cat", "readlink", ecc. .
Then we create $out as a directory and inside it we put the binary.

Note: instead of running plain gcc (or mkdir), it would have been equivalent to run $gcc/bin/gcc (or $coreutils/bin/mkdir).

Enough with nix-repl


Drop out of nix-repl, write a simple.nix file:
with (import <nixpkgs> {});
derivation {
  name = "simple";
  builder = "${bash}/bin/bash";
  args = [ ./simple_builder.sh ];
  inherit gcc coreutils;
  src = ./simple.c;
  system = builtins.currentSystem;
}
Now you can build it with nix-build simple.nix. It will create a symlink "result" in the current directory, pointing to the out path of the derivation.

The nix-build tool does two main jobs:
  • nix-instantiate: parse simple.nix and return the .drv file relative to the parsed derivation set
  • nix-store -r: realise the .drv, which actually builds the derivation.
Finally creates the symlink.

Look the first line of the .nix file. We have an "import" function call nested in a "with" expression. I recall import accepts one argument, a nix file to parse. In this case it parsed a function out of the file.
Afterwards we call the parsed function with the empty set. We saw this already in nix pill 5.

Let me underline it: "import <nixpkgs> {}" are two function calls, not one. Read it like "(import <nixpkgs>) {}".

The final returned value of that import is a set. To simplify it: it's a set of derivations. Using the "with" expression we drop them into the scope. We basically simulated what :l does in nix-repl, so we can easily access derivations such as bash, gcc and coreutils.

Then we meet the inherit keyword. Doing inherit foo, is the same as doing foo = foo. Doing inherit foo bar, is the same as doing foo = foo; bar = bar. Literally.
This syntax only makes sense inside sets. Don't think it's black magic, it's just a convenience to avoid repeating the same name twice, once for the attribute name, once for the variable in the scope.

Next pill


...we will generalize the builder. If you have noticed, we have written two builder.sh files in this post. We would like to have a generic builder script instead, especially since every builder script goes in the nix store: that's a waste.

Again. Is it really that hard to package stuff in Nix? No, here we're studying the fundamentals of Nix.

Pill 8 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.

Tuesday, July 29, 2014

Nix pill 6: our first derivation

Welcome to the sixth Nix pill. In the previous fifth pill we introduced functions and imports. Functions and imports are very simple concepts that allows for building complex abstractions and composition of modules to build a flexible Nix system.

In this post we finally arrived to writing a derivation. Derivations are the building blocks of a Nix system, from a file system view point. The Nix language is used to describe such derivations.

I remind you how to enter the Nix environment: source ~/.nix-profile/etc/profile.d/nix.sh

The derivation function


The derivation built-in function is used to create derivations. I invite you to read the link in the Nix manual about the derivation built-in. A derivation from a Nix language view point is simply a set, with some attributes. Therefore you can pass the derivation around with variables like anything else.
That's where the real power comes in. 

The derivation function receives a set as first argument. This set requires at least the following three attributes:
  • name: the name of the derivation. In the nix store the format is hash-name, that's the name.
  • system: is the name of the system in which the derivation can be built. For example, x86_64-linux.
  • builder: it is the binary program that builds the derivation.
First of all, what's the name of our system as seen by nix?
nix-repl> builtins.currentSystem
"x86_64-linux"
Let's try to fake the name of the system:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> d
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»
Oh oh, what's that? Did it build the derivation? No it didn't, but it did create the .drv file. nix-repl does not build derivations unless you tell to do so.

Digression about .drv files


What's that .drv file? It is the specification of how to build the derivation, without all the Nix language fuzz.
Before continuing, some analogies with the C language:
  • .nix files are like .c files
  • .drv files are intermediate files like .o files. The .drv describes how to build a derivation, it's the bare minimum information.
  • out paths are then the product of the build
Both drv paths and out paths are stored in the nix store as you can see.

What's in that .drv file? You can read it, but it's better to pretty print it.
$ nix-env -i strategoxt
If you feel the above command being too slow (ignore the meaning of the command below, just do it):
$ nix-env -iA strategoPackages018.strategoxt -f '<nixpkgs>'
The installed pp-aterm program can be used to pretty print .drv files:
$ pp-aterm -i /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
Derive(
  [("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname", "", "")]
, []
, []
, "mysystem"
, "mybuilder"
, []
, [ ("builder", "mybuilder")
  , ("name", "myname")
  , ("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname")
  , ("system", "mysystem")
  ]
Ok we can see there's an out path, but it does not exist yet. We never told Nix to build it, but we know beforehand where the build output will be. Why?

Think, if Nix ever built the derivation just because we accessed it in Nix, we would have to wait a long time if it was, say, Firefox. That's why Nix let us know the path beforehand and keep evaluating the Nix expressions, but it's still empty because no build was ever made.

Important: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It's possible however to have content-addressable derivations for e.g. tarballs as we'll see later on.

Many things are empty in that .drv, however I write a summary of the .drv format for you:
  1. The output paths (they can be multiple ones). By default nix creates one out path called "out".
  2. The list of input derivations. It's empty because we are not referring to any other derivation. Otherwise, there would a list of other .drv files.
  3. The system and the builder executable (yes, it's a fake one).
  4. Then a list of environment variables passed to the builder.
That's it, the minimum necessary information to build our derivation.

Important note: the environment variables passed to the builder are just those you see in the .drv plus some other Nix related configuration (number of cores, temp dir, ...). The builder will not inherit any variable from your running shell, otherwise builds would suffer from non-determinism.

Back to our fake derivation


Let's build our really fake derivation:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> :b d
[...]
these derivations will be built:
  /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
building path(s) `/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname'
error: a `mysystem' is required to build `/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv', but I am a `x86_64-linux'
The :b is a nix-repl specific command to build a derivation. You can see more commands with :? .
So in the output you can see that it takes the .drv as information on how to build the derivation. Then it says it's trying to produce our out path. Finally the error we were waiting for: that derivation can't be built on our system.

We're doing the build inside nix-repl, but what if we don't want to use nix-repl?
You can realise a .drv with:
$ nix-store -r /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
You will get the same output as before.

Let's fix the system attribute:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
build error: invalid file name `mybuilder'
A step forward: of course, that "mybuilder" executable does not really exist. Stop for a moment.

What's in a derivation set


I find useful to inspect the returned value from the derivation function for you.
First of all, the returned value is a plain set:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> builtins.isAttrs d
true
nix-repl> builtins.attrNames d
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "system" "type" ]
You can guess what builtins.isAttrs does, it returns true if the argument is a set. While builtins.attrNames returns a list of keys of the given set. Some kind of reflection, you might say.

Start from drvAttrs:
nix-repl> d.drvAttrs
{ builder = "mybuilder"; name = "myname"; system = "mysystem"; }
That's basically the input we gave to the derivation function. Also d.name, d.system and d.builder attributes are straight the ones we gave as input.
nix-repl> (d == d.out)
true
So out is just the derivation itself, it seems weird but the reason is that we only have one output from the derivation. That's also the reason why d.all is a singleton. We'll see multiple outputs later.

The d.drvPath is the path of the .drv file: /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv .

Something interesting is the type attribute. It's "derivation". Nix does add a little of magic to sets with type derivation, but not that much. To let you understand, you can create yourself a set with that type, it's a simple set:
nix-repl> { type = "derivation"; }
«derivation ???»
Of course it has no other information, so Nix doesn't know what to say :-) But you get it, the type = "derivation" is just a convention for Nix and for us to understand the set is a derivation.

When writing packages, we are interested in the outputs. The other metadata is needed for Nix to know how to create the drv path and the out path.
The outPath attribute is the build path in the nix store: /nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname .

Referring to other derivations


Just like dependencies in other package managers, how do we refer to other packages? How do we refer to other derivations in terms of files on the disk?
We use the outPath. The outPath tells where the files are of that derivation. To make it more convenient, Nix is able to do a conversion from a derivation set to a string.
nix-repl> d.outPath
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> builtins.toString d
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
Nix does the "set to string conversion" as long as there is the outPath attribute (much like a toString method in other languages):
nix-repl> builtins.toString { outPath = "foo"; }
"foo"
nix-repl> builtins.toString { a = "b"; }
error: cannot coerce a set to a string, at (string):1:1
Say we want to use binaries from coreutils (ignore the nixpkgs etc.):
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> coreutils
«derivation /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv»
nix-repl> builtins.toString coreutils
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"
Apart the nixpkgs stuff, just think we added to the scope a series of variables. One of them is coreutils. It is the derivation of the coreutils package you all know of from other Linux distributions. 
It contains basic binaries for GNU/Linux systems (you may have multiple derivations of coreutils in the nix store, no worries):
$ ls /nix/store/*coreutils*/bin
[...]
I remind you, inside strings it's possible to interpolate Nix expressions with ${...}:
nix-repl> "${d}"
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> "${coreutils}"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"
That's very convenient, because then we could refer to e.g. the bin/true binary like this:
nix-repl> "${coreutils}/bin/true"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"

An almost working derivation


In the previous attempt we used a fake builder, "mybuilder" which obviously does not exist. But we can use for example bin/true, which always exits with 0 (success).
nix-repl> :l <nixpkgs>
nix-repl> d = derivation { name = "myname"; builder = "${coreutils}/bin/true"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
builder for `/nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-myname.drv' failed to produce output path `/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname'
Another step forward, it executed the builder (bin/true), but the builder did not create the out path of course, it just exited with 0.

Obvious note: everytime we change the derivation, a new hash is created.

Let's examine the new .drv now that we referred to another derivation:

$ pp-aterm -i /nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-myname.drv
Derive(
  [("out", "/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname", "", "")]
, [("/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv", ["out"])]
, []
, "x86_64-linux"
, "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"
, []
, [ ("builder", "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true")
  , ("name", "myname")
  , ("out", "/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname")
  , ("system", "x86_64-linux")
  ]

Aha! Nix added a dependency to our myname.drv, it's the coreutils.drv. Before doing our build, Nix should build the coreutils.drv. But since coreutils is already in our nix store, no build is needed, 
it's already there with out path /nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21 .

When is the derivation built


Nix does not build derivations during evaluation of Nix expressions. In fact, that's why we have to do ":b drv" in nix-repl, or use nix-store -r in the first place.

An important separation is made in Nix:
  • Instantiate/Evaluation time: the Nix expression is parsed, interpreted and finally returns a derivation set. During evaluation, you can refer to other derivations because Nix will create .drv files and we will know out paths beforehand. This is achieved with nix-instantiate.
  • Realise/Build time: the .drv from the derivation set is built, first building .drv inputs (build dependencies). This is achieved with nix-store -r.
Think of it as of compile time and link time like with C/C++ projects. You first compile all source files to object files. Then link object files in a single executable.

In Nix, first the Nix expression (usually in a .nix file) is compiled to .drv, then each .drv is built and the product is installed in the relative out paths.

Conclusion


Is that complicated to create a package for Nix? No it's not.

We're walking through the fundamentals of Nix derivations, to understand how they work, how they are represented.
Packaging in Nix is certainly easier than that, but we're not there yet in this post. More Nix pills are needed.

With the derivation function we provide a set of information on how to build a package, and we get back the information about where the package was built.
Nix converts a set to a string when there's an outPath, that's very convenient. With that, it's easy to refer to other derivations.

When Nix builds a derivation, it first creates a .drv file from a derivation expression, and uses it to build the output. It does so recursively for all the dependencies (inputs). It "executes" the .drv files like a machine. Not much magic after all.


Next pill


...we will finally write our first working derivation. Yes, this post is about "our first derivation", but I never said it was a working one

Pill 7 is available for reading here.

To be notified about the new pill, stay tuned on #NixPills, follow @lethalman or subscribe to the nixpills rss.