Fetchers

Building software with Nix often requires downloading source code and other files from the internet. To this end, we use functions that we call fetchers, which obtain remote sources via various protocols and services.

Nix provides built-in fetchers such as builtins.fetchTarball. Nixpkgs provides its own fetchers, which work differently:

A built-in fetcher will download and cache files at evaluation time and produce a store path. A Nixpkgs fetcher will create a (fixed-output) derivation, and files are downloaded at build time.
Built-in fetchers will invalidate their cache after tarball-ttl expires, and will require network activity to check if the cache entry is up to date. Nixpkgs fetchers only re-download if the specified hash changes or the store object is not available.
Built-in fetchers do not use substituters. Derivations produced by Nixpkgs fetchers will use any configured binary cache transparently.

This significantly reduces the time needed to evaluate Nixpkgs, and allows Hydra to retain and re-distribute sources used by Nixpkgs in the public binary cache. For these reasons, Nix's built-in fetchers are not allowed in Nixpkgs.

The following table summarises the differences:

Fetchers	Download	Output	Cache	Re-download when
`builtins.fetch*`	evaluation time	store path	`/nix/store`, `~/.cache/nix`	`tarball-ttl` expires, cache miss in `~/.cache/nix`, output store object not in local store
`pkgs.fetch*`	build time	derivation	`/nix/store`, substituters	output store object not available

Tip

pkgs.fetchFrom* helpers retrieve snapshots of version-controlled sources, as opposed to the entire version history, which is more efficient. pkgs.fetchgit by default also has the same behaviour, but can be changed through specific attributes given to it.

Caveats

Because Nixpkgs fetchers are fixed-output derivations, an output hash has to be specified, usually indirectly through a hash attribute. This hash refers to the derivation output, which can be different from the remote source itself!

This has the following implications that you should be aware of:

Use Nix (or Nix-aware) tooling to produce the output hash.
When changing any fetcher parameters, always update the output hash. Use one of the methods from . Otherwise, existing store objects that match the output hash will be re-used rather than fetching new content.

Note

A similar problem arises while testing changes to a fetcher's implementation. If the output of the derivation already exists in the Nix store, test failures can go undetected. The invalidateFetcherByDrvHash function helps prevent reusing cached derivations.

Updating source hashes

There are several ways to obtain the hash corresponding to a remote source. Unless you understand how the fetcher you're using calculates the hash from the downloaded contents, you should use the fake hash method.

[]{#sec-pkgs-fetchers-updating-source-hashes-fakehash-method} The fake hash method: In your package recipe, set the hash to one of
""
lib.fakeHash
lib.fakeSha256
lib.fakeSha512

Attempt to build, extract the calculated hashes from error messages, and put them into the recipe.

Warning

You must use one of these four fake hashes and not some arbitrarily-chosen hash. See for details.

Example

   # Update source hash with the fake hash method

Consider the following recipe that produces a plain file:

{ fetchurl }:
fetchurl {
  url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.05/.version";
  hash = "sha256-ZHl1emidXVojm83LCVrwULpwIzKE/mYwfztVkvpruOM=";
}

A common mistake is to update a fetcher parameter, such as url, without updating the hash:

{ fetchurl }:
fetchurl {
  url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version";
  hash = "sha256-ZHl1emidXVojm83LCVrwULpwIzKE/mYwfztVkvpruOM=";
}

This will produce the same output as before! Set the hash to an empty string:

{ fetchurl }:
fetchurl {
  url = "https://raw.githubusercontent.com/NixOS/nixpkgs/23.11/.version";
  hash = "";
}

When building the package, use the error message to determine the correct hash:

$ nix-build
(some output removed for clarity)
error: hash mismatch in fixed-output derivation '/nix/store/7yynn53jpc93l76z9zdjj4xdxgynawcw-version.drv':
        specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-BZqI7r0MNP29yGH5+yW2tjU9OOpOCEvwWKrWCv5CQ0I=
error: build of '/nix/store/bqdjcw5ij5ymfbm41dq230chk9hdhqff-version.drv' failed

Prefetch the source with nix-prefetch-<type> <URL>, where <type> is one of
url
git
hg
cvs
bzr
svn

The hash is printed to stdout.

Prefetch by package source (with nix-prefetch-url '<nixpkgs>' -A <package>.src, where <package> is package attribute name). The hash is printed to stdout.

This works well when you've upgraded the existing package version and want to find out new hash, but is useless if the package can't be accessed by attribute or the package has multiple sources (.srcs, architecture-dependent sources, etc).

Upstream hash: use it when upstream provides sha256 or sha512. Don't use it when upstream provides md5, compute sha256 instead.

A little nuance is that nix-prefetch-* tools produce hashes with the nix32 encoding (a Nix-specific base32 adaptation), but upstream usually provides hexadecimal (base16) encoding. Fetchers understand both formats. Nixpkgs does not standardise on any one format.

You can convert between hash formats with nix-hash.

Extract the hash from a local source archive with sha256sum. Use nix-prefetch-url file:///path/to/archive if you want the custom Nix base32 hash.

Obtaining hashes securely

It's always a good idea to avoid Man-in-the-Middle (MITM) attacks when downloading source contents. Otherwise, you could unknowingly download malware instead of the intended source, and instead of the actual source hash, you'll end up using the hash of malware. Here are security considerations for this scenario:

http:// URLs are not secure to prefetch hashes.
Upstream hashes should be obtained via a secure protocol.
https:// URLs give you more protections when using nix-prefetch-* or for upstream hashes.
https:// URLs are secure when using the fake hash method only if you use one of the listed fake hashes. If you use any other hash, the download will be exposed to MITM attacks even if you use HTTPS URLs.

In more concrete terms, if you use any other hash, the --insecure flag will be passed to the underlying call to curl when downloading content.

[]{#fetchurl}

`fetchurl`

fetchurl returns a fixed-output derivation which downloads content from a given URL and stores the unaltered contents within the Nix store.

It uses {manpage}curl(1) internally, and allows its behaviour to be modified by specifying a few attributes in the argument to fetchurl (see the documentation for attributes curlOpts, curlOptsList, and netrcPhase).

The resulting store path is determined by the hash given to fetchurl, and also the name (or pname and version) values.

If neither name nor pname and version are specified when calling fetchurl, it will default to using the basename of url or the first element of urls. If pname and version are specified, fetchurl will use those values and will ignore name, even if it is also specified.

Inputs

fetchurl requires an attribute set with the following attributes:

url (String; optional) : The URL to download from.

Note

Either url or urls must be specified, but not both.

All URLs of the format specified here are supported.

Default value: "".

urls (List of String; optional) : A list of URLs, specifying download locations for the same content. Each URL will be tried in order until one of them succeeds with some content or all of them fail. See to understand how this attribute affects the behaviour of fetchurl.

Note

Either url or urls must be specified, but not both.

Default value: [].

hash (String; optional) : Hash of the derivation output of fetchurl, following the format for integrity metadata as defined by SRI. For more information, see .

Note

It is recommended that you use the hash attribute instead of the other hash-specific attributes that exist for backwards compatibility.

If hash is not specified, you must specify outputHash and outputHashAlgo, or one of sha512, sha256, or sha1.

Default value: "".

outputHash (String; optional) : Hash of the derivation output of fetchurl in the format expected by Nix. See the documentation on the Nix manual for more information about its format.

Note