lib.fileset: file set functions

[]{#sec-fileset}

The lib.fileset library allows you to work with file sets. A file set is a (mathematical) set of local files that can be added to the Nix store for use in Nix derivations. File sets are easy and safe to use, providing obvious and composable semantics with good error messages to prevent mistakes.

Overview

Basics: - Implicit coercion from paths to file sets

lib.fileset.maybeMissing:

Create a file set from a path that may be missing.

lib.fileset.trace/lib.fileset.traceVal:

Pretty-print file sets for debugging.

lib.fileset.toSource:

Add files in file sets to the store to use as derivation sources.

lib.fileset.toList:

The list of files contained in a file set.

Combinators: - lib.fileset.union/lib.fileset.unions:

Create a larger file set from all the files in multiple file sets.

lib.fileset.intersection:

Create a smaller file set from only the files in both file sets.

lib.fileset.difference:

Create a smaller file set containing all files that are in one file set, but not another one.

Filtering: - lib.fileset.fileFilter:

Create a file set from all files that satisisfy a predicate in a directory.

Utilities: - lib.fileset.fromSource:

Create a file set from a lib.sources-based value.

lib.fileset.gitTracked/lib.fileset.gitTrackedWith:

Create a file set from all tracked files in a local Git repository.

If you need more file set functions, see this issue to request it.

Implicit coercion from paths to file sets

All functions accepting file sets as arguments can also accept paths as arguments. Such path arguments are implicitly coerced to file sets containing all files under that path: - A path to a file turns into a file set containing that single file. - A path to a directory turns into a file set containing all files recursively in that directory.

If the path points to a non-existent location, an error is thrown.

Note

Just like in Git, file sets cannot represent empty directories. Because of this, a path to a directory that contains no files (recursively) will turn into a file set containing no files.

Note

File set coercion does not add any of the files under the coerced paths to the store. Only the toSource function adds files to the Nix store, and only those files contained in the fileset argument. This is in contrast to using paths in string interpolation, which does add the entire referenced path to the store.

Example

Assume we are in a local directory with a file hierarchy like this:

├─ a/
│  ├─ x (file)
│  └─ b/
│     └─ y (file)
└─ c/
   └─ d/

Here's a listing of which files get included when different path expressions get coerced to file sets: - ./. as a file set contains both a/x and a/b/y (c/ does not contain any files and is therefore omitted). - ./a as a file set contains both a/x and a/b/y. - ./a/x as a file set contains only a/x. - ./a/b as a file set contains only a/b/y. - ./c as a file set is empty, since neither c nor c/d contain any files.

`lib.fileset.maybeMissing`

Create a file set from a path that may or may not exist: - If the path does exist, the path is coerced to a file set. - If the path does not exist, a file set containing no files is returned.

Inputs

path

: 1. Function argument

Type

maybeMissing :: Path -> FileSet

Examples

lib.fileset.maybeMissing usage example

# All files in the current directory, but excluding main.o if it exists
difference ./. (maybeMissing ./main.o)

Located at lib/fileset/default.nix:189 in <nixpkgs>.

`lib.fileset.trace`

Incrementally evaluate and trace a file set in a pretty way. This function is only intended for debugging purposes. The exact tracing format is unspecified and may change.

This function takes a final argument to return. In comparison, traceVal returns the given file set argument.

This variant is useful for tracing file sets in the Nix repl.

Inputs

fileset

: The file set to trace.

This argument can also be a path, which gets implicitly coerced to a file set.

val

: The value to return.

Type

trace :: FileSet -> Any -> Any

Examples

lib.fileset.trace usage example

trace (unions [ ./Makefile ./src ./tests/run.sh ]) null
=>
trace: /home/user/src/myProject
trace: - Makefile (regular)
trace: - src (all files in directory)
trace: - tests
trace:   - run.sh (regular)
null

Located at lib/fileset/default.nix:251 in <nixpkgs>.

`lib.fileset.traceVal`

Incrementally evaluate and trace a file set in a pretty way. This function is only intended for debugging purposes. The exact tracing format is unspecified and may change.

This function returns the given file set. In comparison, trace takes another argument to return.

This variant is useful for tracing file sets passed as arguments to other functions.

Inputs

fileset

: The file set to trace and return.

This argument can also be a path, which gets implicitly coerced to a file set.

Type

traceVal :: FileSet -> FileSet

Examples

lib.fileset.traceVal usage example

toSource {
  root = ./.;
  fileset = traceVal (unions [
    ./Makefile
    ./src
    ./tests/run.sh
  ]);
}
=>
trace: /home/user/src/myProject
trace: - Makefile (regular)
trace: - src (all files in directory)
trace: - tests
trace:   - run.sh (regular)
"/nix/store/...-source"

Located at lib/fileset/default.nix:311 in <nixpkgs>.

`lib.fileset.toSource`

Add the local files contained in fileset to the store as a single store path rooted at root.

The result is the store path as a string-like value, making it usable e.g. as the src of a derivation, or in string interpolation:

stdenv.mkDerivation {
  src = lib.fileset.toSource { ... };
  # ...
}

The name of the store path is always source.

Inputs

Takes an attribute set with the following attributes

root (Path; required)

: The local directory path that will correspond to the root of the resulting store path. Paths in strings, including Nix store paths, cannot be passed as root. root has to be a directory.

Note

Changing root only affects the directory structure of the resulting store path, it does not change which files are added to the store. The only way to change which files get added to the store is by changing the fileset attribute.

fileset (FileSet; required)

: The file set whose files to import into the store. File sets can be created using other functions in this library. This argument can also be a path, which gets implicitly coerced to a file set.

Note

If a directory does not recursively contain any file, it is omitted from the store path contents.

Type

toSource :: {
  root :: Path,
  fileset :: FileSet,
} -> SourceLike

Examples

lib.fileset.toSource usage example

# Import the current directory into the store
# but only include files under ./src
toSource {
  root = ./.;
  fileset = ./src;
}
=> "/nix/store/...-source" "Import the current directory into the store
# but only include ./Makefile and all files under ./src
toSource {
  root = ./.;
  fileset = union
    ./Makefile
    ./src;
}
=> "/nix/store/...-source" "Trying to include a file outside the root will fail
toSource {
  root = ./.;
  fileset = unions [
    ./Makefile
    ./src
    ../LICENSE
  ];
}
=> <error> "The root needs to point to a directory that contains all the files
toSource {
  root = ../.;
  fileset = unions [
    ./Makefile
    ./src
    ../LICENSE
  ];
}
=> "/nix/store/...-source" "The root has to be a local filesystem path
toSource {
  root = "/nix/store/...-source";
  fileset = ./.;
}
=> <error>

Located at lib/fileset/default.nix:426 in <nixpkgs>.

`lib.fileset.toList`

The list of file paths contained in the given file set.

Note

This function is strict in the entire file set. This is in contrast with combinators lib.fileset.union, lib.fileset.intersection and lib.fileset.difference.

Thus it is recommended to call toList on file sets created using the combinators, instead of doing list processing on the result of toList.

The resulting list of files can be turned back into a file set using lib.fileset.unions.

Inputs

fileset

: The file set whose file paths to return. This argument can also be a path, which gets implicitly coerced to a file set.

Type

toList :: FileSet -> [ Path ]

Examples

lib.fileset.toList usage example

toList ./.
[ ./README.md ./Makefile ./src/main.c ./src/main.h ]

toList (difference ./. ./src)
[ ./README.md ./Makefile ]

Located at lib/fileset/default.nix:524 in <nixpkgs>.

`lib.fileset.union`

The file set containing all files that are in either of two given file sets. This is the same as unions, but takes just two file sets instead of a list. See also Union (set theory).

The given file sets are evaluated as lazily as possible, with the first argument being evaluated first if needed.

Inputs

fileset1

: The first file set. This argument can also be a path, which gets implicitly coerced to a file set.

fileset2

: The second file set. This argument can also be a path, which gets implicitly coerced to a file set.

Type

union :: FileSet -> FileSet -> FileSet

Examples

lib.fileset.union usage example

# Create a file set containing the file `Makefile`
# and all files recursively in the `src` directory
union ./Makefile ./src "Create a file set containing the file `Makefile`
# and the LICENSE file from the parent directory
union ./Makefile ../LICENSE

Located at lib/fileset/default.nix:569 in <nixpkgs>.

`lib.fileset.unions`

The file set containing all files that are in any of the given file sets. This is the same as union, but takes a list of file sets instead of just two. See also Union (set theory).

The given file sets are evaluated as lazily as possible, with earlier elements being evaluated first if needed.

Inputs

filesets

: A list of file sets. The elements can also be paths, which get implicitly coerced to file sets.

Type

unions :: [ FileSet ] -> FileSet

Examples

lib.fileset.unions usage example

# Create a file set containing selected files
unions [
  # Include the single file `Makefile` in the current directory
  # This errors if the file doesn't exist
  ./Makefile

  # Recursively include all files in the `src/code` directory
  # If this directory is empty this has no effect
  ./src/code

  # Include the files `run.sh` and `unit.c` from the `tests` directory
  ./tests/run.sh
  ./tests/unit.c

  # Include the `LICENSE` file from the parent directory
  ../LICENSE
]

Located at lib/fileset/default.nix:632 in <nixpkgs>.

`lib.fileset.intersection`

The file set containing all files that are in both of two given file sets. See also Intersection (set theory).

The given file sets are evaluated as lazily as possible, with the first argument being evaluated first if needed.

Inputs

fileset1

: The first file set. This argument can also be a path, which gets implicitly coerced to a file set.

fileset2

: The second file set. This argument can also be a path, which gets implicitly coerced to a file set.

Type

intersection :: FileSet -> FileSet -> FileSet

Examples

lib.fileset.intersection usage example

# Limit the selected files to the ones in ./., so only ./src and ./Makefile
intersection ./. (unions [ ../LICENSE ./src ./Makefile ])

Located at lib/fileset/default.nix:683 in <nixpkgs>.

`lib.fileset.difference`

The file set containing all files from the first file set that are not in the second file set. See also Difference (set theory).

The given file sets are evaluated as lazily as possible, with the first argument being evaluated first if needed.

Inputs

positive

: The positive file set. The result can only contain files that are also in this file set. This argument can also be a path, which gets implicitly coerced to a file set.

negative

: The negative file set. The result will never contain files that are also in this file set. This argument can also be a path, which gets implicitly coerced to a file set.

Type

union :: FileSet -> FileSet -> FileSet

Examples

lib.fileset.difference usage example

# Create a file set containing all files from the current directory,
# except ones under ./tests
difference ./. ./tests

let
  # A set of Nix-related files
  nixFiles = unions [ ./default.nix ./nix ./tests/default.nix ];
in
# Create a file set containing all files under ./tests, except ones in `nixFiles`,
# meaning only without ./tests/default.nix
difference ./tests nixFiles

Located at lib/fileset/default.nix:746 in <nixpkgs>.

`lib.fileset.fileFilter`

Filter a file set to only contain files matching some predicate.

Inputs

predicate

: The predicate function to call on all files contained in given file set. A file is included in the resulting file set if this function returns true for it.

This function is called with an attribute set containing these attributes:

name (String): The name of the file
type (String, one of "regular", "symlink" or "unknown"): The type of the file. This matches result of calling builtins.readFileType on the file's path.
hasExt (String -> Bool): Whether the file has a certain file extension. hasExt ext is true only if hasSuffix ".${ext}" name.

This also means that e.g. for a file with name .gitignore, hasExt "gitignore" is true.

Other attributes may be added in the future.

path

: The path whose files to filter

Type

fileFilter ::
  ({
    name :: String,
    type :: String,
    hasExt :: String -> Bool,
    ...
  } -> Bool)
  -> Path
  -> FileSet

Examples

lib.fileset.fileFilter usage example

# Include all regular `default.nix` files in the current directory
fileFilter (file: file.name == "default.nix") ./. "Include all non-Nix files from the current directory
fileFilter (file: ! file.hasExt "nix") ./. "Include all files that start with a "." in the current directory
fileFilter (file: hasPrefix "." file.name) ./. "Include all regular files (not symlinks or others) in the current directory
fileFilter (file: file.type == "regular") ./.

Located at lib/fileset/default.nix:829 in <nixpkgs>.

`lib.fileset.fromSource`

Create a file set with the same files as a lib.sources-based value. This does not import any of the files into the store.

This can be used to gradually migrate from lib.sources-based filtering to lib.fileset.

A file set can be turned back into a source using toSource.

Note

File sets cannot represent empty directories. Turning the result of this function back into a source using toSource will therefore not preserve empty directories.

Inputs

source

: 1. Function argument

Type

fromSource :: SourceLike -> FileSet

Examples

lib.fileset.fromSource usage example

# There's no cleanSource-like function for file sets yet,
# but we can just convert cleanSource to a file set and use it that way
toSource {
  root = ./.;
  fileset = fromSource (lib.sources.cleanSource ./.);
} "Keeping a previous sourceByRegex (which could be migrated to `lib.fileset.unions`),
# but removing a subdirectory using file set functions
difference
  (fromSource (lib.sources.sourceByRegex ./. [
    "^README\.md$"
    # This regex includes everything in ./doc
    "^doc(/.*)?$"
  ])
  ./doc/generated "Use cleanSource, but limit it to only include ./Makefile and files under ./src
intersection
  (fromSource (lib.sources.cleanSource ./.))
  (unions [
    ./Makefile
    ./src
  ]);

Located at lib/fileset/default.nix:908 in <nixpkgs>.

`lib.fileset.gitTracked`

Create a file set containing all Git-tracked files in a repository.

This function behaves like gitTrackedWith { } - using the defaults.

Inputs

path

: The path to the working directory of a local Git repository. This directory must contain a .git file or subdirectory.

Type

gitTracked :: Path -> FileSet

Examples

lib.fileset.gitTracked usage example

# Include all files tracked by the Git repository in the current directory
gitTracked ./. "Include only files tracked by the Git repository in the parent directory
# that are also in the current directory
intersection ./. (gitTracked ../.)

Located at lib/fileset/default.nix:969 in <nixpkgs>.

`lib.fileset.gitTrackedWith`

Create a file set containing all Git-tracked files in a repository. The first argument allows configuration with an attribute set, while the second argument is the path to the Git working tree.

gitTrackedWith does not perform any filtering when the path is a Nix store path and not a repository. In this way, it accommodates the use case where the expression that makes the gitTracked call does not reside in an actual git repository anymore, and has presumably already been fetched in a way that excludes untracked files. Fetchers with such equivalent behavior include builtins.fetchGit, builtins.fetchTree (experimental), and pkgs.fetchgit when used without leaveDotGit.

If you don't need the configuration, you can use gitTracked instead.

This is equivalent to the result of unions on all files returned by git ls-files (which uses --cached by default).

Warning

Currently this function is based on builtins.fetchGit As such, this function causes all Git-tracked files to be unnecessarily added to the Nix store, without being re-usable by toSource.

This may change in the future.

Inputs

options (attribute set) : recurseSubmodules (optional, default: false) : Whether to recurse into Git submodules to also include their tracked files. If true, this is equivalent to passing the --recurse-submodules flag to git ls-files.

path : The path to the working directory of a local Git repository. This directory must contain a .git file or subdirectory.

Type

gitTrackedWith :: { recurseSubmodules :: Bool ? false } -> Path -> FileSet

Examples

lib.fileset.gitTrackedWith usage example

# Include all files tracked by the Git repository in the current directory
# and any submodules under it
gitTracked { recurseSubmodules = true; } ./.

Located at lib/fileset/default.nix:1031 in <nixpkgs>.