WIP: feat: trivial builders V1 #24

Draft
austreelis wants to merge 5 commits from austreelis/feat/trivial-builders-v1 into main AGit
Member

Adds trivial builders:

  • trivial.text (nixpkgs.writeTextFile)
    • draft implementation
    • from nix string
    • from source file(s)
      • concatenate several
    • set executable
    • set target path
    • runtime dependencies
    • set interpreter (shebang, for scripts)
      • set executable by default if interpreter is defined
      • allow an argument for the interpreter
  • trivial.command (nixpkgs.runCommand)
    • draft implementation
    • PATH configuration
    • configure substitution and preferLocalBuild
  • trivial.concat trivial.text (nixpkgs.concatTextFile)
    • draft implementation

    merged as a feature of trivial.text

  • trivial.join (nixpkgs.symlinkJoin)
    • draft implementation
    • abort on conflict
    • log joined paths
    • debug-log actions taken
    • documenting the script
    • benchmarking
  • cleanup
  • document

I'll be expanding/checking that todo-list as I go.

commits are supposed to be reviewed independantly (though I might cherry-pick restructure them later to make them cleaner).

Examples were added to showcase how these can be used, where aux.d is a join of all of them:

❯ nix-build tidepool -A packages.aux.d.latest.packages.x86_64-linux.x86_64-linux.x86_64-linux.package \
  && eza -Tl result/ \
  && bat (fd . -Lt f result/)
dr-xr-xr-x - root  1 Jan  1970 result
dr-xr-xr-x - root  1 Jan  1970 ├── aux
dr-xr-xr-x - root  1 Jan  1970 │   └── example
lrwxrwxrwx - root  1 Jan  1970 │       ├── e -> /nix/store/kmkz57xv4y8hs6jpysd43w1pid1c7a93-aux-e-1.0.0/aux/example/e
lrwxrwxrwx - root  1 Jan  1970 │       ├── f -> /nix/store/c337rxb1qkxsj36pbd6qh1283jaspx6i-aux-f-1.0.0/aux/example/f
lrwxrwxrwx - root  1 Jan  1970 │       └── g -> /nix/store/hsgfpds5rg8xh2cg6i47502xchb00djl-aux-g-1.0.0/aux/example/g
lrwxrwxrwx - root  1 Jan  1970 └── e -> /nix/store/kmkz57xv4y8hs6jpysd43w1pid1c7a93-aux-e-1.0.0/e
───────┬─────────────────────────────
       │ File: result/aux/example/e
───────┼─────────────────────────────
   1   │ meow
───────┴─────────────────────────────
───────┬─────────────────────────────
       │ File: result/aux/example/f
───────┼─────────────────────────────
   1   │ #!/catgirl meow
   2   │ nyah
───────┴─────────────────────────────
───────┬─────────────────────────────
       │ File: result/aux/example/g
───────┼─────────────────────────────
   1   │ I am z
   2   │ ---
   3   │ I am y
   4   │ ---
   5   │ I am x
───────┴─────────────────────────────
───────┬─────────────────────────────
       │ File: result/e
───────┼─────────────────────────────
   1   │ meow
───────┴─────────────────────────────

closes #10

Adds trivial builders: - [x] `trivial.text` (`nixpkgs.writeTextFile`) - [x] draft implementation - [x] from nix string - [x] from source file(s) - [x] concatenate several - [x] set executable - [x] set target path - [ ] ~~runtime dependencies~~ - [x] set interpreter (shebang, for scripts) - [x] set executable by default if interpreter is defined - [x] allow an argument for the interpreter - [x] `trivial.command` (`nixpkgs.runCommand`) - [x] draft implementation - [x] `PATH` configuration - [x] configure substitution and `preferLocalBuild` - [x] ~~`trivial.concat`~~ `trivial.text` (`nixpkgs.concatTextFile`) - [x] draft implementation > merged as a feature of `trivial.text` - [x] `trivial.join` (`nixpkgs.symlinkJoin`) - [x] draft implementation - [x] abort on conflict - [x] log joined paths - [x] debug-log actions taken - [ ] documenting the script - [ ] benchmarking - [ ] cleanup - [ ] document I'll be expanding/checking that todo-list as I go. commits are supposed to be reviewed independantly (though I might cherry-pick restructure them later to make them cleaner). Examples were added to showcase how these can be used, where `aux.d` is a `join` of all of them: ``` ❯ nix-build tidepool -A packages.aux.d.latest.packages.x86_64-linux.x86_64-linux.x86_64-linux.package \ && eza -Tl result/ \ && bat (fd . -Lt f result/) dr-xr-xr-x - root 1 Jan 1970 result dr-xr-xr-x - root 1 Jan 1970 ├── aux dr-xr-xr-x - root 1 Jan 1970 │ └── example lrwxrwxrwx - root 1 Jan 1970 │ ├── e -> /nix/store/kmkz57xv4y8hs6jpysd43w1pid1c7a93-aux-e-1.0.0/aux/example/e lrwxrwxrwx - root 1 Jan 1970 │ ├── f -> /nix/store/c337rxb1qkxsj36pbd6qh1283jaspx6i-aux-f-1.0.0/aux/example/f lrwxrwxrwx - root 1 Jan 1970 │ └── g -> /nix/store/hsgfpds5rg8xh2cg6i47502xchb00djl-aux-g-1.0.0/aux/example/g lrwxrwxrwx - root 1 Jan 1970 └── e -> /nix/store/kmkz57xv4y8hs6jpysd43w1pid1c7a93-aux-e-1.0.0/e ───────┬───────────────────────────── │ File: result/aux/example/e ───────┼───────────────────────────── 1 │ meow ───────┴───────────────────────────── ───────┬───────────────────────────── │ File: result/aux/example/f ───────┼───────────────────────────── 1 │ #!/catgirl meow 2 │ nyah ───────┴───────────────────────────── ───────┬───────────────────────────── │ File: result/aux/example/g ───────┼───────────────────────────── 1 │ I am z 2 │ --- 3 │ I am y 4 │ --- 5 │ I am x ───────┴───────────────────────────── ───────┬───────────────────────────── │ File: result/e ───────┼───────────────────────────── 1 │ meow ───────┴───────────────────────────── ``` closes #10
austreelis added 1 commit 2025-09-29 23:42:59 +00:00
Merges a forest of paths into one directory tree with symlinks.
Conflicts are not handled and will always error out.
We use one symlink for every node that is unique to a given store path.

Performance likely becomes bad when joining lots of derivations, but I
have not investigated. The should algorithm do no unnecessary work
though.

Akin to nixpkgs' symlinkJoin.
requested review from jakehamilton 2025-09-30 00:04:35 +00:00
Author
Member

This is highly non-definitive, but I don't really know if the build function is quite right (I don't know what I'm doing).

Also, the example package (d.nix) is broken until I get the other trivial builders to produce something useful for it to consume.

This is highly non-definitive, but I don't really know if the build function is quite right (I don't know what I'm doing). Also, the example package (`d.nix`) is broken until I get the other trivial builders to produce something useful for it to consume.
austreelis force-pushed austreelis/feat/trivial-builders-v1 from 0bd20a9502 to f780a7698c 2025-10-01 23:10:35 +00:00 Compare
srxl reviewed 2025-10-02 09:21:39 +00:00
@ -0,0 +15,4 @@
description = "Derivations to join under a single store path";
type =
let
initial = lib.types.attrs.of lib.types.derivation;
Member

What's the motivation for accepting an attrset at all here?

What's the motivation for accepting an attrset at all here?
Author
Member

I just followed the package's deps.xx.yy, and I agree with its design that will allow someyhing like

deps = {
  inherit (config.packages.fondation)
    bash
    coreutils
    findutils;
};

Which really is saner than with ...; [ x y z ];

I just followed the package's `deps.xx.yy`, and I agree with its design that will allow someyhing like ```nix deps = { inherit (config.packages.fondation) bash coreutils findutils; }; ``` Which really is saner than `with ...; [ x y z ];`
Member

Hmmm, fair enough. I don't have a particular preference towards either (although I will admit with is a little yucky), but it makes sense to have the option.

Hmmm, fair enough. I don't have a particular preference towards either (although I will admit `with` is a little yucky), but it makes sense to have the option.
srxl marked this conversation as resolved
srxl reviewed 2025-10-02 09:32:17 +00:00
@ -0,0 +108,4 @@
passAsFile = [ "content" ];
});
in
# FIXME: surely we have an assertion system to use instead of this ? right ?
Member

I don't think we have one yet. Probably should do that somewhere...

I don't think we have one yet. Probably should do that somewhere...
Author
Member
opened [aucolotl/lib #8](https://git.auxolotl.org/auxolotl/lib/pulls/8)
austreelis marked this conversation as resolved
RossSmyth reviewed 2025-10-02 19:38:10 +00:00
@ -0,0 +42,4 @@
Path of the program used to execute this file as a script.
If set, a shebang is prepended to the file.
'';
type = lib.types.nullish lib.types.path;
Contributor

Should support passing flags to the interpreters as well.

Should support passing flags to the interpreters as well.
Author
Member

Added it, I'd like to know how you like it :)

Added it, I'd like to know how you like it :)
austreelis force-pushed austreelis/feat/trivial-builders-v1 from f780a7698c to 9e30c5b82f 2025-10-04 13:30:29 +00:00 Compare
austreelis force-pushed austreelis/feat/trivial-builders-v1 from 9e30c5b82f to 5bfeb1cecb 2025-10-04 16:27:13 +00:00 Compare
Author
Member

I initially wanted to support nixpkgs.writeShellApplication-style runtimeDeps, but if we don't want to assume settings.interpreter.executable is a shell (bash and co), we'd need to wrap the resulting script to set PATH, so I'll be leaving it for later, and won't do it in this PR.

I initially wanted to support `nixpkgs.writeShellApplication`-style `runtimeDeps`, but if we don't want to assume `settings.interpreter.executable` is a shell (bash and co), we'd need to wrap the resulting script to set `PATH`, so I'll be leaving it for later, and won't do it in this PR.
austreelis force-pushed austreelis/feat/trivial-builders-v1 from 5bfeb1cecb to e841e5ea11 2025-10-04 16:52:30 +00:00 Compare
Author
Member

And here I go double-posting to say that I thing this PR only needs cleanup and aftercare, I have implemented everything I wanted ! I'd appreciate a review pass, I don't plan to touch the API again or change the builders' behavior :3

And here I go double-posting to say that I thing this PR only needs cleanup and aftercare, I have implemented everything I wanted ! I'd appreciate a review pass, I don't plan to touch the API again or change the builders' behavior :3
saiko reviewed 2025-10-04 18:17:05 +00:00
saiko left a comment
First-time contributor

I would suggest quoting variables always, using x() over function x, using [[ over [, and moving the then/do into the same line as the if/for. That's the most common formatting/constructs used for bash and afaik there's no reason to use [ over [[ ever. And it looks like you wanted to unify the log functions, I would also recommend that.

(Also I'm personally not a fan of set -u because it needs you to write weird constructs like ${var:-} and most people are not used to it, so imo it's trading one pitfall for another, but fine enough if you want the additional strictness)

I would suggest quoting variables always, using `x()` over `function x`, using `[[` over `[`, and moving the `then`/`do` into the same line as the `if`/`for`. That's the most common formatting/constructs used for bash and afaik there's no reason to use `[` over `[[` ever. And it looks like you wanted to unify the log functions, I would also recommend that. (Also I'm personally not a fan of set -u because it needs you to write weird constructs like ${var:-} and most people are not used to it, so imo it's trading one pitfall for another, but fine enough if you want the additional strictness)
@ -0,0 +16,4 @@
echo -n "$1$sep" >&2
done
echo "${1:-}" >&2
}
First-time contributor

This entire function is not used, is it?

This entire function is not used, is it?
@ -0,0 +22,4 @@
echo "[PANIC] $1" >&2
while shift && [ -n "${1:+x}" ]
do echo -e "$1" >&2
done
First-time contributor
panic() {
  echo -n "[PANIC] $1" >&2

  while shift; do
    echo -e "$1" >&2
  done

Why break if the argument is empty? That seems like a pitfall.

``` panic() { echo -n "[PANIC] $1" >&2 while shift; do echo -e "$1" >&2 done ``` Why break if the argument is empty? That seems like a pitfall.
@ -0,0 +34,4 @@
echo -n "$1, " >&2
done
echo "${1:-}" >&2
}
First-time contributor
info() {
  echo -n " [INFO] $1 " >&2

  while shift; do
    echo -n "$1" >&2
    (( $# > 1 )) && echo -n ", " >&2
  done

  echo
}
``` info() { echo -n " [INFO] $1 " >&2 while shift; do echo -n "$1" >&2 (( $# > 1 )) && echo -n ", " >&2 done echo } ```
@ -0,0 +38,4 @@
function debug { true; }
if ((NIX_DEBUG > 0 ))
then {
First-time contributor
debug() {
  :
}

if ((NIX_DEBUG > 0)); then
``` debug() { : } if ((NIX_DEBUG > 0)); then ```
@ -0,0 +46,4 @@
echo -n "$1, " >&2
done
echo "${1:-}" >&2
}
First-time contributor

This is the same as info() except for the tag. Is that what log() was supposed to be for?

This is the same as info() except for the tag. Is that what log() was supposed to be for?
Author
Member

upsy yes, I intended debug, info and panic to use log

upsy yes, I intended `debug`, `info` and `panic` to use `log`
@ -0,0 +96,4 @@
local path="${3:-}"
entry="${entry:+"${entry%%/}/"}$path"
entry="${entry:-/}"
# entry="$(normalize_path "$entry")"
First-time contributor

Was a normalize_path function supposed to be implemented here?

Was a normalize_path function supposed to be implemented here?
@ -0,0 +143,4 @@
while read -d $'\n' -r path
do
push_entry "$origin" "$entry" "$path"
done < <( \
First-time contributor

Pretty sure you don't need most of the \ here.

      done < <(
        find -L "$source" -maxdepth 1 -mindepth 1 -printf %P$'\n' \
        || panic "couldn't walk directory" "$source"
      )
Pretty sure you don't need most of the \ here. ``` done < <( find -L "$source" -maxdepth 1 -mindepth 1 -printf %P$'\n' \ || panic "couldn't walk directory" "$source" ) ```
@ -0,0 +173,4 @@
# # shellcheck disable=2154 # $out is an input
# if ! mkdir "$out" 2>/dev/null
# then panic "output directory $out already exists"
# fi
First-time contributor

Why is this here commented out?

Why is this here commented out?
Author
Member

Yeah I need to clean this up, I'll get to it when I do a cleanup pass.

Yeah I need to clean this up, I'll get to it when I do a cleanup pass.
Author
Member

I'm not used to writing bash, so re function, quoting, [[ etc. I'll take your word for it. The only thing I know for sure is that [ -z ${myVar:+x} ] never requires quoting (it's either "" or "x" so it cannot word-split again, and [ -z ] evaluates to false). But I guess quoting doesn't hurt.

[...] not a fan of set -u because it needs you to write weird constructs like ${var:-} and most people are not used to it, [...]

I do prefer setting -u, I know it's a convention I've seen in nix spaces, and I'd rather document bash's weirdness than try to avoid it I think (though I'm definitely open to changing my mind if I'm the only one holsing that belief).

Thanks for reviewing the scripts, it's dedinitely not my strong point !

I'm not used to writing bash, so re `function`, quoting, `[[` etc. I'll take your word for it. The only thing I know for sure is that `[ -z ${myVar:+x} ]` never requires quoting (it's either `""` or `"x"` so it cannot word-split again, and `[ -z ]` evaluates to false). But I guess quoting doesn't hurt. > [...] not a fan of set -u because it needs you to write weird constructs like ${var:-} and most people are not used to it, [...] I do prefer setting -u, I know it's a convention I've seen in nix spaces, and I'd rather document bash's weirdness than try to avoid it I think (though I'm definitely open to changing my mind if I'm the only one holsing that belief). Thanks for reviewing the scripts, it's dedinitely not my strong point !
austreelis force-pushed austreelis/feat/trivial-builders-v1 from e841e5ea11 to 35741de622 2025-10-06 18:12:53 +00:00 Compare
jakehamilton referenced this pull request from a commit 2025-10-18 22:21:53 +00:00
Member

I stuffed this through hydra even though it's a WIP - it's failing with the same eval errors as #22, I guess it needs a rebase?

I stuffed this through hydra even though it's a WIP - it's failing with the same eval errors as #22, I guess it needs a rebase?
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin +refs/pull/24/head:austreelis/feat/trivial-builders-v1
git checkout austreelis/feat/trivial-builders-v1

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git checkout main
git merge --no-ff austreelis/feat/trivial-builders-v1
git checkout austreelis/feat/trivial-builders-v1
git rebase main
git checkout main
git merge --ff-only austreelis/feat/trivial-builders-v1
git checkout austreelis/feat/trivial-builders-v1
git rebase main
git checkout main
git merge --no-ff austreelis/feat/trivial-builders-v1
git checkout main
git merge --squash austreelis/feat/trivial-builders-v1
git checkout main
git merge --ff-only austreelis/feat/trivial-builders-v1
git checkout main
git merge austreelis/feat/trivial-builders-v1
git push origin main
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: auxolotl/labs#24
No description provided.