feat: add lib.fetchurl function #13

srxl · 2025-10-03T14:43:24Z

srxl commented

2025-10-03 14:43:24 +00:00

Same change as auxolotl/labs#27 - we should use that wrapper here as well.

srxl added 2 commits 2025-10-03 14:43:25 +00:00

feat: add builtin:fetchurl wrapper function to lib 24ae64d724

refactor: use lib.fetchurl to download package sources f763483699

Performed by running: rg --files-with-matches "builtins\.fetchurl" | xargs sed -i -e
's/builtins\.fetchurl/lib\.fetchurl/g'

aleksi referenced this pull request

2025-10-03 18:24:05 +00:00

Allow overriding configuration from outside #14

austreelis referenced this pull request

2025-10-03 19:28:00 +00:00

Allow overriding configuration from outside #14

srxl force-pushed fetchurl from f763483699 to 5418b05ccb

2025-10-04 06:49:28 +00:00

Compare

srxl changed title from ~~WIP: feat: add lib.fetchurl function~~ to feat: add lib.fetchurl function

2025-10-04 07:27:26 +00:00

srxl force-pushed fetchurl from 5418b05ccb to 5ab0169d3e

2025-10-04 07:30:09 +00:00

Compare

srxl referenced this pull request

2025-10-04 08:01:56 +00:00

WIP: implement lib.fetchTarball #15

jakehamilton commented

2025-10-04 20:10:24 +00:00

Is this needed since we talked about moving this to lib?

Is this needed since we talked about moving this to `lib`?

srxl commented

2025-10-04 21:55:22 +00:00

Are we? FWIW, I disagree with doing that - it's specific to bootstrapping, and I don't think it's something we should really encourage use of, especially outside of foundation/tidepool.

jakehamilton commented

2025-10-05 01:29:25 +00:00

@srxl wrote in #13 (comment):

Are we? FWIW, I disagree with doing that - it's specific to bootstrapping, and I don't think it's something we should really encourage use of, especially outside of foundation/tidepool.

These are Nix primitives (fetchurl, fetchTarball, <nix/fetchurl.nix>) without any system requirements. They are provided by Nix itself, we would only be aliasing them on lib like other helpers. Nothing stops a user from using them if we choose not to alias them. The only difference is that it would make 'lib` consistent.

@srxl wrote in https://git.auxolotl.org/auxolotl/foundation/pulls/13#issuecomment-984: > Are we? FWIW, I disagree with doing that - it's specific to bootstrapping, and I don't think it's something we should really encourage use of, especially outside of foundation/tidepool. These are Nix primitives (fetchurl, fetchTarball, <nix/fetchurl.nix>) without any system requirements. They are provided by Nix itself, we would only be aliasing them on `lib` like other helpers. Nothing stops a user from using them if we choose not to alias them. The only difference is that it would make 'lib` consistent.

srd424 commented

2025-10-05 11:27:21 +00:00

For some reason this fails on hydra at the moment, will investigate later: https://hydra.aux-cache.dev/build/2111/nixlog/1/tail

jakehamilton requested changes 2025-10-18 15:33:09 +00:00

default.nix

					
				@ -5,2 +5,2 @@

					pins = import ./npins;

					lib = import pins.lib;

				  pins = import ./npins;

				  lib = (import pins.lib).extend (final: prev: prev // {

jakehamilton commented

2025-10-18 15:33:05 +00:00

Can we put the fetcher in a namespace?

Irenes commented

2025-10-18 19:33:48 +00:00

I attempted to merge this into main locally on my end and test it. After doing that, I get the following:

$ nix flake check --all-systems
warning: unknown flake output 'extras'
error:
       … while checking flake output 'packages'
         at /nix/store/vj4jv17nyaqxsyc9n3ykp8b55qjgipgh-source/flake.nix:23:7:
           22|
           23|       packages = forEachSystem (
             |       ^
           24|         system:

       … while evaluating the attribute 'value'
         at /nix/store/21d458jy4pnb8mwd8ss6wanixkrk3gbh-lib.git-7552ab4/src/options/default.nix:339:9:
          338|       // {
          339|         value = builtins.addErrorContext "while evaluating the option `${identifier}`:" value;
             |         ^
          340|         highestPriority = merged.raw.highestPriority;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: attribute 'fetchurl' missing
       at /nix/store/vj4jv17nyaqxsyc9n3ykp8b55qjgipgh-source/src/stages/stage0/phases/phase00.nix:101:13:
          100|       executable = lib.modules.overrides.default (
          101|         lib.fetchurl {
             |             ^
          102|           name = "hex0-seed";

I think this has something to do with the wrong version of lib being passed through?

Anyway, I agree with Jake that this belongs in lib, so I think we should add it there rather than here? Unless I'm misunderstanding, I think that this is not specific to bootstrapping, but please feel free to tell me more.

I attempted to merge this into main locally on my end and test it. After doing that, I get the following: ``` $ nix flake check --all-systems warning: unknown flake output 'extras' error: … while checking flake output 'packages' at /nix/store/vj4jv17nyaqxsyc9n3ykp8b55qjgipgh-source/flake.nix:23:7: 22| 23| packages = forEachSystem ( | ^ 24| system: … while evaluating the attribute 'value' at /nix/store/21d458jy4pnb8mwd8ss6wanixkrk3gbh-lib.git-7552ab4/src/options/default.nix:339:9: 338| // { 339| value = builtins.addErrorContext "while evaluating the option `${identifier}`:" value; | ^ 340| highestPriority = merged.raw.highestPriority; (stack trace truncated; use '--show-trace' to show the full trace) error: attribute 'fetchurl' missing at /nix/store/vj4jv17nyaqxsyc9n3ykp8b55qjgipgh-source/src/stages/stage0/phases/phase00.nix:101:13: 100| executable = lib.modules.overrides.default ( 101| lib.fetchurl { | ^ 102| name = "hex0-seed"; ``` I think this has something to do with the wrong version of `lib` being passed through? Anyway, I agree with Jake that this belongs in lib, so I think we should add it there rather than here? Unless I'm misunderstanding, I think that this is **not** specific to bootstrapping, but please feel free to tell me more.

aleksi commented

2025-10-24 16:13:06 +00:00

For some reason this fails on hydra at the moment

I just hit the same failure locally, but without this patch. It seemed to be intermittent. Maybe related to gnutar impurity in ctime/mtime?

> For some reason this fails on hydra at the moment I just hit the same failure locally, but without this patch. It seemed to be intermittent. Maybe related to gnutar impurity in ctime/mtime?

Irenes commented

2025-10-24 19:27:38 +00:00

huhhh

the Hydra failure above ends with

+ make 'AR=tcc -ar'
CDPATH="${ZSH_VERSION+.}:" && cd . && /nix/store/1ip5l060r175qx00nlndac68y7xxiiad-bash-2.05b/bin/bash '/build/make-4.4.1/build-aux/missing' aclocal-1.16 -I m4
/build/make-4.4.1/build-aux/missing: line 81: aclocal-1.16: command not found
WARNING: 'aclocal-1.16' is missing on your system.
         You should only need it if you modified 'acinclude.m4' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'aclocal' program is part of the GNU Automake package:
         <https://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <https://www.gnu.org/software/autoconf>
         <https://www.gnu.org/software/m4/>
         <https://www.perl.org/>
make: *** [Makefile:655: aclocal.m4] Error 127

ohhhh.... yes I see. normally it wouldn't need to re-run autotools, because their output is checked into the source tree. something about the timestamps, though, convinced it that the autoconf inputs were newer than their outputs, so it tried to re-run but the dependencies it needs for that don't exist at this stage of bootstrapping.

was this what your local failure looked like as well? the gnutar change should only affect certain corner cases that I don't think actually apply, but something else maybe messed with the timestamps?

huhhh the Hydra failure above ends with ``` + make 'AR=tcc -ar' CDPATH="${ZSH_VERSION+.}:" && cd . && /nix/store/1ip5l060r175qx00nlndac68y7xxiiad-bash-2.05b/bin/bash '/build/make-4.4.1/build-aux/missing' aclocal-1.16 -I m4 /build/make-4.4.1/build-aux/missing: line 81: aclocal-1.16: command not found WARNING: 'aclocal-1.16' is missing on your system. You should only need it if you modified 'acinclude.m4' or 'configure.ac' or m4 files included by 'configure.ac'. The 'aclocal' program is part of the GNU Automake package: <https://www.gnu.org/software/automake> It also requires GNU Autoconf, GNU m4 and Perl in order to run: <https://www.gnu.org/software/autoconf> <https://www.gnu.org/software/m4/> <https://www.perl.org/> make: *** [Makefile:655: aclocal.m4] Error 127 ``` ohhhh.... yes I see. normally it wouldn't need to re-run autotools, because their output is checked into the source tree. something about the timestamps, though, convinced it that the autoconf inputs were newer than their outputs, so it tried to re-run but the dependencies it needs for that don't exist at this stage of bootstrapping. was this what your local failure looked like as well? the gnutar change should only affect certain corner cases that I don't think actually apply, but something else maybe messed with the timestamps?

srd424 commented

2025-10-24 20:29:10 +00:00

This isn't something similar to what we fixed in PR#1 and PR#2, is it? Somehow the mode not always getting unpacked correctly, and that causing problems .. somehow?? My memory is so bad the days, I can't remember that far back ..

This pull request has changes conflicting with the target branch.

default.nix
src/stages/stage1/musl/boot.nix
src/stages/stage1/tinycc/musl.nix

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u fetchurl:srxl-fetchurl

git checkout srxl-fetchurl

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git checkout main

git merge --no-ff srxl-fetchurl

git checkout srxl-fetchurl

git rebase main

git checkout main

git merge --ff-only srxl-fetchurl

git checkout srxl-fetchurl

git rebase main

git checkout main

git merge --no-ff srxl-fetchurl

git checkout main

git merge --squash srxl-fetchurl