auxolotl/labs
26
12
Fork 16
Code Issues 6 Pull requests 9 Projects Releases Packages Wiki Activity Actions

gnutar: Make reproducible by default #43

Merged
Irenes merged 2 commits from RossSmyth/labs:wrapTar into main 2025-10-28 02:25:33 +00:00
Conversation 12 Commits 2 Files changed 3 +53 -4
RossSmyth commented 2025-10-22 18:41:26 +00:00
Contributor
Copy link

Sometimes reproducibility is a concern when tarring and untarring. This should mitigate it.

Sometimes reproducibility is a concern when tarring and untarring. This should mitigate it.
RossSmyth added 1 commit 2025-10-22 18:41:34 +00:00
RossSmyth commented 2025-10-22 18:43:13 +00:00
Author
Contributor
Copy link

I'm putting this in labs first. If it works well I'll probably add it to foundation as well.

I'm putting this in labs first. If it works well I'll probably add it to foundation as well.
RossSmyth commented 2025-10-22 18:52:50 +00:00
Author
Contributor
Copy link

It builds stage3 GCC but fails to build stage4 GCC with this patch

It builds stage3 GCC but fails to build stage4 GCC with this patch
RossSmyth force-pushed wrapTar from ee06186de6 to d3e5c10ab3 2025-10-23 18:00:42 +00:00 Compare
RossSmyth commented 2025-10-23 18:06:35 +00:00
Author
Contributor
Copy link

Tested with
nom build -f tidepool packages.foundation.gcc.latest.packages.x86_64-linux.x86_64-linux.package
and it built successfully.

Tested with `nom build -f tidepool packages.foundation.gcc.latest.packages.x86_64-linux.x86_64-linux.package` and it built successfully.
Irenes reviewed 2025-10-23 23:10:32 +00:00
tidepool/src/packages/foundation/gnutar/versions/1.35-stage1.nix Outdated
@ -111,0 +113,4 @@
# to provide a deterministic substitute for the "current" time. Note that
# 315532800 = 1980-01-01 12:00:00. We use this date because python's wheel
# implementation uses zip archive and zip does not support dates going back to
# 1970.
Irenes commented 2025-10-23 23:10:31 +00:00
Owner
Copy link

Nice approach, this for sure seems like something we should do. @jakehamilton any thoughts on the choice of date?

Sorry to be picking about wording, but is this midnight or noon? It sounds like you're saying it's noon; I can imagine reasons for that choice but would like to be explicit about it.

Also I'd like to better understand the relevance of the zip format; I didn't realize gnutar did zip? Is the concern that files we extract, are then going to be compressed by zip as part of a later, non-gnutar build phase? If that's all it is, I feel like it would be better to deal with that concern in the wheel build process and use 0 as the default time for gnutar, but it's for sure up for discussion.

Nice approach, this for sure seems like something we should do. @jakehamilton any thoughts on the choice of date? Sorry to be picking about wording, but is this midnight or noon? It sounds like you're saying it's noon; I can imagine reasons for that choice but would like to be explicit about it. Also I'd like to better understand the relevance of the zip format; I didn't realize gnutar did zip? Is the concern that files we extract, are then going to be compressed by zip as part of a later, non-gnutar build phase? If that's all it is, I feel like it would be better to deal with that concern in the wheel build process and use 0 as the default time for gnutar, but it's for sure up for discussion.
RossSmyth commented 2025-10-24 03:00:34 +00:00
Author
Contributor
Copy link

Hm, I copied this from SOURCE_DATE_EPOCH in Nixpkgs, but this isn't applicable to gnutar

Note on the epoch var:
https://reproducible-builds.org/specs/source-date-epoch/
https://reproducible-builds.org/docs/source-date-epoch/

This is only relevant if that var is set, and nix-shell is used with Python.

Hm, I copied this from SOURCE_DATE_EPOCH in Nixpkgs, but this isn't applicable to gnutar Note on the epoch var: https://reproducible-builds.org/specs/source-date-epoch/ https://reproducible-builds.org/docs/source-date-epoch/ This is only relevant if that var is set, and nix-shell is used with Python.
Irenes commented 2025-10-24 18:14:56 +00:00
Owner
Copy link

Thanks - most of my questions above no longer apply per the latest changes, but please put that "docs" link in a comment somewhere. It does an amazing job of explaining and contextualizing everything, and I think it will be very helpful for future readers to know about it. Also that's really cool that a spec for this exists.

Thanks - most of my questions above no longer apply per the latest changes, but please put that "docs" link in a comment somewhere. It does an amazing job of explaining and contextualizing everything, and I think it will be very helpful for future readers to know about it. Also that's really cool that a spec for this exists.
RossSmyth marked this conversation as resolved
srd424 commented 2025-10-24 11:29:19 +00:00
Member
Copy link

Hydra seems happy: https://hydra.aux-cache.dev/jobset/aux-prs/labs-pr43

Hydra seems happy: https://hydra.aux-cache.dev/jobset/aux-prs/labs-pr43
RossSmyth force-pushed wrapTar from d3e5c10ab3 to d0e1fb6709 2025-10-24 17:27:10 +00:00 Compare
RossSmyth commented 2025-10-24 17:29:38 +00:00
Author
Contributor
Copy link

I changed it based upon feedback on Matrix so that it uses a hook rather than a wrapper. This is so that the same derivation can be used by end-users and in builds.

I do not think using TAR_OPTIONS would be supported by all build system since some do not inherit all environment variables, I am mainly thinking of Meson. But PATH should almost always be inherited.

I changed it based upon feedback on Matrix so that it uses a hook rather than a wrapper. This is so that the same derivation can be used by end-users and in builds. I do not think using `TAR_OPTIONS` would be supported by all build system since some do not inherit all environment variables, I am mainly thinking of Meson. But PATH should almost always be inherited.
RossSmyth commented 2025-10-24 17:47:40 +00:00
Author
Contributor
Copy link

Hm I'm not sure if this hook is doing anything because it is wrong, but I was able to build packages with it fine.

Hm I'm not sure if this hook is doing anything because it is wrong, but I was able to build packages with it fine.
RossSmyth reviewed 2025-10-24 17:48:20 +00:00
tidepool/src/packages/foundation/gnutar/versions/1.35-stage1.nix Outdated
@ -87,0 +96,4 @@
cat << EOF > "$tar_wrap_dir/tar"
#!${packages.foundation.bash.versions."5.2.15-bootstrap".package}/bin/bash
TAR_OPTIONS="--mtime=@1 --owner=0 --group=0 --numeric-owner --sort=name" "$tar_out/bin/.tar" "\$@"
RossSmyth commented 2025-10-24 17:48:19 +00:00
Author
Contributor
Copy link

There is nothing at /bin/.tar but I was able to build GCC and such

There is nothing at `/bin/.tar` but I was able to build GCC and such
RossSmyth marked this conversation as resolved
RossSmyth commented 2025-10-24 17:51:09 +00:00
Author
Contributor
Copy link

Ok found the issue. Needed to passthru the hooks too.

Ok found the issue. Needed to passthru the hooks too.
RossSmyth force-pushed wrapTar from d0e1fb6709 to bda09169f6 2025-10-24 17:56:21 +00:00 Compare
vlinkz reviewed 2025-10-25 01:53:45 +00:00
vlinkz left a comment
Owner
Copy link

We should also probably add this wrapper hook to bootstrap too

We should also probably add this wrapper hook to bootstrap too
tidepool/src/packages/foundation/gnutar/versions/1.35-stage1.nix Outdated
@ -87,0 +92,4 @@
# We want to ensure we prepend to PATH so our wrapper is picked first.
"aux:wrap:tar" = lib.dag.entry.before [ "unpack" ] ''
tar_wrap_dir=$(mktemp -d)
tar_out=${packages.foundation.gnutar.versions."1.35-stage1".package}
vlinkz commented 2025-10-25 01:51:50 +00:00
Owner
Copy link

Should probably be config.package to ensure we're pointing to this instance of the gnutar package

Should probably be `config.package` to ensure we're pointing to this instance of the gnutar package
RossSmyth commented 2025-10-25 15:59:55 +00:00
Author
Contributor
Copy link

Yeah I asked on matrix if packages have a fixed-point arg as I was unaware of config.package before.

Yeah I asked on matrix if packages have a fixed-point arg as I was unaware of `config.package` before.
RossSmyth marked this conversation as resolved
tidepool/src/packages/foundation/gnutar/versions/1.35-stage1.nix Outdated
@ -87,0 +95,4 @@
tar_out=${packages.foundation.gnutar.versions."1.35-stage1".package}
cat << EOF > "$tar_wrap_dir/tar"
#!${packages.foundation.bash.versions."5.2.15-bootstrap".package}/bin/bash
vlinkz commented 2025-10-25 01:52:56 +00:00
Owner
Copy link

Going to test it out, but this should probably be added to deps.host and then use config.deps.bash here instead to ensure we have the right platform set

Going to test it out, but this should probably be added to `deps.host` and then use `config.deps.bash` here instead to ensure we have the right platform set
RossSmyth commented 2025-10-25 16:01:29 +00:00
Author
Contributor
Copy link

Ok! I wasn't sure how to specify "use a bash that runs on host"

Ok! I wasn't sure how to specify "use a bash that runs on host"
RossSmyth marked this conversation as resolved
RossSmyth force-pushed wrapTar from bda09169f6 to 5847179532 2025-10-25 16:30:39 +00:00 Compare
RossSmyth force-pushed wrapTar from 5847179532 to f716dbd347 2025-10-25 16:32:26 +00:00 Compare
srd424 commented 2025-10-26 07:44:25 +00:00
Member
Copy link

Rebuilt on hydra, still looks OK: https://hydra.aux-cache.dev/jobset/aux-prs/labs-pr43

Rebuilt on hydra, still looks OK: https://hydra.aux-cache.dev/jobset/aux-prs/labs-pr43
Irenes commented 2025-10-28 02:25:23 +00:00
Owner
Copy link

good, yeah, it being a hook sounds like the right way to do it, to me. thanks for incorporating the comment I asked for. I'm merging it :)

good, yeah, it being a hook sounds like the right way to do it, to me. thanks for incorporating the comment I asked for. I'm merging it :)
Irenes merged commit 551bbc60de into main 2025-10-28 02:25:33 +00:00
RossSmyth deleted branch wrapTar 2025-10-28 03:15:24 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
  
Compat
Breaking
Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No labels
Compat
Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: auxolotl/labs#43
No description provided.
Delete branch "RossSmyth/labs:wrapTar"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?