add stdenv #2

2024-05-02T00:47:35Z

vlinkz commented

2024-05-02 00:47:35 +00:00

(Migrated from github.com)

Working towards implementing stdenv from nixpkgs decoupled from the rest of the repo

Edit:
Right now, the package structure is similar to nixpkgs by-name, with modification to allow for multiple versions, ie:

by-name
├── gd
│   ├── gd
│   │   ├── default.nix
│   │   └── packages.nix
│   └── gdbm
│       └── default.nix
...

In this case, package gdbm will be evaluated as gdbm = callPackage ./by-name/gdbm { }
Packages with a packages.nix file, such as gd will be imported as import "${path}/packages.nix" { inherit lib noSysDirs config overlays; } res self super. This idea allows for most (all?) packages and package sets to be placed in by-name.

Todo

build stdenv
build nix
clean up

Working towards implementing stdenv from nixpkgs decoupled from the rest of the repo Edit: Right now, the package structure is similar to nixpkgs [by-name](https://github.com/NixOS/nixpkgs/blob/master/pkgs/by-name/README.md), with modification to allow for multiple versions, ie: ``` by-name ├── gd │ ├── gd │ │ ├── default.nix │ │ └── packages.nix │ └── gdbm │ └── default.nix ... ``` In this case, package `gdbm` will be evaluated as `gdbm = callPackage ./by-name/gdbm { }` Packages with a `packages.nix` file, such as `gd` will be imported as `import "${path}/packages.nix" { inherit lib noSysDirs config overlays; } res self super`. This idea allows for most (all?) packages and package sets to be placed in `by-name`. ### Todo - [x] build stdenv - [x] build nix - [x] clean up

❤️ 2

Sorixelle (Migrated from github.com) reviewed 2024-05-02 00:47:35 +00:00

Sorixelle commented

2024-05-06 06:11:03 +00:00

(Migrated from github.com)

Just checking in - how are we tracking on this PR? I'd like to start using some of this stuff in the JS repo once it's ready.

vlinkz commented

2024-05-06 16:24:17 +00:00

(Migrated from github.com)

Been a bit busy irl so haven't had as much time to work on this as I'd like, but I think I've gotten past the bulk of the issues and now just have to rip all the necessary packages from nixpkgs. In the meantime, you can use the nixPackages output of this repo, and just switch it to auxPackages later on (although might cause friction in migration if packages used aren't in core yet). Regardless at least stdenv should be done sometime this week (hopefully)

Been a bit busy irl so haven't had as much time to work on this as I'd like, but I think I've gotten past the bulk of the issues and now just have to rip all the necessary packages from nixpkgs. In the meantime, you can use the `nixPackages` output of this repo, and just switch it to `auxPackages` later on (although might cause friction in migration if packages used aren't in core yet). Regardless at least stdenv should be done sometime this week (hopefully)

Sorixelle commented

2024-05-13 03:40:21 +00:00

(Migrated from github.com)

Looking at the amount of changes happening here, I worry this is going to become impossible to review once it's ready. Would it be possible to split some of this work into smaller chunks?

Also, have we drafted up any documentation on what our "stdenv"-equivalent actually looks like? It seems like we're copypasting the one from Nixpkgs mostly here, but I'm not sure if we've talked much about if that's the path we want to go down (haven't been keeping super up to date the past few days, apologies)

Looking at the amount of changes happening here, I worry this is going to become impossible to review once it's ready. Would it be possible to split some of this work into smaller chunks? Also, have we drafted up any documentation on what our "stdenv"-equivalent actually looks like? It *seems* like we're copypasting the one from Nixpkgs mostly here, but I'm not sure if we've talked much about if that's the path we want to go down (haven't been keeping super up to date the past few days, apologies)

vlinkz commented

2024-05-13 04:08:45 +00:00

(Migrated from github.com)

Yeah, it's mostly copy-paste from nixpkgs with the (almost) bare minimum to build stdenv and nix (and lix). Right now that does include a lot of packages which can hopefully be reduced over time. I was hoping to build a starting point where we can test things on and allow for language-specific SIGs to start work.

Looking at the amount of changes happening here, I worry this is going to become impossible to review once it's ready. Would it be possible to split some of this work into smaller chunks?

Yeah, I don't think there's any good way around it given the web of dependencies. Most of the stuff tying everything together is in pkgs/top-level, and all of the packages are in pkgs/by-name. There's still a lot of commented-out stuff in top-level/all-packages.nix, but mostly build-support stuff that I'm hoping to move to a separate place, nowhere nearly as bad as the nixpkgs version.

For progress, as of the latest commit, both nix and lix can be built successfully. From putting together this proof of concept, I noticed a few things we'll need to decide upon:

How do we want to implement a testing infrastructure?
How do we want to implement alternative versions of core packages that depend on packages from another SIG?
- Specifically, I noticed that cmake has a version that uses qt5Packages for its UI. Should cmakeWithGui be defined in a KDE/Qt SIG repo, or in top-level?
Some core language packages, such as rust and python3 are required to build some base utilities. How should this be managed? Under the direction of the core SIG, or the language-specific SIG? Either requires strong coordination between the two.

Yeah, it's mostly copy-paste from nixpkgs with the (almost) bare minimum to build stdenv and nix (and lix). Right now that does include a lot of packages which can hopefully be reduced over time. I was hoping to build a starting point where we can test things on and allow for language-specific SIGs to start work. > Looking at the amount of changes happening here, I worry this is going to become impossible to review once it's ready. Would it be possible to split some of this work into smaller chunks? Yeah, I don't think there's any good way around it given the web of dependencies. Most of the stuff tying everything together is in `pkgs/top-level`, and all of the packages are in `pkgs/by-name`. There's still a lot of commented-out stuff in `top-level/all-packages.nix`, but mostly `build-support` stuff that I'm hoping to move to a separate place, nowhere nearly as bad as the nixpkgs version. For progress, as of the latest commit, both `nix` and `lix` can be built successfully. From putting together this proof of concept, I noticed a few things we'll need to decide upon: - How do we want to implement a testing infrastructure? - How do we want to implement alternative versions of core packages that depend on packages from another SIG? - Specifically, I noticed that `cmake` has a version that uses `qt5Packages` for its UI. Should `cmakeWithGui` be defined in a KDE/Qt SIG repo, or in top-level? - Some core language packages, such as `rust` and `python3` are required to build some base utilities. How should this be managed? Under the direction of the core SIG, or the language-specific SIG? Either requires strong coordination between the two.

isabelroses commented

2024-05-13 10:36:27 +00:00

(Migrated from github.com)

@vlinkz you have commited result into the pkgs directory.

@vlinkz you have commited `result` into the `pkgs` directory.

tcmal commented

2024-05-13 13:37:07 +00:00

(Migrated from github.com)

Cool, I was planning on working on this today but you beat me to it.

I think there's some room to reduce the tree further, but that isn't necessarily a priority right now. For reference, this is all the files that appeared to be evaluated when I built lix on x86_64-linux, and this is every derivation.

I also did some work on separating lib out to its own repository, I'll try and make another PR stacked on top of this one in a bit.

Cool, I was planning on working on this today but you beat me to it. I think there's some room to reduce the tree further, but that isn't necessarily a priority right now. For reference, [this](https://pastebin.com/Xx0g6Af9) is all the files that appeared to be evaluated when I built lix on x86_64-linux, and [this](https://pastebin.com/up5fF4Nd) is every derivation. I also did some work on separating `lib` out to its own repository, I'll try and make another PR stacked on top of this one in a bit.

❤️ 2

vlinkz commented

2024-05-13 14:07:18 +00:00

(Migrated from github.com)

I think there's some room to reduce the tree further, but that isn't necessarily a priority right now. For reference, this is all the files that appeared to be evaluated when I built lix on x86_64-linux, and this is every derivation.

Yes, there's definitely more packages than just that directly used it the build, mostly due to optional dependencies not used in the x86_64-linux version of the packages used. For example a lot of packages depend on darwin and it's set of packages, but that doesn't appear in the x86_64-linux evaluation. However without it, the build still fails with missing dependencies. Rather than heavily modify most of the packages to remove such dependencies, I packaged them given they will probably be necessary in the future.

> I think there's some room to reduce the tree further, but that isn't necessarily a priority right now. For reference, [this](https://pastebin.com/Xx0g6Af9) is all the files that appeared to be evaluated when I built lix on x86_64-linux, and [this](https://pastebin.com/up5fF4Nd) is every derivation. Yes, there's definitely more packages than just that directly used it the build, mostly due to optional dependencies not used in the x86_64-linux version of the packages used. For example a lot of packages depend on `darwin` and it's set of packages, but that doesn't appear in the x86_64-linux evaluation. However without it, the build still fails with missing dependencies. Rather than heavily modify most of the packages to remove such dependencies, I packaged them given they will probably be necessary in the future.

👍 1

vlinkz commented

2024-05-13 15:23:53 +00:00

(Migrated from github.com)

While this isn't perfect yet, I've opened this for review so that once/if merged, it can be more straightforward for others to contribute, and so that other SIG repos can start work depending on the packages it exports (stdenv, etc). As mentioned before, given the volume of changes it'll be impossible to review everything at once. I'm proposing we use this as a starting point to bootstrap the core repo and we fix issues, clean up the organization, and make other changes along the way.

While this isn't perfect yet, I've opened this for review so that once/if merged, it can be more straightforward for others to contribute, and so that other SIG repos can start work depending on the packages it exports (`stdenv`, etc). As mentioned before, given the volume of changes it'll be impossible to review everything at once. I'm proposing we use this as a starting point to bootstrap the core repo and we fix issues, clean up the organization, and make other changes along the way.

isabelroses (Migrated from github.com) reviewed 2024-05-13 16:31:10 +00:00

pkgs/pkgs-lib/formats/hocon/test/backwards-compatibility/default.nix

					
				@ -0,0 +1,65 @@

				{ lib, formats, stdenvNoCC, writeText, ... }:

isabelroses (Migrated from github.com) commented

2024-05-13 16:30:21 +00:00

lib and writeText seem to be unused

isabelroses commented

2024-05-13 16:32:05 +00:00

(Migrated from github.com)

There seems to be a lot of dead nix code, I would comment on more of it but GitHub cannot handle changes of this size well.

vlinkz (Migrated from github.com) reviewed 2024-05-13 16:51:57 +00:00

pkgs/pkgs-lib/formats/hocon/test/backwards-compatibility/default.nix

					
				@ -0,0 +1,65 @@

				{ lib, formats, stdenvNoCC, writeText, ... }:

vlinkz (Migrated from github.com) commented

2024-05-13 16:51:56 +00:00

yes, within nixpkgs and within this pr there is a lot of dead code and unused dependencies. I removed a few that I stumbled across putting this together, but I'm certain that there are a lot more instances of this

vlinkz commented

2024-05-13 17:06:09 +00:00

(Migrated from github.com)

There seems to be a lot of dead nix code, I would comment on more of it but GitHub cannot handle changes of this size well.

Yes, in addition to that there is a lot of commented out code such as in perl-packages.nix or python-packages.nix. I left the structure the same as its original nixpkgs form, but we will need to decide on what to keep and how to separate other parts into language-specific SIGs.

I also commented out a number of passthru and passthru.tests attributes that require extra test packages but are not used for package evaluation: libarchive/default.nix#L133-135. We'll need to decide upon how we want to format tests and if we even want to keep them within the package derivations.

> There seems to be a lot of dead nix code, I would comment on more of it but GitHub cannot handle changes of this size well. Yes, in addition to that there is a lot of commented out code such as in [perl-packages.nix](https://github.com/auxolotl/core/blob/072e19ee33382b9bf0789e8214084783a5a0a284/pkgs/by-name/pe/perl/perl-packages.nix) or [python-packages.nix](https://github.com/auxolotl/core/blob/072e19ee33382b9bf0789e8214084783a5a0a284/pkgs/by-name/py/python/python-packages.nix). I left the structure the same as its original nixpkgs form, but we will need to decide on what to keep and how to separate other parts into language-specific SIGs. I also commented out a number of `passthru` and `passthru.tests` attributes that require extra test packages but are not used for package evaluation: [libarchive/default.nix#L133-135](https://github.com/auxolotl/core/blob/072e19ee33382b9bf0789e8214084783a5a0a284/pkgs/by-name/li/libarchive/default.nix#L133-L135). We'll need to decide upon how we want to format tests and if we even want to keep them within the package derivations.

👍 1

isabelroses commented

2024-05-13 17:08:47 +00:00

(Migrated from github.com)

I think the best way to go about this, is to merge this PR. And we can handle what we want to keep in another PR such that we can cleanly see the difference between this and nixpkgs. And also to something working into core.

👍 1

isabelroses (Migrated from github.com) approved these changes 2024-05-13 17:08:57 +00:00

isabelroses (Migrated from github.com) approved these changes 2024-05-13 17:09:07 +00:00

Sign in to join this conversation.

No reviewers

Sorixelle

isabelroses