From 02a45110c9d6b7cd7ecacdb819e31416bec8d65d Mon Sep 17 00:00:00 2001 From: KFears Date: Sat, 4 Oct 2025 20:15:48 +0000 Subject: [PATCH] docs: add basic docs on Tidepool and its package sets (#18) This is very far from perfect, but it lays some groundwork to Tidepool documentation. This follows my (very limited) understanding, and so is very much incomplete and WIP. I would like some help with the concepts I haven't quite understood <3 Reviewed-on: https://git.auxolotl.org/auxolotl/labs/pulls/18 Reviewed-by: Jake Hamilton Co-authored-by: KFears Co-committed-by: KFears --- tidepool/README.md | 173 +++++++++++++++++++++++++++++++- tidepool/src/packages/README.md | 96 ++++++++++++++++++ 2 files changed, 268 insertions(+), 1 deletion(-) create mode 100644 tidepool/src/packages/README.md diff --git a/tidepool/README.md b/tidepool/README.md index 0949df4..feebf29 100644 --- a/tidepool/README.md +++ b/tidepool/README.md @@ -4,7 +4,7 @@ Aux Tidepool is an initial package set built on top of [Aux Foundation](../found are created and managed using [Aux Lib](../lib)'s module system to allow for highly dynamic and extensible configuration. -## Usage +## Installation Packages can be imported both with and without Nix Flakes. To import them using Nix Flakes, add this repository as an input. @@ -26,3 +26,174 @@ let in # ... ``` + +## Usage + +To build a package, you can run the following command inside of the `tidepool` directory: + +```console +nix-build -A packages.foundation.bash.latest.packages.x86_64-linux.x86_64-linux.x86_64-linux.package +``` + +If you have `nix-command` experimental feature enabled, you can v3 CLI instead: + +```console +nix build -f . packages.foundation.bash.latest.packages.x86_64-linux.x86_64-linux.x86_64-linux.package +``` + +Using flakes, the difference is only in calling: + +```console +nix build .#packages.foundation.bash.latest.packages.x86_64-linux.x86_64-linux.x86_64-linux.package +``` + +This interface is verbose, but stable. We are still working out the details for +a friendlier interface! + +The full command syntax looks like this: + +```console +nix-build -A packages.${package-set}.${package}.${version}.packages.${build}.${host}.${target}.package +``` + +Tidepool packages can have multiple versions available. Tidepool also provides aliases +for convenience, currently "stable" and "latest". Aliases go directly after `${package}`: + +```console +nix-build -A packages.${package-set}.${package}.${alias}.packages.${build}.${host}.${target}.package +``` + +The actual available versions live one level deeper, under `versions` namespace: + +```console +nix-build -A packages.${package-set}.${package}.versions.${version}.packages.${build}.${host}.${target}.package +``` + +Here are a few practical examples for Tidepool. We can use a specific Bash version: + +```console +nix-build -A packages.foundation.bash.versions."5.2.15-stage1".packages.x86_64-linux.x86_64-linux.x86_64-linux.package +``` + +Get Bash cross-compiled from `i686-linux` to `x86_64-linux`: + +```console +nix build .#packages.foundation.bash.latest.packages.i686-linux.x86_64-linux.x86_64-linux.package +``` + +Or a native `aarch64-linux` Bash: + +```console +nix build .#packages.foundation.bash.latest.packages.aarch64-linux.aarch64-linux.aarch64-linux.package +``` + +Running this command will: + +- Bootstrap `i686-linux` `gcc` as a dependency of `x86_64-linux` `gcc` +- Incrementally bootstrap the actual `x86_64-linux` `gcc` +- Incrementally bootstrap `aarch64-linux` `gcc` from `x86_64-linux` `gcc` +- Build the native `aarch64-linux` `gcc` +- Build the native `aarch64-linux` `bash` + +It's very early in Tidepool's development, and this syntax and UX is expected to change! For now, +the only syntax available is this developer-facing style. You can expect improvements to be made! + +## Development + +Tidepool is built with Aux's portable submodules. They function similarly to submodules used by NixOS. +Using the submodules, Tidepool provides a nice and easy-to-work-with API. + +Tidepool can be used with flakes or without them. The solution we recommend +for non-flake usage is [npins](https://github.com/andir/npins). +Tidepool exports an attribute set, which can be viewed using `nix repl`. +We can visualize it! + +```console +tidepool +├───builders +├───lib +├───mirrors +├───packages +│ ├───aux +│ │ ├───a +│ │ ├───b +│ │ └───c +│ ├───foundation +│ │ ├───bash +│ │ ├─── ... +│ │ └───zlib +│ ├───context +│ │ └─── ... +│ └───... +├───extend +├───internal +├───new +└───preferences +``` + +There are 4 outputs that share the name with folders under `src/`: + +- `builders` - this is a set of Tidepool builders +- `lib` - this is the Tidepool library. It defines some nice-to-haves +for packaging +- `mirrors` - this is set of mirrors. Many projects, like GNU, provide +many mirrors - you can change a mirror to the one your prefer +- `packages` - this is a set for all of the Tidepool packages + +You may learn more about those attributes in their corresponding folders. + +Additionally, there are several other attributes, which don't have +corresponding folders: + +- `extend` - this is a function to extend Tidepool's package set by providing +additional package modules. Here's how to use it: + +```nix +let + tidepool = import ./tidepool; +in + tidepool.extend { + modules = [ + # Put additional modules you want to include here. + ]; + } +``` + +- `internal` - this is the internal attribute set. Currently, it only brings Tidepool's +dependencies into scope: more concretely, it exposes the `foundation` namespace from +the (perhaps confusingly-named) [Foundation](https://git.auxolotl.org/auxolotl/foundation) +repository, which does the bootstrap chain from a 357-byte file. It is not the same +as Tidepool's `foundation` namespace, which makes those early-bootstrap packages integrate +with Tidepool (e.g., enabling cross-compilation) +- `new` - this is a function that allows to create a new empty package set. +It allows you to create a package set from scratch. Using this function, +you will get access to Tidepool library functions and the interface for defining +your own builders (but you don't get builder implementations). Here's how to use it: + +```nix +let + tidepool = import ./tidepool; + + module = { config }: { + config = { + # Create a new package set from scratch, no packages exist unless you define them. + }; + }; +in + tidepool.new module; +``` + +- `preferences` - this is an attribute set that lists your preferences for the package sets. +It has the following shape: + +```nix +{ + packages = { + allow = { + broken = false; + incompatible = false; + }; + version = "latest"; + }; +} +``` diff --git a/tidepool/src/packages/README.md b/tidepool/src/packages/README.md new file mode 100644 index 0000000..abf0197 --- /dev/null +++ b/tidepool/src/packages/README.md @@ -0,0 +1,96 @@ +# Packages + +In this folder, the Tidepool package sets are stored. + +## Foundation + +The package set under `foundation` is a minimal set to start building everything +else. It builds upon and uses packages from [Foundation](https://git.auxolotl.org/auxolotl/foundation), +our early bootstrap chain. It provides a comprehensive set of packages that are +used to build all other basic Linux utilities, and are often transitive dependencies +of higher-level packages. + +The goal of `foundation` is to build `gcc` and `glibc`, +as well as some supplementary packages, so that other packages can be built with +them. It also sets up a basic Linux environment: `coreutils`, `diffutils`, `bash`, +and other utilities you'd expect to find on a Linux distro. + +To achieve this, we need a lot of other packages built, too! Some +examples include `python`, `xz` and `gzip`. + +Lastly, `foundation` also includes `patchelf`, a simple C utility. +It is not used in `foundation`, but it is very useful for building +more high-level packages: in particular, it allows us to patch +binaries to point to our dependencies. This "unbreaks" proprietary +binaries by making them point to Tidepool-provided dependencies. +Neat! + +The bootstrap chain is based on [full-source](https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/) +bootstrap, made possible with amazing work by Guix's people, +and Emily Trau's effort on porting it to Nix! +It bootstraps a proper `gcc` for `i686-linux`, which can then +be used to cross-compile a `gcc` for other architectures, such +as `x86_64-linux`. + +To build stage1 (`i686-linux`), run: + +```console +nix build --system i686-linux .#packages.foundation.glibc.versions."2.38-stage1-passthrough".packages.i686-linux.x86_64-linux.x86_64-linux.package +``` + +The full bootstrap chain is available in a confusingly-named +[Foundation repository](https://git.auxolotl.org/auxolotl/foundation). + +## Aux + +`aux`, at this time, is a trivial package set that demonstrates +the power of a fully-bootstrapped `foundation`. It consists of 3 +trivial packages: `a`, `b` and `c`. The chain of dependencies +is like this (the arrow points to the **dependency**): + +``` +a -> b -> c +``` + +Package `a` merely depends on `b`. + +Package `b` announces a build option `custom`. It sets up `context` +and sets up a hook after `unpack` phase. It also propagates dependency `c`. +It's worth discussing this in more depth, because it demonstrates a couple of +more advanced features. + +`context` is information that the package provides when it is used as dependency, +which builders and hooks may use. For example, to use a C dependency, you +would need to tell the compiler where to look for relevant headers by appending +a flag. The C dependency can have `context` defined which includes the necessary +compiler flag, and the package that uses this dependency will append this flag +automatically. + +[Nixpkgs](https://github.com/nixos/nixpkgs) uses a different approach to do this. +In Nixpkgs, packages include the `nix-support` subdirectory which contains the +same information as what `context` usually contains. Nixpkgs does this in +the builder code at build-time; Tidepool provides this information at eval time, +which provides the means to type-check the `context`. + +Package `b` provides a `context` to include the necessary directory: `include` under +the Nix store path for package `b`. This Nix store path is provided as an environment +variable from a hook. + +While Tidepool's architecture is subject to change, `context` already +has planned changes for it. When those changes happen - the information +about `context` will be moved to a historical design doc. But until then, +`context` is still relevant! + +`hooks` provide extension points for packages. With `hooks`, you can inject arbitrary build +instructions between, before or after the default build stages. Package `b` does exactly +that: it uses a hook to add entry after `unpack` phase and before others, which exports +an environment variable that contains the Nix store path, that is then used for `context`. + +Propagation ensures that the propagated dependency is also installed, when the package +is installed. Package `b` propagates dependency `c` - this means, when package `b` is +included as a dependency, package `c` is also included automatically. This is especially +useful when working with interpreted languages, such as Python - for them, all dependencies +need to be in a single scope, and propagation makes sure of that. + +Package `c` doesn't have dependencies, but it is otherwise identical +to package `b`.