docs: add basic docs on Tidepool and its package sets (#18)
This is very far from perfect, but it lays some groundwork to Tidepool documentation. This follows my (very limited) understanding, and so is very much incomplete and WIP. I would like some help with the concepts I haven't quite understood <3 Reviewed-on: #18 Reviewed-by: Jake Hamilton <jake.hamilton@hey.com> Co-authored-by: KFears <kfearsoff@gmail.com> Co-committed-by: KFears <kfearsoff@gmail.com>
This commit is contained in:
parent
b67847e7eb
commit
02a45110c9
2 changed files with 268 additions and 1 deletions
|
|
@ -4,7 +4,7 @@ Aux Tidepool is an initial package set built on top of [Aux Foundation](../found
|
|||
are created and managed using [Aux Lib](../lib)'s module system to allow for highly dynamic and
|
||||
extensible configuration.
|
||||
|
||||
## Usage
|
||||
## Installation
|
||||
|
||||
Packages can be imported both with and without Nix Flakes. To import them using Nix Flakes,
|
||||
add this repository as an input.
|
||||
|
|
@ -26,3 +26,174 @@ let
|
|||
in
|
||||
# ...
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
To build a package, you can run the following command inside of the `tidepool` directory:
|
||||
|
||||
```console
|
||||
nix-build -A packages.foundation.bash.latest.packages.x86_64-linux.x86_64-linux.x86_64-linux.package
|
||||
```
|
||||
|
||||
If you have `nix-command` experimental feature enabled, you can v3 CLI instead:
|
||||
|
||||
```console
|
||||
nix build -f . packages.foundation.bash.latest.packages.x86_64-linux.x86_64-linux.x86_64-linux.package
|
||||
```
|
||||
|
||||
Using flakes, the difference is only in calling:
|
||||
|
||||
```console
|
||||
nix build .#packages.foundation.bash.latest.packages.x86_64-linux.x86_64-linux.x86_64-linux.package
|
||||
```
|
||||
|
||||
This interface is verbose, but stable. We are still working out the details for
|
||||
a friendlier interface!
|
||||
|
||||
The full command syntax looks like this:
|
||||
|
||||
```console
|
||||
nix-build -A packages.${package-set}.${package}.${version}.packages.${build}.${host}.${target}.package
|
||||
```
|
||||
|
||||
Tidepool packages can have multiple versions available. Tidepool also provides aliases
|
||||
for convenience, currently "stable" and "latest". Aliases go directly after `${package}`:
|
||||
|
||||
```console
|
||||
nix-build -A packages.${package-set}.${package}.${alias}.packages.${build}.${host}.${target}.package
|
||||
```
|
||||
|
||||
The actual available versions live one level deeper, under `versions` namespace:
|
||||
|
||||
```console
|
||||
nix-build -A packages.${package-set}.${package}.versions.${version}.packages.${build}.${host}.${target}.package
|
||||
```
|
||||
|
||||
Here are a few practical examples for Tidepool. We can use a specific Bash version:
|
||||
|
||||
```console
|
||||
nix-build -A packages.foundation.bash.versions."5.2.15-stage1".packages.x86_64-linux.x86_64-linux.x86_64-linux.package
|
||||
```
|
||||
|
||||
Get Bash cross-compiled from `i686-linux` to `x86_64-linux`:
|
||||
|
||||
```console
|
||||
nix build .#packages.foundation.bash.latest.packages.i686-linux.x86_64-linux.x86_64-linux.package
|
||||
```
|
||||
|
||||
Or a native `aarch64-linux` Bash:
|
||||
|
||||
```console
|
||||
nix build .#packages.foundation.bash.latest.packages.aarch64-linux.aarch64-linux.aarch64-linux.package
|
||||
```
|
||||
|
||||
Running this command will:
|
||||
|
||||
- Bootstrap `i686-linux` `gcc` as a dependency of `x86_64-linux` `gcc`
|
||||
- Incrementally bootstrap the actual `x86_64-linux` `gcc`
|
||||
- Incrementally bootstrap `aarch64-linux` `gcc` from `x86_64-linux` `gcc`
|
||||
- Build the native `aarch64-linux` `gcc`
|
||||
- Build the native `aarch64-linux` `bash`
|
||||
|
||||
It's very early in Tidepool's development, and this syntax and UX is expected to change! For now,
|
||||
the only syntax available is this developer-facing style. You can expect improvements to be made!
|
||||
|
||||
## Development
|
||||
|
||||
Tidepool is built with Aux's portable submodules. They function similarly to submodules used by NixOS.
|
||||
Using the submodules, Tidepool provides a nice and easy-to-work-with API.
|
||||
|
||||
Tidepool can be used with flakes or without them. The solution we recommend
|
||||
for non-flake usage is [npins](https://github.com/andir/npins).
|
||||
Tidepool exports an attribute set, which can be viewed using `nix repl`.
|
||||
We can visualize it!
|
||||
|
||||
```console
|
||||
tidepool
|
||||
├───builders
|
||||
├───lib
|
||||
├───mirrors
|
||||
├───packages
|
||||
│ ├───aux
|
||||
│ │ ├───a
|
||||
│ │ ├───b
|
||||
│ │ └───c
|
||||
│ ├───foundation
|
||||
│ │ ├───bash
|
||||
│ │ ├─── ...
|
||||
│ │ └───zlib
|
||||
│ ├───context
|
||||
│ │ └─── ...
|
||||
│ └───...
|
||||
├───extend
|
||||
├───internal
|
||||
├───new
|
||||
└───preferences
|
||||
```
|
||||
|
||||
There are 4 outputs that share the name with folders under `src/`:
|
||||
|
||||
- `builders` - this is a set of Tidepool builders
|
||||
- `lib` - this is the Tidepool library. It defines some nice-to-haves
|
||||
for packaging
|
||||
- `mirrors` - this is set of mirrors. Many projects, like GNU, provide
|
||||
many mirrors - you can change a mirror to the one your prefer
|
||||
- `packages` - this is a set for all of the Tidepool packages
|
||||
|
||||
You may learn more about those attributes in their corresponding folders.
|
||||
|
||||
Additionally, there are several other attributes, which don't have
|
||||
corresponding folders:
|
||||
|
||||
- `extend` - this is a function to extend Tidepool's package set by providing
|
||||
additional package modules. Here's how to use it:
|
||||
|
||||
```nix
|
||||
let
|
||||
tidepool = import ./tidepool;
|
||||
in
|
||||
tidepool.extend {
|
||||
modules = [
|
||||
# Put additional modules you want to include here.
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
- `internal` - this is the internal attribute set. Currently, it only brings Tidepool's
|
||||
dependencies into scope: more concretely, it exposes the `foundation` namespace from
|
||||
the (perhaps confusingly-named) [Foundation](https://git.auxolotl.org/auxolotl/foundation)
|
||||
repository, which does the bootstrap chain from a 357-byte file. It is not the same
|
||||
as Tidepool's `foundation` namespace, which makes those early-bootstrap packages integrate
|
||||
with Tidepool (e.g., enabling cross-compilation)
|
||||
- `new` - this is a function that allows to create a new empty package set.
|
||||
It allows you to create a package set from scratch. Using this function,
|
||||
you will get access to Tidepool library functions and the interface for defining
|
||||
your own builders (but you don't get builder implementations). Here's how to use it:
|
||||
|
||||
```nix
|
||||
let
|
||||
tidepool = import ./tidepool;
|
||||
|
||||
module = { config }: {
|
||||
config = {
|
||||
# Create a new package set from scratch, no packages exist unless you define them.
|
||||
};
|
||||
};
|
||||
in
|
||||
tidepool.new module;
|
||||
```
|
||||
|
||||
- `preferences` - this is an attribute set that lists your preferences for the package sets.
|
||||
It has the following shape:
|
||||
|
||||
```nix
|
||||
{
|
||||
packages = {
|
||||
allow = {
|
||||
broken = false;
|
||||
incompatible = false;
|
||||
};
|
||||
version = "latest";
|
||||
};
|
||||
}
|
||||
```
|
||||
|
|
|
|||
96
tidepool/src/packages/README.md
Normal file
96
tidepool/src/packages/README.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
# Packages
|
||||
|
||||
In this folder, the Tidepool package sets are stored.
|
||||
|
||||
## Foundation
|
||||
|
||||
The package set under `foundation` is a minimal set to start building everything
|
||||
else. It builds upon and uses packages from [Foundation](https://git.auxolotl.org/auxolotl/foundation),
|
||||
our early bootstrap chain. It provides a comprehensive set of packages that are
|
||||
used to build all other basic Linux utilities, and are often transitive dependencies
|
||||
of higher-level packages.
|
||||
|
||||
The goal of `foundation` is to build `gcc` and `glibc`,
|
||||
as well as some supplementary packages, so that other packages can be built with
|
||||
them. It also sets up a basic Linux environment: `coreutils`, `diffutils`, `bash`,
|
||||
and other utilities you'd expect to find on a Linux distro.
|
||||
|
||||
To achieve this, we need a lot of other packages built, too! Some
|
||||
examples include `python`, `xz` and `gzip`.
|
||||
|
||||
Lastly, `foundation` also includes `patchelf`, a simple C utility.
|
||||
It is not used in `foundation`, but it is very useful for building
|
||||
more high-level packages: in particular, it allows us to patch
|
||||
binaries to point to our dependencies. This "unbreaks" proprietary
|
||||
binaries by making them point to Tidepool-provided dependencies.
|
||||
Neat!
|
||||
|
||||
The bootstrap chain is based on [full-source](https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/)
|
||||
bootstrap, made possible with amazing work by Guix's people,
|
||||
and Emily Trau's effort on porting it to Nix!
|
||||
It bootstraps a proper `gcc` for `i686-linux`, which can then
|
||||
be used to cross-compile a `gcc` for other architectures, such
|
||||
as `x86_64-linux`.
|
||||
|
||||
To build stage1 (`i686-linux`), run:
|
||||
|
||||
```console
|
||||
nix build --system i686-linux .#packages.foundation.glibc.versions."2.38-stage1-passthrough".packages.i686-linux.x86_64-linux.x86_64-linux.package
|
||||
```
|
||||
|
||||
The full bootstrap chain is available in a confusingly-named
|
||||
[Foundation repository](https://git.auxolotl.org/auxolotl/foundation).
|
||||
|
||||
## Aux
|
||||
|
||||
`aux`, at this time, is a trivial package set that demonstrates
|
||||
the power of a fully-bootstrapped `foundation`. It consists of 3
|
||||
trivial packages: `a`, `b` and `c`. The chain of dependencies
|
||||
is like this (the arrow points to the **dependency**):
|
||||
|
||||
```
|
||||
a -> b -> c
|
||||
```
|
||||
|
||||
Package `a` merely depends on `b`.
|
||||
|
||||
Package `b` announces a build option `custom`. It sets up `context`
|
||||
and sets up a hook after `unpack` phase. It also propagates dependency `c`.
|
||||
It's worth discussing this in more depth, because it demonstrates a couple of
|
||||
more advanced features.
|
||||
|
||||
`context` is information that the package provides when it is used as dependency,
|
||||
which builders and hooks may use. For example, to use a C dependency, you
|
||||
would need to tell the compiler where to look for relevant headers by appending
|
||||
a flag. The C dependency can have `context` defined which includes the necessary
|
||||
compiler flag, and the package that uses this dependency will append this flag
|
||||
automatically.
|
||||
|
||||
[Nixpkgs](https://github.com/nixos/nixpkgs) uses a different approach to do this.
|
||||
In Nixpkgs, packages include the `nix-support` subdirectory which contains the
|
||||
same information as what `context` usually contains. Nixpkgs does this in
|
||||
the builder code at build-time; Tidepool provides this information at eval time,
|
||||
which provides the means to type-check the `context`.
|
||||
|
||||
Package `b` provides a `context` to include the necessary directory: `include` under
|
||||
the Nix store path for package `b`. This Nix store path is provided as an environment
|
||||
variable from a hook.
|
||||
|
||||
While Tidepool's architecture is subject to change, `context` already
|
||||
has planned changes for it. When those changes happen - the information
|
||||
about `context` will be moved to a historical design doc. But until then,
|
||||
`context` is still relevant!
|
||||
|
||||
`hooks` provide extension points for packages. With `hooks`, you can inject arbitrary build
|
||||
instructions between, before or after the default build stages. Package `b` does exactly
|
||||
that: it uses a hook to add entry after `unpack` phase and before others, which exports
|
||||
an environment variable that contains the Nix store path, that is then used for `context`.
|
||||
|
||||
Propagation ensures that the propagated dependency is also installed, when the package
|
||||
is installed. Package `b` propagates dependency `c` - this means, when package `b` is
|
||||
included as a dependency, package `c` is also included automatically. This is especially
|
||||
useful when working with interpreted languages, such as Python - for them, all dependencies
|
||||
need to be in a single scope, and propagation makes sure of that.
|
||||
|
||||
Package `c` doesn't have dependencies, but it is otherwise identical
|
||||
to package `b`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue