c8h4/aux-search
1
0
Fork 0
Code Issues Pull requests Packages Releases Activity
aux-search/.github/workflows/cron-flakes.yml

70 lines
2 KiB
YAML
Raw Normal View History

Flake support/frontend (#324) * Setup flake info extraction Prepare data model fro derivations (#1) Add flake info data (#1) Implement fetching general flake info (#1) Expose CLI (#1) Keep cargo happy Add some doc comments Pin to local nixpkgs to excessive downloads Extend visibility of some data objects Add command to extract infomation about defivations (#1) Add call new feature in main (#1) Include more information in derivation (#1) Add log access Always debug log stderr from nix Format nix script Collect systems per package Remove unnecessary imports Create flake Remove top level version field Represent collected systems/version pairs in rust Fix quotation marks in tests Add correct cargo hash Add iconv dependency Return a list from nix script Export as json Undo version by platform distinction Remove nixpkgs override Apply cargo fmt Flatten export structure Allow for complex licenses Prepare using a central nix file Implement nix part o accessing apps Include the correct filename Add accessor for `all` key Access all available information by default Track more information about Apps Run cargo fmt Fix: allow local builds Prepare next version of the flake info tool Include examples and pull script Expose flake info as library Include thiserror for custom errors Define a source data type Collects source types and their metadata, collected in a json file Add command line argument for input files Mutually exclusive with --flake Refactor functions to extract information given a flake identifier Add kind specifier as CLI argument Amend Argument parsing to require eiteher flake or targets to be defined Run extraction for specified flake or list of flakes as specified in a json file Resolves #5 References #7 Use internal tag to distnguich target types Include target falg usage in examples Set include provided source if available (resolves #9) Resolve flake name Update examples Dont include empty license or description Fix a misfomatting in cargot.toml Add elastic dependencies Implement a wrapper around the elasticsearch client Implements pushing exports (#4) Temporarily skip serializing an unimplemented field in elastic output Extract reading source list files from binary Add lazy_static as dependency Implement createing and pushing to elastic index Add elastic options Provide default name and env falbac for elastic index Modify app binary and type as optionals App can be a derivation too Update examples Add more elastic commands Supported: - ensure - clear - push Rename elastic search config struct Add elastic push support to binary Rename flag to enable elastic push Imporve error messages and format binary source Fix nix file incorrectly expecting meta fields Changing flake descriotions to an optional field deserialize git_ref as hash Implement temporary stores and gc of these prevents flakes from accessing store paths Pass extra arguments to nix Update cargo hash and skip integration tests Move flake.nix to root folder and add apps for all components Fix command invocation that fails test Update README(s) Add help for extra arguments (cherry picked from commit be4bc3dd929178bef66114c2201aaa88e47e9add) * Safely read legacyPackages * Read nixosOptions from flake * Update ES Mapping * Show more detailed error and backtrace if available * Try reading options only if key is defined * Format nix script * Add error context when attempting deserialization * Fix derivation representation to fit nix output * Add push elasticsearch settings * Add Flake channel * Rename import module * Remove Flakes Channel * Prepare nixpkgs import * Separate import/export types * Break up nixpkgs package representation * Use the same naming scheme for Nixpkgs entries and flake entries * Document import module * Remove serialization attributes * Reversable type and SerDe implemetation * Add *_reverse fields * Unpublicating export fields * Read from NixOption struct * serialize empty fields as null * Tag export json variants * Serialize a single option-declaration * Format npkgs parse test * Define NixOption Sorry thats too late.. * Parse system key * Make Package output compatible with the frontend * Add Url-only licesnse variant * Add StringOrStruct type * Add accessor method for elements catched by OneOrMany * New utility to flatten in homogenous lists recursively * Add Maintainer type catching maintaiers used in nixpkgs * Format Implementatio n * Remove explicit representation of platforms * Open nixpkgs parser to cover all packages * Convert all imported fields to their export representation * Define reverse fields in ES schema * Format nixpkgs command runner * Expose shorthand to pull a specific channel by command line * Extract utility functions into their own module * Implement AttributeQuery generation * Integrate query in export * Implement *_set attributes * Document purpose of export module * Use more descriptive github blob route * Complete Option Export representation Reuse the same option type for nixpkgs and flakes * Enable nixpkgs option import * Expose nixpkgs option import functions and integrate in binary * Chunk ElasticSearch Bulk operations * Address Example/Default field formatting * Add abort strategy for existing indices * New command line interface using subcommands * Document new interface * Bump version * Add nixpkgs cron job * More precise name for nixpkgs cron job * Add Flakes cron job * Read version from file and fix channel names * Correct file names for flake group import * Fix group command * Run new cron jobs on PR * Update Cargo sources * Integrate new flakes route * Add search type option * Add flake types and messages * Add flake types * Extract Request body builders and expose more types * Combine Package and Option Search * Factor out html body * Dispatch messages by search type to flake page * Correct type naming * Remove Debug instructions and unimplemented flake search type * Do not reload Flake search page while the search subject is unchanged * Implement switching subjects * Fix init type signature * Add url/git based flakes to mapping * Parse flake info * Link to flake repo and show flake maintainer * Fix optional decoded values * Add group and type as search buckts * Show search selection in all cases * Move flake decoding to search * Show flake information for options * Hardcode flakesearch to only search a specified flake index * Improve experimental state notice * Fix category select not present in some cases * Change default flake index * Show correct category title * Add missing imports * Serialize more optional fields as null * elastic-test.rs file not needed anymore * names of the workflows should be immediately obvious. * better jobs names * rename the flakes group * need to provide --elastic-schema-version via the cli option * typo * Fix errors in workflow files * Install flake enabled nix by default * Fix variable substitution * Use string group names * Provide elastic schema verion only through cli * Fix scheduled task name * Improve error reporting * Allow insecure packages here * Add missing imports * Tree-wide: cargo-fmt * only import nixpkgs for now * no importing of flakes * also bump a version * fixing cron-flakes.yml * make it obvious that this is for frontend * missed this when merging * Extract hashes - Split out revision info from flakes - Retrieve current nixpkgs revision from github * Write aliases, enabled by default * Filter additional platforms * Change alias structure * expect channel like nixpkgs identifiers * Don't cause error if push is aborted * Specify correct channel identifier * Allow options to evaluate unfree packages * Retrieve and delete specific aliases * Specify import path correclty * Fix channel warmup * Abort push if channel already indexed * Remove debug pr hook for import action * Fix indentation by tab * Make flakes show again * Fix import group naming in flake wrokflow * Rename flake group to match imported index * Run nixpkgs import on pr activity (Debugging behavior) * Just show literal Examples, resolves #336 * Use actual nixpkgs branch names * Trim derivation/option declaration path, resolves #335 Remove /nix/store/*-source prefix * Fix sidebar width and close button position * Placeholder texts in flake result area * Show flake install info * Don't show package/option selection before search * Make sure install command for nixos is always shown * Add toml source config support * Rewrite current example flake group as toml * Update flake cron job * flake-info: use saner nix packaging method When trying to first work in this I naive approached it with `nix-shell`. That of course lead to the fixed output bollocks failing with a hash mismatch. By making use of the `cargoLock` attribute on `buildRustPackage` we can tame the FOD-beast and only have to provide one hash manually for a single package (that we fetch from a GitHub repository). This also means that updating dependencies will be simpler as the native Cargo.lock file can be used. (cherry picked from commit c3a0e46d1eb56e128e6923e6c493eb836fc81e85) * Update flake lock file * Do not build python import script * add flake names to the title as well * Disable debug imports on prs Co-authored-by: Rok Garbas <rok@garbas.si> Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
2021-08-25 22:40:42 +00:00
name: "Flakes: Hourly import to Elasticsearch"
on:
Translate example/default values from JSON to NIx (#357)
2021-11-11 19:29:44 +00:00
workflow_dispatch:
Flake support/frontend (#324) * Setup flake info extraction Prepare data model fro derivations (#1) Add flake info data (#1) Implement fetching general flake info (#1) Expose CLI (#1) Keep cargo happy Add some doc comments Pin to local nixpkgs to excessive downloads Extend visibility of some data objects Add command to extract infomation about defivations (#1) Add call new feature in main (#1) Include more information in derivation (#1) Add log access Always debug log stderr from nix Format nix script Collect systems per package Remove unnecessary imports Create flake Remove top level version field Represent collected systems/version pairs in rust Fix quotation marks in tests Add correct cargo hash Add iconv dependency Return a list from nix script Export as json Undo version by platform distinction Remove nixpkgs override Apply cargo fmt Flatten export structure Allow for complex licenses Prepare using a central nix file Implement nix part o accessing apps Include the correct filename Add accessor for `all` key Access all available information by default Track more information about Apps Run cargo fmt Fix: allow local builds Prepare next version of the flake info tool Include examples and pull script Expose flake info as library Include thiserror for custom errors Define a source data type Collects source types and their metadata, collected in a json file Add command line argument for input files Mutually exclusive with --flake Refactor functions to extract information given a flake identifier Add kind specifier as CLI argument Amend Argument parsing to require eiteher flake or targets to be defined Run extraction for specified flake or list of flakes as specified in a json file Resolves #5 References #7 Use internal tag to distnguich target types Include target falg usage in examples Set include provided source if available (resolves #9) Resolve flake name Update examples Dont include empty license or description Fix a misfomatting in cargot.toml Add elastic dependencies Implement a wrapper around the elasticsearch client Implements pushing exports (#4) Temporarily skip serializing an unimplemented field in elastic output Extract reading source list files from binary Add lazy_static as dependency Implement createing and pushing to elastic index Add elastic options Provide default name and env falbac for elastic index Modify app binary and type as optionals App can be a derivation too Update examples Add more elastic commands Supported: - ensure - clear - push Rename elastic search config struct Add elastic push support to binary Rename flag to enable elastic push Imporve error messages and format binary source Fix nix file incorrectly expecting meta fields Changing flake descriotions to an optional field deserialize git_ref as hash Implement temporary stores and gc of these prevents flakes from accessing store paths Pass extra arguments to nix Update cargo hash and skip integration tests Move flake.nix to root folder and add apps for all components Fix command invocation that fails test Update README(s) Add help for extra arguments (cherry picked from commit be4bc3dd929178bef66114c2201aaa88e47e9add) * Safely read legacyPackages * Read nixosOptions from flake * Update ES Mapping * Show more detailed error and backtrace if available * Try reading options only if key is defined * Format nix script * Add error context when attempting deserialization * Fix derivation representation to fit nix output * Add push elasticsearch settings * Add Flake channel * Rename import module * Remove Flakes Channel * Prepare nixpkgs import * Separate import/export types * Break up nixpkgs package representation * Use the same naming scheme for Nixpkgs entries and flake entries * Document import module * Remove serialization attributes * Reversable type and SerDe implemetation * Add *_reverse fields * Unpublicating export fields * Read from NixOption struct * serialize empty fields as null * Tag export json variants * Serialize a single option-declaration * Format npkgs parse test * Define NixOption Sorry thats too late.. * Parse system key * Make Package output compatible with the frontend * Add Url-only licesnse variant * Add StringOrStruct type * Add accessor method for elements catched by OneOrMany * New utility to flatten in homogenous lists recursively * Add Maintainer type catching maintaiers used in nixpkgs * Format Implementatio n * Remove explicit representation of platforms * Open nixpkgs parser to cover all packages * Convert all imported fields to their export representation * Define reverse fields in ES schema * Format nixpkgs command runner * Expose shorthand to pull a specific channel by command line * Extract utility functions into their own module * Implement AttributeQuery generation * Integrate query in export * Implement *_set attributes * Document purpose of export module * Use more descriptive github blob route * Complete Option Export representation Reuse the same option type for nixpkgs and flakes * Enable nixpkgs option import * Expose nixpkgs option import functions and integrate in binary * Chunk ElasticSearch Bulk operations * Address Example/Default field formatting * Add abort strategy for existing indices * New command line interface using subcommands * Document new interface * Bump version * Add nixpkgs cron job * More precise name for nixpkgs cron job * Add Flakes cron job * Read version from file and fix channel names * Correct file names for flake group import * Fix group command * Run new cron jobs on PR * Update Cargo sources * Integrate new flakes route * Add search type option * Add flake types and messages * Add flake types * Extract Request body builders and expose more types * Combine Package and Option Search * Factor out html body * Dispatch messages by search type to flake page * Correct type naming * Remove Debug instructions and unimplemented flake search type * Do not reload Flake search page while the search subject is unchanged * Implement switching subjects * Fix init type signature * Add url/git based flakes to mapping * Parse flake info * Link to flake repo and show flake maintainer * Fix optional decoded values * Add group and type as search buckts * Show search selection in all cases * Move flake decoding to search * Show flake information for options * Hardcode flakesearch to only search a specified flake index * Improve experimental state notice * Fix category select not present in some cases * Change default flake index * Show correct category title * Add missing imports * Serialize more optional fields as null * elastic-test.rs file not needed anymore * names of the workflows should be immediately obvious. * better jobs names * rename the flakes group * need to provide --elastic-schema-version via the cli option * typo * Fix errors in workflow files * Install flake enabled nix by default * Fix variable substitution * Use string group names * Provide elastic schema verion only through cli * Fix scheduled task name * Improve error reporting * Allow insecure packages here * Add missing imports * Tree-wide: cargo-fmt * only import nixpkgs for now * no importing of flakes * also bump a version * fixing cron-flakes.yml * make it obvious that this is for frontend * missed this when merging * Extract hashes - Split out revision info from flakes - Retrieve current nixpkgs revision from github * Write aliases, enabled by default * Filter additional platforms * Change alias structure * expect channel like nixpkgs identifiers * Don't cause error if push is aborted * Specify correct channel identifier * Allow options to evaluate unfree packages * Retrieve and delete specific aliases * Specify import path correclty * Fix channel warmup * Abort push if channel already indexed * Remove debug pr hook for import action * Fix indentation by tab * Make flakes show again * Fix import group naming in flake wrokflow * Rename flake group to match imported index * Run nixpkgs import on pr activity (Debugging behavior) * Just show literal Examples, resolves #336 * Use actual nixpkgs branch names * Trim derivation/option declaration path, resolves #335 Remove /nix/store/*-source prefix * Fix sidebar width and close button position * Placeholder texts in flake result area * Show flake install info * Don't show package/option selection before search * Make sure install command for nixos is always shown * Add toml source config support * Rewrite current example flake group as toml * Update flake cron job * flake-info: use saner nix packaging method When trying to first work in this I naive approached it with `nix-shell`. That of course lead to the fixed output bollocks failing with a hash mismatch. By making use of the `cargoLock` attribute on `buildRustPackage` we can tame the FOD-beast and only have to provide one hash manually for a single package (that we fetch from a GitHub repository). This also means that updating dependencies will be simpler as the native Cargo.lock file can be used. (cherry picked from commit c3a0e46d1eb56e128e6923e6c493eb836fc81e85) * Update flake lock file * Do not build python import script * add flake names to the title as well * Disable debug imports on prs Co-authored-by: Rok Garbas <rok@garbas.si> Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
2021-08-25 22:40:42 +00:00
schedule:
- cron: '0 * * * *'
jobs:
hourly-import-channel:
runs-on: ubuntu-latest
strategy:
fail-fast: true
matrix:
group:
- "manual"
env:
RUST_LOG: debug
FI_ES_EXISTS_STRATEGY: recreate
FI_ES_URL: ${{ secrets.ELASTICSEARCH_URL }}
steps:
- name: Checking out the repository
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Installing Nix
Dependabot(deps): Bump cachix/install-nix-action from 14 to 16 (#374) Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 14 to 16. - [Release notes](https://github.com/cachix/install-nix-action/releases) - [Commits](https://github.com/cachix/install-nix-action/compare/v14...v16) --- updated-dependencies: - dependency-name: cachix/install-nix-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-11-22 12:27:31 +00:00
uses: cachix/install-nix-action@v16
Flake support/frontend (#324) * Setup flake info extraction Prepare data model fro derivations (#1) Add flake info data (#1) Implement fetching general flake info (#1) Expose CLI (#1) Keep cargo happy Add some doc comments Pin to local nixpkgs to excessive downloads Extend visibility of some data objects Add command to extract infomation about defivations (#1) Add call new feature in main (#1) Include more information in derivation (#1) Add log access Always debug log stderr from nix Format nix script Collect systems per package Remove unnecessary imports Create flake Remove top level version field Represent collected systems/version pairs in rust Fix quotation marks in tests Add correct cargo hash Add iconv dependency Return a list from nix script Export as json Undo version by platform distinction Remove nixpkgs override Apply cargo fmt Flatten export structure Allow for complex licenses Prepare using a central nix file Implement nix part o accessing apps Include the correct filename Add accessor for `all` key Access all available information by default Track more information about Apps Run cargo fmt Fix: allow local builds Prepare next version of the flake info tool Include examples and pull script Expose flake info as library Include thiserror for custom errors Define a source data type Collects source types and their metadata, collected in a json file Add command line argument for input files Mutually exclusive with --flake Refactor functions to extract information given a flake identifier Add kind specifier as CLI argument Amend Argument parsing to require eiteher flake or targets to be defined Run extraction for specified flake or list of flakes as specified in a json file Resolves #5 References #7 Use internal tag to distnguich target types Include target falg usage in examples Set include provided source if available (resolves #9) Resolve flake name Update examples Dont include empty license or description Fix a misfomatting in cargot.toml Add elastic dependencies Implement a wrapper around the elasticsearch client Implements pushing exports (#4) Temporarily skip serializing an unimplemented field in elastic output Extract reading source list files from binary Add lazy_static as dependency Implement createing and pushing to elastic index Add elastic options Provide default name and env falbac for elastic index Modify app binary and type as optionals App can be a derivation too Update examples Add more elastic commands Supported: - ensure - clear - push Rename elastic search config struct Add elastic push support to binary Rename flag to enable elastic push Imporve error messages and format binary source Fix nix file incorrectly expecting meta fields Changing flake descriotions to an optional field deserialize git_ref as hash Implement temporary stores and gc of these prevents flakes from accessing store paths Pass extra arguments to nix Update cargo hash and skip integration tests Move flake.nix to root folder and add apps for all components Fix command invocation that fails test Update README(s) Add help for extra arguments (cherry picked from commit be4bc3dd929178bef66114c2201aaa88e47e9add) * Safely read legacyPackages * Read nixosOptions from flake * Update ES Mapping * Show more detailed error and backtrace if available * Try reading options only if key is defined * Format nix script * Add error context when attempting deserialization * Fix derivation representation to fit nix output * Add push elasticsearch settings * Add Flake channel * Rename import module * Remove Flakes Channel * Prepare nixpkgs import * Separate import/export types * Break up nixpkgs package representation * Use the same naming scheme for Nixpkgs entries and flake entries * Document import module * Remove serialization attributes * Reversable type and SerDe implemetation * Add *_reverse fields * Unpublicating export fields * Read from NixOption struct * serialize empty fields as null * Tag export json variants * Serialize a single option-declaration * Format npkgs parse test * Define NixOption Sorry thats too late.. * Parse system key * Make Package output compatible with the frontend * Add Url-only licesnse variant * Add StringOrStruct type * Add accessor method for elements catched by OneOrMany * New utility to flatten in homogenous lists recursively * Add Maintainer type catching maintaiers used in nixpkgs * Format Implementatio n * Remove explicit representation of platforms * Open nixpkgs parser to cover all packages * Convert all imported fields to their export representation * Define reverse fields in ES schema * Format nixpkgs command runner * Expose shorthand to pull a specific channel by command line * Extract utility functions into their own module * Implement AttributeQuery generation * Integrate query in export * Implement *_set attributes * Document purpose of export module * Use more descriptive github blob route * Complete Option Export representation Reuse the same option type for nixpkgs and flakes * Enable nixpkgs option import * Expose nixpkgs option import functions and integrate in binary * Chunk ElasticSearch Bulk operations * Address Example/Default field formatting * Add abort strategy for existing indices * New command line interface using subcommands * Document new interface * Bump version * Add nixpkgs cron job * More precise name for nixpkgs cron job * Add Flakes cron job * Read version from file and fix channel names * Correct file names for flake group import * Fix group command * Run new cron jobs on PR * Update Cargo sources * Integrate new flakes route * Add search type option * Add flake types and messages * Add flake types * Extract Request body builders and expose more types * Combine Package and Option Search * Factor out html body * Dispatch messages by search type to flake page * Correct type naming * Remove Debug instructions and unimplemented flake search type * Do not reload Flake search page while the search subject is unchanged * Implement switching subjects * Fix init type signature * Add url/git based flakes to mapping * Parse flake info * Link to flake repo and show flake maintainer * Fix optional decoded values * Add group and type as search buckts * Show search selection in all cases * Move flake decoding to search * Show flake information for options * Hardcode flakesearch to only search a specified flake index * Improve experimental state notice * Fix category select not present in some cases * Change default flake index * Show correct category title * Add missing imports * Serialize more optional fields as null * elastic-test.rs file not needed anymore * names of the workflows should be immediately obvious. * better jobs names * rename the flakes group * need to provide --elastic-schema-version via the cli option * typo * Fix errors in workflow files * Install flake enabled nix by default * Fix variable substitution * Use string group names * Provide elastic schema verion only through cli * Fix scheduled task name * Improve error reporting * Allow insecure packages here * Add missing imports * Tree-wide: cargo-fmt * only import nixpkgs for now * no importing of flakes * also bump a version * fixing cron-flakes.yml * make it obvious that this is for frontend * missed this when merging * Extract hashes - Split out revision info from flakes - Retrieve current nixpkgs revision from github * Write aliases, enabled by default * Filter additional platforms * Change alias structure * expect channel like nixpkgs identifiers * Don't cause error if push is aborted * Specify correct channel identifier * Allow options to evaluate unfree packages * Retrieve and delete specific aliases * Specify import path correclty * Fix channel warmup * Abort push if channel already indexed * Remove debug pr hook for import action * Fix indentation by tab * Make flakes show again * Fix import group naming in flake wrokflow * Rename flake group to match imported index * Run nixpkgs import on pr activity (Debugging behavior) * Just show literal Examples, resolves #336 * Use actual nixpkgs branch names * Trim derivation/option declaration path, resolves #335 Remove /nix/store/*-source prefix * Fix sidebar width and close button position * Placeholder texts in flake result area * Show flake install info * Don't show package/option selection before search * Make sure install command for nixos is always shown * Add toml source config support * Rewrite current example flake group as toml * Update flake cron job * flake-info: use saner nix packaging method When trying to first work in this I naive approached it with `nix-shell`. That of course lead to the fixed output bollocks failing with a hash mismatch. By making use of the `cargoLock` attribute on `buildRustPackage` we can tame the FOD-beast and only have to provide one hash manually for a single package (that we fetch from a GitHub repository). This also means that updating dependencies will be simpler as the native Cargo.lock file can be used. (cherry picked from commit c3a0e46d1eb56e128e6923e6c493eb836fc81e85) * Update flake lock file * Do not build python import script * add flake names to the title as well * Disable debug imports on prs Co-authored-by: Rok Garbas <rok@garbas.si> Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
2021-08-25 22:40:42 +00:00
with:
install_url: https://nixos-nix-install-tests.cachix.org/serve/i6laym9jw3wg9mw6ncyrk6gjx4l34vvx/install
install_options: '--tarball-url-prefix https://nixos-nix-install-tests.cachix.org/serve'
extra_nix_config: |
experimental-features = nix-command flakes
access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}
- uses: cachix/cachix-action@v10
with:
name: nixos-search
signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
- name: Install unstable channel
run: |
nix-channel --add https://nixos.org/channels/nixpkgs-unstable
nix-channel --update
- name: Installing jq
run: |
nix-env -iA nixpkgs.nixFlakes nixpkgs.jq
- name: Building import_scripts
run: |
flake, overlay: Rename package to `flake-info` (#383) The package now has the same name as the `app` and the command (`bin/flake-info`).
2021-12-06 14:01:26 +00:00
nix build ./#packages.x86_64-linux.flake-info
Flake support/frontend (#324) * Setup flake info extraction Prepare data model fro derivations (#1) Add flake info data (#1) Implement fetching general flake info (#1) Expose CLI (#1) Keep cargo happy Add some doc comments Pin to local nixpkgs to excessive downloads Extend visibility of some data objects Add command to extract infomation about defivations (#1) Add call new feature in main (#1) Include more information in derivation (#1) Add log access Always debug log stderr from nix Format nix script Collect systems per package Remove unnecessary imports Create flake Remove top level version field Represent collected systems/version pairs in rust Fix quotation marks in tests Add correct cargo hash Add iconv dependency Return a list from nix script Export as json Undo version by platform distinction Remove nixpkgs override Apply cargo fmt Flatten export structure Allow for complex licenses Prepare using a central nix file Implement nix part o accessing apps Include the correct filename Add accessor for `all` key Access all available information by default Track more information about Apps Run cargo fmt Fix: allow local builds Prepare next version of the flake info tool Include examples and pull script Expose flake info as library Include thiserror for custom errors Define a source data type Collects source types and their metadata, collected in a json file Add command line argument for input files Mutually exclusive with --flake Refactor functions to extract information given a flake identifier Add kind specifier as CLI argument Amend Argument parsing to require eiteher flake or targets to be defined Run extraction for specified flake or list of flakes as specified in a json file Resolves #5 References #7 Use internal tag to distnguich target types Include target falg usage in examples Set include provided source if available (resolves #9) Resolve flake name Update examples Dont include empty license or description Fix a misfomatting in cargot.toml Add elastic dependencies Implement a wrapper around the elasticsearch client Implements pushing exports (#4) Temporarily skip serializing an unimplemented field in elastic output Extract reading source list files from binary Add lazy_static as dependency Implement createing and pushing to elastic index Add elastic options Provide default name and env falbac for elastic index Modify app binary and type as optionals App can be a derivation too Update examples Add more elastic commands Supported: - ensure - clear - push Rename elastic search config struct Add elastic push support to binary Rename flag to enable elastic push Imporve error messages and format binary source Fix nix file incorrectly expecting meta fields Changing flake descriotions to an optional field deserialize git_ref as hash Implement temporary stores and gc of these prevents flakes from accessing store paths Pass extra arguments to nix Update cargo hash and skip integration tests Move flake.nix to root folder and add apps for all components Fix command invocation that fails test Update README(s) Add help for extra arguments (cherry picked from commit be4bc3dd929178bef66114c2201aaa88e47e9add) * Safely read legacyPackages * Read nixosOptions from flake * Update ES Mapping * Show more detailed error and backtrace if available * Try reading options only if key is defined * Format nix script * Add error context when attempting deserialization * Fix derivation representation to fit nix output * Add push elasticsearch settings * Add Flake channel * Rename import module * Remove Flakes Channel * Prepare nixpkgs import * Separate import/export types * Break up nixpkgs package representation * Use the same naming scheme for Nixpkgs entries and flake entries * Document import module * Remove serialization attributes * Reversable type and SerDe implemetation * Add *_reverse fields * Unpublicating export fields * Read from NixOption struct * serialize empty fields as null * Tag export json variants * Serialize a single option-declaration * Format npkgs parse test * Define NixOption Sorry thats too late.. * Parse system key * Make Package output compatible with the frontend * Add Url-only licesnse variant * Add StringOrStruct type * Add accessor method for elements catched by OneOrMany * New utility to flatten in homogenous lists recursively * Add Maintainer type catching maintaiers used in nixpkgs * Format Implementatio n * Remove explicit representation of platforms * Open nixpkgs parser to cover all packages * Convert all imported fields to their export representation * Define reverse fields in ES schema * Format nixpkgs command runner * Expose shorthand to pull a specific channel by command line * Extract utility functions into their own module * Implement AttributeQuery generation * Integrate query in export * Implement *_set attributes * Document purpose of export module * Use more descriptive github blob route * Complete Option Export representation Reuse the same option type for nixpkgs and flakes * Enable nixpkgs option import * Expose nixpkgs option import functions and integrate in binary * Chunk ElasticSearch Bulk operations * Address Example/Default field formatting * Add abort strategy for existing indices * New command line interface using subcommands * Document new interface * Bump version * Add nixpkgs cron job * More precise name for nixpkgs cron job * Add Flakes cron job * Read version from file and fix channel names * Correct file names for flake group import * Fix group command * Run new cron jobs on PR * Update Cargo sources * Integrate new flakes route * Add search type option * Add flake types and messages * Add flake types * Extract Request body builders and expose more types * Combine Package and Option Search * Factor out html body * Dispatch messages by search type to flake page * Correct type naming * Remove Debug instructions and unimplemented flake search type * Do not reload Flake search page while the search subject is unchanged * Implement switching subjects * Fix init type signature * Add url/git based flakes to mapping * Parse flake info * Link to flake repo and show flake maintainer * Fix optional decoded values * Add group and type as search buckts * Show search selection in all cases * Move flake decoding to search * Show flake information for options * Hardcode flakesearch to only search a specified flake index * Improve experimental state notice * Fix category select not present in some cases * Change default flake index * Show correct category title * Add missing imports * Serialize more optional fields as null * elastic-test.rs file not needed anymore * names of the workflows should be immediately obvious. * better jobs names * rename the flakes group * need to provide --elastic-schema-version via the cli option * typo * Fix errors in workflow files * Install flake enabled nix by default * Fix variable substitution * Use string group names * Provide elastic schema verion only through cli * Fix scheduled task name * Improve error reporting * Allow insecure packages here * Add missing imports * Tree-wide: cargo-fmt * only import nixpkgs for now * no importing of flakes * also bump a version * fixing cron-flakes.yml * make it obvious that this is for frontend * missed this when merging * Extract hashes - Split out revision info from flakes - Retrieve current nixpkgs revision from github * Write aliases, enabled by default * Filter additional platforms * Change alias structure * expect channel like nixpkgs identifiers * Don't cause error if push is aborted * Specify correct channel identifier * Allow options to evaluate unfree packages * Retrieve and delete specific aliases * Specify import path correclty * Fix channel warmup * Abort push if channel already indexed * Remove debug pr hook for import action * Fix indentation by tab * Make flakes show again * Fix import group naming in flake wrokflow * Rename flake group to match imported index * Run nixpkgs import on pr activity (Debugging behavior) * Just show literal Examples, resolves #336 * Use actual nixpkgs branch names * Trim derivation/option declaration path, resolves #335 Remove /nix/store/*-source prefix * Fix sidebar width and close button position * Placeholder texts in flake result area * Show flake install info * Don't show package/option selection before search * Make sure install command for nixos is always shown * Add toml source config support * Rewrite current example flake group as toml * Update flake cron job * flake-info: use saner nix packaging method When trying to first work in this I naive approached it with `nix-shell`. That of course lead to the fixed output bollocks failing with a hash mismatch. By making use of the `cargoLock` attribute on `buildRustPackage` we can tame the FOD-beast and only have to provide one hash manually for a single package (that we fetch from a GitHub repository). This also means that updating dependencies will be simpler as the native Cargo.lock file can be used. (cherry picked from commit c3a0e46d1eb56e128e6923e6c493eb836fc81e85) * Update flake lock file * Do not build python import script * add flake names to the title as well * Disable debug imports on prs Co-authored-by: Rok Garbas <rok@garbas.si> Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
2021-08-25 22:40:42 +00:00
- name: Import ${{ matrix.group }} group
run: |
./result/bin/flake-info --push --elastic-schema-version=$(cat ./VERSION) group ./flakes/${{ matrix.group }}.toml ${{ matrix.group }}
if: github.repository == 'NixOS/nixos-search'
- name: Warmup ${{ matrix.group }} channel
run: |
curl ${{ secrets.ELASTICSEARCH_URL }}/$(cat VERSION)-${{ matrix.group }}/_search | jq '.took'
curl ${{ secrets.ELASTICSEARCH_URL }}/$(cat VERSION)-${{ matrix.group }}/_search | jq '.took'
curl ${{ secrets.ELASTICSEARCH_URL }}/$(cat VERSION)-${{ matrix.group }}/_search | jq '.took'
if: github.repository == 'NixOS/nixos-search'
Reference in a new issue Copy permalink