aux-search/.github/workflows/import-to-elasticsearch.yml

122 lines
3.3 KiB
YAML
Raw Permalink Normal View History

name: "Hourly import to Elasticsearch"
on:
workflow_dispatch:
schedule:
- cron: '0 * * * *'
2023-02-07 15:32:04 +00:00
permissions:
contents: read
issues: write
jobs:
Feat/dynamic import (#473) * Make the channel importer versions dynamic * few things fromt he list 1. add nixos-org-configurations as input to flake.nix 2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON. 3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION. 4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there. * Decode nixosChannels in Elm * Use nixosChannels that came via application flag * read nixos channels in github action * defaultNixOSChannel should be calculated * add two pointers where the check should be added * pass nixosChannels to flake-info and remove title, rather calculate it * Add NixosChannels struct validation and validation Error * Read NIXOS_CHANNEL variable * Check channel * Add channel struct to fix parsing NIXOS_CHANNELS * Use `eachDefaultSystem` instead of listing them manually * Add individual dev shells for frontend and backend * Update .github/workflows/import-to-elasticsearch.yml Co-authored-by: Naïm Favier <n@monade.li> * use both development environments by default (as it was) but still provide devShells for each of the subprojects * pkgs.lib → lib everywhere and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION; * Update flake.nix Co-authored-by: Naïm Favier <n@monade.li> * typo * bumping version to test the changes to import-to-elasticsearch github action * some invisibile characters needed to be removed * typo * will this work * typo * forgot the checkout step * add some debugging * typo * read NIXOS_CHANNELS from environment not via argument * fix for the NIXOS_CHANNELS variable Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: ysndr <me@ysndr.de> Co-authored-by: Naïm Favier <n@monade.li>
2022-04-24 21:48:01 +00:00
nixos-channels:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.nixos-channels.outputs.matrix }}
steps:
- name: Checking out the repository
uses: actions/checkout@v4
Feat/dynamic import (#473) * Make the channel importer versions dynamic * few things fromt he list 1. add nixos-org-configurations as input to flake.nix 2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON. 3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION. 4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there. * Decode nixosChannels in Elm * Use nixosChannels that came via application flag * read nixos channels in github action * defaultNixOSChannel should be calculated * add two pointers where the check should be added * pass nixosChannels to flake-info and remove title, rather calculate it * Add NixosChannels struct validation and validation Error * Read NIXOS_CHANNEL variable * Check channel * Add channel struct to fix parsing NIXOS_CHANNELS * Use `eachDefaultSystem` instead of listing them manually * Add individual dev shells for frontend and backend * Update .github/workflows/import-to-elasticsearch.yml Co-authored-by: Naïm Favier <n@monade.li> * use both development environments by default (as it was) but still provide devShells for each of the subprojects * pkgs.lib → lib everywhere and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION; * Update flake.nix Co-authored-by: Naïm Favier <n@monade.li> * typo * bumping version to test the changes to import-to-elasticsearch github action * some invisibile characters needed to be removed * typo * will this work * typo * forgot the checkout step * add some debugging * typo * read NIXOS_CHANNELS from environment not via argument * fix for the NIXOS_CHANNELS variable Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: ysndr <me@ysndr.de> Co-authored-by: Naïm Favier <n@monade.li>
2022-04-24 21:48:01 +00:00
- name: Setup
uses: ./.github/actions/common-setup
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Feat/dynamic import (#473) * Make the channel importer versions dynamic * few things fromt he list 1. add nixos-org-configurations as input to flake.nix 2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON. 3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION. 4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there. * Decode nixosChannels in Elm * Use nixosChannels that came via application flag * read nixos channels in github action * defaultNixOSChannel should be calculated * add two pointers where the check should be added * pass nixosChannels to flake-info and remove title, rather calculate it * Add NixosChannels struct validation and validation Error * Read NIXOS_CHANNEL variable * Check channel * Add channel struct to fix parsing NIXOS_CHANNELS * Use `eachDefaultSystem` instead of listing them manually * Add individual dev shells for frontend and backend * Update .github/workflows/import-to-elasticsearch.yml Co-authored-by: Naïm Favier <n@monade.li> * use both development environments by default (as it was) but still provide devShells for each of the subprojects * pkgs.lib → lib everywhere and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION; * Update flake.nix Co-authored-by: Naïm Favier <n@monade.li> * typo * bumping version to test the changes to import-to-elasticsearch github action * some invisibile characters needed to be removed * typo * will this work * typo * forgot the checkout step * add some debugging * typo * read NIXOS_CHANNELS from environment not via argument * fix for the NIXOS_CHANNELS variable Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: ysndr <me@ysndr.de> Co-authored-by: Naïm Favier <n@monade.li>
2022-04-24 21:48:01 +00:00
CACHIX_SIGNING_KEY: ${{ secrets.CACHIX_SIGNING_KEY }}
- name: NixOS Channels
id: nixos-channels
run: |
nix build --accept-flake-config -L .#nixosChannels
Feat/dynamic import (#473) * Make the channel importer versions dynamic * few things fromt he list 1. add nixos-org-configurations as input to flake.nix 2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON. 3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION. 4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there. * Decode nixosChannels in Elm * Use nixosChannels that came via application flag * read nixos channels in github action * defaultNixOSChannel should be calculated * add two pointers where the check should be added * pass nixosChannels to flake-info and remove title, rather calculate it * Add NixosChannels struct validation and validation Error * Read NIXOS_CHANNEL variable * Check channel * Add channel struct to fix parsing NIXOS_CHANNELS * Use `eachDefaultSystem` instead of listing them manually * Add individual dev shells for frontend and backend * Update .github/workflows/import-to-elasticsearch.yml Co-authored-by: Naïm Favier <n@monade.li> * use both development environments by default (as it was) but still provide devShells for each of the subprojects * pkgs.lib → lib everywhere and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION; * Update flake.nix Co-authored-by: Naïm Favier <n@monade.li> * typo * bumping version to test the changes to import-to-elasticsearch github action * some invisibile characters needed to be removed * typo * will this work * typo * forgot the checkout step * add some debugging * typo * read NIXOS_CHANNELS from environment not via argument * fix for the NIXOS_CHANNELS variable Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: ysndr <me@ysndr.de> Co-authored-by: Naïm Favier <n@monade.li>
2022-04-24 21:48:01 +00:00
channels="{\"channel\": $(< ./result)}"
echo "$channels"
echo "matrix=$channels" >> "$GITHUB_OUTPUT"
Feat/dynamic import (#473) * Make the channel importer versions dynamic * few things fromt he list 1. add nixos-org-configurations as input to flake.nix 2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON. 3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION. 4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there. * Decode nixosChannels in Elm * Use nixosChannels that came via application flag * read nixos channels in github action * defaultNixOSChannel should be calculated * add two pointers where the check should be added * pass nixosChannels to flake-info and remove title, rather calculate it * Add NixosChannels struct validation and validation Error * Read NIXOS_CHANNEL variable * Check channel * Add channel struct to fix parsing NIXOS_CHANNELS * Use `eachDefaultSystem` instead of listing them manually * Add individual dev shells for frontend and backend * Update .github/workflows/import-to-elasticsearch.yml Co-authored-by: Naïm Favier <n@monade.li> * use both development environments by default (as it was) but still provide devShells for each of the subprojects * pkgs.lib → lib everywhere and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION; * Update flake.nix Co-authored-by: Naïm Favier <n@monade.li> * typo * bumping version to test the changes to import-to-elasticsearch github action * some invisibile characters needed to be removed * typo * will this work * typo * forgot the checkout step * add some debugging * typo * read NIXOS_CHANNELS from environment not via argument * fix for the NIXOS_CHANNELS variable Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: ysndr <me@ysndr.de> Co-authored-by: Naïm Favier <n@monade.li>
2022-04-24 21:48:01 +00:00
import-nixpkgs:
Feat/dynamic import (#473) * Make the channel importer versions dynamic * few things fromt he list 1. add nixos-org-configurations as input to flake.nix 2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON. 3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION. 4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there. * Decode nixosChannels in Elm * Use nixosChannels that came via application flag * read nixos channels in github action * defaultNixOSChannel should be calculated * add two pointers where the check should be added * pass nixosChannels to flake-info and remove title, rather calculate it * Add NixosChannels struct validation and validation Error * Read NIXOS_CHANNEL variable * Check channel * Add channel struct to fix parsing NIXOS_CHANNELS * Use `eachDefaultSystem` instead of listing them manually * Add individual dev shells for frontend and backend * Update .github/workflows/import-to-elasticsearch.yml Co-authored-by: Naïm Favier <n@monade.li> * use both development environments by default (as it was) but still provide devShells for each of the subprojects * pkgs.lib → lib everywhere and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION; * Update flake.nix Co-authored-by: Naïm Favier <n@monade.li> * typo * bumping version to test the changes to import-to-elasticsearch github action * some invisibile characters needed to be removed * typo * will this work * typo * forgot the checkout step * add some debugging * typo * read NIXOS_CHANNELS from environment not via argument * fix for the NIXOS_CHANNELS variable Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: ysndr <me@ysndr.de> Co-authored-by: Naïm Favier <n@monade.li>
2022-04-24 21:48:01 +00:00
needs: nixos-channels
runs-on: ubuntu-latest
strategy:
fail-fast: false
Feat/dynamic import (#473) * Make the channel importer versions dynamic * few things fromt he list 1. add nixos-org-configurations as input to flake.nix 2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON. 3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION. 4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there. * Decode nixosChannels in Elm * Use nixosChannels that came via application flag * read nixos channels in github action * defaultNixOSChannel should be calculated * add two pointers where the check should be added * pass nixosChannels to flake-info and remove title, rather calculate it * Add NixosChannels struct validation and validation Error * Read NIXOS_CHANNEL variable * Check channel * Add channel struct to fix parsing NIXOS_CHANNELS * Use `eachDefaultSystem` instead of listing them manually * Add individual dev shells for frontend and backend * Update .github/workflows/import-to-elasticsearch.yml Co-authored-by: Naïm Favier <n@monade.li> * use both development environments by default (as it was) but still provide devShells for each of the subprojects * pkgs.lib → lib everywhere and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION; * Update flake.nix Co-authored-by: Naïm Favier <n@monade.li> * typo * bumping version to test the changes to import-to-elasticsearch github action * some invisibile characters needed to be removed * typo * will this work * typo * forgot the checkout step * add some debugging * typo * read NIXOS_CHANNELS from environment not via argument * fix for the NIXOS_CHANNELS variable Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: ysndr <me@ysndr.de> Co-authored-by: Naïm Favier <n@monade.li>
2022-04-24 21:48:01 +00:00
matrix: ${{ fromJSON(needs.nixos-channels.outputs.matrix) }}
env:
RUST_LOG: debug
FI_ES_EXISTS_STRATEGY: abort
FI_ES_URL: ${{ secrets.ELASTICSEARCH_URL2 }}
steps:
- name: Checking out the repository
uses: actions/checkout@v4
- name: Setup
uses: ./.github/actions/common-setup
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CACHIX_SIGNING_KEY: ${{ secrets.CACHIX_SIGNING_KEY }}
- name: Import ${{ matrix.channel }} channel
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
nix run --accept-flake-config .#flake-info -- --push --elastic-schema-version=$(< VERSION) nixpkgs ${{ matrix.channel }}
if: github.repository_owner == 'NixOS'
- name: Warmup ${{ matrix.channel }} channel
run: |
for (( i = 0; i < 3; i++ )) do
curl -sS ${{ secrets.ELASTICSEARCH_URL2 }}/latest-$(< VERSION)-nixos-${{ matrix.channel }}/_search | jq -c '.took // .'
done
if: github.repository_owner == 'NixOS'
2023-02-07 15:32:04 +00:00
- name: Create issue if failed
if: failure()
uses: JasonEtco/create-an-issue@v2
with:
filename: .github/failing-import-issue.md
search_existing: open
update_existing: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CHANNEL: ${{ matrix.channel }}
import-flakes:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
group:
- "manual"
env:
RUST_LOG: debug
FI_ES_EXISTS_STRATEGY: recreate
FI_ES_URL: ${{ secrets.ELASTICSEARCH_URL2 }}
steps:
- name: Checking out the repository
uses: actions/checkout@v4
- name: Setup
uses: ./.github/actions/common-setup
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CACHIX_SIGNING_KEY: ${{ secrets.CACHIX_SIGNING_KEY }}
- name: Import ${{ matrix.group }} group
run: |
nix run --accept-flake-config .#flake-info -- --push --elastic-schema-version=$(< ./VERSION) group ./flakes/${{ matrix.group }}.toml ${{ matrix.group }}
if: github.repository_owner == 'NixOS'
- name: Warmup ${{ matrix.group }} group
run: |
for (( i = 0; i < 3; i++ )) do
curl -sS ${{ secrets.ELASTICSEARCH_URL2 }}/latest-$(< VERSION)-group-${{ matrix.group }}/_search | jq -c '.took // .'
done
if: github.repository_owner == 'NixOS'