* Render descriptions as Markdown
Following https://github.com/NixOS/nixpkgs/pull/237557, DocBook
descriptions are not supported any more.
We cannot use options.json any more because it still has DocBook
descriptions in 22.11, so read the options from the modules directly.
* Bump VERSION
* flake-info: don't render definition lists
Nothing uses them in nixpkgs and they mess up descriptions that have a
line starting with a tilde.
* Bump VERSION
* Fix `fix-xrefs.lua`'s empty link detection
Since we now go directly from Markdown to HTML, Pandoc sees actually
empty links instead of `???`.
* Bump VERSION
* Search programs provided by a package
Use the `programs.sqlite` database provided with nixpkgs channels to
populate a `package_programs` field so that searches for e.g. `make` return
`gnumake` with a higher priority.
* Bump VERSION
* frontend: show programs
* flake-info: rely on upstream for extracting nixpkgs options
Use the `options.json` generated in nixpkgs to be more in sync with the
manual.
Also fixes usage of `nixpkgs-archive` with local paths.
* Bump VERSION
* Make the channel importer versions dynamic
* few things fromt he list
1. add nixos-org-configurations as input to flake.nix
2. evaluate channels.nix file and export channels via environment variable.
that environment variable (lets call it NIXOS_CHANNELS) should be present
during the build and inside the nix shell. the content of the variable can
be JSON.
3. we pickup the NIXOS_CHANNELS environment variable in
frontend/webpack.config.js and pass it further to webpack process, just
like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION.
4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm
application flag. Just like we do with other variables there.
* Decode nixosChannels in Elm
* Use nixosChannels that came via application flag
* read nixos channels in github action
* defaultNixOSChannel should be calculated
* add two pointers where the check should be added
* pass nixosChannels to flake-info and remove title, rather calculate it
* Add NixosChannels struct validation and validation Error
* Read NIXOS_CHANNEL variable
* Check channel
* Add channel struct to fix parsing NIXOS_CHANNELS
* Use `eachDefaultSystem` instead of listing them manually
* Add individual dev shells for frontend and backend
* Update .github/workflows/import-to-elasticsearch.yml
Co-authored-by: Naïm Favier <n@monade.li>
* use both development environments by default (as it was)
but still provide devShells for each of the subprojects
* pkgs.lib → lib everywhere
and define lib = nixpkgs.lib before the call to eachDefaultSystem
Also, version = lib.fileContents ./VERSION;
* Update flake.nix
Co-authored-by: Naïm Favier <n@monade.li>
* typo
* bumping version to test the changes to import-to-elasticsearch github action
* some invisibile characters needed to be removed
* typo
* will this work
* typo
* forgot the checkout step
* add some debugging
* typo
* read NIXOS_CHANNELS from environment not via argument
* fix for the NIXOS_CHANNELS variable
Co-authored-by: Janne Heß <janne@hess.ooo>
Co-authored-by: ysndr <me@ysndr.de>
Co-authored-by: Naïm Favier <n@monade.li>
* flake-info: fix import of `null` defaults and examples
Currently they're being treated as missing fields, and hence displayed
as "Not given" on the frontend.
* Bump VERSION
* frontend/Options: don't show missing fields
Co-authored-by: Rok Garbas <rok@garbas.si>
* Show default outputs
Following https://github.com/NixOS/nix/pull/6242, we now have access to
default outputs so we can single them out in the frontend.
We can also drop `--out-path`, which speeds up import quite a lot and
doesn't require increasing swap space or allowing broken/unfree
packages anymore.
* Use nixUnstable in CI
* Bump VERSION
As mentioned in [^1] this PR includes a bump in swap size of the github action evaluating nixpkgs.
This is subject to change once a tentative change to Nix is merged.
^1: https://github.com/NixOS/nixos-search/pull/419#issuecomment-1065356169
—
* flake-info: use packages-config.nix straight from nixpkgs
No need to override anything anymore, see discussion at
https://github.com/NixOS/nixos-search/pull/343#issuecomment-1021147104
* flake-info: query package outputs
`package_outputs` is now set to the actual outputs of the derivation
instead of `meta.outputsToInstall`
Also updates nixpkgs to get Rust 1.57 which has HashMap.into_keys
* frontend: show package outputs
* Drop 21.05 channel
* Increase swap space in import-nixpkgs worker
* Bump VERSION
* frontend: improvements and refactoring
- move licenses to the end (without a line break)
- pluralise "Licenses:" conditionally
- change the homepage emoji from a house to a world
- replace backwards List.append pipelines with ++ chains
- avoid text nodes in <ul> outside of <li>
- improve rendering of maintainers
* frontend: sort package outputs
* Extract pandoc function to own module
* Bump schema version
* Make string handling more explicit
* Update logging and pandoc crates
* Improve serializing error handling
* Serialize values as string (for elastic)
* Perform option doc parsing entirely in rust
* Show non pandoc'ed results as code elements
* Parse correct html string
* Change expected `option_type` type to String
* Allow treat string unparsable as html like non-docbook strings
* Improve deserializing error reporting using serde_path_to_error
* Format code
* Setup flake info extraction
Prepare data model fro derivations (#1)
Add flake info data (#1)
Implement fetching general flake info (#1)
Expose CLI (#1)
Keep cargo happy
Add some doc comments
Pin to local nixpkgs to excessive downloads
Extend visibility of some data objects
Add command to extract infomation about defivations (#1)
Add call new feature in main (#1)
Include more information in derivation (#1)
Add log access
Always debug log stderr from nix
Format nix script
Collect systems per package
Remove unnecessary imports
Create flake
Remove top level version field
Represent collected systems/version pairs in rust
Fix quotation marks in tests
Add correct cargo hash
Add iconv dependency
Return a list from nix script
Export as json
Undo version by platform distinction
Remove nixpkgs override
Apply cargo fmt
Flatten export structure
Allow for complex licenses
Prepare using a central nix file
Implement nix part o accessing apps
Include the correct filename
Add accessor for `all` key
Access all available information by default
Track more information about Apps
Run cargo fmt
Fix: allow local builds
Prepare next version of the flake info tool
Include examples and pull script
Expose flake info as library
Include thiserror for custom errors
Define a source data type
Collects source types and their metadata, collected in a json file
Add command line argument for input files
Mutually exclusive with --flake
Refactor functions to extract information given a flake identifier
Add kind specifier as CLI argument
Amend Argument parsing to require eiteher flake or targets to be defined
Run extraction for specified flake or list of flakes as specified in a json file
Resolves#5
References #7
Use internal tag to distnguich target types
Include target falg usage in examples
Set include provided source if available (resolves#9)
Resolve flake name
Update examples
Dont include empty license or description
Fix a misfomatting in cargot.toml
Add elastic dependencies
Implement a wrapper around the elasticsearch client
Implements pushing exports (#4)
Temporarily skip serializing an unimplemented field in elastic output
Extract reading source list files from binary
Add lazy_static as dependency
Implement createing and pushing to elastic index
Add elastic options
Provide default name and env falbac for elastic index
Modify app binary and type as optionals
App can be a derivation too
Update examples
Add more elastic commands
Supported:
- ensure
- clear
- push
Rename elastic search config struct
Add elastic push support to binary
Rename flag to enable elastic push
Imporve error messages and format binary source
Fix nix file incorrectly expecting meta fields
Changing flake descriotions to an optional field
deserialize git_ref as hash
Implement temporary stores and gc of these
prevents flakes from accessing store paths
Pass extra arguments to nix
Update cargo hash and skip integration tests
Move flake.nix to root folder and add apps for all components
Fix command invocation that fails test
Update README(s)
Add help for extra arguments
(cherry picked from commit be4bc3dd929178bef66114c2201aaa88e47e9add)
* Safely read legacyPackages
* Read nixosOptions from flake
* Update ES Mapping
* Show more detailed error and backtrace if available
* Try reading options only if key is defined
* Format nix script
* Add error context when attempting deserialization
* Fix derivation representation to fit nix output
* Add push elasticsearch settings
* Add Flake channel
* Rename import module
* Remove Flakes Channel
* Prepare nixpkgs import
* Separate import/export types
* Break up nixpkgs package representation
* Use the same naming scheme for Nixpkgs entries and flake entries
* Document import module
* Remove serialization attributes
* Reversable type and SerDe implemetation
* Add *_reverse fields
* Unpublicating export fields
* Read from NixOption struct
* serialize empty fields as null
* Tag export json variants
* Serialize a single option-declaration
* Format npkgs parse test
* Define NixOption
Sorry thats too late..
* Parse system key
* Make Package output compatible with the frontend
* Add Url-only licesnse variant
* Add StringOrStruct type
* Add accessor method for elements catched by OneOrMany
* New utility to flatten in homogenous lists recursively
* Add Maintainer type catching maintaiers used in nixpkgs
* Format Implementatio n
* Remove explicit representation of platforms
* Open nixpkgs parser to cover all packages
* Convert all imported fields to their export representation
* Define reverse fields in ES schema
* Format nixpkgs command runner
* Expose shorthand to pull a specific channel by command line
* Extract utility functions into their own module
* Implement AttributeQuery generation
* Integrate query in export
* Implement *_set attributes
* Document purpose of export module
* Use more descriptive github blob route
* Complete Option Export representation
Reuse the same option type for nixpkgs and flakes
* Enable nixpkgs option import
* Expose nixpkgs option import functions and integrate in binary
* Chunk ElasticSearch Bulk operations
* Address Example/Default field formatting
* Add abort strategy for existing indices
* New command line interface using subcommands
* Document new interface
* Bump version
* Add nixpkgs cron job
* More precise name for nixpkgs cron job
* Add Flakes cron job
* Read version from file and fix channel names
* Correct file names for flake group import
* Fix group command
* Run new cron jobs on PR
* Update Cargo sources
* Integrate new flakes route
* Add search type option
* Add flake types and messages
* Add flake types
* Extract Request body builders and expose more types
* Combine Package and Option Search
* Factor out html body
* Dispatch messages by search type to flake page
* Correct type naming
* Remove Debug instructions and unimplemented flake search type
* Do not reload Flake search page while the search subject is unchanged
* Implement switching subjects
* Fix init type signature
* Add url/git based flakes to mapping
* Parse flake info
* Link to flake repo and show flake maintainer
* Fix optional decoded values
* Add group and type as search buckts
* Show search selection in all cases
* Move flake decoding to search
* Show flake information for options
* Hardcode flakesearch to only search a specified flake index
* Improve experimental state notice
* Fix category select not present in some cases
* Change default flake index
* Show correct category title
* Add missing imports
* Serialize more optional fields as null
* elastic-test.rs file not needed anymore
* names of the workflows should be immediately obvious.
* better jobs names
* rename the flakes group
* need to provide --elastic-schema-version via the cli option
* typo
* Fix errors in workflow files
* Install flake enabled nix by default
* Fix variable substitution
* Use string group names
* Provide elastic schema verion only through cli
* Fix scheduled task name
* Improve error reporting
* Allow insecure packages here
* Add missing imports
* Tree-wide: cargo-fmt
* only import nixpkgs for now
* no importing of flakes
* also bump a version
* fixing cron-flakes.yml
* make it obvious that this is for frontend
* missed this when merging
* Extract hashes
- Split out revision info from flakes
- Retrieve current nixpkgs revision from github
* Write aliases, enabled by default
* Filter additional platforms
* Change alias structure
* expect channel like nixpkgs identifiers
* Don't cause error if push is aborted
* Specify correct channel identifier
* Allow options to evaluate unfree packages
* Retrieve and delete specific aliases
* Specify import path correclty
* Fix channel warmup
* Abort push if channel already indexed
* Remove debug pr hook for import action
* Fix indentation by tab
* Make flakes show again
* Fix import group naming in flake wrokflow
* Rename flake group to match imported index
* Run nixpkgs import on pr activity (Debugging behavior)
* Just show literal Examples, resolves#336
* Use actual nixpkgs branch names
* Trim derivation/option declaration path, resolves#335
Remove /nix/store/*-source prefix
* Fix sidebar width and close button position
* Placeholder texts in flake result area
* Show flake install info
* Don't show package/option selection before search
* Make sure install command for nixos is always shown
* Add toml source config support
* Rewrite current example flake group as toml
* Update flake cron job
* flake-info: use saner nix packaging method
When trying to first work in this I naive approached it with
`nix-shell`. That of course lead to the fixed output bollocks failing
with a hash mismatch. By making use of the `cargoLock` attribute on
`buildRustPackage` we can tame the FOD-beast and only have to provide
one hash manually for a single package (that we fetch from a GitHub
repository).
This also means that updating dependencies will be simpler as the native
Cargo.lock file can be used.
(cherry picked from commit c3a0e46d1eb56e128e6923e6c493eb836fc81e85)
* Update flake lock file
* Do not build python import script
* add flake names to the title as well
* Disable debug imports on prs
Co-authored-by: Rok Garbas <rok@garbas.si>
Co-authored-by: Andreas Rammhold <andreas@rammhold.de>
* Reduce the numbers of bucket results on each ChangeBucket event
* Warmup after reindexing, fixes#250
* Using multiple sort keys, fixes#218
* Index also unfree packages, fixes#226, fixes#237
* Add wilcard query, fixes#279
* Fix rendering of default and example values
* bump the version
* reduce nix-instantiate calls
* Allow literalExample in default values, handle null correctly
* using py38 for now
* literalExample is not always string
* in case literalExample is not string process it as we would normally
* Remove 19.09 branch
* also remove 19.09 channel from github actions
* improve testing of empty lists and dicts
Co-authored-by: Rok Garbas <rok@garbas.si>
- still muse multi_match but now with cross_fields type
- add edge n-gram index field since cross_fields works only with match
and this way we can have prefix support
- suffix support is still supported as before by reversing the words in
data and query
- for query we now create variations of all multi_match queries. from 2
words you get 4 queries, from 3 works you get 8 queries and so on.
