Commit graph

248 commits

Author SHA1 Message Date
Till Wegmueller
def11a1dfb
Refactor: Replace redb with rusqlite for catalog handling
- Transitioned the catalog backend from `redb` to `rusqlite` for better compatibility and concurrency.
- Updated `IpsProvider` to use SQLite for package querying, dependency resolution, and obsolescence handling.
- Removed `decode_manifest_bytes_local` and unused `manifest` cache logic.
- Simplified catalog-related functions and integrated `sqlite_catalog` module.
- Enhanced test functions and added schemas for managing SQLite databases.
2026-02-04 22:39:42 +01:00
Till Wegmueller
4ab529f4c7
chore(format): format code 2026-01-25 23:17:49 +01:00
Till Wegmueller
e7dc9d5a4c
Add comment for testing in Solstice CI workflow configuration 2026-01-25 22:02:03 +01:00
Till Wegmueller
04f37476f0
Add Solstice CI workflow with per-OS setup scripts
- Introduced `.solstice/workflow.kdl` for defining Solstice CI workflows, including Linux and illumos builds.
- Added `setup-linux.sh` and `setup-illumos.sh` scripts for per-OS environment preparation.
- Implemented `job.sh` as a legacy script hook for additional build steps.
2026-01-25 20:05:07 +01:00
Till Wegmueller
e1ce390abd
Refactor code to simplify handling and remove unused functions
- Removed `add_package` function from `FileBackend`, consolidating package indexing logic.
- Allowed unused assignments using `#![allow(unused_assignments)]` where necessary.
- Addressed unused imports in integration tests.
- Simplified `Digest` error handling by removing unreachable branches for unknown algorithms.
2026-01-20 22:19:25 +01:00
Till Wegmueller
1c0619ca55
Add pkg6recv package for receiving IPS repositories.
- Introduced `pkg6recv` for downloading packages from IPS repositories with support for recursive dependency fetching.
- Implemented CLI commands with `clap` for source/destination setup, package selection, and publisher defaults.
- Added `ConsoleProgressReporter` for detailed receive progress visibility.
- Enhanced the `Manifest` parser to support JSON format alongside IPS text format.
- Updated `FileBackend` and `RestBackend` repositories to fetch manifests and payloads in new operations.
- Extended `Digest` utilities for payload integrity checks.
- Added tests to verify basic functionality and manifest format preservation.
2026-01-20 20:16:58 +01:00
Till Wegmueller
22178cffd7
Add manual testing setup for pkg6depotd
- Introduced scripts and configurations for manual testing using `anyvm` with OpenIndiana and OmniOS.
- Implemented repository fetching (`fetch_repo.sh`) and server startup (`run_depotd.sh`) scripts.
- Enhanced `pkg6depotd` to support default publisher routes and trailing slashes.
- Updated integration tests to verify new publisher route behavior.
2026-01-20 17:44:36 +01:00
Till Wegmueller
d16e5339ce
chore: Release 2026-01-19 00:28:58 +01:00
Till Wegmueller
d0722bdacd
Enable push and tag in workspace release configuration. 2026-01-19 00:28:41 +01:00
Till Wegmueller
074dad33bf
chore: Release {{version}} 2026-01-19 00:27:43 +01:00
Till Wegmueller
a8174e8da2
Mark package as non-publishable in workspace configuration. 2026-01-19 00:27:17 +01:00
Till Wegmueller
6ee6392b01
Standardize description, version, authors, and edition fields across packages using workspace inheritance. 2026-01-19 00:25:47 +01:00
Till Wegmueller
86e1a2eb53
Update OmniOS workflows and Vagrantfile to replace library/lz4 with compress/lz4 2026-01-18 17:27:54 +01:00
Till Wegmueller
d763bf1c34
Simplify PkgTreeError struct and consolidate error handling logic across main.rs. Update OmniOS workflows and Vagrantfile to include additional libraries and improve configuration consistency. 2026-01-18 17:05:59 +01:00
Till Wegmueller
0e873ee114
Update OmniOS workflows to use ooce packages
- Replaced `extra.omnios` publisher setup and outdated package names with `ooce` packages in `rust.yml` and `release.yml`.
2026-01-18 16:45:43 +01:00
Till Wegmueller
d7c7079374
Add OmniOS extra publisher setup to workflows
- Updated both `rust.yml` and `release.yml` to configure the extra OmniOS publisher for additional package installations.
2026-01-18 16:32:27 +01:00
Till Wegmueller
4d96cf1d2b
Update Illumos build workflows to use OmniOS VM
- Replaced OpenIndiana VM with OmniOS VM in Illumos build steps across `rust.yml` and `release.yml` workflows.
- Updated related workflow labels and comments for consistency.
2026-01-18 16:01:24 +01:00
Till Wegmueller
7fb3793c54
Remove Jenkinsfile, add Illumos builds to GitHub Actions, and document release process
- Deleted the Jenkinsfile, transitioning all CI workflows to GitHub Actions.
- Added Illumos (OpenIndiana) build and release steps to GitHub workflows.
- Created a new release pipeline triggered by version tags starting with `v`.
- Updated README with release instructions using `cargo-release`.
2026-01-18 14:59:24 +01:00
Till Wegmueller
898ec20ad8
Update binaries and workflows for pkg6 rename
- Renamed `pkg6dev` to `pkg6` across build scripts, workflows, and documentation.
- Added support for additional binaries (`pkg6repo` and `pkg6depotd`) in release builds.
- Disabled `RUSTFLAGS` warnings-as-errors policy in workflows for improved flexibility.
- Simplified error handling in `manifest_fmri` with streamlined conditionals.
- Introduced `#[allow(clippy::result_large_err)]` to suppress clippy warnings for large error types.
2026-01-18 14:30:05 +01:00
Till Wegmueller
b8c625a11e
Refactor attribute value access and error handling
- Replaced `values.get(0)` with `values.first()` for improved clarity and consistency across multiple files.
- Simplified conditional checks with `.is_empty()` and `format_args!` to enhance readability.
- Implemented `Default` trait for `VariableMode` directly within the enum.
- Updated error handling by boxing errors in `RepositoryError` and `CatalogError`.
- Replaced redundant `else` blocks with streamlined logic.
2026-01-18 13:00:21 +01:00
Till Wegmueller
9512eb3b12
chore: cargo fmt 2026-01-18 12:51:55 +01:00
Till Wegmueller
8f089656ba
Add search functionality to repository and route handlers (currently disabled)
- Introduced a searchable index with structured `IndexEntry` support for packages, files, directories, and dependencies.
- Added `search` method in `DepotRepo` with wildcard and case-sensitive query handling.
- Created `/search/0` and `/search/1` routes for search API, supporting publishers and token-based queries.
- Updated `SearchIndex` handling to map tokens to detailed `IndexEntry` structures.
- Improved index building to include attributes for files, directories, and dependencies.
2026-01-18 12:29:44 +01:00
Till Wegmueller
b080288114
Add legacy manifest handling with JSON fallback for transaction commits
- Introduced `legacy_manifest_content` field in `Transaction` to allow preserving byte-identical legacy manifests.
- Updated `commit` method to save both JSON and legacy manifests, falling back to JSON if no legacy content is provided.
- Enhanced manifest copy logic to handle both JSON and legacy formats during transaction finalization.
- Improved catalog rebuild to skip `.json` files when a corresponding legacy manifest exists.
2025-12-23 14:09:14 +01:00
Till Wegmueller
7dc475ed2d
Fix FMRI and catalog processing inconsistencies
- Corrected FMRI formatting to `pkg:/name` for entries without a publisher.
- Excluded `pkg.fmri` from summary action extraction in file backend.
- Removed unused signature handling in catalog package processing.
- Reordered `PackageVersionEntry` struct fields for logical consistency.
2025-12-23 13:19:51 +01:00
Till Wegmueller
e0de265d17
Add rebuild_test_repo.sh script for rebuilding test repositories with PostgreSQL packages 2025-12-23 12:33:31 +01:00
Till Wegmueller
b32ace705f
Refactor whitespace handling in JSON serialization and improve ISO-8601 timestamp formatting
- Removed unnecessary spaces in JSON key-value and array formatting for Python-style compatibility.
- Enhanced `format_iso8601_basic` to ensure precise microsecond-level time formatting.
- Updated `write_update_log` to return and store SHA-1 signatures for update log verification.
2025-12-23 12:33:12 +01:00
Till Wegmueller
ff0b9f4319
Add support for file URL without algorithm and refactor JSON serialization for Python-style compatibility
- Added `/file/1/{digest}` route to support file retrieval without specifying the algorithm.
- Implemented a Python-compatible JSON serializer to ensure consistent formatting for catalog artifacts.
- Replaced `HashMap` with `BTreeMap` for deterministic ordering in catalog serialization and updates.
- Updated integration tests to validate the new route functionality and ensure response correctness.
- Refactored `format_iso8601_basic` to improve timestamp formatting consistency.
2025-12-22 22:42:56 +01:00
Till Wegmueller
a921c99eb6
Reorder struct fields for consistency and align with Rust formatting standards. 2025-12-22 20:11:08 +01:00
Till Wegmueller
d2d1c297cc
Refactor to align with Rust formatting guidelines and enhance code readability.
- Adjusted indentation and line breaks for structs, functions, and method calls to comply with Rust formatting standards.
- Improved error message formatting and consistency across `PkgTreeError` instances.
- Restructured long function arguments and chained calls for clarity and maintainability.
- Simplified conditional statements and loops for better readability.
- No functional changes introduced.
2025-12-22 20:10:17 +01:00
Till Wegmueller
d0fcdbec20
Update Vagrantfile with memory configuration, synced folder, and provisioning enhancements
- Added provider-specific memory allocation (8192 MB) for VirtualBox and Libvirt.
- Configured synced folder using rsync with `.git/` and `target/` exclusions.
- Streamlined provisioning by replacing GCC dependencies with Rust and updating package installation list.
- Removed redundant provisioning script for Rust setup.
2025-12-22 19:39:15 +01:00
Till Wegmueller
81f00a0624
Add caching headers, admin routes, and configurable cache max-age support
- Introduced caching headers (`Cache-Control`, `ETag`, `Last-Modified`) for file responses to improve client-side caching.
- Added HTTP admin routes for health check and authentication validation.
- Made `cache_max_age` configurable via server configuration with a default of 3600 seconds.
- Enhanced file handler with metadata-based `Last-Modified` computation.
- Updated integration tests to cover new functionality and ensure correctness.
2025-12-09 20:23:00 +01:00
Till Wegmueller
3457b4acba
Add version parsing, FMRI formatting, and license content resolution
- Enhanced `get_info` handler to parse version components into `Build Release`, `Branch`, and `Packaging Date`.
- Added proper FMRI formatting in `pkg://<publisher>/<name>@<version>` format.
- Implemented license content resolution with file-based lookup, gzip decompression, and content truncation.
- Introduced compressed and uncompressed package size calculations for manifest actions.
- Updated dependencies to include `chrono` and `flate2` for date parsing and gzip decompression.
2025-12-09 16:42:21 +01:00
Till Wegmueller
cff3d5d960
Add content-type handling for JSON catalog artifacts and weak ETag for manifests
- Ensured correct `Content-Type` header for catalog artifacts (`catalog.attrs` and `catalog.*`) in HTTP responses.
- Added SHA-1 based weak ETag generation for manifest responses to improve caching and legacy compatibility.
- Updated `integration_tests` to validate content-type and ETag correctness.
- Added new dependency `sha1` for hashing support.
2025-12-09 16:02:02 +01:00
Till Wegmueller
e87d1a3166
Add batched catalog rebuild support and INI-only repo test
- Introduced `BatchOptions` for configurable batched catalog rebuilds.
- Added `rebuild_catalog_batched` in `file_backend` for batched processing with deterministic package order and progress tracking.
- Updated `rebuild_catalog` to leverage the new batched implementation with default options.
- Added integration test `test_ini_only_repo_serving_catalog` to validate functionality with an INI-only repository configuration.
- Enhanced test coverage for legacy catalog handling via `pkg5.repository`.
2025-12-09 14:23:55 +01:00
Till Wegmueller
bd67e06012
Add catalog_writer module for atomic catalog updates with SHA-1 signatures
- Introduced `write_catalog_attrs`, `write_catalog_part`, and `write_update_log` functions for streamlined and secure file writing.
- Refactored `file_backend` to use `catalog_writer` for managing catalog updates, improving readability and maintainability.
- Updated `save_catalog_attrs`, `save_catalog_part`, and `append_update` to leverage atomic writes and ensure signature computation.
- Replaced manual serialization logic with centralized writing utilities for consistency and error resilience.
- Updated dependencies for JSON handling and signature computation.
2025-12-09 12:49:25 +01:00
Till Wegmueller
a948f87e6f
Add legacy repository support and SHA-1 signature handling
- Introduced fallback for legacy `pkg5.repository` configuration in INI format alongside the existing `pkg6.repository` JSON format.
- Enabled SHA-1 signature computation for compatibility with legacy catalog signatures.
- Added methods to save update logs in legacy format and enhance catalog compatibility.
- Updated dependencies to include `sha1` for hashing.
2025-12-09 12:12:57 +01:00
Till Wegmueller
c4910bb434
Refactor error handling in repository module to include structured error variants with detailed context.
- Replaced string-based `FileReadError`, `FileWriteError`, and `DirectoryCreateError` with structured error types using `PathBuf` and `source`.
- Updated all affected methods in `rest_backend`, `file_backend`, and tests to use the new structured error variants.
- Improved error messages for clarity and debugging by embedding file paths and sources directly into error data.
2025-12-08 23:13:27 +01:00
Till Wegmueller
ee02bf3cf0
Simplify config loading by replacing match with unwrap_or_else for improved readability. 2025-12-08 22:49:35 +01:00
Till Wegmueller
81bc7b8574
Refactor catalog handling by replacing get_legacy_catalog with get_catalog_file_path for improved clarity and consistency.
- Removed now-redundant catalog path resolution logic in `repo.rs`.
- Updated `file_backend` with a unified method for fetching catalog file paths, addressing invalid path prefixes.
- Adjusted HTTP handler to use the updated method.
2025-12-08 22:45:39 +01:00
Till Wegmueller
55decc16ff
Simplify CatalogPart::load by removing manual JSON parsing fallbacks and introducing test coverage for sample catalogs. 2025-12-08 22:39:28 +01:00
Till Wegmueller
84b2c50ed6
Remove deprecated Catalog v0 support and refactor legacy catalog handling.
- Removed `get_catalog` handler and associated Catalog v0 route.
- Updated `get_legacy_catalog` to accept `filename` for enhanced flexibility.
- Adjusted integration tests to validate changes and focus on Catalog v1 support.
- Refactored `SupportedOperation` definitions to exclude Catalog v0.
2025-12-08 22:10:11 +01:00
Till Wegmueller
048dddf487
Refactor get_versions response with structured VersionsResponse model.
- Added `Operation` enum and `SupportedOperation` struct to encapsulate supported operations and versions.
- Updated response generation to dynamically include the `pkg6depotd` version and supported operations.
- Enhanced integration tests to validate version string format changes.
2025-12-08 21:41:02 +01:00
Till Wegmueller
0b3a974ca6
Add REST API v1 endpoints and legacy catalog handling for pkg6depotd
- Expanded repository structure by introducing methods for fetching legacy catalogs, catalog file paths, and repository info.
- Added new REST API v1 endpoints for catalog, manifest, file, and publisher handling.
- Implemented `publisher` handler module with `get_publisher_v0` and `get_publisher_v1` methods to retrieve publisher details in pkg5 format.
- Updated `integration_tests` to validate new endpoints and ensure compatibility with legacy and modern catalog/manifest handling.
- Removed unused dependency `walkdir` and refactored test cases for clarity and efficiency.
2025-12-08 21:36:37 +01:00
Till Wegmueller
cd15e21420
Add repository handling and foundational HTTP routes for pkg6depotd
- Implemented `DepotRepo` for repository access, including methods for catalog path, file path, and manifest retrieval.
- Introduced foundational HTTP routes for catalog, manifest, file, and package info retrieval.
- Added integration tests to validate repository setup and basic server functionality.
- Modularized HTTP handlers for better maintainability and extended them with new implementations like `info` and `manifest` handling.
- Refactored `main` function to simplify initialization and leverage reusable `run` logic in a new `lib.rs`.
- Updated `Cargo.toml` and `Cargo.lock` to include new dependencies: `walkdir` and updated testing utilities.
2025-12-08 20:50:20 +01:00
Till Wegmueller
f2a3bc4d7c
Introduce foundational structure for pkg6depotd
- Added initial implementation of the `pkg6depotd` server with modular components for CLI parsing, configuration management, HTTP handling, repository access, and daemonization.
- Implemented basic server startup logic with a default router and placeholder handlers.
- Integrated telemetry initialization and configuration fallback mechanism for ease of development.
- Updated `Cargo.toml` and `Cargo.lock` to include dependencies necessary for server functionality.
2025-12-08 20:11:05 +01:00
Till Wegmueller
340e58ca09
Add comprehensive documentation for IPS search index design and implementation
- Documented indexer workflow, consistency mechanisms, and file layout for client-side and server-side indexing.
- Detailed fast incremental updates and full rebuild criteria with corresponding file modifications.
- Specified encoding formats, invariants, and validation rules for on-disk index files.
- Enhanced developer documentation for maintaining and troubleshooting the search index.
- Updated VCS configuration to remove unused submodule mapping.
2025-12-08 20:10:30 +01:00
Till Wegmueller
6be608164d
Add detailed documentation for Redb-based IPS Search Index schema and encoding plans
- Introduced a series of planning documents detailing the Redb-based IPS index design, schema specification, and encoding strategies.
- Added a high-level overview of the core search index schema and the use of Redb MVCC transactions for consistency and performance improvements.
- Documented simplified schema definitions avoiding optional elements, focusing on compact encodings.
- Defined transitions to postcard-encoded binary formats, aligning with Rust’s serde for standardized serialization.
- Outlined migration strategies, invariants, error handling, and testing plans for index adoption.
- Enhanced documentation with structured explanations for developers to implement, extend, and migrate seamlessly to the new index model.
2025-12-08 20:10:04 +01:00
Till Wegmueller
33dd228df7
Update VCS mappings to include oi-userland 2025-09-02 16:05:53 +02:00
Till Wegmueller
1198a6a934
Remove Submodule
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
2025-08-31 10:44:35 +02:00
Till Wegmueller
23815a2aab
Expand ManifestBuilder with new helper methods for setting attributes, licenses, links, and dependencies.
- Added `add_set`, `add_license`, `add_link`, and `add_depend` methods for streamlined manifest construction.
- Updated documentation with new examples showcasing the extended `ManifestBuilder` API.
- Enhanced test coverage to validate the new helper methods.
2025-08-31 00:17:15 +02:00