Commit graph

105 commits

Author SHA1 Message Date
Till Wegmueller
b080288114
Add legacy manifest handling with JSON fallback for transaction commits
- Introduced `legacy_manifest_content` field in `Transaction` to allow preserving byte-identical legacy manifests.
- Updated `commit` method to save both JSON and legacy manifests, falling back to JSON if no legacy content is provided.
- Enhanced manifest copy logic to handle both JSON and legacy formats during transaction finalization.
- Improved catalog rebuild to skip `.json` files when a corresponding legacy manifest exists.
2025-12-23 14:09:14 +01:00
Till Wegmueller
7dc475ed2d
Fix FMRI and catalog processing inconsistencies
- Corrected FMRI formatting to `pkg:/name` for entries without a publisher.
- Excluded `pkg.fmri` from summary action extraction in file backend.
- Removed unused signature handling in catalog package processing.
- Reordered `PackageVersionEntry` struct fields for logical consistency.
2025-12-23 13:19:51 +01:00
Till Wegmueller
b32ace705f
Refactor whitespace handling in JSON serialization and improve ISO-8601 timestamp formatting
- Removed unnecessary spaces in JSON key-value and array formatting for Python-style compatibility.
- Enhanced `format_iso8601_basic` to ensure precise microsecond-level time formatting.
- Updated `write_update_log` to return and store SHA-1 signatures for update log verification.
2025-12-23 12:33:12 +01:00
Till Wegmueller
ff0b9f4319
Add support for file URL without algorithm and refactor JSON serialization for Python-style compatibility
- Added `/file/1/{digest}` route to support file retrieval without specifying the algorithm.
- Implemented a Python-compatible JSON serializer to ensure consistent formatting for catalog artifacts.
- Replaced `HashMap` with `BTreeMap` for deterministic ordering in catalog serialization and updates.
- Updated integration tests to validate the new route functionality and ensure response correctness.
- Refactored `format_iso8601_basic` to improve timestamp formatting consistency.
2025-12-22 22:42:56 +01:00
Till Wegmueller
a921c99eb6
Reorder struct fields for consistency and align with Rust formatting standards. 2025-12-22 20:11:08 +01:00
Till Wegmueller
d2d1c297cc
Refactor to align with Rust formatting guidelines and enhance code readability.
- Adjusted indentation and line breaks for structs, functions, and method calls to comply with Rust formatting standards.
- Improved error message formatting and consistency across `PkgTreeError` instances.
- Restructured long function arguments and chained calls for clarity and maintainability.
- Simplified conditional statements and loops for better readability.
- No functional changes introduced.
2025-12-22 20:10:17 +01:00
Till Wegmueller
e87d1a3166
Add batched catalog rebuild support and INI-only repo test
- Introduced `BatchOptions` for configurable batched catalog rebuilds.
- Added `rebuild_catalog_batched` in `file_backend` for batched processing with deterministic package order and progress tracking.
- Updated `rebuild_catalog` to leverage the new batched implementation with default options.
- Added integration test `test_ini_only_repo_serving_catalog` to validate functionality with an INI-only repository configuration.
- Enhanced test coverage for legacy catalog handling via `pkg5.repository`.
2025-12-09 14:23:55 +01:00
Till Wegmueller
bd67e06012
Add catalog_writer module for atomic catalog updates with SHA-1 signatures
- Introduced `write_catalog_attrs`, `write_catalog_part`, and `write_update_log` functions for streamlined and secure file writing.
- Refactored `file_backend` to use `catalog_writer` for managing catalog updates, improving readability and maintainability.
- Updated `save_catalog_attrs`, `save_catalog_part`, and `append_update` to leverage atomic writes and ensure signature computation.
- Replaced manual serialization logic with centralized writing utilities for consistency and error resilience.
- Updated dependencies for JSON handling and signature computation.
2025-12-09 12:49:25 +01:00
Till Wegmueller
a948f87e6f
Add legacy repository support and SHA-1 signature handling
- Introduced fallback for legacy `pkg5.repository` configuration in INI format alongside the existing `pkg6.repository` JSON format.
- Enabled SHA-1 signature computation for compatibility with legacy catalog signatures.
- Added methods to save update logs in legacy format and enhance catalog compatibility.
- Updated dependencies to include `sha1` for hashing.
2025-12-09 12:12:57 +01:00
Till Wegmueller
c4910bb434
Refactor error handling in repository module to include structured error variants with detailed context.
- Replaced string-based `FileReadError`, `FileWriteError`, and `DirectoryCreateError` with structured error types using `PathBuf` and `source`.
- Updated all affected methods in `rest_backend`, `file_backend`, and tests to use the new structured error variants.
- Improved error messages for clarity and debugging by embedding file paths and sources directly into error data.
2025-12-08 23:13:27 +01:00
Till Wegmueller
81bc7b8574
Refactor catalog handling by replacing get_legacy_catalog with get_catalog_file_path for improved clarity and consistency.
- Removed now-redundant catalog path resolution logic in `repo.rs`.
- Updated `file_backend` with a unified method for fetching catalog file paths, addressing invalid path prefixes.
- Adjusted HTTP handler to use the updated method.
2025-12-08 22:45:39 +01:00
Till Wegmueller
55decc16ff
Simplify CatalogPart::load by removing manual JSON parsing fallbacks and introducing test coverage for sample catalogs. 2025-12-08 22:39:28 +01:00
Till Wegmueller
84b2c50ed6
Remove deprecated Catalog v0 support and refactor legacy catalog handling.
- Removed `get_catalog` handler and associated Catalog v0 route.
- Updated `get_legacy_catalog` to accept `filename` for enhanced flexibility.
- Adjusted integration tests to validate changes and focus on Catalog v1 support.
- Refactored `SupportedOperation` definitions to exclude Catalog v0.
2025-12-08 22:10:11 +01:00
Till Wegmueller
0b3a974ca6
Add REST API v1 endpoints and legacy catalog handling for pkg6depotd
- Expanded repository structure by introducing methods for fetching legacy catalogs, catalog file paths, and repository info.
- Added new REST API v1 endpoints for catalog, manifest, file, and publisher handling.
- Implemented `publisher` handler module with `get_publisher_v0` and `get_publisher_v1` methods to retrieve publisher details in pkg5 format.
- Updated `integration_tests` to validate new endpoints and ensure compatibility with legacy and modern catalog/manifest handling.
- Removed unused dependency `walkdir` and refactored test cases for clarity and efficiency.
2025-12-08 21:36:37 +01:00
Till Wegmueller
23815a2aab
Expand ManifestBuilder with new helper methods for setting attributes, licenses, links, and dependencies.
- Added `add_set`, `add_license`, `add_link`, and `add_depend` methods for streamlined manifest construction.
- Updated documentation with new examples showcasing the extended `ManifestBuilder` API.
- Enhanced test coverage to validate the new helper methods.
2025-08-31 00:17:15 +02:00
Till Wegmueller
d78cd9f659
Refactor depend module
- Simplified `split_dir_base` signature by removing unnecessary lifetime annotation.
- Refactored `compute_python_runpaths` to eliminate mutable variable and streamline `insert_default_runpath`.
2025-08-30 23:26:17 +02:00
Till Wegmueller
29ef35f350
Add api module for forge/pkgdev high-level integration
- Introduced `api.rs` to provide a stable, struct-first API surface for building, linting, resolving, and publishing IPS packages.
- Encapsulated existing functionality for better abstraction and usability in external integrations.
- Enhanced `libips` with a high-level repository and transaction interface.
- Added support for dependency generation, manifest transformations, and linting via configurable rulesets.
- Updated documentation with integration flow examples and usage guidelines.
2025-08-30 22:25:45 +02:00
Till Wegmueller
77f02fdfbd
Add depend module for file-level dependency generation
- Introduced `depend.rs` to handle dependency generation for ELF files, scripts, Python modules, and SMF manifests.
- Implemented file classification and analysis logic with configurable bypass rules and runpath handling.
- Added utility functions to resolve file dependencies into manifest actions using a provided repository.
- Updated `Cargo.toml` with `goblin` dependency for ELF processing.
- Enhanced codebase with default runpath insertion, dynamic token expansion, and Python module import detection.
- Included `pkgdepend` documentation for dependency resolution overview.
2025-08-30 18:35:41 +02:00
Till Wegmueller
7cffa6c4e6
Add transformer module for manifest transformation logic
- Introduced `transformer.rs` with a structured approach for parsing and applying transformation rules.
- Added support for operations like `add`, `default`, `delete`, `drop`, `edit`, `emit`, and `set` on attributes, files, directories, and other targets.
- Implemented regex-based matching for patterns and backreference handling in transformations.
- Enhanced manifest modification functionality, including attribute/facet operations and deferred action emission.
- Added comprehensive unit tests to validate transformation rules and their applications.
2025-08-28 23:50:59 +02:00
Till Wegmueller
88e06b488d
Refactor catalog processing to merge parts by version
- Introduced `process_publisher_merged` to consolidate catalog parts per `stem@version` before processing.
- Replaced part-wise processing with merged version logic for efficiency and improved determinism.
- Added `process_catalog_part` dead code annotation and cleaned up outdated logic.
- Enhanced handling of merged actions and signature deduplication across catalog parts.
- Improved logging for import and obsolete package metrics.
2025-08-27 00:05:36 +02:00
Till Wegmueller
6ed8bb5b5b
Improve dependency constraint extraction and branch locking logic
- Updated `extract_constraint` to derive constraints from both optional properties and dependency FMRI version strings.
- Ensured child dependencies inherit parent's branch when not explicitly constrained, mirroring solver behavior.
2025-08-26 21:51:05 +02:00
Till Wegmueller
d77e61f90f
Add solver advice system for user-focused installation guidance
- Introduced a new `advice` module in the solver to provide user-oriented guidance for resolving dependency issues.
- Added structured failure details to `SolverError` for improved diagnosis and advice generation.
- Updated `pkg6` to display actionable advice when dependency resolution fails, including catalog refresh, publisher verification, and incorporation constraints.
- Enhanced CLI tools with a dangling dependency scan to identify and output unresolved stems.
- Improved modularity and caching mechanisms for catalog and manifest handling to support the advice system.
- Refined error messages and logging for better user feedback during installation errors.
2025-08-26 21:09:06 +02:00
Till Wegmueller
9ac8f98b38
Refactor solver and manifest handling
- Replaced `CatalogProvider` with database-backed solution, improving manifest retrieval logic.
- Added fallback and LZ4 decoding support for catalog-stored manifests.
- Enhanced incorporation lock handling with direct database queries.
- Updated sample install script to use `debug` logging for better traceability.
2025-08-21 23:52:11 +02:00
Till Wegmueller
e4bd9a748a
Refactor ImageCatalog and apply manifest logic
- Added support for obsoleted package database with separate handling in `ImageCatalog`.
- Enhanced apply manifest functionality with progress callback support and processing statistics.
- Introduced LZ4 compression for manifest storage in `ImageCatalog`.
- Removed debugging eprintln statements, replaced with structured logging.
- Updated `pkg6` image creation and installation logic to improve user feedback and error handling.
- Updated database initialization and build processes to handle new obsoleted logic.
2025-08-19 22:43:50 +02:00
Till Wegmueller
77147999b3
Add incorporation lock support and update dependencies across the project
- Implemented incorporation lock functionality for packages, including pre-checks, addition, and retrieval methods.
- Enhanced package installation logic to handle incorporation constraints.
- Updated dependency versions in `Cargo.toml` and `Cargo.lock`, aligning with latest compatible releases.
- Introduced new tests to verify incorporation lock behavior, including scenarios with and without locks.
- Refactored code for improved readability and added error handling improvements.
- Fixed minor typos and inconsistencies in comments and log messages.
2025-08-19 14:30:55 +02:00
Till Wegmueller
39124f9df4
Refactor imports and remove unused ActionOrder implementation
- Cleaned up imports across multiple modules for better organization and removed unused dependencies.
- Eliminated `ActionOrder` implementation from `executors` as it's no longer being utilized.
- Simplified error handling in `get_dependencies` using `unwrap_or_else`.
- Optimized test setups with consistent use of `tempdir` helper function.
2025-08-19 11:10:36 +02:00
Till Wegmueller
a2645749b1
Add action executors, action plans, and sample install script
- Implemented `executors` module for applying manifests with support for directories, files, and links.
- Added `action_plan` module to merge and execute install plans from manifests.
- Introduced `run_sample_install.sh` script for testing installations, including dry-run and real execution.
- Enhanced `pkg6` install logic to resolve and apply action plans.
- Improved manifest management with `save_manifest` and repository-based fetching.
2025-08-19 11:06:48 +02:00
Till Wegmueller
d483e2a995
implement manifest fetching and fix CI errors 2025-08-13 23:23:45 +02:00
Till Wegmueller
abf294f38c
remove legacy resolver. 2025-08-13 22:20:21 +02:00
Till Wegmueller
28a18c088a
implement dependency solver based on resolvo 2025-08-13 22:09:19 +02:00
Till Wegmueller
287b692920
fixup catalog locations and import. Add end2end test for that. (ignored by default) 2025-08-13 19:09:22 +02:00
Till Wegmueller
f2922dab11
Fix catalog import logic 2025-08-05 00:44:29 +02:00
Till Wegmueller
92cce0f767
Add debugging utilities and improved package listing
- Add support for debugging database tables with `--stats`, `--dump-all`, and `--dump-table` options.
- Introduce CLI command `DebugDb` for inspecting catalog, obsoleted, and installed tables.
- Enhance `List` command to include an `--all` flag for listing all available packages.
- Update logging for database operations and package queries.
- Add methods to dump and analyze table contents in `InstalledPackages` and `ImageCatalog`.
2025-08-05 00:20:57 +02:00
Till Wegmueller
0ec1c1928a
Update Image and Catalog functionalities with improved catalog handling and new refresh logic
- Introduced `Image::refresh_catalogs` to support refreshing catalogs selectively or fully for specified publishers.
- Enhanced `CatalogManager::load` to handle missing fields gracefully, including fallback mechanisms for `packages`.
- Added detailed logging and error diagnostics for catalog parsing and directory operations.
- Improved publisher-specific directory management, including cleaning and recreating directories during full refresh.
- Updated tests to verify correct behavior for catalog directory clearing and new catalog handling logic.
2025-08-04 23:45:41 +02:00
Till Wegmueller
f31d2e11af
Add support for partial image creation and improve logging
- Introduced the `ImageType` enum to support both `Full` and `Partial` image types during creation.
- Updated `Image::create_image` to accept `ImageType` as a parameter.
- Enhanced logging with detailed debug statements for catalog building, image queries, and package processing.
- Refactored CLI to include a new `--type` option for specifying image type during image creation.
- Simplified path determination logic in `pkg6` with a new `determine_image_path` helper function.
- Removed unused imports and unnecessary file checks for cleaner code.
2025-08-04 23:01:04 +02:00
Till Wegmueller
e27cd35d8d
Add ImageCatalog and InstalledPackages modules for package management
- Introduced `ImageCatalog` for handling catalog-related operations, including initialization, building from publishers, querying, and manifest management.
- Added `InstalledPackages` for managing installed package operations, such as adding, removing, querying, and retrieving manifests.
- Defined database schemas and error types to support both modules.
- Refactored table handling, ensuring consistent transaction scoping with `redb`.
2025-08-04 22:01:38 +02:00
Till Wegmueller
e33ccbe6ec
Remove unused ReadableTable import from libips::image module 2025-08-03 16:34:49 +02:00
Till Wegmueller
45521d0c73
Update RestBackend documentation example for download_catalog changes
- Include `ReadableRepository` and `WritableRepository` imports in the example.
- Update the `download_catalog` method to reflect the new optional parameter.
2025-08-03 14:36:36 +02:00
Till Wegmueller
2777f153c9
Introduce progress reporting infrastructure and error management refinements
- Add `ProgressReporter` trait with `start`, `update`, and `finish` methods for customizable progress tracking during operations.
- Implement `ProgressInfo` struct for detailed progress state reporting, including operation, progress, and context information.
- Create a no-op implementation `NoopProgressReporter` for cases where progress reporting is not needed.
- Enhance `pkg6` with structured error handling using `Pkg6Error` and new diagnostic variants for I/O, JSON, FMRI, and logging errors.
- Update `Image` functionality to leverage progress reporting during catalog downloads and include robust publisher management.
- Introduce `TestProgressReporter` for unit tests to validate progress reporting integration.
2025-08-03 14:28:36 +02:00
Till Wegmueller
f7f017f7b9
Add tests for Image functionality and refactor image handling logic
- Implement unit tests to cover `Image` creation, metadata management, saving/loading, and error handling.
- Add separate constructors for `Full` and `Partial` image types (`new_full`, `new_partial`).
- Introduce robust path handling methods (`metadata_dir`, `image_json_path`) for consistent metadata structure.
- Enhance error diagnostics with `InvalidPath` variant in `ImageError`.
- Update `Cargo.toml` for test module inclusion.
2025-08-02 22:12:37 +02:00
Till Wegmueller
dfe8ac1305
Add support for backward compatibility with legacy repository files
- Introduced creation of `pub.p5i` files for publishers to maintain compatibility with older repository formats.
- Implemented saving repository configuration in legacy INI format (`pkg5.repository`).
- Updated tests to validate the generation and structure of legacy files.
- Added new dependencies (`rust-ini`) and updated `Cargo.toml` and `Cargo.lock` accordingly.
2025-08-02 13:17:49 +02:00
Till Wegmueller
fc00304038
Remove redundant file checks and update test assertions for publisher-specific file paths
- Eliminate obsolete assertions for the root `file` directory as it is no longer created.
- Update tests to validate file paths in `publisher`-specific directories, aligning with the updated repository structure.
- Simplify `create_directories` by removing unnecessary `file` directory creation.
2025-07-31 00:29:53 +02:00
Till Wegmueller
99dd0fe87c
Refactor repository structure to use publisher-specific directories
- Introduce publisher-specific hierarchy (`publisher/<publisher_name>`) for catalog, package, and file storage.
- Replace hardcoded paths with helper methods (`construct_catalog_path`, `construct_package_dir`, `construct_file_path_with_publisher`) for consistent path generation.
- Remove outdated tests and simplify redundant assertions in repository tests.
- Update `CatalogManager` and file handling logic to leverage publisher-specific paths.
- Enhance debug logs to include publisher context where applicable.
2025-07-31 00:18:21 +02:00
Till Wegmueller
845ffec13b
Switch to CBOR for metadata serialization and simplify key management
- Replace `bincode` serialization with `CBOR` for obsoleted package metadata to improve interoperability and error messaging.
- Simplify key handling by directly using FMRI strings as keys, removing the need for intermediate serialization steps.
- Update deserialization, error handling, and logging to align with `CBOR` serialization.
- Add `serde_cbor` as a dependency, updating `Cargo.toml` and `Cargo.lock` accordingly.
- Refactor `ObsoletedPackageKey` to include conversion between FMRI strings and data components.
2025-07-29 23:14:47 +02:00
Till Wegmueller
a5f1b7d815
Refactor obsoleted.rs to optimize database schema and enhance error handling
- Replace `FMRI_TO_HASH_TABLE` and `HASH_TO_METADATA_TABLE` with a unified `FMRI_TO_METADATA_TABLE` for direct FMRI-to-metadata mapping, reducing complexity and eliminating intermediate hash lookups.
- Implement robust error handling and logging in database transactions, serialization, and deserialization procedures.
- Introduce on-the-fly manifest generation for entries with `NULL_HASH`, streamlining workflows for minimal manifest creation.
- Update indexing, querying, and removal logic to align with the new table and error handling improvements.
- Streamline `list_obsoleted_packages` by introducing index-based fetching with structured fallbacks to filesystem methods.
2025-07-29 21:24:32 +02:00
Till Wegmueller
5b4b719b42
Introduce cleanup functionality for obsoleted packages with TTL support
- Add methods to find and clean up obsoleted packages older than a specified TTL (`find_obsoleted_packages_older_than_ttl` and `cleanup_obsoleted_packages_older_than_ttl`) in `libips`.
- Implement a new `CleanupObsoleted` command in `pkg6repo` to handle cleanup operations.
- Update workspace `Cargo.toml` files with unified attributes for better consistency.
- Adjust dependencies (`libips` version patterns, `thiserror` updates) and enhance metadata management for obsoleted packages.
- Enhance repository operations by adding batch processing and robust logging during cleanup.
2025-07-29 19:08:18 +02:00
Till Wegmueller
7633feb36f
Add support for null hash in obsoleted package metadata
- Introduce `NULL_HASH` constant to indicate the absence of original manifest storage.
- Add `new_with_null_hash` method to create metadata for obsoleted packages without storing full manifests.
- Implement on-the-fly generation of minimal manifests with obsoletion attributes for packages with null hash.
- Update `pkg5_import` to use null hash for obsoleted packages, reducing storage requirements.
- Document null hash usage and its benefits in `obsoleted_packages.md`.
2025-07-29 18:40:34 +02:00
Till Wegmueller
9b2f74c5c1
Introduce obsoleted package management system in IPS
- Add `obsoleted.rs` module to handle storing, metadata management, and operations for obsoleted packages.
- Implement commands for marking, listing, searching, restoring, exporting, and importing obsoleted packages (`pkg6repo`).
- Enhance `RepositoryError` with `From` implementations for various error types to manage database and serialization-related errors.
- Introduce reusable data structures for obsoleted package metadata and export representation.
- Update `Cargo.toml` and `Cargo.lock` to include new dependencies (`redb`, `bincode`, etc.).
- Document obsoleted package workflow and integration details in `doc/obsoleted_packages.md` for contributors.
- Refactor repository internals to integrate obsoleted package support without disrupting existing workflow.
- Add robust error handling, logging, and pagination for enhanced usability and scalability.
2025-07-29 16:16:12 +02:00
Till Wegmueller
88b55c4a70
Add serde attribute defaults and improve JSON deserialization handling in manifests
- Introduce `#[serde(skip_serializing_if = "is_empty", default)]` for various optional fields to streamline serialization and ensure defaults are applied during deserialization.
- Add `tracing::debug` logging for enhanced error context in JSON deserialization fallback logic.
- Update tests to reflect changes in manifest parsing, including new cases for the updated JSON format.
2025-07-29 11:38:36 +02:00
Till Wegmueller
81eb4a7447
Use walkdir for efficient recursive traversal in FileBackend::build_search_index
- Replace manual recursive directory scanning with `walkdir` for simplified and efficient file traversal.
- Update dependencies in `Cargo.toml` and `Cargo.lock` to include `walkdir`.
- Enhance code maintainability by replacing redundant logic with streamlined traversal and processing.
2025-07-27 19:48:52 +02:00