Commit graph

79 commits

Author SHA1 Message Date
Till Wegmueller
a2645749b1
Add action executors, action plans, and sample install script
- Implemented `executors` module for applying manifests with support for directories, files, and links.
- Added `action_plan` module to merge and execute install plans from manifests.
- Introduced `run_sample_install.sh` script for testing installations, including dry-run and real execution.
- Enhanced `pkg6` install logic to resolve and apply action plans.
- Improved manifest management with `save_manifest` and repository-based fetching.
2025-08-19 11:06:48 +02:00
Till Wegmueller
d483e2a995
implement manifest fetching and fix CI errors 2025-08-13 23:23:45 +02:00
Till Wegmueller
abf294f38c
remove legacy resolver. 2025-08-13 22:20:21 +02:00
Till Wegmueller
28a18c088a
implement dependency solver based on resolvo 2025-08-13 22:09:19 +02:00
Till Wegmueller
287b692920
fixup catalog locations and import. Add end2end test for that. (ignored by default) 2025-08-13 19:09:22 +02:00
Till Wegmueller
f2922dab11
Fix catalog import logic 2025-08-05 00:44:29 +02:00
Till Wegmueller
92cce0f767
Add debugging utilities and improved package listing
- Add support for debugging database tables with `--stats`, `--dump-all`, and `--dump-table` options.
- Introduce CLI command `DebugDb` for inspecting catalog, obsoleted, and installed tables.
- Enhance `List` command to include an `--all` flag for listing all available packages.
- Update logging for database operations and package queries.
- Add methods to dump and analyze table contents in `InstalledPackages` and `ImageCatalog`.
2025-08-05 00:20:57 +02:00
Till Wegmueller
0ec1c1928a
Update Image and Catalog functionalities with improved catalog handling and new refresh logic
- Introduced `Image::refresh_catalogs` to support refreshing catalogs selectively or fully for specified publishers.
- Enhanced `CatalogManager::load` to handle missing fields gracefully, including fallback mechanisms for `packages`.
- Added detailed logging and error diagnostics for catalog parsing and directory operations.
- Improved publisher-specific directory management, including cleaning and recreating directories during full refresh.
- Updated tests to verify correct behavior for catalog directory clearing and new catalog handling logic.
2025-08-04 23:45:41 +02:00
Till Wegmueller
f31d2e11af
Add support for partial image creation and improve logging
- Introduced the `ImageType` enum to support both `Full` and `Partial` image types during creation.
- Updated `Image::create_image` to accept `ImageType` as a parameter.
- Enhanced logging with detailed debug statements for catalog building, image queries, and package processing.
- Refactored CLI to include a new `--type` option for specifying image type during image creation.
- Simplified path determination logic in `pkg6` with a new `determine_image_path` helper function.
- Removed unused imports and unnecessary file checks for cleaner code.
2025-08-04 23:01:04 +02:00
Till Wegmueller
e27cd35d8d
Add ImageCatalog and InstalledPackages modules for package management
- Introduced `ImageCatalog` for handling catalog-related operations, including initialization, building from publishers, querying, and manifest management.
- Added `InstalledPackages` for managing installed package operations, such as adding, removing, querying, and retrieving manifests.
- Defined database schemas and error types to support both modules.
- Refactored table handling, ensuring consistent transaction scoping with `redb`.
2025-08-04 22:01:38 +02:00
Till Wegmueller
e33ccbe6ec
Remove unused ReadableTable import from libips::image module 2025-08-03 16:34:49 +02:00
Till Wegmueller
45521d0c73
Update RestBackend documentation example for download_catalog changes
- Include `ReadableRepository` and `WritableRepository` imports in the example.
- Update the `download_catalog` method to reflect the new optional parameter.
2025-08-03 14:36:36 +02:00
Till Wegmueller
2777f153c9
Introduce progress reporting infrastructure and error management refinements
- Add `ProgressReporter` trait with `start`, `update`, and `finish` methods for customizable progress tracking during operations.
- Implement `ProgressInfo` struct for detailed progress state reporting, including operation, progress, and context information.
- Create a no-op implementation `NoopProgressReporter` for cases where progress reporting is not needed.
- Enhance `pkg6` with structured error handling using `Pkg6Error` and new diagnostic variants for I/O, JSON, FMRI, and logging errors.
- Update `Image` functionality to leverage progress reporting during catalog downloads and include robust publisher management.
- Introduce `TestProgressReporter` for unit tests to validate progress reporting integration.
2025-08-03 14:28:36 +02:00
Till Wegmueller
f7f017f7b9
Add tests for Image functionality and refactor image handling logic
- Implement unit tests to cover `Image` creation, metadata management, saving/loading, and error handling.
- Add separate constructors for `Full` and `Partial` image types (`new_full`, `new_partial`).
- Introduce robust path handling methods (`metadata_dir`, `image_json_path`) for consistent metadata structure.
- Enhance error diagnostics with `InvalidPath` variant in `ImageError`.
- Update `Cargo.toml` for test module inclusion.
2025-08-02 22:12:37 +02:00
Till Wegmueller
dfe8ac1305
Add support for backward compatibility with legacy repository files
- Introduced creation of `pub.p5i` files for publishers to maintain compatibility with older repository formats.
- Implemented saving repository configuration in legacy INI format (`pkg5.repository`).
- Updated tests to validate the generation and structure of legacy files.
- Added new dependencies (`rust-ini`) and updated `Cargo.toml` and `Cargo.lock` accordingly.
2025-08-02 13:17:49 +02:00
Till Wegmueller
fc00304038
Remove redundant file checks and update test assertions for publisher-specific file paths
- Eliminate obsolete assertions for the root `file` directory as it is no longer created.
- Update tests to validate file paths in `publisher`-specific directories, aligning with the updated repository structure.
- Simplify `create_directories` by removing unnecessary `file` directory creation.
2025-07-31 00:29:53 +02:00
Till Wegmueller
99dd0fe87c
Refactor repository structure to use publisher-specific directories
- Introduce publisher-specific hierarchy (`publisher/<publisher_name>`) for catalog, package, and file storage.
- Replace hardcoded paths with helper methods (`construct_catalog_path`, `construct_package_dir`, `construct_file_path_with_publisher`) for consistent path generation.
- Remove outdated tests and simplify redundant assertions in repository tests.
- Update `CatalogManager` and file handling logic to leverage publisher-specific paths.
- Enhance debug logs to include publisher context where applicable.
2025-07-31 00:18:21 +02:00
Till Wegmueller
845ffec13b
Switch to CBOR for metadata serialization and simplify key management
- Replace `bincode` serialization with `CBOR` for obsoleted package metadata to improve interoperability and error messaging.
- Simplify key handling by directly using FMRI strings as keys, removing the need for intermediate serialization steps.
- Update deserialization, error handling, and logging to align with `CBOR` serialization.
- Add `serde_cbor` as a dependency, updating `Cargo.toml` and `Cargo.lock` accordingly.
- Refactor `ObsoletedPackageKey` to include conversion between FMRI strings and data components.
2025-07-29 23:14:47 +02:00
Till Wegmueller
a5f1b7d815
Refactor obsoleted.rs to optimize database schema and enhance error handling
- Replace `FMRI_TO_HASH_TABLE` and `HASH_TO_METADATA_TABLE` with a unified `FMRI_TO_METADATA_TABLE` for direct FMRI-to-metadata mapping, reducing complexity and eliminating intermediate hash lookups.
- Implement robust error handling and logging in database transactions, serialization, and deserialization procedures.
- Introduce on-the-fly manifest generation for entries with `NULL_HASH`, streamlining workflows for minimal manifest creation.
- Update indexing, querying, and removal logic to align with the new table and error handling improvements.
- Streamline `list_obsoleted_packages` by introducing index-based fetching with structured fallbacks to filesystem methods.
2025-07-29 21:24:32 +02:00
Till Wegmueller
5b4b719b42
Introduce cleanup functionality for obsoleted packages with TTL support
- Add methods to find and clean up obsoleted packages older than a specified TTL (`find_obsoleted_packages_older_than_ttl` and `cleanup_obsoleted_packages_older_than_ttl`) in `libips`.
- Implement a new `CleanupObsoleted` command in `pkg6repo` to handle cleanup operations.
- Update workspace `Cargo.toml` files with unified attributes for better consistency.
- Adjust dependencies (`libips` version patterns, `thiserror` updates) and enhance metadata management for obsoleted packages.
- Enhance repository operations by adding batch processing and robust logging during cleanup.
2025-07-29 19:08:18 +02:00
Till Wegmueller
7633feb36f
Add support for null hash in obsoleted package metadata
- Introduce `NULL_HASH` constant to indicate the absence of original manifest storage.
- Add `new_with_null_hash` method to create metadata for obsoleted packages without storing full manifests.
- Implement on-the-fly generation of minimal manifests with obsoletion attributes for packages with null hash.
- Update `pkg5_import` to use null hash for obsoleted packages, reducing storage requirements.
- Document null hash usage and its benefits in `obsoleted_packages.md`.
2025-07-29 18:40:34 +02:00
Till Wegmueller
9b2f74c5c1
Introduce obsoleted package management system in IPS
- Add `obsoleted.rs` module to handle storing, metadata management, and operations for obsoleted packages.
- Implement commands for marking, listing, searching, restoring, exporting, and importing obsoleted packages (`pkg6repo`).
- Enhance `RepositoryError` with `From` implementations for various error types to manage database and serialization-related errors.
- Introduce reusable data structures for obsoleted package metadata and export representation.
- Update `Cargo.toml` and `Cargo.lock` to include new dependencies (`redb`, `bincode`, etc.).
- Document obsoleted package workflow and integration details in `doc/obsoleted_packages.md` for contributors.
- Refactor repository internals to integrate obsoleted package support without disrupting existing workflow.
- Add robust error handling, logging, and pagination for enhanced usability and scalability.
2025-07-29 16:16:12 +02:00
Till Wegmueller
88b55c4a70
Add serde attribute defaults and improve JSON deserialization handling in manifests
- Introduce `#[serde(skip_serializing_if = "is_empty", default)]` for various optional fields to streamline serialization and ensure defaults are applied during deserialization.
- Add `tracing::debug` logging for enhanced error context in JSON deserialization fallback logic.
- Update tests to reflect changes in manifest parsing, including new cases for the updated JSON format.
2025-07-29 11:38:36 +02:00
Till Wegmueller
81eb4a7447
Use walkdir for efficient recursive traversal in FileBackend::build_search_index
- Replace manual recursive directory scanning with `walkdir` for simplified and efficient file traversal.
- Update dependencies in `Cargo.toml` and `Cargo.lock` to include `walkdir`.
- Enhance code maintainability by replacing redundant logic with streamlined traversal and processing.
2025-07-27 19:48:52 +02:00
Till Wegmueller
87eea546aa
Remove obsolete test_manifest_parsing.rs and refactor FileBackend for robust manifest handling
- Eliminate redundant `test_manifest_parsing.rs` as its functionality is covered during `FileBackend` implementation.
- Define `PackageContentVectors` for streamlined data organization.
- Enhance manifest handling in `FileBackend` with recursive search, improved content extraction, and robust error handling.
- Simplify and centralize manifest parsing logic for maintainability and code clarity.
2025-07-27 16:12:59 +02:00
Till Wegmueller
21de26ae82
Refactor codebase to improve formatting, logging clarity, and error handling
- Standardize formatting by aligning and cleaning up indentation, spacing, and line breaks across multiple modules.
- Enhance logging with improved message formatting for better context and readability.
- Simplify error handling in `pkg6dev`, `pkg6repo`, and core libraries by consolidating redundant patterns and improving clarity.
- Update test cases to reflect formatting and logging changes while extending coverage of edge cases.
- Perform general cleanup, including removing unused imports, refining comments, and ensuring consistent style.
2025-07-27 15:22:49 +02:00
Till Wegmueller
abb997df43
Refactor FileBackend for improved manifest handling and directory structure
- Extract manifest path construction and recursive manifest search into helper methods (`construct_manifest_path` and `find_manifests_recursive`).
- Enhance package information extraction by separating stem and version details from FMRI.
- Adjust directory creation logic to include per-package subdirectories for better organization.
- Simplify and centralize manifest parsing logic, improving error handling and maintainability.
2025-07-27 15:20:45 +02:00
Till Wegmueller
f4d1ce73b6
Refactor repository operations to use immutable references
- Modify `rebuild` and `refresh` methods to take `&self` instead of `&mut self`.
- Replace mutable repository instances in `pkg6dev`, `pkg6repo`, and core library with immutable ones.
- Introduce `debug` logging for enhanced traceability during metadata rebuild and refresh.
- Extract and simplify catalog rebuild logic into a dedicated `rebuild_catalog` method.
- General cleanup: remove redundant methods, reorganize logic, and refine comments across affected files.
2025-07-27 13:43:56 +02:00
Till Wegmueller
7f4ecf6346
Organize repository files into hierarchical directory structure for improved scalability
- Update `FileBackend` to store files in a multi-level directory structure (`xx/yy/hash`) based on hash prefixes.
- Adjust method signatures and logic across `FileBackend`, `pkg6dev`, and `pkg6repo` to support mutable repository operations.
- Add test cases and scripts (`test_file_structure.rs`, `test_repo_operations.sh`) to validate the new file structure.
- Refactor catalog manager to use `RefCell` for interior mutability to enable lazy initialization.
- Clean up redundant and dead code, enhance comments, and replace outdated logic for file handling and metadata operations.
2025-07-27 13:15:52 +02:00
Till Wegmueller
633e742528
Remove backward compatibility logic for root pkg directory manifest copy in file_backend.rs 2025-07-27 12:26:57 +02:00
Till Wegmueller
d00de23d52
Simplify pkg5_import.rs by removing redundant directory creation logic; delegate to Transaction.commit() for handling. 2025-07-27 12:19:38 +02:00
Till Wegmueller
9b271d81b5
Add support for User, Group, Driver, Legacy, and Transform actions in Manifest parsing
- Implement parsing and mapping of new action types: `User`, `Group`, `Driver`, `Legacy`, and `Transform` within `Manifest`.
- Introduce comprehensive test coverage for the newly supported actions, including edge cases for `ftpuser` property in manifests.
- Refactor to leverage `HashSet` for user `services` to avoid duplicates.
- Improve comments and error handling during action parsing for enhanced clarity and robustness.
2025-07-27 11:30:24 +02:00
Till Wegmueller
ad153a65ab
Replace println! statements with structured logging using the tracing crate, refactor error handling to use error! macros for logging, improve comments for clarity, and simplify type usage by utilizing HashMap and HashSet from the prelude. 2025-07-26 23:41:09 +02:00
Till Wegmueller
cb75c045e5
Introduce Pkg6DevError for improved error handling across pkg6dev, replace anyhow with custom error types, enable structured diagnostics with miette, update dependencies, and add tracing for logging. 2025-07-26 21:20:50 +02:00
Till Wegmueller
b451099b54
Refactor tests and stubs: update parsed_manifest assertions with more descriptive values and simplify unused variable patterns in repository rest backend methods. 2025-07-26 18:04:58 +02:00
Till Wegmueller
4649608408
Replace anyhow with custom RepositoryError for improved error specificity and consistency. Remove anyhow dependency. 2025-07-26 16:50:10 +02:00
Till Wegmueller
a33a3246b6
chore(fmt): format with cargo fmt 2025-07-26 12:54:01 +02:00
Till Wegmueller
a5cdb7e673
Add JSON and TSV output to repository commands, integrate serde for struct serialization, and introduce search subcommand with flexible filtering options. 2025-07-26 12:35:44 +02:00
Till Wegmueller
307dfddc7c
Remove TESTING.md, fix CLI argument naming (--repo-version), enhance transaction publisher handling, and improve manifest storage logic. 2025-07-26 11:52:42 +02:00
Till Wegmueller
4f0e2fe066
Add unit and end-to-end tests for repository functionality, including setup scripts, package management operations, and TESTING.md documentation. 2025-07-26 10:52:25 +02:00
Till Wegmueller
ead9ad12af
Refactor repository traits into ReadableRepository and WritableRepository, introduce Repository to combine traits for backward compatibility, and update relevant modules and tests to use the new structure. 2025-07-26 10:34:45 +02:00
Till Wegmueller
a0fcc13033
Introduce Catalog module with structs and functionality for managing package metadata, enhance file_backend and CLI to handle catalog operations, and update dependencies (chrono and others) to support new features. 2025-07-24 00:28:33 +02:00
Till Wegmueller
63f2d1da62
Add search method to repository module for querying packages, update dependencies to support search functionality. 2025-07-23 23:22:24 +02:00
Till Wegmueller
2cb63fbef0
Introduce PackageContents struct for structured representation of package data, refactor show_contents for multiple content types, and enhance handling of manifests with improved filtering and output formatting. 2025-07-23 22:39:49 +02:00
Till Wegmueller
ca7ce75008
Enhance show_contents to parse manifests directly, add filtering by publishers, patterns, and action types, and handle multiple content types (files, directories, links, dependencies, licenses). 2025-07-22 19:39:46 +02:00
Till Wegmueller
5d987ca0cb
Refactor package FMRI handling by introducing Fmri struct, update dependencies, and adjust repository and action modules for structured FMRI data processing
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
2025-07-22 14:10:37 +02:00
Till Wegmueller
e3662eaf23
Introduce PackageInfo struct for structured package data, refactor list_packages to use it, and enhance publisher directory handling
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
2025-07-22 11:57:24 +02:00
Till Wegmueller
7332e0f7b5
Refactor repository info retrieval to return structured data, add detailed publisher information, and update tests accordingly
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
2025-07-22 10:21:16 +02:00
Till Wegmueller
a9584fa6d2
Add package publishing functionality and default publisher management
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
2025-07-21 23:20:19 +02:00
Till Wegmueller
8a32bf3176
Add compression support (Gzip/LZ4) and update dependencies
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
2025-07-21 22:37:08 +02:00