pkg DEPOT 1. Summary This document describes the architecture of a pkg(5) depot server as implemented in pkg.depotd(1m). This includes: an overview of the depot server's filesystem layout, operations provided by the depot server, the interfaces provided by the depot server, and a description of how clients communicate with the depot server for each operation. NOTE: The image packaging system is under development. Changes to interfaces may occur as part of the architectural review process, as shortcomings are identified, and as new features are introduced. Questions about planned or possible change should be asked on pkg-discuss. 2. Discussion A pkg(5) depot server provides a way for clients to interact with package content and metadata contained within a pkg(5) repository, and with the depot server itself. It accomplishes this by providing an HTTP-based interface suitable for usage by pkg(1) clients and web user agents. 2.1. Filesystem Layout The types of information that the depot server stores and/or retrieves can be categorized as follows: - depot data This includes: configuration data, presentation content (such as web page templates), publishing data (e.g. in-flight transactions), and temporary data (e.g. the feed cache). - repository data This includes: catalog information, package content (files), package metadata (manifests), and search data. 2.1.1. Layout The depot server uses the following 'root' directory structures for the storage and retrieval of depot and repository data: - repo_dir (depot and repository data) cfg_cache (depot data) A file containing the cached configuration information for the depot server. catalog/ (repository data) This directory contains the repository catalog and its related metadata. file/ (repository data) This directory contains the file content of packages in the repository. Files are stored using a two-level path fragment, derived from the SHA1-hash of a file's content, assumed to have at least 8 distinct characters. Example: 00/ 0023bb/ 000023bb53fdc7bcf35e62b7b0b353a56d36a504 index/ (repository data) This directory contains the search indices for the repository. pkg/ (repository data) This directory contains the metadata (manifests) for the repository's packages. The manifests for each package are stored in a directory with the same name as the package stem using a URL-encoded filename. Example: entire/ 0.5.11%2C5.11-0.86%3A20080422T234219Z trans/ (depot data) This directory contains in-flight transactions for packages that are waiting for the publication process to complete so that they can be added to the repository's catalog. Each transaction is stored in a directory named after the pending transaction id and contains the manifest waiting for publication to finish stored with the filename of 'manifest'. Example: 1229379580_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081215T221940Z/ manifest updatelog/ (repository data) This directory contains metadata detailing changes to the repository by publishing operations. - content_root (depot data) web/ This directory contains all of the web presentation content for the depot. 2.2. Operations When communicating with the depot server via HTTP, operations are presented via a URL-based mechanism that allows each to be versioned so that changes in protocol can be hidden from older clients as the interfaces to operations evolve. Operations made available by a pkg.depotd(5) server can be accessed via GET or POST, as appropriate for each operation, via a URL such as the following: http://pkg.opensolaris.org/release/manifest/0/SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z The above example can be broken down into four basic components: publisher_origin_url - http://pkg.opensolaris.org/release/ operation_name - manifest protocol_version - 0 operation_arguments - SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z Each of these components can be described as follows: publisher_origin_url - A URL that can be used to access a depot server's repository. operation_name - The name of the operation that the client is wanting to initiate. protocol_version - An integer value representing the version of the operation's protocol spoken by the client. operation_arguments - String data (such as a package FMRI) that is parsed and then used to determine what resource(s) will be used to perform an operation. Some operations expect arguments or data to be passed via POST-based form data, headers, or the request body instead. 2.2.1. Operation Types Each operation that the depot server provides is either designed to interact with a pkg(5) repository, or with the depot server itself. These operations can be categorized as follows: - content These operations are read-only, and retrieve file data that comprises the content of a package in a repository. - depot These operations are read-only, and permit retrieval of: the list of operations that the depot server currently provides (including protocol version and pkg(5) software version), statistics information, and other depot information. - metadata These operations are read-only, and retrieve metadata related to a package FMRI, such as its name, version, etc. stored in a repository's catalog. - publishing These operations alter a repository's catalog, package metadata, and allow storage of package content. - administrative These operations perform administrative tasks on the repository or depot server. 2.2.2. Modes Which types of operations are available is dependent on which mode the depot server is currently operating in: - default In default mode, the depot server allows content, depot, metadata, and publishing operations. - readonly In readonly mode, the depot server allows content, depot, and metadata operations. - mirror In mirror mode, the depot server allows content and depot operations. 2.2.3. Content Operations The pkg.depotd(5) server provides the following operations for retrieving package content: - file Version 0: A GET operation that retrieves the contents of a file, belonging to a package, using a SHA-1 hash of the file's content. Example: URL: http://pkg.opensolaris.org/release/file/0/ a00030db8b91f85d0b7144d0d4ef241a3f1ae28f Expects: A SHA-1 hash of the file's content belonging to a package in the request path. Returns: The contents of the file, compressed using the gzip compression algorithm. Version 1: A GET or POST/PUT operation that retrieves or uploads the contents of a file. For GET requests, it behaves the same as Version 0. For POST/PUT requests, it allows uploading file content to the repository. Example: URL: http://pkg.opensolaris.org/release/file/1/ a00030db8b91f85d0b7144d0d4ef241a3f1ae28f Expects: For GET: A SHA-1 hash of the file's content belonging to a package in the request path. For POST/PUT: The file content in the request body. Returns: For GET: The contents of the file, compressed using the gzip compression algorithm. For POST/PUT: Response status of 200 on success; any other status indicates failure. Version 2: Extends Version 1 to add support for HEAD requests. For GET and POST/PUT requests, it behaves the same as Version 1. For HEAD requests, it returns metadata about the file without returning the file content. Example: URL: http://pkg.opensolaris.org/release/file/2/ a00030db8b91f85d0b7144d0d4ef241a3f1ae28f Expects: For GET/POST/PUT: Same as Version 1. For HEAD: A SHA-1 hash of the file's content belonging to a package in the request path. Returns: For GET/POST/PUT: Same as Version 1. For HEAD: Response headers containing file metadata including compressed attributes in X-Ipkg-Attr-{n} headers. 2.2.4. Depot Operations - versions Version 0: A GET operation that retrieves text data representing what operations are supported and version information about the depot server. Example: URL: http://pkg.opensolaris.org/versions/0/ Expects: Nothing Returns: text/plain data containing the version of the pkg(5) software that the depot is based upon, a list of the operations currently supported, and the protocol version supported for each operation. Sample Output: pkg-server bfc04991436e info 0 search 0 versions 0 catalog 0 manifest 0 add 0 file 0 abandon 0 close 0 open 0 - status Version 0: A GET operation that retrieves a JSON formatted dictionary containing statistics information for the repository being served. Example: URL: http://pkg.opensolaris.org/status/0/ Expects: Nothing Returns: A JSON formatted dictionary containing statistics information for the repository being served. 2.2.5. Meta-data Operations - catalog Version 0: A GET operation that retrieves a text/plain datastream representing a complete catalog or an incremental update to an existing one as requested. Example: URL: http://pkg.opensolaris.org/catalog/0/ Expects: Nothing or the following headers: If-Modified-Since: {ISO 8601 formatted date and time in UTC} Returns: Either the contents of a pkg(5) catalog file, or the entries that were added since the specified date as they are found in the catalog file, separated by newlines. Version 1: A GET operation that retrieves the contents of a specific catalog file by name. Example: URL: http://pkg.opensolaris.org/catalog/1/catalog.attrs Expects: A catalog file name in the request path. Returns: The contents of the specified catalog file. Valid Catalog File Names: The following catalog files are valid for the V1 catalog endpoint: catalog.attrs Contains metadata about the catalog, including creation date, last modified date, package counts, and information about available catalog parts. Sample Output: { "_SIGNATURE": { "sha-1": "8f5c22fd8218f7a0982d3e3fdd01e40671cb9cab" }, "created": "20050614T080000.234231Z", "last-modified": "20090508T161025.686485Z", "package-count": 40802, "package-version-count": 1706, "parts": { "catalog.base.C": { "last-modified": "20090508T161025.686485Z", "signature-sha-1": "9b37ef267ae6aa8a31b878aad4e9baa234470d45" }, "catalog.dependency.C": { "last-modified": "20090508T161025.686485Z", "signature-sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8" }, "catalog.summary.C": { "last-modified": "20090508T161025.686485Z", "signature-sha-1": "b3a6ab53677c7b5f94c9bd551a484d57b54ed6f7" } }, "version": 1 } catalog.base.C Contains the FMRIs of packages in the repository and optional digest values for verifying manifest contents. Sample Output: { "_SIGNATURE": { "sha-1": "9b37ef267ae6aa8a31b878aad4e9baa234470d45" }, "opensolaris.org": { "SUNWipkg": [ { "version": "0.5.11,5.11-0.117:20090623T135937Z", "signature-sha-1": "596f26c4fc725b486faba089071d2b3b35482114" }, { "version": "0.5.11,5.11-0.118:20090707T220625Z", "signature-sha-1": "ab6f26c4fc725b386faca089071d2b3d35482114" } ] } } catalog.dependency.C Contains package FMRIs, depend actions, and set actions for facets or variants. Sample Output: { "_SIGNATURE": { "sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8" }, "opensolaris.org": { "SUNWdvdrw": [ { "version": "5.21.4.10.8,5.11-0.108:20090218T042840Z", "actions": [ "set name=variant.zone value=global value=nonglobal", "set name=variant.arch value=sparc value=i386", "depend fmri=SUNWlibms@0.5.11-0.108 type=require", "depend fmri=SUNWlibC@0.5.11-0.108 type=require", "depend fmri=SUNWcsl@0.5.11-0.108 type=require" ] } ] } } catalog.summary.C Contains package FMRIs and set actions (excluding those for facets or variants). Sample Output: { "_SIGNATURE": { "catalog-sha-1": "b3a6ab53677c7b5f94c9bd551a484d57b54ed6f7" }, "opensolaris.org": { "SUNWdvdrw": [ { "version": "5.21.4.10.8,5.11-0.108:20090218T042840Z", "actions": [ "set name=description value=\"DVD creation utilities\"", "set name=info.classification value=org.opensolaris.category.2008:System/Media" ] } ] } } catalog.summary. Locale-specific versions of the summary catalog, where is an OpenSolaris system locale name. update.. Update logs containing incremental changes to the catalog, where is a UTC date and time in ISO-8601 'reduced accuracy basic format' and is an OpenSolaris system locale name. Sample Output: { "_SIGNATURE": { "sha-1": "0c896321c59fd2cd4344fec074d55ba9c88f75e8" }, "opensolaris.org": { "SUNWdvdrw": [ { "op-type": "add", "op-time": "20090524T042841Z", "version": "5.21.4.10.8,5.11-0.111:20090524T042840Z", "catalog.dependency.C": { "actions": [ "depend fmri=SUNWlibms@0.5.11-0.111 type=require", "depend fmri=SUNWlibC@0.5.11-0.111 type=require", "depend fmri=SUNWcsl@0.5.11-0.111 type=require" ] }, "catalog.summary.C": { "actions": [ "set name=description value=\"DVD creation utilities\"", "set name=info.classification value=org.opensolaris.category.2008:System/Media" ] }, "signature-sha-1": "fe6f26c4fc124b386faca089071d2b3a35482114" } ] } } - info Version 0: A GET operation that retrieves a text/plain description of a package and its licensing information specified by the provided FMRI. Example: URL: http://pkg.opensolaris.org/info/0/entire@0.5.11,5.11-0.101:20081119T235706Z Expects: A URL-encoded pkg(5) FMRI, excluding the 'pkg:/' scheme prefix and publisher information, and including the full version information. Returns: A text/plain representation of the specified package and its licensing information. Sample Output: Name: entire Summary: entire incorporation Publisher: Unknown Version: 0.5.11 Build Release: 5.11 Branch: 0.101 Packaging Date: Wed Nov 19 23:57:06 2008 Size: 0.00 B FMRI: pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z License: - manifest Version 0: A GET operation that retrieves the contents of the manifest file for a package specified by the provided FMRI. Example: URL: http://pkg.opensolaris.org/manifest/0/entire@0.5.11,5.11-0.101:20081119T235706Z Expects: A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix and publisher information and including the full version information. Returns: The contents of the package's manifest file. Version 1: A GET or POST/PUT operation that retrieves or uploads the contents of a manifest file. For GET requests, it behaves the same as Version 0. For POST/PUT requests, it allows uploading manifest content to the repository. Example: URL: http://pkg.opensolaris.org/manifest/1/entire@0.5.11,5.11-0.101:20081119T235706Z Expects: For GET: A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix and publisher information and including the full version information. For POST/PUT: The manifest content in the request body. Returns: For GET: The contents of the package's manifest file. For POST/PUT: Response status of 200 on success; any other status indicates failure. - p5i Version 0: A GET operation that retrieves an application/vnd.pkg5.info datastream representing publisher and package information. This is intended for consumption by clients for the purposes of auto-configuration, metadata management policy determination, and triggering packaging operations such as installation. Example: URL: http://pkg.opensolaris.org/release/p5i/0/SUNWcs Expects: A full or partial URL-encoded pkg(5) FMRI, excluding the publisher prefix. If the partial or full FMRI is valid, it will be added to the datastream as is. If it includes the wildcard character '*', a search of the repository's catalog for matching entries will be performed and the unique set of resulting package stems will be added to the datastream. If no match is found, a 404 error will be raised. Returns: Returns a pkg(5) information datastream based on the repository configuration's publisher information and the provided full or partial FMRI or matching entries. The Content-Type of the response is 'application/vnd.pkg5.info'. - publisher Version 0: A GET operation that retrieves an application/vnd.pkg5.info datastream representing publisher information. This is intended for consumption by clients for auto-configuration and metadata management policy determination. Example: URL: http://pkg.opensolaris.org/release/publisher/0 Expects: Nothing Returns: Returns a pkg(5) information datastream based on the repository configuration's publisher information. The Content-Type of the response is 'application/vnd.pkg5.info'. Version 1: A GET operation that retrieves an application/vnd.pkg5.info datastream representing publisher information for a specific publisher or all publishers. This version provides improved error handling and response formatting. Example: URL: http://pkg.opensolaris.org/release/publisher/1 Expects: Nothing Returns: Returns a pkg(5) information datastream based on the repository configuration's publisher information. The Content-Type of the response is 'application/vnd.pkg5.info'. - search Version 0: A GET operation that retrieves a text/plain list of packages with metadata that matches the specified criteria. Example: URL: http://pkg.opensolaris.org/release/search/0/vim Expects: A URL-encoded token representing the search criteria. Returns: A text/plain list of matching entries, separated by newlines. Each entry consists of a set of four space-separated values: index - what search index the entry was found in action - what package action the entry is related to value - the value that the matched the search criteria package - the fmri of the package that contains the match Results are streamed to the client as they are found. Sample Output: basename pkg:/SUNWvim@7.1.284,5.11-0.101:20081119T230659Z dir usr/share/vim basename pkg:/SUNWvim@7.1.284,5.11-0.93:20080708T171331Z file usr/bin/vim Version 1: A GET or POST operation that retrieves a text/plain list of packages with metadata that matches the specified criteria. This version adds support for POST requests, case-sensitive searches, and returns a validation string to confirm the client is talking to a valid search server. Example: URL: http://pkg.opensolaris.org/release/search/1/vim Expects: For GET: A URL-encoded token representing the search criteria. For POST: Search criteria in the request parameters. Returns: A text/plain list of matching entries, preceded by a validation string. Each entry consists of a set of space-separated values: query_number - which query in the request this result is for return_type - whether the result is an action or package fmri - the fmri of the package that contains the match field - the field that matched (URL-encoded) line - the line that matched (for actions) If no results are found, a 204 No Content status is returned. - index Version 0: A GET operation that provides an administrative interface for search indexing. Returns no output if successful; otherwise the response body will contain the failure details. Example: URL: http://pkg.opensolaris.org/release/index/0/refresh Expects: A command to execute, such as "refresh" to update search indexes. Returns: No output if successful; otherwise the response body will contain the failure details. 2.2.6. Publishing Operations - add Version 0: A POST operation that adds content to an in-flight transaction for the Transaction ID specified. This could either be file content for the package or metadata about the package. This data is not added to the repository for retrieval until a close operation for the specified Transaction ID is executed. Example: URL: http://pkg.opensolaris.org/add/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z HEADERS: X-IPkg-SetAttr1: description=Package Name REQUEST BODY: Expects: A Transaction ID as output by pkgsend(1) in the request path. The file content (if applicable), to be added, in the request body. Any attributes to be set in the headers in the pattern of: X-IPkg-SetAttr{integer}: attr=value Returns: Response status of 200 on success; any other status indicates failure. - abandon Version 0: A GET operation that aborts an in-flight transaction for the Transaction ID specified. This will discard any data related to the transaction. Example: URL: http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z Expects: A Transaction ID as output by pkgsend(1) in the request path. Returns: Response status of 200 on success; any other status indicates failure. - append Version 0: A POST operation that starts an append transaction for the package name specified in the request path. Returns no output. Example: URL: http://pkg.opensolaris.org/append/0/system%2Flibc@0.1-98 Expects: A URL-encoded pkg(5) FMRI (excluding timestamp). Returns: Response status of 200 on success and an identifier for the new transaction in the 'Transaction-ID' response header; any other status indicates failure. - close Version 0: A GET operation that ends an in-flight transaction for the Transaction ID specified. If successful, the corresponding package is added to the repository catalog and is immediately available to repository users. Example: URL: http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z Expects: A Transaction ID as output by pkgsend(1) in the request path. Returns: Response status of 200 on success; any other status indicates failure. - open Version 0: A GET operation that starts an in-flight transaction for the package FMRI specified. Example: URL: http://pkg.opensolaris.org/open/0/system%2Flibc@0.1-98 Expects: A URL-encoded pkg(5) FMRI (excluding timestamp). Returns: Response status of 200 on success and an identifier for the new transaction in the 'Transaction-ID' response header; any other status indicates failure. 2.2.7. Administrative Operations - admin Version 0: A GET operation that executes a specified repository administration operation based on the provided query string. Example: URL: http://pkg.opensolaris.org/release/admin/0?cmd=refresh-index Expects: A command parameter in the query string. Available commands are: rebuild Discard search data and package catalogs and rebuild both. rebuild-indexes Discard search data and rebuild. rebuild-packages Discard package catalogs and rebuild. refresh Update search and package data. refresh-indexes Update search data. (Add packages found in the repository to their related search indexes.) refresh-packages Update package data. (Add packages found in the repository to their related catalog.) Returns: No output if successful; otherwise the response body will contain the failure details. 3. References