ips/doc/pkg5_docs/depot.txt

pkg
DEPOT

1.  Summary

    This document describes the architecture of a pkg(5) depot server as
    implemented in pkg.depotd(1m).  This includes: an overview of the depot
    server's filesystem layout, operations provided by the depot server,
    the interfaces provided by the depot server, and a description of how
    clients communicate with the depot server for each operation.

    NOTE: The image packaging system is under development.  Changes to
    interfaces may occur as part of the architectural review process, as
    shortcomings are identified, and as new features are introduced.
    Questions about planned or possible change should be asked on
    pkg-discuss.

2.  Discussion

    A pkg(5) depot server provides a way for clients to interact with package
    content and metadata contained within a pkg(5) repository, and with the
    depot server itself.  It accomplishes this by providing an HTTP-based
    interface suitable for usage by pkg(1) clients and web user agents.

2.1.  Filesystem Layout

    The types of information that the depot server stores and/or retrieves can
    be categorized as follows:

    - depot data
        This includes: configuration data, presentation content (such as
        web page templates), publishing data (e.g. in-flight transactions),
        and temporary data (e.g. the feed cache).

    - repository data
        This includes: catalog information, package content (files), package
        metadata (manifests), and search data.

2.1.1.  Layout

    The depot server uses the following 'root' directory structures for the
    storage and retrieval of depot and repository data:

    - repo_dir (depot and repository data)
        cfg_cache (depot data)
            A file containing the cached configuration information for the
            depot server.

        catalog/ (repository data)
            This directory contains the repository catalog and its related
            metadata.

        file/ (repository data)
            This directory contains the file content of packages in the
            repository.

            Files are stored using a two-level path fragment, derived from the
            SHA1-hash of a file's content, assumed to have at least 8 distinct
            characters.

            Example:
                00/
                    0023bb/
                        000023bb53fdc7bcf35e62b7b0b353a56d36a504

        index/ (repository data)
            This directory contains the search indices for the repository.

        pkg/ (repository data)
            This directory contains the metadata (manifests) for the
            repository's packages.

            The manifests for each package are stored in a directory with the
            same name as the package stem using a URL-encoded filename.

            Example:
                entire/
                    0.5.11%2C5.11-0.86%3A20080422T234219Z

        trans/ (depot data)
            This directory contains in-flight transactions for packages that
            are waiting for the publication process to complete so that they
            can be added to the repository's catalog.

            Each transaction is stored in a directory named after the pending
            transaction id and contains the manifest waiting for publication
            to finish stored with the filename of 'manifest'.

            Example:
                1229379580_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081215T221940Z/
                    manifest

        updatelog/ (repository data)
            This directory contains metadata detailing changes to the repository
            by publishing operations.

    - content_root (depot data)

        web/
            This directory contains all of the web presentation content for the
            depot.

2.2.  Operations

    When communicating with the depot server via HTTP, operations are presented
    via a URL-based mechanism that allows each to be versioned so that changes
    in protocol can be hidden from older clients as the interfaces to operations
    evolve.

    Operations made available by a pkg.depotd(5) server can be accessed via GET
    or POST, as appropriate for each operation, via a URL such as the following:

        http://pkg.opensolaris.org/release/manifest/0/SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z

    The above example can be broken down into four basic components:

        publisher_origin_url    - http://pkg.opensolaris.org/release/
        operation_name          - manifest
        protocol_version        - 0
        operation_arguments     - SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z

    Each of these components can be described as follows:

        publisher_origin_url    - A URL that can be used to access a depot
                                  server's repository.

        operation_name          - The name of the operation that the client is
                                  wanting to initiate.

        protocol_version        - An integer value representing the version of
                                  the operation's protocol spoken by the client.

        operation_arguments     - String data (such as a package FMRI) that is
                                  parsed and then used to determine what
                                  resource(s) will be used to perform an
                                  operation.  Some operations expect arguments
                                  or data to be passed via POST-based form data,
                                  headers, or the request body instead.

2.2.1.  Operation Types

    Each operation that the depot server provides is either designed to interact
    with a pkg(5) repository, or with the depot server itself.  These operations
    can be categorized as follows:

    - content
        These operations are read-only, and retrieve file data that comprises
        the content of a package in a repository.

    - depot
        These operations are read-only, and permit retrieval of: the list of
        operations that the depot server currently provides (including protocol
        version and pkg(5) software version), statistics information, and other
        depot information.

    - metadata
        These operations are read-only, and retrieve metadata related to a
        package FMRI, such as its name, version, etc. stored in a repository's
        catalog.

    - publishing
        These operations alter a repository's catalog, package metadata, and
        allow storage of package content.

2.2.2. Modes

    Which types of operations are available is dependent on which mode the depot
    server is currently operating in:

        - default
            In default mode, the depot server allows content, depot, metadata,
            and publishing operations.

        - readonly
            In readonly mode, the depot server allows content, depot, and
            metadata operations.

        - mirror
            In mirror mode, the depot server allows content and depot
            operations.

2.2.3.  Content Operations

    The pkg.depotd(5) server provides the following operations for retrieving
    package content:

    - file
        Version 0:
            A GET operation that retrieves the contents of a file, belonging to a
            package, using a SHA-1 hash of the file's content.

            Example:
                URL:
                http://pkg.opensolaris.org/release/file/0/
                a00030db8b91f85d0b7144d0d4ef241a3f1ae28f

            Expects:
                A SHA-1 hash of the file's content belonging to a package in the
                request path.

            Returns:
                The contents of the file, compressed using the gzip compression
                algorithm.

2.2.2.  Depot Operations

    - versions
        Version 0:
            A GET operation that retrieves text data representing what operations
            are supported and version information about the depot server.

            Example:
                URL:
                http://pkg.opensolaris.org/versions/0/

            Expects:
                Nothing

            Returns:
                text/plain data containing the version of the pkg(5) software that
                the depot is based upon, a list of the operations currently
                supported, and the protocol version supported for each
                operation.

            Sample Output:
                pkg-server bfc04991436e
                info 0
                search 0
                versions 0
                catalog 0
                manifest 0
                add 0
                file 0
                abandon 0
                close 0
                open 0

2.2.2.  Meta-data Operations

    - catalog
        Version 0:
            A GET operation that retrieves a text/plain datastream
            representing a complete catalog or an incremental update to an
            existing one as requested.

            Example:
                URL:
                http://pkg.opensolaris.org/catalog/0/

            Expects:
                Nothing or the following headers:
                    If-Modified-Since: {ISO 8601 formatted date and time in UTC}

            Returns:
                Either the contents of a pkg(5) catalog file, or the entries
                that were added since the specified date as they are found
                in the catalog file, separated by newlines.

    - info
        Version 0:
            A GET operation that retrieves a text/plain description of a
            package and its licensing information specified by the provided
            FMRI.

            Example:
                URL:
                http://pkg.opensolaris.org/info/0/entire@0.5.11,5.11-0.101:20081119T235706Z

            Expects:
                A URL-encoded pkg(5) FMRI, excluding the 'pkg:/' scheme prefix
                and publisher information, and including the full version
                information.

            Returns:
                A text/plain representation of the specified package and its
                licensing information.

            Sample Output:
                Name: entire
                Summary: entire incorporation
                Publisher: Unknown
                Version: 0.5.11
                Build Release: 5.11
                Branch: 0.101
                Packaging Date: Wed Nov 19 23:57:06 2008
                Size: 0.00 B
                FMRI: pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z

                License:

    - manifest
        Version 0:
            A GET operation that retrieves the contents of the manifest file for
            a package specified by the provided FMRI.

            Example:
                URL:
                http://pkg.opensolaris.org/manifest/0/entire@0.5.11,5.11-0.101:20081119T235706Z

            Expects:
                A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix
                and publisher information and including the full version
                information.

            Returns:
                The contents of the package's manifest file.

    - p5i
        Version 0:
                A GET operation that retrieves an application/vnd.pkg5.info
                datastream representing publisher and package information.
                This is intended for consumption by clients for the purposes
                of auto-configuration, metadata management policy determination,
                and triggering packaging operations such as installation.

            Example:
                URL:
                http://pkg.opensolaris.org/release/p5i/0/SUNWcs

            Expects:
                A full or partial URL-encoded pkg(5) FMRI, excluding the
                publisher prefix.  If the partial or full FMRI is valid, it will
                be added to the datastream as is.  If it includes the wildcard
                character '*', a search of the repository's catalog for matching
                entries will be performed and the unique set of resulting
                package stems will be added to the datastream.  If no match is
                found, a 404 error will be raised.

            Returns:
                Returns a pkg(5) information datastream based on the repository
                configuration's publisher information and the provided full or
                partial FMRI or matching entries.  The Content-Type of the
                response is 'application/vnd.pkg5.info'.

    - publisher
        Version 0:
                A GET operation that retrieves an application/vnd.pkg5.info
                datastream representing publisher information.  This is intended
                for consumption by clients for auto-configuration and metadata
                management policy determination.

            Example:
                URL:
                http://pkg.opensolaris.org/release/publisher/0

            Expects:
                Nothing

            Returns:
                Returns a pkg(5) information datastream based on the repository
                configuration's publisher information.  The Content-Type of the
                response is 'application/vnd.pkg5.info'.

    - search
        Version 0:
            A GET operation that retrieves a text/plain list of packages with
            metadata that matches the specified criteria.

            Example:
                URL:
                http://pkg.opensolaris.org/release/search/0/vim

            Expects:
                A URL-encoded token representing the search criteria.

            Returns:
                A text/plain list of matching entries, separated by newlines.
                Each entry consists of a set of four space-separated values:

                    index   - what search index the entry was found in

                    action  - what package action the entry is related to

                    value   - the value that the matched the search criteria

                    package - the fmri of the package that contains the match

                Results are streamed to the client as they are found.

            Sample Output:
                basename pkg:/SUNWvim@7.1.284,5.11-0.101:20081119T230659Z dir usr/share/vim
                basename pkg:/SUNWvim@7.1.284,5.11-0.93:20080708T171331Z file usr/bin/vim

2.2.3.  Publishing Operations

    - add
        Version 0:
            A POST operation that adds content to an in-flight transaction for
            the Transaction ID specified.  This could either be file content
            for the package or metadata about the package.

            This data is not added to the repository for retrieval until a close
            operation for the specified Transaction ID is executed.

            Example:
                URL:
                http://pkg.opensolaris.org/add/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z

                HEADERS:
                X-IPkg-SetAttr1: description=Package Name

                REQUEST BODY:

            Expects:
                A Transaction ID as output by pkgsend(1) in the request path.
                The file content (if applicable), to be added, in the request
                body.  Any attributes to be set in the headers in the pattern
                of:
                    X-IPkg-SetAttr{integer}: attr=value

            Returns:
                Response status of 200 on success; any other status indicates
                failure.

    - abandon
        Version 0:
            A GET operation that aborts an in-flight transaction for the
            Transaction ID specified.  This will discard any data related to
            the transaction.

            Example:
                URL:
                http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z

            Expects:
                A Transaction ID as output by pkgsend(1) in the request path.

            Returns:
                Response status of 200 on success; any other status indicates
                failure.

    - close
        Version 0:
            A GET operation that ends an in-flight transaction for the
            Transaction ID specified.  If successful, the corresponding package
            is added to the repository catalog and is immediately available to
            repository users.

            Example:
                URL:
                http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z

            Expects:
                A Transaction ID as output by pkgsend(1) in the request path.

            Returns:
                Response status of 200 on success; any other status indicates
                failure.

    - open
        Version 0:
            A GET operation that starts an in-flight transaction for the
            package FMRI specified.

            Example:
                URL:
                http://pkg.opensolaris.org/open/0/system%2Flibc@0.1-98

            Expects:
                A URL-encoded pkg(5) FMRI (excluding timestamp).

            Returns:
                Response status of 200 on success and an identifier for the new
                transaction in the 'Transaction-ID' response header; any other
                status indicates failure.

3.  References