+
+Actuators
+~~~~~~~~~
+
+Reboot necessity.
+
+Those system configuration steps which can be deferred.
+
+Variants and facets
+~~~~~~~~~~~~~~~~~~~
+
+Packaging considerations
+========================
+
+Many of the good packaging criteria present trade-offs among themselves. It
+will often be difficult to satisfy all requirements equally. These criteria are
+presented in order of importance; however, this sequence is meant to serve as a
+flexible guide depending on the circumstances. Although each of these criteria
+is important, it is up to you to optimize these requirements to produce a good
+set of packages.
+
+Optimize for Client-Server Configurations
+-----------------------------------------
+
+You should consider the various patterns of software use
+(client and server) when laying out packages. Good packaging
+design divides the affected files to optimize installation of each
+configuration type. For example, for a network protocol implementation,
+it should be possible to install the client without necessarily
+installing the server.
+
+Package by Functional Boundaries
+--------------------------------
+
+Packages should be self-contained and distinctly identified with a set of
+functionality. For example, a package containing UFS should contain all UFS
+utilities and be limited to only UFS binaries.
+
+Packages should be organized from a customer's point of view into functional
+units.
+
+Package Along License or Royalty Boundaries
+-------------------------------------------
+
+Put code that requires royalty payments due to contractual agreements or
+that has distinct software license terms in a dedicated package or group
+of packages. Do not to disperse the code into more packages than
+necessary.
+
+Overlap in Packages
+-------------------
+
+When constructing the packages, ensure that duplicate files are eliminated when
+possible. Unnecessary duplication of files results in support and version
+difficulties. If your product has multiple packages, constantly compare the
+contents of these packages for redundancies.
+
+Sizing Considerations
+---------------------
+
+Size is package-specific and depends on other criteria. For example, the
+maximum size of /opt should be considered. When possible, a good package should
+not contain only one or two files or contain extremely large numbers of files.
+There are cases where a smaller or larger package might be appropriate to
+satisfy other criteria.
+
+Licensing Considerations for Packages
+-------------------------------------
+
+If you are distributing software that uses licensing, there are several things
+you need to consider:
+
+ - Business operations
+ - Communication with users
+ - Technology
+
+*Business Operations.* Before you begin distributing licensed software, set up
+your business operations to distribute, price, and track licenses. There are a
+variety of ways to distribute licenses, such as fax, electronic mail, or an 800
+telephone number. You need to choose a method of distribution and set up all
+the necessary processes. You also need to consider whether licenses need to be
+upgraded with the software and how this will be done.
+
+Pricing policy and types of licenses must also be considered. You must consider
+how the product is used and what kinds of licenses your users will need to use
+the product effectively. Single user licenses may not be appropriate for many
+situations.
+
+*Communication with Users.* Before you implement licensing, you need to inform
+your users, particularly if the product has not been licensed in the past.
+
+When you do implement licensing, you may want to consider implementing it
+gradually. The first step would be monitoring the use of licenses, followed by
+warning that the software is being used without a license, and finally, denying
+the use of the software.
+
+*Technology.* If you are going to use a commercial product for licensing, there
+are many things to consider when making your choice. You need to decide what
+your priorities are. For example, is ease of administration and use most
+important? Or is enforcing the licensing policy more important?
+
+You also need to consider whether the software will be used in a heterogeneous
+or homogeneous environment and whether standards are important. You may also
+want to look at the security provided by the product. Is it easy to get around
+the enforcement of licenses?
+
+The issues involved in choosing a commercial product will vary depending on
+the kind of application and your reasons for implementing licensing.
+
+
+Naming your package
+===================
+
+.. include:: guide-naming-conventions.rst
+
+Decorating your package
+=======================
+
+.. include:: guide-metadata-conventions.rst
+
+A full packaging example
+========================
+
+Delivery examples
+=================
+
+In the following sections, we give examples of delivering specific
+resources via the image packaging system.
+
+Device driver
+-------------
+
+XXX ``e1000g``
+
+Versioned interpreter
+---------------------
+
+XXX perl
+XXX ``verexec``
+
+|smf5| Service
+--------------
+
+XXX pkg.depotd
+
+GNOME Desktop Elements
+----------------------
+
+XXX pick a specific Gnome application
+
+Coordinating groups of packages using incorporations
+----------------------------------------------------
+
+Renaming a package
+------------------
+
+Making a package version obsolete
+---------------------------------
+
+Moving a file between packages
+------------------------------
+
+Migrating an existing package
+=============================
+
+Migrating a System V Package
+----------------------------
+
+XXX pkgsend
+
+Migrating a |tar1| archive
+--------------------------
+
+XXX pkgsend
+
+Publishing from an installation directory
+-----------------------------------------
+
+Pre-publication tools
+=====================
+
+pkgmogrify, pkgdepend, pkgdiff, and pkgfmt.
+
+----------------
+Depot operations
+----------------
+
+Distributing packages with a depot
+==================================
+
+XXX thread tuning
+
+Using Apache HTTPD as a reverse proxy cache
+-------------------------------------------
+
+recommended
+
+can speed up operations
+
+Running a content mirror
+------------------------
+
+From DVD
+
+Via rsync
+
+Long-term operations
+--------------------
+
+Splitting and spreading package retrieval load
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+can be load balanced
+
+Tuning search
+~~~~~~~~~~~~~
+
+Publishing packages with a depot
+================================
+
+running in r/w mode
+
+XXX could we automate snapshots?
+
+----------------
+Packaging client
+----------------
+
+.. section-autonumbering
+ :start: 1
+
+.. include:: guide-pkg-states.rst
+
+- protocol / network format
+ - client side REST API
+ - publication side REST API
+
+Retrieval protocol operations
+=============================
+
+Publication protocol operations
+===============================
+
+.. include:: guide-publication-protocol.rst
+
+Other protocol operations
+=========================
+
+- versions
+ Version 0
+
+ A GET operation that retrieves the current set of operations
+ offered by the contacted depot.
+
+ Example:
+
+ URL: http://pkg.opensolaris.org/versions/0
+
+ Expects:
+
+ No body.
+
+ Returns:
+
+ List of operations and versions, one operation per line, space
+ separated list of versions for each operation.
+- search
+ Version 1
+
+ A GET operation that presents a query to the search index
+ capability of the contacted depot.
+
+
+--------------------------------
+Package depots and other servers
+--------------------------------
+
+.. section-autonumbering
+ :start: 1
+
+.. include:: guide-repository-format.rst
+
+|depotd1m| implementation
+=========================
+
+.. include:: guide-implementation-depot.rst
+
+--------------------------------
+Appendix: Reference manual pages
+--------------------------------
+
+pkg(1)
+======
+
+.. raw:: html
+
+
+
+.. raw:: html
+ :file: ../src/man/pkg.1
+
+.. raw:: html
+
+
+
+.. raw:: latex
+
+ \begin{verbatim}
+
+.. raw:: latex
+ :file: ../src/man/pkg.1
+
+.. raw:: latex
+
+ \end{verbatim}
+
+pkgrecv(1)
+==========
+
+.. raw:: html
+
+
+
+.. raw:: html
+ :file: ../src/man/pkgrecv.1
+
+.. raw:: html
+
+
+
+.. raw:: latex
+
+ \begin{verbatim}
+
+.. raw:: latex
+ :file: ../src/man/pkgrecv.1
+
+.. raw:: latex
+
+ \end{verbatim}
+
+pkgsend(1)
+==========
+
+.. raw:: html
+
+
+
+.. raw:: html
+ :file: ../src/man/pkgsend.1
+
+.. raw:: html
+
+
+
+.. raw:: latex
+
+ \begin{verbatim}
+
+.. raw:: latex
+ :file: ../src/man/pkgsend.1
+
+.. raw:: latex
+
+ \end{verbatim}
+
+pkg.depotd(1M)
+==============
+
+.. raw:: html
+
+
+
+.. raw:: html
+ :file: ../src/man/pkg.depotd.1m
+
+.. raw:: html
+
+
+
+.. raw:: latex
+
+ \begin{verbatim}
+
+.. raw:: latex
+ :file: ../src/man/pkg.depotd.1m
+
+.. raw:: latex
+
+ \end{verbatim}
+
+pkg(5)
+======
+
+.. raw:: html
+
+
+
+.. raw:: html
+ :file: ../src/man/pkg.5
+
+.. raw:: html
+
+
+
+.. raw:: latex
+
+ \begin{verbatim}
+
+.. raw:: latex
+ :file: ../src/man/pkg.5
+
+.. raw:: latex
+
+ \end{verbatim}
+
+---------------------------
+Appendix: Protocol details
+---------------------------
+
+------------------------------------------
+Appendix: Architectural process materials
+------------------------------------------
+
+.. raw:: html
+
+
+
+.. raw:: html
+ :file: one-pager-main.txt
+
+.. raw:: html
+
+
+
+
diff --git a/doc/pkg5_docs/guide-metadata-conventions.rst b/doc/pkg5_docs/guide-metadata-conventions.rst
new file mode 100644
index 0000000..7086138
--- /dev/null
+++ b/doc/pkg5_docs/guide-metadata-conventions.rst
@@ -0,0 +1,329 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+Tags and attributes
+-------------------
+
+Definitions
+~~~~~~~~~~~
+
+ Both packages and actions within a package can carry metadata, which
+ we informally refer to as attributes and tags. Both attributes and
+ tags have a name and one or more values.
+
+ Attributes
+ settings that apply to an entire package. Introduction
+ of an attribute that causes different deliveries by the client could
+ cause a conflict with the versioning algebra, so we attempt to avoid
+ them.
+
+ Tags
+ settings that affect individual files within a package.
+
+Attribute and tag syntax and namespace
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Syntax
+``````
+
+Naming
+^^^^^^
+
+ The syntax for attributes and tags is similar to that used for
+ |pkg5| and |smf5| FMRIs.
+
+ [org_prefix,][name][:locale]
+
+ The organizational prefix is a forward-ordered or reverse-ordered
+ domain name, followed by a comma. The name field is an identifier
+ which may have a prefix ending in a period to allocate the namespace.
+ If the locale field is omitted, the default locale is "C", a 7-bit
+ ASCII locale.
+
+ Each of these fields is [A-Za-z][A-Za-z0-9\_-.]*.
+
+Manifests
+^^^^^^^^^
+
+ In package manifests, the syntax for an attribute is:
+
+ set name= value= [value= ...]
+
+ In package manifests, tags are included in the action line
+ for the action they apply to:
+
+ [...] = [= ...]
+
+Unprefixed attributes and tags
+``````````````````````````````
+
+ All unprefixed attributes and tags are reserved for use by the
+ framework.
+
+ Generally, unprefixed tags are introduced in the definition of an
+ action.
+
+Attributes and tags common to all packages
+``````````````````````````````````````````
+
+ Attributes and tags starting with "pkg." or "info." are for attributes
+ common to all packages, regardless of which particular OS platforms that
+ a specific package may target. "pkg" attributes are used by the
+ packaging system itself, while "info" attributes are purely informational,
+ possibly for use by other software.
+
+Common attributes
+^^^^^^^^^^^^^^^^^
+
+ pkg.summary
+ A short, descriptive name for the package.
+ pkg.summary:fr would be the descriptive name in French.
+ Exact numerical version strings are discouraged in the
+ descriptive name.
+
+ Example: "Apache HTTPD Web Server 2.x"
+
+ pkg.description
+ A descriptive paragraph for the package. Exact numerical version
+ strings can be embedded in the paragraph.
+
+ pkg.detailed-url
+ One or more URLs to pages or sites with further information about
+ the package. pkg.detailed-url:fr would be the URL to a page with
+ information in French.
+
+ pkg.renamed
+ A value of "true" indicates the package has been renamed or split
+ into the packages listed in the depend actions.
+
+ pkg.obsolete
+ A value of "true" indicates the package is obsolete and should be
+ removed on upgrade.
+
+ pkg.human-version
+ For components whose upstream version isn't a dot-separated sequence
+ of nonnegative integers (OpenSSL's 0.9.8r, for example), this
+ attribute can be set to that string, and will be displayed when
+ appropriate. It cannot be used in an FMRI to install a particular
+ version; package authors must still convert the version into a
+ sequence of integers.
+
+ variant.*
+ See facets.txt
+
+Common tags
+^^^^^^^^^^^
+
+ disable_fmri
+ See "Actuators" section of |pkg5|
+
+ facet.*
+ See facets.txt
+
+ reboot-needed
+ See "Actuators" section of |pkg5|
+
+ refresh_fmri
+ See "Actuators" section of |pkg5|
+
+ restart_fmri
+ See "Actuators" section of |pkg5|
+
+ suspend_fmri
+ See "Actuators" section of |pkg5|
+
+ variant.*
+ See facets.txt
+
+Informational attributes
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following attributes are not necessary for correct package installation,
+but having a shared convention lowers confusion between publishers and
+users.
+
+info.classification
+ A list of labels classifying the package into the categories
+ shared among |pkg5| graphical clients.
+
+ Values currently used for OpenSolaris are prefixed with
+ ``org.opensolaris.category.2008:`` and must match one of the
+ categories listed in ``src/gui/data/opensolaris.org.sections``
+
+info.keyword
+ A list of additional terms that should cause this package to be
+ returned by a search.
+
+info.maintainer
+ A human readable string describing the entity providing the
+ package. For an individual, this string is expected to be their
+ name, or name and email.
+
+info.maintainer-url
+ A URL associated with the entity providing the package.
+
+info.upstream
+ A human readable string describing the entity that creates the
+ software. For an individual, this string is expected to be
+ their name, or name and email.
+
+info.upstream-url
+ A URL associated with the entity that creates the
+ software delivered within the package.
+
+info.source-url
+ A URL to the source code bundle, if appropriate, for the package.
+
+info.repository-url
+ A URL to the source code repository, if appropriate, for the
+ package.
+
+info.repository-changeset
+ A changeset ID for the version of the source code contained in
+ info.repository-url.
+
+Attributes common to all packages for an OS platform
+````````````````````````````````````````````````````
+
+ Each OS platform is expected to define a string representing that
+ platform. For example, the |OS_Name| platform is represented by
+ the string "opensolaris".
+
+OpenSolaris attributes
+^^^^^^^^^^^^^^^^^^^^^^
+
+ org.opensolaris.arc-caseid
+ One or more case identifiers (e.g., PSARC/2008/190) associated with
+ the ARC case or cases associated with the component(s) delivered by
+ the package.
+
+ org.opensolaris.smf.fmri
+ One or more FMRI's representing SMF services delivered by this
+ package. Automatically generated by |pkgdepend1| for packages
+ containing SMF service manifests.
+
+ opensolaris.zone
+ Obsolete - replaced by variant.opensolaris.zone.
+
+ variant.opensolaris.zone
+ See facets.txt
+
+OpenSolaris tags
+^^^^^^^^^^^^^^^^
+
+ opensolaris.zone
+ Obsolete - replaced by variant.opensolaris.zone.
+
+ variant.opensolaris.zone
+ See facets.txt
+
+Organization specific attributes
+````````````````````````````````
+
+ Organizations wishing to provide a package with additional metadata
+ or to amend an existing package's metadata (in a repository that
+ they have control over) must use an organization-specific prefix.
+ For example, a service organization might introduce
+ ``service.example.com,support-level`` or
+ ``com.example.service,support-level`` to describe a level of support
+ for a package and its contents.
+
+Attributes specific to certain types of actions
+```````````````````````````````````````````````
+
+ Each type of action also has specific attributes covered in the
+ documentation of those actions. These are generally documented
+ in the section of the |pkg5| manual page for that action.
+
+Attributes specific to certain types of file
+````````````````````````````````````````````
+
+ These would generally appear on file actions for files in a specific
+ format.
+
+ elfarch, elfbits, elfhash
+
+ Data about ELF format binary files (may be renamed in the future
+ to info.file.elf.*). Automatically generated during package
+ publication. See the "File Actions" section of |pkg5|.
+
+ info.file.font.name
+
+ The name of a font contained in a given file. There may be multiple
+ values per file for formats which collect multiple typefaces into a
+ single file, such as .ttc (TrueType Collections), or for font aliases.
+ May also be provided in localized variants, such as a Chinese font
+ providing both info.file.font.name:en and info.file.font.name:zh for
+ the English and Chinese names for the font.
+
+ info.file.font.xlfd
+
+ An X Logical Font Description (XLFD) for a font contained in a
+ given file. Should match an XLFD listed in fonts.dir or fonts.alias
+ for the file. There may be multiple values per file due to font
+ aliases.
+
+
+.. 3.3. Attributes best avoided
+
+.. built-on release
+
+.. One problem we may run into is packages that have been built on a
+ release newer than that on the image. These packages should be
+ evaluated as unsuitable for the image, and not offered in the graph.
+ There are a few ways to handle this case:
+
+.. 1. Separate repository. All packages offered by a repository were
+ built on a known system configuration. This change requires
+ negotiation between client and server for a built-on match
+ condition. It also means that multiple repositories are needed
+ for a long lifecycle distribution.
+
+.. 2. Attributes. Each package comes with a built-on attribute. This
+ means that clients move from one built-on release to another
+ triggered by conditions set by the base package on the client.
+ Another drawback is that it becomes impossible to request a
+ specific package by an FMRI, without additional knowledge.
+
+.. 3. Additional version specifier. We could extend
+ release,branch:timestamp to release,built,branch:timestamp--or
+ fold the built and branch version together. Since the built
+ portion must reserve major.minor.micro, that means we move to a
+ package FMRI that looks like
+
+.. coreutils@6.7,5.11.0.1:timestamp
+
+.. This choice looks awkward. We could instead treat the built
+ portion as having a default value on a particular client. Then
+ the common specifier would be
+
+.. name@release[,build]-branch:timestamp
+
+.. build would be the highest available valid release for the
+ image.
+
+.. The meaning of the built-on version could be controversial. A
+ simple approach would be to base it on base/minimal's release,
+ rather than uname(1) output.
+
+
+
diff --git a/doc/pkg5_docs/guide-naming-conventions.rst b/doc/pkg5_docs/guide-naming-conventions.rst
new file mode 100644
index 0000000..d1d97ec
--- /dev/null
+++ b/doc/pkg5_docs/guide-naming-conventions.rst
@@ -0,0 +1,184 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+
+Package naming conventions
+--------------------------
+
+Definitions
+~~~~~~~~~~~
+
+To be consistent with the system, following the introduction of the
+fault management architecture, each package is named by an FMRI in the
+``pkg:`` scheme. That is, we have::
+
+ pkg://publisher/pkg_name@version
+
+The publisher is generally expected to be a forward or reverse domain
+name identifying the publisher from which a package can be retrieved.
+Publishers which cannot be determined to be a domain name are
+legitimate, but optional functionality, like automatic server discovery
+for a particular publisher, may fail to work. In the examples that
+follow, we use "opensolaris.org" as a generic publisher.
+
+Although RFC 2396 usage would suggest using the "authority" term, we
+instead call it the publisher name, as the role of this section of the
+FMRI is to identify the source for the package, which we consider a
+publication of one or more software components.
+
+The pkg_name, like service names, can be split into a category,
+subcategories, and a basename. This namespace might be populated
+with "manifest" and other metadata endpoints, as well as the SHA-1
+names of the package's included files. (Although the direct access
+to properties of the ``svc`` FMRI scheme has been rarely used.)
+
+A "group package" is a package that depends upon (minimum versions
+of) other packages, as well as optionally delivering files or other
+actions. An "incorporating package" is a group package that places
+forward constraints upon the versions of the packages it depends upon,
+which restricts the interval of valid package versions to one the author
+of the incorporation believes functional.
+
+
+Namespace
+~~~~~~~~~
+
+Single namespace, separate publishers
+``````````````````````````````````````
+
+The primary design point of the package namespace is to allow
+multiple package producers to co-exist in a single namespace, so
+that images can switch between equivalent components from different
+producers.
+
+Domain-name-based escape
+````````````````````````
+
+At any point in the category hierarchy, a safe namespace can be created
+by using the forward or reverse domain name, either as a subcategory or
+as a comma-led prefix to a subcategory or package base name. (This
+scheme is similar to FMRI namespace escapes in smf(5), although we are
+eliminating use of stock symbol prefixes.) Generally, safe namespaces
+are only needed when the components delivered by a package need to be
+distinguished from those delivered by a package in the single namespace;
+reasons for distinguishing the components might include construction of
+a collection of components, insulated from changes in the wider system,
+or for legal reasons.
+
+For instance, when example.com wishes to publish the "whiskers"
+package without reference to a larger namespace convention it can
+use any of the following examples::
+
+ pkg://opensolaris.org/.../com.example/whiskers
+
+ pkg://opensolaris.org/.../com.example,whiskers
+
+ pkg://opensolaris.org/.../com.example,software/whiskers
+
+and so forth.
+
+Locally reserved namespace
+``````````````````````````
+
+The top-level "site" category is reserved for use by the entity that
+administrates the server. Neither the organizations producing the
+operating system nor those providing additional software components may
+use the site category.
+
+The top-level "vendor" category is reserved for use by organizations
+providing additional. The leading subcategory must be a domain. That
+is, if example.com wishes to publish the "whiskers" package in the
+vendor category, it would use a package name like::
+
+ pkg://opensolaris.org/vendor/example.com/whiskers
+
+Use of the "vendor" category, and the vendor-escaped packages below, is
+likely only to be of interest to organizations that redistribute
+software from multiple source vendors.
+
+Additional reserved namespace
+`````````````````````````````
+
+.. admonition:: ARC Notice
+
+ Some or all of these reservations may be eliminated or reduced when
+ the single namespace convention reaches its final form.
+
+The top-level "cluster", "feature", "group", and "metacluster"
+categories are all reserved for future use.
+
+Single namespace conventions
+````````````````````````````
+
+Discussion
+^^^^^^^^^^
+
+Packaging systems and distributions appear to have explicit
+categories, subcategories, and potentially larger groups; some
+distributions have explicit fields for these values, others use
+tagging or multi-valued fields, which allows them to classify
+certain packages multiply. For the FMRI namespace case, the system
+is similar to a packaging system with multiple, single-valued,
+category fields.
+
+There appear to be two standard approaches to categorizing packages:
+
+ 1. By what primary class of thing a package delivers.
+
+ 2. By what area of functionality a package's contents address.
+
+In the first approach, we get strict top-level categories like
+"application", "driver", "library", and "system" or "kernel", as
+well as potentially overlapping categories like "utility" and
+"tool". Use of the leading subcategory is limited, and generally
+given to the subsystem involved. A relatively detailed worked
+example of the X11 subsystem under this scheme is given in
+
+http://mail.opensolaris.org/pipermail/pkg-discuss/2008-February/001838.html
+
+XXX This reference is now wrong.
+
+In the second, we would also see categories like these, but leading
+subcategory is much more likely to classify according to
+functionality, so that we would see "application/mail",
+"application/web", "application/compiler", and so forth. Most
+network packaging systems appear to classify in this fashion.
+
+An appealing variation of the second form is to rotate all of the
+non-"application" packages under a "system" mega-category, such that
+all of the leaf packages (with the possible exception of device
+drivers) are exposed at the category level. Table 1 shows some
+example transformations under this scheme.
+
+.. table:: Rotating non-application categories under system.
+
+ ========================= ====================
+ FROM TO
+ ========================= ====================
+ application/web/firefox web/firefox
+ application/compiler/gcc4 compiler/gcc4
+ library/c system/library/c
+ kernel/generic system/kernel/generic
+ ========================= ====================
+
+
+
diff --git a/doc/pkg5_docs/guide-pkg-states.rst b/doc/pkg5_docs/guide-pkg-states.rst
new file mode 100644
index 0000000..8e12a9a
--- /dev/null
+++ b/doc/pkg5_docs/guide-pkg-states.rst
@@ -0,0 +1,110 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+.. :vim set expandtab:
+
+.. _Package States:
+
+Package states
+==============
+
+By a *package state*, we mean an image's recorded state for a particular
+package version. We might also refer to these as "client states".
+Within an image, each package version can have only one state, as given
+in the following diagram::
+
+ 0
+ |
+ v
+ IN_CATALOG ---------> OUT_CATALOG
+ | ^
+ +--->---+---<-------+ |
+ | | | |
+ | v | |
+ | INSTALLED --> FROZEN |
+ | | |
+ | | |
+ | v |
+ +-- PRESERVED |
+ | | |
+ | | |
+ | v |
+ +-- DELETED -------------------+
+
+
+ 0 -> IN_CATALOG:
+ A catalog update with new entries.
+
+ IN_CATALOG:
+ An entry for this package is available in the locally installed
+ catalog.
+
+ IN_CATALOG -> OUT_CATALOG:
+ Entry formerly present on local catalog is no longer published by any
+ repository. (Package never locally installed.)
+
+ OUT_CATALOG:
+ Although a formerly known package, no entry for this package is
+ available in the locally installed catalog. An INSTALLED or
+ FROZEN package can never be OUT_CATALOG, as the system will
+ preserve the entry until the package is no longer in a locally
+ public state.
+
+ IN_CATALOG -> INSTALLED:
+ Transition takes place on package installation.
+
+ INSTALLED -> FROZEN:
+ Transition takes place if manually frozen or frozen by virtue of
+ reference from another package group.
+
+ FROZEN -> INSTALLED:
+ Manually unfrozen, or unfrozen by reference drop due to
+ change in formerly referring package group.
+
+ INSTALLED -> PRESERVED:
+ Old copies moved aside during upgrade of package components, but
+ not removed.
+
+ PRESERVED -> DELETED:
+ Old copies removed.
+
+ DELETED -> OUT_CATALOG:
+ Package has been removed from client catalog. Client software
+ would take a PRESERVED package through DELETED automatically to
+ reach OUT_CATALOG.
+
+ PRESERVED -> INSTALLED:
+ Package reinstalled or reversed.
+
+ DELETED -> INSTALLED:
+ Package reinstalled.
+
+ XXX How does the ZFS snapshot (that we might have taken prior to an
+ operation) get represented in the state? Is there an image state
+ machine model as well?
+
+ XXX Need a substate of INSTALLED for damaged packages.
+
+ XXX Need a substate of INSTALLED for packages where the global zone
+ portion is available, but local installation has not finished. Can
+ we generalize this state for all diskless installs?
+
diff --git a/doc/pkg5_docs/guide-publication-protocol.rst b/doc/pkg5_docs/guide-publication-protocol.rst
new file mode 100644
index 0000000..f09b310
--- /dev/null
+++ b/doc/pkg5_docs/guide-publication-protocol.rst
@@ -0,0 +1,104 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+ - add
+ Version 0:
+ A POST operation that adds content to an in-flight transaction for
+ the Transaction ID specified. This could either be file content
+ for the package or metadata about the package.
+
+ This data is not added to the repository for retrieval until a close
+ operation for the specified Transaction ID is executed.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/add/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
+
+ HEADERS:
+ X-IPkg-SetAttr1: description=Package Name
+
+ REQUEST BODY:
+
+ Expects:
+ A Transaction ID as output by pkgsend(1) in the request path.
+ The file content (if applicable), to be added, in the request
+ body. Any attributes to be set in the headers in the pattern
+ of:
+
+ X-IPkg-SetAttr{integer}: attr=value
+
+ Returns:
+ Response status of 200 on success; any other status indicates
+ failure.
+
+ - abandon
+ Version 0:
+ A GET operation that aborts an in-flight transaction for the
+ Transaction ID specified. This will discard any data related to
+ the transaction.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
+
+ Expects:
+ A Transaction ID as output by pkgsend(1) in the request path.
+
+ Returns:
+ Response status of 200 on success; any other status indicates
+ failure.
+
+ - close
+ Version 0:
+ A GET operation that ends an in-flight transaction for the
+ Transaction ID specified. If successful, the corresponding package
+ is added to the repository catalog and is immediately available to
+ repository users.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
+
+ Expects:
+ A Transaction ID as output by pkgsend(1) in the request path.
+
+ Returns:
+ Response status of 200 on success; any other status indicates
+ failure.
+
+ - open
+ Version 0:
+ A GET operation that starts an in-flight transaction for the
+ package FMRI specified.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/open/0/system%2Flibc@0.1-98
+
+ Expects:
+ A URL-encoded pkg(5) FMRI (excluding timestamp).
+
+ Returns:
+ Response status of 200 on success and an identifier for the new
+ transaction in the 'Transaction-ID' response header; any other
+ status indicates failure.
+
diff --git a/doc/pkg5_docs/guide-repository-format.rst b/doc/pkg5_docs/guide-repository-format.rst
new file mode 100644
index 0000000..96c4ee4
--- /dev/null
+++ b/doc/pkg5_docs/guide-repository-format.rst
@@ -0,0 +1,106 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+.. _Repository file system layout:
+
+Repository file system layout
+=============================
+
+File system Layout
+------------------
+
+The types of information that the depot server stores and/or retrieves can
+be categorized as follows:
+
+ - depot data
+ This includes: configuration data, presentation content (such as
+ web page templates), publishing data (e.g. in-flight transactions),
+ and temporary data (e.g. the feed cache).
+
+ - repository data
+ This includes: catalog information, package content (files), package
+ metadata (manifests), and search data.
+
+Layout
+~~~~~~
+
+ The depot server uses the following 'root' directory structures for the
+ storage and retrieval of depot and repository data:
+
+ - repo_dir (depot and repository data)
+ cfg_cache (depot data)
+ A file containing the cached configuration information for the
+ depot server.
+
+ catalog/ (repository data)
+ This directory contains the repository catalog and its related
+ metadata.
+
+ file/ (repository data)
+ This directory contains the file content of packages in the
+ repository.
+
+ Files are stored using a two-level path fragment, derived from the
+ SHA1-hash of a file's content, assumed to have at least 8 distinct
+ characters.
+
+ Example:
+ 00/
+ 0023bb/
+ 000023bb53fdc7bcf35e62b7b0b353a56d36a504
+
+ index/ (repository data)
+ This directory contains the search indices for the repository.
+
+ pkg/ (repository data)
+ This directory contains the metadata (manifests) for the
+ repository's packages.
+
+ The manifests for each package are stored in a directory with the
+ same name as the package stem using a URL-encoded filename.
+
+ Example:
+ entire/
+ 0.5.11%2C5.11-0.86%3A20080422T234219Z
+
+ trans/ (depot data)
+ This directory contains in-flight transactions for packages that
+ are waiting for the publication process to complete so that they
+ can be added to the repository's catalog.
+
+ Each transaction is stored in a directory named after the pending
+ transaction id and contains the manifest waiting for publication
+ to finish stored with the filename of 'manifest'.
+
+ Example:
+ 1229379580_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081215T221940Z/
+ manifest
+
+ updatelog/ (repository data)
+ This directory contains metadata detailing changes to the repository
+ by publishing operations.
+
+ - content_root (depot data)
+
+ web/
+ This directory contains all of the web presentation content for the
+ depot.
diff --git a/doc/pkg5_docs/guide-retrieval-protocol.rst b/doc/pkg5_docs/guide-retrieval-protocol.rst
new file mode 100644
index 0000000..519e2c0
--- /dev/null
+++ b/doc/pkg5_docs/guide-retrieval-protocol.rst
@@ -0,0 +1,228 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+
+Depot Operations
+----------------
+
+ - versions
+ Version 0:
+ A GET operation that retrieves text data representing what operations
+ are supported and version information about the depot server.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/versions/0/
+
+ Expects:
+ Nothing
+
+ Returns:
+ text/plain data containing the version of the pkg(5) software that
+ the depot is based upon, a list of the operations currently
+ supported, and the protocol version supported for each
+ operation.
+
+ Sample Output:
+ pkg-server bfc04991436e
+ info 0
+ search 0
+ versions 0
+ catalog 0
+ manifest 0
+ add 0
+ file 0
+ abandon 0
+ close 0
+ open 0
+
+Metadata Operations
+-------------------
+
+ - catalog
+ Version 0:
+ A GET operation that retrieves a text/plain datastream
+ representing a complete catalog or an incremental update to an
+ existing one as requested.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/catalog/0/
+
+ Expects:
+ Nothing or the following headers:
+ If-Modified-Since: {ISO 8601 formatted date and time in UTC}
+
+ Returns:
+ Either the contents of a pkg(5) catalog file, or the entries
+ that were added since the specified date as they are found
+ in the catalog file, separated by newlines.
+
+ - info
+ Version 0:
+ A GET operation that retrieves a text/plain description of a
+ package and its licensing information specified by the provided
+ FMRI.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/info/0/entire@0.5.11,5.11-0.101:20081119T235706Z
+
+ Expects:
+ A URL-encoded pkg(5) FMRI, excluding the 'pkg:/' scheme prefix
+ and publisher information, and including the full version
+ information.
+
+ Returns:
+ A text/plain representation of the specified package and its
+ licensing information.
+
+ Sample Output:
+ Name: entire
+ Summary: entire incorporation
+ Publisher: Unknown
+ Version: 0.5.11
+ Build Release: 5.11
+ Branch: 0.101
+ Packaging Date: Wed Nov 19 23:57:06 2008
+ Size: 0.00 B
+ FMRI: pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z
+
+ License:
+
+ - manifest
+ Version 0:
+ A GET operation that retrieves the contents of the manifest file for
+ a package specified by the provided FMRI.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/manifest/0/entire@0.5.11,5.11-0.101:20081119T235706Z
+
+ Expects:
+ A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix
+ and publisher information and including the full version
+ information.
+
+ Returns:
+ The contents of the package's manifest file.
+
+ - p5i
+ Version 0:
+ A GET operation that retrieves an application/vnd.pkg5.info
+ datastream representing publisher and package information.
+ This is intended for consumption by clients for the purposes
+ of auto-configuration, metadata management policy determination,
+ and triggering packaging operations such as installation.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/release/p5i/0/SUNWcs
+
+ Expects:
+ A full or partial URL-encoded pkg(5) FMRI, excluding the
+ publisher prefix. If the partial or full FMRI is valid, it will
+ be added to the datastream as is. If it includes the wildcard
+ character '*', a search of the repository's catalog for matching
+ entries will be performed and the unique set of resulting
+ package stems will be added to the datastream. If no match is
+ found, a 404 error will be raised.
+
+ Returns:
+ Returns a pkg(5) information datastream based on the repository
+ configuration's publisher information and the provided full or
+ partial FMRI or matching entries. The Content-Type of the
+ response is 'application/vnd.pkg5.info'.
+
+ - publisher
+ Version 0:
+ A GET operation that retrieves an application/vnd.pkg5.info
+ datastream representing publisher information. This is intended
+ for consumption by clients for auto-configuration and metadata
+ management policy determination.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/release/publisher/0
+
+ Expects:
+ Nothing
+
+ Returns:
+ Returns a pkg(5) information datastream based on the repository
+ configuration's publisher information. The Content-Type of the
+ response is 'application/vnd.pkg5.info'.
+
+ - search
+ Version 0:
+ A GET operation that retrieves a text/plain list of packages with
+ metadata that matches the specified criteria.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/release/search/0/vim
+
+ Expects:
+ A URL-encoded token representing the search criteria.
+
+ Returns:
+ A text/plain list of matching entries, separated by newlines.
+ Each entry consists of a set of four space-separated values:
+
+ index - what search index the entry was found in
+
+ action - what package action the entry is related to
+
+ value - the value that the matched the search criteria
+
+ package - the fmri of the package that contains the match
+
+ Results are streamed to the client as they are found.
+
+ Sample Output:
+ basename pkg:/SUNWvim@7.1.284,5.11-0.101:20081119T230659Z dir usr/share/vim
+ basename pkg:/SUNWvim@7.1.284,5.11-0.93:20080708T171331Z file usr/bin/vim
+
+Content Operations
+------------------
+
+ The pkg.depotd(5) server provides the following operations for retrieving
+ package content:
+
+ - file
+ Version 0:
+ A GET operation that retrieves the contents of a file, belonging to a
+ package, using a SHA-1 hash of the file's content.
+
+ Example:
+ URL:
+ http://pkg.opensolaris.org/release/file/0/
+ a00030db8b91f85d0b7144d0d4ef241a3f1ae28f
+
+ Expects:
+ A SHA-1 hash of the file's content belonging to a package in the
+ request path.
+
+ Returns:
+ The contents of the file, compressed using the gzip compression
+ algorithm.
+
diff --git a/doc/pkg5_docs/guide-txn-states.rst b/doc/pkg5_docs/guide-txn-states.rst
new file mode 100644
index 0000000..bdf010f
--- /dev/null
+++ b/doc/pkg5_docs/guide-txn-states.rst
@@ -0,0 +1,114 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+.. :vim set expandtab:
+
+.. _Transaction States:
+
+Transaction states
+==================
+
+On a depot open for publication, a new package version may be in the
+process of publication. As the package version is built up, it goes
+through a series of *transaction states*. We may also refer to these as
+"server states".
+
+We phrase the state machine in terms of a single removal state,
+ABANDONED, which covers both the never-created package instance (even
+with a series of never-finished transaction events). It may be more
+appropriate to separate the ABANDONED state into TX_ABANDONED and
+PKG_DELETED.
+
+This summary leaves us with a state transition diagram like::
+
+ 0
+ |
+ |
+ v
+ +--> TRANSACTING --> ABANDONED <--+
+ | | ^ |
+ | | | |
+ | v | |
+ | SUBMITTED ----> INCOMPLETE |
+ | | | |
+ | | | |
+ +--- PUBLISHED <---------+ |
+ | |
+ | |
+ +------------------------+
+
+ 0 -> TRANSACTING
+ On initial package creation.
+
+ TRANSACTING -> ABANDONED
+ If initial package transaction never committed, commitment
+ failed, or explicitly dropped.
+
+ TRANSACTING -> SUBMITTED
+ On successful package transaction commitment. Packages with
+ syntax errors or immediate inconsistencies would have failed in
+ commitment.
+
+ SUBMITTED:
+ The package modified by the transaction is known by a specific
+ version. Its contents are in the repository.
+
+ SUBMITTED -> INCOMPLETE
+ If commitment included a deferred inconsistency (package
+ dependency is the only expected form), then the package is left
+ in the incomplete state.
+
+ INCOMPLETE:
+ The package with the specific version string is on the
+ incomplete list. Its contents are in the repository.
+
+ INCOMPLETE -> ABANDONED
+ If incomplete package explicitly removed. (Possibly by
+ timeout from arrival in INCOMPLETE.)
+
+ SUBMITTED -> PUBLISHED
+ If commitment had no deferred inconsistencies, then the package
+ is considered ready for publication.
+
+ INCOMPLETE -> PUBLISHED
+ If the deferred inconsistencies, upon reevaluation, are now held
+ to no longer be inconsistent, then the package is considered
+ ready for publication.
+
+ PUBLISHED:
+ The package with the specific version string is present in the
+ catalog. Its contents remain in the repository.
+
+ PUBLISHED -> ABANDONED
+ On manual request for package decommissioning, the package will
+ be moved to the abandoned state.
+
+ ABANDONED:
+ A package with a specific version string is no longer in the
+ catalog or on the incomplete list. Its contents, if they were
+ in the repository, should be removed from the repository.
+
+ XXX ARCHIVED might be a special state connected to PUBLISHED, or
+ merely a substate. An archived package has its manifest and
+ contents in the repository, but cannot be installed on a client.
+ The point of including ARCHIVED packages is to allow client
+ deduction on a system not installed by the pkg system.
diff --git a/doc/pkg5_docs/history.txt b/doc/pkg5_docs/history.txt
new file mode 100644
index 0000000..e17ddcc
--- /dev/null
+++ b/doc/pkg5_docs/history.txt
@@ -0,0 +1,221 @@
+
+pkg
+HISTORY
+
+1. Summary
+
+The intent of history is to enable pkg clients to maintain a record of all
+image-modifying operations that they perform.
+
+2. Discussion
+
+2.1 Operations
+
+ The following operations are currently recorded by the clients that are
+ part of pkg(5):
+
+ add-publisher
+ image-create
+ image-set-attributes
+ image-update
+ install
+ purge-history
+ rebuild-index
+ refresh-publisher
+ remove-publisher
+ set-publisher
+ set-preferred-publisher
+ uninstall
+ update-publisher
+
+ These operations were chosen because they have the potential to alter the
+ behavior of pkg(5) operations or because they modify an image.
+
+3. History Entries
+
+3.1 Information Recorded
+
+ Each history entry contains a record of basic information about the client
+ performing the operation, along with a mix of required and optional
+ information about the operation performed.
+
+ The following information about the client is always recorded:
+
+ client_name - The name of the client performing the operation
+ (e.g. pkg, packagemanager)
+
+ client_version - The value of pkg.VERSION at the time the client
+ performed the operation (e.g. 2e5300c4f0a4+)
+
+ client_args - The command-line arguments used when executing the
+ client (e.g. /usr/bin/pkg install foo)
+
+ The following information about the operation performed is always recorded:
+
+ operation_name - The name of the operation performed (refer to 2.1)
+
+ start_time - When the operation started (e.g. 20080916T214726Z)
+
+ end_time - When the operation ended (e.g. 20080916T214800Z)
+
+ userid - The id of the user that performed the operation
+ (e.g. 0)
+
+ username - The username of the user that performed the operation
+ (e.g. root)
+
+ result - The outcome of the operation and the reason for it
+ (e.g. Failed, Transport) This maps to the pkg history
+ output columns of 'outcome' and 'reason'.
+
+ The following information about the operation will be recorded if provided
+ by the client or api:
+
+ start_state - Information about the operation requested before any
+ evaluation of that request is performed (e.g. an
+ image plan before evaluation)
+
+ end_state - Information about the operation requested after
+ evaluation of that request has been performed (e.g.
+ image plan after evaluation)
+
+ errors - Any errors that were encountered while performing the
+ operation (i.e. tracebacks, exceptions, etc.)
+
+ be - The name of the boot environment on which the
+ operation was performed
+
+ be_uuid - The uuid corresponding to the boot environment
+
+ new_be - The name of any new boot environment that was created
+ while performing the operation.
+
+ new_be_uuid - The uuid corresponding to the new boot environment
+
+ snapshot - The name of the snapshot that was taken as a result of
+ this operation. If the operation completed
+ successfully and the snapshot was destroyed, this
+ this information is not stored.
+
+3.2 Storage
+
+ Each history entry is recorded in a XML file located in the metadata
+ directory of an image (e.g. /var/pkg) in a directory named "history".
+ Each XML file is named after the pattern %Y%m%dT%H%M%SZ-sequence.xml.
+ Where sequence is a numeric value appended so that when multiple
+ operations are performed on an image within the same second (e.g. -02,
+ -03, etc.) entries are still written correctly.
+
+3.3 File Format
+
+ It should be noted that this format description is that of a private
+ interface and is subject to change.
+
+ History entries are recorded in a XML file with a fairly simplistic
+ structure. The format is as follows:
+
+ The first line of each file is a standard xml header along with the
+ encoding used to save the information. This allows the pkg history
+ command to correctly display this information if the client changes
+ locales later.
+
+
+
+ The second element of every history XML file is a root element used to
+ contain the client and operation information. Only one per file ever
+ occurs.
+
+
+
+ The client element has a name and version attribute used to record
+ client_name and client_version. Only one per file ever occurs.
+
+
+
+ The args element contains one or more "arg" (argument) elements
+ that match each element of the system argument list used to
+ execute the client. It has no attributes. Only one per file
+ ever occurs.
+
+
+
+ Each arg element contains a CDATA element to encapsulate
+ the raw information for the argument. It has no attributes.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ The operation element has the following attributes: name,
+ start_time, end_time, userid, and username. Optional attributes
+ are be, snapshot and new_be, indicating the boot environment the
+ operation was applied to, the snapshot taken during this operation,
+ and any new boot environment created as a result.
+ It can also contain the following, optional elements: start_state,
+ end_state, and errors. It only occurs once per file.
+
+
+
+ The start_state element is used to store information about the
+ operation performed before a request is evaluated (e.g. image
+ plan before evaluation). It always contains a CDATA element with
+ the related information and only occurs once per file. It has no
+ attributes.
+
+
+
+
+
+ The end_state element is used to store information about the
+ operation performed after a request is evaluated (e.g. image
+ plan after evaluation). It always contains a CDATA element with
+ the related information and only occurs once per file. It has no
+ attributes.
+
+
+ pkg:/SUNWvim@7.1.284,5.11-0.96:20080825T192756Z
+ None -> pkg:/SUNWcsl@0.5.11,5.11-0.96:20080825T183047Z
+ None -> pkg:/SUNWpool@0.5.11,5.11-0.96:20080825T191747Z
+ None -> pkg:/SUNWlxml@2.6.31,5.11-0.96:20080825T191518Z
+ None -> pkg:/SUNWzlib@1.2.3,5.11-0.96:20080825T193230Z
+ None -> pkg:/SUNWlibms@0.5.11,5.11-0.96:20080825T191411Z
+ None -> pkg:/SUNWopenssl@0.9.8,5.11-0.96:20080825T194948Z
+ None -> pkg:/SUNWlibsasl@0.5.11,5.11-0.96:20080825T191417Z
+ None -> pkg:/SUNWpr@0.5.11,5.11-0.96:20080825T192030Z
+ None -> pkg:/SUNWtls@0.5.11,5.11-0.96:20080825T192639Z
+ ]]>
+
+ The errors element contains one or more "error" elements
+ that encapsulate the information about each error that was
+ recorded during the operation. It has no attributes, and
+ only occurs once per file.
+
+
+
+ Each error element contains a CDATA element to encapsulate
+ the raw information about the error. It has no attributes.
+
+
+
+
+
+
+
+
diff --git a/doc/pkg5_docs/image.rst b/doc/pkg5_docs/image.rst
new file mode 100644
index 0000000..6c45f8e
--- /dev/null
+++ b/doc/pkg5_docs/image.rst
@@ -0,0 +1,198 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+.. :vim:set expandtab:
+
+IMAGES
+------
+
+
+Image types
+~~~~~~~~~~~
+
+ Each collection of installed package instances is some kind of
+ image. We envision three kinds:
+
+ - entire images
+ An entire image contains (and can contain) all appropriate
+ packages from a repository. It has a machine type and release.
+
+ - partial images
+ A partial image contains (and can contain) all appropriate
+ packages from a repository. An partial image is tied to an
+ entire image and has an identified list of non-writeable
+ directory paths. It inherits its machine type and release.
+
+ - user images
+ A user image consists entirely of relocatable packages. It is
+ tied to an entire image for dependency satisfaction of
+ non-relocatable packages. It inherits its machine type and
+ release.
+
+Image configuration
+~~~~~~~~~~~~~~~~~~~
+
+Configuration inheritance
+`````````````````````````
+
+ Some aspects of configuration can be shared between all images. For
+ instance, a user image may specify publishers beyond those encoded
+ into the system defaults. So, a user image must have authoritative
+ configuration, but be able to draw on the parent image's state for
+ default settings.
+
+ Operations on partial images and non-live entire images may need to
+ access image configuration data when smf(5) for that image is
+ unavailable. So images of these kinds must have cached
+ configuration state.
+
+ Roughly, these aspects are sufficient to constrain our configuration
+ behaviour for user and entire images:
+
+ [user image]
+ look up local configuration
+ if undefined, request properties from svc://application/pkg
+ if unavailable, examine parent's configuration cache
+ if undefined or unavailable, use hard-coded default
+
+ [entire image]
+ request properties from svc://application/pkg
+ if unavailable, examine configuration cache
+ if undefined or unavailable, use hard-coded default
+
+ Partial images could have differing behaviour depending on whether
+ the operation is invoked from within the partial image's "packaging
+ context" (as it would be for an operation issued from within the
+ zone), or outside it (operations on the entire image and its
+ associated partial images).
+
+ For the first case, the configuration strategy is the same as that
+ for an entire image. For the second case, we could do
+
+ [partial image, external context]
+
+ - examine partial image configuration cache
+
+ - if unavailable, request properties from svc://application/pkg
+
+ - if undefined or unavailable, examine entire image configuration
+ cache
+
+ - if undefined, use hard-coded default
+
+ For certain properties (or even certain packages), it may be
+ inappropriate to let the partial image configurations drift from
+ that of the entire image.
+
+Configuration components
+````````````````````````
+
+ List of publishers. For each publisher, we have a prefix, an
+ origin URL, a list of mirror URLs, and annotations.
+
+ publisher_[escaped_name]/ Property group of type "com.sun.pkg,publisher"
+ /prefix pkg: publisher
+ /origin http:, https:, or ftp: URL
+ /mirrors list of URLs
+ /disabled boolean indicating whether publisher should be used
+
+ Image properties. The image has a collection of simple properties,
+ like machine type and policies that control the behavior of the pkg(5)
+ software within that image. Policies are properties with a boolean value.
+
+ properties/
+ /pursue-latest (not currently used)
+ /require-optional (not currently used)
+ /display-copyrights (not currently used)
+ /flush-content-cache-on-success
+ should downloaded compressed files be removed
+ after a successful install
+ /preferred-publisher preferred publisher for unknown package lookups
+ /title title of image for use in GUIs and other UIs
+ /description longer description of the content of the image
+
+ /[various]
+
+ Entire images have a property group for each tied partial image.
+
+ partial_[escaped_name]/ Property group of type "com.sun.pkg,partial"
+
+ /path Filesystem path
+
+ /vdomainname Defined if this image is a virtual
+ domain controlled on this system.
+
+ /vdomaintype "xen", "zone", ...
+
+ (XXX Should we instead assume that each of Zones and Xen will
+ acquire service FMRIs per zone/domain?)
+
+
+Image-filesystem interactions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ The intent is to utilize snapshot-capable filesystems to provide a
+ rollback mechanism that includes both the pkg(1M)-driven
+ modifications as well as subsequent modifications from configuration
+ methods, etc. In most cases, we are considering a filesystem with
+ capabilities similar to ZFS.
+
+ XXX Is fssnap_ufs(1M) sufficient to build something meaningful for
+ UFS?
+
+ With appropriate policies set, the image plan, prior to execution
+ must snapshot the involved filesystems.
+
+ There seem to be two options:
+
+ 1. The image can build its list of filesystems and types (plus
+ other attributes, like mutability.
+
+ 2. A list of filesystems is given to each package plan, the package
+ plan then evaluates its proposed actions against that list, and
+ offers a method to return the affected subset of the list.
+
+ In this case, we must also determine whether or not we are
+ restricted to clones (because one or more packages in the image
+ plan require kernel-restart) or are potentially live.
+
+ XXX Either of these approaches is applicable in the image/substrate
+ model, where the substrate owns the low-level properties specific to
+ a particular "place to write data".
+
+ In the case that one or more filesystems in the image is or are not
+ capable of snapshots, we have two choices:
+
+ - take no snapshot, as image-revert not possible in any safe or
+ complete sense
+
+ - take a set of snapshots that lead to a revert point that requires
+ manual forcing
+
+ We must warn about images of this kind, unless policy settings allow
+ us otherwise. Since we want to allow and understand "zfs
+ split"-style operations [1], we also need to determine if a snapshot
+ set taken before a split can sensibly be restored after the split
+ operation.
+
+
+[1] (The zfs split RFE.)
diff --git a/doc/pkg5_docs/image.txt b/doc/pkg5_docs/image.txt
new file mode 100644
index 0000000..9247bf6
--- /dev/null
+++ b/doc/pkg5_docs/image.txt
@@ -0,0 +1,170 @@
+
+pkg
+IMAGES
+
+:vim:set expandtab:
+
+1. Summary
+
+
+2. Discussion
+
+2.1. Image types
+
+ Each collection of installed package instances is some kind of
+ image. We envision three kinds:
+
+ - entire images
+ An entire image contains (and can contain) all appropriate
+ packages from a repository. It has a machine type and release.
+
+ - partial images
+ A partial image contains (and can contain) all appropriate
+ packages from a repository. An partial image is tied to an
+ entire image and has an identified list of non-writeable
+ directory paths. It inherits its machine type and release.
+
+ - user images
+ A user image consists entirely of relocatable packages. It is
+ tied to an entire image for dependency satisfaction of
+ non-relocatable packages. It inherits its machine type and
+ release.
+
+2.2. Image configuration
+
+2.2.1. Configuration inheritance
+
+ Some aspects of configuration can be shared between all images. For
+ instance, a user image may specify publishers beyond those encoded
+ into the system defaults. So, a user image must have authoritative
+ configuration, but be able to draw on the parent image's state for
+ default settings.
+
+ Operations on partial images and non-live entire images may need to
+ access image configuration data when smf(5) for that image is
+ unavailable. So images of these kinds must have cached
+ configuration state.
+
+ Roughly, these aspects are sufficient to constrain our configuration
+ behaviour for user and entire images:
+
+ [user image]
+ look up local configuration
+ if undefined, request properties from svc://application/pkg
+ if unavailable, examine parent's configuration cache
+ if undefined or unavailable, use hard-coded default
+
+ [entire image]
+ request properties from svc://application/pkg
+ if unavailable, examine configuration cache
+ if undefined or unavailable, use hard-coded default
+
+ Partial images could have differing behaviour depending on whether
+ the operation is invoked from within the partial image's "packaging
+ context" (as it would be for an operation issued from within the
+ zone), or outside it (operations on the entire image and its
+ associated partial images).
+
+ For the first case, the configuration strategy is the same as that
+ for an entire image. For the second case, we could do
+
+ [partial image, external context]
+ examine partial image configuration cache
+ if unavailable, request properties from svc://application/pkg
+ if undefined or unavailable, examine entire image configuration
+ cache
+ if undefined, use hard-coded default
+
+ For certain properties (or even certain packages), it may be
+ inappropriate to let the partial image configurations drift from
+ that of the entire image.
+
+2.2.2. Configuration components
+
+ List of publishers. For each publisher, we have a prefix, an
+ origin URL, a list of mirror URLs, and annotations.
+
+ publisher_[escaped_name]/ Property group of type "com.sun.pkg,publisher"
+ /prefix pkg: publisher
+ /origin http:, https:, or ftp: URL
+ /mirrors list of URLs
+ /disabled boolean indicating whether publisher should be used
+
+ Image properties. The image has a collection of simple properties,
+ like machine type and policies that control the behavior of the pkg(5)
+ software within that image. Policies are properties with a boolean value.
+
+ properties/
+ /pursue-latest (not currently used)
+ /require-optional (not currently used)
+ /display-copyrights (not currently used)
+ /flush-content-cache-on-success
+ should downloaded compressed files be removed
+ after a successful install
+ /preferred-publisher preferred publisher for unknown package lookups
+ /title title of image for use in GUIs and other UIs
+ /description longer description of the content of the image
+ /[various]
+
+ Entire images have a property group for each tied partial image.
+
+ partial_[escaped_name]/ Property group of type "com.sun.pkg,partial"
+ /path Filesystem path
+ /vdomainname Defined if this image is a virtual
+ domain controlled on this system.
+ /vdomaintype "xen", "zone", ...
+
+ (XXX Should we instead assume that each of Zones and Xen will
+ acquire service FMRIs per zone/domain?)
+
+
+2.3 Image-filesystem interactions
+
+ The intent is to utilize snapshot-capable filesystems to provide a
+ rollback mechanism that includes both the pkg(1M)-driven
+ modifications as well as subsequent modifications from configuration
+ methods, etc. In most cases, we are considering a filesystem with
+ capabilities similar to ZFS.
+
+ XXX Is fssnap_ufs(1M) sufficient to build something meaningful for
+ UFS?
+
+ With appropriate policies set, the image plan, prior to execution
+ must snapshot the involved filesystems.
+
+ There seem to be two options:
+
+ 1. The image can build its list of filesystems and types (plus
+ other attributes, like mutability.
+
+ 2. A list of filesystems is given to each package plan, the package
+ plan then evaluates its proposed actions against that list, and
+ offers a method to return the affected subset of the list.
+
+ In this case, we must also determine whether or not we are
+ restricted to clones (because one or more packages in the image
+ plan require kernel-restart) or are potentially live.
+
+ XXX Either of these approaches is applicable in the image/substrate
+ model, where the substrate owns the low-level properties specific to
+ a particular "place to write data".
+
+ In the case that one or more filesystems in the image is or are not
+ capable of snapshots, we have two choices:
+
+ - take no snapshot, as image-revert not possible in any safe or
+ complete sense
+
+ - take a set of snapshots that lead to a revert point that requires
+ manual forcing
+
+ We must warn about images of this kind, unless policy settings allow
+ us otherwise. Since we want to allow and understand "zfs
+ split"-style operations [1], we also need to determine if a snapshot
+ set taken before a split can sensibly be restored after the split
+ operation.
+
+
+3. References
+
+[1] (The zfs split RFE.)
diff --git a/doc/pkg5_docs/kvm.md b/doc/pkg5_docs/kvm.md
new file mode 100644
index 0000000..eef90d2
--- /dev/null
+++ b/doc/pkg5_docs/kvm.md
@@ -0,0 +1,183 @@
+
+# KVM branded-zone support
+
+KVM branded zones are configured mainly via custom attributes in the zone
+configuration.
+
+To get started, `pkg install system/zones/brand/kvm` and configure a zone with the
+kvm brand and the appropriate attributes; see the example zone at the end of
+this page.
+
+To troubleshoot problems if the zone fails to start, review the log file
+which will be created at `/path/to/zone/root/tmp/init.log`
+
+### Attributes
+
+| Attribute | Default | Syntax | Example
+| --- | --- | --- | ---
+| bootdisk1 | | path[,serial=] | tank/hdd/kvm1
+| bootorder | cd | \[c\]\[d\]\[n\]
+| cdrom3 | | path to ISO | /data/iso/FreeBSD-11.1-RELEASE-amd64-bootonly.iso
+| cpu | qemu64 |
+| console | pipe,id=console0,path=/dev/zconsole4 | options |
+| disk1 | | path[,serial=] | tank/hdd/kvm2,serial=1234
+| diskN2 | | path[,serial=] | tank/hdd/kvm2,serial=1234
+| diskif | virtio | virtio,ahci
+| netif | virtio-net-pci | virtio-net-pci,e1000
+| ram | 1G | n(G\|M) | 8G
+| type | generic | generic
+| vcpus | 1 | n | 16
+| vnc4 | off | off,on,options | unix:/tmp/vm.vnc
+| extra | | extra arguments for hypervisor |
+
+#### Notes
+
+
+- You will also need to pass the underlying disk device through to the zone via a device entry, see the example below;
+- Use diskN to specify the slot into which the disk will be placed. A plain disk tag will be put in the lowest available slot.
+- The ISO file needs passing through to the zone via a lofs mount, see the example below;
+- Setting vnc to on is the same as setting it to unix=/tmp/vm.vnc.
+- You can connect to the virtual machine console from the global zone with zlogin -C zonename;
+
+
+### Example zone
+
+The following example zone is shown twice, once in info format and once in
+export (showing the necessary commands for creation). Note that the example
+shows setting the `allowed-address` attribute for the network interface -
+this does not configure the address within the virtual machine but rather
+prevents the use of any other address (L3 protection).
+
+```
+bloody# zonecfg -z oi info
+zonename: oi
+zonepath: /data/zone/oi
+brand: kvm
+autoboot: false
+bootargs:
+pool:
+limitpriv:
+scheduling-class:
+ip-type: exclusive
+hostid:
+fs-allowed:
+fs:
+ dir: /tank/iso/OI-hipster-minimal-20180427.iso
+ special: /tank/iso/OI-hipster-minimal-20180427.iso
+ raw not specified
+ type: lofs
+ options: [ro,nodevices]
+net:
+ address not specified
+ allowed-address: 10.0.0.112/24
+ defrouter not specified
+ global-nic not specified
+ mac-addr not specified
+ physical: oi0
+ vlan-id not specified
+device:
+ match: /dev/zvol/rdsk/tank/hdd/oi0
+device:
+ match: /dev/zvol/rdsk/tank/hdd/oi1
+device:
+ match: /dev/zvol/rdsk/tank/hdd/oi2
+attr:
+ name: vcpus
+ type: string
+ value: 16
+attr:
+ name: ram
+ type: string
+ value: 4G
+attr:
+ name: cdrom
+ type: string
+ value: /tank/iso/OI-hipster-minimal-20180427.iso
+attr:
+ name: vnc
+ type: string
+ value: on
+attr:
+ name: bootdisk
+ type: string
+ value: tank/hdd/oi0
+attr:
+ name: disk
+ type: string
+ value: tank/hdd/oi1
+attr:
+ name: disk1
+ type: string
+ value: tank/hdd/oi2,serial=1234
+```
+
+```
+bloody# zonecfg -z oi export
+create -b
+set zonepath=/data/zone/oi
+set brand=kvm
+set autoboot=false
+set ip-type=exclusive
+add fs
+set dir=/tank/iso/OI-hipster-minimal-20180427.iso
+set special=/tank/iso/OI-hipster-minimal-20180427.iso
+set type=lofs
+add options ro
+add options nodevices
+end
+add net
+set allowed-address=10.0.0.112/24
+set physical=oi0
+end
+add device
+set match=/dev/zvol/rdsk/tank/hdd/oi0
+end
+add device
+set match=/dev/zvol/rdsk/tank/hdd/oi1
+end
+add device
+set match=/dev/zvol/rdsk/tank/hdd/oi2
+end
+add attr
+set name=vcpus
+set type=string
+set value=16
+end
+add attr
+set name=ram
+set type=string
+set value=4G
+end
+add attr
+set name=cdrom
+set type=string
+set value=/tank/iso/OI-hipster-minimal-20180427.iso
+end
+add attr
+set name=vnc
+set type=string
+set value=on
+end
+add attr
+set name=bootdisk
+set type=string
+set value=tank/hdd/oi0
+end
+add attr
+set name=disk
+set type=string
+set value=tank/hdd/oi1
+end
+add attr
+set name=disk1
+set type=string
+set value=tank/hdd/oi2,serial=1234
+end
+```
+
+You can connect to kvm vga console of zone with vncviewer and socat. For example:
+```
+# socat TCP-LISTEN:5500 UNIX-CONNECT:/data/zone/oi/root/tmp/vm.vnc
+$ vncviewer localhost:5500
+```
+
diff --git a/doc/pkg5_docs/license.txt b/doc/pkg5_docs/license.txt
new file mode 100644
index 0000000..502348a
--- /dev/null
+++ b/doc/pkg5_docs/license.txt
@@ -0,0 +1,523 @@
+pkg(5): image packaging system
+
+LICENSE ACTIONS ACCEPTANCE PROPOSAL
+
+1. Overview
+
+ License actions provide a way to deliver the textual data
+ contained within licenses, copyright notices, disclaimers, or
+ other legally-related documents that need to be associated with
+ the contents of a package. They also provide a way for packages
+ to provide guidance to pkg(5) clients as to how this information
+ should be presented or if it requires acceptance by the user
+ before the package can be delivered into an image.
+
+ This proposal has the following core goals for the implementation
+ of license acceptance functionality:
+
+ * Support for non-interactive and interactive package operations
+
+ * Enablement of package creators to provide guidance to pkg(5)
+ client API consumers as to how licensing or other informational
+ data should be presented and interacted with
+
+ * Enablement of administrators and users to query and report on
+ the licensing or other informational data related to packages
+ contained within an image
+
+ * Enablement of administrators and users to restrict what packages
+ can be delivered into an image based on the package's provided
+ licensing information.
+
+ To achieve these goals, changes must be made to the following
+ pkg(5) components:
+
+ * License Actions
+
+ * Image configuration
+
+ * Client API
+
+ * BUI: pkg.depotd(1m)
+
+ * CLI: pkg(1), pkgsend(1)
+
+ * GUI: packagemanager(1), updatemanager(1)
+
+ This proposal omits the GUI programs as a separate team will
+ develop and deliver the enhancements related to the changes
+ discussed here.
+
+2. License Action attribute changes and additions
+
+ Currently, the license attribute of license actions is not
+ restrictive enough in what characters are allowed for the
+ name of the license. To ensure cross-platform compatibility
+ and consistent naming, it is proposed that the license
+ attribute's definition be amended as follows:
+
+ license This attribute provides a description for the license
+ to help describe the license in a meaningful way. Some
+ examples might include:
+
+ - "GPL v2 only"
+ - "GPL with Specific Exception"
+
+ A recommended list of descriptions for common open-
+ source licenses, ang guidelines for custom licenses
+ will be added to the license action documentation to
+ encourage consistency. This value must be unique
+ within a package.
+
+ To support license acceptance functionality, new attributes for
+ license actions are needed to allow packages to provide guidance
+ to pkg(5) API consumers:
+
+ class This optional attribute is primarily for use in
+ filter and query operations. It is intended that
+ only short, descriptive text be used here. As
+ such, the content of this value will be limited to
+ the characters [A-Za-z][A-Za-z0-9 _-.,]*. While
+ this value is optional, its usage is strongly
+ encouraged, and so a recommended list of classes
+ for common open-source licenses, and guidelines
+ for custom licenses will be added to the license
+ action documentation. This value must be unique
+ within a package.
+
+ must-accept A boolean value indicating whether this license
+ must be accepted by a user before the related
+ package can be installed or updated. Acceptable
+ values are "true" or "false". Omission of this
+ attribute must be considered equivalent to
+ "false". The reason for this is that many
+ licenses (especially copyleft ones) do not
+ require this and clients should not prompt for it
+ unless provided guidance to do so by the package
+ creator.
+
+ must-display A boolean value indicating whether the content of
+ the action's payload must be displayed by clients
+ during packaging operations. Omission of this
+ value must be considered equivalent to "false".
+
+3. Image configuration additions
+
+ Note that changes proposed in this section implicitly depend on
+ on upcoming changes to the image configuration format for the
+ client. As such, how policies, etc. are stored is not discussed
+ here other than to note that they will be stored within the image
+ configuration.
+
+ To enable administrators and users to effectively manage the
+ packages contained within an image, users need to be able to
+ define, on a per-publisher basis, what the behaviour of the
+ packaging system and clients should be in the following key
+ areas related to licenses:
+
+ * filtering
+
+ Administrators and users need to be able to define a policy
+ that can be used to determine what packages are delivered into
+ an image based on licensing information defined in packages.
+
+ * acceptance
+
+ Administrators and users need to be able to define a policy
+ that can be used to determine what behaviour clients should
+ exhibit when encountering a license that requires explicit
+ acceptance.
+
+ It is believed that a clear delineation between image properties
+ (which describe the image itself) and policies (which provide
+ guidance to pkg(5) clients) will be helpful to end-users from a
+ usage and documentation standpoint. As such, a new policy system
+ has been devised to allow users to provide guidance to the pkg(5)
+ system and clients on a per-publisher (and/or global) level based
+ on the policy in question. Per-publisher values will override
+ the global value based on the publisher of the package that is
+ being evaluated by the client API.
+
+ The following new policies will be created, and can all be set at
+ a global or per-publisher level:
+
+ license-policy A keyword indicating what the behaviour of clients
+ should be when the license of a package requires
+ acceptance. The following keywords are supported:
+
+ accept Automatically accepts any license with
+ must-accept=true after license
+ filtering has been applied.
+
+ decline Automatically declines any license
+ with must-accept=false after license
+ filtering has been applied.
+
+ explicit Requires explicit acceptance of any
+ licenses with must-accept=true by the
+ user after licensing filtering has
+ been applied. This could be
+ implemented as an interactive prompt,
+ or by a failure of the client with a
+ requirement to pass an explicit
+ command-line option for acceptance.
+ This is the default value.
+
+ license-accept A list of values that will be used to mark any
+ license with a matching description or class as
+ accepted automatically if they require acceptance.
+ This value is undefined by default.
+
+ license-decline A list of values that will be used to mark any
+ license with a matching description or class as
+ declined automatically, regardless of whether they
+ require acceptance. This value is undefined by
+ default.
+
+ license-display A keyword indicating what the behaviour of the
+ client should be when displaying the text of
+ license actions. The following keywords are
+ supported:
+
+ all Suggests clients display the text of all
+ license actions, but must display the text
+ of license actions that have
+ must-display=true.
+
+ auto The default value for the image
+ configuration, which indicates that
+ clients must display the text of license
+ actions that have must-display=true.
+
+ The following, existing image properties will become policies and
+ can only be set on a global basis:
+
+ flush-content-cache-on-success
+ pursue-latest
+ require-optional
+
+ The following, existing image properties will become policies and
+ can be set on a global or per-publisher basis:
+
+ send-uuid
+
+ The following, existing image properties will be removed as they
+ were never implemented and are being replaced by functionality
+ discussed in this proposal:
+
+ display-copyrights
+
+ Finally, it should be noted that the existing 'property' subcommands
+ cannot be used to alter or view policies and the proposed 'policy'
+ subcommands likewise cannot be used to alter or view properties.
+
+4. Client API
+
+4.1 pkg.client.api
+
+ The client API, to enable license acceptance functionality
+ enforcement and control for clients, will need to change in
+ the following ways:
+
+ * Plan creation will be changed to analyze and determine license
+ acceptance and or build a list of licenses and their acceptance
+ status for packages being installed or updated.
+
+ * The ImagePlan object will have a new method named 'get_licenses'
+ added. After plan creation is finished, consumers may call this
+ method to retrieve a list of tuples containing a LicenseInfo
+ object, whether the license is allowed by image policy, and the
+ current acceptance status of the license as detailed later. The
+ default acceptance status of a license will be 'declined' unless
+ otherwise defined by image policy or operation policy.
+
+ * The ImagePlan object will have a new method named
+ 'set_license_status' which will allow callers to mark the explicit
+ acceptance of a package's license by a user:
+
+ set_license_status(fmri, class=None, description=None, status)
+
+ Either 'class' or 'description' must be provided.
+
+ * To allow pkg.client.api consumers access to set_license_status
+ and get_licenses(), equivalent wrapper functions will be added
+ to the PlanDescription object provided by the API.
+
+ * Two new exceptions will be added to pkg.client.api_errors. The
+ first will be named 'PlanExecutionError' and be used as a base
+ exception class for all plan execution errors. The second
+ exception will be named 'PlanExecutionLicenseError' and will
+ inherit from the first exception, while being used to indicate
+ that a licensing related error occurred.
+
+ * Plan execution will be changed to verify that each license
+ that has must-accept=True has been marked as accepted. If
+ any license has not been marked as accepted, and requires
+ acceptance, a 'PlanExecutionLicenseError' will be raised.
+
+ * Plan execution will be changed to record each package that is
+ part of the operation using pkg.client.history. The full FMRI,
+ the classes and descriptions identifying the licenses contained
+ within the package, and acceptance status of each license within
+ a package will be recorded. The following statuses will be used
+ to indicate a package's license acceptance:
+
+ * accepted
+ Indicates that the license was accepted through the API,
+ presumably by the user.
+
+ * accepted-policy
+ Indicates that the license was automatically accepted
+ based on image policy.
+
+ * declined
+ Indicates that the license required acceptance and was not
+ marked as accepted.
+
+ * declined-policy
+ Indicates that the license was automatically rejected
+ based on image policy.
+
+ * not-applicable
+ Indicates that the license did not require acceptance and
+ was not declined by policy.
+
+4.2 pkg.client.history
+
+ To ease logging of license acceptance information, and to
+ accurately reflect operation failure, the following changes will
+ be made to pkg.client.history:
+
+ * New operation result constants will be added:
+
+ RESULT_FAILED_LICENSE_DECLINED
+ Used to indicate that an operation failed because a
+ license that required acceptance was not marked as
+ accepted.
+
+ RESULT_FAILED_LICENSE_POLICY
+ Used to indicate that an operation failed because a
+ license was declined based on image policy.
+
+ * log_operation_error() will be changed to ensure that the new
+ 'PlanExecutionLicenseError' triggers an appropriate failure
+ result as noted above.
+
+ LOG_PKG_OP_INSTALL
+ LOG_PKG_OP_UPDATE
+ LOG_PKG_OP_REMOVE
+
+ * A new method to log license status information about a package
+ will be added:
+
+ log_pkg_license(class=None, description=None, status)
+
+ Either 'class' or 'description' must be provided.
+
+ * A new method to log operation information about a package will
+ be added:
+
+ log_pkg_operation(operation, old_fmri, new_fmri)
+
+5. BUI
+
+5.1. pkg.depotd(1m)
+
+ The pkg.depotd(1m) server that pkg(5) provides will be changed
+ such that its output matches that of the CLI when listing licenses
+ as much as possible, such as showing the "license" attribute of
+ a license action.
+
+6. CLI
+
+6.1. pkg(1)
+
+ The pkg(1) client that pkg(5) provides needs to be enhanced to
+ provide the following functionality:
+
+
+ * A mechanism to temporarily override client behaviour, provide
+ intial policy values during image-creation, or to provide
+ explicit user intent (such as license acceptance).
+
+ A new command-line option as shown below will be added to the
+ pkg(1) client that will allow its usage with applicable sub-
+ commands. Policy names not applicable to the corresponding
+ subcommand will effectively be ignored (such as pursue-latest
+ with the 'fix' subcommand).
+
+ --policy =
+
+ The following subcommands will allow this option:
+
+ * contents
+ * fix
+ * history
+ * image-update
+ * info
+ * install
+ * list
+ * uninstall
+ * verify
+
+ image-create will also support the --policy option, but instead of
+ being used as a temporary override, it will be used to specify the
+ permanent defaults for the image being created.
+
+ * Graceful, informative failure due to license-related exceptions
+
+ If a license-related exception occurs during plan execution, a
+ failure message similar to the following will be displayed:
+
+ The following packages are not permitted by image policy
+ due to the current image's license policies:
+
+ --------------------------------------------------
+ License A
+ --------------------------------------------------
+ Foo
+ Bar
+
+ The following packages contain licenses that require
+ acceptance before they can be installed (use --accept
+ to accept these licenses):
+
+ --------------------------------------------------
+ License B
+ --------------------------------------------------
+ Baz
+ Quux
+
+ Note that it may be possible for the same license to be marked
+ as requiring acceptance in one package and not another, and so
+ the display above may indicate the same license more than
+ once.
+
+ * Display of licenses that require it during packaging operations
+
+ pkg(1) will be changed to retrieve and display the text of
+ any licenses that must be displayed or have been requested
+ for display after plan creation. The default value is based
+ on image policy as noted in section 3.
+
+ The display of the license text will be the same as the
+ proposed output for the 'info' subcommand as noted below.
+
+ * Improvement of license display by 'info' subcommand
+
+ Currently, the info subcommand will display license text for
+ one or more given FMRIs but does so without any visual
+ separation other than a newline between distinct license text
+ output, does not indicate which package the license belongs
+ to, and does not provide any clear indication of the identity
+ of a license (e.g. CDDL, BSD, etc.).
+
+ The output of this subcommand will be changed so that the
+ description and class of the license will be shown and a visual
+ separator will be placed between the output of each license.
+ The revised output is anticipated to be similar to this:
+
+ ------------------------------------------------------------
+ Package: Foo
+ License: copyright
+
+ Copyright 2009 Example Company, Inc. All Rights Reserved.
+
+ ------------------------------------------------------------
+ Package: Foo
+ License: Termsv1 "Example Company's License Description v1"
+
+ Lorem ipsum dolor sit amet, consectetur adipiscing elit.
+ Vivamus in risus sem, id blandit justo. Maecenas nulla massa,
+ mollis sed lobortis a, placerat vel elit. Quisque eleifend leo
+ ipsum. Donec feugiat gravida molestie. Nulla facilisi. Nullam
+ sit amet ligula sed mauris tempor fermentum quis at purus.
+
+ ------------------------------------------------------------
+ Package: Bar
+ License: Termsv2 "Example Company's License Description v2"
+
+ Lorem ipsum dolor sit amet, consectetur adipiscing elit.
+ Vivamus in risus sem, id blandit justo. Maecenas nulla massa,
+ mollis sed lobortis a, placerat vel elit.
+
+ * A set-policy subcommand will be added to allow setting policy
+ values that functions as follows:
+
+ set-policy [-p publisher] policy-name policy-value ... ...
+
+ If -p is not provided, the global policy value will be changed.
+
+ One or more pairs of policy name and policy value may be provided
+ provided to this command to allow setting multiple values using a
+ single command invocation.
+
+ Some options may not be set at the publisher level and this will
+ result in an error message such as:
+
+ $ pkg set-policy -p opensolaris.org flush-content-cache-on-success
+ pkg: set-policy: flush-content-cache-on-success is a global policy
+ and cannot be set on a per-publisher basis.
+
+ * An unset-policy subcommand will be added to allow unsetting or
+ resetting policy values that functions as follows:
+
+ unset-policy [-p publisher] policy-name ...
+
+ If -p is not provided, the global policy value will be changed.
+
+ Some options may not be set at the publisher level and this will
+ result in an error message such as:
+
+ $ pkg unset-policy -p opensolaris.org \
+ flush-content-cache-on-success
+ pkg: unset-policy: flush-content-cache-on-success is a global
+ policy and cannot be unset on a per-publisher basis.
+
+ * A policy subcommand will be added for retrieving policy values
+ that functions as follows:
+
+ policy [-H] [-p publisher] policy-name ...
+
+ If -p is not provided, then all matching policy values will be
+ shown instead of those of a specific publisher. 'All' may be
+ specified to show global values that do not apply to a specific
+ publisher.
+
+ If -H is provided, headers will be omitted.
+
+ One or more policy names may be specified separated by whitespace.
+ If no policy-names are specified, all policies will be displayed.
+
+ Example output:
+
+ $ pkg policy
+ PUBLISHER POLICY VALUE
+ All license-accept
+ All license-decline
+ All license-display auto
+ All license-policy decline
+ opensolaris.org license-accept accept
+
+ $ pkg policy license-accept
+ PUBLISHER POLICY VALUE
+ All license-accept
+ opensolaris.org license-accept accept
+
+ $ pkg policy license-accept license-display
+ PUBLISHER POLICY VALUE
+ All license-accept
+ All license-display auto
+ opensolaris.org license-accept accept
+
+ $ pkg policy license-accept -H
+ All license-accept
+ opensolaris.org license-accept accept
+
+ $ pkg policy license-accept -p All
+ PUBLISHER POLICY VALUE
+ All license-accept decline
+
+ $ pkg policy license-accept -p opensolaris.org
+ PUBLISHER POLICY VALUE
+ opensolaris.org license-accept accept
diff --git a/doc/pkg5_docs/linked-images.txt b/doc/pkg5_docs/linked-images.txt
new file mode 100644
index 0000000..f9c4124
--- /dev/null
+++ b/doc/pkg5_docs/linked-images.txt
@@ -0,0 +1,1265 @@
+.. This document is formatted using reStructuredText, which is a Markup
+ Syntax and Parser Component of Docutils for Python. An html version
+ of this document can be generated using the following command:
+ rst2html.py doc/linked-images.txt > doc/linked-images.html
+
+============================
+Ips pkg(5) zones integration
+============================
+
+:Author: Edward Pilatowicz
+:Version: 0.6
+
+.. sectnum::
+.. contents::
+
+Introduction
+============
+
+To allow for support of pkg(5) within zones and the automatic management
+and upgrading of zones during global zone pkg(5) operations, this
+proposal describes enhancements to pkg(5) to support "linked images".
+
+The pkg(5) linked images functionality proposed herein is intended to
+be generic, with zones being one type of linked images that can be
+supported. In addition to supporting zones linked images we also propose
+supporting another "system" type of linked images. The details of how
+these linked image types differ will be explained in more detail below.
+
+Another goal of the pkg(5) linked image support is to make all the
+linked image functionality visible to other pkg(5) subsystems common,
+and to isolate code dealing with the different linked image types within
+well defined modules contained within the linked image subsystem. IE,
+while other pkg(5) subsystems may need to be aware of linked images,
+they should not have to worry about specific linked image types.
+(zones, system, etc.)
+
+Linked images will have properties associated with them. The set of
+available properties may vary across different linked image types. The
+storage location for these properties values (IE, linked image metadata)
+may be plugin specific. For the "system" plugin, property data
+configuration will live within a /var/pkg configuration file. For the
+"zones" linked image plugin, property configuration will be derived from
+the zones subsystem (some properties will have implicit values, others
+may be derived from zonecfg(1m) parameters, etc.)
+
+
+Zones and linked images
+=======================
+
+The design for pkg(5) linked images as proposed herein is primarily
+being driven by the need to support zones. Hence, before jumping into
+the specifics of linked images support it is worth discussing how zones
+user are expected to interact with linked images, and also the
+requirements that zones have upon the design of linked images.
+
+
+Zone users and linked images
+----------------------------
+
+Ideally, zone users should be unaware of all linked image functionality
+proposed herein. They should never need to run any of the proposed
+pkg(1) linked image commands. Linked images will do their work
+automatically under the hood when users do operations like:
+
+- Run pkg(1) in the global zone. In this case pkg(1) operations will
+ automatically keep non-global zones in sync with the global zone.
+
+- Run pkg(1) in a non-global zone. In this case pkg(1) will prevent
+ operations that would allow a non-global zone to go out of sync with
+ the global zone.
+
+- Run zoneadm(1m) install/uninstall/detach/attach/etc. In this case the
+ zone brand callbacks will utilize with the pkg(5) linked image
+ functionality to manage zones as linked images.
+
+
+Zones requirements
+------------------
+
+Zones is a OS level virtualization technology that allows us to run
+multiple userland execution environment on a single kernel. Each of
+these userland execution environments is a separate virtual machine,
+with it's own filesystem, processes, network stack, etc. The default
+execution environment (IE, the one you get when you install Solaris or
+OpenSolaris) is called the global zone. Subsequent execution
+environments are called non-global zones and can be created and managed
+via zonecfg(1M) and zoneadm(1M).
+
+All the zones on a system share the same kernel, and the kernel is
+managed by the global zone. Non-global zones can not supply their own
+kernel modules. Hence, any software within a zone which spans the
+user/kernel boundary must be kept in sync with the kernel installed in
+the global zone. This puts constraints on what software can be installed
+within zones. The basic requirements here can be summed up as:
+
+- Software installed in a non-global zone that depends on specific
+ kernel functionality must be kept in sync with the kernel software
+ installed within the global zone. examples:
+ libzfs and drv/zfs (system/file-system/zfs)
+ libdladm (system/library) and drv/dld (system/kernel)
+
+- Software installed in a non-global that communicates to the global
+ zone via private interfaces must be kept in sync with the kernel
+ software installed within the global zone. examples:
+ zones proxy (system/zones)
+ libdladm (system/library) and dladm (system/network)
+
+- Software that depends on specific kernel functionality can only be
+ installed in a non-global zone if the corresponding kernel
+ functionality is installed within the global zone.
+
+Since non-global zones are separate virtual machines with their own
+filesystems and software, these machines (and their software contents)
+may not be trusted by the global zone. Hence any software management
+operations being done on non-global zone should not be able to affect
+the global zone. Effectively this means that all actions performed on a
+zone cannot safely be done from the global zone environment. This means
+that software management operations initiated from a global zone will
+either have to "enter" the zone (if it's running) to perform their
+operation, or they must take special precautions when accessing zone
+images to ensure that all filesystem operations are performed safely.
+The basic requirements here can be summed up as:
+
+- Software management operations will need to span multiple processes
+ operating in different zones.
+
+- Since zones are untrusted execution environment, global zone pkg(5)
+ operations should not read data from non-global zone. IE, any data
+ flow required to support linked images must be from a global zone to a
+ non-global zone, and never in the reverse direction. Also, write
+ accesses to a non-global zone from the global zone should be kept to
+ an absolute minimum, and any such accesses must be provably safe via
+ code inspection.
+
+Lastly, since non-global zones are separate virtual machines, they will
+normally not have access to the contents of the global zone. Yet the
+software that can be run in non-global zones is constrained by the
+software installed in the global zone. Hence:
+
+- All the constraints required to perform software management operations
+ within a non-global zone must be persistently available within that
+ zones image/filesystem.
+
+
+Zones non-requirements
+----------------------
+
+While developing linked images, existing Solaris 10 zones users have
+asked how linked images will enable them to continue to do certain zones
+administrative operations that they are familiar with. Unfortunately,
+some of these operations were possible mainly as side effects of the
+zones and SVR4 packaging implementations on Solaris 10. Since pkg(5) is
+significantly changing the way zones are used and managed, some of these
+legacy operations will no longer be possible and users will need to
+adopt new ways of managing their zones. So here are some of the
+requests from Solaris 10 users that we're not initially addressing with
+linked images.
+
+
+**The ability to install a patch on all zones on a system.**
+
+Patching of systems with pkg(5) will be substantially different from the
+patching of Solaris 10 system. This change is the administrative model
+is being driven by pkg(5) and is largely orthogonal to zones. With
+pkg(5), patching of systems will be done via pkg(1) update, where
+repositories are populated with the latests versions of packages that
+all system should be running, and when systems are image-updated they
+will automatically install the latest versions of software available
+within the repositories they are using. Hence, if an administrator
+wants to install updated packages onto a system with zones, they should
+push the updated packages into their repositories and update their
+zones.
+
+
+**The ability to install a package on all zones on a system.**
+
+Previously in Solaris 10, the package content of zones closely tracked
+the package content of the global zone. In most cases installing a
+package in the global zone would result in that package being installed
+in all zones. With pkg(5) the contents of zones are much more decoupled
+from the contents of global zones. We will not be providing a way to
+install packages across multiple zones. In general, system should only
+contain the software that they need for their operation. Hence, this
+software should be installed at zone installation time. Or if added
+later, it needs to be added directly into the zone image (via a pkg(1)
+install run within that zone image).
+
+We may subsequently create new mechanisms to help with operations like
+the ones above. Such mechanisms might take the form of recursive
+operation support, a simple image content policy mechanism, or some
+larger system life cycle management mechanism that defines the package
+software content of systems for their entire deployment (instead of just
+at install time).
+
+
+Possible linked image types
+===========================
+
+So in addition to supporting zones linked image, the design herein has
+also considered other future possible types of linked images. So before
+going into the details of the linked image design it's worth while to
+first describe some possible linked image types to understand when and
+where they might be used.
+
+
+**Zones linked images**
+
+ Support for zones linked images is included in this proposal. Users
+ will not be able to directly create or manage zones linked images.
+ Instead the system will automatically manage zones linked images when
+ zones are used. Zones linked images should provide us with with a
+ means to do the following:
+
+ - Allow for the creation of non-global zone images where the contents
+ of those images is constrained by the software installed in the
+ global zone.
+
+ - Allow for the updating of software within non-global zones while
+ taking into account the constraints placed upon the zone by the
+ software installed in the global zone.
+
+ - Allow pkg(5) operations initiated from (and operating on) the global
+ zone to update non-global zone images as required to keep them in
+ sync with software being updated in the global zone.
+
+ - Allow for pkg(5) operations initiated directly upon non-global zone
+ image to take into account the constraints placed upon then by the
+ software installed in the global zone.
+
+ - Allow for the auditing of non-global zones to verify that their
+ software contents are in sync with the software installed in the
+ global zone.
+
+
+**System linked images**
+
+ Support for system linked images is included in this proposal. These
+ types of linked images will be very similar to zone linked images. All
+ the features listed above that will be available for zones linked
+ images will also be available for system linked images. But unlike
+ zone linked images, these images can be created via new pkg(1)
+ interfaces.
+
+ Support for system linked images is included in this proposal because
+ it is anticipated that system linked images will be used internally
+ within Solaris. Also, having a "system" linked image type will
+ greatly facilitate the testing of linked image functionality, the
+ large majority of which is common to all linked image types.
+
+
+**User linked images**
+
+ Support for user linked images is NOT included in this proposal. These
+ type of images would be managed very differently from zones or system
+ linked images. User linked images could provide us with the following
+ functionality:
+
+ - Allow for arbitrary users to create user linked images, where the
+ contents of that image are in sync with the software installed in
+ another image.
+
+ - Allow for a user to update the software within a user linked image
+ while staying in sync with the software installed in another image.
+
+ - Allow for the auditing of a user linked image to verify that their
+ software contents are in sync with the software installed in another
+ image.
+
+ So here's an example of how a user linked image might work. Say a user
+ named Ed wants to run a copy of SpamAssassin on a system named
+ jurassic, but jurassic doesn't have SpamAssassin installed. Ed could
+ then create a "user" linked image that is linked to the jurassic
+ system image. Then he could install SpamAssassin into that image.
+ Pkg(5) would install a version of SpamAssassin that is compatible with
+ the software contents already installed on jurassic. (In the process
+ of doing this pkg(5) would also install into the user image any
+ dependencies required by SpamAssassin that were missing from
+ jurassic.) Ed could then run this version of SpamAssassin without
+ having to worry about his software being out of sync with the system.
+ The system administrator would have no knowledge of user images that
+ might be created on a system, hence, if the administrator updates the
+ software on a system then any user images may now be out of sync. In
+ this case, Ed would be able to perform an audit of all his user linked
+ images to determine if they are in sync with their specified policy
+ and the contents of the system they are being used on. If they were
+ out of sync (due to the system being updated) he could then initiate a
+ sync operation to update them and bring them back in sync.
+
+
+**Diskless client linked images**
+
+ Support for diskless client linked images is NOT included in this
+ proposal. These types of images would probably not be managed
+ directly, but would probably be managed indirectly by diskless client
+ management tools. One possible deployment model for diskless clients
+ would be to create a parent image which represents the standard
+ diskless client configuration deployment, and then to create linked
+ images all parented to that one parent image. These linked images
+ would actually be the diskless client root filesystems. Subsequent
+ software management operations could be performed on the parent image,
+ with changes automatically propagated to all the client images
+ automatically based on the content policy of the linked images.
+
+
+**Distributed linked images**
+
+ Support for Distributed linked images is NOT included in this
+ proposal. But if pkg(5) functionality was accessible over the network
+ via rad interfaces, it should be possible to create linked image
+ relationships that span multiple machines. This could be used in a
+ manager similar to diskless clients where a master deployment image is
+ created on one machine, and this master image is then linked to
+ machines which deploy the image. Subsequently, updates to the master
+ image would automatically be propagated to deployment machines, based
+ of the content management policy being used.
+
+
+Out of scope
+============
+
+There are many components which will probably be required to allow
+pkg(5) and zones to work seamlessly together. This proposal does not
+address all of them. Specifically, the following features are not being
+delivered by this project.
+
+
+**User, diskless, and distributed linked images.**
+
+ While this project will provide basic support for creating system and
+ zone linked images, it will not provide support for any other types of
+ linked linked images, even though other types of linked images may be
+ discussed within this document.
+
+
+**Offline zone install.** [1]_
+
+ This proposal does nothing to address the current requirement that an
+ active network connection to a valid package repository be available
+ during most pkg(5) operations. If anything, this proposal will
+ increase the number of operations that may require such a connection
+ to be available.
+
+
+**The system repository.** [2]_
+
+ When managing zone linked images, it's critical that certain packages
+ be kept in sync between global and non-global zones. This means that
+ zones must have access to any publishers that contain packages that
+ must be kept in sync between the images, regardless of the zones
+ publisher configuration. Some additional complications to this problem
+ are that for zones which are not running, we may not be able to
+ instantiate their network configuration, so we may not be able to talk
+ to any of their configured network based publishers. The planned
+ solution to address these problems is to create "system repository".
+ The system repository would allow a linked image to access any
+ publishers configured within the parent image that contain packages
+ which needed to be kept in sync with the child image.
+
+ Since delivery of the system repository is out of scope for this
+ project, initially zones linked image support will rely on zones
+ having a publisher configuration which provides access to all the
+ packages required to keep the zone in sync.
+
+
+**Zones image locking.** [3]_
+
+ When performing pkg(5) operations in a global zone that affect the
+ contents of the global zone, we need a locking mechanism to prevent
+ concurrent operations from being done within a non-global zone. We
+ also need a locking mechanism that allows for the reverse. Ie, if a
+ zone is in the middle of a pkg(5) operation we don't want to initiate
+ an operation from the global zone which might also require updating
+ that zone. Other scenarios that we will probably also need to protect
+ against include a zone changing state while we're performing a pkg(5)
+ operation on that zone. For example, if we're installing a package in
+ a global zone which results in us also installing a package in a
+ non-global zone, we need to prevent that non-global zone from
+ rebooting while we're installing the package. Another example is that
+ if we're installing or removing a package from a global zone, we will
+ probably want to prevent a concurrent install of a non-global zone
+ since that could results in the freshly installed zone being out of
+ sync with the global zone. This proposal does not address any of the
+ possible race conditions wrt pkg(5) and zone linked image operations.
+
+
+**Pkg(5)/beadm(1M) image filesystem management.** [4]_
+
+ Currently, beadm(1M) allows for versioning (via snapshotting and
+ cloning) of multiple filesystem in a global zone image, assuming all
+ those filesystems are zfs based and are all children of the current
+ root filesystem. beadm(1M) treats any other filesystems in the current
+ BE as shared and lofs mounts them into any other alternate BE. This
+ means that if any pkg(5) software is installed into these "shared"
+ filesystems that software will become out of sync wrt some BE. Pkg(5)
+ and beadm(1M) do not perform any check to ensure that all the software
+ being installed is not being installed into "shared" filesystems. This
+ same problem also affects zones. This proposal does not address this
+ issue in any way and assumes that eventually pkg(5) or beadm(1M) will
+ provide more flexible and robust functionality to manage images that
+ span multiple filesystem.
+
+
+**Beadm(1M) support for linked images.**
+
+ Currently, beadm(1M) can be run within the global zone to manage
+ global zone boot environments and their associated zones. But
+ beadm(1M) does not support running within a non-global zone and
+ creating snapshots of zone boot environments from within a zone.
+ Beadm(1M) support within a zone is a requirements for supporting
+ pkg(5) image-update within a zone. (Since image-update only operates
+ on alternate boot environments.) It is also the case that other pkg(5)
+ operations may refuse to operate on the currently running image and
+ may only be run on cloned and offline boot environments. Since
+ beadm(1M) can not be run within a zone, we can't create cloned boot
+ environments within a zone, so none of these operations will be
+ supported.
+
+ Additionally, beadm(1M) is currently aware of zones images, but
+ eventually, beadm(1m) should probably become linked image aware, since
+ all linked images should be snapshotted, cloned, and upgraded in sync.
+ Enhancing beadm(1M) will be required to support non-zone types of
+ linked images that live outside the current boot environment. Once
+ this project delivers initial pkg(5) linked image interfaces it will
+ be possible to update beadm(1M) to consume these interfaces so that it
+ can be aware of all linked images on the system, instead of just zone
+ images. These beadm(1M) enhancements are out of the scope of this
+ proposal.
+
+
+pkg(5) linked image overview
+============================
+
+pkg(5) linked images will always always have a parent and child
+relationship. Parent images may have multiple children, but each child
+image may only have one parent. It's possible for there to be multiple
+levels of linked images (i.e., nested linked image), for example, you
+could have a system linked image, which is a child of a zone linked
+image, which is the child of a global zone image.
+
+
+pkg(5) linked image name
+------------------------
+
+All pkg(5) linked images are uniquely identified by a name. A fully
+qualified linked image name is :.
+The linked image name must begin with an alphanumeric character and can
+contain alphanumeric characters, underscores (_), hyphens (-), and dots
+(.). Additional restrictions on the format may be
+defined by the linked image plugin handling that type of linked image.
+
+
+pkg(5) linked image attach mode
+-------------------------------
+
+As previously mentioned, all linked images will have a parent/child
+relationship. But there are two distinct ways that this parent/child
+link can be established, which in turn determines what operations are
+possible on each image and how the linked image relationship is managed.
+
+First, a parent may link/attach a child image to itself, in which case,
+the parent image will be authoritative for the linked image
+relationship. This means that:
+
+- The parent image is aware of the child image.
+
+- The child image will know that it is a linked image, but it will
+ not know the location of it's parent image.
+
+- Linked image properties can only be modified from the parent image and
+ are treated as read-only within the child image.
+
+- Packaging operations on the parent image which require updating child
+ images to keep them in sync will attempt to do so automatically.
+
+Second, an image may make itself into a child by linking/attaching
+itself to a parent. In this case the parent has no awareness of the
+child image and the child image is authoritative for and must manage the
+linked image relationship. This means that:
+
+- The parent image is unaware of the child image.
+
+- The child image will know that it is a linked image, and it will
+ know the location of the parent image.
+
+- Linked image properties only exist within (and there for must be
+ modified from within) the child image.
+
+- Packaging operations on the parent image will not directly affect the
+ child image. It is the responsibility of the child image to make sure
+ that it remains in sync with it's parent.
+
+In the former image linking mode, the parent must "push" constraint (and
+property) information to child images. While in the latter mode the
+child will "pull" constraint information from the parent.
+
+Zones linked images will exclusively use the push linking mode.
+System linked images will support both the push and pull linking modes.
+
+
+pkg(5) linked image properties
+------------------------------
+
+As mentioned earlier, each child linked image will have properties
+associated with it.
+
+In the case of both push and pull based child images, linked image
+property information will always be stored within a private file and
+directory in the pkg(5) image metadata directory. Currently either
+/.org.opensolaris,pkg/linked/ or /var/pkg/linked/.
+
+In the case of push based parent images, the property information for
+child images is accessible through a plugin for managing each different
+type of linked image. This allows each linked image type to manage the
+storage of linked image properties. In the case of system linked
+images, child linked image properties will be stored within the image
+configuration. In the case of Zones linked images, linked image
+properties may either be fixed or derived from different zonecfg(1m)
+options.
+
+Initially the following properties will be supported:
+
+**li-name**
+
+ This is a linked image name (as described above).
+
+ This property is common to all linked image types, both push and
+ pull based.
+
+ Notably, this property always refers to a child linked image and
+ never a parent image. This is because the linked image name encodes
+ what type of linked image the child is.
+
+**li-mode**
+
+ This indicates the attach mode of the linked image child (as
+ described above), either push or pull.
+
+ This property is common to all linked image types, both push and
+ pull based.
+
+**li-path**
+
+ This is the filesystem path by which one linked image is accessible
+ from another.
+
+ In the case of a parent image with a push based child, this property
+ will point to the child image. From within the push based child,
+ this property will not be present. In the case of pull based
+ children, this property will point to the parent linked image.
+
+**li-recurse**
+
+ This property specifies if recursive linked image operations should
+ continue to recurse within a linked child.
+
+ This property only exists for push based children of a parent image.
+
+ By default, for zones this property will be set to false. This means
+ that if we do an recursive pkg(5) operation in the global zone that
+ affects a non-global zone, we will update that non-global zone, but we
+ will ignore any children of that non-global zone. So if for example,
+ a non-global zone administrator has created a push based child of the
+ non-global zone image, global zone pkg operations will not recurse
+ into that child image.
+
+
+pkg(5) linked image interfaces
+==============================
+
+
+pkg(1) linked image interfaces introduction
+-------------------------------------------
+
+Linked image support has an impact on the behavior and interfaces of
+many pkg(5) operations. Hence, before jumping into all the new linked
+image interfaces it makes sense to discuss some of the common linked
+image behaviors and options to pkg(1) cli interfaces.
+
+pkg(5) operation recursion
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ Any pkg(1) subcommand which installs or removes packages from a linked
+ image may attempt to recurse into linked child images to attempt to
+ keep those child images in sync with the parent. There are two things
+ to note about this recursive behavior.
+
+ First, the initial recursion into child images by a pkg(5) operation
+ on a parent image has no relation to the li-recure linked image
+ property. This property is only used when dealing with multiple
+ levels of recursion. If this property is false for a child image, a
+ recursive operation will still descend into that child, but it will
+ not continue to recursively descended into children of that child.
+
+ Second, it's important to realize that recursion doesn't imply that
+ the operation being performed on the parent will also be performed on
+ all it's children. Recursion into children is only done to keep child
+ images in sync. To clarify this point it's worth explicitly
+ describing how this impacts common pkg(5) operations.
+
+ **pkg(1) install** - When an install recurses into child linked images,
+ it does so only to keep those images in sync. Hence, if a user
+ installs a new package in a parent image, the requested package will
+ not automatically be installed into child images. Although if a
+ user "upgrades" a package in the parent image by installing a newer
+ version of that package, and that package also happens to be kept in
+ sync between the parent and child images, then the recursive sync
+ done as part of the install operation will result in the newer
+ package being installed in the child image.
+
+ **pkg(1) update** - This is handled similarly to a pkg(1) install.
+ When we recurse into the child we do not do a pkg(1) update. We
+ will only update the minimum number of packages required to keep the
+ child in sync with the planned contents of the parent image.
+
+ **pkg(1) uninstall** - When an uninstall recurses into a child, it will
+ not uninstall the specified packages within that child. Hence if a
+ random un-synced package is removed from a parent image it will not
+ be touched in a child image. But if the user tries to remove
+ a synced package which is installed within a child image, the
+ removal will fail. The user will have to remove the package from
+ child images before they can remove it from the parent image.
+
+
+pkg(1) linked image common cli options: ignoring children
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ As mention above, any pkg(1) subcommand which installs or removes
+ packages from a linked image may attempt to recurse into linked child
+ images to attempt to keep those child images in sync with the parent.
+ There for, these pkg(1) commands will all get new options to allow
+ them to ignore linked child images. These pkg(1) commands include::
+
+ attach-linked [-I | -i ]
+ change-facet [-I | -i ]
+ change-variant [-I | -i ]
+ image-update [-I | -i ]
+ install [-I | -i ]
+ set-property-linked [-I | -i ]
+ sync-linked [-I | -i ]
+ uninstall [-I | -i ]
+
+ Note that the list of commands above does not include pkg(1) commands
+ like verify, fix, and change-facet, since those commands will never
+ result in packages being installed or removed from an image.
+ Unintuitively, set-property-linked is included in the list above since
+ it may be used to change linked image properties which may results in
+ in packaging contents changes in that image, which in turn may need to
+ be propagated to linked children.
+
+ When performing one of the above operations, the operation is first
+ planned in the parent image, and if there are required packaging
+ changes then we will recurse to each child image and create a plan to
+ keep each child in sync with the planned changes in the parent. If a
+ plan which keeps a child image in sync cannot be created then the
+ operation will fail before any image is modified. In this case, if
+ the administrator wants to retry the requested operation they will
+ need to do one of the following before that operation will succeed:
+
+ - Detach the child image preventing the operation.
+
+ - Modify the package contents of the child image that is preventing
+ the operation such that the operation can succeed.
+
+ - Pass the -i option to the requested pkg(1) command,
+ there by telling it to ignore the specified child image that is
+ preventing the operation.
+
+ - Use the -I option to requested pkg(1) command, there by telling
+ it to ignore all child images.
+
+ Notably, in certain cases it's possible for a push based child linked
+ image to exist but not be accessible from the parent. An example of
+ this would be when a non-root user runs a pkg(1) command, all zone
+ linked image paths will not be accessible. If pkg(1) is attempting to
+ do any operation which may recurse into child images, all children
+ must be accessible and if any child is not accessible the operations
+ will fail and no updates will be made to any image.
+
+
+pkg(1) linked image common cli options: selecting children
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ Some of the proposed linked image pkg(1) commands support common
+ arguments that allow the caller to specify which linked image they
+ should operate on. These new commands and options are::
+
+ property-linked [-l ]
+ set-property-linked [-l ]
+
+ audit-linked [-a|-l ]
+ detach-linked [-a|-l ]
+ sync-linked [-a|-l ]
+
+ If one the above commands is run without any arguments, then the
+ command will be preformed on the current image with the assumption
+ that the current image is a linked child image. If the current image
+ is not a child image an error will be generated.
+
+ If the "-l " option is specified to one of the commands
+ above, then it's assumed that the current image has a child by that
+ name and the requested operation is run for that child.
+
+ If the "-a" option is specified to one of the commands above, then the
+ command is run for all the children of the current image.
+
+
+pkg(1) linked image common cli options: syncing parent metadata
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ If a child image is linked to a parent in pull mode, then certain
+ pkg(1) subcommands will always attempt to update linked image metadata
+ from their parent. This update will fail if the parent image is not
+ accessible. Hence, a new --no-parent-sync option is available for to
+ skip this parent metadata sync. The pkg(1) commands which support
+ this option are:
+
+ attach [--no-parent-sync]
+ audit [--no-parent-sync]
+ change-facet [--no-parent-sync]
+ change-variant [--no-parent-sync]
+ image-update [--no-parent-sync]
+ install [--no-parent-sync]
+ list [--no-parent-sync]
+ sync [--no-parent-sync]
+
+
+pkg(1) linked image cli interfaces
+----------------------------------
+
+pkg(1) list-linked
+~~~~~~~~~~~~~~~~~~
+
+ **pkg list-linked [-H]**
+
+ List all known child images associated with the current image. This
+ lists the name and path for each child image.
+
+ Here's an example of this command when run by a non-root user on a
+ system with zones::
+
+ $ pkg list-linked
+ NAME RELATIONSHIP PATH
+ - self /
+ zone:dhcp child /export/zones/dhcp/root
+ zone:z1 child /export/zones/z1/root
+ zone:z3 child /export/zones/z3/root
+ system:child child /child
+
+
+pkg(1) property-linked
+~~~~~~~~~~~~~~~~~~~~~~
+
+ **pkg property-linked [-H] [-l ] [propname ...]**
+
+ List all property values associated with a linked image. If no linked
+ image is specified then if the current image is assumed to be a child
+ linked image and it's properties are listed.
+
+ Here's an example of this command when run on an image that is not
+ linked::
+
+ $ pkg -R /tmp/a list-linked
+ $
+
+ Here's an example of this command when run on an image that has
+ children but is not itself a child::
+
+ $ pkg property-linked
+ PROPERTY VALUE
+ li-altroot /
+ li-path /
+
+ Here's an example of this command when run by a non-root user on a
+ system with zones::
+
+ $ pkg property-linked -l zone:dhcp
+ PROPERTY VALUE
+ li-altroot /
+ li-model push
+ li-name zone:dhcp
+ li-path /export/zones/dhcp/root
+
+ Here's an example of this command when run by a root user directly on
+ a zone/child image::
+
+ # pkg -R /export/zones/dhcp/root property-linked
+ PROPERTY VALUE
+ li-altroot /export/zones/dhcp/root
+ li-model push
+ li-name zone:dhcp
+ li-path /export/zones/dhcp/root/
+
+
+pkg(1) audit-linked
+~~~~~~~~~~~~~~~~~~~
+
+ **pkg audit-linked [--no-parent-sync] [-a|-l ]**
+
+ Audit the package contents of a linked image to see if it's in sync
+ with the contents of it's parent image and it's content policy.
+
+ Here's an example of this command when run on an image that has no
+ parent::
+
+ $ pkg audit-linked
+ pkg: Linked image exception(s):
+ Current image is not a linked child: /
+
+ Here's an example of this command run by root on a system with zones::
+
+ # pkg audit-linked -a
+ NAME STATUS
+ zone:dhcp diverged
+ zone:z1 diverged
+ zone:z3 diverged
+ system:child synced
+
+
+pkg(1) sync-linked
+~~~~~~~~~~~~~~~~~~
+
+ | **pkg sync-linked [-a|-l ]**
+ | **[--no-parent-sync] [--no-pkg-updates] [--linked-md-only]**
+ | **[-nvq] [--accept] [--licenses] [--no-index] [--no-refresh]**
+
+ Sync will attempt to bring an image into sync with it's policy.
+
+ A sync operation may upgrade packages or install new packages into an
+ image while bringing the image into sync. If the user wants to
+ prevent a sync from installing new packages or updating existing
+ packages, they can specify the --no-pkg-updates option. If the image
+ cannot be synced without installing or updating packages, then this
+ option will cause the sync operation to fail.
+
+ If the caller doesn't want to make any packaging updates to the child
+ image then they can specify the --linked-md-only option. (This option
+ implies the --no-pkg-updates option.) When this option is specified
+ the package contents of an image will not be modified, and the only
+ result of the operation is that the parent content data stored within
+ the child image will be updated.
+
+ Since a sync operation may modify the packaging contents of an image
+ it is very similar to a pkg(1) install operation, there for the sync
+ command also must support of the same options as the pkg(1) install
+ command. Those common options are::
+
+ [-nvq] [--accept] [--licenses] [--no-index] [--no-refresh]
+
+ Here's an example of this command when run on an image that has no
+ parent::
+
+ $ pkg sync-linked
+ pkg: detach-linked failed (linked image exception(s)):
+ Current image is not a linked child: /
+
+ Here's an example of this command run by root on a system with zones::
+
+ # pkg sync-linked -l zone:dhcp -v
+ Solver: [ Variables: 2356 Clauses: 60384 Iterations: 2 State: Succeeded]
+ ...
+ Maintained incorporations: None
+
+ Package version changes:
+ ...
+ pkg://opensolaris.org/entire@0.5.11,5.11-0.125:20091014T044127Z -> pkg://opensolaris.org/entire@0.5.11,5.11-0.139:20100511T142142Z
+ ...
+
+
+pkg(1) set-property-linked
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ | **pkg set-property-linked [-l ]**
+ | **[--no-parent-sync] [--no-pkg-updates] [--linked-md-only]**
+ | **[-nvq] [--accept] [--licenses] [--no-index] [--no-refresh]**
+ | ** **
+
+ This command will attempt to update the specified linked image
+ property.
+
+ Certain linked image properties may be read-only, and which properties
+ are read-only may vary between different types of linked images. When
+ dealing with push based child images, all linked image properties are
+ treated as read-only within the child.
+
+ Since a set-property-linked operation can change a linked image's
+ content policy, this command may need to sync a child image with it's
+ parent. Hence, the set-property-linked command also supports many of
+ the same options as the pkg(1) sync-link command. Those common
+ options are::
+
+ [--no-parent-sync] [--no-pkg-updates] [--linked-md-only]
+ [-nvq] [--accept] [--licenses] [--no-index] [--no-refresh]
+
+
+pkg(1) attach-linked
+~~~~~~~~~~~~~~~~~~~~
+
+ | **pkg attach-linked (-c|-p)**
+ | **[--no-pkg-updates] [--linked-md-only]**
+ | **[-nvq] [--accept] [--licenses] [--no-index] [--no-refresh]**
+ | **[--prop-linked = ...]**
+ | ** **
+
+ The attach-linked command is used to establish a linked image
+ relationship. This command may not be supported for all linked image
+ types. (For example, zone linked images cannot be attached via this
+ command.)
+
+ If a parent image want to link a child image to itself, the -c option
+ should be specified. This creates a child with a push mode of
+ operation. If a child image wants to attach to a parent image, the -p
+ option should be specified. This creates a child with a pull mode of
+ operation.
+
+ When linking images the user may specify optional linked image
+ property values. Not all properties are settable via these values.
+ (The allowable properties may change in the future and may also be
+ linked image type plugin specific.)
+
+ Normally when linking images together, a sync of the child image is
+ automatically done. If the child image can not be synced the attach
+ will fail. Since an attach operation tries to sync a child image with
+ it's parent, the attach-linked command must also support many of the
+ same options as the pkg(1) sync-link command. Those common options
+ are::
+
+ [--no-pkg-updates] [--linked-md-only]
+ [-nvq] [--accept] [--licenses] [--no-index] [--no-refresh]
+
+ Currently, linked image relationships cannot be created at image
+ creation time. A linked image relationship can only be established at
+ between two existing images. The reason for this is mainly cli
+ complexity. Specifically, supporting this would require that pkg(1)
+ image-create accept all the same commands options as the pkg(1)
+ attach-linked command.
+
+ Here's an example of this command attaching a push child image::
+
+ # pkg -R attach-linked -v -c system:child /child
+ Create boot environment: No
+ Rebuild boot archive: No
+ Services:
+ None
+
+
+pkg(1) detach-linked
+~~~~~~~~~~~~~~~~~~~~
+
+ **pkg detach-linked [-a|-l ]**
+
+ The detach-linked command will end a linked image relationship. This
+ command may not be supported for all linked image types. (For
+ example, zone linked images cannot be detached via this command.)
+
+ Here's an example of this command when run on an image that is not
+ linked::
+
+ # pkg detach-linked
+ pkg: detach-linked failed (linked image exception(s)):
+ Current image is not a linked child: /
+
+ Here's an example of this command when run directly on a child that
+ linked to a parent via a push mode relationship::
+
+ # pkg -R /child detach-linked
+ pkg: detach-linked failed (linked image exception(s)):
+ Parent linked to child, can not detach child: /child
+
+ Here's an example of this command detaching a child image::
+
+ # pkg detach-linked -l system:b -v
+ Create boot environment: No
+ Rebuild boot archive: No
+ Services:
+ None
+
+
+
+pkg(5) linked image manifest metadata
+-------------------------------------
+
+When dealing with zones, package publishers need a way to specify which
+packages must be kept in sync between zone images. In Solaris 10 this
+was done via a SVR4 packaging attribute (SUNW_PKG_ALLZONES). With
+pkg(5) we will create a new manifest depend actions for this
+purpose::
+
+ depend type=parent
+
+If a package contains this dependency and it is being installed into an
+image which is not a linked child then this dependency is ignored. If a
+package containing this image is being installed into a child image,
+then it's required that the same package be installed within the parent
+image. This dependency has an implicit fmri value which is equal to the
+package fmri which contains the dependency. Also, when matching fmris
+in the parent image, the fmri version matching algorithm employed is the
+same as that used for satisfying "incorporate" dependencies. (Ie, an
+exact version match is required, not a equal than or greater version
+match.)
+
+So packages that must be kept in sync between the global and non-global
+zone should have the following dependency action in their manifest::
+
+ depend type=parent variant.opensolaris.zone=nonglobal
+
+The dependency above will need to be set for any package which delivers
+software that may operate across zone boundaries on a system. This
+would include:
+
+- Any software which delivers a kernel component of any kind.
+
+- Any software which delivers interfaces which may span a zone boundary.
+ Some examples would include:
+
+ - pkg(5) - Since zones are different linked images, pkg(5) by
+ definition must manage images that span zones.
+
+ - libdladm - When this library is used inside a zone it will will
+ communicate (via private interfaces) to the dlmgmtd daemon inside
+ the global zone.
+
+Initially, all of ON and pkg(5) will set this property for all packages
+they deliver. In time, the number of packages delivered from the ON and
+pkg(5) gates with this attribute set will be reduced.
+
+This property will also be a public interface since anyone delivering
+software via pkg(5) may fall into one of the categories above. Examples
+of third parties applications which fall into this category would
+include things like VirtualBox, VxVM/VxFS, ClearCase, etc.
+
+
+pkg(5) linked image api.py interfaces
+-------------------------------------
+
+There will need to be changes made to the pkg(5) api.py interfaces to
+support linked images.
+
+The new pkg(5) api.py interfaces will attempt to minimize the amount of
+change required for existing pkg(5) api.py clients which do not wish to
+provide support for managing linked images directly. (This should
+include every api.py client except for the pkg(1) cli). Aside from
+tweaks to existing api.py interfaces, the most significant impact to
+existing api.py consumers will be a new set of linked image related
+exceptions and errors that may be generated by api.py interface calls.
+
+
+
+pkg(5) linked image internals
+=============================
+
+
+pkg(5) linked image child operations
+------------------------------------
+
+When operating on child linked images, pkg(1) will access those images
+in one of two ways.
+
+1) A parent image may access a child image directly to write linked
+image property and parent content information into the child image.
+
+2) For all other operations, the pkg(1) linked image code will spawn a
+new pkg(1) cli process to manipulate the child image. For system images
+this will be a normal pkg(1) process started with the -R option. Zones
+linked images will initially operate in the same way as system images.
+
+
+pkg(5) linked image operation staging
+-------------------------------------
+
+Currently pkg(1) has multiple stages of operation when manipulating
+images. The stages most relevant to linked image operations are:
+
+1) Package install/uninstall/upgrade planning
+2) Package action planning
+3) Package content downloading
+4) Action execution
+
+When dealing with linked images we want to be able to perform most of
+the operations above in lock step across multiple images. We want to be
+able to do planning for all child images (and potentially children of
+children) before beginning any of the later stages of execution (this
+will allows us to report problems early.). Before beginning any
+operation which modifies an image we also want to make sure that we've
+downloaded any required content for updating any linked images.
+
+Also, depending on how many packages are installed within an pkg(5)
+image, stages 1 and 2 above can be very memory intensive, and when
+planning operations across many images we want to be careful not to run
+a system out of memory.
+
+Hence, this project will create a mechanism where using pkg(5) we can
+execute one of the specific stages above, save the results to disk, and
+then resume our operation to perform a later stage, only using
+previously saved results. This will be done by adding the following
+private options to pkg(1):
+
+ | **pkg [--runid= --stage=(pubcheck|plan|prepare|execute)] ...**
+
+The --stage option will specify which specific operation the pkg(1)
+command should perform. If --stage=pubcheck is specified, the pkg(1)
+command will verify that the current images publisher configuration is a
+superset of the parent images publisher configuration. If --stage=plan
+is specified, the pkg(1) command will plan what packages to
+install/uninstall/upgrade, write that information to disk, and exit. If
+--stage=prepare is specified, pkg(1) will read a previously created
+install/uninstall/upgrade plan from disk (an error will be generated if
+there is no existing package plan on disk), and then download any
+required contents. Lastly, --stage=execute will cause pkg(1) to read
+existing package and action plans and execute them.
+
+When writing package plans to disk they are stored in json format.
+
+It's possible that we may have multiple non-image-modifying operations
+in progress on a single image at one. (For example, using the -n option
+to make pkg(1) commands.) Hence, we introduce a --runid option that
+allows the caller to specify a number (N) to uniquely identify the saved
+package and action plans. The --runid options is required when using
+the --stage option.
+
+These new options are private and users should never specify them. They
+will be used internally by the linked image code when invoking pkg(1)
+commands to operate on child images.
+
+
+zones(5) and pkg(5) integration
+===============================
+
+This project will update the zones smf service script and the zones ipkg
+brand callback to utilize the new linked image pkg(1) cli and api.py
+interfaces.
+
+zones smf service changes
+-------------------------
+
+During system boot the zones smf service (svc:/system/zones) will be
+enhanced such that it does a linked-audit of all installed ipkg branded
+zones on the system. If any zones fail the linked audit, the zones
+service will enter the maintenance state. If a zone has the zonecfg(1m)
+autoboot property set to true, but fails a linked audit, that zone will
+not be booted.
+
+ipkg brand callback changes
+---------------------------
+
+The zones ipkg brand boot callback will be updated to audit zone images
+before attempting to boot them. If a zone image fails to audit then it
+will also fail to boot.
+
+The zones install callback will be modified such that it uses linked
+functionality when creating and populating zones images.
+
+The zones attach and detach callbacks will be modified so that they
+automatically use the linked image attach and detach functionality. By
+default, a zoneadm attach will use the pkg(1) attach-linked
+--no-pkg-updates option, unless the zoneadm(1m) attach -u option is
+specified (there by allowing package updates).
+
+Zoneadm(1m) move will not require any new callbacks or updates since
+zones linked image metadata (like the zone path) will not be cached by
+the linked image subsystem. Instead, internally, the zones linked image
+plugin will obtain all zones linked image metadata directly from the
+zones subsystem.
+
+
+
+Related Bugs
+============
+
+The linked image support proposed above is covered by the following
+bugs:
+
+- | 16148 need linked image support for zones, phase 1
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=16148
+
+- | 6969956 need pkg metadata indicating pkgs which must be synced between zones
+ | http://monaco.sfbay.sun.com/detail.jsf?cr=6969956
+ | 7633 need pkg metadata indicating pkgs which must be synced between zones
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=7633
+
+The following bugs are all prerequisites for most the future linked
+images and zones follow on work described above:
+
+- | 16149 need system repository to facilitate linked image support
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=16149
+
+- | 6964121 tmpfs rename should update ->v_path
+ | http://monaco.sfbay.sun.com/detail.jsf?cr=6964121
+
+The future improvements to linked images and zones which are required
+for fully integrated zones support are covered by:
+
+- | 16150 need linked image support for zones, phase 2
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=16150
+
+- | 6978239 pkg(5) needs a zones state locking mechanism
+ | http://monaco.sfbay.sun.com/detail.jsf?cr=6978239
+
+Other bugs which are related to pkg(5) and zones(5) which are not being
+addressed by this work include:
+
+- | 1947 Offline zone creation is impossible
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=1947
+
+- | 13986 incorporation/metapackage needed for zone install
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=13986
+
+- | 15343 pkg needs to be able to lock BEs
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=15343
+
+- | 15342 libbe (and beadm) need BE write lock support
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=15342
+
+- | 16258 libbe support for zones
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=16258
+
+- | 16838 pkg should take into account libbe's ideas about shared and nonshared filesystems
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=16838
+
+- | 6988152 beadm/libbe needs support for zone/linked dataset management
+ | http://monaco.sfbay.sun.com/detail.jsf?cr=6988152
+
+- | 6998842 Zones Proxy for the pkg(5) System Repository
+ | http://monaco.sfbay.sun.com/detail.jsf?cr=6998842
+
+
+PTL Entries
+===========
+
+- | 8362 Pkg support for Zones phase 1
+ | http://sac.sfbay/projectlog/ptl/dashboard.php?UniqueID=8362
+
+- | 8724 Pkg support for Zones phase 2
+ | http://sac.sfbay/projectlog/ptl/dashboard.php?UniqueID=8724
+
+- | 8725 pkg(5) System Repository
+ | http://sac.sfbay/projectlog/ptl/dashboard.php?UniqueID=8725
+
+- | 8726 Zones Proxy for the pkg(5) System Repository
+ | http://sac.sfbay/projectlog/ptl/dashboard.php?UniqueID=8726
+
+- | 8727 Updater Branded Zones
+ | http://sac.sfbay/projectlog/ptl/dashboard.php?UniqueID=8727
+
+- | 8728 Zone State Locking
+ | http://sac.sfbay/projectlog/ptl/dashboard.php?UniqueID=8728
+
+
+References
+==========
+
+.. [1] | 1947 Offline zone creation is impossible
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=1947
+
+.. [2] | 16149 need system repository to facilitate linked image support
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=16149
+
+.. [3] | 15343 pkg needs to be able to lock BEs
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=15343
+ | 15342 libbe (and beadm) need BE write lock support
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=15342
+ | 6978239 pkg(5) needs a zones state locking mechanism
+ | http://monaco.sfbay.sun.com/detail.jsf?cr=6978239
+
+.. [4] | 16838 pkg should be aware of image and filesystem boundaries
+ | https://defect.opensolaris.org/bz/show_bug.cgi?id=16838
diff --git a/doc/pkg5_docs/macros.rst b/doc/pkg5_docs/macros.rst
new file mode 100644
index 0000000..6ea5f82
--- /dev/null
+++ b/doc/pkg5_docs/macros.rst
@@ -0,0 +1,53 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+
+.. Hyperlinks
+
+.. _pkg-project: http://opensolaris.org/os/project/pkg/
+
+.. _pkg-opensolaris-release: http://pkg.opensolaris.org/release/
+
+.. _pkg-opensolaris-dev: http://pkg.opensolaris.org/dev/
+
+
+.. Substitutions
+
+.. |pkg5| replace:: ``pkg``\ (5)
+
+.. |smf5| replace:: ``smf``\ (5)
+
+.. |tar1| replace:: ``tar``\ (1)
+.. |pkg1| replace:: ``pkg``\ (1)
+.. |pkgsend1| replace:: ``pkgsend``\ (1)
+.. |pkgrecv1| replace:: ``pkgrecv``\ (1)
+.. |pkgdepend1| replace:: ``pkgdepend``\ (1)
+.. |pkgdiff1| replace:: ``pkgdiff``\ (1)
+.. |pkgfmt1| replace:: ``pkgfmt``\ (1)
+.. |pkgmogrify1| replace:: ``pkgmogrify``\ (1)
+
+.. |depotd1m| replace:: ``pkg.depotd``\ (1M)
+
+.. |beadm1m| replace:: ``beadm``\ (1M)
+.. |libbe3lib| replace:: ``libbe``\ (3LIB)
+
+.. |OS_Name| replace:: OpenSolaris
diff --git a/doc/pkg5_docs/manifest_preprocessor.txt b/doc/pkg5_docs/manifest_preprocessor.txt
new file mode 100644
index 0000000..73a5eea
--- /dev/null
+++ b/doc/pkg5_docs/manifest_preprocessor.txt
@@ -0,0 +1,119 @@
+A ON manifest pre-processor needs the following attributes:
+
+1) Can selectively "comment out" parts of the
+ manifest.
+
+2) Can perform string substitution in manifests.
+
+3) Can include other files to be part of this manifest
+ using either inline include statements or command
+ line arguments. In either case, a search path
+ for these files can be specified via one or more
+ -I options ala cpp.
+
+4) Can transform actions matching certain
+ criteria, using specifications delivered as part of the
+ manifest (or one of the included files).
+
+Importer currently does the first two via simple recursive
+string substitution:
+
+Macros are of the form: $(MACRONAME) and can have
+any string value, including another macro. Macros
+are found in each line, repeatedly substituted
+until no more are found. Expressions of macro
+form w/o a value are passed through w/o modification.
+
+Eliding lines from a manifest is done by using a
+macro at the begining of the line that has a '#'
+value, which is the manifest comment character.
+As a practical matter, the macro names are assigned
+in the importer Makefile to facilitate understanding;
+eg
+
+$(sparc_ONLY) has the value '#' unless the current
+build is sparc; $(i386_ONLY) is similar. On sparc,
+$(ARCH64) is sparcv9, etc.
+
+The importer files then use these macros to allow
+a single file to be used for both architectures.
+
+For including files, the easiest mechanism would be
+some unique (distinct from either a comment, macro
+or action line) syntax such as:
+
+
+
+The specified path is searched w/o substitution for
+if it starts w/ "/"; otherwise any parameters specifed
+via -I are preprended and then searched for in order.
+
+Any files specified w/ -A would be searched for and appended
+to the manifest as if they had be specified w/
+at the end of the file.
+
+Once the manifest preprocessor had performed macro substitution
+and file inclusion, the last step would be executing any transformative
+steps specified in the files. These transformative steps would
+contain a set of matching criteria and the desired effect on the action:
+
+ operation>
+
+matching criteria would be of the form
+
+file|dir|signature|... any action types
+path=var/svc/manifest/.*xml
+mode=0?1[0-7][0-7][0-7]
+
+when python regexp rules would be used to find matching attributes.
+If multiple attributes are specified they all must be true for the
+transform to take effect, with exception that specifying multiple
+action types will match any specified.
+
+operation specification looks like this:
+
+drop # discard this action
+edit attribute regexp [replace] # apply regexp to value of attribute;
+ # if match occurs, substitute replace for
+ # portion of attribute value that matched.
+ # If replace is omitted, remove the
+ # the attribute values that match regexp
+set attribute value # set specified attribute to value
+add attribute value # add value to attribute
+
+Examples:
+
+Add tags to smf manifest so they're properly imported:
+
+ add refresh_fmri svc:/system/manifest-import:default>
+
+Add tags to gnome icon files to rebuild icon cache:
+
+ add restart_fmri svc:/application/desktop-cache/icon-cache:default>
+
+Change action location (including link targets) from usr/openwin to usr/X11:
+
+ edit path usr/openwin usr/X11>
+ edit target openwin X11>
+
+
+Command line
+------------
+
+pkgmog [-D macro=value] ... [-I includepath]... [-A filename] ... [inputfile [outputfile]]
+
+Where:
+
+inputfile is stdin if omitted
+outputfile is stdout if omitted
+
+-D defines the value of the specified macro; may be empty
+
+-I adds the specified path to search for included files
+
+-A appends the file to the end of the input as if the last line in the file
+ included it
+
+(pkgmog is short for package transmogrify)
+
+
diff --git a/doc/pkg5_docs/mediated-links.txt b/doc/pkg5_docs/mediated-links.txt
new file mode 100644
index 0000000..1c44973
--- /dev/null
+++ b/doc/pkg5_docs/mediated-links.txt
@@ -0,0 +1,269 @@
+Problem Statement
+=================
+
+It's common to want multiple versions of a component installed on a system.
+In order for this to work, each version must have no coincident pathnames.
+For most pathnames, it's easy to put them in a versioned directory or give
+them a versioned name. But for some frequently used pathnames (an
+executable expected to be in $PATH, or a man page), it's highly convenient
+to provide an unversioned path (possibly as a link to a versioned path) for
+casual use.
+
+This of course leads to a conflict when multiple component versions all
+want to provide the same unversioned path. We need a mechanism by which we
+can manage these paths.
+
+The most typical desired behavior for these paths is that they refer to the
+latest version available; that is, if component foo has versions 1.2, 1.3,
+and 1.4 installed, then unversioned pathnames for foo should refer to the
+1.4 version. However, there may be reasons that the package vendor, the
+site administrator, or even the local system administrator may want to
+override this choice. Since properly written code uses the versioned
+pathnames, the reference should be allowed to switch at arbitrary times.
+
+Another related problem is the delivery of different components which
+provide similar functionality. This differs from the previously stated
+problem in that the components don't share the same versioning space, and
+so need a different namespace from which the desired component can be
+chosen.
+
+Finally, a versioned component may also have different implementations
+(indeed, there may be multiple implementation axes), so we need to handle
+both problems at once.
+
+
+Proposal
+========
+
+We assert that this problem can be restricted to pathnames which can be
+replaced by symbolic links. Although a solution could apply to other
+filesystem entry types, the implementations would be significantly more
+difficult, and likely wouldn't provide any real benefit.
+
+To that, we introduce the notion of "mediated links", being symlinks or
+hardlinks which are subject -- when necessary -- to mediation in the case of
+conflict. Mediated links are link or hardlink actions with some extra metadata
+that allows the packaging system to treat conflicts specially.
+
+The one required attribute is "mediator", whose value denotes the entry in
+the namespace shared by all the pathnames participating in a given
+mediation group. For instance, the mediator attribute value for all
+/usr/bin/python and /usr/share/man/man1/python.1 links might be "python".
+The value of this attribute must be an alphanumeric string.
+
+For a link mediated solely by version, the "mediator-version" attribute is
+required. This is the version (expressed as a dot-separated sequence of
+nonnegative integers) of the interface described by the "mediator"
+attribute, and not necessarily related to the version of the implementation
+delivered by the package, or by the version in the package FMRI. For
+instance, links participating in the "python" mediation might have
+"mediator-version" set to "2.4", "2.6", "3.2", etc.
+
+When resolving the conflict in mediated links, pkg(5) will normally choose
+the link with the greatest value of "mediator-version". The package vendor
+may override this selection by adding the attribute "mediator-priority"
+with the value "vendor" to its preferred link in a package it delivers.
+Similarly, site administrators may override the selection by version or the
+vendor override by adding the attribute "mediator-priority" with the value
+"site" to its preferred link. The local system administrator may override
+this selection with the commandline specified below.
+
+Mediation may be performed by implementation in addition to or instead of
+by version. In this case, the attribute is "mediator-implementation" and
+the value must be a string containing only alphanumeric characters and '-'
+(preferably short and descriptive). Implementation strings are not considered
+to be ordered, and so one must be chosen explicitly, or the system will choose
+arbitrarily (first in lexical order). As with versioned mediation,
+"mediator-priority=vendor" and "mediator-priority=site" can be used to override
+the default behavior, and the local administrator can set the preference
+explicitly.
+
+If the implementation itself can be or is versioned, then the value may
+have a '@' and a version appended to it. If multiple versions of an
+implementation exist, the default will be to choose the greatest as defined
+by those versions, similarly to the purely version-mediated scenario.
+
+By default, every time a package operation is performed, the "best" mediation
+will be selected and then applied to all packages. However, the local
+administrator may explicitly set both the version and implementation for a
+mediation, or only one of them, in which case the system will select the "best"
+mediation with the matching component.
+
+All links delivered to a given pathname must participate in mediation or not
+at all. If some links delivered to a given pathname are mediated, and some
+are not, package operations will fail with a conflicting error message. Likewise,
+all mediated links delivered to a given pathname must share the same mediator.
+
+However, not all mediator versions and implementations need to provide a link at
+a given path. If a version and/or implementation doesn't provide a link, then the
+link is removed when that version is selected.
+
+The commandline used for locally administering mediated links is as
+follows:
+
+ - pkg set-mediator [-V ] [-I ] mediator ...
+
+ Sets the mediator to a given version and/or implementation.
+
+ - pkg unset-mediator [-V] [-I] mediator ...
+
+ Reverts the mediator to the system default. If the -V option is
+ supplied, revert the version-mediated aspect to the system default, but leave
+ the implementation mediation in place. If the -I option is supplied,
+ revert the implementation-mediated aspect to the system default, but
+ leave the version mediation in place. If neither -V or -I are specified,
+ both the version and implementation will be reverted to the system default.
+
+ - pkg mediator [ ...]
+
+ Display information about the mediators on the system. When one or
+ more mediators are specified, display information only about the specified
+ mediators. The information should include the mediator name, selected
+ version and implementation, and how each component was chosen by the system.
+
+Examples
+========
+
+Mediation by version
+--------------------
+
+The canonical case for version mediation is a language platform. In this
+case, we'll investigate Python. Two paths that might be mediated are the
+python interpreter executable and its manpage. The runtime/python-26
+package would deliver
+
+ file path=usr/bin/python2.6 ...
+ file path=usr/share/man/man1/python2.6.1 ...
+ link path=usr/bin/python target=python2.6 mediator=python \
+ mediator-version=2.6
+ link path=usr/share/man/man1/python.1 target=python2.6.1 mediator=python \
+ mediator-version=2.6
+
+and the runtime/python-24 package would deliver
+
+ file path=usr/bin/python2.4 ...
+ file path=usr/share/man/man1/python2.4.1 ...
+ link path=usr/bin/python target=python2.4 mediator=python \
+ mediator-version=2.4
+ link path=usr/share/man/man1/python.1 target=python2.4.1 mediator=python \
+ mediator-version=2.4
+
+If only runtime/python-26 were installed, then /usr/bin/python would be
+linked to python2.6. Likewise, if only runtime/python-24 were installed,
+then /usr/bin/python would be linked to python2.4. If both were installed,
+then by default, 2.6 would be found to be greater than 2.4, and the link
+would point to python2.6.
+
+If the vendor of the two packages wished, they could deliver a package
+"runtime/python" which would simply have a dependency on the desired
+default version.
+
+If the local administrator preferred /usr/bin/python to be version 2.4,
+then she could run
+
+ pkg set-mediator -V 2.4 python
+
+after which running "python" would run the 2.4 interpreter, and "man
+python" would display the manual for the 2.4 interpreter.
+
+
+Vendor override
+---------------
+
+If, in the above scenario, the vendor of runtime/python-26 and
+runtime/python-24 wished to override the default behavior and have the 2.4
+interpreter be the default, then the runtime/python-24 package would
+deliver
+
+ file path=usr/bin/python2.4 ...
+ file path=usr/share/man/man1/python2.4.1 ...
+ link path=usr/bin/python target=python2.4 mediator=python \
+ mediator-version=2.4 mediator-priority=vendor
+ link path=usr/share/man/man1/python.1 target=python2.4.1 mediator=python \
+ mediator-version=2.4 mediator-priority=vendor
+
+Thus in the case where both runtime/python-26 and runtime/python-24 were
+installed, /usr/bin/python would point to python2.4, despite 2.6 being
+greater than 2.4. The "runtime/python" package would likely then contain a
+a dependency on "runtime/python-24". The local administrator could
+override this choice to 2.6 by running
+
+ pkg set-mediator -V 2.6 python
+
+
+Mediation by implementation
+---------------------------
+
+The canonical case for implementation mediation is the editor vi. There
+are multiple implementations of vi, but there is no versioned specification
+which each implements. We will examine vim, nvi, and the legacy Solaris
+vi.
+
+In the package editor/vim, we deliver
+
+ file path=usr/bin/vim
+ link path=usr/bin/vi target=vim mediator=vi mediator-implementation=vim
+
+In the package editor/svr4-vi, we deliver
+
+ file path=usr/has/bin/vi
+ link path=usr/bin/vi target=../has/bin/vi mediator=vi mediator-implementation=svr4
+
+And in the package editor/nvi, we deliver
+
+ file path=usr/bin/nvi
+ link path=usr/bin/vi target=nvi mediator=vi mediator-implementation=nvi
+
+If editor/vim were installed on the system without either of the others,
+then /usr/bin/vi would point to vim. If either of the other packages were
+later added, the /usr/bin/vi would continue to run vim, until the local
+administrator were to run
+
+ pkg set-mediator -I svr4 vi
+
+which would switch the link to run legacy Solaris vi. If the packages were
+all installed simultaneously, the link would be chosen arbitrarily. The
+vendor ought to deliver a vendor priority tag:
+
+ link path=usr/bin/vi target=vim mediator=vi mediator-implementation=vim \
+ mediator-priority=vendor
+
+so that the desired default is always chosen.
+
+
+Mediation with multiple implementations
+---------------------------------------
+
+Vim can be configured with a handful of different compile-time options.
+For this example, we'll focus on --with-features=tiny (the minimal
+configuration) and --with-features=huge (the maximal configuration, sans
+GUI and external languages).
+
+As before, we deliver the package editor/svr4-vi:
+
+ file path=usr/has/bin/vi
+ link path=usr/bin/vi target=../has/bin/vi mediator=vi mediator-implementation=svr4
+
+We deliver two vim packages -- one for tiny, one for huge -- with a new
+"vim" implementation-based mediator, set to "tiny" for the tiny package and
+"huge" for the huge package. In addition, an implementation-based mediated
+link is delivered to /usr/bin/vi which targets vim if the vi mediator is
+set to vim. Since these two actions are identical, there is no conflict.
+
+Here is vim-tiny:
+
+ file path=usr/bin/vim-tiny
+ link path=usr/bin/vim target=vim-tiny mediator=vim mediator-implementation=tiny
+ link path=usr/bin/vi target=vim mediator=vi mediator-implementation=vim
+
+and editor/vim-huge:
+
+ file path=usr/bin/vim-huge
+ link path=usr/bin/vim target=vim-huge mediator=vim mediator-implementation=huge
+ link path=usr/bin/vi target=vim mediator=vi mediator-implementation=vim
+
+Thus to ensure that /usr/bin/vi is the huge vim, the administrator would
+have to run
+
+ pkg set-mediator -I huge vim
+ pkg set-mediator -I vim vi
diff --git a/doc/pkg5_docs/multi-platform.rst b/doc/pkg5_docs/multi-platform.rst
new file mode 100644
index 0000000..7576586
--- /dev/null
+++ b/doc/pkg5_docs/multi-platform.rst
@@ -0,0 +1,57 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+pkg
+MULTI-PLATFORM
+
+The core pkg(5) technology is generic enough to be useful across multiple platforms
+(e.g. Windows and Linux). The full range of supported platforms are listed here:
+http://wikis.sun.com/display/IpsBestPractices/OS+Platform+Support
+
+The following modules within the pkg(5) source base are multi-platform:
+ - the CLIs (client.py, publish.py, depot.py, pull.py)
+ - src/modules (the core of pkg(5))
+ - src/tests (except the CLI tests do not run on Windows)
+ - src/man
+ - src/web
+ - src/po (except for the GUI messages which are OpenSolaris-only)
+
+The following modules are not multi-platform (only supported on OpenSolaris):
+ - src/brand
+ - src/gui, src/um and the start scripts (packagemanger.py, updatemanager.py,
+ and updatemanagernotifier.py)
+ - pkgdefs
+ - SMF support: src/svc-pkg-depot, src/pkg-server.xml, src/pkg-update.xml
+ - src/util
+
+
+The following modules are only used for non-OpenSolaris support:
+ - src/scripts
+
+Multi-platform support is focused on providing support for user images as the
+operating system software is not delivered for other platforms using pkg(5).
+
+Development best practices for writing multi-platform pkg(5) code are available
+here: http://opensolaris.org/os/project/pkg/devinfo/bestpractices/.
+
+Information about using multi-platform pkg(5) and pre-built binaries
+are available here: http://wikis.sun.com/display/IpsBestPractices
diff --git a/doc/pkg5_docs/multi-platform.txt b/doc/pkg5_docs/multi-platform.txt
new file mode 100644
index 0000000..cbc0048
--- /dev/null
+++ b/doc/pkg5_docs/multi-platform.txt
@@ -0,0 +1,34 @@
+pkg
+MULTI-PLATFORM
+
+The core pkg(5) technology is generic enough to be useful across multiple platforms
+(e.g. Windows and Linux). The full range of supported platforms are listed here:
+http://wikis.sun.com/display/IpsBestPractices/OS+Platform+Support
+
+The following modules within the pkg(5) source base are multi-platform:
+ - the CLIs (client.py, publish.py, depot.py, pull.py)
+ - src/modules (the core of pkg(5))
+ - src/tests (except the CLI tests do not run on Windows)
+ - src/man
+ - src/web
+ - src/po (except for the GUI messages which are OpenSolaris-only)
+
+The following modules are not multi-platform (only supported on OpenSolaris):
+ - src/brand
+ - src/gui, src/um and the start scripts (packagemanger.py, updatemanager.py,
+ and updatemanagernotifier.py)
+ - pkgdefs
+ - SMF support: src/svc-pkg-depot, src/pkg-server.xml, src/pkg-update.xml
+ - src/util
+
+The following modules are only used for non-OpenSolaris support:
+ - src/scripts
+
+Multi-platform support is focused on providing support for user images as the
+operating system software is not delivered for other platforms using pkg(5).
+
+Development best practices for writing multi-platform pkg(5) code are available
+here: http://opensolaris.org/os/project/pkg/devinfo/bestpractices/.
+
+Information about using multi-platform pkg(5) and pre-built binaries
+are available here: http://wikis.sun.com/display/IpsBestPractices
diff --git a/doc/pkg5_docs/on-disk-format.txt b/doc/pkg5_docs/on-disk-format.txt
new file mode 100644
index 0000000..4f13243
--- /dev/null
+++ b/doc/pkg5_docs/on-disk-format.txt
@@ -0,0 +1,1050 @@
+pkg(5): image packaging system
+
+This information is Copyright (c) 2010, Oracle and/or its affiliates.
+All rights reserved.
+
+ON-DISK FORMAT PROPOSAL
+
+1. Introduction
+ 1.1. Date of This Document:
+
+ 06/02/2010
+
+ 1.2. Name of Document Author/Supplier:
+
+ Shawn Walker, Oracle,
+ on behalf of the pkg(5) project team
+
+ 1.3. Acknowledgements:
+
+ This document is largely based on comments from the following
+ individuals to whom the author is exceedingly indebted to:
+
+ - Danek Duvall
+ - Mike Gerdts
+ - Stephen Hahn
+ - Krister Johansen
+ - Dan Price
+ - Brock Pytlik
+ - Bart Smaalders
+ - Peter Tribble
+
+2. Project Summary
+
+ 2.1. Project Description:
+
+ "...the repository can be archived up, put on a CD, memory
+ stick, 2D barcode, and protected by the Black Knight, fire
+ moats, komodo dragons, etc." - Danek Duvall
+
+ pkg(5) is primarily a network-oriented binary packaging system.
+ Although some of the tools it provides support filesystem-based
+ operations for publication, the primary expected use for package
+ operations (such as install, update, search, etc.) is between an
+ intelligent client and one or more servers that provide access
+ to a package repository and/or other interactive services.
+
+ This project seeks to define and establish an on-disk format
+ (and corresponding container format), for the pkg(5) system,
+ with the intent that it can enable the ubiquitous, transparent
+ use of package data from filesystem-based resources.
+
+ The changes proposed by this project are evolutionary, not
+ revolutionary, in nature. In particular, this project seeks
+ to refine and adopt the existing repository format used by the
+ pkg(5) depot server as the on-disk format. Supplementary to
+ that, it also seeks the addition of a container format to ease
+ provisioning of the on-disk format, and the unification of the
+ scheme used by the client and server to store package data.
+
+ 2.2. Problem Area:
+
+ For some deployments, network-based package data access is not
+ possible or is undesirable. Concerns often cited in this area
+ include:
+
+ - lack of access control or ability to easily integrate with
+ existing access control systems,
+
+ - inability to rely on alternative (or existing) provisioning
+ arrangements (such as NFS-based file servers),
+
+ - environmental or procedural requirements that prohibit the
+ ability to or use of a network-based service,
+
+ - characteristics of network protocols (such as HTTP, etc.) that
+ artificially limit functionality or performance (as opposed to
+ iSCSI or other alternatives),
+
+ - ease of administration of filesystem-based resources, and
+
+ - ease of transferring package data.
+
+3. Project Technical Description:
+ 3.1. Details:
+
+ This project defines an on-disk format (and corresponding con-
+ tainer format) that is intended for the supplemental or complete
+ provisioning of package data at all stages of the package life-
+ cycle. That is, when package data is published, stored by the
+ client or server, or otherwise used during package operations.
+
+ The on-disk format (defined in detail later in this document)
+ is intended to be distributable in its raw form (a pre-defined
+ structure of directories and files) or within a container format
+ (such as a zip file, etc.).
+
+ Out of necessity, the use of filesystem-based resources (such as
+ those provided by the on-disk format) will sometimes limit the
+ operations that can be performed to a subset of those normally
+ available when interacting with a network-based repository. For
+ example, search and publisher configuration may not be possible,
+ and purely interactive services such as the BUI (Browser UI)
+ offered by the depot server for a repository, RSS feeds, and
+ others will not be available.
+
+ Because of the wide-ranging impact of the changes required to
+ implement this functionality, it is intended that the project
+ be implemented in the following sequence:
+
+ - Client Support for filesystem-based Repository Access
+
+ - Depot Storage, Client Transport and Publication Tool Update
+
+ - Client Storage and Image Format Update
+
+ - Client and Depot Support for On-Disk Archive Format
+
+ 3.2. Bug/RFE Number(s):
+
+ As an example of the kinds of defects and RFEs intended to be
+ resolved by this project, see the following selection of
+ defect.opensolaris.org bug IDs:
+
+2152 standalone package support needed (on-disk format)
+166 depot doesn't set directory mode when creating directories
+2086 validate that a repository is really a repository in pkg.depotd
+6335 publisher repo with invalid certificate information shouldn't
+ prevent querying other repos
+6576 pkg install/update support for temporary publisher origins desired
+6940 depot support for file:// URI desired
+7213 ability to remove published packages
+7273 manifests should be arranged in a hierarchy by publisher
+7276 /var/pkg metadata needs reorg (looks busy)
+8433 client and pull need to refer to refer to "repository" instead of
+ "server"
+8722 advanced repository metadata store needed
+8725 versioning information for depot and repository metadata needed
+9571 CachedManifest should be named FactoredManifest
+9572 CachedManifest should allow consumers to specify cache location
+9872 publication api should use new transport subsystem
+9933 ability to control repository creation behaviour or removal of it
+10244 caching dictionaries as a class variable prevents multi-image and
+ repo search
+11362 Image update dying when trying to talk to a disabled and offline
+ publisher
+11740 publishers with installed packages should not be removable
+12814 publisher prefixes should be forcibly lower-cased or case
+ insensitive
+14802 ability to have separate read / write download caches
+15320 pkgsend will traceback if unable to parse server error response
+15371 repository property defaults opensolaris.org-specific
+
+ 3.3. In Scope:
+
+ Filesystem-based data resourcing for package operations.
+
+ 3.4. Out of Scope:
+
+ Package signing and fine-grained access control for package
+ repositories.
+
+4. On-Disk Format Technical Description:
+ 4.1. Overview:
+
+ The on-disk format is intended to exist both in a raw format as
+ a pre-defined structure of directories and files, and in an
+ archive format which is primarily a simple container for
+ the raw format.
+
+ 4.2. Raw Format:
+
+ 4.2.1. Goals:
+ The goals for the raw on-disk format include:
+
+ - unification of client and server package data storage
+ for data common to both,
+
+ - transparent usage of package data regardless of operation
+ or use by client or server,
+
+ - ease in composition and decomposition of package data
+ stored within by publisher or package,
+
+ - re-use of existing publication tools for on-disk format,
+
+ - enablement of future publication tools to automatically
+ be able to manipulate or use on-disk format, and
+
+ - ease of provisioning.
+
+ 4.2.2. Raw Format specification:
+
+ The pkg(5) repository format is a set of directories and
+ files that conform to a pre-defined structure.
+
+ For a version 3 repository (the current format), the
+ structure is as follows:
+
+ /
+ catalog/
+
+ index/
+
+ file/
+ /
+
+ pkg/
+ /
+
+ trans/
+
+ cfg_cache (optional repository configuration file)
+
+ Version 4 of the repository format eliminates the potential
+ for unintended collisions between package metadata from
+ different publishers and simplifies composition and decomp-
+ osition of repository content. The top-level is an optional
+ shared storage space for data common to all publishers in
+ the repository, while the publisher subdirectory contains
+ data specific to a publisher. It is essentially a nested
+ repository format, and can be defined as follows:
+
+ /
+ file/ (optional)
+ publisher/ (optional)
+ / (optional)
+ catalog/ (optional)
+
+ file/ (optional)
+ /
+
+ index/ (optional)
+ pkg/ (optional)
+ /
+
+ trans/ (optional)
+
+ pub.p5i (optional)
+ pkg5.repository (required)
+
+ By default, repository operations will store data in the
+ publisher-specific location found under publisher/
+ for new repositories.
+
+ In the case that the top-level file/ directory is used,
+ automatic decomposition of contents into its publisher-
+ specific components will not be possible unless
+ corresponding package manifests are also available.
+
+ To support easy composition, filtering, and creation of
+ package archives, directories above marked with the text
+ '(optional)' must not be required. The behaviour of
+ consumers accessing the contents of the repository should
+ be as follows based on the directory accessed:
+
+ - file/
+ This optional directory serves as a place to store file
+ data for more than one publisher. Package files are
+ stored in gzip format using a sha1sum of the file as the
+ filename, and then the first two letters of the filename
+ as the parent directory's name.
+
+ - publisher//catalog/
+ If absent, consumers should determine the list of
+ packages available based on the manifest files present
+ in the publisher/ subdirectory. If present, consumers
+ should expect v1 (or newer) catalog files, or none at
+ all, to be contained within.
+
+ - publisher//file/
+ Consumers should always check this subdirectory first
+ (if present) when retrieving package file data if the
+ publisher is known. Package files are stored in gzip
+ format using a sha1sum of the file as the filename, and
+ then the first two letters of the filename as the parent
+ directory's name.
+
+ - publisher//index/
+ If absent, search functionality should be disabled for
+ this publisher, or a fallback to 'slow manifest-based
+ search' performed. If present, consumers should expect
+ v1 (or newer) search files, or none at all, to be con-
+ tained within.
+
+ - publisher//pkg/
+ If absent, search must be disabled for this publisher
+ even if index is present. If present, manifests are
+ stored in pkg(5) manifest format using the uri-encoded
+ version of the package FMRI as the filename, and using
+ the uri-encoded package FMRI stem (name) as the parent
+ directory's name.
+
+ - publisher//trans/
+ If absent, this directory will be created during
+ publication operations. If present, in progress
+ transaction data is stored in a directory named
+ by the open time of the transaction as a UTC UNIX
+ timestamp plus an '_' and the URI-encoded package
+ FRMI. As an example:
+
+ 1245176111_pkg%3A%2FBRCMbnx%400.5.11%2C5.11-0.116
+ %3A20090616T181511Z
+
+ - publisher//pub.p5i
+ This pkg(5) information (p5i) file should contain
+ suggested configuration information for clients such as
+ origins, mirrors, alias, etc. Consumers can use this to
+ provide clients with initial or suggested configuration
+ information for a given publisher. If not present, the
+ publisher's identity should be assumed based on the
+ directory structure, while the refresh interval should
+ be assumed to be 4 hours.
+
+ - pkg5.repository
+ This file serves as an identifier and a place to store
+ configuration information specific to the repository.
+ It *is not* an equivalent to the existing cfg_cache
+ file which will no longer be used. Its format and
+ structure are as follows:
+
+ [repository]
+ version =
+
+ Any information found in the cfg_cache used in the previous
+ repository format related to a publisher is now stored in
+ the pub.p5i file for the related publisher. (Examples of
+ information include origins, mirrors, maintainer info,
+ etc.) As a result, the cfg_cache file is no longer used.
+
+ Any depot-specific properties, such as the feed icon, logo,
+ etc. are now completely managed using SMF or a user-provided
+ configuration file. This change was made not only to sim-
+ plify configuration, but to separate depot configuration
+ from repsitory configuration.
+
+ An example version 4 repository might be structured as
+ follows:
+
+ /
+ publisher/
+ example.com/
+ catalog/
+ catalog.attrs
+ catalog.base.C
+ file/
+ ff/
+ fffff277f5a8fb63e57670afc178415c2c5e706d
+ index/
+ __at_depend
+ ...
+ pkg/
+ package%2Fpkg/
+ 0.5.11%2C5.11-0.136%3A20100327T063139Z
+ trans/
+ 1245176111_pkg%3A%2FBRCMbnx%400.5.11%2C5.11-0.116
+ %3A20090616T181511Z
+ pub.p5i
+ example.net/
+ catalog/
+ catalog.attrs
+ catalog.base.C
+ file/
+ af/
+ affff277f5a8fb63e57670afc178415c2c5e706d
+ index/
+ __at_depend
+ ...
+ pkg/
+ package%2Fpkg/
+ 0.5.11%2C5.11-0.133%3A20090327T062137Z
+ trans/
+ 1245176111_pkg%3A%2FFAAMbnx%400.5.11%2C5.11-0.139
+ %3A20100616T181511Z
+ pub.p5i
+
+ pkg5.repository:
+ [repository]
+ version = 4
+
+ 4.3. Archive Format:
+
+ 4.3.1. Requirements:
+
+ The requirements for the on-disk archive format include:
+
+ - support for archives greater than 8GB in size,
+
+ - support for files in archive greater than 4GB in size,
+
+ - support for efficient storage of hard links,
+
+ - support for pathnames sigificantly greater than > 255
+ characters in length,
+
+ - core Python bindings exist or can be easily created using
+ an existing library,
+
+ - can be a container of compressed files, as opposed to a
+ compressed container of uncompressed files,
+
+ - open, royalty-free, well-documented format with wide
+ platform support and acceptance,
+
+ - multi-threaded decompression and compression possible,
+
+ - creation and basic manipulation of package archives
+ possible using widely-available tools,
+
+ - simple composition and filtering of its content should be
+ possible, and
+
+ - random access to the archive contents must be possible
+ without reading the entire archive file.
+
+ 4.3.2. Candidates:
+
+ A number of potential archive formats have been considered
+ for use, including:
+
+ - 7z (7-Zip)
+ - cpio
+ - pax (portable archive exchange format)
+ - ZIP
+
+ The evaluations provided for each format here are not in-
+ tended to be exhaustive; rather they focus on the specific
+ requirements of this project. For more information about
+ these formats, and the documents used to evaluate them,
+ please refer to section 6 of this proposal.
+
+ 4.3.3. 7z Evaluation:
+
+ The 7z format was rejected for the following reasons:
+
+ - Does not permit random access to archive contents or
+ requires the entire archive file to access the contents
+ and adding this would require a custom variation of 7z.
+
+ - Although the 7z format supports compression methods other
+ than LZMA, a primary motivator for using 7z would be the
+ ability to use LZMA natively as part of the conatiner
+ format. However, the tradeoffs in terms of CPU and memory
+ footprint currently make LZMA unsuitable for pkg(5) when
+ compared to other compression algorithms such as those
+ used by gzip(1).
+
+ - Use of the 7z format would require integration of the LZMA
+ SDK (which also provides a basic 7z API in C) and the cre-
+ ation of python bindings or the integration of a third
+ party's (such as pylzma).
+
+ - No native support for extended attributes or UNIX owner/
+ group permissions.
+
+ 4.3.4. cpio Evaluation:
+
+ The cpio format doesn't natively support random access to
+ archive contents, but the format itself doesn't prevent
+ this. An index could be added first file in the archive
+ with the information needed to provide fast, random access
+ to the archive contents.
+
+ The cpio format was rejected for the following reasons:
+
+ - The length of pathnames in cpio archives is limited to
+ 256 characters for the portable format.
+
+ - Available tools vary significantly in maximum archive size
+ support.
+
+ - The portable cpio format stores a copy of the file data
+ with every hard link in an archive instead of simply
+ storing a pointer to the source file in the archive.
+
+ 4.3.4. PAX Evaluation:
+
+ The PAX format meets all of the requirements except that of
+ random access to archive contents. However, the format
+ itself doesn't prevent this. A table of contents file could
+ be supplied as the first file in the archive with the info-
+ rmation needed to provide fast, random access to the con-
+ tainer contents.
+
+ 4.3.5. ZIP Evaluation:
+
+ The ZIP format meets all of the requirements listed above
+ (assuming that ZIP64 extensions are used), with the ex-
+ ceptions listed below for which it was rejected:
+
+ - The use or implementation of some of the functionality
+ documented in the .ZIP file format requires a license from
+ PKWARE.
+
+ - While random archive content access is possible, the ZIP
+ file format stores the index for the archive at the end of
+ the archive (as opposed to the beginning). This increases
+ the number of round trips that would be required for
+ potential remote random content access. It also means
+ that extraction requires multiple seeks to the end of the
+ file before any content can be extracted from the archive,
+ which can be detrimental to performance for some media
+ types (optical, etc.).
+
+ 4.3.6. Evaluation Conclusion:
+
+ Based on the requirements set forth in section 4.3.1, the
+ PAX format was selected as the on-disk archive format
+ for pkg(5) packages. However, to enable efficient access
+ to the archive contents, an index file needs to be present
+ as the first file in the archive.
+
+ Early evaluations of an unoptimised prototype were performed
+ using a repository containing all packages for build 136 and
+ unbundleds. The on-disk size of the repository was appox-
+ imately 4.98G. The resulting archive was 5.0G in size, with
+ an archive index file 9.7M in size (when the index was comp-
+ ressed using gzip).
+
+ First time access to the prototype archive for extraction of
+ a single file after creation yielded a total time of approx-
+ imately 5 seconds compared to approximately 36-42 seconds
+ for utilities such as pax(1), tar(1), or gtar(1).
+
+ Creation of the archive took 7 minutes, 35 seconds on a
+ custom-built Intel Core 2 DUO E8400, with 8GB Memory,
+ and a 1TB 10000 RPM SATA Drive w/ 64MB Cache.
+
+ 4.3.7. Package Archive Specification:
+
+ pkg(5) archive files will have an extension of 'p5p' which
+ will stand for 'pkg(5) package'. The format of these
+ archives matches that defined by IEEE Std 1003.1, 2004 for
+ the pax Interchange Format, with the exception that the
+ first archive entry is tagged with an extended pax archive
+ header that specifies the archive version and the version
+ of the pkg(5) API that was used to write it. In addition,
+ the file for the first archive entry must be the index
+ file file for the package archive. The layout can be
+ visualised as follows:
+
+ .--------------------------------------------------------.
+ | ustar header for pax header global archive data |
+ .--------------------------------------------------------.
+ | pax global extended header data for archive |
+ .--------------------------------------------------------.
+ | ustar header for pax header for archive index file |
+ .--------------------------------------------------------.
+ | pax extended header data for archive index file |
+ .--------------------------------------------------------.
+ | ustar header for package archive index file |
+ .--------------------------------------------------------.
+ | file data for package archive index file |
+ .--------------------------------------------------------.
+ | remaining archive data |
+ .________________________________________________________.
+
+ The archive and API version is stored in the header of the
+ index file instead of the global header for two reasons:
+ first, any headers in the global header are treated as
+ though they apply to every entry in the archive, and
+ secondly, the pax specification states that global headers
+ should not be used with interchange media that could suffer
+ partial data loss during transport. Since the archive
+ version primarily serves as a way for clients to reliably
+ determine if a "standard" pax archive versus one with an
+ index is being read, this approach seems reasonable.
+
+ The reason for this limitation is to ensure that clients
+ performing selective archive extraction can be guaranteed
+ to find the location and size of the package archive index
+ file without knowing the size of the header for the index
+ file in advance (this layout ensures that clients can
+ find the archive index and/or identify the archive in
+ the first 2048 bytes).
+
+ In addition, pkg(5) archives in this format make remote,
+ selective archive access possible. For example, a client
+ could request the first 2048 bytes of a pkg(5) archive file
+ from a remote repository, identify the offsets of the index
+ and then retrieve it using a HTTP/1.1 byte-ranges request.
+ Once it has the archive index file, it can then perform
+ additional byte-range requests to selectively transfer the
+ the data for a set of specific files from the archive. This
+ convention also optimises access to the archive for sources
+ that are heavily biased towards sequential reads.
+
+ The index file must be named using the following template
+ and be compressed using the gzip format described by RFCs
+ 1951 and 1952, and formatted according to section 4.3.8:
+
+ p5p.index..v.gz
+
+ is an integer in string form that
+ indicates which index file this is. The number only
+ exists so that each index file can remain unique in
+ the archive. An archive may contain multiple index
+ files to support fast archive additions.
+
+ is an integer in string form that
+ indicates the version of the index file. The initial
+ version for this proposal will be '0'.
+
+ However, if the first file in the archive is found to not
+ use the layout or format shown above, or any of the index
+ files in the archive are not in a format supported by the
+ client (version too old or too new), the archive must be
+ treated as a standard pax archive and some operations may
+ not be possible or experience degraded performance. The
+ same is also true if the index file is found to not match
+ the archive contents.
+
+ All entries in the archive (excluding any archive index
+ files) must conform to the repository layout specified in
+ section 4.2.2 of this proposal.
+
+ Since a pkg(5) repository can contain one or more packages,
+ pkg(5) archive files can also contain the data for one or
+ more packages. This allows easy redistribution of a single
+ package and all of its dependencies in a single file.
+
+ Finally, it should be noted that only ascii character path-
+ names are expected in the archive as the raw repository
+ format does not use or support unicode pathnames.
+
+ 4.3.8. Package Archive Index Specification:
+
+ The pkg(5) archive index file enables fast, efficient access
+ to the contents of an archive. It contains an entry for all
+ files in the archive excluding the index file itself in the
+ following format (also referred to as index format version
+ 0):
+
+ NULNULNULNUL
+ NULNL
+
+ is a string containing the pathname of the file
+ in the archive using only ascii characters. It can be
+ up to 65,535 bytes in length.
+
+ is an unsigned long long integer in string form
+ containing the relative offset in bytes of the first
+ header block for the file in the archive. The offset is
+ relative to the end of the last block of the index file
+ in the archive they are listed in.
+
+ is an unsigned long long integer in string
+ form containing the size of the file's entry in bytes
+ in the archive (including archive headers and trailers
+ for the entry).
+
+ is an unsigned long long integer in string form
+ containing the size of the file in bytes in the archive.
+
+ is a single character representing the type
+ of the file in the archive. Possible values are:
+ 0 Regular File
+ 1 Hard Link
+ 2 Symbolic Link
+ 5 Directory or subdirectory
+
+ All values not listed above are reserved for future
+ use. Unrecognised values should be treated as a
+ regular file.
+
+ An example set of entries would appear as follows:
+
+ pkg5.repositoryNUL0NUL546NUL2560NUL0NUL
+ pkgNUL2560NUL0NUL1536NUL5NUL
+ pkg/service%2Ffault-managementNUL4096NUL0NUL1536NUL5NUL
+
+ It should be noted that other possible formats were
+ evaluated for the index file, including those based
+ on: JSON, XDR, and python's pack. However, all other
+ formats were found to be deficient for one or more
+ of the following reasons:
+
+ - larger in size
+
+ - no streaming support (required entire index file be
+ loaded into memory)
+
+ - significantly greater parsing times using currently
+ available Python libraries
+
+ - required developing an envelope format that could
+ contain the encoded data
+
+5. Proposed Changes:
+
+ 5.1. Client Support for filesystem-based Repository Access:
+
+ The pkg.client.api provided by pkg(5) will be updated to allow
+ access to repositories via the filesystem. All functionality
+ normally offered by pkg.depotd will be supported.
+
+ pkg(1) and packagemanager(1) will be modified to support the
+ use of URIs using the 'file' scheme. No user visible changes
+ will be made to any existing subcommands or options except
+ that URIs using the 'file' scheme will be allowed.
+
+ When accessing repositories using the 'file' scheme, clients
+ by default will not copy package file data into the client's
+ cache (e.g. /var/pkg/download). Instead, the transport system
+ will treat configured repositories as an additional read-only
+ cache.
+
+ 5.2. Depot Storage, Client Transport and Publication Tool Update:
+
+ The pkg.server.repository module will be updated to support
+ the new repository format outlined in section 4.2.2. Existing
+ repositories will not automatically be upgraded, while new
+ repositories will use the new format. A new administrative
+ command detailed below has been introduced to allow upgrading
+ existing repositories to the new format.
+
+ These changes will automatically allow the client to access
+ repositories in the new format when using filesystem-based
+ access. Older clients will remain unable to access repo-
+ sitories in the new format.
+
+ The client transport system will be updated to support all
+ publication operations and the publication tools and project
+ private APIs will be changed to use the client transport
+ system.
+
+ The '-d' option of pkgrecv(1) will be changed such that if
+ the name of a file with a '.p5p' extension is specified,
+ and that file does not already exist, a pkg(5) archive
+ file will be created containing the specified packages.
+ If the file already exists, it will exit with an error.
+ When pkgrecv(1) creates pkg(5) archive files, it will omit
+ catalog and index data.
+
+ Due to the transport changes above, pkgrecv(1) will also
+ be able to use pkg(5) archive files as a source of package
+ data. pkgsend(1) will not support the use of pkg(5)
+ archive files as a destination due to the publication
+ model it currently uses.
+
+ To support the expanded multiple publisher version 4 format
+ of repositories, the depot server will be updated to respond
+ to requests as follows:
+
+ - If clients include the publisher prefix as part of the request
+ path, then responses will be for that specific publisher's
+ data. For example:
+
+ http://localhost/dev/opensolaris.org/manifest/
+ 0/opensolaris.org/backup%2Fareca/7.1%2C5.11-0.134
+ %3A20100302T005731Z
+
+ http://localhost/dev/file/0/opensolaris.org/
+ 2ce6c746c85cd7ac44571d094b53c5fe1bfc32c8
+
+ - The default publisher specified in the depot configuration
+ will be used when responding to requests for operations that
+ do not include the publisher prefix. For example:
+
+ http://localhost/dev/manifest/0/
+ backup%2Fareca/7.1%2C5.11-0.134%3A20100302T005731Z
+
+ ...provides a response identical to the first case where the
+ publisher prefix was provided as part of the request. Those
+ expecting to maintain a large population of older clients
+ should reassign publisher URLs down a level, to include the
+ publisher explicitly although this is not required for
+ correct operation.
+
+ A new utility named pkgrepo will be added to facilitate the
+ creation and management of pkg(5) repositories. It will have
+ the following global options:
+
+ -s repo_uri_or_path
+ A URI or path specifying the location of a pkg(5)
+ package repository.
+
+ -? / --help
+
+ It will have the following subcommands:
+
+ create
+ Creates a pkg(5) repository at the specified location.
+ Can only be used with filesystem-based repositories.
+
+ publisher [ ...]
+ Lists the publishers of packages in the repository:
+
+ PUBLISHER PACKAGES VERSIONS UPDATED
+
+
+ ...
+
+ rebuild
+ Discards any catalog, search or other cached informaqtion
+ found in the repository and then re-creates it based on
+ the current contents of the repository. Can only be used
+ with filesystem-based repositories.
+
+ refresh
+ By default, catalogs any new packages found in the repo-
+ sitory and updates search indices. This is intended for
+ use with deferred publication (--no-catalog or --no-index
+ options of pkgsend). Can only be used with filesystem-based
+ repositories.
+
+ Options:
+ --no-catalog - doesn't add new packages
+ --no-index - doesn't refresh search indices
+
+ remove fmri_pattern ...
+ Removes the specified package(s) from the repository.
+ If more than one match is found for any given pattern,
+ the exact FMRI must be provided.
+
+ upgrade
+ Can only be used with filesystem-based repositories.
+ Upgrades the repository to the most current format if
+ possible.
+
+ Has these options:
+
+ -n determine whether the upgrade could be formed and exit
+
+ -v show a summary of what will be done, the current format
+ of the repository and what it will be upgraded to
+
+ 5.3. Client Storage and Image Format Update:
+
+ To simplify and unify the storage format used by the client,
+ and pkg(5) repositories, the format of the client image
+ will be changed to use the structure described below.
+
+ For a version 3 image (the current format), the structure is as
+ follows:
+
+
+ download/
+ /
+
+ file/
+ gui_cache/
+ history/
+ index/
+ lost+found/
+ pkg/
+ /
+ /
+ manifest
+ manifest.
+ publisher/
+ /
+ catalog/
+ certs/ (optional)
+ last_refreshed (optional)
+ state/
+ installed/
+
+ known/
+
+ tmp/
+ cfg_cache
+ lock
+
+ For a version 4 image (the proposed format), the structure is
+ as follows:
+
+
+ cache/
+ index/
+
+ publisher/
+ /
+ catalog/
+
+ pkg/
+ /
+ /
+
+ tmp/
+
+ gui_cache/
+
+ history/
+
+ license/
+ /
+
+ lost+found/
+
+ publisher/
+ /
+ certs/
+
+
+ ssl/
+ client ssl certificates>
+ state/
+ installed/
+
+ known/
+
+ pkg5.image (client configuration file; was cfg_cache)
+
+ A new property named 'version' will be added to the image
+ and will be readonly (cannot be set using the set-property
+ subcommand of pkg(1)).
+
+ Existing images will not automatically be upgraded to the new
+ format. To enable the upgrading of existing images to newer
+ formats, the following subcommands will be added:
+
+ update-format
+ Updates the format of the client's image to the current
+ format if possible.
+
+ 5.4. Client and Depot Support for On-Disk Archive Format:
+
+ The pkg.server.repository module will be updated to support
+ the serving of a repository in readonly mode using a pkg(5)
+ archive file.
+
+ The pkg.client.api transport system will be updated to support
+ the usage of a pkg(5) archive file as an origin for package
+ data.
+
+ To support the specification of temporary origins, the install
+ and update subcommands will be modified by adding a '-g' option
+ to specify additional temporary package origin URIs or
+ the path to a pkg(5) archive file or pkg(5) info file. The
+ '-g' option may be specified multiple times. As an example:
+
+ $ pkg install -g /path/to/foo.p5p \
+ -g http://mytemprepo:10000/ \
+ -g file:/path/to/bar.p5p \
+ foo bar localpkg
+
+ pkg(5) archive files used as a source of package data during an
+ install or update operation will have their content cached by
+ the client before the operation begins. Any publishers found
+ in the archive will be temporarily added to the image if they do
+ not already exist. Publishers that were temporarily added but
+ not used during the operation will be removed after operation
+ completion or failure. Any package FMRIs or patterns provided
+ will be matched using only the sources provided using '-g'.
+
+ The pkg list and pkg info commands will also be updated by
+ adding the '-g' option described above, with the exception
+ that the '-g' option may only be specified once, and only
+ the source named will be used for the operation.
+
+ Using '-g' with the pkg list subcommand implies '-n' by default,
+ unless '-f' is specified; it also implies '-a'. To list all
+ versions, the '-f' option must be used. As an example:
+
+ $ pkg list -g /path/to/foo.p5p
+ NAME (PUBLISHER) VERSION STATE UFOXI
+ bar (example.com) 1.0-0.133 known -----
+ foo (example.com) 1.0-0.133 installed -----
+
+ $ pkg list -g file:/path/to/foo.p5p
+ NAME (PUBLISHER) VERSION STATE UFOXI
+ bar (example.com) 1.0-0.133 known -----
+ foo (example.com) 1.0-0.133 installed -----
+
+ $ pkg list -f -g http://example.com/multi_foo.p5p
+ NAME (PUBLISHER) VERSION STATE UFOXI
+ foo (example.com) 1.0-0.133 installed u----
+ foo (example.com) 2.0-0.133 known u----
+ foo (example.com) 3.0-0.133 known -----
+
+ $ pkg list -g file:/path/to/repo
+ NAME (PUBLISHER) VERSION STATE UFOXI
+ repopkg (example.com) 2.0-0.133 known -----
+
+ $ pkg list -g http://myrepo:10000
+ NAME (PUBLISHER) VERSION STATE UFOXI
+ localpkg (example.org) 3.0-0.133 known -----
+
+ Using '-g' with the pkg info subcommand implies '-r'. The '-l'
+ option cannot be used in combination with '-g'. As an example:
+
+ $ pkg info -g /path/to/bundle.p5p
+ Name: bar
+ Summary: A useful complement to foo.
+ State: Not Installed
+ ...
+ Name: foo
+ Summary: Provides useful utilities.
+ State: Installed
+ ...
+
+ '-g' was chosen for the option usage described above to match
+ the '-g' already used by set-publisher and image-create for
+ origins, and due to the unfortunate existing usage of '-s'
+ by the 'pkg list' subcommand.
+
+6. Reference Documents:
+
+ Project team members and community members have provided a number of
+ informal comments that served as the basis for the goals of this
+ project:
+
+ - "new on-disk format?", 18 Jan. 2008:
+ http://markmail.org/thread/2kg6w5bfwp4x3knc
+
+ - "reorganising the repository and client metadata", 23. Sep. 2009:
+ http://markmail.org/thread/stfrosvx3v6if2fi
+
+ - "ZAP - Zip Archive Packaging", Sep. 2007:
+ http://markmail.org/thread/ijyq3mlrhaofccgx
+
+ In addition, the following materials were referenced when writing
+ this proposal:
+
+ - "7z", 12 Apr. 2010:
+ http://en.wikipedia.org/wiki/7z
+
+ - "RFC2616: HTTP/1.1 Header Field Definitions", 01 Sep. 2004:
+ http://www.w3.org/Protocols/rfc2616/
+ rfc2616-sec14.html#sec14.35.1
+
+ - "cpio", 21 Mar. 2010:
+ http://en.wikipedia.org/wiki/Cpio
+
+ - "copy file archives in and out", 26 Mar. 2007:
+ http://heirloom.sourceforge.net/man/cpio.1.html
+
+ - "The gzip file format", Date Unknown:
+ http://www.gzip.org/format.txt
+
+ - "DragonFly File Formats Manual, cpio -- format of cpio archive
+ files"
+ http://leaf.dragonflybsd.org/cgi/web-man?command=cpio§ion=5
+
+ - "A Quick Benchmark: Gzip vs. Bzip2 vs. LZMA", 31 May. 2005:
+ http://tukaani.org/lzma/benchmarks.html
+
+ - "Lempel Ziv Markov Algorithm and 7-Zip", 7 Feb. 2008:
+ http://blogs.sun.com/clayb/entry/lempel_ziv_markov_algorithm_and
+
+ - "The Open Group Base Specifications Issue 6: pax Interchange
+ Format, IEEE Std 1003.1, 2004 Edition"
+ http://www.opengroup.org/onlinepubs/009695399/utilities/
+ pax.html#tag_04_100_13_01
+
+ - ".ZIP File Format Specification", 28 Sep. 2007:
+ http://www.pkware.com/documents/casestudies/APPNOTE.TXT
+
+ - "ZIP (file format)", 17 Apr. 2010:
+ http://en.wikipedia.org/wiki/ZIP_%28file_format%29
diff --git a/doc/pkg5_docs/one-pager-main.txt b/doc/pkg5_docs/one-pager-main.txt
new file mode 100644
index 0000000..4da1a6d
--- /dev/null
+++ b/doc/pkg5_docs/one-pager-main.txt
@@ -0,0 +1,427 @@
+Template Version: @(#)onepager.txt 1.31 07/08/08 SMI
+
+This information is Copyright 2008 Sun Microsystems
+
+1. Introduction
+ 1.1. Project/Component Working Name:
+
+ pkg(5): image packaging system
+
+ 1.2. Name of Document Author/Supplier:
+
+ Stephen Hahn, Sun Microsystems,
+ on behalf of the pkg(5) project team
+
+ 1.3. Date of This Document:
+
+ 03/10/2008
+
+ 1.4. Name of Major Document Customer(s)/Consumer(s):
+ 1.4.1. The Community you expect to review your project:
+
+ Install and Packaging CG
+
+ 1.4.2. The ARC(s) you expect to review your project:
+
+ PSARC
+
+ 1.5. Email Aliases:
+ 1.5.2. Responsible Engineer:
+
+ stephen.hahn@sun.com
+
+ 1.5.4. Interest List:
+
+ pkg-discuss@opensolaris.org
+
+2. Project Summary
+ 2.1. Project Description:
+
+ The image packaging system, pkg(5), is a portable software
+ packaging and delivery system intended to allow efficient,
+ observable, and controllable transitions between known
+ configurations of software content. pkg(5) will subsume the
+ functionality of the of the packaging and patching utilities
+ included in historical Solaris releases. A primary goal for
+ this project is to improve and extend the usability and
+ functionality of our packaging system.
+
+ The project includes a set of recommended changes to the
+ existing software groupings--the package definitions--in an
+ attempt to produce a more rational and flexible organization of
+ the current components.
+
+ 2.2. Risks and Assumptions:
+
+ We intend to preserve the legacy packaging system functionality
+ to support compatibility of existing packages. We believe that,
+ if migration and compatibility practices are made available, the
+ provision of a new packaging mechanism will be followed by
+ adoption.
+
+ We strongly believe that the refactoring and renaming of the
+ existing package graph is not achievable with reasonable cost
+ and duration with the existing packaging/patching/installation
+ software. We also believe compatibility with the existing graph
+ can be preserved, to support the earlier assumption about
+ preserving legacy package operations.
+
+ We also believe that, for the majority of the operating system's
+ development and deployment needs, binary software delivery is
+ preferred over source-based build delivery.
+
+3. Business Summary
+ 3.1. Problem Area:
+
+ Deficits in the current packaging, patching, and installation
+ tool set affect potentially all parties interacting with the
+ historical Solaris releases and successors. Such deficits
+ include:
+
+ - lack of support for dependency-based retrieval during package
+ installation, from one or more network repositories,
+
+ - coarse and incorrect dependencies, limiting use for
+ construction of appliances or other specific-purpose systems,
+
+ - lack of versioning and control over change,
+
+ - forced interactivity,
+
+ - integration with virtualized systems, particularly patching
+ performance,
+
+ - reliance of the installer on hidden information, limiting
+ participation in system upgrade scenarios,
+
+ - lack of safety, specifically around package completeness and
+ alternate package contexts,
+
+ - high developer costs around package and patch creation and
+ maintenance,
+
+ - lack of support for unprivileged package and patch
+ installation,
+
+ - lack of awareness of ZFS and smf(5),
+
+ - late or no correctness checking, and
+
+ - minimal ease of use.
+
+ Additionally, the absence of a portable and efficient
+ cross-platform software delivery system places additional costs
+ upon teams that must deliver software for multiple platforms,
+ such as enterprise middleware vendors.
+
+ 3.2. Market/Requester:
+
+ Distribution providers and software content providers have
+ requested substantial changes to the legacy packaging system.
+ The requested changes focus on reducing maintenance costs and
+ increasing development efficiencies.
+
+ Various customers of the historical Solaris release have asked
+ for substantial capabilities not present in the legacy packaging
+ system.
+
+ Finally, multiplatform packaging capabilities are of interest to
+ a number of software content providers.
+
+ 3.3. Business Justification:
+
+ See 3.1 and 3.2.
+
+ 3.4. Competitive Analysis:
+
+ Every major operating system vendor--and most upcoming new
+ vendors--offers a form of networked software delivery and
+ updates. Well known companies with such technologies are
+ Microsoft, Red Hat, Apple, and Canonical; new companies include
+ rPath. Non-profit entities with equivalent technology include
+ the Debian Project.
+
+ 3.5. Opportunity Window/Exposure:
+
+ In order for OpenSolaris-based systems to remain competitive in
+ software delivery functionality, Solaris 10 should be the last
+ Minor release with a packaging system that fails to meet the
+ needs stated in 3.1 and 3.2.
+
+ 3.6. How will you know when you are done?:
+
+ In terms of basic capabilities, we can examine each component.
+ Project completion on the retrieval side can be measured by
+ achieving the capability of managing mixed content from a
+ variety of publishers, with potentially distinct entitlement
+ regimes. On the publication side, completion of the initial
+ project is reached once the goals around dependency and
+ correctness checking (and failure handling) are met for both the
+ server and the publication client. Finally, ease of use (or
+ familiarity) must match or exceed that of other leading
+ packaging systems.
+
+ In terms of the product as a whole, we must be able to upgrade,
+ with some statement about limitations on fidelity, a system
+ installed using the legacy packaging components such that it can
+ be further updated using the image packaging system.
+
+4. Technical Description:
+ 4.1. Details:
+
+ pkg(5) is a network-oriented binary packaging system. Although
+ it will have on-disk representations for versioned packages, the
+ primary expected use for installation of software will be
+ between an intelligent client and one or more relatively simple
+ servers.
+
+ The project defines a client-server publication mechanism. The
+ publication client offers up transactions on packages. The
+ server evaluates transactions from the publication client.
+ Transations that are deemed to be complete and/or safe by the
+ server are then made available to the retrieval client.
+
+ The initial transport will be HTTP and HTTPS, protocols around
+ which most sites have developed mature access policies. Support
+ for most common HTTP/HTTPS load-balancing, redirection, and
+ proxying techniques will be implemented, making the system easy
+ to deploy in a variety of scenarios. Additional transports may
+ be investigated during the course of the project or as future
+ work.
+
+ The project does not define a default mechanism for building
+ software as part of the packaging process. The project team
+ believes strongly that software builds are a separate function,
+ and probably also agrees that different kinds of software may
+ require different build techniques.
+
+ More controversially, the project, in an attempt to increase
+ system safety and to reduce developer burden, removes the notion
+ of arbitrary context scripting from packaging. (This removal
+ means that the legacy packaging system must remain on the system
+ for long-term compatibility.) Empirical evidence from the
+ prototype phase has so far borne out this decision.
+
+ 4.2. Bug/RFE Number(s):
+
+ As an example of the kinds of defects and RFEs intended to be
+ resolved by this project, we present the following selection of
+ bug IDs from the past 15 years:
+
+1105830 pkgadd and pkgrm should be able to handle dependency ordering
+1149607 Package dependencies hidden within a cluster.
+1165888 RFE: allow non-root users to install software using the package mechanis
+1184238 patches should be fully managed by package utilites
+1208431 pkgrm with no arguments defaults to all
+1249015 pkgadd requires root access
+4202113 pkginfo command is ridiculously slow
+4240078 pkgadd should not allow an intel package to install on Sparc and visa-ve
+4385316 RFE Support pkgadd of clusters
+4480153 Improvements desired for pkg management
+4762470 pkgadd: soft dependencies
+4795539 pkgadd should check dependencies of all packages provided on the command
+4847723 rem_drv in preremove scripts should have consistent usage model
+4939605 grep-friendly pkgchk -l variant desired
+5012345 request for tool to list package dependencies
+6208580 pkgadd/pkgrm should be smarter about dependancies
+6246595 Sun's package management needs improvement
+6491381 Create audit log for packaging and patch commads
+
+ In contrast,
+
+1181241 wants to split large binary across multiple floppies with pkgmk
+
+ will not be addressed by this proposal.
+
+ 4.3. In Scope:
+
+ Package-service delivery and containment relationships.
+
+ Package installation behaviour in virtualized environments.
+
+ 4.4. Out of Scope:
+
+ Specific operational scenarios for repositories operated by Sun
+ Microsystems.
+
+ Provision of a GUI/BUI for package management.
+
+ Specific package contents and manifests.
+
+ 4.5. Interfaces:
+
+ pkg(5) will present a substantial set of new and modified interfaces
+ to the core system. In particular, documented definitions of
+
+ - retrieval client CLI,
+
+ - publication client CLI,
+
+ - administrative and server CLIs,
+
+ - client metadata representations,
+
+ - server metadata representations,
+
+ - retrieval and publication protocol operations,
+
+ - a dynamic language API to access packaging functions,
+
+ - an on-disk package format,
+
+ - package metadata conventions,
+
+ - available package constituents ("actions"), and
+
+ - package naming and versioning conventions,
+
+ will be presented as interfaces introduced by this project.
+
+ It is possible that some of the nominally private interfaces
+ associated with legacy packaging will be affected; at a minimum,
+ files previously delivered via legacy packaging will no longer
+ be tracked by the legacy system. This outcome could result in a
+ correctly functioning system that presents very differently in
+ terms of file-package membership when interrogated using the
+ legacy packaging API.
+
+ Various components of the project will be introduced at each
+ stability and/or commitment level. The components are being
+ engineered such that the public interfaces can be evolved
+ compatibly, once the initial development is complete. (In fact,
+ the prototype is expected to support this evolution during the
+ development phase.)
+
+ The components are currently implemented in Python
+ (PSARC/2005/532, PSARC/2005/555, PSARC/2006/666).
+
+ 4.6. Doc Impact:
+
+ The project expects to provide reference manual pages for each
+ of the groups of interface identified above. Furthermore, the
+ project expects to provide a Developer's Guide to replace the
+ current Application Packager's Guide.
+
+ 4.7. Admin/Config Impact:
+
+ Substantial new capabilities in software installation will
+ become available.
+
+ A related project to produce a package manipulation GUI is being
+ pursued.
+
+ 4.8. HA Impact:
+
+ None known; dependent on specific impacts of legacy packaging on
+ these capabilities.
+
+ 4.9. I18N/L10N Impact:
+
+ Commands will require localization, as will any publically
+ committed library or equivalent APIs.
+
+ 4.10. Packaging & Delivery:
+
+ All package, cluster, and metacluster boundaries will be
+ examined in the course of the project.
+
+ The primary upgrade mechanism for operating systems using pkg(5)
+ will be achieved via pkg(5) components; this mechanism is
+ expected to replace the current standalone upgrade and
+ LiveUpgrade paths. The replacement is expected, in concert with
+ the SnapUpgrade project, to present capabilities that equal or
+ exceed those of LiveUpgrade.
+
+ 4.11. Security Impact:
+
+ In the current implementation, the protocol is built atop access
+ to HTTP and/or HTTPS. Accordingly, the server side will
+ potentially listen on ports associated with those services.
+
+ The server and client side will require access to key and
+ certificate management interfaces.
+
+ A mechanism for signing repository catalogs and package
+ manifests will be a part of the publication interface. A
+ corresponding mechanism for verifying signed catalogs and
+ manifests will be implemented for the installation client.
+ These mechanisms will apply to both packages in a network
+ package repository and packages in their on-disk representation.
+
+ 4.12. Dependencies:
+
+ The project is dependent on SnapUpgrade for coherent collection,
+ organization, and activation of filesystem snapshots.
+
+5. Reference Documents:
+ Project site:
+
+ http://opensolaris.org/os/project/pkg/
+
+ Project team members have written a number of informal essays on
+ various goals--problems to solve, outcomes to avoid, hopes to
+ realize--on aspects of the project:
+
+ - General observations:
+
+ http://blogs.sun.com/sch/entry/observations_on_packaging
+
+ - On testability and complexity costs with the current patching
+ methods:
+
+ http://blogs.sun.com/barts/entry/rethinking_patching
+
+ - Eliminating scripting in a packaging system
+
+ http://blogs.sun.com/sch/entry/pkg_1_a_no_scripting
+
+ - Keep software builds separate from software delivery:
+
+ http://blogs.sun.com/sch/entry/pkg_leaving_the_build_system
+
+ - Keeping critical metadata back the packaging system, rather
+ than in the installer:
+
+ http://blogs.sun.com/sch/entry/pkg_no_more_installer_magic
+
+ Related efforts in the Caiman project:
+
+ - Snap Upgrade,
+ http://opensolaris.org/os/project/caiman/Snap_Upgrade/
+
+ - Distribution Constructor,
+ http://opensolaris.org/os/project/caiman/Constructor/
+
+6. Resources and Schedule:
+ 6.1. Projected Availability:
+
+ 2008
+
+ 6.2. Cost of Effort:
+
+ Unable to estimate at present time.
+
+ 6.4. Product Approval Committee requested information:
+ 6.4.1. Consolidation or Component Name:
+
+ ON
+
+ 6.5. ARC review type:
+
+ Standard.
+
+ 6.6. ARC Exposure: open
+ 6.6.1. Rationale: Part of OpenSolaris
+
+7. Prototype Availability:
+
+ 7.1. Prototype Availability:
+
+ Prototype exit criteria are: ability to support multiple transports,
+ some access control capability, constrained dependency support,
+ bulk of OpenSolaris-specific actions.
+
+ 7.2. Prototype Cost:
+
+ Unable to estimate.
+
diff --git a/doc/pkg5_docs/parallel-linked-images.txt b/doc/pkg5_docs/parallel-linked-images.txt
new file mode 100644
index 0000000..c17570e
--- /dev/null
+++ b/doc/pkg5_docs/parallel-linked-images.txt
@@ -0,0 +1,205 @@
+.. This document is formatted using reStructuredText, which is a Markup
+ Syntax and Parser Component of Docutils for Python. An html version
+ of this document can be generated using the following command:
+ rst2html.py doc/parallel-linked-images.txt >doc/parallel-linked-images.html
+
+======================
+Parallel Linked Images
+======================
+
+:Author: Edward Pilatowicz
+:Version: 0.1
+
+
+Problems
+========
+
+Currently linked image recursion is done serially and in stages. For
+example, when we perform an "pkg update" on an image then for each child
+image we will execute multiple pkg.1 cli operations. The multiple pkg.1
+invocations on a single child image correspond with the following
+sequential stages of pkg.1 execution:
+
+1) publisher check: sanity check child publisher configuration against
+ parent publisher configuration.
+2) planning: plan fmri and action changes.
+3) preparation: download content needed to execute planned changes.
+4) execution: execute planned changes.
+
+So to update an image with children, we invoke pkg.1 four times for each
+child image. This architecture is inefficient for multiple reasons:
+
+- we don't do any operations on child images in parallel
+
+- when executing multiple pkg.1 invocations to perform a single
+ operation on a child image, we are constantly throwing out and
+ re-initializing lots of pkg.1 state.
+
+To make matters worse, when as we execute stages 3 and 4 on a child
+image the pkg client also re-executes previous stages. For example,
+when we start stage 4 (execution) we re-execute stages 2 and 3. So for
+each child we update we end up invoking stage 2 three times, and stage 3
+twice. This leads to bugs like 18393 (where it seems that we download
+packages twice). It also means that we have caching code buried within
+the packaging system that attempts to cache internal state to disk in an
+effort to speed up subsequent re-runs of previous stages.
+
+
+Solutions
+=========
+
+
+Eliminate duplicate work
+------------------------
+
+We want to eliminate a lot of the duplicate work done when executing
+packaging operations on children in stages. To do this we will update
+the pkg client api to allow callers to:
+
+- Save an image plan to disk.
+- Load an image plan from disk.
+- Execute a loaded plan from disk without first "preparing" it. (This
+ assumes that the caller has already "prepared" the plan in a previous
+ invocation.)
+
+In addition to eliminating duplicated work during staged execution, this
+will also allow us to stop caching intermediate state internally within
+the package system. Instead client.py will be enhanced to cache the
+image plan and it will be the only component that knows about "staging".
+
+To allow us to save and restore plans, all image plan data will be saved
+within a PlanDescription object, and we will support serializing this
+object into a json format. The json format for saved image plans is an
+internal, unstable, and unversioned private interface. We will not
+support saving an image plan to disk and then executing it later with a
+different version of the packaging system on a different host. Also,
+even though we will be adding data into the PlanDescription object we
+will also not be exposing any new information about an image plan to via
+the PlanDescription object to api consumers.
+
+An added advantage of allowing api consumers to save an image plan to
+disk is that it should help with our plans to have the api.gen_plan_*()
+functions to be able to return PlanDescription object for child images.
+A file descriptor (or path) associated with a saved image plan would be
+one way for child images to pass image plans back to their parent (which
+could then load them and yield them as results to api.gen_plan_*()).
+
+
+Update children in parallel
+---------------------------
+
+We want to enhance the package client so that it can update child images
+in parallel.
+
+Due to potential resource constraints (cpu, memory, and disk io) we
+cannot entirely remove the ability to operate on child images serially.
+Instead, we plan to allow for a concurrency setting that specifies how
+many child images we are willing to update in parallel. By default when
+operating on child images we will use a concurrency setting of 1, this
+maintains the current behavior of the packaging system. If a user wants
+to specify a higher concurrency setting, they can use the "-C N" option
+to subcommands that recurse (like "install", "update", etc) or they can
+set the environment variable "PKG_CONCURRENCY=N". (In both cases N is
+an integer which specifies the desired concurrency level.)
+
+Currently, pkg.1 worker subprocesses are invoked via the pkg.1 cli
+interfaces. When switching to parallel execution this will be changed
+to use a json encoded rpc execution model. This richer interface is
+needed to allow worker processes to pause and resume execution between
+stages so that we can do multi-staged operations in a single process.
+
+Unfortunately, the current implementation does not yet retain child
+processes across different stages of execution. Instead, whenever we
+start a new stage of execution, we spawn one process for each child
+images, then we make a remote procedure call into N images at once
+(where N is our concurrency level). When an RPC returns, that child
+process exits and we start a call for the next available child.
+
+Ultimately, we'd like to move to model where we have a pool of N worker
+processes, and those processes can operate on different images as
+necessary. These processes would be persistent across all stages of
+execution, and ideally, when moving from one stage to another these
+processes could cache in memory the state for at least N child images so
+that the processes could simply resume execution where they last left
+off.
+
+The client side of this rpc interface will live in a new module called
+PkgRemote. The linked image subsystem will use the PkgRemote module to
+initiate operations on child images. One PkgRemote instance will be
+allocated for each child that we are operating on. Currently, this
+PkgRemote module will only support the sync and update operations used
+within linked images, but in the future it could easily be expanded to
+support other remote pkg.1 operations so that we can support recursive
+linked image operations (see 7140357). When PkgRemote invokes an
+operation on a child image it will fork off a new pkg.1 worker process
+as follows:
+
+ pkg -R /path/to/linked/image remote --ctlfd=5
+
+this new pkg.1 worker process will function as an rpc server which the
+client will make requests to.
+
+Rpc communication between the client and server will be done via json
+encoded rpc. These requests will be sent between the client and server
+via a pipe. The communication pipe is created by the client, and its
+file descriptor is passed to the server via fork/exec. The server is
+told about the pipe file descriptor via the --ctlfd parameter. To avoid
+issues with blocking IO, all communication via this pipe will be done by
+passing file descriptors. For example, if the client wants to send a
+rpc request to the server, it will write that rpc request into a
+temporary file and then send the fd associated with the temporary file
+over the pipe. Any reply from the server will be similarly serialized
+and then sent via a file descriptor over the pipe. This should ensure
+that no matter the size of the request or the response, we will not
+block when sending or receiving requests via the pipe. (Currently, the
+limit of fds that can be queued in a pipe is around 700. Given that our
+rpc model includes matched requests and responses, it seems unlikely
+that we'd ever hit this limit.)
+
+In the pkg.1 worker server process, we will have a simple json rpc
+server that lives within client.py. This server will listen for
+requests from the client and invoke client.py subcommand interfaces
+(like update()). The client.py subcommand interfaces were chosen to be
+the target for remote interfaces for rpc calls for the following
+reasons:
+
+- Least amount of encoding / decoding. Since these interfaces are
+ invoked just after parsing user arguments, they mostly involve simple
+ arguments (strings, integers, etc) which have a direct json encoding.
+ Additionally, the return values from these calls are simple return
+ code integers, not objects, which means the results are also easy to
+ encode. This means that we don't need lots of extra serialization /
+ de-serialization logic (for things like api exceptions, etc).
+
+- Output and exception handling. The client.py interfaces already
+ handle exceptions and output for the client. This means that we don't
+ have to create new output classes and build our own output and
+ exception management handling code, instead we leverage the existing
+ code.
+
+- Future recursion support. Currently when recursing into child images
+ we only execute "sync" and "update" operations. Eventually we want to
+ support pkg.1 subcommand recursion into linked images (see 7140357)
+ for many more operations. If we do this, the client.py interfaces
+ provide a nice boundary since there will be an almost 1:1 mapping
+ between parent and child subcommand operations.
+
+
+Child process output and progress management
+--------------------------------------------
+
+Currently, since child execution happens serially, all child images have
+direct access to standard out and display their progress directly there.
+Once we start updating child images in parallel this will no longer be
+possible. Instead, all output from children will be logged to temporary
+files and displayed by the parent when a child completes a given stage
+of execution.
+
+Additionally, since child images will no longer have access to standard
+out, we will need a new mechanism to indicate progress while operating
+on child images. To do this we will have a progress pipe between each
+parent and child image. The child image will write one byte to this
+pipe whenever one of the ProgressTracker`*_progress() interfaces are
+invoked. The parent process can read from this pipe to detect progress
+within children and update its user visible progress tracker
+accordingly.
diff --git a/doc/pkg5_docs/pkg-guide-web.css b/doc/pkg5_docs/pkg-guide-web.css
new file mode 100644
index 0000000..3fc9e79
--- /dev/null
+++ b/doc/pkg5_docs/pkg-guide-web.css
@@ -0,0 +1,67 @@
+/*
+ * This specification may be enough for adequate printing.
+ */
+@page {
+ margin: 2.5cm;
+}
+
+body {
+ font-family: serif;
+ margin: 5%;
+}
+
+h1, h2, h3, h4, h5, h6 {
+ font-family: sans-serif;
+}
+
+a {
+ text-decoration: none;
+ color: #4444aa;
+}
+
+a:hover {
+ color: #222288;
+ background-color: #eeeeee;
+}
+
+.subtitle {
+ font-style: italic;
+}
+
+th .docinfo-name {
+ text-align: right;
+}
+
+div .admonition {
+ border-color: #aaaaaa;
+
+ border-right-style: none;
+ border-left-style: none;
+
+ border-top-style: solid;
+ border-top-width: thick;
+
+ border-bottom-style: solid;
+ border-bottom-width: thin;
+
+ margin-right: 20em;
+}
+
+div .sidebar {
+ border-color: #ffaaaa;
+ border-width: thin;
+ float: right;
+ width: 20em;
+ background-color: #eeeecc;
+}
+
+p .sidebar-title {
+ font-family: sans-serif;
+ text-size: 24px;
+
+}
+
+p .topic-title {
+ font-family: sans-serif;
+ text-size: 24px;
+}
diff --git a/doc/pkg5_docs/pkg-states.rst b/doc/pkg5_docs/pkg-states.rst
new file mode 100644
index 0000000..51ff23c
--- /dev/null
+++ b/doc/pkg5_docs/pkg-states.rst
@@ -0,0 +1,195 @@
+.. CDDL HEADER START
+
+.. The contents of this file are subject to the terms of the
+ Common Development and Distribution License (the "License").
+ You may not use this file except in compliance with the License.
+
+.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+ or http://www.opensolaris.org/os/licensing.
+ See the License for the specific language governing permissions
+ and limitations under the License.
+
+.. When distributing Covered Code, include this CDDL HEADER in each
+ file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+ If applicable, add the following below this CDDL HEADER, with the
+ fields enclosed by brackets "[]" replaced with your own identifying
+ information: Portions Copyright [yyyy] [name of copyright owner]
+
+.. CDDL HEADER END
+
+
+.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+
+
+.. :vim set expandtab:
+
+PACKAGE STATES
+--------------
+
+
+Server states
+~~~~~~~~~~~~~
+
+ We phrase the state machine in terms of a single removal state,
+ ABANDONED, which covers both the never-created package instance
+ (even with a series of never-finished transaction events). It may
+ be more appropriate to separate the ABANDONED state into
+ TX_ABANDONED and PKG_DELETED.
+
+ This leaves us with a state transition diagram like::
+
+ 0
+ |
+ |
+ v
+ +--> TRANSACTING --> ABANDONED <--+
+ | | ^ |
+ | | | |
+ | v | |
+ | SUBMITTED ----> INCOMPLETE |
+ | | | |
+ | | | |
+ +--- PUBLISHED <---------+ |
+ | |
+ | |
+ +------------------------+
+
+ 0 -> TRANSACTING
+ On initial package creation.
+
+ TRANSACTING -> ABANDONED
+ If initial package transaction never committed, commitment
+ failed, or explicitly dropped.
+
+ TRANSACTING -> SUBMITTED
+ On successful package transaction commitment. Packages with
+ syntax errors or immediate inconsistencies would have failed in
+ commitment.
+
+ SUBMITTED:
+ The package modified by the transaction is known by a specific
+ version. Its contents are in the repository.
+
+ SUBMITTED -> INCOMPLETE
+ If commitment included a deferred inconsistency (package
+ dependency is the only expected form), then the package is left
+ in the incomplete state.
+
+ INCOMPLETE:
+ The package with the specific version string is on the
+ incomplete list. Its contents are in the repository.
+
+ INCOMPLETE -> ABANDONED
+ If incomplete package explicitly removed. (Possibly by
+ timeout from arrival in INCOMPLETE.)
+
+ SUBMITTED -> PUBLISHED
+ If commitment had no deferred inconsistencies, then the package
+ is considered ready for publication.
+
+ INCOMPLETE -> PUBLISHED
+ If the deferred inconsistencies, upon reevaluation, are now held
+ to no longer be inconsistent, then the package is considered
+ ready for publication.
+
+ PUBLISHED:
+ The package with the specific version string is present in the
+ catalog. Its contents remain in the repository.
+
+ PUBLISHED -> ABANDONED
+ On manual request for package decommissioning, the package will
+ be moved to the abandoned state.
+
+ ABANDONED:
+ A package with a specific version string is no longer in the
+ catalog or on the incomplete list. Its contents, if they were
+ in the repository, should be removed from the repository.
+
+ XXX ARCHIVED might be a special state connected to PUBLISHED, or
+ merely a substate. An archived package has its manifest and
+ contents in the repository, but cannot be installed on a client.
+ The point of including ARCHIVED packages is to allow client
+ deduction on a system not installed by the pkg system.
+
+Client states
+~~~~~~~~~~~~~
+
+ Within an image, each package version can have only one state, as
+ given in the following diagram::
+
+ 0
+ |
+ v
+ IN_CATALOG ---------> OUT_CATALOG
+ | ^
+ +--->---+---<-------+ |
+ | | | |
+ | v | |
+ | INSTALLED --> FROZEN |
+ | | |
+ | | |
+ | v |
+ +-- PRESERVED |
+ | | |
+ | | |
+ | v |
+ +-- DELETED -------------------+
+
+
+ 0 -> IN_CATALOG:
+ A catalog update with new entries.
+
+ IN_CATALOG:
+ An entry for this package is available in the locally installed
+ catalog.
+
+ IN_CATALOG -> OUT_CATALOG:
+ Entry formerly present on local catalog is no longer published by any
+ repository. (Package never locally installed.)
+
+ OUT_CATALOG:
+ Although a formerly known package, no entry for this package is
+ available in the locally installed catalog. An INSTALLED or
+ FROZEN package can never be OUT_CATALOG, as the system will
+ preserve the entry until the package is no longer in a locally
+ public state.
+
+ IN_CATALOG -> INSTALLED:
+ Transition takes place on package installation.
+
+ INSTALLED -> FROZEN:
+ Transition takes place if manually frozen or frozen by virtue of
+ reference from another package group.
+
+ FROZEN -> INSTALLED:
+ Manually unfrozen, or unfrozen by reference drop due to
+ change in formerly referring package group.
+
+ INSTALLED -> PRESERVED:
+ Old copies moved aside during upgrade of package components, but
+ not removed.
+
+ PRESERVED -> DELETED:
+ Old copies removed.
+
+ DELETED -> OUT_CATALOG:
+ Package has been removed from client catalog. Client software
+ would take a PRESERVED package through DELETED automatically to
+ reach OUT_CATALOG.
+
+ PRESERVED -> INSTALLED:
+ Package reinstalled or reversed.
+
+ DELETED -> INSTALLED:
+ Package reinstalled.
+
+ XXX How does the ZFS snapshot (that we might have taken prior to an
+ operation) get represented in the state? Is there an image state
+ machine model as well?
+
+ XXX Need a substate of INSTALLED for damaged packages.
+
+ XXX Need a substate of INSTALLED for packages where the global zone
+ portion is available, but local installation has not finished. Can
+ we generalize this state for all diskless installs?
+
diff --git a/doc/pkg5_docs/pkg-states.txt b/doc/pkg5_docs/pkg-states.txt
new file mode 100644
index 0000000..90a1268
--- /dev/null
+++ b/doc/pkg5_docs/pkg-states.txt
@@ -0,0 +1,165 @@
+
+pkg
+PACKAGE STATES
+
+:vim set expandtab:
+
+1. Server states
+
+ We phrase the state machine in terms of a single removal state,
+ ABANDONED, which covers both the never-created package instance
+ (even with a series of never-finished transaction events). It may
+ be more appropriate to separate the ABANDONED state into
+ TX_ABANDONED and PKG_DELETED.
+
+ 0
+ |
+ |
+ v
+ +--> TRANSACTING --> ABANDONED <--+
+ | | ^ |
+ | | | |
+ | v | |
+ | SUBMITTED ----> INCOMPLETE |
+ | | | |
+ | | | |
+ +--- PUBLISHED <---------+ |
+ | |
+ | |
+ +------------------------+
+
+ 0 -> TRANSACTING
+ On initial package creation.
+
+ TRANSACTING -> ABANDONED
+ If initial package transaction never committed, commitment
+ failed, or explicitly dropped.
+
+ TRANSACTING -> SUBMITTED
+ On successful package transaction commitment. Packages with
+ syntax errors or immediate inconsistencies would have failed in
+ commitment.
+
+ SUBMITTED:
+ The package modified by the transaction is known by a specific
+ version. Its contents are in the repository.
+
+ SUBMITTED -> INCOMPLETE
+ If commitment included a deferred inconsistency (package
+ dependency is the only expected form), then the package is left
+ in the incomplete state.
+
+ INCOMPLETE:
+ The package with the specific version string is on the
+ incomplete list. Its contents are in the repository.
+
+ INCOMPLETE -> ABANDONED
+ If incomplete package explicitly removed. (Possibly by
+ timeout from arrival in INCOMPLETE.)
+
+ SUBMITTED -> PUBLISHED
+ If commitment had no deferred inconsistencies, then the package
+ is considered ready for publication.
+
+ INCOMPLETE -> PUBLISHED
+ If the deferred inconsistencies, upon reevaluation, are now held
+ to no longer be inconsistent, then the package is considered
+ ready for publication.
+
+ PUBLISHED:
+ The package with the specific version string is present in the
+ catalog. Its contents remain in the repository.
+
+ PUBLISHED -> ABANDONED
+ On manual request for package decommissioning, the package will
+ be moved to the abandoned state.
+
+ ABANDONED:
+ A package with a specific version string is no longer in the
+ catalog or on the incomplete list. Its contents, if they were
+ in the repository, should be removed from the repository.
+
+ XXX ARCHIVED might be a special state connected to PUBLISHED, or
+ merely a substate. An archived package has its manifest and
+ contents in the repository, but cannot be installed on a client.
+ The point of including ARCHIVED packages is to allow client
+ deduction on a system not installed by the pkg system.
+
+2. Client states
+
+ 0
+ |
+ v
+ IN_CATALOG ---------> OUT_CATALOG
+ | ^
+ +--->---+---<-------+ |
+ | | | |
+ | v | |
+ | INSTALLED --> FROZEN |
+ | | |
+ | | |
+ | v |
+ +-- PRESERVED |
+ | | |
+ | | |
+ | v |
+ +-- DELETED -------------------+
+
+
+ 0 -> IN_CATALOG:
+ A catalog update with new entries.
+
+ IN_CATALOG:
+ An entry for this package is available in the locally installed
+ catalog.
+
+ IN_CATALOG -> OUT_CATALOG:
+ Entry formerly present on local catalog is no longer published by any
+ repository. (Package never locally installed.)
+
+ OUT_CATALOG:
+ Although a formerly known package, no entry for this package is
+ available in the locally installed catalog. An INSTALLED or
+ FROZEN package can never be OUT_CATALOG, as the system will
+ preserve the entry until the package is no longer in a locally
+ public state.
+
+ IN_CATALOG -> INSTALLED:
+ Transition takes place on package installation.
+
+ INSTALLED -> FROZEN:
+ Transition takes place if manually frozen or frozen by virtue of
+ reference from another package group.
+
+ FROZEN -> INSTALLED:
+ Manually unfrozen, or unfrozen by reference drop due to
+ change in formerly referring package group.
+
+ INSTALLED -> PRESERVED:
+ Old copies moved aside during upgrade of package components, but
+ not removed.
+
+ PRESERVED -> DELETED:
+ Old copies removed.
+
+ DELETED -> OUT_CATALOG:
+ Package has been removed from client catalog. Client software
+ would take a PRESERVED package through DELETED automatically to
+ reach OUT_CATALOG.
+
+ PRESERVED -> INSTALLED:
+ Package reinstalled or reversed.
+
+ DELETED -> INSTALLED:
+ Package reinstalled.
+
+ XXX How does the ZFS snapshot (that we might have taken prior to an
+ operation) get represented in the state? Is there an image state
+ machine model as well?
+
+ XXX Need a substate of INSTALLED for damaged packages.
+
+ XXX Need a substate of INSTALLED for packages where the global zone
+ portion is available, but local installation has not finished. Can
+ we generalize this state for all diskless installs?
+
diff --git a/doc/pkg5_docs/pkgs-and-groups.txt b/doc/pkg5_docs/pkgs-and-groups.txt
new file mode 100644
index 0000000..f34f301
--- /dev/null
+++ b/doc/pkg5_docs/pkgs-and-groups.txt
@@ -0,0 +1,138 @@
+
+PSARC/2008/190
+pkg(5): image packaging system
+
+PACKAGES AND GROUPS
+
+1. Definitions
+
+ To be consistent with the system, following the introduction of the
+ fault management architecture, each package is named by an FMRI in
+ the "pkg:" scheme. That is, we have
+
+ pkg://publisher/pkg_name@version
+
+ The publisher is generally expected to be a forward or reverse
+ domain name identifying the publisher from which a package can be
+ retrieved. Publishers which cannot be determined to be a domain
+ name are legitimate, but optional functionality, like automatic
+ server discovery for a particular publisher, may fail to work.
+ In the examples that follow, we use "opensolaris.org" as a generic
+ publisher.
+
+ The pkg_name, like service names, can be split into a category,
+ subcategories, and a basename. This namespace might be populated
+ with "manifest" and other metadata endpoints, as well as the SHA-1
+ names of the package's included files. (Although the direct access
+ to properties of the svc FMRI scheme has been rarely used.)
+
+ A "group package" is a package that depends upon (minimum versions
+ of) other packages, as well as optionally delivering files or other
+ actions. An "incorporation" is a group package that places forward
+ constraints upon the versions of the packages it depends upon, which
+ restricts the interval of valid package versions to one the author
+ of the incorporation believes functional.
+
+2. Namespace
+
+2.1. Single namespace, separate publishers
+
+ The primary design point of the package namespace is to allow
+ multiple package producers to co-exist in a single namespace, so
+ that images can switch between equivalent components from different
+ producers.
+
+2.2. Domain-name-based escape
+
+ At any point in the category hierarchy, a safe namespace can be
+ created by using the forward or reverse domain name, either as a
+ subcategory or as a comma-led prefix to a subcategory or package
+ base name. (This scheme is similar to FMRI namespace escapes in
+ smf(5), although we are eliminating use of stock symbol prefixes.)
+
+ For instance, when example.com wishes to publish the "whiskers"
+ package without reference to a larger namespace convention it can
+ use any of the following examples
+
+ pkg://opensolaris.org/.../example.com/whiskers
+ pkg://opensolaris.org/.../com.example/whiskers
+
+ pkg://opensolaris.org/.../example.com,whiskers
+ pkg://opensolaris.org/.../com.example,whiskers
+
+ pkg://opensolaris.org/.../example.com,software/whiskers
+ pkg://opensolaris.org/.../com.example,software/whiskers
+
+ and so forth.
+
+2.2. Locally reserved namespace
+
+ The top-level "site" category is reserved for use by the entity that
+ administrates the server. Neither the organizations producing the
+ operating system nor those providing additional software components
+ may use the site category.
+
+ The top-level "vendor" category is reserved for use by organizations
+ providing additional. The leading subcategory must be a domain.
+ That is, if example.com wishes to publish the "whiskers" package in
+ the vendor category, it would use a package name like
+
+ pkg://opensolaris.org/vendor/example.com/whiskers
+
+2.3. Additional reserved namespace
+
+ The top-level "cluster", "feature", "group", "metacluster", and
+ "service" categories are all reserved for future use.
+
+ Inception note: some or all of these reservations may be eliminated
+ or reduced when the single namespace convention reaches its final
+ form.
+
+2.4. Single namespace conventions
+
+2.4.1. Discussion
+
+ Packaging systems and distributions appear to have explicit
+ categories, subcategories, and potentially larger groups; some
+ distributions have explicit fields for these values, others use
+ tagging or multi-valued fields, which allows them to classify
+ certain packages multiply. For the FMRI namespace case, the system
+ is similar to a packaging system with multiple, single-valued,
+ category fields.
+
+ There appear to be two standard approaches to categorizing packages:
+
+ 1. By what primary class of thing a package delivers.
+
+ 2. By what area of functionality a package's contents address.
+
+ In the first approach, we get strict top-level categories like
+ "application", "driver", "library", and "system" or "kernel", as
+ well as potentially overlapping categories like "utility" and
+ "tool". Use of the leading subcategory is limited, and generally
+ given to the subsystem involved. A relatively detailed worked
+ example of the X11 subsystem under this scheme is given in
+
+ http://mail.opensolaris.org/pipermail/pkg-discuss/2008-February/001838.html
+
+ In the second, we would also see categories like these, but leading
+ subcategory is much more likely to classify according to
+ functionality, so that we would see "application/mail",
+ "application/web", "application/compiler", and so forth. Most
+ network packaging systems appear to classify in this fashion.
+
+ An appealing variation of the second form is to rotate all of the
+ non-"application" packages under a "system" mega-category, such that
+ all of the leaf packages (with the possible exception of device
+ drivers) are exposed at the category level. Table 1 shows some
+ example transformations under this scheme.
+
+ FROM TO
+ application/web/firefox web/firefox
+ application/compiler/gcc4 compiler/gcc4
+ library/c system/library/c
+ kernel/generic system/kernel/generic
+
+ Table 1: Rotating non-application categories under system.
+
+
diff --git a/doc/pkg5_docs/protocol-versioning.txt b/doc/pkg5_docs/protocol-versioning.txt
new file mode 100644
index 0000000..6a085f4
--- /dev/null
+++ b/doc/pkg5_docs/protocol-versioning.txt
@@ -0,0 +1,22 @@
+For obvious compatibility reasons, the over-the-wire protocol needs to be
+versioned. Since we're using HTTP, it makes sense to simply version the
+URLs used.
+
+The scheme is pretty simple:
+
+ http://host:port/operation/version(/extra)?
+
+where "version" is a simple integer.
+
+Open questions:
+
+ - Are there operations where we don't want to require a version?
+
+ - Is there a use for a "300 Multiple Choices" HTTP return code?
+
+ - If we always try the client-optimal version first and then back off
+ linearly, we'll want to cache the version that succeeded so we don't
+ spend all of our time trying versions that don't exist.
+
+ - Alternatively, we can call "versions" to find out what versions the
+ server supports for each operation, and cache that.
diff --git a/doc/pkg5_docs/publication.txt b/doc/pkg5_docs/publication.txt
new file mode 100644
index 0000000..4b8799e
--- /dev/null
+++ b/doc/pkg5_docs/publication.txt
@@ -0,0 +1,33 @@
+
+pkg
+PUBLICATION AND TRANSACTIONS
+
+:vim set expandtab:
+
+1. Problem
+
+ We prefer having "fat" packages, to having a plurality of speciated
+ nodes in our version tree. That is, we hope to have exception
+ packages have a single platform attribute. This means that we have
+
+ build files on platform A and B
+ commit fat package containing (A, B) files
+
+ as the gross ordering of operations. Taken strictly, this ordering
+ would be too restrictive, particularly once the set of platforms
+ which might participate in a package goes beyond two.
+
+2. Ways to handle multi-platform transactions
+
+ Treat every transaction on v as an update to the version, v. A
+ platform-specific update would include both common files and the
+ platform-tagged or architecture-tagged files. If common files
+ differed between v:t_1 and v:t_2, then this transaction isn't
+ valid--it's branch modifying.
+
+ Otherwise, the v:t_2 package contains the common files and the set
+ of flavoured files submitted so far.
+
+ (This approach evades the problem of expecting a platform-specific
+ update to only replace files with alternate versions.)
+
diff --git a/doc/pkg5_docs/publisher_search_order.txt b/doc/pkg5_docs/publisher_search_order.txt
new file mode 100644
index 0000000..77d5f50
--- /dev/null
+++ b/doc/pkg5_docs/publisher_search_order.txt
@@ -0,0 +1,98 @@
+
+In order to select packages from a variety of sources, pkg(5) supports
+the configuration of multiple publishers. We've started with the
+concept of a preferred publisher, but some ambiguity has arisen as to
+what should happen when publishers are deleted, added, etc. In
+addition, the single level of selection afforded by the preferred
+designation appears to be somewhat limiting; at the same time, the
+rules for selecting amongst multiple publishers of the same package
+need clarification rather than additional complexity.
+
+We propose the idea of publisher search order, starting with the
+preferred publisher. Adding a publisher adds a new publisher at the
+end of the search order; if a publisher is made preferred it is moved
+to the front of the search order. Deleting or disabling a publisher
+causes it to be removed from the search order. Re-enabling a publisher
+causes it to added at the end of the search order.
+
+When a package is nominated for installation without an explicit
+publisher being specified, the first publisher found in the search
+order to provide the package is selected. Once installed, subsequent
+updates to that package by default always occur from that publisher,
+even if a publisher earlier in the search order starts publishing a
+package with the same name. This behavior of a publisher is
+characterized as "sticky", and is the default. It can be disabled on
+a per-publisher basis, and such disablement is useful mostly for
+developers seeking to replace a portion of their packages w/
+development versions. If a publisher is made "non-sticky", its
+packages are searched for as on initial installation on every update -
+no preference is afforded by the previous installation. Deleted/disabled
+publishers are made non-sticky.
+
+Each selection of the publisher for a package is made independently
+according to the algorithms above; there is no implicit inheritance
+of publisher across dependencies of any type.
+
+The above suggests the following additions to the set-publisher
+subcommand of pkg:
+
+set-publisher [--search-before=publisher] [--search-after=publisher] publisher
+set-publisher [--sticky] [--non-sticky] publisher
+
+
+--search-before=publisherB publisherA causes publisher A to be moved
+from its current place in the search order to be just ahead of publisher B.
+
+--search-after=publisherB publisherA causes publisher A to be moved
+from its current place in the search order to be just behind publisher B.
+
+Specifying --non-sticky causes this publisher not to automatically
+be selected for all updates to packages already installed from this
+publisher; instead, publishers searched earlier are automatically preferred.
+--sticky causes the original behavior to be restored for subsequent
+updates.
+
+
+Use cases
+---------
+
+Normal user getting packages from pkg.opensolaris.org and datahavens.org:
+
+ 1) Installs system as per usual, points preferred to usual
+ best available publisher - say, pkg.opensolaris.org
+ 2) Adds new publisher datahavens.org to acquire mplayer.
+ Without specifying search order, new publisher is appended
+ to the current order.
+
+Project Developer:
+
+ 1) Installs system as per usual, points preferred to usual
+ best available publisher - say, pkg.opensolaris.org
+ 2) Adds new publisher "MyOwnRepo" pointing at his
+ latest and greatest bits. Note that his repo is lowest
+ in search order, but since his package names are unique no issues
+ arise.
+
+Contrib User that prefers supported bits:
+
+ 1) Installs system as per usual, points preferred to usual
+ best available publisher - say, pkg.opensolaris.org
+ 2) Adds contrib repo after p.o.o, and makes it non-sticky.
+ 3) Adds packages from contrib as desired.
+ 4) When and if packages move to p.o.o, they're automatically
+ updated from p.o.o on the next image update or
+ pkg install ....
+
+OpenSolaris developer:
+
+ 1) Installs system as per usual, points preferred to usual
+ best available publisher - say, pkg.opensolaris.org/dev
+ 2) Adds new preferred publisher "MyOwnRepo" pointing at his
+ latest and greatest bits; making it preferred places it
+ first in the search order.
+ 3) Pkg.opensolaris.org is made non-sticky to cause pkgs from MyOwnRepo
+ to replace those from pkg.opensolaris.org on next update
+ 4) If additional fresh bits are required, additional development
+ repos can be added and placed ahead of pkg.opensolaris.org
+ in the search order; in this way multiple consolidations
+ can be kept at the bleeding edge.
diff --git a/doc/pkg5_docs/razor.txt b/doc/pkg5_docs/razor.txt
new file mode 100644
index 0000000..16a7b21
--- /dev/null
+++ b/doc/pkg5_docs/razor.txt
@@ -0,0 +1,92 @@
+
+Requirements gathering for new packaging system
+-----------------------------------------------
+
+Some of these requirements will be satisfied in part by the use of ZFS
+as a root filesystem.
+
+In no particular order:
+
+A new packaging system must:
+
+* replace existing patching/upgrade/live upgrade/Jumpstart/Jet/SUC/...
+ functionality; there should be one way of managing all software change on
+ Solaris 11.
+
+* allow fine grain control of installation contents to support minimization
+ A customer should be able to select the desired functionality, and have the
+ system bring in the closure of the dependency graph.
+
+* support the installation of packages to a directory for diskless and Xen
+ configs.
+
+* be repository based to faciltate efficient software distribution.
+
+* support the user's connection to multiple repositories to provide
+ different types of software, newer software, different vendors/suppliers,
+ etc.
+
+* deal with zones:
+ * maintain global zone, whole root zones in sync
+ * cope w/ directory level split between global and local zones, or
+ eliminate them.
+
+* allow a package to be installed in alternative (non default) locations.
+
+* allow the installation of multiple instances/revisions of the same package
+ in different locations.
+
+* manage package dependencies in multiple ways:
+ * allow a set of packages to be managed as a group; all packages
+ must transition together. Groups may include other groups.
+ * allow specification of a minimum version level
+ * dependency graphs need not be acyclic
+
+* permit the selection of alternative software streams available from a single
+ repository.
+
+* permit the "tagging" of packages with interesting information such as
+ external packaging version number, features provided (at least partially)
+ by this packages, etc.
+
+* permit the creation of alternative package branches to represent either early
+ platform introduction or customer-specific fixes that are later merged into
+ the mainline.
+
+* manage updates to client system in a transactional fashion; either we run the
+ old bits or the new bits, never some of each.
+
+* support secure upgrading through firewalls, etc, w/o special handling,
+ ports opened, etc, on the client side. It must be possible to both allow
+ and disallow anonymous access to the repository, and offer fine grain access
+ controls.
+
+* be robust in the face of filesystem damage on the client side. It must be
+ possible to identify where the system doesn't match the packaging information,
+ and to be able to repair any damage by restoring damaged components from the
+ repository.
+
+* be open source, and included in OpenSolaris. All the tools necessary to build
+ and distribute OpenSolaris via a repository should be part of OpenSolaris.
+
+* support interactive development. A developer should be readily able to
+ create a new repository, make changes, build, commit the changes to the
+ repository and update his machine with those changes and reboot.
+
+* be of acceptable performance. Upgrade operations should not reacquire files
+ already in place on the system. As much as possible, packaging operations
+ should be order (1) rather then order(number of zones). Packaging operations
+ should not be significantly affected by the number of packages already
+ installed on the system
+
+A new packaging system must not:
+
+* require changing the build system of consolidations contributing to
+ software respository. Different parts of OpenSolaris build in many different
+ ways; forcing them all to be the same is unacceptable.
+
+* require the use of non-local resources for clients; security conscious
+ companies must be able to run their own repositories completely disconnected
+ from any external networks.
+
+
diff --git a/doc/pkg5_docs/repository.txt b/doc/pkg5_docs/repository.txt
new file mode 100644
index 0000000..d4160fb
--- /dev/null
+++ b/doc/pkg5_docs/repository.txt
@@ -0,0 +1,105 @@
+pkg
+REPOSITORIES
+
+1. Summary
+
+2. Discussion
+
+2.1. Repository configuration
+
+2.2.1. Configuration inheritance
+
+ The pkg.depotd(5) server may need to access repository configuration data when
+ smf(5) is unavailable. As such, a repository must have a cached
+ configuration state.
+
+ Roughly, these aspects are sufficient to constrain our configuration
+ behavior:
+
+ request properties from svc://application/pkg
+ if unavailable, examine configuration cache
+ if undefined or unavailable, use hard-coded defaults
+
+2.2.2. Configuration components
+
+ Repository attributes. The repository has a collection of simple
+ attributes for providing various bits of metadata and configuration
+ information. Note that changing these values requires a restart
+ of any pkg.depotd processes referencing the repository so that
+ changes will be reflected in operations and output.
+
+ repository/ Property group of type "application"
+ /name A short, descriptive name for the repository.
+
+ Examples: "opensolaris.org base repository"
+ "opensolaris.org contrib repository"
+
+ /description A descriptive paragraph for the repository.
+
+ /maintainer A human readable string describing the entity
+ maintaining the repository. For an individual,
+ this string is expected to be their name, or
+ name and email.
+
+ Examples: "Project Indiana"
+ "Project Indiana "
+
+ /maintainer_url A URL associated with the entity maintaining the
+ repository.
+
+ Example:
+ "http://www.opensolaris.org/os/project/indiana/"
+
+ /detailed_url One or more URLs to pages or sites with further
+ information about the repository.
+
+ Example: "http://www.opensolaris.org/"
+
+ feed/ Property group of type "application"
+ /id A Universally Unique Identifier (UUID) used to
+ permanently, uniquely identify the feed.
+ Changing this value can have unexpected effects
+ on feed consumers. In addition, when serving
+ multiple copies of a repository, each copy's
+ cfg_cache must have the same value for this
+ attribute.
+
+ /name A short, descriptive name for RSS/Atom feeds
+ generated by the depot serving the repository.
+
+ Example: "opensolaris.org packaging feed"
+
+ /description A descriptive paragraph for the feed.
+
+ /authority A fully-qualified domain name or email address
+ that will be used to generate a unique
+ identifier for each entry in the feed. Changing
+ this value can have unexpected effects on feed
+ consumers.
+
+ Examples: "opensolaris.org"
+ "indiana-discuss@opensolaris.org"
+
+ /icon A filename of a small image that will be used to
+ visually represent the feed (e.g. web
+ links to the repository, or the icon shown in a
+ user agent's address bar). This file must be
+ located in the depot server's inst_root
+ directory.
+
+ /logo A filename of a large image that will be used by
+ user agents to visually brand or identify the
+ feed. This file must be located in the depot's
+ inst_root directory.
+
+ /window A numeric value representing the number of
+ hours, before the feed for the repository was
+ last generated, to include when creating the feed
+ for the repository updatelog. The default value
+ is "24".
+
+ Example: "48"
+
+3. References
+
diff --git a/doc/pkg5_docs/rest.txt b/doc/pkg5_docs/rest.txt
new file mode 100644
index 0000000..c3c3596
--- /dev/null
+++ b/doc/pkg5_docs/rest.txt
@@ -0,0 +1,38 @@
+
+pkg
+Use of REST
+
+We use REST over HTTP as the primary networking API between client and
+server. That choice means that
+
+- the client can be reimplemented,
+
+- the server can be reimplemented, and
+
+- decorations on a transaction, like authentication, encryption, and
+ redirection, can be handled using the enormous technology set around
+ HTTP.
+
+The first two are true of any well-defined protocol, but we benefit in
+this case from the wide availability of HTTP client and server
+implementation starting points.
+
+Installer API:
+
+GET /catalog
+GET /version/pkg_name
+GET /depend/pkg_name
+GET /data/pkg_name[/from[/to]]
+
+Packager API:
+
+POST /trans/pkg_name
+ Returning a transaction ID
+POST /add/trans_id/type
+ Plus metadata in submitted headers and contents as file body.
+GET /summary/trans_id
+POST /meta/trans_id/require
+ Dependency metadata in submitted headers.
+POST /commit/trans_id
+ Returning package URL.
+
diff --git a/doc/pkg5_docs/rfes.txt b/doc/pkg5_docs/rfes.txt
new file mode 100644
index 0000000..99711cf
--- /dev/null
+++ b/doc/pkg5_docs/rfes.txt
@@ -0,0 +1,45 @@
+
+1. [sch] Put size in manifest so that we can do GET with offset to speed
+ retrieval. Example: reget on Windows.
+
+2. [billm] ACLs and extended attributes. Are these both new action types?
+
+3. [sch] Do we need other forms of signing, beyond publisher and depot SSL
+ certificates? Is it best to enforce a CA-based model, or should web
+ of trust also be allowed?
+
+4. [dp] Package deletion should include a "subsumed-by" or "replaced-by". A
+ good example is when we stopped including Mozilla, its final
+ version--expressing the deletion--should have stated subsumption by
+ Firefox and Thunderbird.
+
+5. [barts] Minimization boundaries based on setuid binaries.
+
+6. [barts] Feature tagging. This should be offered via leaf packages
+ and grouping packages.
+
+7. [dp] Present timestamp as UTC YYYYMMDDHHMMSS, as opposed to Unix
+ seconds.
+
+8. [lianep] Be able to preserve specific files, even though the
+ package no longer provides them. This one's tricky: if you state,
+ "release file" in version 2, then an upgrade from v 1 to v 3 would
+ miss it (as v 2's manifest is not consulted). These kind of actions
+ could be treated as choking, or we could always examine intermediate
+ manifests.
+
+9. [psa] Take a snapshot [of each affected filesystem] between every
+ package update operation in a larger image transaction, as opposed
+ to at the image transaction boundaries only.
+
+10. [sch] Examine use of alternative, HTTP 1.1-friendly URL loading
+ modules. (Example: Duke's urlgrabber.)
+
+11. [pelegri] Use of package-level metadata to provide additional
+ information, such as links to training/learning resources,
+ declarations of related packages, endorsements by certifying
+ publishers.
+
+12. [sch] Support use of :timestamp field for "newer" and "older"
+ queries.
+
diff --git a/doc/pkg5_docs/sat.txt b/doc/pkg5_docs/sat.txt
new file mode 100644
index 0000000..c76209b
--- /dev/null
+++ b/doc/pkg5_docs/sat.txt
@@ -0,0 +1,238 @@
+
+It's become clear that our approach to evaluating
+packaging dependencies and constraints needs to become
+a lot more sophisticated as we've been trying to make
+packaging metadata more accurately reflect the way
+we build and test packages.
+
+A significant part of the difficulty is dealing with
+externally produced packages; if a variety of versions
+are available we may need to iteratively test multiple
+versions, evaluating their dependencies to find one that
+is compatible w/ the constraints that may be active on
+the current image.
+
+One method of doing this sort of automated decision making
+is to cast the problem as a series of boolean expressions,
+and then apply a SAT solver to find a solution. These notes
+describe the results of my experiments w/ the minisat solver
+Stephen posted some time ago....
+
+Notes:
+--------
+
+1) The presence of a particular package version is a
+ single boolean variable; True if it's present,
+ False if not.
+
+ The problem set handed to the SAT solver is a series
+ of clauses; each clause are boolean variables (or
+ their negation) or'd together. All clauses must be
+ true for the solution to exist.
+
+ The clauses need to encode the fact that only one version
+ of a package may be installed at a time, and also encode
+ all different package dependencies and constraints.
+
+2) Each package has some number of versions, inherently ordered.
+ Only one version of a package may be installed at a time
+
+ pkg a -> a.1, a.2, a.3, a.4
+ pkg b -> b.1, b.2, b.3, b.4
+
+ Thus for "a":
+
+ !a.1 | !a.2
+ !a.1 | !a.3
+ !a.1 | !a.4
+ !a.2 | !a.3
+ !a.2 | !a.4
+ !a.3 | !a.4
+
+ where !a represents the negation of a.
+
+ This means that for N versions, we have N(N-1)/2 clauses;
+ pruning older non-accessible versions will be required to
+ bound memory consumption.
+
+3) Each version of a package may have dependencies on other
+ packages, either w/ or w/o a version. The version specification
+ will likely not be fully specified (eg multiple versions
+ may satisfy this requirement).
+
+4) dependencies may be of the following types:
+
+ required: fmri specifies minimum acceptable version
+
+ if a.1 requires b.2, b.3 or b.4:
+ !a.1 | b.2 | b.3 | b.4
+
+ optional: if present, fmri must be at this level or greater
+ if a.1 optionally requires b.3:
+ !a.1 | !b.1
+ !a.1 | !b.2
+
+ incorporate: if present, pkg must match fmri
+
+ if a.1 incorporates b.3:
+ !a.1 | !b.1
+ !a.1 | !b.2
+ !a.1 | !b.4
+
+ exclude: if present, pkg must be less that version in fmri:
+
+ if a.1 excludes b.3,
+
+ !a.1 | !b.3
+ !a.1 | !b.4
+
+ All of these are linear in the number of package versions
+ either meeting or failing to meet the dependency.
+
+5) To express state, the presence of a package is encoded as a
+ clause. We compute the matching fmris and then construct
+ a clause that matches one of those fmris. Specifying a single
+ version requires that version to be present in the solution;
+ we can also specify current version or newer, or any version of
+ a package.
+
+6) The SAT solver will find a particular solution to our packaging
+ problem, but there is no way of "preferring" newer packages, and
+ preventing the introduction of extraneous unneeded packages.
+ As a result, external optimization in the form of repeated
+ solution attempts w/ additional constraints is necessary.
+ The following algorithm has been implemented:
+
+ The packaging problem to be solved is expressed as a series of
+ boolean constraints, and a solution obtained. Then, for each
+ fmri appearing in the solution vector, all older versions are
+ excluded; in other words, if a.3 is part of the solution, then
+ subsequent solutions will not contain a.1 or a.2. Then a single
+ vector is added that is the negation of the just found vector,
+ and another solution is found. For example:
+
+ if solution is a.2, b.3, z.10, we add
+
+ # first negations of older versions
+ !a.1
+ !b.1
+ !b.2
+ !z.1
+ !z.2
+ ...
+ !z.9
+ # now negate just found solution
+ !a.2 | !b.3 | !z.10
+
+ The latter vector requires that the new solution not contain
+ a.2 and b.3 and z.10; since we've excluded older versions we
+ will either get a vector that eliminates one of the packages
+ as unneeded (if dependencies allow) or one that has newer
+ versions of one of the needed pkgs.
+
+ We repeat the above process until a solution cannot be found;
+ the last found solution must therefore be the most optimal one.
+
+ The above technique may fail to find the overall optimal
+ solution if newer packages have incorporation dependencies
+ on earlier versions of their dependencies. This is expected
+ to be rare. Pruning the solution space to eliminate older
+ packages is necessary due to rapid solution space growth if
+ there are multiple versions that satisfy dependencies.
+
+
+7) In order to prevent rapid growth of clause count as the
+ number of versions of packages increases, trimming the
+ solution space is essential. I'm currently using the
+ following techniques:
+
+ 1) install a new package on existing system
+
+ identify any existing installed constraints,
+ and trim pkg catalog to eliminate versions
+ outside those constraints.
+
+ trim pkg catalog to exclude all pkg older than
+ those already installed
+
+ input to solver is trimmed catalog, and
+ vectors selecting any version of already installed
+ pkgs that meet constraints, plus a vector selected
+ any version of desired pkg.
+
+ 2) upgrade to latest version of all available pkgs
+
+ identify any existing installed constraints,
+ and trim pkg catalog to eliminate versions
+ OLDER than those constraints.
+
+ trim pkg catalog to exclude all pkg older than
+ those already installed
+
+ input to solver is trimmed catalog, and
+ vectors selecting any version of already installed
+ pkgs
+
+ 3) upgrade to specified version
+
+ identify any existing installed constraints,
+ and trim pkg catalog to eliminate versions
+ OLDER than those constraints.
+
+ trim pkg catalog to exclude all pkg older than
+ those already installed
+
+ input to solver is trimmed catalog, and
+ vectors selecting any version of already installed
+ pkgs, plus vector(s) selecting desired constraint(s).
+
+8) One of the most difficult aspects of using a SAT solver
+ is providing a reasonable error message when no solution
+ can be found.
+
+
+ Some techniques that I'm experimenting with include:
+
+ Explicitly checking for obvious non-starters (pkg
+ version earlier than already installed, pkg version that
+ violates constraints on system) prior to passing to SAT
+ solver. This is needed to permit trimming in any case.
+
+ Using the pruned catalog to quickly evaluate the effect
+ of constraints.
+
+
+Implementation details
+-------------------------
+
+combine catalog object w/ list of installed pkgs and proposed
+changes:
+
+class pkg_solver(object):
+ def __init__(self, catalog, existing_fmris):
+
+ def solve_install(existing_freezes, proposed_fmris):
+ """tries to find solution that adds specified fmris
+ to existing set; any existing packages containing
+ incorporate dependencies which are at most only depended on
+ by name (no version) are frozen."""
+
+ def solve_reinstall(existing_freezes, proposed_fmris):
+ """tries to find solution that replaces existing version
+ with specified version; this one allows stuff to go
+ backwards if specified on command line"""
+
+ def solve_uninstall(existing_freezes, proposed_fmris):
+ """tries to remove specified package"""
+
+ def solve_update_all(existing_freezes):
+ """find most recent version of all packages"""
+
+ solve* routines return a list of tuples (old_version, new_version)
+ for each fmri that is changing; new installs have None as old_version,
+ removals have None as new_version. A returned empty list indicates
+ that no action is needed.
+
+ A failure to find a solution throws an exception,
+ pkg_solver.No_Solution_Found.
+
diff --git a/doc/pkg5_docs/scripting.txt b/doc/pkg5_docs/scripting.txt
new file mode 100644
index 0000000..72beb39
--- /dev/null
+++ b/doc/pkg5_docs/scripting.txt
@@ -0,0 +1,121 @@
+ Getting rid of install/upgrade scripting
+
+The design of SVR4 packaging relies heavily on the use of scripting to
+achieve common packaging operations. These scripts are delivered by
+the package developers, and run in multiple installation contexts,
+which may include install, alternate roots, zones, cross architecture
+and cross OS versions. This has caused a host of complex and thorny
+issues, and led directly to the IPS team's decision to eliminate the
+use of scripting in IPS.
+
+Instead, the IPS architecture requires software to be largely
+self-assembling; changes in configuration are either detected at boot
+time or the appropriate SMF services are restarted in the case of live
+installation of packages. [In the few cases when this is not possible
+due to assembly being needed for boot, the required support is being
+built into IPS directly; such cases are actually rare].
+
+Some concrete examples of how such "self-assembly" can be realized may
+be worthwhile:
+
+The recent hostid project moved the storage of the hostid information
+on i386 architecture machines from the sysinit kernel module (which
+was bpatch'd during install) into the /etc/hostid file. The initial
+design for handling upgrade added code to both BFU and the installer,
+which decoded the hostid from the sysinit module and created the
+hostid file during upgrade. Freshly installed systems had no hostid
+file, so the kernel generated one dyanmically and a SMF service
+created the /etc/hostid file after boot. Since neither bfu nor
+install code runs during IPS image-update operations, this strategy
+did not work on OpenSolaris and the hostid was not preserved on
+upgrade.
+
+To fix this, the kernel was modified to search for a sysinit module
+and read the hostid from that if /etc/hostid didn't already exist.
+Only if that was not present did the kernel generate a new hostid. As
+a result of this change, the upgrade code could be eliminated from
+both BFU and upgrade, and the system upgrade process just works the
+same in all cases.
+
+The key design pattern here is to have the consumer of the
+configuration file (here, genunix) handle the old file format/location
+if the new file doeesn't exist; writing out the new file resolves the
+upgrade process and completes the self-assembly.
+
+Another common problem is the configuration file management problem.
+A canonical example might be /etc/security/exec_attr. Here many
+different packages contribute lines into this file. With SVR4
+packages, a package's file fragments are merged in by the class action
+script i.rbac during installation or upgrade. Removal is problematic
+on uninstall and is not attempted.
+
+For IPS, the proposed solution is to have each package deliver its
+portion of this file into a separate directory, using the name of the
+package (suitably escaped, of course) as the file name. A service
+that runs at boot determines whether or not the file needs to be
+re-composited from the file fragments. This cleanly handles both
+install and uninstall.
+
+The key design pattern here is to have packages deliver their
+components as files, which is something the packaging system is
+designed to do. Assembly of the composited file is left to boot time
+and an SMF service, which deals with older package fragments, etc.
+
+Another approach to the configuration file problem is to change the
+format of the file. A classic problem is the management of
+/etc/driver_aliases, which is "maintained" by add_drv and update_drv.
+This file maintains a mapping between a PCI id and the driver that
+supports that device. Note that rather than maintaining a text file
+which would be read in by the kernel at boot, an alternate approach
+would be to encode the same information as a directory of symbolic
+links from the PCI id to the name of the driver. This would eliminate
+the potential confusion of two different drivers trying to own the
+same PCI id, the need for locking the file during updates, and the
+need to run programs on install and uninstall to manage this file
+since the links would be simply installed and uninstalled as part of
+the normal packaging operations.
+
+The design pattern here is also simple - the filesystem already
+maintains a single namespace, locking, conflict detection, etc;
+reusing these attributes saves time and effort.
+
+Another common problem is moving files, links and directories around
+the filesystem and between packages; this is often handled via
+scripting in the case of SVR4 packages. In IPS, normal (not editable)
+files can simply move between packages; this results in their removal
+from the original location and then their re-installation in their new
+location. To insure that editable files retain their customizations,
+the receiving package must annotate the actions that deliver the new
+file name/location with an attribute that defines the original owning
+packages and location. This looks like this:
+
+original_name=SUNWfoo:etc/foo.txt
+
+indicating that the name of original owning package was SUNWfoo and
+that the original path was etc/foo.txt. Note that no matter where
+this file moves to subsequently, it should always maintain this
+identifier.
+
+Directories in IPS are automatically reference counted, and are
+deleted when there are no longer any explicit declarations of
+those directories or no files under packaging system control in
+the directory. Left-over files & directories not under packaging
+control are moved to a lost-and-found directory under var/pkg;
+this is needed to support changing directories to symbolic links,
+etc., w/o scripting.
+
+There is not yet a simple method to move unpackaged files, etc
+from an old location to a new one; developers facing this problem
+should move those files as part of their self-assembly at first
+boot and avoid removing the references to the original directories
+to prevent their movement to lost-and-found during upgrade.
+We are considering various solutions to this problem.
+
+In the short term, those developers working on OpenSolaris have a
+foot in both worlds - since OpenSolaris is built from Nevada SVR4
+packages, their code needs to work in both environments. With
+the exception of driver post-install scripts (which are converted
+by the folks working on OpenSolaris to IPS driver actions) the goal
+is to remove all post-install/class action processing other than
+SMF service restarting on live systems using the patterns shown
+here.
diff --git a/doc/pkg5_docs/search.txt b/doc/pkg5_docs/search.txt
new file mode 100644
index 0000000..2fe2e31
--- /dev/null
+++ b/doc/pkg5_docs/search.txt
@@ -0,0 +1,74 @@
+
+pkg
+SEARCH
+
+1. Goals
+
+ i. Provide relevant information
+ ii. Provide a consistently fast response
+ iii. Make responses consistent between local and remote search
+ iv. Provide the user with a good interface to the information
+ v. Allow seamless recovery when search fails
+ vi. Ensure the index is (almost) always in a consistent state
+
+2. Approach
+
+ From a high level, there are two components to search: the
+ indexer, which maintains the information needed for search; the
+ query engine, which actually performs a search of the information
+ provided. The indexer is responsible for creating and updating the
+ indexes and ensuring they're always in a consistent state. It does this
+ by maintaining a set of inverted indexes as text files (details of which
+ can be found in the comments at the top of indexer.py). On the server
+ side, it's hooked into the publishing code so that the index is updated
+ each time a package is published. If indexing is already happening when
+ packages are published, they're queued and another update to the indexes
+ happens once the current run is finished. On the client side, it's
+ hooked into the install, image-update, and uninstall code so that each
+ of those actions are reflected in the index.
+
+ The query engine is responsible for processing the text from the user,
+ searching for that token in its information, and giving the client code
+ the information needed for a reasonable response to the user. It must
+ ensure that the information it uses is in a consistent state. On the
+ server, an engine is created during the server initialization. It reads
+ in the files it needs and stores the data internally. When the server gets
+ a search request from a client, it hands the search token to the query
+ engine. The query engine ensures that it has the most recent information
+ (locking and rereading the files from disk if necessary) and then searches
+ for the token in its dictionaries. On the client, the process is the same
+ except that the indexes are read from disk each time instead of being stored
+ because a new instance of pkg is started for each search.
+
+3. Details
+
+ Search reserves the $ROOT/index directory for its use on both the client
+ and the server. It also creates a TMP directory inside index which it stores
+ indexes in until it's ready to migrate them to the the proper directory.
+
+ indexer.py contains detailed information about the files used to store the
+ index and their formats.
+
+ 3.1 Locking
+
+ The indexes use a version locking protocol. The requirements for the
+ protocol are:
+ the writer never blocks on readers
+ any number of readers are allowed
+ readers must always have consistent data regardless the
+ writer's actions
+ To implement these features, several conventions must be observed. The
+ writer is responsible for updating these files in another location,
+ then moving them on top of existing files so that from a reader's
+ perspective, file updates are always atomic. Each file in the index has
+ a version in the first line. The writer is responsible for ensuring that
+ each time it updates the index, the files all have the same version
+ number and that version number has not been previously used. The writer
+ is not responsible for moving multiple files atomically, but it should
+ make an effort to have files in $ROOT/index be out of sync for as short
+ a time as is possible.
+
+ The readers are responsible for ensuring that the files their reading
+ the indexes from are a consistent set (have identical version
+ numbers). consistent_open in search_storage takes care of this
+ functionality.
diff --git a/doc/pkg5_docs/server_api_versions.txt b/doc/pkg5_docs/server_api_versions.txt
new file mode 100644
index 0000000..3007773
--- /dev/null
+++ b/doc/pkg5_docs/server_api_versions.txt
@@ -0,0 +1,252 @@
+Version 12:
+ Incompatible with clients using versions 0-11.
+
+ pkg.server.api changed as follows:
+ * New constants 'PKG_STATE_RENAMED' and 'PKG_STATE_OBSOLETE' were
+ added to assist in determining package state information returned
+ from functions to the CatalogInterface class.
+
+ * The fmris() function of CatalogInterface now has an 'ordered'
+ parameter to control whether FMRIs are returned in ascending name,
+ descending version order.
+
+ * A new function named 'gen_allowed_packages' was added to
+ CatalogInterface to determine a list of packages allowed based on
+ the FMRI of a package that incorporates other packages.
+
+ * A new function named 'gen_packages' was added to
+ CatalogInterface as a replacement for the removed functions
+ get_matching_version_fmris() and get_matching_pattern_fmris(). It
+ provides superior filtering capabilities and includes package state
+ and attribute information.
+
+Version 11:
+ Incompatible with clients using versions 0-10.
+
+ pkg.server.api changed as follows:
+ * A new property named 'version' was added to the CatalogInterface
+ class to expose the version format of the catalog.
+
+ * The 'rename_requests' property was removed from the ConfigInterface
+ class as the stat it represented was for functionality that was
+ never implemented.
+
+ * A new property named 'publisher' that returns the Publisher
+ object for the publisher related to the request was added to the
+ RequestInterface class.
+
+Version 10:
+ Incompatible with clients using versions 0-9.
+
+ pkg.server.api changed as follows:
+ * The PackageInfo.PREF_PUBLISHER property and data was removed.
+
+ * The CatalogInterface.INFO_MULTI_MATCH property was removed and
+ the info() method no longer detects and considers multiple
+ matches for a single pattern an error.
+
+Version 9:
+ Incompatible with clients using versions 0-8.
+
+ pkg.server.api changed as follows:
+ * All feed related properties retrieved or set using the
+ (get|set)_repo_* methods are no longer valid. Instead,
+ all feed-related configuration information must be
+ retrieved using the new depot configuration methods
+ found in the ConfigInterface class.
+
+Version 8:
+ Incompatible with clients using versions 0-7.
+
+ pkg.server.api changed as follows:
+ * The return type of CatalogInterface.search changed so that instead
+ of returning the fmri string in the result, it returns a PkgFmri
+ object.
+
+Version 7:
+ Compatible with clients using version 6.
+
+ CatalogInterface has changed as follows:
+ * A new function named 'info' was added to provide a way to
+ retrieved package information. It returns a PackageInfo
+ object representing the set of available package information.
+ See pydoc 'pkg.client.api.PackageInfo' for more information.
+
+ * A new function named 'get_entry_all_variants' was added to
+ provide access to the list of variants for a given package.
+
+ pkg.server.api changed as follows:
+ * New classes representing package metadata and license
+ information were added to pkg.api_common.
+
+ pkg.server.api_errors changed as follows:
+ * A new exception named 'UnrecognizedOptionsToInfo' was added.
+
+Version 6:
+Incompatible with clients using versions 0-5.
+ CatalogInterface:
+ * get_matching_pattern_fmris() and get_matching_version_fmris()
+ now return a tuple of (fmris, unmatched). Where 'fmris' is
+ a list of matching FMRIs, and 'unmatched' is a dict of
+ unmatched patterns or versions indexed by match criteria.
+
+ * package_count now returns the number of unique packages in the
+ catalog instead of the unique number of package versions.
+
+ * package_version_count, a new property, was added that contains
+ the number of unique package versions in the catalog.
+
+ ConfigInterface:
+ * get_repo_attrs was renamed to get_repo_properties
+
+ * get_repo_attr_value was renamed to get_repo_property_value
+
+Version 5:
+Compatible with clients using Versions 3-4.
+ ConfigInterface.get_repo_attr_value() has changed as follows:
+ * Section 'feed' attribute 'authority' has been removed. It has been
+ replaced by section 'publisher' attribute 'prefix'.
+
+ * Section 'publisher' with attributes 'alias' and 'prefix' was added.
+
+ * New attributes were added for section 'repository': 'collection_type',
+ 'legal_uris', 'mirrors', 'origins', 'refresh_seconds',
+ 'registration_uri', and 'related_uris'. See the pydoc for
+ pkg.server.api.ConfigInterface for details.
+
+Version 4:
+Compatible with clients using Version 3.
+
+Changes:
+ CatalogInterface.search() has changed as follows:
+ * A docstring has been added; see pydoc pkg.server.api for details.
+
+ * Added optional keyword 'matching_version' that allows consumers to
+ filter search results based on version.
+
+ * Added optional, boolean keyword 'return_latest' that causes only the
+ the newest versions of packages to be returned when 'return_type'
+ is Query.RETURN_PACKAGES.
+
+Version 3:
+Incompatible with clients using Versions 0-2.
+
+Changes:
+ CatalogInterface.search() has changed as follows:
+ * Added optional, boolean keyword 'case_sensitive'. This indicates
+ whether a case-sensitive search should be performed.
+
+ * Added optional keyword argument 'return_type'. This determines
+ whether results should be returned as Query.RETURN_ACTIONS or
+ Query.RETURN_PACKAGES.
+
+ * Added optional, integer keyword argument 'start_point'. This
+ specifies how many matching results should be skipped before
+ returning anything.
+
+ * Added optional, integer keyword argument 'num_to_return'. This
+ indicates how many results should be returned before the search
+ is aborted.
+
+ * search_done() was removed. Previously, after calling search(), api
+ consumers would have to call search_done(). This is no longer
+ necessary.
+
+Version 2:
+Incompatible with clients using Versions 0-1.
+Changes:
+CatalogInterface.get_matching_version_fmris() no longer accepts the constraint
+parameter. However, as before, it expects a list of version strings for the
+'versions' parameter. These version strings may now contain the wildcard
+characters '*' and '?'.
+
+Version 1:
+Incompatible with clients using Version 0.
+Changes:
+CatalogInterface.search_done() was added to perform cleanup after all results
+have been retrieved from search().
+
+CatalogInterface.search() now returns a generator object, instead of a list
+object, that requires that CatalogInterface.search_done() is called after all
+of the desired results have been retrieved for proper cleanup.
+
+Version 0:
+Initial api version, containing the following:
+class BaseInterface
+ -- used to instantiate other interface objects
+
+class CatalogInterface
+ def __init__(self, version_id, base):
+ def fmris(self):
+ def get_matching_pattern_fmris(self, patterns):
+ def get_matching_version_fmris(self, versions,
+ constraint=pkg.version.CONSTRAINT_AUTO):
+
+ @property
+ last_modified
+
+ @property
+ package_count
+
+ def search(self, token):
+
+ @property
+ search_available
+
+class ConfigInterface
+ def __init__(self, version_id, base):
+
+ @property
+ catalog_requests
+
+ @property
+ content_root
+
+ @property
+ file_requests
+
+ @property
+ filelist_requests
+
+ @property
+ filelist_file_requests
+
+ @property
+ in_flight_transactions
+
+ @property
+ manifest_requests
+
+ @property
+ mirror
+
+ @property
+ readonly
+
+ @property
+ rename_requests
+
+ @property
+ web_root
+
+ def get_repo_attrs(self):
+
+ def get_repo_attr_value(self, section, attr):
+
+class RequestInterface
+ def __init__(self, version_id, base):
+
+ def get_accepted_languages(self):
+
+ def get_rel_path(self, uri):
+
+ def log(self, msg):
+
+ @property
+ params
+
+ @property
+ path_info
+
+ def url(self, path='', qs='', script_name=None, relative=None):
+
diff --git a/doc/pkg5_docs/signed_manifests.txt b/doc/pkg5_docs/signed_manifests.txt
new file mode 100644
index 0000000..9c62308
--- /dev/null
+++ b/doc/pkg5_docs/signed_manifests.txt
@@ -0,0 +1,239 @@
+ Manifest signing
+ ----------------
+
+Manifests in IPS contain all the packaging metadata - file
+permissions, ownership, content hashes, etc, and are stored and
+transmitted as a simple text file with one line per action. During
+download or when a system is checked for compliance, the manifest
+contents are compared to the files to determine whether or not a
+package is correctly received/installed. Given their importance,
+verifying that all manifests are correct and reflect the original
+publisher's intent is an important part of system validation.
+Cryptographic signatures protecting the integrity of all actions form
+a Merkle hash tree that includes the delivered binaries such that
+complete verification of the installed software is possible. There
+are other uses for manifest signing beyond validation; signatures can
+also be used to indicate approval by other organizations or parties.
+For example, the internal QA organization could sign manifests of
+Sun-delivered packages once they had be determined to be qualified for
+production use; policy settings could mandate such approvals prior to
+installation.
+
+As a result, it is a useful characteristic for signatures to be
+independent of other signatures in a manifest; it should be possible
+to add (or remove) signatures (but not other actions) in a manifest
+without invalidating the other signatures that are present. This
+feature also facilitates production handoffs, with signatures used
+along the path to indicate completion along the way; subsequent steps
+can optionally remove previous signatures at any time without ill-effect.
+
+The need to treat signatures differently during hash computation
+suggests that the signature itself should be easily distinguished from
+other sorts of package metadata; this leads us to a new signature
+action, of the form:
+
+signature algorithm= \
+ value= \
+ chain="" \
+ version=
+
+The payload and pkg.chain_certs attributes represent the packaging hash of the
+pem file(s) containing the x.509 certificate(s) downloadable from the
+originating repository; the value is the signed hash of the manifest's message
+text, prepared as discussed below. The payload certificate is the certificate
+which verifies the value in pkg.sigval. The other certificates presented need
+to form a certificate path that leads from the payload certificate to the trust
+anchor(s) that was established as part of the publisher configuration.
+
+To compute the signature of a manifest, first a canonical representation of the
+manifest is created. (See "Computing the manifest's message text" below.) The
+message text is then given to the signature algorithm along with the private key
+and the result is the value of the signature.
+
+Two types of signature algorithms are currently supported. The first is rsa
+group of signature algorithms. An example is "rsa-sha256". The bit after the
+dash specifies the hash algorithm to use to change the message text into a
+single value the rsa algorithm can use.
+
+The second type of signature algorithm is compute the hash only. This type of
+algorithm exists primarily for testing and process verification purposes and
+presents the hash as the signature value. A signature action of this type is
+indicated by the lack of a payload certificate hash. This type of signature
+action is verified if the image is configured to check signatures. Its
+presence however does not count as a signature if signatures are required.
+
+signature algorithm= value= \
+ version=
+
+Additional signature types (pgp, for example) may be added in the future.
+
+Additional metadata can be added to a signature if desired, as with any
+other action.
+
+Policies may be set for the image or for specific publishers. The policies
+include ignoring signatures, verifying existing signatures, requiring
+signatures, and requiring that specific common names must be seen in the chain
+of trust. Other policies may be added in the future.
+
+Computing the manifest's message text:
+--------------------------------------
+
+Manifests have an interesting property: the lines in a manifest may be
+reordered without affecting the meaning of the manifest. As a result,
+manifest order is not preserved and subject to change during package
+publication processing. It is thus necessary for our manifest signing
+to be independent of presented line order, or the action ordering
+algorithm used for installation, as that may change over time.
+
+Straightforward C-locale alphabetical sorting of attributes within
+actions, the multiple values within those attributes, and across actions
+can be used to enforce a consistent ordering for signature purposes and
+is not subject to change for a given manifest.
+
+In order to allow other signatures to be added or removed from a
+manifest, computation of the manifest message text does not include
+other signatures; in order to protect metadata on the signature
+itself, the signature being produced or verified is included in the
+message text at the end, aside of course from the actual signature
+value itself.
+
+For example, take the following manifest:
+set name=fmri value=foo@1.0
+dir path=foo/bar group=sys
+signature cert1 algorithm=rsa-sha256 value=val1 random_attr=baz
+signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee
+
+The text used to compute val1 is:
+dir group=sys path=foo/bar
+set name=fmri value=foo@1.0
+signature cert1 algorithm=rsa-sha256 value= random_attr=baz
+
+The text used to compute val2 is:
+dir group=sys path=foo/bar
+set name=fmri value=foo@1.0
+signature cert2 another_attr=whee algorithm=rsa-sha256 value=
+
+Including the text of the signature action itself prevents the signature action
+from being modified. Ensuring that the text used for compute val1 contains no
+mention of the second signature (or any other signature) allows signatures to be
+added and removed freely.
+
+Verification of merged signatures:
+----------------------------------
+
+We produce "fat" packages (containing variants such as different
+architectures, debug vs non-debug kernel, etc) by producing manifests
+for each variant, and then merging them. Actions that are the same
+between variants being merge are left unmodified; those that are
+different receive a variant tag (see facets.txt). If it is considered
+useful to have signatures persist and be useful across such merges,
+some additional steps are required to verify such signatures after
+merging.
+
+Generally, signatures will be unique to their variant, thus they will
+be tagged with variant tags after merging. To verify signatures
+post-merge, the evaluation process has three steps after other signature actions
+have been removed from consideration.
+
+1) Any variant tags present on the signature are assumed to have been added
+after the merge. Thus, all actions whose variants do not match the signature's
+variants are removed from inclusion in the message text.
+2) Since the variant tags were not present when the manifest was signed, they
+need to be removed from the attributes of each of the remaining actions as
+well. This includes removing the set attributes which define the variants for
+the package.
+3) Restore any set actions in the existing_pkg_vars attribute which
+describe variants which have been removed from the text.
+
+For example, consider the following manifest:
+set name=fmri value=foo@1.0
+set name=variant.arch value=sparc value=i386
+set name=variant.debug value=true value=false
+dir path=foo1 group=sys
+dir path=foo2 variant.arch=sparc
+dir path=foo2/d variant.arch=sparc variant.debug=true
+dir path=foo2/nd variant.arch=sparc variant.debug=false
+dir path=foo3 variant.arch=i386
+signature cert1 algorithm=rsa-sha256 value=val1 random_attr=baz \
+ variant.arch=sparc variant.debug=true
+signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee \
+ variant.arch=i386 existing_pkg_vars="variant.arch=i386"
+
+To compute the message text for the first signature, the first step is applied
+producing the following text:
+set name=fmri value=foo@1.0
+set name=variant.arch value=sparc value=i386
+set name=variant.debug value=true value=false
+dir path=foo1 group=sys
+dir path=foo2 variant.arch=sparc
+dir path=foo2/d variant.arch=sparc variant.debug=true
+signature cert1 algorithm=rsa-sha256 value=val1 random_attr=baz \
+ variant.arch=sparc variant.debug=true
+
+After the second step is applied, the text becomes:
+set name=fmri value=foo@1.0
+dir path=foo1 group=sys
+dir path=foo2
+dir path=foo2/d
+signature cert1 algorithm=rsa-sha256 value=val1 random_attr=baz
+
+Since the third step doesn't apply to this signature, the above text becomes the
+canonical message text.
+
+Computing the message text for the second signature proceeds like this:
+After the first step the text is:
+set name=fmri value=foo@1.0
+set name=variant.arch value=sparc value=i386
+set name=variant.debug value=true value=false
+dir path=foo1 group=sys
+dir path=foo3 variant.arch=i386
+signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee \
+ variant.arch=i386 existing_pkg_vars="variant.arch=i386"
+
+After the second step, the text is:
+set name=fmri value=foo@1.0
+dir path=foo1 group=sys
+dir path=foo3
+signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee \
+ existing_pkg_vars="variant.arch=i386"
+
+In the third step, we restore the set action which was present in the original
+manifest. The text becomes:
+set name=fmri value=foo@1.0
+set name=variant.arch value=i386
+dir path=foo1 group=sys
+dir path=foo3
+signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee \
+ existing_pkg_vars="variant.arch=i386"
+
+The existing_pkg_vars attribute allows whether a package variant set
+action was present or not at the time of signing to be determined
+deterministically.
+
+Publication of signed manifests:
+--------------------------------
+
+Publishing a signed manifest is a two step process. First the package is
+published, unsigned, to a repository. The package is then updated in place,
+using pkgsign, appending a signature action to the manifest in the repository
+but leaving the package, including its timestamp, intact. This process allows a
+signature action to be added by someone other than the publisher without
+invalidating the publisher's signature. For example, the QA department of a
+company may want to sign all packages that are installed internally to indicate
+they have been approved for use, but not republish the packages which would
+create a new timestamp and invalidate the signature of the original publisher.
+
+The disadvantage of this approach is that a fmri no longer represents a single
+manifest eternally. Eventually, this problem is solved by having the client
+ensure that the hash of the manifest it loads for the fmri matches the hash in
+the catalog. Until that feature is implemented, it is imperative that
+publishers ensure that no client has access to an unsigned manifest which they
+plan to sign in the future, or to return to the QA example, that no client
+inside the organization sees a manifest for a fmri without a signature which the
+QA plans to sign in the future.
+
+pkgsign is able to update a package in place through the use of a new publishing
+operation called append. This operation opens a transaction to modify the
+manifest of a fmri which is already in the repository. When the transaction is
+closed, the signed manifest replaces the existing manifest and the catalog is
+updated to reflect the new hash of the manifest.
diff --git a/doc/pkg5_docs/system_repository.txt b/doc/pkg5_docs/system_repository.txt
new file mode 100644
index 0000000..ed19cf2
--- /dev/null
+++ b/doc/pkg5_docs/system_repository.txt
@@ -0,0 +1,106 @@
+System Repository and Publishers
+
+Introduction:
+
+Linked images, and zones in particular, must keep certain packages
+in sync with the global zone in order to be functional. The global zone will
+constrain packages within the non-global zones and configure special publishers
+in the non-global zone (NGZ). These publishers (henceforth called system
+publishers) are special because the non-global zone cannot make certain kinds of
+modifications to them. Among the forbidden operations for the non-global zone on
+the system publishers are deleting, disabling, removing or replacing origins
+provided by the system repository, and any other operations which might prevent
+the solver from meeting the constraints imposed by the constraint package. The
+global zone must provide the means for the non-global zone to configure itself
+with system publishers by providing information like origins. The global zone
+also has to provide a connection to the system publishers' repositories which is
+available even in a scratch zone.
+
+
+The Data path:
+
+The pkg client in the NGZ uses the system repository in the global zone as a
+proxy to the system publishers. To ensure that a communication path between the
+pkg client in the NGZ and the system repository in the global zone always
+exists, the zone proxy client and the zone proxy daemon were created.
+
+The zone proxy client runs in the NGZ. When started, it creates a socket which
+listens on an inet port on 127.0.0.1 in the NGZ. It passes the file descriptor
+for this socket to the zone proxy daemon in the global zone via a door call. The
+zone proxy daemon listens for connections on the file descriptor. When zone
+proxy daemon receives a connection, it proxies the connection to the system
+depot. The system depot is an Apache instance running in the global zone which
+provides connectivity to and configuration of publishers.
+
+The system depot acts as a proxy for the http and https repositories for
+the publishers it provides. When proxying to https repositories, it uses the
+keys and certificates in the global zone to identify itself and verify the
+server's identity. It also provides a http interface to the file repositories
+for the publishers it provides as well as serving publisher and image
+configuration via the syspub/0 response.
+
+
+Configuration:
+
+The syspub/0 response is a p5s file. The p5s file contains publisher
+configuration and image configuration. Currently, the only image configuration
+it contains is the publisher search order for the provided publishers, but other
+information may be added to the response as needed. In addition to the basic
+collection of publisher information, the p5s file also contains a list of URIs
+which the pkg client should proxy to via the system depot instead of contacting
+them directly. When creating a p5s file, the URI for origins and mirrors can
+be transformed. HTTPS URIs are transformed to HTTP URIs since the system depot
+will be doing the SSL communication, not the pkg client. File URIs are
+transformed into HTTP URIs with a special format. The URIs contain the special
+token "" which the p5s parser knows to replace with the URI of the zone
+proxy client. The rest of the URI contains the prefix of the publisher, then
+the sha1 hash of the global zone path to the file repository.
+
+The information for the syspub/0 response comes from the global zone's image's
+configuration. The application/pkg/system-repository service is responsible for
+transforming the image configuration into an Apache configuration file and
+causing the system depot to reread its configuration. The global zone
+pkg client restarts the application/pkg/system-repository service whenever the
+image's publisher configuration changes.
+
+The Apache configuration file, written by sysrepo.py using the Mako template
+sysrepo_httpd.conf.mako does two things:
+
+ * enables an Apache proxy accepting only requests to the configured
+ publisher http URIs (and utilizes a file-based proxy cache) only listening
+ on the loopback interface by default.
+
+ * adds a series of mod_rewrite RewriteRule and Alias directives to allow
+ the Apache instance to gain access to configured file:// publishers.
+
+The Apache URIs are accessible from the system repository configured at
+'' for a given publisher '' are below.
+
+We serve static responses to the following URIs, the content being either
+a known versions file, or the p5s file mentioned above:
+
+ http:///versions/0/
+ http:///syspub/0
+
+For access to file:// repositories, we use rewrite rules and Alias directives to
+access the file repository contents. In order to allow repositories with
+different paths, yet prevent exposing those paths to system repository clients,
+we use an SHA-1 hash of the path, and include that in the URI (below, this hash
+is represented by "HASH") The URIs we accept for file:// repositories are:
+
+ http:////HASH/file/1/
+ http:////HASH/manifest/0/
+ http:////HASH/publisher/0
+ http:///