CloudNebulaProject/ips
1
0
Fork 0
mirror of https://codeberg.org/Toasterson/ips.git synced 2026-04-10 13:20:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Import pkg5 docs as reference for new code.

Browse source
This commit is contained in:
Till Wegmueller 2025-07-22 11:57:44 +02:00
parent e3662eaf23
commit 2e69b277ed
No known key found for this signature in database
89 changed files with 20320 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

126
doc/pkg5_docs/Makefile Normal file
View file

@ -0,0 +1,126 @@
#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License (the "License").
# You may not use this file except in compliance with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#
# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
PWD:sh = pwd
RST2HTML_FLAGS = --rfc-references \
--no-generator \
--time \
--stylesheet-path=$(CSS) \
--link-stylesheet
RST2LATEX = rst2latex.py
RST2LATEX_FLAGS = --documentclass=ireport --documentoptions=12pt
draft := RST2LATEX_FLAGS = --documentclass=idraft --documentoptions=12pt
.MAKE.STATE:
M4_ARGS = -DCOMPLETE=1
WEB_INCLUDES = macros.rsi web.css
output/%.rsi: output %.rst
gm4 $(M4_ARGS) $< > $@
TXT_FRAGMENTS = \
apt.txt \
catalog.txt \
deduction.txt \
elf-jar-handling.txt \
es-requirements.txt \
file-metadata.txt \
filter.txt \
image.txt \
multi-platform.txt \
one-pager-main.txt \
protocol-versioning.txt \
publication.txt \
razor.txt \
rest.txt \
rfes.txt \
transaction-order.txt \
usr-setuid-bins.txt \
versions.txt \
wos-conversion.txt \
wos-filetype-stats.txt \
xml.txt
RST_FRAGMENTS = \
actions.rst \
depot.rst \
guide-basic-ops.rst \
guide-implementation-depot.rst \
guide-metadata-conventions.rst \
guide-naming-conventions.rst \
guide-pkg-states.rst \
guide-publication-protocol.rst \
guide-repository-format.rst \
guide-retrieval-protocol.rst \
guide-txn-states.rst \
RST_JOINS = \
guide-main.rst
CSS=pkg-guide-web.css
HTML_DIR = html-out
FILE_URL = file://$(PWD)/$(HTML_DIR)
HTML_CSS = $(HTML_DIR)/$(CSS)
HTML_FRAGMENTS = $(RST_FRAGMENTS:%.rst=$(HTML_DIR)/%.html)
HTML_JOINS = $(RST_JOINS:%.rst=$(HTML_DIR)/%.html)
.KEEP_STATE:
pdf: guide-main.pdf
draft: pdf
html: $(HTML_DIR) .WAIT $(HTML_CSS) $(HTML_FRAGMENTS) $(HTML_JOINS)
$(HTML_JOINS): $(HTML_FRAGMENTS)
html-out:
-mkdir $@
# If we are embedding stylesheets, then we may wish to have this rule also
# depend on $(CSS).
$(HTML_DIR)/%.html: %.rst macros.rst
rst2html.py $(RST2HTML_FLAGS) $< > $@
$(HTML_DIR)/$(CSS): $(CSS)
cp $(CSS) $@
%.pdf: %.rst macros.rst $(RSI_INCLUDES)
$(RST2LATEX) $(RST2LATEX_FLAGS) $< > $(<:%.rst=%.tex)
pdflatex $(<:%.rst=%.tex)
pdflatex $(<:%.rst=%.tex)
clean:
rm -f $(HTML_FRAGMENTS) $(RSI_INCLUDES)
clobber: clean

60
doc/pkg5_docs/README.guide Normal file
View file

@ -0,0 +1,60 @@
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
README.guide
0. Format
We're writing the Developer Guide in reStructured Text. Once we're
happy with the content, we'll output XML and convert that to
DocBook-XML to hand over to Documentation for final production.
Heading in rST are based on over- and underlining. Heading levels
are a document-specific convention; our levels are as follows:
==== ----
==== , ---- , ==== , ---- , ~~~~, ````, ^^^^
Inconsistencies with this ordering will lead to document build
failure.
1. Files
guide-main.rst is the Developer Guide's outline. It contains the
main document directives; it may also contain shorter sections,
prior to being broken out into separate files.
Each file should begin with its appropriate heading for the Guide as
a whole.
1.1. Titles
Titles are always mixed case. Particular levels may be set in
capitals or small capitals by style sheet.
2. Rest of the directory
Files not starting with guide- are legacy documents from earlier in
the project. If you are rewriting one of these to fit into the
Guide, please rename them; if you don't feel you're the owner,
extract the content, and add an "XXX Copied to guide-....rst" in the
original file.
3. Tools
You can obtain docutils by using easy_install via
$ pfexec easy_install docutils
With a straight docutils installation, you can build the "html"
target out of the Makefile.
To build the PDF version, you will need a pdflatex-capable TeX
installation, such as TeX Live, which is available for download at
http://www.tug.org/texlive/
The Makefile and docutils expect that the TeX executables are
available via the path.

62
doc/pkg5_docs/TODO Normal file
View file

@ -0,0 +1,62 @@
pkg
TODO
During the prototype phase, this is merely a list of items not fully
discussed, written up or implemented.
1. Algorithms/Logic
- Preferred publisher fallback
- a discussion of catalogs, comparisons between, and
operations on
- per-package publisher association
- Catalog update mechanism
- event oriented
- digested
- Freeze/constraint flow
- statement
- back tracking
- Snapshot/revert safety
- Image/substrate relationship
- Image operational history
- Conventions for automated repository discovery
- Peer/neighborhood cooperation/verification/etc.
- Depot management
- store forever and demote from catalog
- conserve disk space
- client as depot?
- Compatibility options
- understand SysV as fixed version:timestamp packages
from a legacy publisher?
2. Formats/interfaces
- Versioned formats (event, full) for catalog
- Versioned format for manifest
- Versioned format for marshalled transactions ("new package
format")
- Practices around REST and versions (entry points or data?)
3. Data analysis
- continued examination of version and patch history of S9 and
S10 (S8?)
- write up worked examples (e.g. telnet on both sides
of private kernel API change)
4. Coding
- Server split to transaction server and plain old HTTP server
sides
- SSH tunnel support in pkgsend(1)

257
doc/pkg5_docs/actions.rst Normal file
View file

@ -0,0 +1,257 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
.. :vim:set expandtab:
Supported Actions
-----------------
We need to support an extensible set of "actions", which we define as
reversible operations that a package can request to enable its later
function on the target image.
Packages need a limited set of operations on individual files to
manipulate the configuration. The current class actions are given in
Appendix A. It appears that if "manifest" and "rbac" were supported,
along with some management of editable files (preserve, renamenew,
initd, renameold), then the remaining operations could be deferred to
image instantiation.
From the legacy packaging system, we can inspect the class action
scripts and the postinstall scripts to identify the set of common
actions.
depend
Declare dependency on other packages.
directory
All directories.
driver
Package contains device driver Module loading will be disabled
during operations on live images.
file
All other files. Preservation and rename handling are managed as
optional tags.
hardlink, link
All hard and symbolic links.
service
Package contains service description. Inventoried services will
be temporarily disabled during operations on live images.
set
Set a package tag.
user, group
Package requires user, group, or other package-reference managed
resource.
XXX Is this set sufficient to boot? Should we somehow discriminate
them from non-booting actions?
legacy
Record package attributes into legacy packaging metadata.
license
License files, which deliver into the image metadata rather than
the image's filesystems.
XXX Do we have a hard-reboot, reconfigure-reboot, and/or soft-reboot
action? Otherwise we are going to build path knowledge somewhere else
in the packaging system.
Interface summary::
<interface>
<action name="dependency" payload="false" commitment="Committed" />
<action name="directory" payload="false" commitment="Committed" />
<action name="hardlink" payload="false" commitment="Committed" />
<action name="legacy" payload="false" commitment="Committed" />
<action name="license" payload="true" commitment="Committed" />
<action name="link" payload="false" commitment="Committed" />
<action name="driver" payload="false" commitment="Committed" />
<action name="file" payload="true" commitment="Committed" />
<action name="group" payload="false" commitment="Committed" />
<action name="service" payload="true" commitment="Committed" />
<action name="user" payload="false" commitment="Committed" />
</interface>
1. Custom actions
It is discouraged, but certainly possible to deliver custom actions
into the appropriate $PYTHONROOT/vendor-packages/pkg directory, by
including those actions in a separate package that the new package
requires, and invoking the pkg(1) client twice--once to deliver the
custom actions and once to use them to install the new package.
(Rescanning pkg.actions would complicate the image plan/package plan
evaluations.)
The deployer may wish to deny such actions from operating. For this
case, the set of known actions is fixed elsewhere in the pkg modules
and updated with subsequent versions. A global and per-image policy,
known-actions-only, allows the deployer to disallow operations on
packages utilizing actions of unknown provenance.
Interface::
<interface>
<policy name="known-actions-only" scope="global,image"
type="boolean" commitment="Committed">
Deployer control over execution of unknown actions.
</policy>
<interface>
Appendix A. Current class actions on Solaris NV.
Command output
> $ grep -v none /tmp/summary | \
> egrep 1\ \[ef\]\ | \
> cut -d \ -f 3 | \
> sort | uniq -c | sort -nr
> 152 manifest
> 129 preserve
> 45 renamenew
> 32 rbac
> 30 initd
> 30 fontsdir
> 17 fontsalias
> 16 appservenv
> 10 ttmapsdir
> 10 encodingsdir
> 8 renameold
> 6 build
> 5 tiservices
> 5 master
> 5 asenv
> 4 smfyes
> 4 services
> 4 immodules
> 4 fontsupr
> 4 fontsscale
> 4 fontenc
> 3 OWconfig
> 2 smfno
> 2 smf
> 2 sendmail
> 2 sed
> 2 owfontpath
> 2 fonttmap
> 2 devlink
> 2 append
> 1 ypnicknames
> 1 vfstab
> 1 ttysrch
> 1 ttydefs
> 1 ttmapkoi8
> 1 ttmap13
> 1 syslogconf
> 1 svmpreserve
> 1 sshdconfig
> 1 sock2path
> 1 shadow
> 1 sdconf
> 1 scsivhciconf
> 1 scsa2usbconf
> 1 sampleslist
> 1 rOWconfig
> 1 qlc
> 1 publickey
> 1 powerconf
> 1 policyconf
> 1 pkcs11confbase
> 1 passwd
> 1 papersize
> 1 pamconf
> 1 opensslcnf
> 1 nsswitch
> 1 nscd
> 1 nfssecconf
> 1 netconfig
> 1 ncalogd
> 1 ncakmod
> 1 nametomajor
> 1 minorperm
> 1 mailxrc
> 1 mach
> 1 logindevperm
> 1 logadmconf
> 1 localprofile
> 1 locallogin
> 1 krbconf
> 1 keytable
> 1 kclasses
> 1 kcfconfbase
> 1 iuap
> 1 iscsiconf
> 1 ipsecalgsbase
> 1 initupdate
> 1 inittab
> 1 init
> 1 inetdconf
> 1 ibnexconf
> 1 hosts
> 1 group
> 1 ftpusers
> 1 ftpaccess
> 1 fstypes
> 1 fpconf
> 1 fonttmap9
> 1 fonttmap7
> 1 fonttmap5
> 1 fonttmap4
> 1 fonttmap2
> 1 fonttmap15
> 1 fontenc9
> 1 fontenc7
> 1 fontenc5
> 1 fontenc4
> 1 fontenc13
> 1 etcsystem
> 1 etcrpc
> 1 etcremote
> 1 etcprofile
> 1 EtcDefLu
> 1 drvalias
> 1 dialers
> 1 dhcpinittab
> 1 devpolicy
> 1 devlinktab
> 1 defsu
> 1 defrpcnisd
> 1 defpasswd
> 1 defnfs
> 1 deflogin
> 1 definit
> 1 cronroot
> 1 configmapconf
> 1 bootenvrc
> 1 automaster
> 1 ataconf
> 1 adpconf
> 1 AddNoUpdate
>

213
doc/pkg5_docs/actions.txt Normal file
View file

@ -0,0 +1,213 @@
PSARC/2008/190
pkg(5): image packaging system
SUPPORTED ACTIONS
We need to support an extensible set of "actions", which we define as
reversible operations that a package can request to enable its later
function on the target image.
Packages need a limited set of operations on individual files to
manipulate the configuration. The current class actions are given in
Appendix A. It appears that if "manifest" and "rbac" were supported,
along with some management of editable files (preserve, renamenew,
initd, renameold), then the remaining operations could be deferred to
image instantiation.
From the legacy packaging system, we can inspect the class action
scripts and the postinstall scripts to identify the set of common
actions.
depend Declare dependency on other packages.
directory All directories.
driver Package contains device driver
Module loading will be disabled during
operations on live images.
file All other files. Preservation and rename handling are
managed as optional tags.
hardlink,
link All hard and symbolic links.
service Package contains service description
Inventoried services will be temporarily
disabled during operations on live images.
set Set a package attribute.
user,
group Package requires user, group, or other package-reference
managed resource.
XXX Is this set sufficient to boot? Should we somehow discriminate
them from non-booting actions?
legacy Record package attributes into legacy packaging
metadata.
license License files, which deliver into the image metadata
rather than the image's filesystems.
XXX Do we have a hard-reboot, reconfigure-reboot, and/or soft-reboot
action? Otherwise we are going to build path knowledge somewhere else
in the packaging system.
<interface>
<action name="dependency" payload="false" commitment="Committed" />
<action name="directory" payload="false" commitment="Committed" />
<action name="hardlink" payload="false" commitment="Committed" />
<action name="legacy" payload="false" commitment="Committed" />
<action name="license" payload="true" commitment="Committed" />
<action name="link" payload="false" commitment="Committed" />
<action name="driver" payload="false" commitment="Committed" />
<action name="file" payload="true" commitment="Committed" />
<action name="group" payload="false" commitment="Committed" />
<action name="service" payload="true" commitment="Committed" />
<action name="user" payload="false" commitment="Committed" />
</interface>
1. Custom actions
It is discouraged, but certainly possible to deliver custom actions
into the appropriate $PYTHONROOT/vendor-packages/pkg directory, by
including those actions in a separate package that the new package
requires, and invoking the pkg(1) client twice--once to deliver the
custom actions and once to use them to install the new package.
(Rescanning pkg.actions would complicate the image plan/package plan
evaluations.)
The deployer may wish to deny such actions from operating. For this
case, the set of known actions is fixed elsewhere in the pkg modules
and updated with subsequent versions. A global and per-image policy,
known-actions-only, allows the deployer to disallow operations on
packages utilizing actions of unknown provenance.
<interface>
<policy name="known-actions-only" scope="global,image"
type="boolean" commitment="Committed">
Deployer control over execution of unknown actions.
</policy>
<interface>
Appendix A. Current class actions on Solaris NV.
$ grep -v none /tmp/summary | egrep 1\ \[ef\]\ | cut -d \ -f 3 | sort | uniq -c | sort -nr
152 manifest
129 preserve
45 renamenew
32 rbac
30 initd
30 fontsdir
17 fontsalias
16 appservenv
10 ttmapsdir
10 encodingsdir
8 renameold
6 build
5 tiservices
5 master
5 asenv
4 smfyes
4 services
4 immodules
4 fontsupr
4 fontsscale
4 fontenc
3 OWconfig
2 smfno
2 smf
2 sendmail
2 sed
2 owfontpath
2 fonttmap
2 devlink
2 append
1 ypnicknames
1 vfstab
1 ttysrch
1 ttydefs
1 ttmapkoi8
1 ttmap13
1 syslogconf
1 svmpreserve
1 sshdconfig
1 sock2path
1 shadow
1 sdconf
1 scsivhciconf
1 scsa2usbconf
1 sampleslist
1 rOWconfig
1 qlc
1 publickey
1 powerconf
1 policyconf
1 pkcs11confbase
1 passwd
1 papersize
1 pamconf
1 opensslcnf
1 nsswitch
1 nscd
1 nfssecconf
1 netconfig
1 ncalogd
1 ncakmod
1 nametomajor
1 minorperm
1 mailxrc
1 mach
1 logindevperm
1 logadmconf
1 localprofile
1 locallogin
1 krbconf
1 keytable
1 kclasses
1 kcfconfbase
1 iuap
1 iscsiconf
1 ipsecalgsbase
1 initupdate
1 inittab
1 init
1 inetdconf
1 ibnexconf
1 hosts
1 group
1 ftpusers
1 ftpaccess
1 fstypes
1 fpconf
1 fonttmap9
1 fonttmap7
1 fonttmap5
1 fonttmap4
1 fonttmap2
1 fonttmap15
1 fontenc9
1 fontenc7
1 fontenc5
1 fontenc4
1 fontenc13
1 etcsystem
1 etcrpc
1 etcremote
1 etcprofile
1 EtcDefLu
1 drvalias
1 dialers
1 dhcpinittab
1 devpolicy
1 devlinktab
1 defsu
1 defrpcnisd
1 defpasswd
1 defnfs
1 deflogin
1 definit
1 cronroot
1 configmapconf
1 bootenvrc
1 automaster
1 ataconf
1 adpconf
1 AddNoUpdate

25
doc/pkg5_docs/apt.txt Normal file
View file

@ -0,0 +1,25 @@
apt-get install pkg_name[-]
suffix with - to remove
apt-get upgrade
upgrades all packages
will "keep back" packages in case of broken dependencies (absent
dependencies), or if a package has an additional dependency
(resolve with apt-get install)
apt-get dist-upgrade
upgrades everything, handling relationship changes
apt-get clean; apt-get autoclean
remove cached packages under varying conditions
/etc/apt/apt.conf
config file
contains APT::Default-Release "version";
/etc/apt/preferences
allows pinning of packages
pin priorities allow packages to be blocked from installation,
or from being upgraded, plus intermediate behaviours

204
doc/pkg5_docs/bhyve.md Normal file
View file

@ -0,0 +1,204 @@
# Bhyve branded-zone support
Bhyve branded zones are configured mainly via custom attributes in the zone
configuration.
To get started, `pkg install system/zones/brand/bhyve` and configure a zone with the
bhyve brand and the appropriate attributes; see the example zone at the end of
this page.
To troubleshoot problems if the zone fails to start, review the log file
which will be created at `/path/to/zone/root/tmp/init.log`
### Attributes
| Attribute | Default | Syntax | Example
| --- | --- | --- | ---
| acpi | on | on,off
| bootdisk<sup>1</sup> | | path[,serial=<serno>] | tank/hdd/bhyve1
| bootorder | cd | \[c\]\[d\]
| bootrom<sup>3</sup> | BHYVE_RELEASE | firmware name\|path to firmware | BHYVE_DEBUG
| cdrom<sup>4</sup> | | path to ISO | /data/iso/FreeBSD-11.1-RELEASE-amd64-bootonly.iso
| console | /dev/zconsole<sup>6</sup> | options | socket,/tmp/vm.com1,wait
| disk<sup>1</sup> | | path[,serial=<serno>] | tank/hdd/bhyve2,serial=1234
| diskN<sup>2</sup> | | path[,serial=<serno>] | tank/hdd/bhyve2,serial=1234
| diskif | virtio-blk | virtio-blk,ahci-hd
| hostbridge | i440fx | i440fx,q35,amd,netapp,none
| netif | virtio-net-viona | virtio-net-viona,e1000
| ram | 1G | n(G\|M) | 8G
| type | generic | generic,windows,openbsd
| vcpus | 1 | [[cpus=]numcpus][,sockets=n][,cores=n][,threads=n]] | cpus=16,sockets=2,cores=4,threads=2
| vnc<sup>5</sup> | off | off,on,options | socket,/tmp/vm.vnc,w=1024,h=768,wait
| extra | | extra arguments for hypervisor |
#### Notes
<ol>
<li>You will also need to pass the underlying disk device through to the zone via a <i>device</i> entry, see the example below;</li>
<li>Use diskN to specify the slot into which the disk will be placed. A plain <i>disk</i> tag will be put in the lowest available slot.</li>
<li>Available firmware files can be found in <i>/usr/share/bhyve/firmware/</i>;</li>
<li>The ISO file needs passing through to the zone via a lofs mount, see the example below;</li>
<li>Setting vnc to <i>on</i> is the same as setting it to <i>unix=/tmp/vm.vnc</i>.</li>
<li>You can connect to the virtual machine console from the global zone with <i>zlogin -C zonename</i>;</li>
</ol>
### Example zone
The following example zone is shown twice, once in info format and once in
export (showing the necessary commands for creation). Note that the example
shows setting the `allowed-address` attribute for the network interface -
this does not configure the address within the virtual machine but rather
prevents the use of any other address (L3 protection).
```
hipster# zonecfg -z bhyve info
zonename: bhyve
zonepath: /data/zone/bhyve
brand: bhyve
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
fs-allowed:
fs:
dir: /tank/iso/FreeBSD-11.1-RELEASE-amd64-bootonly.iso
special: /tank/iso/FreeBSD-11.1-RELEASE-amd64-bootonly.iso
raw not specified
type: lofs
options: [ro,nodevices]
net:
address not specified
allowed-address: 10.0.0.112/24
defrouter not specified
physical: bhyve0
device:
match: /dev/zvol/rdsk/tank/hdd/bhyve0
device:
match: /dev/zvol/rdsk/tank/hdd/bhyve1
device:
match: /dev/zvol/rdsk/tank/hdd/bhyve2
attr:
name: vcpus
type: string
value: cpus=16,sockets=2,cores=4,threads=2
attr:
name: ram
type: string
value: 4G
attr:
name: bootrom
type: string
value: BHYVE_DEBUG
attr:
name: console
type: string
value: socket,/tmp/vm.com1
attr:
name: hostbridge
type: string
value: amd
attr:
name: bootdisk
type: string
value: tank/hdd/bhyve0
attr:
name: disk
type: string
value: tank/hdd/bhyve1
attr:
name: disk1
type: string
value: tank/hdd/bhyve2,serial=1234
attr:
name: cdrom
type: string
value: /tank/iso/FreeBSD-11.1-RELEASE-amd64-bootonly.iso
attr:
name: vnc
type: string
value: unix=/tmp/vm.vnc,wait
```
```
hipster# zonecfg -z bhyve export
create -b
set zonepath=/data/zone/bhyve
set brand=bhyve
set autoboot=false
set ip-type=exclusive
add fs
set dir=/tank/iso/debian-9.4.0-amd64-netinst.iso
set special=/tank/iso/debian-9.4.0-amd64-netinst.iso
set type=lofs
add options ro
add options nodevices
end
add net
set allowed-address=10.0.0.112/24
set physical=bhyve0
end
add device
set match=/dev/zvol/rdsk/tank/hdd/bhyve0
end
add device
set match=/dev/zvol/rdsk/tank/hdd/bhyve1
end
add device
set match=/dev/zvol/rdsk/tank/hdd/bhyve2
end
add attr
set name=vcpus
set type=string
set value=cpus=16,sockets=2,cores=4,threads=2
end
add attr
set name=ram
set type=string
set value=4G
end
add attr
set name=bootrom
set type=string
set value=BHYVE_DEBUG
end
add attr
set name=console
set type=string
set value=socket,/tmp/vm.com1
end
add attr
set name=hostbridge
set type=string
set value=amd
end
add attr
set name=bootdisk
set type=string
set value=tank/hdd/bhyve0
end
add attr
set name=disk
set type=string
set value=tank/hdd/bhyve1
end
add attr
set name=disk1
set type=string
set value=tank/hdd/bhyve2,serial=1234
end
add attr
set name=cdrom
set type=string
set value=/tank/iso/debian-9.4.0-amd64-netinst.iso
end
add attr
set name=vnc
set type=string
set value=unix=/tmp/vm.vnc,wait
end
```

54
doc/pkg5_docs/boot_environments.txt Normal file
View file

@ -0,0 +1,54 @@
As of 2010/03, we always create new boot environments
when doing an image-update, and never do so for
install, instead failing the operation should an action
tagged w/ reboot-needed be affected. We also always
create new boot environments for change-variant and
change-facet operations, and never for fix.
This needs work.
Instead, I'd like to propose the following:
1) We support the creation of a new optional backup
boot environment before executing the operation
on the live image. Non-live images can easily
use snapshots for this; this option would have
no effect in the case that the operation creates
a new BE.
2) All image-modifying operations on a live image
would create a new boot environment only if needed.
This behavior could be altered via command line flags
to always create a new boot environment or to never
create one, failing instead if the operation needs
one.
pkg install
pkg fix
would both get the following new options:
[--backup-be]
[--backup-be-name name]
[--no-new-be | --always-new-be]
[--be-name name]
pkg image-update
pkg change-facet
pkg change-variant
would all get the following new options:
[--backup-be]
[--backup-be-name name]
[--no-new-be | --always-new-be]
So, to retain the current behavior,
invoke pkg install --no-new-be ....
invoke pkg image-update --always-new-be ...
pkg change-facet --always-nes-be ...
pkg change-variant --always-new-be ...
- Bart

43
doc/pkg5_docs/case-outline.txt Normal file
View file

@ -0,0 +1,43 @@
PSARC/2008/190
pkg(5): image packaging system
CASE OUTLINE
- introduction and overview
- multiplatform aspects
- client use
- pkg states
- developer / operator use
- transaction states
- security model
- pkg
- use of SSL
- relationship with libbe, ZFS
- pkgsend / pkgrecv
- authentication
- pkg.depotd
- reverse proxy
- horizontal use
- image relationships
- partial images
- user images, unlinked
- user images, linked
- depot relationships
- client to depot
- depot to depot
- protocol / network format
- client side REST API
- publication side REST API
- on-disk format
- repo format
- image format
- actions
- smf(5) configuration transition
- app configuration
- migration and compatibility

1177
doc/pkg5_docs/catalog-v1.txt Normal file
View file

File diff suppressed because it is too large Load diff

105
doc/pkg5_docs/catalog.txt Normal file
View file

@ -0,0 +1,105 @@
pkg
CATALOG AND INVENTORY OPERATIONS
[Suspended] Premise 1: From the client's perspective, the catalog is just
another package. (It's only special in that its contents drive
automatic package operations.)
Because we pull catalogs from multiple repositories, managing N
packages for N repositories seems overcomplicated.
Premise 2: From the server perspective, the catalog is a file whose
contents are updated as a result of the set of packaging operations that
have occurred within the server's repository. (No operation ever
affects the catalog directly.)
1. Server catalog
The primary content of the catalog is the relationships between
packages. We've already said that there are packages that depend on
other packages and packages that incorporate other packages. These
latter packages, the base- and stack-style packages, are really packages
that, in addition to listing a set of packages, also freeze those
packages, such that it takes a relaxation of the base/stack package for
the client to modify the incorporated packages.
The detailed manifest for each package contains the dependencies,
contents, and actions. But, as it seems that the set of incorporation
relationships are of particular interest for reporting and determining
package retrieval ordering, we place incorporation dependencies in the
catalog as well.
It's reasonably clear that the package repository needs to have
observations of package popularity, so that the most aged packages can
be purged, once unneeded.
2. Client catalog and client inventory
The client catalog is a compilation of package sourcing information from
a collection of repositories, in the form of one server catalog per
repository. Its purpose is to advertise what package versions are
"available", which means that it is the union of the repositories it
sources.
In contrast, the client inventory is the set of packages currently in an
installation state on the system, stored in some representation. One
form of this representation will be in the form of a server catalog.
The inventory is stored under
/var/pkg/db
If some instance of which@2.16 is installed, we would have a directory
/var/pkg/db/pkg/[publisher?]/which/2.16
publisher
manifest
state
XXX Can we have two packages with the same name, but from different
publishers?
An open question is whether we store one or more reverse indices in the
inventory. We could have indices into basenames, pathnames, and
contents, which would look something like:
/var/pkg/db/index/
{basename,path}/
[escaped basename or full path]/
named links to content files
content/
[SHA1 hash of contents] -> package directory
Unfortunately, since FILE_MAX < PATH_MAX, the escaped representation for
full paths is insufficient, and we might need to provide a hash-based
directory name, and a metadata file with the full path.
2.1. Multiple catalogs
If a package is available from two or more repositories, we have a
choice: we can prefer a repository (as we would for a request for a
first installation of a package), or we can store both catalogue
entries.
In a directory-based approach, we might have
/var/pkg/catalog/repository_publisher/category/package/versions
with example
/var/pkg/catalog/pkg.opensolaris.org/application/which/2.16
/var/pkg/catalog/blueslugs.com/application/which/2.16
In a file-based approach, we would have
/var/pkg/catalog/[escaped_catalog_URL]
...
XXX Investigate efficiency of encoding versions as entries in
a file versus encoding as separate files
XXX Are FMRI-level categories actually a help, or is a detailed,
hierarchical namespace going to get in the way?

1408
doc/pkg5_docs/client_api_versions.txt Normal file
View file

File diff suppressed because it is too large Load diff

19
doc/pkg5_docs/deduction.txt Normal file
View file

@ -0,0 +1,19 @@
pkg
CLIENT DEDUCTION AND SELF-REPAIR
In the case of a partially installed system, a system installed by a
previous packaging regime, or a system that has been somehow damaged,
it is possible for the pkg system to identify the current system's
state via a deductive process.
That is, the client can examine an image's contents and determine, by
interrogation, the precise version of each package installed in the
image. If a specific file is missing, then the client can move the
package from (INSTALLED, .) to (INSTALLED, damaged) and record the
operation it would take to restore the package's state (via restoring
the contents).
XXX Most efficient algorithm for the minimum set of queries to
identify the specific version of a package an image contains?

44
doc/pkg5_docs/depot.rst Normal file
View file

@ -0,0 +1,44 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
pkg
DEPOT
1. Summary
This document describes the architecture of a pkg(5) depot server as
implemented in pkg.depotd(1m). This includes: an overview of the depot
server's filesystem layout, operations provided by the depot server,
the interfaces provided by the depot server, and a description of how
clients communicate with the depot server for each operation.
XXX Overview moved to ``guide-implementation-depot.rst``.
XXX Retrieval operations moved to ``guide-retrieval-protocol.rst``.
XXX Publication operations moved to ``guide-publication-protocol.rst``.
XXX Layout moved to ``guide-repository-format.rst``.
XXX Operations moved to ``guide-publication-protocol.rst``.

468
doc/pkg5_docs/depot.txt Normal file
View file

@ -0,0 +1,468 @@
pkg
DEPOT
1. Summary
This document describes the architecture of a pkg(5) depot server as
implemented in pkg.depotd(1m). This includes: an overview of the depot
server's filesystem layout, operations provided by the depot server,
the interfaces provided by the depot server, and a description of how
clients communicate with the depot server for each operation.
NOTE: The image packaging system is under development. Changes to
interfaces may occur as part of the architectural review process, as
shortcomings are identified, and as new features are introduced.
Questions about planned or possible change should be asked on
pkg-discuss.
2. Discussion
A pkg(5) depot server provides a way for clients to interact with package
content and metadata contained within a pkg(5) repository, and with the
depot server itself. It accomplishes this by providing an HTTP-based
interface suitable for usage by pkg(1) clients and web user agents.
2.1. Filesystem Layout
The types of information that the depot server stores and/or retrieves can
be categorized as follows:
- depot data
This includes: configuration data, presentation content (such as
web page templates), publishing data (e.g. in-flight transactions),
and temporary data (e.g. the feed cache).
- repository data
This includes: catalog information, package content (files), package
metadata (manifests), and search data.
2.1.1. Layout
The depot server uses the following 'root' directory structures for the
storage and retrieval of depot and repository data:
- repo_dir (depot and repository data)
cfg_cache (depot data)
A file containing the cached configuration information for the
depot server.
catalog/ (repository data)
This directory contains the repository catalog and its related
metadata.
file/ (repository data)
This directory contains the file content of packages in the
repository.
Files are stored using a two-level path fragment, derived from the
SHA1-hash of a file's content, assumed to have at least 8 distinct
characters.
Example:
00/
0023bb/
000023bb53fdc7bcf35e62b7b0b353a56d36a504
index/ (repository data)
This directory contains the search indices for the repository.
pkg/ (repository data)
This directory contains the metadata (manifests) for the
repository's packages.
The manifests for each package are stored in a directory with the
same name as the package stem using a URL-encoded filename.
Example:
entire/
0.5.11%2C5.11-0.86%3A20080422T234219Z
trans/ (depot data)
This directory contains in-flight transactions for packages that
are waiting for the publication process to complete so that they
can be added to the repository's catalog.
Each transaction is stored in a directory named after the pending
transaction id and contains the manifest waiting for publication
to finish stored with the filename of 'manifest'.
Example:
1229379580_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081215T221940Z/
manifest
updatelog/ (repository data)
This directory contains metadata detailing changes to the repository
by publishing operations.
- content_root (depot data)
web/
This directory contains all of the web presentation content for the
depot.
2.2. Operations
When communicating with the depot server via HTTP, operations are presented
via a URL-based mechanism that allows each to be versioned so that changes
in protocol can be hidden from older clients as the interfaces to operations
evolve.
Operations made available by a pkg.depotd(5) server can be accessed via GET
or POST, as appropriate for each operation, via a URL such as the following:
http://pkg.opensolaris.org/release/manifest/0/SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z
The above example can be broken down into four basic components:
publisher_origin_url - http://pkg.opensolaris.org/release/
operation_name - manifest
protocol_version - 0
operation_arguments - SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z
Each of these components can be described as follows:
publisher_origin_url - A URL that can be used to access a depot
server's repository.
operation_name - The name of the operation that the client is
wanting to initiate.
protocol_version - An integer value representing the version of
the operation's protocol spoken by the client.
operation_arguments - String data (such as a package FMRI) that is
parsed and then used to determine what
resource(s) will be used to perform an
operation. Some operations expect arguments
or data to be passed via POST-based form data,
headers, or the request body instead.
2.2.1. Operation Types
Each operation that the depot server provides is either designed to interact
with a pkg(5) repository, or with the depot server itself. These operations
can be categorized as follows:
- content
These operations are read-only, and retrieve file data that comprises
the content of a package in a repository.
- depot
These operations are read-only, and permit retrieval of: the list of
operations that the depot server currently provides (including protocol
version and pkg(5) software version), statistics information, and other
depot information.
- metadata
These operations are read-only, and retrieve metadata related to a
package FMRI, such as its name, version, etc. stored in a repository's
catalog.
- publishing
These operations alter a repository's catalog, package metadata, and
allow storage of package content.
2.2.2. Modes
Which types of operations are available is dependent on which mode the depot
server is currently operating in:
- default
In default mode, the depot server allows content, depot, metadata,
and publishing operations.
- readonly
In readonly mode, the depot server allows content, depot, and
metadata operations.
- mirror
In mirror mode, the depot server allows content and depot
operations.
2.2.3. Content Operations
The pkg.depotd(5) server provides the following operations for retrieving
package content:
- file
Version 0:
A GET operation that retrieves the contents of a file, belonging to a
package, using a SHA-1 hash of the file's content.
Example:
URL:
http://pkg.opensolaris.org/release/file/0/
a00030db8b91f85d0b7144d0d4ef241a3f1ae28f
Expects:
A SHA-1 hash of the file's content belonging to a package in the
request path.
Returns:
The contents of the file, compressed using the gzip compression
algorithm.
2.2.2. Depot Operations
- versions
Version 0:
A GET operation that retrieves text data representing what operations
are supported and version information about the depot server.
Example:
URL:
http://pkg.opensolaris.org/versions/0/
Expects:
Nothing
Returns:
text/plain data containing the version of the pkg(5) software that
the depot is based upon, a list of the operations currently
supported, and the protocol version supported for each
operation.
Sample Output:
pkg-server bfc04991436e
info 0
search 0
versions 0
catalog 0
manifest 0
add 0
file 0
abandon 0
close 0
open 0
2.2.2. Meta-data Operations
- catalog
Version 0:
A GET operation that retrieves a text/plain datastream
representing a complete catalog or an incremental update to an
existing one as requested.
Example:
URL:
http://pkg.opensolaris.org/catalog/0/
Expects:
Nothing or the following headers:
If-Modified-Since: {ISO 8601 formatted date and time in UTC}
Returns:
Either the contents of a pkg(5) catalog file, or the entries
that were added since the specified date as they are found
in the catalog file, separated by newlines.
- info
Version 0:
A GET operation that retrieves a text/plain description of a
package and its licensing information specified by the provided
FMRI.
Example:
URL:
http://pkg.opensolaris.org/info/0/entire@0.5.11,5.11-0.101:20081119T235706Z
Expects:
A URL-encoded pkg(5) FMRI, excluding the 'pkg:/' scheme prefix
and publisher information, and including the full version
information.
Returns:
A text/plain representation of the specified package and its
licensing information.
Sample Output:
Name: entire
Summary: entire incorporation
Publisher: Unknown
Version: 0.5.11
Build Release: 5.11
Branch: 0.101
Packaging Date: Wed Nov 19 23:57:06 2008
Size: 0.00 B
FMRI: pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z
License:
- manifest
Version 0:
A GET operation that retrieves the contents of the manifest file for
a package specified by the provided FMRI.
Example:
URL:
http://pkg.opensolaris.org/manifest/0/entire@0.5.11,5.11-0.101:20081119T235706Z
Expects:
A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix
and publisher information and including the full version
information.
Returns:
The contents of the package's manifest file.
- p5i
Version 0:
A GET operation that retrieves an application/vnd.pkg5.info
datastream representing publisher and package information.
This is intended for consumption by clients for the purposes
of auto-configuration, metadata management policy determination,
and triggering packaging operations such as installation.
Example:
URL:
http://pkg.opensolaris.org/release/p5i/0/SUNWcs
Expects:
A full or partial URL-encoded pkg(5) FMRI, excluding the
publisher prefix. If the partial or full FMRI is valid, it will
be added to the datastream as is. If it includes the wildcard
character '*', a search of the repository's catalog for matching
entries will be performed and the unique set of resulting
package stems will be added to the datastream. If no match is
found, a 404 error will be raised.
Returns:
Returns a pkg(5) information datastream based on the repository
configuration's publisher information and the provided full or
partial FMRI or matching entries. The Content-Type of the
response is 'application/vnd.pkg5.info'.
- publisher
Version 0:
A GET operation that retrieves an application/vnd.pkg5.info
datastream representing publisher information. This is intended
for consumption by clients for auto-configuration and metadata
management policy determination.
Example:
URL:
http://pkg.opensolaris.org/release/publisher/0
Expects:
Nothing
Returns:
Returns a pkg(5) information datastream based on the repository
configuration's publisher information. The Content-Type of the
response is 'application/vnd.pkg5.info'.
- search
Version 0:
A GET operation that retrieves a text/plain list of packages with
metadata that matches the specified criteria.
Example:
URL:
http://pkg.opensolaris.org/release/search/0/vim
Expects:
A URL-encoded token representing the search criteria.
Returns:
A text/plain list of matching entries, separated by newlines.
Each entry consists of a set of four space-separated values:
index - what search index the entry was found in
action - what package action the entry is related to
value - the value that the matched the search criteria
package - the fmri of the package that contains the match
Results are streamed to the client as they are found.
Sample Output:
basename pkg:/SUNWvim@7.1.284,5.11-0.101:20081119T230659Z dir usr/share/vim
basename pkg:/SUNWvim@7.1.284,5.11-0.93:20080708T171331Z file usr/bin/vim
2.2.3. Publishing Operations
- add
Version 0:
A POST operation that adds content to an in-flight transaction for
the Transaction ID specified. This could either be file content
for the package or metadata about the package.
This data is not added to the repository for retrieval until a close
operation for the specified Transaction ID is executed.
Example:
URL:
http://pkg.opensolaris.org/add/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
HEADERS:
X-IPkg-SetAttr1: description=Package Name
REQUEST BODY:
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
The file content (if applicable), to be added, in the request
body. Any attributes to be set in the headers in the pattern
of:
X-IPkg-SetAttr{integer}: attr=value
Returns:
Response status of 200 on success; any other status indicates
failure.
- abandon
Version 0:
A GET operation that aborts an in-flight transaction for the
Transaction ID specified. This will discard any data related to
the transaction.
Example:
URL:
http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
Returns:
Response status of 200 on success; any other status indicates
failure.
- close
Version 0:
A GET operation that ends an in-flight transaction for the
Transaction ID specified. If successful, the corresponding package
is added to the repository catalog and is immediately available to
repository users.
Example:
URL:
http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
Returns:
Response status of 200 on success; any other status indicates
failure.
- open
Version 0:
A GET operation that starts an in-flight transaction for the
package FMRI specified.
Example:
URL:
http://pkg.opensolaris.org/open/0/system%2Flibc@0.1-98
Expects:
A URL-encoded pkg(5) FMRI (excluding timestamp).
Returns:
Response status of 200 on success and an identifier for the new
transaction in the 'Transaction-ID' response header; any other
status indicates failure.
3. References

184
doc/pkg5_docs/dev-guide/Makefile Normal file
View file

@ -0,0 +1,184 @@
#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License (the "License").
# You may not use this file except in compliance with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#
#
# Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
#
# A simple Makefile to build the dev-guide.
# Examples:
#
# $ make book (makes the pdf using rst2pdf)
# $ make xmlbook (makes docbook using rst2docbook)
# $ make check (makes pdfs from each book fragment)
# $ make chpt1.pdf (makes a single pdf)
# $ make -e IGNORE_ERRORS=true chpt4.pdf
# $ make clobber
#
#
# Note: in order for the pdf to build, we need to have
# library/python-2/python-imaging-26 installed, due to
# the png logo we use in macros.txt, referenced by
# developer-guide.txt
#
# set this to make the build to ignore errors in RST documents.
IGNORE_ERRORS=
BOOK_FRAGMENTS= developer-guide.txt \
chpt1.txt \
chpt2.txt \
chpt3.txt \
chpt4.txt \
chpt5.txt \
chpt6.txt \
chpt7.txt \
chpt8.txt \
chpt9.txt \
chpt10.txt \
chpt11.txt \
chpt12.txt \
chpt13.txt \
chpt14.txt \
appendix-a.txt \
appendix-b.txt
PROTO=../../proto/dev-guide
DOCTMP=doctmp
DOCTOOLS=doctools
TOOLSPATH=$(DOCTOOLS)/lib/python2.7/site-packages
VENVPATH=$(DOCTOOLS)/venv
PIP=$(VENVPATH)/bin/pip
VIRTUALENV=$(DOCTOOLS)/bin/virtualenv
VENV=PYTHONPATH=$(TOOLSPATH) $(VIRTUALENV)
SED=/usr/gnu/bin/sed
FLOWTABLES=$(VENVPATH)/lib/python2.7/site-packages/rst2pdf/flowables.py
RST2PDF=env LD_LIBRARY_PATH=/usr/lib/libjpeg8-turbo/lib $(VENVPATH)/bin/rst2pdf -s $(STYLE) $(BOOK_OPT)
STYLE=dev-guide.style
# XXX unused at present - unsure if rst2docbook is correct, or whether we want
# to go to the docutils xml, then apply XSLT et al to get to docbook instead.
# For now, $RST2DOCBOOK gets used.
RST2XML=PYTHONPATH=$(TOOLSPATH) $(DOCTOOLS)/bin/rst2xml.py --no-generator --no-file-insertion
RST2DOCBOOK=PYTHONPATH=$(TOOLSPATH) $(DOCTOOLS)/bin/rst2docbook.py \
--no-generator --no-file-insertion --doctype=book
DOCBOOKRST_URL=http://docutils.sourceforge.net/sandbox/oliverr/docbook
DOCBOOKRST_CMD=echo $(DOCBOOKRST_URL) | sed -e 's/http:\/\///g'
DOCBOOKRST_PATH=$(DOCBOOKRST_CMD:sh)
# when building the pdf book, we add specific options here
BOOK_OPT=
EASY_INSTALL=PYTHONPATH=$(TOOLSPATH) /usr/bin/easy_install
GPATCH=/usr/bin/gpatch
WGET=/usr/bin/wget
VERSION_CMD= git log -1 --pretty=format:'%h'
VERSION=$(VERSION_CMD:sh)
DATE_CMD=date +%F:%R:%S-%Z
DATE=$(DATE_CMD:sh)
VERSTAMP=$(DATE) $(VERSION)
all: install
install: venv proto book
# pull down a local copy of rst2pdf and rst2docbook.py
tools: venv proto
venv: $(VENVPATH)
$(VENVPATH): $(VIRTUALENV)
$(VENV) $(VENVPATH)
$(PIP) install rst2pdf
# https://github.com/rst2pdf/rst2pdf/pull/774
grep -q '^import reportlab$$' $(FLOWTABLES) || $(SED) -i -e '/^import re$$/aimport reportlab' $(FLOWTABLES)
$(VIRTUALENV): proto
$(EASY_INSTALL) --prefix=$(DOCTOOLS) virtualenv
proto: doctmp doctools
mkdir -p $(PROTO)
@cp macros.txt $(PROTO)
@echo ".. |version| replace:: $(VERSTAMP)" >> $(PROTO)/macros.txt
doctools:
mkdir -p $(TOOLSPATH)
doctmp:
mkdir $(DOCTMP)
# tries to build all fragments, then builds the book itself
check: $(BOOK_FRAGMENTS:%.txt=$(PROTO)/%.pdf) book
copy: $(BOOK_FRAGMENTS:%.txt=$(PROTO)/%.txt)
book: booktxt
$(MAKE) BOOK_OPT=-b1 $(PROTO)/book.pdf
xmlbook: booktxt
$(MAKE) $(PROTO)/book.xml
booktxt: proto
# Convert bold/italic mentions of "Chapter x" into hyperlinks
# and concatenate into a single file, to render our book
cat $(BOOK_FRAGMENTS) | \
gsed -re 's#\*\*(Chapter [0-9]+)\*\*#`\1`_#g' | \
gsed -re 's#\*(Chapter [0-9]+)\*#`\1`_#g' | \
gsed -re 's#\*\*(Appendix [AB])\*\*#`\1`_#g' | \
gsed -re 's#\*(Appendix [AB])\*#`\1`_#g' > $(PROTO)/book.txt
@# XXX this is ugly, but means we get to reuse the %.pdf & %.xml targets
cp $(PROTO)/book.txt book.txt
$(PROTO)/%.txt: proto
cp $*.txt $(PROTO)
@cat $(PROTO)/macros.txt >> $(PROTO)/$*.txt
$(PROTO)/%.pdf: tools proto $(PROTO)/%.txt
@print "creating $(PROTO)/$*.pdf"
$(RST2PDF) -o $@ $(PROTO)/$*.txt 2> $(DOCTMP)/$*.rst-output.txt
@if [ -s $(DOCTMP)/$*.rst-output.txt ]; then \
print "Errors/warnings found in $*.txt"; \
cat $(DOCTMP)/$*.rst-output.txt; \
if [ -z "$(IGNORE_ERRORS)" ]; then \
rm $(DOCTMP)/$*.rst-output.txt; \
exit 1;\
fi; \
fi;
$(PROTO)/%.xml: tools proto $(PROTO)/%.txt
@print "creating $(PROTO)/$*.xml"
$(RST2DOCBOOK) $(PROTO)/$*.txt $@
# convenience targets to build a single fragment
%.txt: $(PROTO)/%.txt
%.pdf: $(PROTO)/%.txt $(PROTO)/%.pdf
clean:
rm -rf $(PROTO) $(DOCTMP)
clobber: clean
rm -rf $(DOCTOOLS)

132
doc/pkg5_docs/dev-guide/appendix-a.txt Normal file
View file

@ -0,0 +1,132 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Appendix A
----------
Classifying Packages
....................
The following are defined values for the package attribute
``info.classification`` with scheme ``org.opensolaris.category.2008``, used by
the Package Manager GUI to display possible packages. A typical entry
as used in a package manifest might be::
set name=info.classification value=\
"org.opensolaris.category.2008:System/Administration and Configuration"
Note that category and subcategory are separated by a "/".
As usual, spaces in the attribute value require quoting.
Defined categories and subcategories for values are:
Meta Packages
- Builds
- Releases
- Developer Tools
- AMP Stack
- Office Tools
Applications
- Accessories
- Configuration and Preferences
- Games
- Graphics and Imaging
- Internet
- Office
- Panels and Applets
- Plug-ins and Run-times
- Sound and Video
- System Utilities
- Universal Access
Desktop (GNOME)
- Documentation
- File Managers
- Libraries
- Localizations
- Scripts
- Sessions
- Theming
- Trusted Extensions
- Window Managers
Development
- C
- C++
- Databases
- Distribution Tools
- Editors
- Fortran
- GNOME and GTK+
- GNU
- High Performance Computing
- Integrated Development Environments
- Java
- Objective C
- Observability
- Other Languages
- PHP
- Perl
- Python
- Ruby
- Source Code Management
- Suites
- System
- X11
Drivers
- Display
- Media
- Networking
- Other Peripherals
- Ports
- Storage
System
- Administration and Configuration
- Core
- Databases
- Enterprise Management
- File System
- Fonts
- Hardware
- Internationalization
- Libraries
- Localizations
- Media
- Multimedia Libraries
- Packaging
- Printing
- Security
- Services
- Shells
- Software Management
- Text Tools
- Trusted
- Virtualization
- X11
Web Services
- Application and Web Servers
- Communications

263
doc/pkg5_docs/dev-guide/appendix-b.txt Normal file
View file

@ -0,0 +1,263 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Appendix B
----------
Converting SVR4 Packages to IPS
...............................
This appendix covers conversion of packages from SVR4 to IPS and highlights some
aspects of the conversion that should be given special attention.
*Chapter 4* goes into detail on how to package software in IPS. Developers
with build environments that currently produce SVR4 packages should convert
their build processes following the example in that chapter, rather
than continuing to build SVR4 packages then converting those packages to IPS.
As with *Chapter 4*, the fundamental steps to packaging any software in IPS
are:
* Generate a package manifest.
* Add necessary metadata to the generated manifest.
* Evaluate dependencies.
* Add any facets or actuators that are needed.
* Check the package with |pkglint|.
* Publish the package.
* Test the package.
These steps remain essentially the same for SVR4 to IPS conversion and we will
not repeat their explanations. There are a few steps that warrant more
detailed explanations, and which are covered in this appendix.
In this appendix, a sample SVR4 package which is similar to
the IPS package created in *Chapter 4* is used.
Generate a Package Manifest
~~~~~~~~~~~~~~~~~~~~~~~~~~~
``pkgsend generate`` has support for scanning several different sources in
order to generate manifests. In *Chapter 4*, we used a simple directory as the
source. The |pkgsend| utility can also read SVR4 packages, consulting the
|pkgmap| file in that package, rather than the directory inside the package that
contains the files delivered.
While scanning the ``prototype`` file, |pkgsend| also looks for entries that could
cause problems when converting the package to IPS. The |pkgsend| utility
reports those problems and prints the generated manifest.
In this example, a SVR4 package will be used that has a ``pkginfo`` file::
VENDOR=My Software Inc.
HOTLINE=Please contact your local service provider
PKG=MSFTmypkg
ARCH=i386
DESC=A sample SVR4 package of My Sample Package
CATEGORY=system
NAME=My Sample Package
BASEDIR=/
VERSION=11.11,REV=2011.10.17.14.08
CLASSES=none manpage
PSTAMP=linn20111017132525
MSFT_DATA=Some extra package metadata
.. raw:: pdf
PageBreak
and a corresponding ``prototype`` file::
i pkginfo
i copyright
i postinstall
d none opt 0755 root bin
d none opt/mysoftware 0755 root bin
d none opt/mysoftware/lib 0755 root bin
f none opt/mysoftware/lib/mylib.so.1 0644 root bin
d none opt/mysoftware/bin 0755 root bin
f none opt/mysoftware/bin/mycmd 0755 root bin
d none opt/mysoftware/man 0755 root bin
d none opt/mysoftware/man/man1 0755 root bin
f none opt/mysoftware/man/man1/mycmd.1 0644 root bin
Running |pkgsend| on the SVR4 package built using these files, the following
IPS manifest is generated::
$ pkgsend generate ./MSFTmypkg | pkgfmt
pkgsend generate: ERROR: script present in MSFTmypkg: postinstall
set name=pkg.summary value="My Sample Package"
set name=pkg.description value="A sample SVR4 package of My Sample Package"
set name=pkg.send.convert.msft-data value="Some extra package metadata"
dir path=opt owner=root group=bin mode=0755
dir path=opt/mysoftware owner=root group=bin mode=0755
dir path=opt/mysoftware/bin owner=root group=bin mode=0755
file reloc/opt/mysoftware/bin/mycmd path=opt/mysoftware/bin/mycmd owner=root \
group=bin mode=0755
dir path=opt/mysoftware/lib owner=root group=bin mode=0755
file reloc/opt/mysoftware/lib/mylib.so.1 path=opt/mysoftware/lib/mylib.so.1 \
owner=root group=bin mode=0644
dir path=opt/mysoftware/man owner=root group=bin mode=0755
dir path=opt/mysoftware/man/man1 owner=root group=bin mode=0755
file reloc/opt/mysoftware/man/man1/mycmd.1 \
path=opt/mysoftware/man/man1/mycmd.1 owner=root group=bin mode=0644
legacy pkg=MSFTmypkg arch=i386 category=system \
desc="A sample SVR4 package of My Sample Package" \
hotline="Please contact your local service provider" \
name="My Sample Package" vendor="My Software Inc." \
version=11.11,REV=2011.10.17.14.08
license install/copyright license=MSFTmypkg.copyright
There are several points to note in the output above:
* The ``pkg.summary`` and ``pkg.description`` attributes were automatically
created, using the data from the ``pkginfo`` file.
* A ``set`` action was generated from the extra parameter in ``pkginfo``
file. These are set beneath the ``pkg.send.convert.*`` namespace, with
the intention that developers will use |pkgmogrify| transforms to convert
these to a more appropriate attribute name.
* A ``legacy`` action was generated, using data from the SVR4 pkginfo file.
* A ``license`` action was generated, pointing to the copyright file used
in the SVR4 package.
* An error message was emitted pointing to a scripting operation that can't
be converted.
Checking again, we can see a non-zero return code from ``pkgsend generate``, and
the error message again::
$ pkgsend generate MSFTmypkg > /dev/null
pkgsend generate: ERROR: script present in MSFTmypkg: postinstall
$ echo $?
1
In this case, the package is using a ``postinstall`` script that can't be
converted directly to an IPS equivalent. The script must be manually inspected.
This is the ``postinstall`` script in the package::
#!/usr/bin/sh
catman -M /opt/mysoftware/man
Its effects can easily be replaced by using a ``restart_fmri`` *actuator*
pointing to an existing SMF service, ``svc:/application/man-index:default`` as
described in *Chapter 4*. Also, see *Chapter 9* for further discussion of
actuators.
``pkgsend generate`` will also check for the presence of class-action scripts
and will produce error messages indicating which scripts should be examined.
It is impossible to give examples for every package scripting scenario that a
developer can encounter when converting SVR4 packages to IPS packages. In IPS,
the needed functionality probably can be implemented by using an existing action
type or SMF service.
See *Chapter 3* for details about the *action types* available, and *Chapter 9*
for a discussion on *actuators*.
.. raw:: pdf
PageBreak
Verify the Package
~~~~~~~~~~~~~~~~~~
We'll assume that any additional package metadata needed has been added to the
manifest, and that dependency generation and resolution has been performed as per
*Chapter 4*. Our next step is running |pkglint| on the package.
A common source of errors when converting old SVR4 packages is mismatched
attributes between directories delivered in the SVR4 package and those
delivered by other packages on the system.
In this case, the directory action for ``/opt`` in the sample manifest
has different attributes than those defined by the system packages.
Recall that in *Chapter 3*, we discussed the directory action, stating that
all reference-counted actions must have the same attributes. When trying to
install the version of ``mypkg`` that has been generated so far, an error will
occur::
# pkg install mypkg
Creating Plan /
pkg install: The requested change to the system attempts to install multiple actions
for dir 'opt' with conflicting attributes:
1 package delivers 'dir group=bin mode=0755 owner=root path=opt':
pkg://mypublisher/mypkg@1.0,5.11-0:20111017T020042Z
2 packages deliver 'dir group=sys mode=0755 owner=root path=opt':
pkg://openindiana.org/developer/build/onbld@0.5.11,5.11-2018.0.0.18233:20190417T014131Z
pkg://openindiana.org/SUNWcs@0.5.11,5.11-2018.0.0.18233:20190417T022040Z
These packages may not be installed together. Any non-conflicting set may
be, or the packages must be corrected before they can be installed.
To catch the error before publishing the package, rather than at install-time,
|pkglint| can be used with a reference repository::
$ pkglint -c ./cache -r file:///scratch/oi-repo ./mypkg.mf.res
Lint engine setup...
PHASE ITEMS
4 4292/4292
Starting lint run...
WARNING opensolaris.manifest001.1 Missing attribute 'org.opensolaris.consolidation' in pkg:/mypkg@1.0,5.11
ERROR pkglint.dupaction007 path opt is reference-counted but has different attributes across 3
duplicates: group: bin -> mypkg group: sys -> developer/build/onbld SUNWcs
In particular, notice the error message it produces about /opt having incorrect
attributes. The extra ``ldomsmanager`` package that |pkglint|
reports was in the reference package repository, but was not installed on the
test system, so it did not show up in the errors reported previously by
``pkg install``.
Other Considerations
~~~~~~~~~~~~~~~~~~~~
While it is possible to install SVR4 packages directly on a system running
IPS, we strongly recommend against this.
Apart from the ``legacy`` action, described in *Chapter 3*, there are no links
between the two packaging systems, and they do not reference package metadata
from each other.
IPS has commands such as ``pkg verify`` which can determine whether packaged
content has been installed correctly. However if another packaging system
legitimately installs packages, or runs install scripts that modify packaged
files from IPS, errors might result.
Commands such as ``pkg fix`` or ``pkg revert`` could overwrite files
that were delivered by a SVR4 package as well as an IPS package, potentially
causing the packaged applications to malfunction.
Similarly, commands such as ``pkg install``, which normally check for duplicate
actions and common attributes on reference-counted actions, could also fail to
detect potential errors when files from a different packaging system conflict.
With these pitfalls in mind, and given the comprehensive package development
tool chain in IPS, developing IPS packages instead of SVR4 packages is
recommended for OpenIndiana.

288
doc/pkg5_docs/dev-guide/chpt1.txt Normal file
View file

@ -0,0 +1,288 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 1
---------
Design Goals and Concepts
.........................
This chapter discusses IPS design goals and concepts, and discusses
some of the implications of those choices.
IPS is designed to eliminate some long-standing issues with previous
software distribution, installation and maintenance mechanisms
that have caused significant problems for users,
developers/maintainers and ISVs.
Principle IPS design concepts and goals include:
* Minimize planned downtime by making software update possible
while machines are in production.
* Minimize unplanned downtime by supporting quick reboot to
known working software configurations.
* Automate, as much as possible, the installation of new software or
updates to existing software.
* Resolve the difficulties with ever-increasing software size and
limited distribution media space.
* Ensure that it is possible to determine whether or not a package is
correctly installed as defined by the author (publisher) of the
package; such a check should not be spoofable.
* Incorporate mechanisms to allow for the easy virtualization of OpenIndiana
at a variety of levels - and zones in particular.
* Reduce the effort required to generate patches/upgrades for existing
systems.
* Allow other software publishers (ISVs and end-users themselves) to
create and publish packages using IPS.
These goals led fairly directly to the following ideas:
* Leverage ZFS snapshot and clone facilities to dynamically create
boot environments on an as-needed basis.
This means that:
* OpenIndiana requires ZFS as the root file system; zone
file systems need to be on ZFS as well.
* Users can create as many boot environments as desired.
* The packaging system can automatically create boot environments
on an as-needed basis, either for backup purposes prior to
modifying the running system, or for installation of a new
version of the OS.
* Eliminate duplicated mechanisms and code used to install, patch
and update the operating system.
This results in several significant changes to the way the operating
system is maintained. In particular:
* All OS software updates and patching are done directly with
IPS.
* Any time a new package is installed, it is already exactly
at the correct version.
* The requirement for unspoofable verification of package installation
has interesting consequences:
* If a package needs to support installation in multiple ways, those ways
must be specified by the developer, so the verification process could
take this into account.
* Scripting is inherently unverifiable since we cannot determine the
intent of the script writer. This, along with other issues
mentioned later, led to the elimination of scripting during
packaging operations.
* There can be no mechanism for the package to edit its own manifest,
since verification is then impossible.
* If the administrator wants to install a package in a manner
incompatible with the original publisher's definition, we should
enable the administrator to easily republish the package he wants
to alter so that the scope of his changes are clear, not lost
across upgrades, and can be verified in the same manner as the
original package.
* The need to avoid size restrictions led to a software repository
model, accessed using several different methods. Different
repository sources can be composited to provide a complete set of
packages, and repositories can be distributed as a single file. In
this manner, no single media is ever required to contain all the
available software. In order to support disconnected/firewalled
operations, tools are provided to copy and merge repositories.
* The desire to enable multiple (possibly competing) software
publishers led us to driving all the packaging metadata into the
packages themselves, so no master database of all packages,
dependencies, etc. exists. A catalog of available packages from a
software publisher is part of the repository for performance
reasons, but it can be regenerated from the data contained in the
packages at will.
Software Self-Assembly
......................
Given the goals and ideas above, IPS introduces the general concept of *software
self-assembly*: Any collection of installed software on a system should be able
to build itself into a working configuration when that system is booted, by the
time the packaging operation completes, or at software runtime.
Software self-assembly eliminates the need for install-time scripting in IPS. The
software is responsible for its own configuration rather than relying on the
packaging system to perform that configuration on behalf of the software.
Software self-assembly also enables the packaging system to safely operate on
alternate images, such as boot environments that are not currently booted, or
offline zone roots. In addition, since the self-assembly is performed only on
the running image, the package developer does not need to cope with
cross-version or cross-architecture run-time contexts.
There are obviously some aspects of preparing an operating system image that
must be done before boot, and IPS manages this transparently. These items
include updating boot blocks, preparing a boot archive (ramdisk), and on some
architectures, managing the menu of boot choices.
Several idioms are employed to facilitate software self-assembly:
* **Actions**
*Actions* are the atomic units of software delivery in IPS. Each action
delivers a single software object - either a file system object, such as a
*file*, *directory* or *link*, or a more complex software construct, such
as a *user*, *group* or *driver*. These more complex action types,
previously handled by SVR4 class action scripts no longer require scripting.
Actions, grouped together into *packages*, can be installed, updated and
removed from both live images as well as offline images.
While IPS allows for the set of known action types to be extended in the
packaging system, during development we have found that the action types
delivered at present are sufficient for all packaged software in
OpenIndiana. It is not expected that package developers will need to
create new action types.
Actions are discussed in more detail in *Chapter 3*.
* **Composition**
Rather than maintaining complex configuration files, that require
extensive scripting in order to update each configuration file during
packaging operations, IPS encourages package authors to deliver fragments
of the complete configuration file.
The packaged application either accesses those fragments directly when
reading its configuration, or the fragments can be assembled into the
complete configuration file before reading it.
A good example of this is the ``/etc/user_attr`` configuration file, used
by OpenIndiana to configure extended attributes for roles and users on
the system.
Special service ``svc:/system/rbac:default`` now can be used to regenerate
``/etc/user_attr`` from the separate files delivered into the directory
``/etc/user_attr.d``. Multiple packages deliver fragments of the complete
configuration. IPS action delivering these files are marked with
``restart_fmri`` attribute wich causes service restart when these
fragments are installed, removed or updated. Now no additional scripting
is required to update this file.
Obviously this requires that the software is written with composition in
mind, which isn't always possible.
* **Actuators & SMF services**
An *actuator* is a tag applied to any *action* delivered by the packaging
system that causes a system change when that action is installed, removed,
or updated.
These changes are typically implemented as SMF services.
We can create SMF services that are responsible for configuring software
directly, or constructing configuration files using data delivered in the
SMF manifest or sourced from files installed on the system.
Since SMF services have a rich syntax to express dependencies, we can
ensure that each service only runs when all of its dependencies have been
met.
Designing Your Package
......................
Many of the good packaging criteria present trade-offs among themselves. It
will often be difficult to satisfy all requirements equally. These criteria are
presented in order of importance; however, this sequence is meant to serve as a
flexible guide depending on the circumstances. Although each of these criteria
is important, it is up to you to optimize these requirements to
produce a good set of packages.
Naming Your Package
~~~~~~~~~~~~~~~~~~~
OpenIndiana uses a hierarchical naming strategy for IPS packages. Wherever
possible, design your package names to fit into the same scheme. Try to keep the
last part of your package name reasonably unique such that ``pkg install
<name>`` doesn't report conflicts.
Optimize for Client-Server Configurations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You should consider the various patterns of software use (client and
server) when laying out packages. Good packaging design divides the
affected files to optimize installation of each configuration
type. For example, for a network protocol implementation, it should be
possible to install the client without necessarily installing the
server. Note that if client and server share implementation
components, a base package containing the shared bits is necessary.
Package by Functional Boundaries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Packages should be self-contained and distinctly identified with a set of
functionality. For example, a package containing ZFS should contain all ZFS
utilities and be limited to only ZFS binaries.
Packages should be organized from a customer's point of view into functional
units.
Package Along License or Royalty Boundaries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Put code that requires royalty payments due to contractual agreements or
that has distinct software license terms in a dedicated package or group
of packages. Do not disperse the code into more packages than
necessary.
Overlap in Packages
~~~~~~~~~~~~~~~~~~~
Packages that overlap (deliver differing content to the same
file system locations, for example) cannot be installed at the same
time. Since this error might not be caught until final planning for
installation, it can provide a poor user experience, though
|pkglint| can help to detect this during the package authoring process.
If the package content must differ, declare an exclude dependency so that
IPS will understand that these packages are not to be installed together.
Sizing Considerations
~~~~~~~~~~~~~~~~~~~~~
A package represents (modulo *facets*, discussed later) a single unit
of software, and is either installed or not installed. Packages that are
always installed together should be combined. Since IPS downloads only
changed files on update, even large packages update quickly if change is
limited.

452
doc/pkg5_docs/dev-guide/chpt10.txt Normal file
View file

@ -0,0 +1,452 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 10
----------
Advanced Update
................
This chapter deals with more complex package update issues, and describes
several features in IPS designed to simplify these problems.
For most update operations, IPS will automatically do exactly what is
needed to install updated packages. There are some cases, however,
that require the developer to provide additional information to IPS.
For performance reasons, the solver works purely on the dependency
information included in packages. Packages whose dependencies indicate
that they can be installed at the same time but whose content conflicts
cause conflict checking to fail in pre-installation.
An example of conflicting content is two packages installing the same
file. If conflict checking fails, the user must try different package
versions and then manually specify acceptable versions.
Ensuring that conflicting packages cannot be installed due to constraining
dependencies is a responsibility of the package developer. As mentioned in
*Chapter 4*, |pkglint| can assist with this task.
Renaming, Merging and Splitting Packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Often, the desired organization of a software component changes, whether because
of mistakes in the original packages, changes in the product or its usage over
time, or changes in the surrounding software environment. Also, sometimes
just the name of a package needs to change. When contemplating such changes,
thought must be given to the customer who is upgrading their system to ensure
that unintended side effects do not occur.
Three types of package reorganization are discussed in this section, in order of
increasingly complex considerations for pkg update:
1. Renaming single packages
2. Merging two packages
3. Splitting a package
Renaming a Single Package
`````````````````````````
Simple renames are straightforward. IPS provides a mechanism
to indicate that a package has been renamed. To rename a package, publish a
new version of the existing package with the following two actions:
* A ``set`` action in the following form::
set name=pkg.renamed value=true
* A ``require`` dependency on the new package
A renamed package cannot deliver contents other than depend or set actions.
The new package **must** ensure that it cannot be installed at the same
time as the original package before the rename. If both packages are
covered by the same incorporation dependency, this is automatic.
If not, the new package must contain an ``optional`` dependency on the old
package at the renamed version. This ensures that the solver will not
select both packages, which would fail conflict checking.
Anyone installing this renamed package will automatically receive the
new named package, since it is a dependency of the old version. If a
renamed package is not depended upon by any other packages, it is
automatically removed from the system. The presence of older software
can cause a number of renamed packages to be shown as ``installed``; when
that older software is removed the renamed packages are automatically
removed as well.
Packages can be renamed multiple times without issue, although this is not
recommended as it can be confusing to users.
Merging Two Packages
````````````````````
Merging packages is straightforward as well. The following two cases are
examples of merging packages:
* One package absorbs another package at the renamed version.
* Two packages are renamed to the same new package name.
One Package Absorbs Another
,,,,,,,,,,,,,,,,,,,,,,,,,,,
Suppose package *A@2* will absorb package *B@3*. Simply
rename package *B* to package *A@2*; remember to include an optional
dependency in *A@2* on *B@3* unless both packages are incorporated so
they update in lockstep as above. A user upgrading *B* to *B@3*
will now get *A* installed, which has absorbed *B*.
Two Packages Are Renamed
,,,,,,,,,,,,,,,,,,,,,,,,
In this case, simply rename both packages to the name of the new merged
package, including two ``optional`` dependencies on the old packages in
the new one if they are not otherwise constrained.
Splitting a Package
```````````````````
When you split a package, rename each resulting new package as described in
`Renaming a Single Package`_. If one of the resulting new packages is not
renamed, the pre-split and post-split versions of that package are not
compatible and might violate dependency logic when the end user tries to update
the package.
Rename the original package, including multiple ``require`` dependencies on all
new packages that resulted from the split. This ensures that any package that
had a dependency on the original package will get all the new pieces.
Some components of the split package can be absorbed into existing packages as a
merge. See `One Package Absorbs Another`_.
Obsoleting Packages
~~~~~~~~~~~~~~~~~~~
Package obsoletion is the mechanism by which packages are emptied of
contents and are removed from the system. Such a package does not
satisfy ``require`` dependencies, so an installed package with a ``require``
dependency on a package that becomes obsolete will prevent update
unless a newer version of the installed package is available that does
not contain the ``require`` dependency.
A package is made obsolete by publishing a new version with no content except
for the following ``set`` action:
::
set name=pkg.obsolete value=true
A package can be made non-obsolete by publishing newer versions.
Users who updated through the obsoletion will lose this package, while those
who did not will not.
Preserving Editable Files During Package Renaming or Path Changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
One common issue with updating packages is the migration of editable files,
either in the file system or between packages. IPS attempts to migrate
editable files that move between packages (for example, as the result of a
rename) if the file is not renamed and the path of the file has not changed.
However, if the path changes, the following must be done for the user's
customizations to be preserved:
If the ``file`` action in the old package does not contain the attribute
``original_name``, that attribute must be added. Set the value to the
original name of the package, followed by a colon and then the path to
the file without a leading '/'. Once this is present on an editable file,
it must not be changed. This value acts as a unique identifier for all
moves going forward so that regardless of the number of versions
skipped on an update, the user's content is properly preserved.
Moving Unpackaged Contents on Directory Removal or Rename
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Normally, unpackaged contents are salvaged when the containing
directory is removed, because the last reference to it disappears.
When a directory changes names, the packaging system treats this
as the removal of the old directory and the creation of a new one.
Any editable files that are still in the directory when the directory
is renamed or removed are salvaged.
If the old directory has unpackaged content such as log files that
should be moved to the new directory, this can be done with the
``salvage-from`` attribute if placed on the new directory.
For example, suppose we want to rename a directory from::
/opt/mydata/log
to::
/opt/yourdata/log
In the same package version that removes the former directory and
introduces the latter directory, include the following attribute on the ``dir``
action that creates ``/opt/yourdata/log``:
::
salvage-from=opt/mydata/log
Any unpackaged contents of any time are migrated to the new location.
The ``salvage-from`` attribute is covered later in this chapter, when
discussing data that should be shared between boot environments.
Delivering Multiple Implementations of a Given Application
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In some cases, it can be desirable to deliver multiple implementations of a
given application, having all implementations available on the system,
but with one implementation set as the *preferred* implementation.
The preferred implementation would have symlinks to its binaries
installed, say, to ``/usr/bin`` for ease of discovery. We would also like to
allow the administrator to change the preferred implementation as required,
without having to add or remove any additional packages.
A good example of this would be where we have several versions of GCC
installed, each in their own package, but would like ``/usr/bin/gcc``
to always point to our preferred version.
IPS uses the concept of *mediated links* for this purpose. A mediated link
is a symbolic link that is controlled by the ``pkg set-mediator`` and
``pkg unset-mediator`` commands, documented in the |pkg| man page.
The ``link`` actions in the packages that deliver different implementations
of that application are said to participate in a *mediation*.
.. raw:: pdf
PageBreak
The following attributes can be set on ``link`` actions to control how
mediated links are delivered:
mediator
Specifies the entry in the mediation namespace shared by all
path names participating in a given mediation group (for example
``python``).
Link mediation can be performed based on ``mediator-version`` and
``mediator-implementation``. All mediated links for a given path name
must specify the same ``mediator``. However, not all mediator versions
and implementations need to provide a link at a given path. If a
mediation does not provide a link, then the link is removed when that
mediation is selected.
A mediator, in combination with a specific version and/or
implementation represents a *mediation* that can be selected for use
by the packaging system.
mediator-version
Specifies the version (expressed as a dot-separated sequence
of non-negative integers) of the interface described by the
``mediator`` attribute. This attribute is required if ``mediator``
is specified and ``mediator-implementation`` is not. A local
system administrator can explicitly set the version to use. The
value specified should generally match the version of the package
delivering the link (for example, ``runtime/python-26`` should use
``mediator-version=2.6``), although this is not required.
mediator-implementation
Specifies the implementation of the mediator for use in addition to
or instead of the ``mediator-version``. Implementation strings are
not considered to be ordered. A string is arbitrarily selected by
|pkg5| if not explicitly specified by a system administrator.
The value can be a string of arbitrary length composed of
alpha-numeric characters and spaces. If the implementation itself can
be or is versioned, then the version should be specified at the
end of the string, after a '@' (expressed as a dot-separated
sequence of non-negative integers). If multiple versions of an
implementation exist, the default behavior is to select the
implementation with the highest version.
If only one instance of an implementation-mediation link at a
particular path is installed on a system, then that one is
chosen automatically. If future links at the path are installed,
the link will not be switched unless a vendor, site, or local
override applies, or if one of the links is version-mediated.
mediator-priority
When resolving conflicts in mediated links, |pkg5| normally
chooses the link with the greatest value of ``mediator-version`` or
based on ``mediator-implementation`` if that is not possible. This
attribute is used to specify an override for the normal conflict
resolution process.
If this attribute is not specified, the default mediator selection
logic is applied.
* If the value is ``vendor``, the link is preferred over those
that do not have a ``mediator-priority`` specified.
* If the value is ``site``, the link is preferred over those that
have a value of ``vendor`` or that do not have a
``mediator-priority`` specified.
A local system administrator can override the selection logic
described above.
Here are two sample manifests that participate in a mediation for the link
``/usr/bin/myapp``::
set name=pkg.fmri value=pkg://test/myapp-impl-1@1.0,5.11:20111021T035233Z
file path=usr/myapp/5.8.4/bin/myapp group=sys mode=0755 owner=root
link path=usr/bin/myapp target=usr/myapp/5.8.4/bin/myapp mediator=myapp mediator-version=5.8.4
::
set name=pkg.fmri value=pkg://test/myapp-impl-2@1.0,5.11:20111021T035239Z
file path=usr/myapp/5.12/bin/myapp group=sys mode=0755 owner=root
link path=usr/bin/myapp target=usr/myapp/5.12/bin/myapp mediator=myapp mediator-version=5.12
.. raw:: pdf
PageBreak
We can install both of these packages to the same image::
$ pkg list myapp-impl-1 myapp-impl-2
NAME (PUBLISHER) VERSION IFO
myapp-impl-1 1.0 i--
myapp-impl-2 1.0 i--
Using the ``pkg mediator`` command, we can see the mediations in use::
$ pkg mediator
MEDIATOR VER. SRC. VERSION IMPL. SRC. IMPLEMENTATION
myapp local 5.12 system
$ ls -al usr/bin/myapp
lrwxrwxrwx 1 root sys 23 Oct 21 16:58 usr/bin/myapp -> usr/myapp/5.12/bin/myapp
We can see which other packages participate in the ``myapp`` mediation using
``pkg search``::
$ pkg search -ro path,target,mediator,mediator-version,pkg.shortfmri ::mediator:myapp
PATH TARGET MEDIATOR MEDIATOR-VERSION PKG.SHORTFMRI
usr/bin/myapp usr/myapp/5.12/bin/myapp myapp 5.12 pkg:/myapp-impl-2@1.0
usr/bin/myapp usr/myapp/5.8.4/bin/myapp myapp 5.8.4 pkg:/myapp-impl-1@1.0
We can also change the mediation as desired::
# pkg set-mediator -V 5.8.4 myapp
Packages to update: 2
Mediators to change: 1
Create boot environment: No
Create backup boot environment: No
PHASE ITEMS
Indexing Packages 2/2
PHASE ACTIONS
Update Phase 1/1
PHASE ITEMS
Image State Update Phase 2/2
Reading Existing Index 8/8
Indexing Packages 2/2
# ls -al usr/bin/myapp
lrwxrwxrwx 1 root sys 24 Oct 21 17:02 usr/bin/myapp -> usr/myapp/5.8.4/bin/myapp
Delivering Directories To Be Shared Across Boot Environments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In general, IPS doesn't support delivery of packaged contents to datasets that
span boot environments (BEs). This is because such shared contents, if updated in
one boot environment, might not meet the definitions for other boot environments.
For example, we could foresee a case where a ``pkg verify`` of packaged content
that was delivered with different attributes by packages in two separate boot
environments, yet shared between them, would result in in errors.
However, some of the unpackaged files (the files stored in the file system that
were not delivered by any IPS package) found in a boot environment must be shared
across boot environments to preserve normal system operation in the
face of multiple boot environments.
Some examples include ``/var/mail``, ``/var/log`` and the like. Customers are
likely to place such data on separate datasets as well, or on remote file
servers. However, creating per-directory datasets would mean that many datasets
would be created per zone, which is not desirable.
The goal can be achieved using a shared dataset, mounted into the BE during boot,
with symbolic links from locations inside the BE pointing into that dataset.
Inside the BE, applications deliver primordial directory structure to a *.migrate*
staging directory.
As noted above, no packaged file content should be shared between boot
environments, furthermore, it is not possible or desirable to share any
file system objects other than files.
Update is supported from older versions of a package that did not share
content. Use a ``salvage-from`` attribute as discussed in
`Moving Unpackaged Contents on Directory Removal or Rename`_ and shown in
the example below.
The package should no longer deliver the old directory.
During boot, a script can be run as part of an SMF method script to move file
content from the *.migrate* directory into the shared dataset. This script is
responsible for recreating the directory structure that it finds under the
*.migrate* directory in the boot environment, and moving file contents from the
*.migrate* directory to the shared dataset.
For example, for a package that previously delivered the action::
dir path=opt/myapplication/logs owner=daemon group=daemon mode=0755
we first create a dataset ``rpool/OPTSHARE`` (which can be used by other shared
content from ``/opt``) This dataset creation could alternatively be done by the
SMF method script during boot::
# zfs create rpool/OPTSHARE
# zfs set mountpoint=/opt/share rpool/OPTSHARE
A package can then deliver a symbolic link from their previously packaged
directory to an as-yet nonexistent target beneath ``/opt/share``::
link path=opt/myapplication/logs target=../../opt/share/myapplication/logs
Packages can now deliver the directory into this *.migrate* area::
dir path=opt/.migrate/myapplication/logs owner=daemon group=daemon \
mode=0755 reboot-needed=true salvage-from=/opt/myapplication/logs
We use the ``salvage-from`` attribute to move files from the old location into
the *.migrate* directory.
We require a ``reboot-needed`` actuator for these directory entries in order to
properly support updates of |Immutable Zones| mentioned in *Chapter 1*, which
boot as far as the ``svc:/milestone/self-assembly-complete:default`` milestone
in read/write mode if self-assembly is required, before rebooting read-only.
See the discussion of ``file-mac-profile`` in the |zonecfg| manual page for more
on |Immutable Zones|.
Our SMF service, on reboot, will then move any salvaged directory content into
the shared dataset, and the symbolic links from ``/opt/myapplication`` point
into that shared dataset.

361
doc/pkg5_docs/dev-guide/chpt11.txt Normal file
View file

@ -0,0 +1,361 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 11
----------
Signing Packages
................
One important consideration in the design of IPS was being able to
validate that the software installed on the customer's machine was
actually as originally specified by the publisher. This ability to
validate the installed system is key for both the customer and the
support engineering staff.
To support this validation, manifests can be signed in IPS with the
signatures becoming part of the manifest. Signatures are represented
as actions like all other manifest content. Since manifests contain
all the package metadata - file permissions, ownership, content
hashes, etc., a ``signature`` action that validates that the manifest has
not be altered since it was published is an important part of system
validation.
The ``signature`` actions form a tree that includes the delivered binaries
such that complete verification of the installed software is possible.
There are other uses for manifest signing beyond validation; signatures
can also be used to indicate approval by other organizations or parties.
For example, the internal QA organization could sign manifests of packages
once it was determined the packages were qualified for production use.
Policy could mandate such approvals prior to installation.
As a result, a useful characteristic for signatures is to be
independent of other signatures in a manifest. Signatures can be
added or removed without invalidating the other signatures
that might be present. This feature also facilitates production
hand offs, with signatures used along the path to indicate completion
along the way. Subsequent steps can optionally remove previous
signatures at any time without ill effect.
``signature`` actions look like this::
signature <hash of certificate> algorithm=<signature algorithm> \
value=<signature value> \
chain="<hashes of certs needed to validate primary certificate>" \
version=<pkg version of signature>
The payload and ``chain`` attributes represent the packaging
hash of the PEM (Privacy Enhanced Mail) files, containing the
x.509 certificates downloadable from the originating repository.
The value is the signed hash of the manifest's message text, prepared
as discussed below. The payload certificate is the certificate which
verifies the value in ``value``.
The other certificates presented needs to form a
certificate path that leads from the payload certificate to the trust
anchors that were established as part of the publisher configuration.
Two types of signature algorithms are currently supported. The first
is the RSA group of signature algorithms; an example is ``rsa-sha256``.
The bit after the dash specifies the hash algorithm to use to change
the message text into a single value the RSA algorithm can use.
The second type of signature algorithm is compute the hash only. This
type of algorithm exists primarily for testing and process
verification purposes and presents the hash as the signature value. A
signature action of this type is indicated by the lack of a payload
certificate hash. This type of signature action is verified if the
image is configured to check signatures. Its presence however does
not count as a signature if signatures are required::
signature algorithm=<hash algorithm> value=<hash> \
version=<pkg version of signature>
Additional metadata can be added to a signature if desired, as with
any other action. Such metadata is protected by that signature.
Policies can be set for the image or for specific publishers. The
policies include ignoring signatures, verifying existing signatures,
requiring signatures, and requiring that specific common names must be
seen in the chain of trust. Other policies might be added in the
future.
Publishing a signed manifest is a two step process:
1. Publish the package unsigned to a repository.
2. Update the package in place, using |pkgsign| to append
a signature action to the manifest in the repository.
This process leaves the package intact, including its timestamp.
This process enables a signature action to be added
by someone other than the publisher without invalidating the original
publisher's signature. For example, the QA department of a company
might want to sign all packages that are installed internally to
indicate they have been approved for use, but not republish the
packages, which would create a new timestamp and invalidate the
signature of the original publisher.
Note that |pkgsign| is the only way to publish a signed package. If one
attempts to publish a package already containing a signature, that
signature is removed and a warning is emitted. The |pkgsign| man page
contains examples of how to use |pkgsign|.
One current restriction to be aware of is that signature actions with variants
are ignored. That means that doing a |pkgmerge| on a pair of manifests will
invalidate any signatures which were previously applied. Signing the package
should be the last step of the package development before the package is tested.
|pkgsign| does not perform all the possible checks for its inputs when signing
packages. This means it's important to check signed packages to ensure that
they can be properly installed after being signed. What follows are some of the
errors that can appear when attempting to install or update a signed package
along with explanations of what the errors mean and how to solve the problem.
Errors Involving Signed Packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A signed package can fail to install or update for reasons that are unique to
signed packages. For example, if a package's signature fails to verify, or if
its chain of trust can't be verified or anchored to a trusted certificate, the
package will fail to install.
When installing signed packages, certain image properties will influence the
checks that are performed on packages. These properties are:
* ``signature-policy``
* ``signature-required-names``
* ``trust-anchor-directory``
See the |pkg| man page for further information about these properties, and their
permitted values.
What follows are some examples of different failure paths and what can
be done to resolve them.
Example 1: Chain Certificate Not Found
``````````````````````````````````````
::
pkg install: The certificate which issued this certificate:
/C=US/ST=California/L=Menlo Park/O=pkg5/CN=cs1_ch1_ta3/emailAddress=cs1_ch1_ta3
could not be found. The issuer is:
/C=US/ST=California/L=Menlo Park/O=pkg5/CN=ch1_ta3/emailAddress=ch1_ta3
The package involved is:pkg://test/example_pkg@1.0,5.11-0:20110919T184152Z
The error shown above happens when a certificate in the chain of trust is
missing or otherwise erroneous.
In this case, there were three certificates in the chain of trust when the
package was signed. It was rooted in the trust anchor, a certificate named
``ta3``. ``ta3`` signed a chain cert named ``ch1_ta3``.
``ch1_ta3`` signed a code signing certificate named ``cs1_ch1_ta3``. When |pkg|
tried to install the package, it was able to locate the code signing
certificate, ``cs1_ch1_ta3``, but it couldn't locate the chain certificate,
``ch1_ta3``, so the chain of trust could not be established.
The most common cause of this problem is failing to provide the right
certificate(s) to the ``-i`` option of |pkgsign|.
Example 2: Authorized Certificate Not Found
```````````````````````````````````````````
::
pkg install: The certificate which issued this certificate:
/C=US/ST=California/L=Menlo Park/O=pkg5/CN=cs1_cs8_ch1_ta3/emailAddress=cs1_cs8_ch1_ta3
could not be found. The issuer is:
/C=US/ST=California/L=Menlo Park/O=pkg5/CN=cs8_ch1_ta3/emailAddress=cs8_ch1_ta3
The package involved is:pkg://test/example_pkg@1.0,5.11-0:20110919T201101Z
The error shown above is similar to the error in *Example 1* but has a different
cause.
In this case, the package was signed using the ``cs1_cs8_ch1_ta3``
certificate, which was signed by the ``cs8_ch1_ta3`` certificate.
The problem is that the ``cs8_ch1_ta3`` certificate wasn't authorized to sign
other certificates. (To be specific, the ``cs8_ch1_ta3`` certificate had the
``basicConstraints`` extension set to ``CA:false`` and marked critical.)
When |pkg| verifies the chain of trust, it doesn't find a certificate which was
allowed to sign the ``cs1_cs8_ch1_ta3`` certificate. Since the chain of trust
can't be verified from the leaf to the root, |pkg| prevents the package from
being installed.
Example 3: Untrusted Self-Signed Certificate
````````````````````````````````````````````
::
pkg install: Chain was rooted in an untrusted self-signed certificate.
The package involved is:pkg://test/example_pkg@1.0,5.11-0:20110919T185335Z
The error shown above happens when a chain of trust ends in a self-signed
certificate which isn't trusted by the system.
When a developer creates a chain of certificates using ``openssl`` for testing,
the root certificate is usually self-signed, since there's little reason to have
an outside company verify a certificate only used for testing.
In a test situation, there are two solutions.
The first is to add the self-signed certificate which is the root of the chain
of trust into ``/etc/certs/CA`` and refresh the ``system/ca-certificates``
service.
This mirrors the likely situation customers will encounter where a production
package is signed with a certificate that's ultimately rooted in a certificate
that's delivered with the operating system as a trust anchor.
The second solution is to approve the self-signed certificate for the publisher
which offers the package for testing by using the ``--approve-ca-cert``
option for the ``set-publisher`` subcommand to |pkg|.
Example 4: Signature Value Does Not Match Expected Value
````````````````````````````````````````````````````````
::
pkg install: A signature in pkg://test/example_pkg@1.0,5.11-0:20110919T195801Z
could not be verified for this reason:
The signature value did not match the expected value. Res: 0
The signature's hash is 0ce15c572961b7a0413b8390c90b7cac18ee9010
The error shown above happens when the value on the signature action could not
be verified using the certificate which the action claims was paired with the
key used to sign the package.
There are two possible causes for an error like this.
The first is that the package has been changed since it was signed. This
is unlikely to happen since |pkgsend| will strip existing signature actions
during publication (since the new timestamp the package will get will invalidate
the old signature) but is possible if the package's manifest has been hand
edited since signing.
The second, and most likely cause, is that the key and certificate used to the
sign the package weren't a matched pair. If the certificate given to the
``-c`` option of |pkgsign| wasn't created with the key given to the
``-k`` option of |pkgsign|, the package is signed, but its signature won't
be verified.
Example 5: Unknown Critical Extension
`````````````````````````````````````
::
pkg install: The certificate whose subject is
/C=US/ST=California/L=Menlo Park/O=pkg5/CN=cs2_ch1_ta3/emailAddress=cs2_ch1_ta3
could not be verified because it uses a critical extension that pkg5 cannot
handle yet. Extension name:issuerAltName
Extension value:<EMPTY>
The error above happens when a certificate in the chain of trust uses a critical
extension which |pkg| doesn't understand.
Until |pkg| learns how to process that critical extension, the only solution is
to regenerate the certificate without the problematic critical extension.
Example 6: Unknown Extension Value
``````````````````````````````````
::
pkg install: The certificate whose subject is
/C=US/ST=California/L=Menlo Park/O=pkg5/CN=cs5_ch1_ta3/emailAddress=cs5_ch1_ta3
could not be verified because it has an extension with a value that pkg(5)
does not understand.
Extension name:keyUsage
Extension value:Encipher Only
The error above is similar to the error in *Example 5* except that the problem is
not with an unfamiliar critical extension but with a value that |pkg| doesn't
understand for an extension which |pkg| does understand.
In this case, |pkg| understands the ``keyUsage`` extension, but doesn't
understand the value ‘``Encipher Only``.’ The error will look the same whether
the extension in question is critical or not.
The solution, until |pkg| learns about the value in question, is to remove the
value from the extension, or remove the extension entirely.
Example 7: Unauthorized Use of Certificate
``````````````````````````````````````````
::
pkg install: The certificate whose subject is
/C=US/ST=California/L=Menlo Park/O=pkg5/CN=ch1_ta3/emailAddress=ch1_ta3
could not be verified because it has been used inappropriately.
The way it is used means that the value for extension keyUsage must include
'DIGITAL SIGNATURE' but the value was 'Certificate Sign, CRL Sign'.
The error above occurs when a certificate has been used for a purpose for which
it was not authorized.
In this case, the certificate ``ch1_ta3`` has been used to sign the package.
It's ``keyUsage`` extension means that it's only valid to use that certificate
to sign other certificates and CRL's.
Example 8: Unexpected Hash Value
````````````````````````````````
::
pkg install: Certificate
/tmp/ips.test.7149/0/image0/var/pkg/publisher/test/certs/0ce15c572961b7a0413b8390c90b7cac18ee9010
has been modified on disk. Its hash value is not what was expected.
The error above means what it says.
The certificate at the provided path is used to verify the package being
installed but the hash of the contents on disk don't match what the signature
action thought they should be.
This indicates that the certificate has been changed since it was last retrieved
from the publisher.
The simple solution is to remove the certificate and allow |pkg| to download
the certificate again.
Example 9: Revoked Certificate
``````````````````````````````
::
pkg install: This certificate was revoked:
/C=US/ST=California/L=Menlo Park/O=pkg5/CN=cs1_ch1_ta4/emailAddress=cs1_ch1_ta4
for this reason: None
The package involved is: pkg://test/example_pkg@1.0,5.11-0:20110919T205539Z
The error above indicates the certificate in question, which was in the chain of
trust for the package to be installed, was revoked by the issuer of the
certificate.

156
doc/pkg5_docs/dev-guide/chpt12.txt Normal file
View file

@ -0,0 +1,156 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 12
----------
Handling Non-Global Zones
.........................
This chapter describes how IPS handles zones and discusses those cases where
package developers should be aware of zones.
Packaging Considerations For Non-Global Zones
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Developing packages which work consistently with zones usually involves little to
no additional work. However, a few situations call for close attention from
developers. When considering zones and packaging there are two questions which
need to be answered:
* Does anything in my package have an interface which crosses the boundary
between the global zone and non-global zones?
* How much of the package should be installed in the non-global zone?
Does The Package Cross the Global, Non-Global Zone boundary?
````````````````````````````````````````````````````````````
If a package delivers both kernel and userland functionality, and both sides
of that interface must be updated accordingly, then the package must be updated
in any zones that contain that package whenever the package in the non-global
zone gets updated.
This can be done using a ``parent`` dependency in the package being developed.
If a single package delivers both sides of the interface, then a ``parent``
dependency on ``feature/package/dependency/self`` will ensure that the global
zone and the non-global zones contain the same version of the package,
preventing version skew across the interface.
The dependency will also ensure that if the package is in a non-global zone,
then it is also present in the global zone.
If the interface spans multiple packages, then the package containing the
non-global zone side of the interface must contain a ``parent`` dependency on
the package which delivers the global zone side of the interface. The
``parent`` dependency is also discussed in *Chapter 6*.
How Much of a Package Should Be Installed in a Non-Global Zone?
```````````````````````````````````````````````````````````````
If the answer to this question is "all of it" (and that's typically the case)
then nothing needs to be done to the package to enable it to function properly.
For consumers of the package, though, it can be reassuring to see that the
package author properly considered zone installation and decided that this
package can function in a zone.
For that reason, developers should explicitly state that their
package functions in both global and non-global zones. This is done by adding
the following action to the manifest::
set name=variant.opensolaris.zone value=global value=nonglobal
If no content in the package can be installed in a non-global zone (for example
a package which only delivers kernel modules or drivers), then the package
should specify that it cannot be installed in a zone. This is done by adding
the following action to the manifest::
set name=variant.opensolaris.zone value=global
If some but not all of the content in the package can be installed in a
non-global zone, then take the following steps:
1. Use the following ``set`` action to state that the package can be installed
in both global and non-global zones:
::
set name=variant.opensolaris.zone value=global value=nonglobal
2. Identify the actions that are only relevant in either the global or
non-global zone. The global-zone-only actions should have the attribute
``variant.opensolaris.zone=global``. Similarly, actions that only apply in
non-global zones should have the attribute
``variant.opensolaris.zone=nonglobal``.
If a package has a ``parent`` dependency or has pieces which are different in
global and non-global zones, it's important to test that the package works as
expected in the non-global zone as well as the global zone. If the package has
a ``parent`` dependency on itself, then the global zone should configure the
repository which delivers the package as one of its origins. The package should
be installed in the global zone first, and then in the non-global zone for
testing.
Troubleshooting Zones
~~~~~~~~~~~~~~~~~~~~~
Occasionally problems might be encountered when trying to install the package in
the non-global zone.
Typically the first steps to take to attack the problem are to ensure that the
following services are online in the global zone:
* ``svc:/application/pkg/zones-proxyd:default``
* ``svc:/application/pkg/system-repository:default``
and that the following service is online in the non-global zone:
* ``svc:/application/pkg/zones-proxy-client:default``
These three services provide publisher configuration to the non-global zone and
a communication channel that the non-global zone can use to make requests to the
repositories assigned to the system publishers served from the global zone.
Remember that you won't be able to update the package in the non-global zone,
since it has a ``parent`` dependency on itself. Initiating the update from the
global zone and allowing the linked image code in |pkg| to update the non-global
zone is the right solution.
Once the package is installed in the non-global zone, testing its functionality
can begin.
If the package does not have a ``parent`` dependency on itself, then it's not
necessary to configure the publisher in the global zone nor install the package
there. Further, updating the package in the global zone will not update it in
the non-global zone, causing potentially unexpected results when testing the
older non-global zone package.
The simplest solution in this situation is to make the publisher available to
the non-global zone and install and update the package from within the zone.
If the zone cannot access the publisher's repositories, then configuring the
publisher in the global zone will allow the ``zones-proxy-client`` and
``system-repository`` services to proxy access to the publisher for the
non-global zone. In that case, it's still best to install and update
the package in the non-global zone.

227
doc/pkg5_docs/dev-guide/chpt13.txt Normal file
View file

@ -0,0 +1,227 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 13
----------
How IPS Features Are Used when Packaging the OpenIndiana OS
..............................................................
This chapter describes how IPS is used to package OpenIndiana, and how
the various dependency types are used to define working package sets for the OS.
We include this chapter to give another concrete example of how IPS can be used
to manage a complex set of software, and talk about some of the IPS features
that were used.
Versioning
~~~~~~~~~~
In *Chapter 3* we discussed the ``pkg.fmri`` attribute, and talked about the
different components of the version field, describing how the version field can
be used to support different models of software development.
This section explains how OpenIndiana uses the version field, and is provided
to give an insight into the reasons why a fine-grained versioning scheme can be
useful. Developers **do not** have to follow the same versioning scheme as
OpenIndiana.
Given a sample package::
pkg://openindiana.org/system/library@0.5.11,5.11-2018.0.0.18233:20190417T022656Z
This is how the version field ``0.5.11,5.11-2018.0.0.18233:20190417T022656Z``
is broken down:
0.5.11
The component version. For packages that are provided by illumos-gate,
this is the OS major.minor version. For packages developed outside,
this is the upstream version. For example, the Apache Web Server in the
package:
``pkg:/web/server/apache-24@2.4.39,5.11-2018.0.0.0:20190406T083404Z``
has the component version 2.4.39.
5.11
This is the build version. This is used to define the OS release that
this package was built for and should always be 5.11 for packages created
for OpenIndiana.
2018.0.0.18233
This is the branch version. OpenIndiana uses the following notation
for the branch version in this release:
* 2018: Major release number. Usually corresponds to the current year
* 0: Release minor number. Can be incremented on significant updates.
* 0: Update number. Usually incremented when a lot of components need rebuilding.
* 18233: Component revision, incremented for each component update.
In this case it refers to illumos-gate commit number.
20190417T022656Z
This is the timestamp, defined when the package was published.
Incorporations
~~~~~~~~~~~~~~
OpenIndiana is delivered by a set of packages, with each group of packages
constrained by an incorporation.
Each incorporation roughly represents the organization that developed each
group of packages, though there are some cross-incorporation dependencies
within the packages themselves. The following is a list of the incorporation
packages in OpenIndiana:
* ``pkg:/consolidation/cde/cde-incorporation``
* ``pkg:/consolidation/dbtg/dbtg-incorporation``
* ``pkg:/consolidation/install/install-incorporation``
* ``pkg:/consolidation/jdmk/jdmk-incorporation``
* ``pkg:/consolidation/man/man-incorporation``
* ``pkg:/consolidation/nspg/nspg-incorporation``
* ``pkg:/consolidation/osnet/osnet-incorporation``
* ``pkg:/consolidation/userland/userland-incorporation``
* ``pkg:/consolidation/X/X-incorporation``
Each of these incorporations includes:
* general package metadata
* ``incorporate`` dependencies, sometimes with ``variant.arch`` variants
to denote dependencies that are specific to a given architecture
* a ``license`` action that ensures that when the incorporation is
installed, a license is displayed
Each of the packages delivered on the system contains a ``require`` dependency
on one of these incorporations.
OpenIndiana also includes a special package called ``entire``.
The ``entire`` constrains dependency on userland-incorporation.
facet.version-lock.*
~~~~~~~~~~~~~~~~~~~~
Some of the incorporations, listed above use ``facet.version-lock.*`` facets,
which were discussed in *Chapter 6*.
For example, looking at the ``pkg:/consolidation/userland/userland-incorporation``
package, we see::
.
.
depend type=incorporate \
fmri=library/python/subversion@1.9.7-2018.0.0.1 \
depend facet.version-lock.library/python/subversion=true
depend type=incorporate \
fmri=library/security/libassuan@2.1.3-2018.0.0.1 \
facet.version-lock.library/security/libassuan=true
depend type=incorporate \
fmri=network/chat/ircii@0.2011.11.15-2018.0.0.1 \
facet.version-lock.network/chat/ircii=true
.
.
etc.
enabling the administrator to allow certain packages to float free from the
constraints of the incorporation package.
Notably, the ``entire`` package also contains version-lock facet, allowing
userland incorporation to be removed. However, this can result in a system
which is difficult to fix, and this package should only be unlocked
on development systems.
.. raw:: pdf
PageBreak
Informational attributes
~~~~~~~~~~~~~~~~~~~~~~~~
The following attributes are not necessary for correct package installation,
but having a shared convention lowers confusion between publishers and
users.
info.classification
See *Chapter 3* under "Set actions", and *Appendix A*.
info.keyword
A list of additional terms that should cause this package to be
returned by a search.
info.maintainer
A human readable string describing the entity providing the
package. For an individual, this string is expected to be their
name, or name and email.
info.maintainer-url
A URL associated with the entity providing the package.
info.upstream
A human readable string describing the entity that creates the
software. For an individual, this string is expected to be
their name, or name and email.
info.upstream-url
A URL associated with the entity that creates the
software delivered within the package.
info.source-url
A URL to the source code bundle, if appropriate, for the package.
info.repository-url
A URL to the source code repository, if appropriate, for the
package.
info.repository-changeset
A changeset ID for the version of the source code contained in
info.repository-url.
OpenIndiana Attributes
~~~~~~~~~~~~~~~~~~~~~~~~~
org.opensolaris.arc-caseid
One or more case identifiers (e.g., PSARC/2008/190) associated with
the ARC case (Architecture Review Committee) or cases associated with the
component delivered by the package.
org.opensolaris.smf.fmri
One or more FMRIs representing SMF services delivered by this
package. These attributes are automatically generated by |pkgdepend|
for packages containing SMF service manifests.
OpenIndiana Tags
~~~~~~~~~~~~~~~~~~~
variant.opensolaris.zone
See *Chapter 12*
Organization Specific Attributes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Organizations wanting to provide a package with additional metadata
or to amend an existing package's metadata (in a repository that
they have control over) must use an organization-specific prefix.
For example, a service organization might introduce
``service.example.com,support-level`` or
``com.example.service,support-level`` to describe a level of support
for a package and its contents.

147
doc/pkg5_docs/dev-guide/chpt14.txt Normal file
View file

@ -0,0 +1,147 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 14
----------
Republishing Packages
.....................
This chapter describes how administrators can modify existing packages for local
conditions.
Occasionally administrators need to override attributes or modify packages they
did not produce. This might be to replace a portion of the package with an
internal implementation, or something as simple as removing binaries not
permitted on systems.
While other packaging systems provide various mechanisms to "force"
installation, IPS focuses instead on making it easy to republish an
existing package with the desired modifications. This makes upgrade
much easier since new versions can be re-published with the same
modifications. It also enables the rest of the packaging system
to function normally since instead of forcing IPS to ignore changes,
packages reflect the correct, installed state of the system.
Of course, running a system with a republished package can cause
issues with the support organization if any connection is suspected
between observed problems and the modified package.
The basic steps are as follows:
1. Use |pkgrecv| to download the package to be re-published in a *raw* format
to a specified directory. All of the files are named by their hash value,
and the manifest is named ``manifest``. Remember to set any required
proxy configuration in the ``http_proxy`` environment variable.
2. Use |pkgmogrify| to modify the manifest in the desired manner. Any
timestamp from the internal package FMRI should be removed to prevent
confusion during publication as it is ignored.
If changes are significant, running the resulting package through |pkglint|,
as shown in *Chapter 4*, is a good idea.
3. Republish the package with |pkgsend|. Note that this republication
will strip the package of any signatures that are present and will
ignore any timestamp specified by ``pkg.fmri``. To prevent a warning
message you might want to remove signature actions in the |pkgmogrify| step.
If the administrator doesn't have permission to publish to the
original source of the package, they can create a repository with
|pkgrepo|, then use ``pkg set-publisher --search-before=<original>``
to have the client look for packages from the new repository before
falling back to the original publisher.
4. Optionally, sign the package using |pkgsign| so that internal processes can
be followed. Packages should be signed before they are installed (even
during testing) to prevent client caching issues.
.. raw:: pdf
PageBreak
Example 1: Change Package Metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here's a simple example, where we change the ``pkg.summary`` field
to be ‘"IPS has lots of features"’ instead of whatever was there originally,
and republish to our new repository::
$ mkdir republish; cd republish
$ pkgrecv -d . --raw -s http://openindiana.org/hipster/ package/pkg
$ cd package* # package name contains a '/', and is url-encoded.
$ cd * # we pulled down just the latest package by default
$ cat > fix-pkg
# change value of pkg.summary
<transform set name=pkg.summary -> edit value '.*' "IPS has lots of features">
# delete any signature actions
<transform signature -> drop>
# remove timestamp from fmri so we get our own
<transform set name=pkg.fmri -> edit value ":20.+" "">
^D
$ pkgmogrify manifest fix-pkg > new-manifest
$ pkgrepo create ./mypkg
$ pkgsend -s ./mypkg publish -d . new-manifest
Example 2: Change Package Publisher
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Another common use case is to republish packages under a new publisher
name.
This can be useful, for example, when consolidating the packages from several
different development teams' repositories into a single repository for
integration testing.
Again, this can be achieved using the steps in *Example 1*, using
``pkgrecv --raw``, running a |pkgmogrify| transform on the resulting
manifest, then republishing the transformed manifest.
A sample transform to change the publisher to "mypublisher" is::
<transform set name=pkg.fmri -> edit value pkg://[^/]+/ pkg://mypublisher/>
Iterating over all packages in the repository can be done with a simple shell
script, that uses the output from ``pkgrecv --newest`` to process only the
newest packages from the repository.
In the script below, we've saved the transform above in a file called
``change-pub.mog``, and want to in republish from ``development-repo``
to a new repository ``mypublisher``, changing the package publisher along the
way::
#!/usr/bin/ksh93
pkgrepo create mypublisher
pkgrepo -s mypublisher set publisher/prefix=mypublisher
mkdir incoming
for package in $(pkgrecv -s ./development-repo --newest); do
pkgrecv -s development-repo -d incoming --raw $package
done
for pdir in incoming/*/* ; do
pkgmogrify $pdir/manifest change-pub.mog > $pdir/manifest.newpub
pkgsend -s mypublisher publish -d $pdir $pdir/manifest.newpub
done
If necessary, we could modify this script to select only certain packages,
make additional changes to the versioning scheme of the packages, and show
progress as it republishes each package, for example.

118
doc/pkg5_docs/dev-guide/chpt2.txt Normal file
View file

@ -0,0 +1,118 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 2
---------
Package Lifecycle
.................
This chapter provides an overview of the software package lifecycle with IPS.
Software packages go through a detailed lifecycle with IPS. Understanding
the various phases of the package lifecycle will help the developer and
administrator optimize their results. The following sections provide a
high-level description of each state in the package lifecycle:
Creation
~~~~~~~~
Packages can be created by anybody. IPS does not impose any particular
software build system or directory hierarchy on the part of package
authors. More detail about package creation is available in *Chapter 4*.
Aspects of package creation are discussed throughout the remaining
chapters of this guide.
Publication
~~~~~~~~~~~
Packages are published to an IPS repository, either via HTTP or
to the file system. If desired, once packages are published they can
converted to a ``.p5p`` package archive file. To access software from an IPS
repository, the repository can be added to the system, using the
``pkg set-publisher`` command, or accessed as a temporary source, using the
``-g`` flag to |pkg|. Examples of package publication are shown in
*Chapter 4*.
Installation
~~~~~~~~~~~~
Packages can be installed on a system, either from an IPS repository,
accessed over http://, https:// or file:// URLs, or installed directly
from a ``.p5p`` package archive. Package installation is described in more
detail in *Chapter 5*.
Updates
~~~~~~~
Updated versions of packages might become available, either
published to an IPS repository, or delivered as a new ``.p5p`` package
archive.
Installed packages can then be brought up to date, either individually,
or as part of an entire system update.
It is important to note that IPS does not use the same concept of
"patching" as the SVR4 packaging system did: all changes to packaged
software are delivered by updated packages.
The packaging system is optimized to install only the changed portions
delivered by an updated package, but essentially, package
updates are performed in much the same way as package installs. Package
updating is described in more detail in *Chapter 5*.
Renaming
~~~~~~~~
During a package's lifecycle, it might be desirable to rename a package.
Often this is done for organizational reasons or to refactor packages.
Examples of package refactoring would be where there is an interest in
combining several packages into a single package, breaking a single
package into multiple smaller packages, or a combination of the two.
IPS gracefully handles actions that move between packages, and has
capabilities to allow old package names to persist on the system,
automatically installing the new packages when a user asks to install
a renamed package. Package renaming is described in more detail in
*Chapter 10*.
Obsoletion
~~~~~~~~~~
Eventually a package might reach the end of its life. A package
publisher might decide that a package will no longer be supported,
and that it will not have any more updates made available. IPS
allows publishers to mark such packages as obsolete.
Obsolete packages can no longer be used as a target for most
dependencies from other packages, and any packages upgraded to an
obsolete version are automatically removed from the system. Package
obsoletion is described in more detail in *Chapter 10*.
Removal
~~~~~~~
Finally, a package can be removed from the system assuming that no other
packages have dependencies on it. Package removal is described in more
detail in *Chapter 5*.

933
doc/pkg5_docs/dev-guide/chpt3.txt Normal file
View file

@ -0,0 +1,933 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 3
---------
Basic Terminology
.................
This chapter defines IPS terms and describes the IPS components.
Image
~~~~~
IPS is designed to install packages in an image. An image is a directory
tree, and can be mounted in a variety of locations as needed. Images
are of three types:
Full
in a full image, all dependencies are resolved within the
image itself and IPS maintains the dependencies in a consistent
manner;
Zone
in a zone image, IPS maintains the zone consistent with its
global zone as defined by dependencies in the packages;
User
not yet fully functional for OpenIndiana.
In general, images are created or cloned by other software (installers,
|beadm|, |zonecfg|, etc) rather than directly by the user.
Package
~~~~~~~
IPS deals with all software installed on a system in the granularity of
packages. Every package is represented by a *fault management resource
identifier* (FMRI), consisting of a publisher, a name, and a version, with
the scheme ‘``pkg``’. For example::
pkg://openindiana.org/system/library@0.5.11,5.11-2018.0.0.18233:20190417T022656Z
Here, ‘``openindiana.org``’ is the publisher, ‘``system/library``’ is the package
name, and ‘``0.5.11,5.11-2018.0.0.18233:20190417T022656Z``’ is the version.
Package names are hierarchical with an arbitrary number of components
separated by forward slash (‘``/``’) characters. Package names form a
single namespace across publishers; packages with the same name and
version but different publishers are assumed to be interchangeable in terms
of external dependencies and interfaces. Package name components are case
sensitive and must start with a letter or number, but can include
underscores (‘``_``’), dashes (‘``-``’), periods (‘``.``’), and plus signs
(‘``+``’) in later positions.
FMRIs can appear and can be referred to in abbreviated form. The scheme
is typically unnecessary, leaving the FMRI to start with either a double
slash (‘``//``’) or a single slash (‘``/``’). When the first slash is
doubled, the first word following the slash is the publisher name. When
there is only a single leading slash, no publisher name is present, and the
package name is considered complete, or ‘rooted’.
Further abbreviation is possible by eliding leading components of package
names. For instance, ``/driver/network/e1000g`` can be reduced to
``network/e1000g``, or even simply ``e1000g``.
Note that such abbreviation mighy cause the packaging client to
complain about ambiguous package names, in which case disambiguation can
always be achieved by specifying the full, rooted name. Typically package
names are chosen to reduce possible ambiguities, even when referred to
solely by their last component. Some trailing components are common,
however; in such cases, the last two components should be unambiguous.
Scripts should generally refer to packages by their full, rooted names.
It is not possible to construct an abbreviated FMRI that contains a
publisher name and only trailing package name components.
The version is also often unnecessary; packages referred to without version
will generally resolve to the latest version of the package that can be
installed. As explained below, versions themselves need not be complete.
FMRIs can also be referred to with patterns, where an asterisk (‘``*``’)
can match any portion of a package name. Thus ``/driver/*/e1000g`` will
expand to ``/driver/network/e1000g``, as will ``/dri*00g``.
Version
```````
A package version consists of four sequences of integer numbers,
separated by punctuation. The elements in the first three sequences
are separated by dots, and the sequences are arbitrarily long.
Leading zeros in version components (e.g. ‘``01.1``’ or ‘``1.01``’) are
forbidden, to allow for unambiguous sorting by package version.
An example version is::
0.5.11,5.11-2018.0.0.18233:20190417T022656Z
The first part is the component version. For components that are
are provided by illumos-gate, this is the OS major.minor version.
For a component with its own development life cycle, this sequence
is the dotted release number, such as ‘``2.4.10``’.
The second part, which if present must follow a comma, is the build
version. OpenIndiana uses this to denote the release of the OS for
which the package was compiled.
The third part, which if present must follow a dash, is the branch version,
providing vendor-specific information. This can be incremented when the
packaging metadata is changed, independently of the component; can contain
a build number; or provide some other information.
The fourth part, which if present must follow a colon, is a timestamp.
It represents when the package was published in the GMT timezone, and is
automatically updated when the package is published.
The package versions are ordered using left-to-right precedence; thus
the timestamp is the least significant part of the version space; the
number immediately after the ‘``@``’ is the most significant.
If required, ``pkg.human-version`` can be used to hold a human-readable
version string, however the versioning scheme described above must
also be present. The human-readable version string is only used for
display purposes, and is documented later in this chapter.
By allowing arbitrary version lengths, IPS can accommodate a variety
of different models for supporting software. Within the confines
of a given component version, a package author can use the build or branch
versions and assign one portion of the versioning scheme to security
updates, another for paid vs. unpaid support updates, another for minor bug
fixes, etc.
A version can also be the token ‘``latest``’, which is substituted for the
latest version known.
We discuss how OpenIndiana implements versioning in *Chapter 13*.
Publisher
~~~~~~~~~
A publisher is an entity that develops and constructs packages. A
publisher name, or prefix, is used to identify this source in a unique
manner. The use of Internet domains or registered trademarks is
encouraged, since it provides a natural namespace partitioning.
Package clients combine all specified sources of packages for a given
publisher when computing packaging solutions. Publisher names can
include upper and lower case letters, numbers, dashes and periods; the same
characters as a valid hostname.
.. raw:: pdf
PageBreak
Action
~~~~~~
Actions are used to define the software that comprises a package; they
define the data needed to create this software component. When creating
packages, the developer expresses the package contents as a set of actions
then saves those to a *package manifest* file.
Actions look like this:
.. parsed-literal::
*action_name* *attribute1*\=\ *value1* *attribute2*\=\ *value2* ...
As a concrete example::
dir path=a/b/c group=sys mode=0755 owner=root
The first field identifies this as a ``dir`` (or directory) action; the
``name=value`` attributes describe the familiar properties of that
directory. In the cases where the action has data associated with it,
such as a file, the action looks like this::
file 11dfc625cf4b266aaa9a77a73c23f5525220a0ef path=etc/release owner=root \
group=sys mode=0444 chash=099953b6a315dc44f33bca742619c636cdac3ed6 \
pkg.csize=139 pkg.size=189 variant.arch=i386
Here the second attribute (without a ``name=`` prefix), called the
payload, is the SHA-1 hash of the file. This attribute can alternatively
appear as a regular attribute with the name ``hash``; if both forms are
present they must have the same value.
Action metadata is freely extensible; additional attributes can be
added to actions as desired. Attribute names cannot include spaces,
quotes, or equals signs (‘``=``’). Attribute values can have all of those,
although values with spaces must be enclosed in single or double quotes. Single
quotes need not be escaped inside of a double-quoted string, and vice
versa, though a quote can be prefixed with a backslash (‘``\``’) so as not
to terminate the quoted string. Backslashes can be escaped with
backslashes. It is recommended that custom attributes use a reverse
domain name or similar unique prefix to prevent accidental namespace
overlap.
Multiple attributes with the same name can be present and are
treated as unordered lists.
Note that manifests are largely created using programs; it is not
expected that that developers produce complete manifests by hand, but
rather create skeletons with the minimal non-redundant information, and
have the rest filled in with tools such as |pkgmogrify| and |pkgdepend|.
Most actions have key attributes; this attribute is what makes this
action unique from all others in the image. For file system
objects, this is the path for that object.
Types of Actions
~~~~~~~~~~~~~~~~
There are currently twelve action types in IPS. The following
sections describe each action type, and the attributes that
define these actions. The action types are detailed in the |pkg5| man
page, and are repeated here for reference.
Each section contains an example action, as it would appear in a manifest
during package creation. Other attributes might be automatically added
to the action during publication.
File Actions
````````````
The ``file`` action is by far the most common action, and represents an
‘ordinary file’. The file action references a payload, and has four
standard attributes:
path
The file system path where the file is installed.
This is a file action's key attribute. These are relative
to the root of the image.
mode
The access permissions (in numeric form) of the
file. These are simple permissions only, not ACLs.
owner
The name of the user that owns the file.
group
The name of the group that owns the file.
The payload is a positional attribute in that it is not
named. It is the first word after the action name. In a published
manifest, it is the SHA-1 hash of the file contents.
If present in a manifest that has yet to be published, it
represents the path where the payload can be found. See
|pkgsend|. The ``hash`` attribute can be used instead of the
positional attribute, should the value include an equals
sign. Both can be used in the same action. However, the
hashes must be identical.
Other attributes include:
preserve
This specifies that the file's contents should
not be overwritten on upgrade if the contents
are determined to have changed since the file
was installed or last upgraded. On initial
installs, if an existing file is found, the file
is salvaged (stored in ``/var/pkg/lost+found``).
* If the value of ``preserve`` is ``renameold``, then the
existing file is renamed with the extension
``.old``, and the new file is put in its place.
* If the value of ``preserve`` is ``renamenew``, then the
existing file is left alone, and the new file is
installed with the extension ``.new``.
* If the value of ``preserve`` is ``legacy``, then this
file is not installed for initial package
installs. On upgrades, any existing file is
renamed with the extension ``.legacy``, and then the
new file is put in its place.
* If the value of ``preserve`` is ``true`` (or a value not
listed above, such as ``strawberry``), then the
existing file is left alone, and the new file is
not installed. Other values with specific meanings might
be added in future, so using ``true`` should be used if
this functionality is required.
overlay
This specifies whether the action allows other
packages to deliver a file at the same location
or whether it delivers a file intended to overlay
another. This functionality is intended for
use with configuration files that do not participate
in any self-assembly (for example,
``/etc/motd``) and that can be safely overwritten.
* If ``overlay`` is not specified, multiple packages
cannot deliver files to the same location.
* If the value of ``overlay`` is ``allow``, one other
package is allowed to deliver a file to the same
location. This value has no effect unless the
``preserve`` attribute is also set.
* If the value of ``overlay`` is ``true``, the file
delivered by the action overwrites any other
action that has specified ``allow``.
Changes to the installed file are preserved based on the
value of the ``preserve`` attribute of the overlaying
file. On removal, the contents of the file are
preserved if the action being overlaid is still
installed, regardless of whether the ``preserve``
attribute was specified. Only one action can
overlay another, and the ``mode``, ``owner``, and ``group``
attributes must match.
original_name
This attribute is used to handle editable
files moving from package to package or
from place to place, or both. The form this
takes is the name of the originating package,
followed by a colon and the original
path to the file. Any file being deleted is
recorded either with its package and path,
or with the value of the ``original_name``
attribute if specified. Any editable file
being installed that has the ``original_name``
attribute set uses the file of that name if
it is deleted as part of the same packaging
operation.
Note that once set, this attribute should never
change even if the package or file are repeatedly renamed;
this will permit upgrade to occur from all previous versions.
revert-tag
This attribute is used to tag editable
files that should be reverted as a set.
Multiple ``revert-tag`` values can be specified
The file reverts to its manifest-defined
state when ``pkg revert`` is invoked
with any of those tags specified. See
|pkg|.
Specific types of file can have additional attributes. For ELF files,
the following attributes are recognized:
elfarch
The architecture of the ELF file. This will is the output of
``uname -p`` on the architecture for which the file is built.
elfbits
This is ``32`` or ``64``.
elfhash
This is the hash of the ‘interesting’ ELF
sections in the file. These are the sections
that are mapped into memory when the binary is loaded.
These are the only sections necessary to consider when
determining whether the executable behavior of two binaries
will differ.
An example ``file`` action is::
file path=usr/bin/pkg owner=root group=bin mode=0755
Directory Actions
`````````````````
The ``dir`` action is like the ``file`` action in that it represents
a file system object, except that it represents a directory
instead of an ordinary file. The ``dir`` action has the same
four standard attributes as the ``file`` action (``path``, ``owner``,
``group`` and ``mode``), and ``path`` is the key attribute.
Directories are reference counted in IPS. When the last
package that either explicitly or implicitly references a
directory no longer does so, that directory is removed. If
that directory contains unpackaged file system objects,
those items are moved into ``/var/pkg/lost+found``.
To move unpackaged contents into a new directory, the following
attribute might be useful:
salvage-from
This names a directory of salvaged items. A
directory with such an attribute inherits on
creation the salvaged directory contents if
they exist.
During installation, |pkg| will check that all instances of a given
directory action on the system have the same owner, group and mode
attributes, and will not install the action if conflicting actions
will exist on the system as a result of the operation.
An example of a ``dir`` action is::
dir path=usr/share/lib owner=root group=sys mode=0755
Link Actions
````````````
The ``link`` action represents a symbolic link. The ``link`` action
has the following standard attributes:
path
The file system path where the symbolic link is
installed. This is a ``link`` action's key attribute.
target
The target of the symbolic link. The file system object
to which the link resolves.
The ``link`` action also takes attributes that allow for multiple
versions or implementations of a given piece of software to be
installed on the system at the same time. Such links are *mediated*,
and allow administrators to easily toggle which links point to which
version or implementation as desired. These *mediated links* are
discussed in *Chapter 10*.
An example of a ``link`` action is::
link path=usr/lib/libpython2.6.so target=libpython2.6.so.1.0
Hardlink Actions
````````````````
The ``hardlink`` action represents a hard link. It has the same
attributes as the link action, and ``path`` is also its key attribute.
An example of a ``hardlink`` action is::
hardlink path=opt/myapplication/hardlink target=foo
Set Actions
```````````
The ``set`` action represents a package-level attribute, or metadata,
such as the package description.
The following attributes are recognized:
name
The name of the attribute.
value
The value given to the attribute.
The ``set`` action can deliver any metadata the package author chooses.
However, there are a number of well-defined attribute names that have
specific meaning to the packaging system.
pkg.fmri
The name and version of the containing package.
info.classification
One or more tokens that a |pkg5| client can use
to classify the package. The value should have
a scheme (such as ``org.opensolaris.category.2008``
or ``org.acm.class.1998``) and the actual
classification, such as ``Applications/Games``,
separated by a colon (‘``:``’). The scheme is
used by the |packagemanager| GUI. A set of
``info.classification`` values is included in
*Appendix A*.
pkg.summary
A brief synopsis of the description. This is
output with ``pkg list -s`` at the end of each
line, as well as in one line of the output of
``pkg info``, so it should be no longer than
sixty characters. It should describe *what* a
package is, and should refrain from repeating
the name or version of the package.
pkg.description
A detailed description of the contents and
functionality of the package, typically a
paragraph or so in length. It should describe
*why* someone might want to install the package.
pkg.obsolete
When ``true``, the package is marked obsolete. An
obsolete package can have no actions other than
more ``set`` actions, and must not be marked renamed.
Package obsoletion is covered in *Chapter 10*
pkg.renamed
When ``true``, the package has been renamed.
There must be one or more ``depend`` actions in
the package as well which point to the package
versions to which this package has been renamed.
A package cannot be marked both renamed and
obsolete, but otherwise can have any number of
``set`` actions. Package renaming is covered in
*Chapter 10*.
pkg.human-version
The version scheme used by IPS is strict, and
does not allow for letters or words in the
``pkg.fmri`` version field. If there is a commonly
used human-readable version available for a given
package, that can be set here, and is displayed
by IPS tools. It does not get used as a basis for
version comparison and cannot be used in place of
the ``pkg.fmri`` version.
Some additional informational attributes, as well as some used by
OpenIndiana are described in *Chapter 13*.
An example of a ``set`` action is::
set name=pkg.summary value="Image Packaging System"
.. raw:: pdf
PageBreak
Driver Actions
``````````````
The driver action represents a device driver. The driver
action does not reference a payload. The driver files themselves
must be installed as ``file`` actions. The following
attributes are recognized (see ``add_drv(1M)`` for more information):
name
The name of the driver. This is usually, but
not always, the file name of the driver
binary. This is the ``driver`` action's key
attribute.
alias
This represents an alias for the driver. A
given driver can have more than one ``alias``
attribute. No special quoting rules are
necessary.
class
This represents a driver class. A given
driver can have more than one ``class`` attribute.
perms
This represents the file system permissions
for the driver's device nodes.
clone_perms
This represents the file system permissions
for the clone driver's minor nodes for this
driver.
policy
This specifies additional security policy for
the device. A given driver can have more than
one ``policy`` attribute, but no minor device
specification can be present in more than one
attribute.
privs
This specifies privileges used by the driver.
A given driver can have more than one ``privs``
attribute.
devlink
This specifies an entry in ``/etc/devlink.tab``.
The value is the exact line to go into the
file, with tabs denoted by ‘``\t``’. See
``devlinks(1M)`` for more information. A given
driver can have more than one ``devlink``
attribute.
An example of a driver action is::
driver name=vgatext \
alias=pciclass,000100 \
alias=pciclass,030000 \
alias=pciclass,030001 \
alias=pnpPNP,900 variant.arch=i386 variant.opensolaris.zone=global
Depend Actions
``````````````
The ``depend`` action represents an inter-package dependency. A package
can depend on another package because the first requires functionality
in the second for the functionality in the first to work, or even to
install. Dependencies are covered in more detail in *Chapter 6*.
The following attributes are recognized:
fmri
The FMRI representing the target of the dependency. This is the
dependency action’s key attribute. The FMRI value must not
include the publisher. The package name is assumed to be
complete (that is, rooted), even if it does not begin with a forward
slash (‘``/``’).
Dependencies of type ``require-any`` can have multiple ``fmri``
attributes. A version is optional on the ``fmri`` value, though
for some types of dependencies, an FMRI with no version has no
meaning.
The FMRI value cannot use asterisks, and cannot use the
``latest`` token for a version.
type
The type of the dependency.
* If the value is ``require``, then the target package
is required and must have a version equal to
or greater than the version specified in the
``fmri`` attribute. If the version is not specified,
any version satisfies the dependency. A
package cannot be installed if any of its
required dependencies cannot be satisfied.
* If the value is ``optional``, then the target, if present, must
be at the specified version level or greater.
* If the value is ``exclude``, then the containing package cannot
be installed if the target is present at the specified
version level or greater. If no version is specified, the
target package cannot be installed concurrently with the
package specifying the dependency.
* If the value is ``incorporate``, then the dependency is
optional, but the version of the target package is
constrained. See *Chapter 6* for a discussion of
constraints and freezing.
* If the value is ``require-any``, then any one of multiple target
packages as specified by multiple ``fmri`` attributes can satisfy
the dependency, following the same rules as the ``require``
dependency type.
* If the value is ``conditional``, the target is required
only if the package defined by the ``predicate`` attribute is present
on the system.
* If the value is ``origin``, the target must, if present,
be at the specified value or better on the image to be modified
prior to installation. If the value of the ``root-image`` attribute
is ``true``, the target must be present on the image rooted at
‘``/``’ in order to install this package.
* If the value is ``group``, the target is required unless the
package is on the image avoid list. Note that obsolete packages
silently satisfy the ``group`` dependency. See the ``avoid``
subcommand in the |pkg| man page.
* If the value is ``parent``, then the dependency is ignored if
the image is not a child image, such as a zone. If the image
is a child image then the target is required to be present
in the parent image. The version matching for a ``parent``
dependency is the same as that used for ``incorporate``
dependencies.
predicate
The FMRI representing the predicate for ``conditional``
dependencies.
root-image
Has an effect only for ``origin`` dependencies as mentioned above.
An example of a ``depend`` action is::
depend fmri=crypto/ca-certificates type=require
License Actions
```````````````
The ``license`` action represents a license or other informational
file associated with the package contents. A package can deliver
licenses, disclaimers, or other guidance to the package installer
through the use of the license action.
The payload of the license action is delivered into the image
metadata directory related to the package, and should only contain
human-readable text data. It should not contain HTML or any
other form of markup. Through attributes, license actions can
indicate to clients that the related payload must be displayed
and/or require acceptance of it. The method of display and/or
acceptance is at the discretion of clients.
The following attributes are recognized:
license
This attribute provides a meaningful description
for the license to assist users in determining
the contents without reading the license text
itself. Some example values include:
* ABC Co. Copyright Notice
* ABC Co. Custom License
* Common Development and Distribution License 1.0 (CDDL)
* GNU General Public License 2.0 (GPL)
* GNU General Public License 2.0 (GPL) Only
* MIT License
* Mozilla Public License 1.1 (MPL)
* Simplified BSD License
Wherever possible, including the version of the
license in the description is recommended as shown
above. The license value must be unique within a package.
must-accept
When ``true``, this license must be accepted by a
user before the related package can be installed
or updated. Omission of this attribute is
equivalent to ``false``. The method of
acceptance (interactive or configuration-based,
for example) is at the discretion of clients.
must-display
When ``true``, the action's payload must be displayed
by clients during packaging operations. Omission of
this value is considered equivalent to ``false``.
This attribute should not be used for copyright
notices, only actual licenses or other material
that must be displayed during operations. The
method of display is at the discretion of
clients.
The ``license`` attribute is the key attribute for the license action.
An example of a ``license`` action is::
license license="Apache v2.0"
Legacy Actions
``````````````
The ``legacy`` action represents package data used by the legacy SVR4
packaging system. The attributes associated with this action are
added into the legacy system’s databases so that the tools
querying those databases can operate as if the legacy package were
actually installed. In particular, this should be sufficient to
convince the legacy system that the package named by the ``pkg``
attribute is installed on the system, so that the package can be used to
satisfy SVR4 dependencies.
The following attributes, named in accordance with the parameters in
|pkginfo|, are recognized:
category
The value for the CATEGORY parameter. The default value
is ``system``.
desc
The value for the DESC parameter.
hotline
The value for the HOTLINE parameter.
name
The value for the NAME parameter. The default value is
‘``none provided``’.
pkg
The abbreviation for the package being installed. The
default value is the name from the FMRI of the package.
vendor
The value for the VENDOR parameter.
version
The value for the VERSION parameter. The default value is
the version from the FMRI of the package.
The ``pkg`` attribute is the key attribute for the legacy action.
An example of a ``legacy`` action is::
legacy pkg=SUNWcsu arch=i386 category=system \
desc="core software for a specific instruction-set architecture" \
hotline="Please contact your local service provider" \
name="Core Solaris, (Usr)" vendor=illumos \
version=11.11,REV=2009.11.11
Signature Actions
`````````````````
Signature actions are used as part of the support for package signing in
IPS. They are covered in detail in *Chapter 11*.
User Actions
````````````
The user action defines a UNIX user as defined in ``/etc/passwd``,
``/etc/shadow``, ``/etc/group``, and ``/etc/ftpd/ftpusers`` files.
Users defined with this action have entries added to the
appropriate files.
The following attributes are recognized:
username
The unique name of the user.
password
The encrypted password of the user. The default
value is ‘``*LK*``’.
uid
The unique numeric ID of the user. The default value
is the first free value under 100.
group
The name of the user's primary group. This must be
found in ``/etc/group``.
gcos-field
The real name of the user, as represented in the GECOS
field in ``/etc/passwd``. The default value is the
value of the ``username`` attribute.
home-dir
The user's home directory. The default value is
‘``/``’.
login-shell
The user's default shell. The default value is
empty.
group-list
Secondary groups to which the user belongs.
See ``group(4)``.
ftpuser
Can be set to ``true`` or ``false``. The default
value of ``true`` indicates that the user is
permitted to login via FTP. See ``ftpusers(4)``.
lastchg
The number of days between January 1, 1970,
and the date that the password was last modified.
The default value is empty.
min
The minimum number of days required between
password changes. This field must be set to 0
or above to enable password aging. The default
value is empty.
max
The maximum number of days the password is
valid. The default value is empty. See ``shadow(4)``.
warn
The number of days before password expires
that the user is warned.
inactive
The number of days of inactivity allowed for
that user. This is counted on a per-machine
basis. The information about the last login
is taken from the machine's lastlog file.
expire
An absolute date expressed as the number of
days since the UNIX Epoch (January 1, 1970).
When this number is reached, the login can no
longer be used. For example, an ``expire`` value
of 13514 specifies a login expiration of
January 1, 2007.
flag
Set to empty.
For more information on the values of these attributes, see
the ``shadow(4)`` man page.
A example of a user action is::
user gcos-field="pkg(5) server UID" group=pkg5srv uid=97 username=pkg5srv
Group Actions
`````````````
The group action defines a UNIX group as defined in
``group(4)``. No support is present for group passwords. Groups
defined with this action initially have no user list. Users
can be added with the user action. The following attributes
are recognized:
groupname
The value for the name of the group.
gid
The group's unique numeric id. The default
value is the first free group under 100.
An example of a group action is::
group groupname=pkg5srv gid=97
Repository
~~~~~~~~~~
A software repository contains packages for one or more publishers.
Repositories can be configured for access in a variety of different
ways: HTTP, HTTPS, file (on local storage or via NFS or SMB) and as a
self-contained package archive file, usually with the ``.p5p`` extension.
Package archives allow for convenient distribution of IPS packages,
and are discussed further in *Chapter 4*.
A repository accessed via HTTP or HTTPS has a server process (|pkg.depotd|)
associated with it; in the case of file repositories, the repository
software runs as part of the accessing client.
Repositories are created with the |pkgrepo| command, and package archives
are created with the |pkgrecv| command.

637
doc/pkg5_docs/dev-guide/chpt4.txt Normal file
View file

@ -0,0 +1,637 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 4
---------
Packaging Software with IPS
...........................
This chapter describes how to package your software with IPS.
Packaging software with IPS is usually straightforward due to
amount of automation that is provided. Automation avoids
repetitive tedium since that seems to be the principle cause
of most packaging bugs.
Publication in IPS consists of the following steps:
1. Generate a package manifest.
2. Add necessary metadata to the generated manifest.
3. Evaluate dependencies.
4. Add any facets or actuators that are needed.
5. Verify the package.
6. Publish the package.
7. Test the package.
Each step is covered in the following sections.
Generate a Package Manifest
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The easiest way to get started is to organize the component files into the same
directory structure that you want on the installed system.
This can be done with ``install`` target in Makefiles, or if the software
you want to package is already in a tarball, unpacking the tarball
into a subdirectory. For many open source software packages that use
``autoconf(1)``, setting the DESTDIR environment variable to point to the
desired prototype area accomplishes this.
Suppose your software consists of a binary, a library and a man page,
and you want to install this software in a directory under ``/opt`` named
``mysoftware``. You should create a directory (named ``proto`` in the
examples) in your build area under which your software appears; e.g::
proto/opt/mysoftware/lib/mylib.so.1
proto/opt/mysoftware/bin/mycmd
proto/opt/mysoftware/man/man1/mycmd.1
Now, let's generate a manifest for this proto area. We pipe it
through |pkgfmt| to format the manifest so that is more readable. Assuming
that the ``proto`` directory is in the current working directory::
$ pkgsend generate proto | pkgfmt > mypkg.p5m.1
.. raw:: pdf
PageBreak
Examining the file, you will see it contains the following lines::
dir path=opt group=bin mode=0755 owner=root
dir path=opt/mysoftware group=bin mode=0755 owner=root
dir path=opt/mysoftware/bin group=bin mode=0755 owner=root
dir path=opt/mysoftware/lib group=bin mode=0755 owner=root
dir path=opt/mysoftware/man group=bin mode=0755 owner=root
dir path=opt/mysoftware/man/man1 group=bin mode=0755 owner=root
file opt/mysoftware/bin/mycmd path=opt/mysoftware/bin/mycmd group=bin \
mode=0755 owner=root
file opt/mysoftware/lib/mylib.so.1 path=opt/mysoftware/lib/mylib.so.1 \
group=bin mode=0644 owner=root
file opt/mysoftware/man/man1/mycmd.1 path=opt/mysoftware/man/man1/mycmd.1 \
group=bin mode=0644 owner=root
The path of the files to be packaged appears twice in the file action:
* The first word after the word ‘``file``’ describes the location
of the file in the proto area.
* The path in the ‘``path=``’ attribute specifies the location
where the file is to be installed.
This double entry enables you to modify the installation location without
modifying the ``proto`` area. This capability can save significant time, for example
if you repackage software that was designed for installation on a different
operating system.
Also, note that ``pkgsend generate`` has applied defaults for directory
owners and groups. In the case of ``/opt``, the defaults are not correct;
we'll just delete that directory, since it's delivered by other packages
already on the system and |pkg| would not install the package if the
attributes of ``/opt`` conflicted with those already on the system.
Add Necessary Metadata to the Generated Manifest
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A package should define the following metadata.
See also "Set Actions" in *Chapter 3*.
* ``pkg.fmri`` defines the name and version of the package as described in
"Package" in *Chapter 3*. A description of OpenIndiana versioning can
be found in *Chapter 13*
* ``pkg.description`` is a description of the contents of the
package.
* ``pkg.summary`` is a one-line synopsis of the description.
* ``variant.arch`` enumerates the architectures for which this package
is suitable. If the entire package can be installed on any
architecture, this can be omitted. Producing packages that have
different components for different architectures is discussed in
*Chapter 7*.
* ``info.classification`` is a grouping scheme used by |packagemanager|,
the IPS GUI. The supported values are shown in *Appendix A*.
In this case, we pick an arbitrary one for our sample package.
In addition, we will add a link action to ``/usr/share/man/index.d`` pointing to
our ``man`` directory, and discuss this link when covering *facets* and
*actuators* later in this chapter.
Rather than modifying the generated manifest directly, we'll use
|pkgmogrify| to edit the generated manifest. A full description
of how |pkgmogrify| can be used to modify package manifests can be
found in *Chapter 8*.
In this example the macro capability is used to define the architecture,
as well as regular expression matching for the directory to elide
from the manifest.
Now we create a small file containing the information we want to
add to the manifest, as well as the transform needed to drop the ``opt``
directory from the manifest::
set name=pkg.fmri value=mypkg@1.0,5.11-0
set name=pkg.summary value="This is our example package"
set name=pkg.description value="This is a full description of \
all the interesting attributes of this example package."
set name=variant.arch value=$(ARCH)
set name=info.classification \
value=org.opensolaris.category.2008:Applications/Accessories
link path=usr/share/man/index.d/mysoftware target=opt/mysoftware/man
<transform dir path=opt$->drop>
Running |pkgmogrify| over ``mypkg.p5m.1`` with the above lines in
a file named mypkg.mog::
$ pkgmogrify -DARCH=`uname -p` mypkg.p5m.1 mypkg.mog | pkgfmt > mypkg.p5m.2
Examining the file we see::
set name=pkg.fmri value=mypkg@1.0,5.11-0
set name=pkg.description \
value="This is a full description of all the interesting attributes of this example package. "
set name=pkg.summary value="This is our example package"
set name=info.classification \
value=org.opensolaris.category.2008:Applications/Accessories
set name=variant.arch value=i386
link path=usr/share/man/index.d/mysoftware target=opt/mysoftware/man
dir path=opt/mysoftware group=bin mode=0755 owner=root
dir path=opt/mysoftware/bin group=bin mode=0755 owner=root
dir path=opt/mysoftware/lib group=bin mode=0755 owner=root
dir path=opt/mysoftware/man group=bin mode=0755 owner=root
dir path=opt/mysoftware/man/man1 group=bin mode=0755 owner=root
file opt/mysoftware/bin/mycmd path=opt/mysoftware/bin/mycmd group=bin \
mode=0755 owner=root
file opt/mysoftware/lib/mylib.so.1 path=opt/mysoftware/lib/mylib.so.1 \
group=bin mode=0644 owner=root
file opt/mysoftware/man/man1/mycmd.1 path=opt/mysoftware/man/man1/mycmd.1 \
group=bin mode=0644 owner=root
link path=usr/share/man/index.d/mysoftware target=../../../../opt/mysoftware/man
Note that the directory action defining ``opt`` has been removed, and the
manifest contents from ``mypkg.mog`` have been added to our package.
Evaluate Dependencies
~~~~~~~~~~~~~~~~~~~~~
Use the |pkgdepend| command to automatically generate dependencies for the
package. The generated depend actions are defined in *Chapter 3* and discussed
further in *Chapter 6*.
Dependency generation is composed of two separate steps:
1. Determine the files on which our software depends.
2. Determine the packages that contain those files.
These steps are referred to as *dependency generation* and
*dependency resolution* and are performed using the ``generate`` and ``resolve``
subcommands of |pkgdepend|, respectively.
.. raw:: pdf
PageBreak
First, we'll generate our dependencies::
$ pkgdepend generate -md proto mypkg.p5m.2 | pkgfmt > mypkg.p5m.3
The ``-m`` option causes |pkgdepend| to include the entire manifest in
its output, and the ``-d`` option passes the ``proto`` directory to the command.
In this new file, we see::
set name=pkg.fmri value=mypkg@1.0,5.11-0
set name=pkg.description \
value="This is a full description of all the interesting attributes of this example package."
set name=pkg.summary value="This is our example package"
set name=info.classification \
value=org.opensolaris.category.2008:Applications/Accessories
set name=variant.arch value=i386
dir path=opt/mysoftware group=bin mode=0755 owner=root
dir path=opt/mysoftware/bin group=bin mode=0755 owner=root
dir path=opt/mysoftware/lib group=bin mode=0755 owner=root
dir path=opt/mysoftware/man group=bin mode=0755 owner=root
dir path=opt/mysoftware/man/man1 group=bin mode=0755 owner=root
file opt/mysoftware/bin/mycmd path=opt/mysoftware/bin/mycmd group=bin \
mode=0755 owner=root
file opt/mysoftware/lib/mylib.so.1 path=opt/mysoftware/lib/mylib.so.1 \
group=bin mode=0644 owner=root
file opt/mysoftware/man/man1/mycmd.1 path=opt/mysoftware/man/man1/mycmd.1 \
group=bin mode=0644 owner=root
link path=usr/share/man/index.d/mysoftware target=../../../../opt/mysoftware/man
depend fmri=__TBD pkg.debug.depend.file=libc.so.1 \
pkg.debug.depend.reason=opt/mysoftware/bin/mycmd \
pkg.debug.depend.type=elf type=require pkg.debug.depend.path=lib \
pkg.debug.depend.path=opt/mysoftware/lib pkg.debug.depend.path=usr/lib
depend fmri=__TBD pkg.debug.depend.file=libc.so.1 \
pkg.debug.depend.reason=opt/mysoftware/lib/mylib.so.1 \
pkg.debug.depend.type=elf type=require pkg.debug.depend.path=lib \
pkg.debug.depend.path=usr/lib
|pkgdepend| has added notations about a dependency on ``libc.so.1`` by both
``mylib.so.1`` and ``mycmd``. Note that the internal dependency between
``mycmd`` and ``mylib.so.1`` is currently silently elided by |pkgdepend|.
Now we need to resolve these dependencies. To resolve dependencies,
|pkgdepend| examines the packages currently installed on the machine used
for building the software. By default, |pkgdepend| puts its output in
``mypkg.p5m.3.res``. Note that this takes a while to run as it loads lots of
information about the system on which it is running. |pkgdepend| will resolve
many packages at once if you want to amortize this time over all packages;
running it on one package at a time is not time efficient.
::
$ pkgdepend resolve -m mypkg.p5m.3
.. raw:: pdf
PageBreak
When this completes, ``mypkg.p5m.3.res`` contains::
set name=pkg.fmri value=mypkg@1.0,5.11-0
set name=pkg.description \
value="This is a full description of all the interesting attributes of this example package."
set name=pkg.summary value="This is our example package"
set name=info.classification \
value=org.opensolaris.category.2008:Applications/Accessories
set name=variant.arch value=i386
dir path=opt/mysoftware group=bin mode=0755 owner=root
dir path=opt/mysoftware/bin group=bin mode=0755 owner=root
dir path=opt/mysoftware/lib group=bin mode=0755 owner=root
dir path=opt/mysoftware/man group=bin mode=0755 owner=root
dir path=opt/mysoftware/man/man1 group=bin mode=0755 owner=root
file opt/mysoftware/bin/mycmd path=opt/mysoftware/bin/mycmd group=bin \
mode=0755 owner=root
file opt/mysoftware/lib/mylib.so.1 path=opt/mysoftware/lib/mylib.so.1 \
group=bin mode=0644 owner=root
file opt/mysoftware/man/man1/mycmd.1 path=opt/mysoftware/man/man1/mycmd.1 \
group=bin mode=0644 owner=root
link path=usr/share/man/index.d/mysoftware target=opt/mysoftware/man
depend fmri=pkg:/system/library@0.5.11,5.11-0.175.0.0.0.2.1 type=require
|pkgdepend| has converted the notation about the file dependency on
``libc.so.1`` to a package dependency on ``pkg:/system/library`` which delivers
that file.
We recommended that developers use |pkgdepend| to generate dependencies,
rather than declaring ``depend`` actions manually. Manual dependencies can
become incorrect or unnecessary as the package contents might change over time.
This could happen, for example, when a file that an application depends on gets
moved to a different package. Any manually declared dependencies on that package
would then be out of date.
Some manually declared dependencies might be necessary if |pkgdepend| is unable
to determine dependencies completely, in which case we recommend that comments
are added to the manifest to explain the nature of each dependency.
Add Any Facets or Actuators That Are Needed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Facets and actuators are discussed in more detail in *Chapter 7* and
*Chapter 9*. Facets allow us to denote actions that are not required but can
be optionally installed. Actuators specify system changes that must occur
when an action in our package is installed, updated, or removed.
Since we are delivering a man page in ``opt/mysoftware/man/man1`` we would
like to add a facet to indicate that documentation is optional.
We would also like an SMF service, ``svc:/application/man-index:default``, to be
restarted when our package is installed, so that our man page is included in the
index. The 'restart_fmri' actuator can perform that task.
The ``man-index`` service looks in ``/usr/share/man/index.d`` for symbolic
links to directories containing man pages, adding the target of each link to
the list of directories it scans, hence our earlier addition of that link to
our man pages. This is a good example of the *self-assembly* idiom that was
discussed in *Chapter 1*, and is used throughout the packaging of the OS
itself.
OpenIndiana ships with a set of |pkgmogrify| transforms that were used to
package the the operating system, in ``/usr/share/pkg/transforms``. These
transforms are discussed in more detail in *Chapter 8*.
The file called ``documentation`` contains the transforms that are closest to
what we need here, though since we're delivering our man page to ``/opt``,
we'll use the ``documentation`` transforms file as a guide, and use the
following transforms instead. These transforms include a regular expression
``opt/.+/man(/.+)?`` which match all paths beneath ``opt`` that contain a
``man`` subdirectory::
<transform dir file link hardlink path=opt/.+/man(/.+)? -> \
default facet.doc.man true>
<transform file path=opt/.+/man(/.+)? -> \
add restart_fmri svc:/application/man-index:default>
We can run our manifest through this transform using::
$ pkgmogrify mypkg.p5m.3.res /tmp/doc-transform | pkgfmt > mypkg.p5m.4.res
which changes the three man-page-related actions in our manifest, from::
dir path=opt/mysoftware/man group=bin mode=0755 owner=root
dir path=opt/mysoftware/man/man1 group=bin mode=0755 owner=root
file opt/mysoftware/man/man1/mycmd.1 path=opt/mysoftware/man/man1/mycmd.1 \
group=bin mode=0644 owner=root
to::
dir path=opt/mysoftware/man owner=root group=bin mode=0755 facet.doc.man=true
dir path=opt/mysoftware/man/man1 owner=root group=bin mode=0755 \
facet.doc.man=true
file opt/mysoftware/man/man1/mycmd.1 path=opt/mysoftware/man/man1/mycmd.1 \
owner=root group=bin mode=0644 \
restart_fmri=svc:/application/man-index:default facet.doc.man=true
For efficiency, we could have included this transform when originally adding
metadata to our package, before running |pkgdepend|.
Verify the Package
~~~~~~~~~~~~~~~~~~
The last thing we need to do before publication is run |pkglint| on our
manifest. This helps us determine whether we've made any errors while writing
the manifest that we'd like to catch before publication. Some of the errors
that |pkglint| can catch are ones also caught either at publication time, or
when a user tries to install a package, but obviously, we'd like to catch
errors as early as possible in the package authoring process.
For example, |pkglint| checks that the package doesn't
deliver files already owned by another package, and that all metadata for
shared, reference-counted actions (such as directories) is consistent across
packages.
There are two modes in which to run |pkglint|:
* Directly on the manifest itself
* On the manifest, also referencing a repository
For developers who want to quickly check the validity of their manifests, using
the first form is usually sufficient. The second form is recommended to be run
*at least once* before publication to a repository.
By referencing a repository, |pkglint| can perform additional checks to ensure
that the package interacts well with other packages in that repository.
The full list of checks that |pkglint| performs can be shown with ``pkglint -L``.
Detailed information on how to enable, disable and bypass particular checks
is given in the |pkglint| man page. It also details how to extend |pkglint| to
run additional checks.
In the case of our test package, we see::
$ pkglint mypkg.p5m.4.res
Lint engine setup...
Starting lint run...
WARNING opensolaris.manifest001.1 Missing attribute
'org.opensolaris.consolidation' in pkg:/mypkg@1.0,5.11-0
WARNING pkglint.action005.1 obsolete dependency check skipped: unable
to find dependency pkg:/system/library@0.5.11-0.168 for
pkg:/mypkg@1.0,5.11-0
These warnings are acceptable for our purposes:
* ``opensolaris.manifest001.1`` is warning us that we haven't declared a tag
that is generally only required for bundled OpenIndiana software, so we
can ignore this warning.
* ``pkglint.action005.1`` is warning us that |pkglint| wasn't able to find a
package called ``pkg:/system/library@0.5.11-0.168`` which we have generated
a dependency on. Since |pkglint| was called with just the manifest file as
an argument, it does not know which repository that package is present in,
hence the warning.
When |pkglint| is run with a ``-r`` flag referencing a repository containing
the package that our test package has a dependency on, we see::
$ pkglint -c ./oi-reference -r http://pkg.openindiana.org/hipster mypkg.p5m.4.res
Lint engine setup...
PHASE ITEMS
4 4292/4292
Starting lint run...
WARNING opensolaris.manifest001.1 Missing attribute 'org.opensolaris.consolidation' in pkg:/mypkg@1.0,5.11-0
$
Publish the Package
~~~~~~~~~~~~~~~~~~~
Now that our package is created, dependencies are added, and it has been
checked for correctness, we can publish the package.
IPS provides three different ways to deliver a package:
* Publish to a local file-based repository
* Publish to a remote HTTP-based repository
* Convert to a ``.p5p`` package archive
Generally, publishing to a file-based repository is sufficient while testing
a package.
If the package needs to be transferred to other machines which cannot access
the package repositories, converting one or more packages to a package archive
can be convenient.
The package can also be published directly to an HTTP repository, hosted
on a machine with a read/write instance of ``svc:/application/pkg/server``
(which in turn runs |pkg.depotd|).
We do not generally recommend this method of publication since there are no
authorization/authentication checks on the incoming package when publishing over
HTTP. Publishing to HTTP repositories can be convenient on secure networks or
when testing the same package across several machines if NFS or SMB access to
the file repository is not possible.
Installing packages over HTTP (or preferably HTTPS) is fine, however.
Local File Repositories
```````````````````````
|pkgrepo| can be used to create and manage repositories. We choose a
location on our system, create a repository, then set the default publisher
for that repository::
$ pkgrepo create /scratch/my-repository
$ pkgrepo -s /scratch/my-repository set publisher/prefix=mypublisher
$ find /scratch/my-repository/
/scratch/my-repository/
/scratch/my-repository/pkg5.repository
We can now use ``pkgsend`` to publish our package, and ``pkgrepo`` to examine
the repository afterwards::
$ pkgsend -s /scratch/my-repository/ publish -d proto mypkg.p5m.4.res
pkg://mypublisher/mypkg@1.0,5.11-0:20111012T034303Z
PUBLISHED
$ pkgrepo -s /scratch/my-repository info
PUBLISHER PACKAGES STATUS UPDATED
mypublisher 1 online 2011-10-12T03:43:04.117536Z
The file repository can then be served over HTTP or HTTPS using |pkg.depotd|
if required.
Package Archives
````````````````
Package archives enable you to distribute groups of packages in a single file.
We can use |pkgrecv| to create package archives from package repositories,
and vice versa.
Package archives can be easily downloaded from an existing website, copied to
a USB key or burned to a DVD for installation in cases where a package
repository is not available.
In the case of our simple file repository above, we can create an archive
from this repository with the following command::
$ pkgrecv -s /scratch/my-repository -a -d myarchive.p5p mypkg
Retrieving packages for publisher mypublisher ...
Retrieving and evaluating 1 package(s)...
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 3/3 0.7/0.7
ARCHIVE FILES STORE (MB)
myarchive.p5p 14/14 0.7/0.7
We can list the newest available packages from a repository using pkgrepo::
$ pkgrepo -s /scratch/my-repository list '*@latest'
PUBLISHER NAME O VERSION
mypublisher mypkg 1.0,5.11-0:20111012T033207Z
This output can be useful when constructing scripts to create archives with
the latest versions of all packages from a given repository.
Temporary repositories or package archives provided with the ``-g`` flag for
``pkg install`` and other package operations cannot be used on systems with
child or parent images (non-global zones have a child/parent relationship with
the global zone) since the system repository does not get temporarily configured
with that publisher information.
Package archives can be set as sources of local publishers in non-global zones,
however.
Test the Package
~~~~~~~~~~~~~~~~
Having published our package, we are interested in seeing whether it has been
packaged properly.
In this example, we ensure that our user has the *Software Installation*
Profile, in order to be able to install packages without root privileges,
then we add the publisher in our repository to the system::
$ sudo su
Password:
# usermod -P 'Software Installation' myuser
Found user in files repository.
UX: usermod: myuser is currently logged in, some changes may not take effect
until next login.
^D
$ pfexec pkg set-publisher -p /scratch/my-repository
pkg set-publisher:
Added publisher(s): mypublisher
You can use ``pkg install`` -nv to see what the install command will do without
making any changes. The following example actually installs the package::
$ pfexec pkg install mypkg
Packages to install: 1
Create boot environment: No
Create backup boot environment: No
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 3/3 0.7/0.7
PHASE ACTIONS
Install Phase 15/15
PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2
PHASE ITEMS
Reading Existing Index 8/8
Indexing Packages 1/1
.. raw:: pdf
PageBreak
We can then examine the software as it was delivered on the system::
$ find /opt/mysoftware/
/opt/mysoftware/
/opt/mysoftware/bin
/opt/mysoftware/bin/mycmd
/opt/mysoftware/lib
/opt/mysoftware/lib/mylib.so.1
/opt/mysoftware/man
/opt/mysoftware/man/man-index
/opt/mysoftware/man/man-index/term.doc
/opt/mysoftware/man/man-index/.index-cache
/opt/mysoftware/man/man-index/term.dic
/opt/mysoftware/man/man-index/term.req
/opt/mysoftware/man/man-index/term.pos
/opt/mysoftware/man/man1
/opt/mysoftware/man/man1/mycmd.1
In addition to the binaries and man page showing up, we can see that the system
has also generated the man page indexes as a result of our actuator restarting
the ``man-index`` service.
We can see that ``pkg info`` shows the metadata that we added to our package::
$ pkg info mypkg
Name: mypkg
Summary: This is our example package
Description: This is a full description of all the interesting attributes of
this example package.
Category: Applications/Accessories
State: Installed
Publisher: mypublisher
Version: 1.0
Build Release: 5.11
Branch: 0
Packaging Date: October 12, 2011 03:43:03 AM
Size: 1.75 MB
FMRI: pkg://mypublisher/mypkg@1.0,5.11-0:20111012T034303Z
We can also see that ``pkg search`` returns hits when querying for files in
our package::
$ pkg search -l mycmd.1
INDEX ACTION VALUE PACKAGE
basename file opt/mysoftware/man/man1/mycmd.1 pkg:/mypkg@1.0-0

237
doc/pkg5_docs/dev-guide/chpt5.txt Normal file
View file

@ -0,0 +1,237 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 5
---------
Installing, Removing, and Updating Software Packages
....................................................
This chapter describes how the IPS client works internally when installing,
updating and removing the software installed in an image.
Understanding basically how |pkg| works will help administrators and developers
better understand the various errors that can occur, and allow them to more
quickly resolve package dependency problems.
How package changes are performed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following steps are executed when |pkg| is invoked to modify the software
installed on the machine:
* Check input for errors
* Determine the system end-state
* Run basic checks
* Run the solver
* Optimize the solver results
* Evaluate actions
* Download content
* Execute actions
* Process actuators
When operating on the global zone, during execution of the steps above, |pkg|
can execute operations on any non-global zones on the machine, for example
to ensure that dependencies are correct between the global and non-global zones,
or to download content or execute actions for non-global zones. *Chapter 12*
has more detail about zones.
In the following sections, we'll describe each of these steps.
Check Input for Errors
``````````````````````
We perform basic error checking on the options presented on the command line.
Determine the System End State
``````````````````````````````
A description of the desired end state of the system is
constructed. In the case of updating all packages in the image this
might be something like *"all the packages currently installed, or
newer versions of them"*. In the case of package removal, it would
be *"all the packages currently installed without this one"*.
IPS tries hard to determine what the user intends this end state to look
like. In some cases, IPS might determine an end state that is not what the
user intended, even though that end state does match what the user requested.
When troubleshooting, it is best to be as *specific* as possible. The
following command is not specific::
# pkg update
If this command fails with a message such as ``No updates available for this
image``, then you might want to try a more specific command such as the
following command::
# pkg update "*@latest"
This command defines the end state more exactly, and can produce more
directed error messages.
Run Basic Checks
````````````````
The desired end state of the system is reviewed to make sure that a
solution appears possible. During this basic review, |pkg| checks that a
plausible version exists of all dependencies, and that desired packages
do not exclude each other.
If an obvious error exists, then |pkg| will print an appropriate error
message and exit.
Run the Solver
``````````````
The solver forms the core of the computation engine used by |pkg5|
to determine the packages that can be installed, updated or removed,
given the constraints in the image and constraints introduced by any
new packages for installation.
This problem is an example of a *Boolean satisfiability problem*,
and can be solved by a |SAT solver|.
The various possible choices for all the packages are assigned
boolean variables, and all the dependencies between those packages,
any required packages, etc. are cast as boolean expressions in
conjunctive normal form.
The set of expressions generated is passed to |MiniSAT|. If MiniSAT
cannot find any solution, the error handling code attempts to walk
the set of installed packages and the attempted operation, and print
the reasons that each possible choice was eliminated.
If the currently installed set of packages meet the requirements but
no other does, |pkg| will report that there is nothing to do.
As mentioned in a previous section, the error message generation and
specificity is determined by the inputs to |pkg|. Being as specific as
possible in commands issued to |pkg| will produce the most useful error
messages.
If on the other hand MiniSAT finds a possible solution, we begin
optimization.
Optimize the Solver Results
```````````````````````````
The optimization phase is necessary because there is no way of
describing some solutions as more desirable than others to a SAT
solver.
Instead, once a solution is found, IPS adds constraints
to the problem to separate less desirable choices,
and to separate the current solution as well. We then repeatedly
invoke MiniSAT and repeat the above operation until no more
solutions are found. The last successful solution is taken as the
best one.
Clearly, the difficulty of finding a solution is proportional to
the number of possible solutions. Being more specific about the desired
result will produce solutions more quickly.
Evaluate Actions
````````````````
Once the set of package FMRIs that best satisfy the posed
problem is found, the evaluation phase begins.
In this phase, we compare the packages currently installed on the
system with the end state, and compare package manifests
of old and new packages to determine three lists:
* Actions that are being removed
* Actions that are being added
* Actions that are being updated
The action lists are then updated so that:
* directory and link actions are reference counted, mediated link
processing is done
* hardlinks are marked for repair if their target file is updated.
This is done because updating a target of a hardlink in a manner that
is safe for currently executing processes breaks the hard links.
* editable files moving between packages are correctly handled
so that any user edits are not lost.
* the action lists are sorted so that removals, additions and
updates occur in the correct order.
All the currently installed packages are then cross-checked to
make sure that no packages conflict. That is, ensuring that two
packages do not attempt to deliver a file to the same location,
ensuring that directory attributes for the same directory agree
between packages, etc.
If conflicts exist, these are reported and |pkg| exits with an error message.
Finally, the action lists are scanned to determine if any
SMF services need to be restarted if this operation is performed,
whether or not this change can be applied to a running system,
whether the boot archive needs to be rebuilt and whether the
amount of space required is available, etc.
Download Content
````````````````
If |pkg| is running without the ``-n`` flag, processing continues
to the download phase.
For each action that requires content, we download any required
files by hash and cache them. This step can take some time if
the amount of content to be retrieved is large.
Once downloading is complete, if the change is to be applied to
a live system (image is rooted at '/') and a reboot is required,
the running system is cloned and the target image is switched to
the clone.
Execute Actions
```````````````
Executing actions involves actually performing the install or
remove methods specific to each action type on the image.
Execution begins with all the removal actions being executed. If
any unexpected content is found in directories being removed from
the system, that content is placed in ``/var/pkg/lost+found``.
Execution then proceeds to install and update actions. Note that all
the actions have been blended across all packages. Thus all the
changes in a single package operation are applied to the system at once
rather than package by package. This permits packages to depend on each
other, exchange content, etc. safely. For details on how files
are updated, see the description of the ``file`` action in *Chapter 3*.
Process Actuators
`````````````````
If we're updating a live system, any pending actuators are executed
at this point. These are typically SMF service restarts and refreshes.
Once these are launched, we update the local search indicies. We discuss
actuators in more detail in *Chapter 9*
Lastly, if needed, we update the boot archive.

369
doc/pkg5_docs/dev-guide/chpt6.txt Normal file
View file

@ -0,0 +1,369 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 6
---------
Specifying Dependencies
.......................
Dependencies define how packages are related.
IPS provide a variety of different dependency types as discussed in
*Chapter 3*. In this chapter we go into more detail about how each
dependency type can be used to control the software that is installed.
In IPS, a package cannot be installed unless all package dependencies are
satisfied. IPS allows packages to be mutually dependent (to have circular
dependencies). IPS also allows packages to have different kinds of
dependencies on the same package at the same time.
Dependency Types
~~~~~~~~~~~~~~~~
Each section below contains an example ``depend`` action, as it would appear in
a manifest during package creation.
``require``
```````````
The most basic type of dependency is the ``require`` dependency.
If a package A@1.0 contains a ``require`` dependency on package B@2,
it means that if A@1.0 is installed, a version of B at 2 or higher
must be installed as well.
This acceptance of higher versioned packages reflects the implicit
expectation of binary compatibility in newer versions of existing
packages.
These dependencies are typically used to express functional dependencies
such as libraries or interpreters such as Python, Perl, etc. The version
portion of the specified FMRI can be omitted; it indicates that any
version will suffice. The latter might not be actually true, but
if other dependencies constrain the version adequately, this might
save some effort.
An example ``require`` dependency is::
depend fmri=pkg:/system/library type=require
``require-any``
```````````````
The ``require-any`` dependency is used if more than one package will
satisfy a functional requirement. IPS will pick one of the packages to
install if the dependency is not already satisfied.
A typical use might be to ensure that at least one version of Perl
was installed on the system, for example. The versioning is handled
in the same manner as for the ``require`` dependency.
An example ``require-any`` dependency is::
depend type=require-any fmri=pkg:/editor/gnu-emacs/gnu-emacs-gtk \
fmri=pkg:/editor/gnu-emacs/gnu-emacs-no-x11 \
fmri=pkg:/editor/gnu-emacs/gnu-emacs-x11
``optional``
````````````
The ``optional`` dependency is similar to the ``require`` dependency, but
the specified package need not be installed. However, if it is present,
it must be at the specified version or greater.
This type of dependency is typically used to handle cases where packages
transfer content.
In this case, each version of the package post-transfer
would contain an optional dependency on the other package's
post-transfer version, so it would be impossible to install
incompatible versions of the two packages.
Omitting the version on an optional dependency makes the dependency a no-op,
but is permitted.
An example ``optional`` dependency is::
depend fmri=pkg:/x11/server/xorg@1.9.99 type=optional
``conditional``
```````````````
The ``conditional`` dependency is similar to the ``require`` dependency as
well, except that a predicate attribute is present. If the package specified
in the value of the predicate attribute is present on the system at the
specified or greater version, the conditional dependency is treated as a
``require`` dependency, otherwise it is ignored.
This type of dependency is most often used to install optional extensions
to a package if the requisite base packages are present on the system.
For example, an editor package that has both X11 and terminal versions might
place the X11 version in a separate package, and include a conditional
dependency on the X11 version from the text version with the existence
of the requisite X client library package as the predicate.
An example conditional dependency is::
depend type=conditional fmri=library/python-2/pycurl-26 \
predicate=runtime/python-26
``group``
`````````
The ``group`` dependency is used to construct groups of packages.
The group dependency will ignore the version specified; any version of the
named package satisfies this dependency. The named package
is required unless the package has been placed on the avoid list (see |pkg|),
the package is rejected with ``pkg install --reject``, or the package is
uninstalled with ``pkg uninstall``.
These three options enable administrators to 'deselect' packages that are the
subject of a group dependency. IPS will remember this and not re-install the
package during an update unless it becomes required by another dependency.
If the new dependency is removed by another subsequent operation, then the
package is uninstalled again.
A good example of how to use these dependencies is to construct packages
containing group dependencies on packages that are needed for typical uses
of a system.
The administrator could install all that apply and know that over subsequent
updates to newer versions of the OS, the appropriate packages would be added
to his system.
An example ``group`` dependency is::
depend fmri=package/pkg type=group
``origin``
``````````
The ``origin`` dependency exists to resolve upgrade issues that require
intermediate transitions. The default behavior is to specify the minimum
version of a package (if installed) that must be present on the system being
updated.
For example, a typical use might be a database package version 5 that supports
upgrade from version 3 or greater, but not earlier versions.
In this case, version 5 would have an origin dependency on itself at version 3.
Thus, if version 5 was being fresh installed, installation would proceed; but
if version 1 of the package was installed, one could not upgrade directly to this
version.
Thus, ``pkg update database-package`` would not select version 5
in this case but would pick version 3 instead as the latest possible
version it could upgrade to.
The behavior of this dependency can be modified by the ``root-image`` attribute
being set to ``true``; in this case the named package must be at the specified
version or greater if it is present in the running system, rather than the image
being updated.
This is generally used for operating system issues such as dependencies on boot
block installers.
An example ``origin`` dependency is::
depend fmri=pkg:/database/mydb@3.0 type=origin
``parent``
``````````
The ``parent`` dependency is used for zones or other child images. In
this case, the dependency is only checked in the child image, and specifies a
package and version that must be present in the parent image or global
zone. The version specified must match to the level of precision
specified.
For example, if the ``parent`` dependency is on A@2.1, then any version of A
beginning with 2.1. will match. This dependency is often used to require that
packages are kept in sync between non-global zones and the global zone, and as
a short cut a special package name ``feature/package/dependency/self`` is used
as a synonym for the exact version of the package that contains this dependency.
This is used to keep key operating system components, such as ``libc.so.1``
installed in the zone synchronized with the kernel installed in the global zone.
The ``parent`` dependency is also discussed in *Chapter 12*.
An example ``parent`` dependency is::
depend type=parent fmri=feature/package/dependency/self \
variant.opensolaris.zone=nonglobal
``incorporate``
```````````````
The ``incorporate`` dependency is heavily used in OpenIndiana to ensure that
compatible versions of software are installed together.
The basic mechanism is like that of an ``optional`` dependency, except
that the version matching is that of the ``parent`` dependency: if this package
is present, it must be at the specified version to the level
specified.
How these dependencies are typically used is that many of
them are placed in the same package to define a surface in the package
version space that is compatible. Packages that contain such sets of
incorporate dependencies are often called *incorporations*; it is
typical to define such for sets of software packages that are built
together and are not separately versioned, like much of the kernel.
An example ``incorporate`` dependency is::
depend type=incorporate \
fmri=pkg:/driver/network/e1000g@0.5.11,5.11-2018.0.0.18233
``exclude``
```````````
The ``exclude`` dependency is seldom used. It allows the containing
package to preclude installation with the specified package at the
specified version or higher.
Note that if the version is omitted, no version of the specified package
can be installed with the containing package. These constraints can be
frustrating to administrators, and should be avoided where possible.
An example exclude dependency is::
depend fmri=pkg:/x11/server/xorg@1.10.99 type=exclude
Constraints and Freezing
~~~~~~~~~~~~~~~~~~~~~~~~
Constraints
```````````
Through the careful use of the various types of ``depend`` actions described
above, packages can define the ways in which they are allowed to be upgraded.
In general, we often desire that a group of packages installed on a system
be supported and upgraded as a group: Either all packages in the group are
updated, or none of the packages in the group are updated. As mentioned earlier,
this is the reason for using the ``incorporate`` dependency in OpenIndiana.
The following three partial package manifests show the relationship
between the ``foo`` and ``bar`` packages and the ``myincorp`` incorporation
package:
::
set name=pkg.fmri value=foo@1.0
dir path=foo owner=root group=bin mode=0755
depend fmri=myincorp type=require
::
set name=pkg.fmri value=bar@1.0
dir path=bar owner=root group=bin mode=0755
depend fmri=myincorp type=require
::
set name=pkg.fmri value=myincorp@1.0
depend fmri=foo@1.0 type=incorporate
depend fmri=bar@1.0 type=incorporate
The ``foo`` and ``bar`` packages both have a ``require`` dependency on the
``myincorp`` incorporation. The ``myincorp`` package has ``incorporate``
dependencies such that ``foo`` and ``bar`` can be upgraded to
**at most** version 1.0, to the level of granularity defined by the version
number and, if installed, must be **at least** at version 1.0 or greater.
That is, an ``incorporate`` dependency on version 1.0 allows for
1.0.1, 1.0.2.1, etc. but doesn't allow version 1.1, version 2.0, version 0.9,
etc. When we deliver a new incorporation package, one that has ``incorporate``
dependencies at a higher version, we will allow ``foo`` and ``bar`` to upgrade
to those versions instead (assuming that the incorporation package is also being
upgraded).
Note here that because ``foo`` and ``bar`` both have ``require`` dependencies on
the ``myincorp`` package, the incorporation package must always be installed.
However, conflicting with the requirement we stated at the beginning of this
section, there are some situations where we might want to relax the
incorporation constraint.
Perhaps ``bar`` can function independently of ``foo``, but we'd like ``foo``
to remain within the series of versions our incorporation constrains it to.
We can relax the incorporation constraints using *facets*, allowing the
administrator to effectively *disable* certain ``incorporate`` dependencies.
Facets are discussed in more detail in *Chapter 7*. Briefly, facets
are special attributes that can be applied to actions within a package to enable
authors to mark those actions as optional.
When actions are marked with facet attributes in this manner, the actions
containing those facets can be enabled or disabled using the
``pkg change-facet`` command.
By convention, facets that optionally install ``incorporate`` dependencies are
named ``facet.version-lock.<name>``, where *name* is the package name containing
that ``depend`` action.
So, for example, using the example above, we could have the following
incorporation::
set name=pkg.fmri value=myincorp@1.0
depend fmri=foo@1.0 type=incorporate
depend fmri=bar@1.0 type=incorporate facet.version-lock.bar=true
This incorporation includes our ``depend`` action by default, constraining
``bar`` to version 1.0. The following command relaxes this constraint::
pkg change-facet version-lock.bar=false
The ``bar`` package is free from the incorporation constraints, and can be
upgraded to version 2.0 if necessary.
Freezing
````````
So far, all of the discussion has been around constraints that have been applied
during the package authoring process, by modifying the package manifests
themselves.
|pkg| also has a means for the administrator to apply constraints to the system
at runtime.
Using the ``pkg freeze`` command, the administrator can prevent a given
package from being updated past either its current version, or a version
specified on the command line. This capability is effectively the same as an
``incorporate`` dependency.
See the |pkg| man page for more information on the ``freeze`` command.
In order to apply more complex dependencies to an image, it is necessary to
create and install a package that includes those dependencies.

251
doc/pkg5_docs/dev-guide/chpt7.txt Normal file
View file

@ -0,0 +1,251 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 7
---------
Allowing Variations
...................
In this chapter we explore how variants and facets are used in
IPS to provide different installation options to the end user.
Variants
~~~~~~~~
Since OpenIndiana supports multiple architectures, one common error
made with the SVR4 packaging system was the accidental installation
of packages for an incorrect architecture. With the introduction of
software repositories, the prospect of maintaining a separate
repository for each supported architecture seemed unappealing
to ISVs and error prone for customers. As a result, IPS supports
installation of a single package on multiple architectures.
The mechanism that implements this feature is called a
*variant*. A variant enables the properties of the target image
to determine which software components are actually installed.
A variant has two parts: its name, and the list of possible values.
The variants defined in OpenIndiana are:
======================== =================
Name Values
======================== =================
variant.arch sparc, i386
variant.opensolaris.zone global, nonglobal
variant.debug.* true, false
======================== =================
Variants appear in two places in a package:
* A ``set`` action names the variant and defines the values that
apply to this package.
* Any action that can only be installed for a subset of the
variant values has a tag that specifies the name of the
variant and the value on which it is installed.
For example, a package that delivers the symbolic link ``/var/ld/64``
might include::
set name=variant.arch value=sparc value=i386
dir group=bin mode=0755 owner=root path=var/ld
dir group=bin mode=0755 owner=root path=var/ld/amd64 \
variant.arch=i386
dir group=bin mode=0755 owner=root path=var/ld/sparcv9 \
variant.arch=sparc
link path=var/ld/32 target=.
link path=var/ld/64 target=sparcv9 variant.arch=sparc
link path=var/ld/64 target=amd64 variant.arch=i386
Note that components that are delivered on both sparc and i386 receive
no variant tag, but those delivered to one architecture or the other
receive the appropriate tag. It is perfectly reasonable for actions
to contain multiple tags for different variant names; there might be
debug and nondebug binaries for both sparc and i386.
In OpenIndiana, kernel components are commonly elided from packages
installed in zones, as they serve no useful purpose. Thus, they are
marked with the ``opensolaris.zone`` variant set to ``global`` so that they
are not installed in non-global zones. This is typically done in the
manifest during publication with a |pkgmogrify| rule. Thus the packages
from the i386 and sparc builds are already marked for zones. We then
use the |pkgmerge| command to take the packages from the sparc and i386
builds and merge them together. This is far more reliable and faster
than attempting to construct such packages manually.
The ``variant.debug.*`` portion of the variant namespace is explicitly
predefined to have a default version of ``false``; thus, developers
can provide debug versions of their components, tagged with the
appropriate variant, and users can select that variant if problems
arise. Remember that variants are set per image, so selecting a
suitable name that is unique at the appropriate resolution for that
piece of software is important.
In addition, any variant tags not described here are assumed to
have a default value of ``false`` in the image. This allows
the definition of custom variants not explicitly set in the
image for use in packages.
Note that variant tags are applied to any actions that differ between
architectures during merging; this includes dependencies, ``set`` actions,
etc. Packages that are marked as not supporting one of the variant values
of the current image are not considered for installation.
The |pkgmerge| man page provides several examples of merging packages. Note
that it will merge across multiple different variants at the same time if
needed.
Facets
~~~~~~
Often, package developers have optional portions of their software
that actually belong with the main body, but some people might not
want to install. Some examples are localization files for different
locales, man pages and other documentation, header files needed
only by developers or DTrace users.
Traditionally, such optional content has been placed in separate
packages with an arbitrarily selected naming convention (such as
appending ``-dev`` or ``-devel`` to the package name) enabling administrators
to select the optional content.
This has led to various problems, such as adding a new locale
for all the software on a system being a rather irritating task,
as the admin has to discover all the necessary packages by examining
the lists of available packages.
IPS has implemented a mechanism similar to variants called *facets*
to deal with this problem. Like variants, facets have a name and
a value. The value is either set to ``true`` or ``false`` in the
image. The default value is ``true``. The facet namespace is hierarchal,
with matching rules such that the longest match wins.
For example, the default value for all facets is ``true``; the |pkg| client
implicitly sets ``facet.*`` to ``true``.
Documentation in OpenIndiana packages is tagged with the type of
documentation. For example, man pages are tagged with
``facet.doc.man=true`` in the package manifests.
The following commands include man pages but exclude all other documentation
from being installed in this image::
# pkg change-facet facet.doc.*=false
# pkg change-facet facet.doc.man=true
Similarly, the following commands install only the German localization in this
image::
# pkg change-facet facet.locale.*=false
# pkg change-facet facet.locale.de=true
If an action contains multiple facet tags, the action is installed if the value
of any of the facet tags is ``true``.
The ``pkg facet`` command is useful in determining which facets are
set in the image.
The package developer can use |pkgmogrify| to quickly tag his
man pages, localizations, etc. using regular expressions to
match the different types of files. This is described in detail
in *Chapter 8*.
Facets can also be used to manage dependencies, essentially turning
dependencies on and off, depending on whether the facet is set. See
*Chapter 6* for a discussion of ``facet.version-lock.*``.
OpenIndiana facets that might be of use for software developers include:
======================== ================== ==================
facet.devel facet.locale.es_BO facet.locale.lt_LT
facet.doc facet.locale.es_CL facet.locale.lv
facet.doc.man facet.locale.es_CO facet.locale.lv_LV
facet.doc.pdf facet.locale.es_CR facet.locale.mk
facet.doc.info facet.locale.es_DO facet.locale.mk_MK
facet.doc.html facet.locale.es_EC facet.locale.ml
facet.locale facet.locale.es_ES facet.locale.ml_IN
facet.locale.af facet.locale.es_GT facet.locale.mr
facet.locale.af_ZA facet.locale.es_HN facet.locale.mr_IN
facet.locale.ar facet.locale.es_MX facet.locale.ms
facet.locale.ar_AE facet.locale.es_NI facet.locale.ms_MY
facet.locale.ar_BH facet.locale.es_PA facet.locale.mt
facet.locale.ar_DZ facet.locale.es_PE facet.locale.mt_MT
facet.locale.ar_EG facet.locale.es_PR facet.locale.nb
facet.locale.ar_IQ facet.locale.es_PY facet.locale.nb_NO
facet.locale.ar_JO facet.locale.es_SV facet.locale.nl
facet.locale.ar_KW facet.locale.es_US facet.locale.nl_BE
facet.locale.ar_LY facet.locale.es_UY facet.locale.nl_NL
facet.locale.ar_MA facet.locale.es_VE facet.locale.nn
facet.locale.ar_OM facet.locale.et facet.locale.nn_NO
facet.locale.ar_QA facet.locale.et_EE facet.locale.no
facet.locale.ar_SA facet.locale.eu facet.locale.or
facet.locale.ar_TN facet.locale.fi facet.locale.or_IN
facet.locale.ar_YE facet.locale.fi_FI facet.locale.pa
facet.locale.as facet.locale.fr facet.locale.pa_IN
facet.locale.as_IN facet.locale.fr_BE facet.locale.pl
facet.locale.az facet.locale.fr_CA facet.locale.pl_PL
facet.locale.az_AZ facet.locale.fr_CH facet.locale.pt
facet.locale.be facet.locale.fr_FR facet.locale.pt_BR
facet.locale.be_BY facet.locale.fr_LU facet.locale.pt_PT
facet.locale.bg facet.locale.ga facet.locale.ro
facet.locale.bg_BG facet.locale.gl facet.locale.ro_RO
facet.locale.bn facet.locale.gu facet.locale.ru
facet.locale.bn_IN facet.locale.gu_IN facet.locale.ru_RU
facet.locale.bs facet.locale.he facet.locale.ru_UA
facet.locale.bs_BA facet.locale.he_IL facet.locale.rw
facet.locale.ca facet.locale.hi facet.locale.sa
facet.locale.ca_ES facet.locale.hi_IN facet.locale.sa_IN
facet.locale.cs facet.locale.hr facet.locale.sk
facet.locale.cs_CZ facet.locale.hr_HR facet.locale.sk_SK
facet.locale.da facet.locale.hu facet.locale.sl
facet.locale.da_DK facet.locale.hu_HU facet.locale.sl_SI
facet.locale.de facet.locale.hy facet.locale.sq
facet.locale.de_AT facet.locale.hy_AM facet.locale.sq_AL
facet.locale.de_BE facet.locale.id facet.locale.sr
facet.locale.de_CH facet.locale.id_ID facet.locale.sr_ME
facet.locale.de_DE facet.locale.is facet.locale.sr_RS
facet.locale.de_LI facet.locale.is_IS facet.locale.sv
facet.locale.de_LU facet.locale.it facet.locale.sv_SE
facet.locale.el facet.locale.it_CH facet.locale.ta
facet.locale.el_CY facet.locale.it_IT facet.locale.ta_IN
facet.locale.el_GR facet.locale.ja facet.locale.te
facet.locale.en facet.locale.ja_JP facet.locale.te_IN
facet.locale.en_AU facet.locale.ka facet.locale.th
facet.locale.en_BW facet.locale.ka_GE facet.locale.th_TH
facet.locale.en_CA facet.locale.kk facet.locale.tr
facet.locale.en_GB facet.locale.kk_KZ facet.locale.tr_TR
facet.locale.en_HK facet.locale.kn facet.locale.uk
facet.locale.en_IE facet.locale.kn_IN facet.locale.uk_UA
facet.locale.en_IN facet.locale.ko facet.locale.vi
facet.locale.en_MT facet.locale.ko_KR facet.locale.vi_VN
facet.locale.en_NZ facet.locale.ks facet.locale.zh
facet.locale.en_PH facet.locale.ks_IN facet.locale.zh_CN
facet.locale.en_SG facet.locale.ku facet.locale.zh_HK
facet.locale.en_US facet.locale.ku_TR facet.locale.zh_HK
facet.locale.en_ZW facet.locale.ky facet.locale.zh_SG
facet.locale.eo facet.locale.ky_KG facet.locale.zh_TW
facet.locale.es_AR facet.locale.lg
======================== ================== ==================

193
doc/pkg5_docs/dev-guide/chpt8.txt Normal file
View file

@ -0,0 +1,193 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 8
---------
Modifying Package Manifests Programmatically
............................................
This chapter explains how package manifests can be machine edited to permit the
automated annotation and checking of package manifests.
*Chapter 4* covers the basics for how to publish packages. These techniques are
all that is necessary to publish a package, but when publishing a large package,
a large number of packages, or publishing packages over a period of time, there
can be aspects which involve a significant time commitment for doing repetitive
tasks.
For example, one rule that's used when publishing OpenIndiana is that all
kernel modules should be tagged as requiring a reboot.
One option would be to impose this constraint through human examination and
intervention, but that would be costly and likely error prone.
A second option would be to write a script or program that would handle tagging
these actions. The difficulty here is that to be sure the program tagged
actions correctly, it would need to parse the actions. This can certainly be
done, but it duplicates a lot of functionality already in the IPS framework.
The third, and best, option is to use |pkgmogrify|, provided by IPS, to
transform the package manifests in repeatable ways.
There are two types of rules |pkgmogrify| understands, transform and include.
Transform rules are used to modify actions. Include rules cause other
files to be processed.
Transform Rules
~~~~~~~~~~~~~~~
When publishing OpenIndiana, we made the assumption that all files delivering
in a subdirectory named "``kernel``" should be treated as kernel modules. This
is the rule used to do the tagging::
<transform file path=.*kernel/.+ -> default reboot-needed true>
..
* The rule is enclosed with ‘``<``’ and ‘``>``’. The portion of the rule to
the left of the ‘``->``’ is the selection section or matching section. The
portion to the right of the ‘``->``’ is the execution section of the
operation.
* ‘``transform``’ is the type of the rule
* ‘``file``’ means this rule is for ``file`` actions
* ‘``path=.*kernel/.+``’ means that only file actions with a path attribute
that matches the regular expression ‘``.*kernel/.+``’ are transformed
* ‘``default``’ means that what follows should be added to a matching action
unless a value for the attribute has already been set
* ‘``reboot-needed``’ is the attribute being set
* ‘``true``’ is the value of the attribute
The selection or matching section of a transform rule can restrict by action
type and by action attribute values. The |pkgmogrify| man page goes into detail
about how these matching rules work, but the typical uses are for selecting
actions which deliver to certain areas of the file system.
.. raw:: pdf
PageBreak
For example, a rule that began like this
.. parsed-literal::
<transform file dir link hardlink path=usr/bin.\* -> *[operation]*>
could be used to ensure that ``usr/bin`` and everything delivered inside of it
defaulted to the correct user or group. There is a long list of operations
which |pkgmogrify| can perform, detailed in the man page, which enable a package
developer to programmatically add, remove, set, and edit actions' attributes as
well as add and remove entire actions.
Include Rules
~~~~~~~~~~~~~
Include rules enable transforms to be spread across multiple files and subsets
reused by different manifests. Suppose a developer needs to deliver two
packages: A and B. Both packages should have their ``source-url`` set to the
same url, but only package B should have its files in ``/etc`` set to be
``group=sys``.
In the manifest for package A, an include rule which pulls in the file with the
``source-url`` transform should be added. In package B, an include rule which
pulls in the file containing the file ``group`` setting transform should be
added.
Finally, an include rule which pulls in the file with the ``source-url``
transform should be added to either package B or the file with the transform
that sets the group.
Transform Order
~~~~~~~~~~~~~~~
Transforms are applied in the order in which they are encountered in a file.
The ordering can be used to simplify the matching portions of transforms.
Suppose all files delivered in ``/foo`` should have a default group of ``sys``,
except those delivered in ``/foo/bar``, which should have a default group of
``bin``.
It's certainly possible to write a complex regular expression which matches all
paths that begin with ``/foo``, except those that begin with ``/foo/bar``. Using
the ordering of transforms makes it much simpler.
When ordering default transforms, always go from most specific to most general.
Otherwise the latter rules will never be used.
In this case, the two rules would look like this::
<transform file path=foo/bar/.* -> default group bin>
<transform file path=foo/.* -> default group sys>
Using transforms to add an action using the matching described above would be
difficult since the package developer would need to find a pattern which matched
each package delivered once and only once.
|pkgmogrify| creates synthetic actions to help with this issue. As |pkgmogrify|
processes manifests, for each manifest that sets the ``pkg.fmri`` attribute,
a synthetic *pkg action* is created by |pkgmogrify|. The package developer can
match against the ``pkg`` action as if it was actually in the manifest.
For example, suppose a package developer wanted to add to every package an
action containing the website (foo.com) where the source code for the delivered
software can be found. The following transform accomplishes that::
<transform pkg -> emit set info.source-url=http://foo.com>
.. raw:: pdf
PageBreak
Packaged Transforms
~~~~~~~~~~~~~~~~~~~
As a convenience to developers, a set of the transforms that were used when
packaging OpenIndiana itself are available in ``/usr/share/pkg/transforms``.
At the time of writing, these are:
**developer**
Sets ``facet.devel`` on ``*.h`` header files delivered to
``/usr/.*/include``, archive and lint libraries, ``pkg-config(1)`` data
files, and ``autoconf(1)`` macros.
**documentation**
Sets a variety of ``facet.doc.*`` facets on documentation files.
**locale**
Sets a variety of ``facet.locale.*`` facets on files which are
locale-specific.
**smf-manifests**
Adds a ``restart_fmri`` actuator pointing to the
``svc:/system/manifest-import:default`` on any packaged SMF manifests
so that the system will import that manifest after the package is
installed.

248
doc/pkg5_docs/dev-guide/chpt9.txt Normal file
View file

@ -0,0 +1,248 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Chapter 9
---------
Causing System Change With SMF
..............................
This chapter explains how to use the Service Management Facility (SMF) to
automatically handle any necessary system changes that should occur as a result
of package installation.
The package developer must determine which actions, when initially installed,
updated or removed should cause a change to the system. For each of those
actions, the package developer needs to determine which existing service
provides the desired system change, or write a new service which provides the
needed functionality and ensure that service is delivered to the system.
When the set of actions has been determined, those actions must be tagged in
the package manifest with the correct *actuator* in order to cause that system
change to occur.
As discussed in *Chapter 1*, some system changes are needed to employ the
*software self-assembly* concept used by OpenIndiana and IPS, but system
changes are not limited to this role.
We'll discuss the available actuators in the next section and then provide some
examples.
Actuators
~~~~~~~~~
The following tags can be added to any action in a manifest:
reboot-needed
`````````````
This actuator takes the value ``true`` or ``false``. This actuator declares
that installation, removal or update of the tagged action requires a
reboot when IPS is operating on a live image.
The following actuators are related to SMF services, and are the ones we will
focus on in this chapter.
SMF Actuators
`````````````
SMF actuators take a single SMF FMRI as a value, possibly including globbing
characters to match multiple FMRIs. If the same FMRI is tagged by multiple
actions, possibly across multiple packages being operated on, IPS will only
trigger that actuator once.
The following list of SMF actuators describes the effect on the service FMRI
that is the value of each named actuator:
disable_fmri
The given service should be disabled prior to the package operation
being performed
refresh_fmri
The given service should be refreshed after the package operation has
completed
restart_fmri
The given service should be restarted after the package operation has
completed
suspend_fmri
The given service should be temporarily suspended prior to the package
operation and enabled once it has completed
Delivering an SMF Service
~~~~~~~~~~~~~~~~~~~~~~~~~
A package that delivers a new SMF service usually needs a system change. The
package delivers the SMF manifest file and method script, and the packaged
application requires that the SMF service it delivers must be available
after package installation.
SVR4 post-install scripting could run an SMF command to restart the
``svc:/system/manifest-import:default`` service.
In IPS, the action delivering the manifest file into ``lib/svc/manifest`` or
``var/svc/manifest`` should instead be tagged with the actuator:
``restart_fmri=svc:/system/manifest-import:default``.
The actuator ensures that when the manifest is added, updated, or removed, the
``manifest-import`` service is restarted, causing the service delivered by that
SMF manifest to be added, updated, or removed.
If the package is added to a live-system, this action is performed once all
packages have been added to the system during that packaging operation. If the
package is added to an alternate boot environment, this action is performed
during the first boot of that boot environment.
A Service That Runs Once
~~~~~~~~~~~~~~~~~~~~~~~~
Another common example is a system change that performs one-time configuration
of the new software environment.
In the package delivering our application, we would include the following
actions::
file path=opt/myapplication/bin/run-once.sh owner=root group=sys mode=0755
file path=lib/svc/manifest/application/myapplication-run-once.xml owner=root group=sys \
mode=0644 restart_fmri=svc:/system/manifest-import:default
The SMF method script for the service could contain anything that is needed to
further configure our application, or modify the system so that our application
runs efficiently. In this example, we'll just have it write a simple log
message.
Generally, we also want to ensure that the SMF service only performs work if the
application has not already been configured.
Another approach would be to package the service separate from the application
itself, then have the method script remove the package that contains the
service.
Our method script is::
#!/usr/bin/sh
. /lib/svc/share/smf_include.sh
assembled=$(/usr/bin/svcprop -p config/assembled $SMF_FMRI)
if [ "$assembled" == "true" ] ; then
exit $SMF_EXIT_OK
fi
svccfg -s $SMF_FMRI setprop config/assembled = true
svccfg -s $SMF_FMRI refresh
echo "This is output from our run-once method script"
When testing a method script, it is advisable to run ``pkg verify``
before and after installing the package that runs the actuator.
Compare the output of each run to ensure that the script doesn't attempt to
modify any files that are not marked as editable.
.. raw:: pdf
PageBreak
Our SMF service manifest is::
<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type='manifest' name='MyApplication:run-once'>
<service
name='application/myapplication/run-once'
type='service'
version='1'>
<single_instance />
<dependency
name='fs-local'
grouping='require_all'
restart_on='none'
type='service'>
<service_fmri value='svc:/system/filesystem/local:default' />
</dependency>
<dependent
name='myapplication_self-assembly-complete'
grouping='optional_all'
restart_on='none'>
<service_fmri value='svc:/milestone/self-assembly-complete' />
</dependent>
<instance enabled='true' name='default'>
<exec_method
type='method'
name='start'
exec='/opt/myapplication/bin/run-once.sh'
timeout_seconds='0'/>
<exec_method
type='method'
name='stop'
exec=':true'
timeout_seconds='0'/>
<property_group name='startd' type='framework'>
<propval name='duration' type='astring' value='transient' />
</property_group>
<property_group name='config' type='application'>
<propval name='assembled' type='boolean' value='false' />
</property_group>
</instance>
</service>
</service_bundle>
Note that the SMF service has a ``startd/duration`` property set to
``transient`` so that |svc.startd| doesn't track processes for this service.
Also note that it adds itself as a dependency to the ``self-assembly-complete``
system milestone.
Self-Assembly Hints
~~~~~~~~~~~~~~~~~~~
Here are some additional hints when writing SMF methods to support
self-assembly:
Timestamps
``````````
In an SMF method script, it can be efficient to use the output of ``ls -t`` on a
directory of packaged configuration file fragments, using ``head -1`` to select
the most recently changed version. The timestamp of this file can be compared
with the timestamp of the unpackaged configuration file which is compiled from
those fragments. This comparison can be used when deciding whether the service
needs to recompile the configuration file.
This can be useful if the process of compiling a configuration file from those
fragments is expensive to perform each time the method script runs.
Timeouts
````````
In the example SMF service used in this chapter, we had a ``timeout_seconds``
value of ``0`` for the start method. This means that SMF will wait indefinitely
for self-assembly to complete.
Depending on circumstances, developers might want to impose a finite timeout on
their self-assembly processes, enabling SMF to drop the service to
``maintenance`` if something goes wrong. This can assist the developer when
debugging.

389
doc/pkg5_docs/dev-guide/dev-guide.style Normal file
View file

@ -0,0 +1,389 @@
# List any fonts you would like to embed in the PDF here
embeddedFonts: []
# Default page setup. Can be overridden by including other
# stylesheets with -s
pageSetup:
size: A4
width: null
height: null
margin-top: 2cm
margin-bottom: 2cm
margin-left: 2cm
margin-right: 2cm
margin-gutter: 0cm
spacing-header: 5mm
spacing-footer: 5mm
# The first template is one of the 'pageTemplates"
# (See next section)
firstTemplate: coverPage
# pageTemplates can be accessed with the .. raw:: pdf PageBreak command
pageTemplates:
coverPage:
frames: [
[0cm, 0cm, 100%, 100%]
]
showHeader : false
showFooter : false
emptyPage:
frames: []
[0cm, 0cm, 100%, 100%]
showHeader : false
showFooter : false
oneColumn:
frames: []
[0cm, 0cm, 100%, 100%]
showHeader : true
showFooter : true
twoColumn:
frames: []
[0cm, 0cm, 49%, 100%]
[51%, 0cm, 49%, 100%]
showHeader : true
showFooter : true
threeColumn:
frames: []
[2%, 0cm, 29.333%, 100%]
[35.333%, 0cm, 29.333%, 100%]
[68.666%, 0cm, 29.333%, 100%]
showHeader : true
showFooter : true
cutePage:
frames: []
[0%, 0%, 100%, 100%]
showHeader : true
showFooter : true
defaultFooter : ###Page###
defaultHeader : ###Section###
fontsAlias:
stdBold: Times-Bold
stdBoldItalic: Times-BoldItalic
stdFont: Times-Roman
stdItalic: Times-Italic
stdSans: Helvetica
stdSansBold: Helvetica-Bold
stdSansItalic: Helvetica-Oblique
stdSansBoldItalic: Helvetica-BoldOblique
stdMono: Courier
stdMonoItalic: Courier-Oblique
stdMonoBold: Courier-Bold
stdMonoBoldItalic: Courier-BoldOblique
stdSerif: Times-Roman
linkColor: navy
styles:
base:
parent: null
fontName: stdFont
fontSize: 10
leading: 12
leftIndent: 0
rightIndent: 0
firstLineIndent: 0
alignment: TA_LEFT
spaceBefore: 0
spaceAfter: 0
bulletFontName: stdFont
bulletFontSize: 10
bulletIndent: 0
textColor: black
backColor: null
wordWrap: null
borderWidth: 0
borderPadding: 0
borderColor: null
borderRadius: null
allowWidows: false
allowOrphans: false
hyphenation: false
kerning: false
underline: false
strike: false
commands: []
normal:
parent: base
title-reference:
parent: normal
fontName: stdItalic
bodytext:
parent: normal
spaceBefore: 6
alignment: TA_JUSTIFY
hyphenation: true
toc:
parent: normal
blockquote:
parent: bodytext
leftIndent: 20
lineblock:
parent: bodytext
line:
parent: lineblock
spaceBefore: 0
toc1:
parent: toc
fontName: stdBold
toc2:
parent: toc
leftIndent: 20
toc3:
parent: toc
leftIndent: 40
toc4:
parent: toc
leftIndent: 60
toc5:
parent: toc
leftIndent: 80
toc6:
parent: toc
leftIndent: 100
toc7:
parent: toc
leftIndent: 100
toc8:
parent: toc
leftIndent: 100
toc9:
parent: toc
leftIndent: 100
toc10:
parent: toc
leftIndent: 100
toc11:
parent: toc
leftIndent: 100
toc12:
parent: toc
leftIndent: 100
toc13:
parent: toc
leftIndent: 100
toc14:
parent: toc
leftIndent: 100
toc15:
parent: toc
leftIndent: 100
footer:
parent: normal
alignment: TA_CENTER
header:
parent: normal
alignment: TA_CENTER
attribution:
parent: bodytext
alignment: TA_RIGHT
figure:
parent: bodytext
alignment: TA_CENTER
commands: []
[VALIGN, [ 0, 0 ], [ -1, -1 ], TOP ]
[ALIGN, [ 0, 0 ], [ -1, -1 ], CENTER ]
colWidths: [100%]
figure-caption:
parent: bodytext
fontName: stdItalic
alignment: TA_CENTER
figure-legend:
parent: bodytext
bullet-list:
parent: bodytext
spaceAfter: 4
commands: []
[VALIGN, [ 0, 0 ], [ -1, -1 ], TOP ]
[RIGHTPADDING, [ 0, 0 ], [ 1, -1 ], 0 ]
colWidths: ["20", null]
bullet-list-item:
parent: bodytext
item-list:
parent: bodytext
spaceBefore: 0
commands: []
[VALIGN, [ 0, 0 ], [ -1, -1 ], TOP ]
[RIGHTPADDING, [ 0, 0 ], [ 1, -1 ], 0 ]
colWidths: [20pt,null]
item-list-item:
parent: bodytext
definition-list-term:
parent: normal
fontName: stdBold
spaceBefore: 4
spaceAfter: 0
keepWithNext: false
definition-list-classifier:
parent: normal
fontName: stdItalic
definition:
parent: bodytext
firstLineIndent: 0
bulletIndent: 0
spaceBefore: 0
colWidths: [20pt,null]
commands: []
[VALIGN, [ 0, 0 ], [ -1, -1 ], TOP ]
[LEFTPADDING, [ 0, 0 ], [ -1, -1 ], 0 ]
[BOTTOMPADDING, [ 0, 0 ], [ -1, -1 ], 0 ]
[RIGHTPADDING, [ 0, 0 ], [ -1, -1 ], 0 ]
fieldname:
parent: bodytext
alignment: TA_RIGHT
fontName: stdBold
fieldvalue:
parent: bodytext
rubric:
parent: bodytext
textColor: darkred
alignment: TA_CENTER
italic:
parent: bodytext
fontName: stdItalic
heading:
parent: normal
fontName: stdSansBold
keepWithNext: true
spaceBefore: 12
spaceAfter: 6
title:
parent: heading
fontSize: 200%
fontName: stdSansBold
alignment: TA_CENTER
keepWithNext: false
spaceAfter: 10
subtitle:
parent: title
spaceBefore: 12
fontName: stdSansBold
fontSize: 75%
heading1:
parent: heading
fontName: stdSansBold
fontSize: 175%
heading2:
parent: heading
fontName: stdSansBold
fontSize: 150%
heading3:
parent: heading
fontName: stdSansBold
fontSize: 125%
heading4:
parent: heading
fontName: stdSansBold
heading5:
parent: heading
fontName: stdSansBold
heading6:
parent: heading
fontName: stdSansBold
topic-title:
parent: heading3
sidebar-title:
parent: heading3
sidebar-subtitle:
parent: heading4
titletable:
parent:table
colWidths: ["50%", "50%" ]
commands: []
headertable:
parent:table
colWidths: [ "100%" ]
commands: []
[ "LINEBELOW", [ 0, 0 ], [ 1, -1 ], .25, "black" ]
footertableleft:
parent:table
colWidths: [ "10%", "90%" ]
commands: []
footertableright:
parent:table
colWidths: [ "100%" ]
commands: []
footerleft:
fontName: stdSans
fontSize: 65%
alignment: left
footerright:
fontName: stdSans
fontSize: 65%
alignment: right
pageleft:
fontName: stdSans
alignment: left
pageright:
fontName: stdSans
alignment: right

198
doc/pkg5_docs/dev-guide/developer-guide.txt Normal file
View file

@ -0,0 +1,198 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
Packaging and Delivering Software with the Image Packaging System
=================================================================
A developer's guide
^^^^^^^^^^^^^^^^^^^
.. raw:: pdf
Spacer 0 100
.. class:: figure
|oilogo|
.. raw:: pdf
SetPageCounter 0 arabic
Spacer 0 450
.. class:: titletable
+--------------------------+--------------------+
| | |
| | |version| |
| | |
+--------------------------+--------------------+
.. header::
.. oddeven::
.. class:: headertable
+-----------------------------------------------+
| .. class:: footerright |
| |
| ###Section### |
| |
+-----------------------------------------------+
.. class:: headertable
+-----------------------------------------------+
| .. class:: footerleft |
| |
| ###Section### |
| |
+-----------------------------------------------+
.. footer::
.. oddeven::
.. class:: footertableright
+------------------------------------------------------+
| .. class:: pageright |
| |
| ###Page### |
+------------------------------------------------------+
.. class:: footertableleft
+---------------------+--------------------------------+
| .. class:: pageleft | .. class:: footerleft |
| | |
| ###Page### | ###Title### |
+---------------------+--------------------------------+
.. raw:: pdf
PageBreak oneColumn
::
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
This document is provided for information purposes only and the
contents hereof are subject to change without notice. This document is
not warranted to be error-free, nor subject to any other warranties or
conditions, whether expressed orally or implied in law, including
implied warranties and conditions of merchantability or fitness for a
particular purpose. We specifically disclaim any liability with
respect to this document and no contractual obligations are formed
either directly or indirectly by this document. This document may not
be reproduced or transmitted in any form or by any means, electronic
or mechanical, for any purpose, without our prior written permission.
Oracle and Java are registered trademarks of Oracle and/or its
affiliates. Other names may be trademarks of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel
Corporation. All SPARC trademarks are used under license and are
trademarks or registered trademarks of SPARC International, Inc. AMD,
Opteron, the AMD logo, and the AMD Opteron logo are trademarks or
registered trademarks of Advanced Micro Devices. UNIX is a registered
trademark licensed through X/Open Company, Ltd. 0611
Preface
-------
In OpenIndiana system software is packaged with
the Image Packaging System, or IPS. IPS takes care of
installing new software and upgrading that software.
This manual is for developers and advanced administrators who want
to better understand IPS, how to use it to package their own
software, and want to understand how OpenIndiana is packaged with
IPS.
Special attention is given to the underlying design
concepts and design patterns so that readers can more readily
understand and utilize the more advanced features of IPS.
How this book is organized
--------------------------
**Chapter 1** - *Design Goals and Concepts*, outlines the basic
design philosophy of IPS and its expression as software
patterns.
**Chapter 2** - *Package Lifecycle*, provides an overview of the
software package lifecycle with IPS.
**Chapter 3** - *Basic Terminology*, lays out the basic terminology
and describes the various components that form IPS.
**Chapter 4** - *Packaging Software with IPS*, gets
the new user started constructing their own packages.
**Chapter 5** - *Installing, Removing, and Updating Software Packages*,
shows basic operation of pkg(1).
**Chapter 6** - *Specifying Dependencies*, explains the different types
of IPS dependencies and how they can be used to construct working
software systems.
**Chapter 7** - *Allowing Variations*, explains how variants, facets
and mediated links are used to allow software publishers to
define multiple installation forms for their packages.
**Chapter 8** - *Modifying Package Manifests Programmatically*, explains
how package manifests can be machine edited to permit the automated
annotation and checking of package manifests.
**Chapter 9** - *Causing System Change With SMF*, explains how to
use the Service Management Facility to automatically handle
any necessary system changes that should occur as a result of
package installation.
**Chapter 10**, *Advanced Update*, deals with more complex package
upgrade issues, and describes several features in IPS designed
to simplify these problems.
**Chapter 11**, *Signing Packages*, explains how package signing works
and how developers and QA organizations can sign either new
or existing, already signed packages.
**Chapter 12**, *Handling Non-Global Zones*, describes how IPS handles
zones and discusses those cases where package developers should be
aware of zones.
**Chapter 13**, *How IPS Features Are Used When Packaging the OpenIndiana
OS*, describes how the packages for the operating system are
constructed and how the various dependency types in IPS are used to
define working package sets.
**Chapter 14**, *Republishing Packages*, describes how administrators
can modify existing packages if needed for local conditions.
**Appendix A**: *Classifying Packages*, contains info.classification
scheme definitions.
**Appendix B**: *Converting SVR4 packages to IPS*, gives an example of
converting an SVR4 package to IPS, and highlights some areas that might
need special attention.

44
doc/pkg5_docs/dev-guide/macros.txt Normal file
View file

@ -0,0 +1,44 @@
.. |oilogo| image:: ../../doc/dev-guide/oi_logo.jpg
.. Some basic macros, possibly links to man pages?
.. |packagemanager| replace:: ``packagemanager(1)``
.. |pkg| replace:: ``pkg(1)``
.. |pkg.depotd| replace:: ``pkg.depotd(1M)``
.. |pkg.sysrepo| replace:: ``pkg.sysrepo(1M)``
.. |pkg5| replace:: ``pkg(5)``
.. |pkgdepend| replace:: ``pkgdepend(1)``
.. |pkgdiff| replace:: ``pkgdiff(1)``
.. |pkgfmt| replace:: ``pkgfmt(1)``
.. |pkglint| replace:: ``pkglint(1)``
.. |pkgmerge| replace:: ``pkgmerge(1)``
.. |pkgmogrify| replace:: ``pkgmogrify(1)``
.. |pkgrecv| replace:: ``pkgrecv(1)``
.. |pkgrepo| replace:: ``pkgrepo(1)``
.. |pkgsend| replace:: ``pkgsend(1)``
.. |pkgsign| replace:: ``pkgsign(1)``
.. |pm-updatemanager| replace:: ``pm-updatemanager(1)``
.. |pkginfo| replace:: ``pkginfo(4)``
.. |pkgmap| replace:: ``pkgmap(4)``
.. |svc.startd| replace:: ``svc.startd(1M)``
.. |beadm| replace:: ``beadm(1M)``
.. |zonecfg| replace:: ``zonecfg(1M)``
.. Terminology - we want hyperlinks to the right places for these eventually
.. and only use these macros the first time we mention these.
.. |Immutable Zones| replace:: Immutable Zones
.. |Immutable Zone| replace:: Immutable Zone
.. |ZFS| replace:: ZFS
.. |SMF| replace:: SMF
.. External links - we may fold these into Terminology above
.. |SAT solver| replace:: `SAT solver`_
.. _SAT solver: http://en.wikipedia.org/wiki/Boolean_satisfiability_problem
.. |MiniSAT| replace:: `MiniSAT`_
.. _MiniSAT: http://minisat.se

BIN
doc/pkg5_docs/dev-guide/oi_logo.jpg Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 8.4 KiB

77
doc/pkg5_docs/directory_perms.txt Normal file
View file

@ -0,0 +1,77 @@
Getting directory permissions correct and consistent
between packages is a common problem in distributions;
it's often made worse by sets of packages that attempt
to be installable across multiple versions of the OS.
This is a resolvable problem in the small, but getting
all packages consistent everywhere is clearly untenable,
esp. if directory permissions change over time.
Several ways of dealing w/ this problem suggest themselves:
0) continue as we have been
Pro - easy to do
Con - annoying verification errors, inconsistent
results depending on order of package installation.
1) Fail package installation if new package has different
permissions than existing (already installed) directories.
Pro - easy, solves consistency problem
Con - pushes problem onto user of package, since
problem is caught at install time. Makes changes
very hard.
2) Define a directory permission in just one package,
and make all packages that install into that directory
depend on that package.
Pro - easy to understand.
Con - difficult to manage, leads to a lot of packages
if granularity of directory installations is fine.
ISV implementation more difficult.
Another approach that we're considering is the following:
*) Use a directory of template files (identified by pkg name)
that define default directory permissions, uid & gid.
In this file, both explicit specifications and matching
rules are permitted.
For example:
/etc/dirperms.d/SUNWcs might contain:
/* user=root group=bin mode=755
/usr user=root group=sys mode=755
/var user=root group=sys mode=755
/var/pkg/* user=root group=root mode=755
Explicit matches are always favored, and the
longest possible match is preferred as well.
We anticipate that few packages will actually deliver such
files; the default one in SUNWcs should do for most. Conflicting
permissions in templates cause error messages.
*) The default directory permissions would be applied to
* directories w/o explicit permissions
* directories where package manifests explicitly
conflict in directory permissions
We anticipate that this mechanism should greatly reduce the
difficulty of getting directory permissions correct, as most
packages can simply not specify them.
Possible problem is that different packages could deliver
conflicting template specifications. In this case, the
effect is undefined, and pkg verify will complain about
this situation.

39
doc/pkg5_docs/elf-jar-handling.txt Normal file
View file

@ -0,0 +1,39 @@
pkg
ELF (AND MAYBE JAR) DEPENDENCY HANDLING
ELF files give us ISA information, required libraries with versions, and
potentially provided interface versions. This latter information is
missing from a large set of libraries on OpenSolaris, built without
versioned interfaces.
It seems that an ELF file, on upload, can be tested for its ELF
dependencies, and that these files in turn can be tested for presence.
If any of the latter file's revisions provide versions, then these can
be tested and translated into a minimum package requirement for the
package containing the introduced ELF file.
We always want the latest version surface, so we don't want to send a
new header file without the corresponding ELF objects its changes caused
to be generated.
It seems that ELF differences are only useful for determining whether an
incoming transaction is significant. For instance, RE delivers a group
of change that actually contains no change to the non-ELF files, and the
change to the ELF files is non-ELF significant. In this case, perhaps
the delivery should fail/warn.
XXX Can we do something similar for JAR files? Tasting JAR files is
outlined in $SRC/cmd/file/file.c:zipfile(), since a JAR file is a zip
file with extra header information. [1]
XXX Can we do something similar for Perl or Python (or any language with
internal versioning statements)?
XXX There's really no way to tie the kernel to libc in the current
system, or in pkg(1M). Should we be adding a simple signature, or
reusing library versioning, or must it be left to a human (expressing it
via pkg(1M) dependencies)?
[1] Sun Microsystems, Inc., JAR File Specification,
http://java.sun.com/j2se/1.4.2/docs/guide/jar/jar.html

38
doc/pkg5_docs/es-requirements.txt Normal file
View file

@ -0,0 +1,38 @@
pkg
ENTERPRISE SYSTEM REQUIREMENTS
1. High level requirements.
In no particular order.
- multi-install Some packages must be able to be installed multiple
times (in separate locations) on the filesystem.
- non-root install
- multi-platform Includes Windows.
2. Initial assessment.
Multi-install and non-root install are handled by the user image
type, which allows a per-user install with a separate pkg client cache.
The user image is defined by an image configuration specifying its type,
its root directory, and its parent image (for satisfying non-relocatable
packages).
Question for ES folks: does "the system" need to (or want to) know about
non-root installs in order to be able to patch / upgrade them along with
normal installs (or send out reminders to do so), or will non-root installs
be completely up to the user to manage? I can see a site policy that
requires security patches even for non-root installs; this would require
such installations to register with the system so they can be found.
Meeting of 07/04/05 suggested that ES does not consider this as
a requirement. It might be a best practice for the user to
register each user image with an update service (privileged as
the user, and not necessarily global).
Not sure how to proceed on the Windows requirement. Most of the current
design is portable across POSIX/ELF systems.

125
doc/pkg5_docs/facets.txt Normal file
View file

@ -0,0 +1,125 @@
Package facets and variants
---------------------------
Traditionally, packaging systems have placed optional components
of a package in separate packages, and established conventions
for naming such components, such as -localization-locale,
-devel, -doc, etc. This method of ad-hoc decomposition
makes it more difficult for GUI programs to offer the appropriate
choices when selecting components, makes the introduction of new
optional components difficult and makes installing documentation
after the fact a painful process.
Packaging options also exist which are mutually exclusive; the typical
example is which architecture the package supports. One cannot select
both sparc and x86, since the two architecture's files intersect or
collide. Other examples include debug vs non-debug binaries, and
global vs nonglobal zones.
In pkg(5), options that may be selected or not selected, such as various
locales, documentation, etc., are referred to as facets. Options which
are mutually exclusive are called variants. Both variants and facets
appear as tags on IPS actions, and result in the action being
selected or de-selected for installation. Some examples are:
Name values
--------------------------------------------
facet.locale.* true, false
facet.doc.man true, false
facet.doc true, false
facet.debug.* true, false
facet.devel.* true, false
facet.optional.* true, false
variant.arch sparc, i386, zos
variant.debug.* true, false
An action that is tagged w/ a facet or variant that is not selected
will be automatically excluded; actions w/o facets or variants are
always included. A single action may have multiple facet and variant
tags; an example would be an architecture-specific header file that's
used by developers:
file 8e7247b269fd116345abbf1aa9000a3d81ed871b chash=1fe53e8e2d0ad25bae13e1fd622c50397a2389ce group=bin mode=0644 owner=root path=usr/include/sys/regset.h variant.arch=x86 facet.devel.headers=true pkg.csize=4002 pkg.size=12457
This implies that facets and variants are evaluated ANDed together;
if any of the variant tags do not match, the action is not installed.
On the other hand, the facet tags are OR'd together; if any of the
facets match the action is installed.
Facets and variants are tags, and as such can initially be
set on any action, including dependencies, etc. This can make
testing problematic, however. To simplify matters, variants and
facets are set at the image level. Package developers desiring
fine grained control of their componentry are advised to use
unique facet tags.
In order to simplify grouping of facets, wildcarding is supported
when setting facets, but not variants. For example,
facet.doc.* matches facet.doc.man, facet.doc.info and facet.doc.html.
For ease of installation and backwards compatibility, facets that
are unspecified in the image are automatically included except for those
starting with 'facet.debug.' or 'facet.optional.'; for the same reasons,
any variant matching the name variant.debug.* is automatically set to
false. When multiple image facet patterns match, the longest match is
selected. For example, the image may have:
facet.locale.* false
facet.locale.en_US(utf8) true
Actions marked w/ facet.locale.de would match facet.locale.*
and thus not be installed, but actions matching facet.locale.en_US(utf8)
would match both patterns; since facet.locale.en_US(utf8) is
longer than facet.locale.* that logic would prevail. Note that
exact matches are always preferred.
A more useful example would be installing the french locale as
spoken in France. This consists of files tagged
facet.locale.fr, which tags files which should be installed for all
French locales,
and
facet.locale.fr_FR, which is for France in particular,
but not
facet.locale.fr_CA, which is for Canada.
Setting the following facets insures this selection:
facet.locale.* false # install only locales we specify
facet.locale.fr true
facet.locale.fr_FR true
Changing either variants or facets for an installed image
effectively causes re-evaluation of the actions in the installed
packages, and may or may not be done live depending on the impact
of the change.
Because of the need to select the appropriate variant types prior to
installation or parsing manifests, only variant.debug.* variants can
used with pkg(5) without making explicit changes to the source
code. Developers are encourged to design their components not to
intersect in the filesystem so that facets may be used rather than
variants.
Proposed facets and variants in initial implementation:
Name default comments
-----------------------------------------------------
facet.* true implements default "all facets are included"
facet.locale.* true should be set to false if individual locales are selected
facet.doc.info true
facet.doc.man true
facet.doc.html true
facet.devel true
facet.debug.<all> false
facet.optional.<all> false
facet.platform.sun4u true
facet.platform.sun4v true
variant.arch one of sparc, i386 as set by platform code
variant.opensolaris.zone either global or nonglobal as set by image type
variant.debug.<all> false
variant.<unknown> false all unknown variants default to 'false'

53
doc/pkg5_docs/file-metadata.txt Normal file
View file

@ -0,0 +1,53 @@
pkg
File and other metadata
We would like to be efficient with our storage and bandwidth
utilization. We would also like to avoid a class of inconsistency
errors preventable for a class of file objects (ELF binaries and
libraries, potentially also Java JAR files).
We have to label binaries and platform sensitive files (such as
endian-specific binary formats).
We also need to store ownership, modification and creation times, and
permissions. We may need to store ACLs of various kinds.
Some data is interpreted by the pkg server (to process a filter request
by the client), some by the pkg client (to establish times and
permissions).
pkg pkg_fmri keywords
file platform type path permissions user group type-specific-hash
link link-type dst_path src_path
dir path permissions user group
Envisioned types are "raw" and "elf". Potentially also "jar".
Envisioned platforms are "i86pc", "sun4". ISA distinctions are handled
by the filter options available on "elf" type files.
The leading column is called the *action*. If we do a driver action, it
would be multi-file (since the .conf file is associated). Drivers may
have to be expressed as a non-action. Actions are always reversible in
some fashion.
Example postinstall invocations of add_drv:
1. SUNWuedg
add_drv ${BASEDIR_OPT} -m '* 0666 root sys' \
./SUNWuedg/postinstall- -i "${USBSER_EDGE_ALIASES} \"${NEW_ALIAS}\"" -n usbser_edge
2. SUNWav1394
add_drv -b "${BASEDIR}" -m "${DRVPERM}" -i "${DRVALIAS}" -n ${DRV}
3. SUNWpd
/usr/sbin/add_drv -b ${BASEDIR} -i '"pnpALI,1533,3" "ns87317-ecpp"' \
./SUNWpd/postinstall- -m '* 0666 root sys' -n ecpp ||
4. SUNWlsimega
check_add_drv -i '"pci1028,13" "pci1000,407" "pci1000,407.1000.532" "pci1000,408" "pci1000,408.1000.2" "pci1000,1960"' -b "$BASEDIR" -c scsi lsimega

56
doc/pkg5_docs/filter.txt Normal file
View file

@ -0,0 +1,56 @@
pkg
FILTERING
We start with all actions in a manifest.
We apply each element in the filter chain in order. For each element, we
eliminate all remaining actions which don't match the element.
Simple elements have a single filter:
arch=i386
means eliminate all actions which have an "arch" attribute that isn't
"i386".
More complicated elements can represent an intersection of filters:
arch=i386 & debug=true
means eliminate all actions which have both "arch" and "debug" attributes,
but whose values, respectively, aren't "i386" and "true" (e.g., eliminate
all non-debug i386 actions and all sparc actions, debug and non-debug);
doc=true & locale=fr
means eliminate all actions which have both "doc" and "locale" attributes
and which aren't "true" and "fr", respectively (no, I'm not sure what
"doc=false" means, other than a reason to refactor). That is, strip out
all documentation that isn't French (but keep other localized actions, such
as message files, images, etc.).
Elements can also represent a union of filters:
locale=fr | locale=sv
means eliminate all actions which have a "locale" attribute that isn't
either "fr" or "sv".
We'll probably want to mix intersection and union, too:
doc=true & (locale=fr | locale=sv)
And we want a fallback mechanism, allowing us to get the "best of" a
particular attribute. Four examples:
locale=fr_FR.UTF-8;fr_FR;fr;C
platform=SUNW,Sun-Fire-V240;sun4u
debug=true;false
debug=false;true
Any actions that haven't been eliminated by the end of the chain remain to
be installed. At that point in time, if more than one action exists for
any given object, the evaluation fails (can't have two copies of
/usr/bin/ls just because you remembered to specify architecture but not
debug-ness).

173
doc/pkg5_docs/guide-basic-ops.rst Normal file
View file

@ -0,0 +1,173 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
.. _Basic operations:
|pkg5| basic operations
=======================
We provide a brief summary of the client operations and their invocation
via |pkg1|, the command line client. Use of ``PackageManager`` and
other graphical clients is covered by the *Image Packaging System User's
Guide* for |OS_Name|, or by user documentation of the software
component utilizing image packaging.
Listing installed and available packages
----------------------------------------
Since an image may include hundreds of installed packages, and publisher
repositories may contains hundreds or thousands more available packages,
we note that the ``list`` subcommand has a number of options for filtering
these sets of packages into specific groups. To see the set of packages
currently installed in an image, the plain subcommand is used::
/usr/bin/pkg list
With the ``-u`` option, ``list`` will restrict its output to those
installed packages for which upgrade--newer versions--are available.
This option is a convenient way to detect if newer software has been
released by one or more of the publishers you've subscribed to.
With the ``-a`` option, ``list`` will list all known packages for the
subscribed publishers.
Installing a package
--------------------
To install a package, we first need to know the name of the package.
Package names are part of a larger set of named entites, called
*fault-managed resources*, and referred to using Fault-Managed Resource
Identifiers (FMRIs). A package FMRI contains the name of the package's
publisher, a common label for the package, and a version specifier. For
instance, a valid package FMRI for the Image Packaging System is::
pkg://opensolaris.org/package/pkg@0.5.11-0.137
which identifies this instance of ``package/pkg`` as the package version
published by ``opensolaris.org`` for Build 137 of the development
version of the operating system.
|pkg1| has an abbreviation expansion algorithm so that shorter strings
than the full FMRI can be used to match package names for installation.
Once we know the FMRI, installation uses the ``install`` subcommand to
|pkg1|::
$ pfexec pkg install package/pkg
|pkg1| will calculate a valid version of ``package/pkg`` to install
within the image, and then display progress as it retrieves and executes
the actions that make up that version of the package.
Removing a package
------------------
Removal is the opposite of installation, so package removal is done via
the ``uninstall`` subcommand to |pkg1|. As an example, to remove the
HTML Tidy library::
$ pfexec pkg uninstall text/tidy
Searching for package content
-----------------------------
Image packaging supports both local and remote search. Local search is
confined to the packages installed within the image, while remote search
queries repositories associated with the active list of publishers.
The search facility is versatile, and can be used to query metadata,
dependencies, and contents for the union of the sets of packages the
publishers each offer.
Basic search is available via the ``search`` subcommand, with the ``-l``
option to keep the search local to the present image::
$ pkg search -l bash
INDEX ACTION VALUE PACKAGE
basename dir etc/bash pkg:/shell/bash@4.0.28-0.137
basename dir usr/share/bash pkg:/shell/bash@4.0.28-0.137
basename file usr/bin/bash pkg:/shell/bash@4.0.28-0.137
pkg.fmri set opensolaris.org/shell/bash pkg:/shell/bash@4.0.28-0.137
By default, the ``search`` subcommand searches remote repositories. In
this mode, it is useful for searching for specific filenames, such as
the name of an executable or an include file::
$ pkg search xpath.h
INDEX ACTION VALUE PACKAGE
basename file usr/include/libxml2/libxml/xpath.h pkg:/library/libxml2@2.7.6-0.139
Depot servers may also be queried from a web browser, by setting the
location to that of the running depot. A browser user interface is
provided for search results.
XXX More one search -p, specifically. Maybe also complex queries.
Definitely a longer explanation of search here or in the SAG.
Adding a publisher
------------------
Image packaging allows the retrieval and installation of software
package from a variety of publishers, each of whom may offer different,
interesting components. To add an additional publisher, use the
``set-publisher`` subcommand::
/usr/bin/pkg set-publisher ....
In some cases, a publisher's servers may be unavailable. Although
|pkg1| attempts to minimize the impact of downtime or unreachable
repositories, it can be simpler to
disable a publisher for the duration of the outage. ``set-publisher``
makes it easy to disable a publisher::
/usr/bin/pkg set-publisher --disable ....
Over time, a publisher may no longer offer software interesting for your
system. |pkg1| performance is generally not affected by additional
publisher entries, but it is good policy to trim unused configuration
from one's system. To remove a publisher, use ``pkg unset-publisher``::
/usr/bin/pkg unset-publisher ....
XXX .p5i file based publisher addition.
Initializing an image
---------------------
Although generally not needed for typical operations, when testing newly
created packages, it may be convenient to create a temporary image that
can be deleted after testing has been completed. To create an image,
use the ``image-create`` subcommand. An initial publisher must be
specified::
/usr/bin/pkg image-create ....
To delete an image, simply remove the directory containing the image.
On a typical |OS_Name| installation, the system image starts at the root
of the filesystem ('``/``'), with the packaging metadata stored in
``/var/pkg``.
|pkg1| attempts to determine which image to operate upon automatically,
by scanning its invocation directory. We can identify the target image
via the ``-R image_directory`` option to any |pkg1| invocation.

101
doc/pkg5_docs/guide-implementation-depot.rst Normal file
View file

@ -0,0 +1,101 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
When communicating with the depot server via HTTP, operations are
presented via a URL-based mechanism that allows each to be versioned so
that changes in protocol can be hidden from older clients as the
interfaces to operations evolve.
Operations made available by a |depotd1m| server can be accessed via
GET or POST, as appropriate for each operation, via a URL such as the
following:
http://pkg.opensolaris.org/release/manifest/0/SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z
The above example can be broken down into four basic components:
publisher_origin_url - http://pkg.opensolaris.org/release/
operation_name - manifest
protocol_version - 0
operation_arguments - SUNWvim%407.1.284%2C5.11-0.101%3A20081119T230659Z
Each of these components can be described as follows:
publisher_origin_url - A URL that can be used to access a depot
server's repository.
operation_name - The name of the operation that the client is
wanting to initiate.
protocol_version - An integer value representing the version of
the operation's protocol spoken by the client.
operation_arguments - String data (such as a package FMRI) that is
parsed and then used to determine what
resource(s) will be used to perform an
operation. Some operations expect arguments
or data to be passed via POST-based form data,
headers, or the request body instead.
Operation Types
---------------
Each operation that the depot server provides is either designed to interact
with a pkg(5) repository, or with the depot server itself. These operations
can be categorized as follows:
- content
These operations are read-only, and retrieve file data that comprises
the content of a package in a repository.
- depot
These operations are read-only, and permit retrieval of: the list of
operations that the depot server currently provides (including protocol
version and pkg(5) software version), statistics information, and other
depot information.
- metadata
These operations are read-only, and retrieve metadata related to a
package FMRI, such as its name, version, etc. stored in a repository's
catalog.
- publishing
These operations alter a repository's catalog, package metadata, and
allow storage of package content.
Modes
-----
Which types of operations are available is dependent on which mode the
depot server is currently operating in:
- default
In default mode, the depot server allows content, depot, metadata,
and publishing operations.
- readonly
In readonly mode, the depot server allows content, depot, and
metadata operations.
- mirror
In mirror mode, the depot server allows content and depot
operations.

945
doc/pkg5_docs/guide-main.rst Normal file
View file

@ -0,0 +1,945 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
=================================
Image Packaging Developer's Guide
=================================
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*PSARC/2008/190:* pkg(5): image packaging system
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:Authors: David Comay, Danek Duvall, Tim Foster, Stephen Hahn, Krister Johansen,
Dan Price, Brock Pytlik, Bart Smaalders, Shawn Walker
:Organization: http://hub.opensolaris.org/bin/view/Project+pkg/
:Contact: pkg-discuss@opensolaris.org
:Status: Interim draft
:Date: 2010-04-19
.. include:: macros.rst
.. sectnum::
.. contents:: Table of Contents
:depth: 3
--------
Overview
--------
.. section-autonumbering
:start: 1
:depth: 2
About this document
===================
.. admonition:: ARC Notice
For the purposes of review, this document--a draft of the
developer guide--contains additional information that may be omitted
from the final guide. Such information is generally set off from the
main text.
In this text, *The Image Packaging Developer's Guide*, we cover a range
of topics covering technical aspects of software delivery using |pkg5|,
the image packaging system. As you complete Parts I, II, and III of the
document, you should feel comfortable
* with basic |pkg5| principles of operation and use,
* with authoring packages, decorated with correct metadata and named
according to convention,
* renaming previously published packages, and
* managing simple and complex |depotd1m| deployments.
Parts IV and V focus on implementing components capable of replacing
the default image packaging retrieval clients and depot servers,
respectively.
Introduction
============
The Image Packaging System, or |pkg5|, is a software delivery mechanism
that enables the publication and retrieval of versioned, interdependent
software components. The goal of this mechanism is to always produce a
functioning image, in which each component in that image has its
dependencies on other components met in a self-consistent fashion.
Usually, when we refer to an image, we are referring to the *system
image* associated with an instance of the |OS_Name| operating system,
but an image can also be used to deliver an independent set of software
components, or a set linked in some fashion to yet another image.
Inter-image relationships allow |pkg5| to be used in a wide variety of
software deployment contexts.
The idea of a versioned component comes from recognizing that, in
general, software components change over time, as bugs are fixed, new
features introduced, component boundaries changed, or the component
itself becomes irrelevant. We can look at the history of a component as
a series of versions of that component, persisting indefinitely. In the
following chapters, the relationship over time of a component to other
components, and to itself, will be revisited. The intent is to
emphasize that each software component has a lifecycle, and that there
are representative states and operations in |pkg5| as a component moves
through that lifecycle.
There are a number of features in |pkg5| that make it particularly
appealing in a network context. In general, all |pkg5| metadata is
searchable remotely, easing the discovery of available software. |pkg5|
content delivery is bandwidth-efficient, sending only needed data to the
client, and always in a compressed form. Each publisher's depot
software has public information, which allows the identification of
other locations for software retrieval, such as mirrors for that depot
or publishers of related software. These features are discussed as they
apply to each of the components.
|pkg5|, although it has a command line client, was developed to support
both interactive use with other clients and programmatic use by higher
level management software. We generally illustrate operations by
reviewing the command line client. For developers and package
publishers, we will identify where package metadata, such as additional
information about the package or a specific object delivered by the
package, can influence these different client uses. We, in addition,
describe the more complex publication scenarios, such as signed
manifests and obsolescence. For operators and administrators, we
discuss security, in the form of access control, and depot management.
Our final sections focus on |pkg5| extensions, such as entitlement
support.
Key concepts
============
In this section, we attempt to review the key concepts and indicate
where further discussion and examples can be found in this Guide.
Packages may be installed into images; image contents are the
consequence of package installations.
Packages, package versions, actions, and actuators
--------------------------------------------------
Package
A package represents a set of related resources that can be
installed into a file system location on a computing system. In the
image packaging system, packages are distinguished by their package
names and by their package versions. Only a single version of a
package may be installed to a file system location.
Package names
Since the name of a package often is the easiest way to identify its
contents, the selection of a package name is an important step in
package publication. The set of package names, or *package
namespace* follows conventions to allow publishers to have uniquely
named packages. As an example, the package name for the image
packaging system is ``pkg://opensolaris.org/package/pkg``. Package
names can be matched on unique substrings, which is convenient for
interactive use, so we will often abbreviate package names to their
unique portion. Thus, for command line invocations, we will often
write ``pkg install package/pkg`` to update the image packaging
system's package.
Package versions
A package version is a multiple component object. Two versions of a
particular package can always be ordered via a comparison: they
will either be equal, or one will be less than the other. In the
image packaging system, a version consists of three sub-versions and
a timestamp. For instance, a recent version of ``package/pkg`` is
``0.5.11,5.11-0.138:20100430T013957Z``.
Package states
Each package version has a specific state in each image. For
instance a package version may be *installed* or merely *known* by
having an entry in a catalog.
See `Package States`_.
Package tags and file attributes
As briefly mentioned, packages and their content may have additional
information associated with them. There are both mandatory and
optional metadata items that package publisher can provide and
with which client software is expected to comply. Additionally,
specific publishers may wish to provide additional metadata for
their own use, or for use with the search facility.
Actions
The resources that a package delivers are called actions, as
they cause changes to the system on which the package is installed.
Actions that differ between the current and proposed package
versions are delivered; actions that do not differ are not
delivered.
There are a number of action types for delivering content to the
file system, such as ``file``, ``directory``, and ``link``, as well
as action types that deliver new system metadata, like the ``user``
action, which delivers a new or modified user account. The file
within the package that contains its set of actions is called a
*manifest*. An arbitrary amount of metadata may be added to any
action; such metadata items are called *tags*.
Actuators
A special kind of action tag is the actuator, which identifies a
side effect this action will have when installed on the in-use image
on a running system (a "live image"). Actuators have no effect on
non-live images. One typical actuator use is to request a restart
of a particular |smf5| service instance. This request is made by
specifying the FMRI as the value for the ``restart_fmri`` actuator.
Images and image types
----------------------
Image
We refer to a file system location configured to receive package
installations as an image. Each collection of installed package
instances is some kind of image. We envision three kinds:
- entire images
An entire image contains (and can contain) all appropriate
packages from a repository. It has a machine type and release.
- partial images
A partial image contains (and can contain) all appropriate
packages from a repository. An partial image is tied to an
entire image and has an identified list of non-writeable
directory paths. It inherits its machine type and release.
- user images
A user image consists entirely of relocatable packages. It is
tied to an entire image for dependency satisfaction of
non-relocatable packages. It inherits its machine type and
release.
Publishers, catalogs, and repositories
--------------------------------------
Publishers
Each package has a publisher, which represents the person or
organization that has assembled the package and makes it available
for installation.
Catalogs
Since a publisher can make one or more packages available, each
publisher generally provides a list of all packages and versions
currently available for retrieval. The file that contains this list
is called a *catalog*.
Repository
Although a package [will be able to] be distributed independently,
publishers with many packages may find it easier to publish that
collection to a repository, which is a structured set of directories
and files that contains the packages, their metadata, and their
contents. A *depot* is a network server that can respond to
requests for specific operations on [one or more] repositories.
Dependencies and constraints
----------------------------
Packages can have relationships with other packages, such as a
``require`` relationship, where the presence of a package is mandatory
for a second package to be installed. Each declaration of a
relationship in a package's metadata is done using a ``depend`` action,
which expresses the relationship as a *dependency*. Dependency types
include
``require``
A require dependency states that the package mentioned by the
``fmri`` attribute of the ``depend`` action must also be installed
into the image, with version at or above the specified version. If
no version is given, any version is allowed.
``optional``
An optional dependency states that the package mentioned by the
``fmri`` attribute, if a version is installed in the image, that
version must be at or above the specified version.
``exclude``
An exclude dependency states that, if the package version or a
successor mentioned by the ``fmri`` attribute is installed in the
image, the current package (containing the ``depend`` action) cannot
be installed.
``incorporate``
An incorporation dependency expresses a special type of
relationship, where the presence of the dependent package is
constrained to the version range implied by the ``fmri`` portion of
the ``depend`` action. Incorporation dependencies are used to
restrict the set of package versions to a known and, presumably,
tested subset of all available versions.
``require-any``
A require-any dependency acts as a require dependency, except that
multiple fmris may be specified, and any one being present will satisfy
the dependency.
``conditional``
A conditional dependency states that the package mentioned by the
``fmri`` attribute of the ``depend`` action must also be installed
into the image, with version at or above the specified version if the
package named by the ``predicate`` attribute is present on the system
at the specified level or higher.
``origin``
A origin dependency states that the the package mentioned by
the ``fmri`` attribute must be at the specified level or higher (if present)
prior to installation of the containing package. If the attribute ``root-image``
is present and has the value ``true``, the image that is checked is the
image rooted at '/'.
|OS_Name|-specific concepts
-----------------------------
Migration and compatibility
|pkg5| supplants the historical Solaris Operating System packaging
tools, although these are still supported for compatibility. Moving
a software component to the Image Packaging System requires some
planning.
|smf5| configuration transition
Certain resource types, such as manual pages and desktop icons, are
delivered by multiple applications. New |smf5| service instances
have been provided to simplify the integration of standard resources.
Application configuration
For some applications, specific operations must be taken after
package installation or removal. In addition to general techniques
for handling such operations, we review specific utilities and
services introduced for handling typical cases.
Commands
========
- **Retrieval clients.** |pkg1|, PackageManager. UpdateManager.
- use of SSL
- relationship with |beadm1m|, |libbe3lib|, ZFS
- **Publication and manipulation clients.** |pkgsend1| and |pkgrecv1|
- |pkgdepend1|, |pkgdiff1|, |pkgfmt1|, |pkgmogrify1|
- authentication
- **Depot servers.** |depotd1m|
- reverse proxy
- horizontal use
Other commands affected.
.. include:: guide-basic-ops.rst
---------------------------------
Authoring and publishing packages
---------------------------------
.. section-autonumbering
:start: 1
Getting started
===============
Pick a publisher name, based on a DNS domain you control.
Running a depot
---------------
An |smf5| service instance of ``pkg/server`` is provided with a default
OpenSolaris installation.
set publisher name
set read-write to enabled
Publishing to files
-------------------
Transaction basics
==================
If we examine the process of publishing a particular version of a
package, we can see there are three sets of decisions:
* package contents, or what files or other resources are delivered by
this version of the package;
* package name, or what the package is called; and
* package metadata, or what additional information is provided to
describe the package's purpose within a larger system.
Before we discuss these topics, we review how publication works.
.. admonition:: ARC notice
Additional options for access control to publication operations are
under consideration.
To publish a package, we use the |pkgsend1| command to open a transaction
with an active package depot, to send actions, which are the resources
and metadata delivered by the package, and finally to close the
transaction. Upon the request for the transaction to be closed, the
depot daemon will, for a valid submission, update its catalog and search
indices to reflect the newly published package version. Clients, such
as |pkg1|, can then refresh their local catalogs and retrieve the new
package version.
Let's work through the above description in the form of an example.
In the following example, we wish to publish a simple package that
provides the ``/etc/release`` file, which, on a typical |OS_Name|
system, contains some descriptive information about this particular
release of the operating system.
.. include:: guide-txn-states.rst
Supported Actions
-----------------
|pkg5| supports an extensible set of "actions", which are defined as
reversible operations that a package can request to enable its later
function on the target image.
.. admonition:: ARC notice
Packages need a limited set of operations on individual files to
manipulate the configuration. The current class actions are given in
Appendix A. It appears that if "manifest" and "rbac" were supported,
along with some management of editable files (preserve, renamenew,
initd, renameold), then the remaining operations could be deferred to
image instantiation.
We can analyze the typical delivered components of a working operating
system to identify the set of common actions. The decision to provide
an action for a specific class of resource is strongly influenced by the
need for elements of that class to be fully configured for system boot to
complete. Resources that can be configured after initial boot are
generally not provided with actions, and are expected to use ``file``
actions to deliver content and an |smf5| service to derive and assemble
any necessary local configuration.
depend
Declare dependency on other packages.
directory
All directories.
driver
Package contains device driver Module loading will be disabled
during operations on live images.
file
All other files. Preservation and rename handling are managed as
optional tags.
hardlink, link
All hard and symbolic links.
set
Set a package tag.
user, group
Package requires user, group, or other package-reference managed
resource.
legacy
Record package attributes into legacy packaging metadata.
license
License files, which deliver into the image metadata rather than
the image's filesystems.
signature
Deliver a cryptographic signature for the containing manifest.
Interface summary::
<interface>
<action name="dependency" payload="false" commitment="Committed" />
<action name="directory" payload="false" commitment="Committed" />
<action name="hardlink" payload="false" commitment="Committed" />
<action name="legacy" payload="false" commitment="Committed" />
<action name="license" payload="true" commitment="Committed" />
<action name="link" payload="false" commitment="Committed" />
<action name="driver" payload="false" commitment="Committed" />
<action name="file" payload="true" commitment="Committed" />
<action name="group" payload="false" commitment="Committed" />
<action name="user" payload="false" commitment="Committed" />
<action name="signature" payload="false" commitment="Committed" />
</interface>
.. sidebar:: Custom actions
It is discouraged, but certainly possible to deliver custom actions
into the appropriate ``$PYTHONROOT/vendor-packages/pkg directory``, by
including those actions in a separate package that the new package
requires, and invoking the |pkg1| client twice--once to deliver the
custom actions and once to use them to install the new package.
(Rescanning pkg.actions would complicate the image plan/package plan
evaluations.)
XXX The deployer may wish to deny such actions from operating. For this
case, the set of known actions is fixed elsewhere in the pkg modules
and updated with subsequent versions. A global and per-image policy,
known-actions-only, allows the deployer to disallow operations on
packages utilizing actions of unknown provenance.
Interface::
<interface>
<policy name="known-actions-only" scope="global,image"
type="boolean" commitment="Committed">
Deployer control over execution of unknown actions.
</policy>
<interface>
Actuators
~~~~~~~~~
Reboot necessity.
Those system configuration steps which can be deferred.
Variants and facets
~~~~~~~~~~~~~~~~~~~
Packaging considerations
========================
Many of the good packaging criteria present trade-offs among themselves. It
will often be difficult to satisfy all requirements equally. These criteria are
presented in order of importance; however, this sequence is meant to serve as a
flexible guide depending on the circumstances. Although each of these criteria
is important, it is up to you to optimize these requirements to produce a good
set of packages.
Optimize for Client-Server Configurations
-----------------------------------------
You should consider the various patterns of software use
(client and server) when laying out packages. Good packaging
design divides the affected files to optimize installation of each
configuration type. For example, for a network protocol implementation,
it should be possible to install the client without necessarily
installing the server.
Package by Functional Boundaries
--------------------------------
Packages should be self-contained and distinctly identified with a set of
functionality. For example, a package containing UFS should contain all UFS
utilities and be limited to only UFS binaries.
Packages should be organized from a customer's point of view into functional
units.
Package Along License or Royalty Boundaries
-------------------------------------------
Put code that requires royalty payments due to contractual agreements or
that has distinct software license terms in a dedicated package or group
of packages. Do not to disperse the code into more packages than
necessary.
Overlap in Packages
-------------------
When constructing the packages, ensure that duplicate files are eliminated when
possible. Unnecessary duplication of files results in support and version
difficulties. If your product has multiple packages, constantly compare the
contents of these packages for redundancies.
Sizing Considerations
---------------------
Size is package-specific and depends on other criteria. For example, the
maximum size of /opt should be considered. When possible, a good package should
not contain only one or two files or contain extremely large numbers of files.
There are cases where a smaller or larger package might be appropriate to
satisfy other criteria.
Licensing Considerations for Packages
-------------------------------------
If you are distributing software that uses licensing, there are several things
you need to consider:
- Business operations
- Communication with users
- Technology
*Business Operations.* Before you begin distributing licensed software, set up
your business operations to distribute, price, and track licenses. There are a
variety of ways to distribute licenses, such as fax, electronic mail, or an 800
telephone number. You need to choose a method of distribution and set up all
the necessary processes. You also need to consider whether licenses need to be
upgraded with the software and how this will be done.
Pricing policy and types of licenses must also be considered. You must consider
how the product is used and what kinds of licenses your users will need to use
the product effectively. Single user licenses may not be appropriate for many
situations.
*Communication with Users.* Before you implement licensing, you need to inform
your users, particularly if the product has not been licensed in the past.
When you do implement licensing, you may want to consider implementing it
gradually. The first step would be monitoring the use of licenses, followed by
warning that the software is being used without a license, and finally, denying
the use of the software.
*Technology.* If you are going to use a commercial product for licensing, there
are many things to consider when making your choice. You need to decide what
your priorities are. For example, is ease of administration and use most
important? Or is enforcing the licensing policy more important?
You also need to consider whether the software will be used in a heterogeneous
or homogeneous environment and whether standards are important. You may also
want to look at the security provided by the product. Is it easy to get around
the enforcement of licenses?
The issues involved in choosing a commercial product will vary depending on
the kind of application and your reasons for implementing licensing.
Naming your package
===================
.. include:: guide-naming-conventions.rst
Decorating your package
=======================
.. include:: guide-metadata-conventions.rst
A full packaging example
========================
Delivery examples
=================
In the following sections, we give examples of delivering specific
resources via the image packaging system.
Device driver
-------------
XXX ``e1000g``
Versioned interpreter
---------------------
XXX perl
XXX ``verexec``
|smf5| Service
--------------
XXX pkg.depotd
GNOME Desktop Elements
----------------------
XXX pick a specific Gnome application
Coordinating groups of packages using incorporations
----------------------------------------------------
Renaming a package
------------------
Making a package version obsolete
---------------------------------
Moving a file between packages
------------------------------
Migrating an existing package
=============================
Migrating a System V Package
----------------------------
XXX pkgsend
Migrating a |tar1| archive
--------------------------
XXX pkgsend
Publishing from an installation directory
-----------------------------------------
Pre-publication tools
=====================
pkgmogrify, pkgdepend, pkgdiff, and pkgfmt.
----------------
Depot operations
----------------
Distributing packages with a depot
==================================
XXX thread tuning
Using Apache HTTPD as a reverse proxy cache
-------------------------------------------
recommended
can speed up operations
Running a content mirror
------------------------
From DVD
Via rsync
Long-term operations
--------------------
Splitting and spreading package retrieval load
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
can be load balanced
Tuning search
~~~~~~~~~~~~~
Publishing packages with a depot
================================
running in r/w mode
XXX could we automate snapshots?
----------------
Packaging client
----------------
.. section-autonumbering
:start: 1
.. include:: guide-pkg-states.rst
- protocol / network format
- client side REST API
- publication side REST API
Retrieval protocol operations
=============================
Publication protocol operations
===============================
.. include:: guide-publication-protocol.rst
Other protocol operations
=========================
- versions
Version 0
A GET operation that retrieves the current set of operations
offered by the contacted depot.
Example:
URL: http://pkg.opensolaris.org/versions/0
Expects:
No body.
Returns:
List of operations and versions, one operation per line, space
separated list of versions for each operation.
- search
Version 1
A GET operation that presents a query to the search index
capability of the contacted depot.
--------------------------------
Package depots and other servers
--------------------------------
.. section-autonumbering
:start: 1
.. include:: guide-repository-format.rst
|depotd1m| implementation
=========================
.. include:: guide-implementation-depot.rst
--------------------------------
Appendix: Reference manual pages
--------------------------------
pkg(1)
======
.. raw:: html
<pre>
.. raw:: html
:file: ../src/man/pkg.1
.. raw:: html
</pre>
.. raw:: latex
\begin{verbatim}
.. raw:: latex
:file: ../src/man/pkg.1
.. raw:: latex
\end{verbatim}
pkgrecv(1)
==========
.. raw:: html
<pre>
.. raw:: html
:file: ../src/man/pkgrecv.1
.. raw:: html
</pre>
.. raw:: latex
\begin{verbatim}
.. raw:: latex
:file: ../src/man/pkgrecv.1
.. raw:: latex
\end{verbatim}
pkgsend(1)
==========
.. raw:: html
<pre>
.. raw:: html
:file: ../src/man/pkgsend.1
.. raw:: html
</pre>
.. raw:: latex
\begin{verbatim}
.. raw:: latex
:file: ../src/man/pkgsend.1
.. raw:: latex
\end{verbatim}
pkg.depotd(1M)
==============
.. raw:: html
<pre>
.. raw:: html
:file: ../src/man/pkg.depotd.1m
.. raw:: html
</pre>
.. raw:: latex
\begin{verbatim}
.. raw:: latex
:file: ../src/man/pkg.depotd.1m
.. raw:: latex
\end{verbatim}
pkg(5)
======
.. raw:: html
<pre>
.. raw:: html
:file: ../src/man/pkg.5
.. raw:: html
</pre>
.. raw:: latex
\begin{verbatim}
.. raw:: latex
:file: ../src/man/pkg.5
.. raw:: latex
\end{verbatim}
---------------------------
Appendix: Protocol details
---------------------------
------------------------------------------
Appendix: Architectural process materials
------------------------------------------
.. raw:: html
<pre>
.. raw:: html
:file: one-pager-main.txt
.. raw:: html
</pre>

329
doc/pkg5_docs/guide-metadata-conventions.rst Normal file
View file

@ -0,0 +1,329 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Tags and attributes
-------------------
Definitions
~~~~~~~~~~~
Both packages and actions within a package can carry metadata, which
we informally refer to as attributes and tags. Both attributes and
tags have a name and one or more values.
Attributes
settings that apply to an entire package. Introduction
of an attribute that causes different deliveries by the client could
cause a conflict with the versioning algebra, so we attempt to avoid
them.
Tags
settings that affect individual files within a package.
Attribute and tag syntax and namespace
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Syntax
``````
Naming
^^^^^^
The syntax for attributes and tags is similar to that used for
|pkg5| and |smf5| FMRIs.
[org_prefix,][name][:locale]
The organizational prefix is a forward-ordered or reverse-ordered
domain name, followed by a comma. The name field is an identifier
which may have a prefix ending in a period to allocate the namespace.
If the locale field is omitted, the default locale is "C", a 7-bit
ASCII locale.
Each of these fields is [A-Za-z][A-Za-z0-9\_-.]*.
Manifests
^^^^^^^^^
In package manifests, the syntax for an attribute is:
set name=<attribute name> value=<value> [value=<value2> ...]
In package manifests, tags are included in the action line
for the action they apply to:
<action> [...] <tag name>=<tag value> [<tag name>=<tag value2> ...]
Unprefixed attributes and tags
``````````````````````````````
All unprefixed attributes and tags are reserved for use by the
framework.
Generally, unprefixed tags are introduced in the definition of an
action.
Attributes and tags common to all packages
``````````````````````````````````````````
Attributes and tags starting with "pkg." or "info." are for attributes
common to all packages, regardless of which particular OS platforms that
a specific package may target. "pkg" attributes are used by the
packaging system itself, while "info" attributes are purely informational,
possibly for use by other software.
Common attributes
^^^^^^^^^^^^^^^^^
pkg.summary
A short, descriptive name for the package.
pkg.summary:fr would be the descriptive name in French.
Exact numerical version strings are discouraged in the
descriptive name.
Example: "Apache HTTPD Web Server 2.x"
pkg.description
A descriptive paragraph for the package. Exact numerical version
strings can be embedded in the paragraph.
pkg.detailed-url
One or more URLs to pages or sites with further information about
the package. pkg.detailed-url:fr would be the URL to a page with
information in French.
pkg.renamed
A value of "true" indicates the package has been renamed or split
into the packages listed in the depend actions.
pkg.obsolete
A value of "true" indicates the package is obsolete and should be
removed on upgrade.
pkg.human-version
For components whose upstream version isn't a dot-separated sequence
of nonnegative integers (OpenSSL's 0.9.8r, for example), this
attribute can be set to that string, and will be displayed when
appropriate. It cannot be used in an FMRI to install a particular
version; package authors must still convert the version into a
sequence of integers.
variant.*
See facets.txt
Common tags
^^^^^^^^^^^
disable_fmri
See "Actuators" section of |pkg5|
facet.*
See facets.txt
reboot-needed
See "Actuators" section of |pkg5|
refresh_fmri
See "Actuators" section of |pkg5|
restart_fmri
See "Actuators" section of |pkg5|
suspend_fmri
See "Actuators" section of |pkg5|
variant.*
See facets.txt
Informational attributes
^^^^^^^^^^^^^^^^^^^^^^^^
The following attributes are not necessary for correct package installation,
but having a shared convention lowers confusion between publishers and
users.
info.classification
A list of labels classifying the package into the categories
shared among |pkg5| graphical clients.
Values currently used for OpenSolaris are prefixed with
``org.opensolaris.category.2008:`` and must match one of the
categories listed in ``src/gui/data/opensolaris.org.sections``
info.keyword
A list of additional terms that should cause this package to be
returned by a search.
info.maintainer
A human readable string describing the entity providing the
package. For an individual, this string is expected to be their
name, or name and email.
info.maintainer-url
A URL associated with the entity providing the package.
info.upstream
A human readable string describing the entity that creates the
software. For an individual, this string is expected to be
their name, or name and email.
info.upstream-url
A URL associated with the entity that creates the
software delivered within the package.
info.source-url
A URL to the source code bundle, if appropriate, for the package.
info.repository-url
A URL to the source code repository, if appropriate, for the
package.
info.repository-changeset
A changeset ID for the version of the source code contained in
info.repository-url.
Attributes common to all packages for an OS platform
````````````````````````````````````````````````````
Each OS platform is expected to define a string representing that
platform. For example, the |OS_Name| platform is represented by
the string "opensolaris".
OpenSolaris attributes
^^^^^^^^^^^^^^^^^^^^^^
org.opensolaris.arc-caseid
One or more case identifiers (e.g., PSARC/2008/190) associated with
the ARC case or cases associated with the component(s) delivered by
the package.
org.opensolaris.smf.fmri
One or more FMRI's representing SMF services delivered by this
package. Automatically generated by |pkgdepend1| for packages
containing SMF service manifests.
opensolaris.zone
Obsolete - replaced by variant.opensolaris.zone.
variant.opensolaris.zone
See facets.txt
OpenSolaris tags
^^^^^^^^^^^^^^^^
opensolaris.zone
Obsolete - replaced by variant.opensolaris.zone.
variant.opensolaris.zone
See facets.txt
Organization specific attributes
````````````````````````````````
Organizations wishing to provide a package with additional metadata
or to amend an existing package's metadata (in a repository that
they have control over) must use an organization-specific prefix.
For example, a service organization might introduce
``service.example.com,support-level`` or
``com.example.service,support-level`` to describe a level of support
for a package and its contents.
Attributes specific to certain types of actions
```````````````````````````````````````````````
Each type of action also has specific attributes covered in the
documentation of those actions. These are generally documented
in the section of the |pkg5| manual page for that action.
Attributes specific to certain types of file
````````````````````````````````````````````
These would generally appear on file actions for files in a specific
format.
elfarch, elfbits, elfhash
Data about ELF format binary files (may be renamed in the future
to info.file.elf.*). Automatically generated during package
publication. See the "File Actions" section of |pkg5|.
info.file.font.name
The name of a font contained in a given file. There may be multiple
values per file for formats which collect multiple typefaces into a
single file, such as .ttc (TrueType Collections), or for font aliases.
May also be provided in localized variants, such as a Chinese font
providing both info.file.font.name:en and info.file.font.name:zh for
the English and Chinese names for the font.
info.file.font.xlfd
An X Logical Font Description (XLFD) for a font contained in a
given file. Should match an XLFD listed in fonts.dir or fonts.alias
for the file. There may be multiple values per file due to font
aliases.
.. 3.3. Attributes best avoided
.. built-on release
.. One problem we may run into is packages that have been built on a
release newer than that on the image. These packages should be
evaluated as unsuitable for the image, and not offered in the graph.
There are a few ways to handle this case:
.. 1. Separate repository. All packages offered by a repository were
built on a known system configuration. This change requires
negotiation between client and server for a built-on match
condition. It also means that multiple repositories are needed
for a long lifecycle distribution.
.. 2. Attributes. Each package comes with a built-on attribute. This
means that clients move from one built-on release to another
triggered by conditions set by the base package on the client.
Another drawback is that it becomes impossible to request a
specific package by an FMRI, without additional knowledge.
.. 3. Additional version specifier. We could extend
release,branch:timestamp to release,built,branch:timestamp--or
fold the built and branch version together. Since the built
portion must reserve major.minor.micro, that means we move to a
package FMRI that looks like
.. coreutils@6.7,5.11.0.1:timestamp
.. This choice looks awkward. We could instead treat the built
portion as having a default value on a particular client. Then
the common specifier would be
.. name@release[,build]-branch:timestamp
.. build would be the highest available valid release for the
image.
.. The meaning of the built-on version could be controversial. A
simple approach would be to base it on base/minimal's release,
rather than uname(1) output.

184
doc/pkg5_docs/guide-naming-conventions.rst Normal file
View file

@ -0,0 +1,184 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Package naming conventions
--------------------------
Definitions
~~~~~~~~~~~
To be consistent with the system, following the introduction of the
fault management architecture, each package is named by an FMRI in the
``pkg:`` scheme. That is, we have::
pkg://publisher/pkg_name@version
The publisher is generally expected to be a forward or reverse domain
name identifying the publisher from which a package can be retrieved.
Publishers which cannot be determined to be a domain name are
legitimate, but optional functionality, like automatic server discovery
for a particular publisher, may fail to work. In the examples that
follow, we use "opensolaris.org" as a generic publisher.
Although RFC 2396 usage would suggest using the "authority" term, we
instead call it the publisher name, as the role of this section of the
FMRI is to identify the source for the package, which we consider a
publication of one or more software components.
The pkg_name, like service names, can be split into a category,
subcategories, and a basename. This namespace might be populated
with "manifest" and other metadata endpoints, as well as the SHA-1
names of the package's included files. (Although the direct access
to properties of the ``svc`` FMRI scheme has been rarely used.)
A "group package" is a package that depends upon (minimum versions
of) other packages, as well as optionally delivering files or other
actions. An "incorporating package" is a group package that places
forward constraints upon the versions of the packages it depends upon,
which restricts the interval of valid package versions to one the author
of the incorporation believes functional.
Namespace
~~~~~~~~~
Single namespace, separate publishers
``````````````````````````````````````
The primary design point of the package namespace is to allow
multiple package producers to co-exist in a single namespace, so
that images can switch between equivalent components from different
producers.
Domain-name-based escape
````````````````````````
At any point in the category hierarchy, a safe namespace can be created
by using the forward or reverse domain name, either as a subcategory or
as a comma-led prefix to a subcategory or package base name. (This
scheme is similar to FMRI namespace escapes in smf(5), although we are
eliminating use of stock symbol prefixes.) Generally, safe namespaces
are only needed when the components delivered by a package need to be
distinguished from those delivered by a package in the single namespace;
reasons for distinguishing the components might include construction of
a collection of components, insulated from changes in the wider system,
or for legal reasons.
For instance, when example.com wishes to publish the "whiskers"
package without reference to a larger namespace convention it can
use any of the following examples::
pkg://opensolaris.org/.../com.example/whiskers
pkg://opensolaris.org/.../com.example,whiskers
pkg://opensolaris.org/.../com.example,software/whiskers
and so forth.
Locally reserved namespace
``````````````````````````
The top-level "site" category is reserved for use by the entity that
administrates the server. Neither the organizations producing the
operating system nor those providing additional software components may
use the site category.
The top-level "vendor" category is reserved for use by organizations
providing additional. The leading subcategory must be a domain. That
is, if example.com wishes to publish the "whiskers" package in the
vendor category, it would use a package name like::
pkg://opensolaris.org/vendor/example.com/whiskers
Use of the "vendor" category, and the vendor-escaped packages below, is
likely only to be of interest to organizations that redistribute
software from multiple source vendors.
Additional reserved namespace
`````````````````````````````
.. admonition:: ARC Notice
Some or all of these reservations may be eliminated or reduced when
the single namespace convention reaches its final form.
The top-level "cluster", "feature", "group", and "metacluster"
categories are all reserved for future use.
Single namespace conventions
````````````````````````````
Discussion
^^^^^^^^^^
Packaging systems and distributions appear to have explicit
categories, subcategories, and potentially larger groups; some
distributions have explicit fields for these values, others use
tagging or multi-valued fields, which allows them to classify
certain packages multiply. For the FMRI namespace case, the system
is similar to a packaging system with multiple, single-valued,
category fields.
There appear to be two standard approaches to categorizing packages:
1. By what primary class of thing a package delivers.
2. By what area of functionality a package's contents address.
In the first approach, we get strict top-level categories like
"application", "driver", "library", and "system" or "kernel", as
well as potentially overlapping categories like "utility" and
"tool". Use of the leading subcategory is limited, and generally
given to the subsystem involved. A relatively detailed worked
example of the X11 subsystem under this scheme is given in
http://mail.opensolaris.org/pipermail/pkg-discuss/2008-February/001838.html
XXX This reference is now wrong.
In the second, we would also see categories like these, but leading
subcategory is much more likely to classify according to
functionality, so that we would see "application/mail",
"application/web", "application/compiler", and so forth. Most
network packaging systems appear to classify in this fashion.
An appealing variation of the second form is to rotate all of the
non-"application" packages under a "system" mega-category, such that
all of the leaf packages (with the possible exception of device
drivers) are exposed at the category level. Table 1 shows some
example transformations under this scheme.
.. table:: Rotating non-application categories under system.
========================= ====================
FROM TO
========================= ====================
application/web/firefox web/firefox
application/compiler/gcc4 compiler/gcc4
library/c system/library/c
kernel/generic system/kernel/generic
========================= ====================

110
doc/pkg5_docs/guide-pkg-states.rst Normal file
View file

@ -0,0 +1,110 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
.. :vim set expandtab:
.. _Package States:
Package states
==============
By a *package state*, we mean an image's recorded state for a particular
package version. We might also refer to these as "client states".
Within an image, each package version can have only one state, as given
in the following diagram::
0
|
v
IN_CATALOG ---------> OUT_CATALOG
| ^
+--->---+---<-------+ |
| | | |
| v | |
| INSTALLED --> FROZEN |
| | |
| | |
| v |
+-- PRESERVED |
| | |
| | |
| v |
+-- DELETED -------------------+
0 -> IN_CATALOG:
A catalog update with new entries.
IN_CATALOG:
An entry for this package is available in the locally installed
catalog.
IN_CATALOG -> OUT_CATALOG:
Entry formerly present on local catalog is no longer published by any
repository. (Package never locally installed.)
OUT_CATALOG:
Although a formerly known package, no entry for this package is
available in the locally installed catalog. An INSTALLED or
FROZEN package can never be OUT_CATALOG, as the system will
preserve the entry until the package is no longer in a locally
public state.
IN_CATALOG -> INSTALLED:
Transition takes place on package installation.
INSTALLED -> FROZEN:
Transition takes place if manually frozen or frozen by virtue of
reference from another package group.
FROZEN -> INSTALLED:
Manually unfrozen, or unfrozen by reference drop due to
change in formerly referring package group.
INSTALLED -> PRESERVED:
Old copies moved aside during upgrade of package components, but
not removed.
PRESERVED -> DELETED:
Old copies removed.
DELETED -> OUT_CATALOG:
Package has been removed from client catalog. Client software
would take a PRESERVED package through DELETED automatically to
reach OUT_CATALOG.
PRESERVED -> INSTALLED:
Package reinstalled or reversed.
DELETED -> INSTALLED:
Package reinstalled.
XXX How does the ZFS snapshot (that we might have taken prior to an
operation) get represented in the state? Is there an image state
machine model as well?
XXX Need a substate of INSTALLED for damaged packages.
XXX Need a substate of INSTALLED for packages where the global zone
portion is available, but local installation has not finished. Can
we generalize this state for all diskless installs?

104
doc/pkg5_docs/guide-publication-protocol.rst Normal file
View file

@ -0,0 +1,104 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
- add
Version 0:
A POST operation that adds content to an in-flight transaction for
the Transaction ID specified. This could either be file content
for the package or metadata about the package.
This data is not added to the repository for retrieval until a close
operation for the specified Transaction ID is executed.
Example:
URL:
http://pkg.opensolaris.org/add/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
HEADERS:
X-IPkg-SetAttr1: description=Package Name
REQUEST BODY:
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
The file content (if applicable), to be added, in the request
body. Any attributes to be set in the headers in the pattern
of:
X-IPkg-SetAttr{integer}: attr=value
Returns:
Response status of 200 on success; any other status indicates
failure.
- abandon
Version 0:
A GET operation that aborts an in-flight transaction for the
Transaction ID specified. This will discard any data related to
the transaction.
Example:
URL:
http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
Returns:
Response status of 200 on success; any other status indicates
failure.
- close
Version 0:
A GET operation that ends an in-flight transaction for the
Transaction ID specified. If successful, the corresponding package
is added to the repository catalog and is immediately available to
repository users.
Example:
URL:
http://pkg.opensolaris.org/abandon/0/1228870796_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081210T005956Z
Expects:
A Transaction ID as output by pkgsend(1) in the request path.
Returns:
Response status of 200 on success; any other status indicates
failure.
- open
Version 0:
A GET operation that starts an in-flight transaction for the
package FMRI specified.
Example:
URL:
http://pkg.opensolaris.org/open/0/system%2Flibc@0.1-98
Expects:
A URL-encoded pkg(5) FMRI (excluding timestamp).
Returns:
Response status of 200 on success and an identifier for the new
transaction in the 'Transaction-ID' response header; any other
status indicates failure.

106
doc/pkg5_docs/guide-repository-format.rst Normal file
View file

@ -0,0 +1,106 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
.. _Repository file system layout:
Repository file system layout
=============================
File system Layout
------------------
The types of information that the depot server stores and/or retrieves can
be categorized as follows:
- depot data
This includes: configuration data, presentation content (such as
web page templates), publishing data (e.g. in-flight transactions),
and temporary data (e.g. the feed cache).
- repository data
This includes: catalog information, package content (files), package
metadata (manifests), and search data.
Layout
~~~~~~
The depot server uses the following 'root' directory structures for the
storage and retrieval of depot and repository data:
- repo_dir (depot and repository data)
cfg_cache (depot data)
A file containing the cached configuration information for the
depot server.
catalog/ (repository data)
This directory contains the repository catalog and its related
metadata.
file/ (repository data)
This directory contains the file content of packages in the
repository.
Files are stored using a two-level path fragment, derived from the
SHA1-hash of a file's content, assumed to have at least 8 distinct
characters.
Example:
00/
0023bb/
000023bb53fdc7bcf35e62b7b0b353a56d36a504
index/ (repository data)
This directory contains the search indices for the repository.
pkg/ (repository data)
This directory contains the metadata (manifests) for the
repository's packages.
The manifests for each package are stored in a directory with the
same name as the package stem using a URL-encoded filename.
Example:
entire/
0.5.11%2C5.11-0.86%3A20080422T234219Z
trans/ (depot data)
This directory contains in-flight transactions for packages that
are waiting for the publication process to complete so that they
can be added to the repository's catalog.
Each transaction is stored in a directory named after the pending
transaction id and contains the manifest waiting for publication
to finish stored with the filename of 'manifest'.
Example:
1229379580_pkg%3A%2Fsystem%2Flibc%400.1%2C5.11-98%3A20081215T221940Z/
manifest
updatelog/ (repository data)
This directory contains metadata detailing changes to the repository
by publishing operations.
- content_root (depot data)
web/
This directory contains all of the web presentation content for the
depot.

228
doc/pkg5_docs/guide-retrieval-protocol.rst Normal file
View file

@ -0,0 +1,228 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Depot Operations
----------------
- versions
Version 0:
A GET operation that retrieves text data representing what operations
are supported and version information about the depot server.
Example:
URL:
http://pkg.opensolaris.org/versions/0/
Expects:
Nothing
Returns:
text/plain data containing the version of the pkg(5) software that
the depot is based upon, a list of the operations currently
supported, and the protocol version supported for each
operation.
Sample Output:
pkg-server bfc04991436e
info 0
search 0
versions 0
catalog 0
manifest 0
add 0
file 0
abandon 0
close 0
open 0
Metadata Operations
-------------------
- catalog
Version 0:
A GET operation that retrieves a text/plain datastream
representing a complete catalog or an incremental update to an
existing one as requested.
Example:
URL:
http://pkg.opensolaris.org/catalog/0/
Expects:
Nothing or the following headers:
If-Modified-Since: {ISO 8601 formatted date and time in UTC}
Returns:
Either the contents of a pkg(5) catalog file, or the entries
that were added since the specified date as they are found
in the catalog file, separated by newlines.
- info
Version 0:
A GET operation that retrieves a text/plain description of a
package and its licensing information specified by the provided
FMRI.
Example:
URL:
http://pkg.opensolaris.org/info/0/entire@0.5.11,5.11-0.101:20081119T235706Z
Expects:
A URL-encoded pkg(5) FMRI, excluding the 'pkg:/' scheme prefix
and publisher information, and including the full version
information.
Returns:
A text/plain representation of the specified package and its
licensing information.
Sample Output:
Name: entire
Summary: entire incorporation
Publisher: Unknown
Version: 0.5.11
Build Release: 5.11
Branch: 0.101
Packaging Date: Wed Nov 19 23:57:06 2008
Size: 0.00 B
FMRI: pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z
License:
- manifest
Version 0:
A GET operation that retrieves the contents of the manifest file for
a package specified by the provided FMRI.
Example:
URL:
http://pkg.opensolaris.org/manifest/0/entire@0.5.11,5.11-0.101:20081119T235706Z
Expects:
A URL-encoded pkg(5) FMRI excluding the 'pkg:/' scheme prefix
and publisher information and including the full version
information.
Returns:
The contents of the package's manifest file.
- p5i
Version 0:
A GET operation that retrieves an application/vnd.pkg5.info
datastream representing publisher and package information.
This is intended for consumption by clients for the purposes
of auto-configuration, metadata management policy determination,
and triggering packaging operations such as installation.
Example:
URL:
http://pkg.opensolaris.org/release/p5i/0/SUNWcs
Expects:
A full or partial URL-encoded pkg(5) FMRI, excluding the
publisher prefix. If the partial or full FMRI is valid, it will
be added to the datastream as is. If it includes the wildcard
character '*', a search of the repository's catalog for matching
entries will be performed and the unique set of resulting
package stems will be added to the datastream. If no match is
found, a 404 error will be raised.
Returns:
Returns a pkg(5) information datastream based on the repository
configuration's publisher information and the provided full or
partial FMRI or matching entries. The Content-Type of the
response is 'application/vnd.pkg5.info'.
- publisher
Version 0:
A GET operation that retrieves an application/vnd.pkg5.info
datastream representing publisher information. This is intended
for consumption by clients for auto-configuration and metadata
management policy determination.
Example:
URL:
http://pkg.opensolaris.org/release/publisher/0
Expects:
Nothing
Returns:
Returns a pkg(5) information datastream based on the repository
configuration's publisher information. The Content-Type of the
response is 'application/vnd.pkg5.info'.
- search
Version 0:
A GET operation that retrieves a text/plain list of packages with
metadata that matches the specified criteria.
Example:
URL:
http://pkg.opensolaris.org/release/search/0/vim
Expects:
A URL-encoded token representing the search criteria.
Returns:
A text/plain list of matching entries, separated by newlines.
Each entry consists of a set of four space-separated values:
index - what search index the entry was found in
action - what package action the entry is related to
value - the value that the matched the search criteria
package - the fmri of the package that contains the match
Results are streamed to the client as they are found.
Sample Output:
basename pkg:/SUNWvim@7.1.284,5.11-0.101:20081119T230659Z dir usr/share/vim
basename pkg:/SUNWvim@7.1.284,5.11-0.93:20080708T171331Z file usr/bin/vim
Content Operations
------------------
The pkg.depotd(5) server provides the following operations for retrieving
package content:
- file
Version 0:
A GET operation that retrieves the contents of a file, belonging to a
package, using a SHA-1 hash of the file's content.
Example:
URL:
http://pkg.opensolaris.org/release/file/0/
a00030db8b91f85d0b7144d0d4ef241a3f1ae28f
Expects:
A SHA-1 hash of the file's content belonging to a package in the
request path.
Returns:
The contents of the file, compressed using the gzip compression
algorithm.

114
doc/pkg5_docs/guide-txn-states.rst Normal file
View file

@ -0,0 +1,114 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
.. :vim set expandtab:
.. _Transaction States:
Transaction states
==================
On a depot open for publication, a new package version may be in the
process of publication. As the package version is built up, it goes
through a series of *transaction states*. We may also refer to these as
"server states".
We phrase the state machine in terms of a single removal state,
ABANDONED, which covers both the never-created package instance (even
with a series of never-finished transaction events). It may be more
appropriate to separate the ABANDONED state into TX_ABANDONED and
PKG_DELETED.
This summary leaves us with a state transition diagram like::
0
|
|
v
+--> TRANSACTING --> ABANDONED <--+
| | ^ |
| | | |
| v | |
| SUBMITTED ----> INCOMPLETE |
| | | |
| | | |
+--- PUBLISHED <---------+ |
| |
| |
+------------------------+
0 -> TRANSACTING
On initial package creation.
TRANSACTING -> ABANDONED
If initial package transaction never committed, commitment
failed, or explicitly dropped.
TRANSACTING -> SUBMITTED
On successful package transaction commitment. Packages with
syntax errors or immediate inconsistencies would have failed in
commitment.
SUBMITTED:
The package modified by the transaction is known by a specific
version. Its contents are in the repository.
SUBMITTED -> INCOMPLETE
If commitment included a deferred inconsistency (package
dependency is the only expected form), then the package is left
in the incomplete state.
INCOMPLETE:
The package with the specific version string is on the
incomplete list. Its contents are in the repository.
INCOMPLETE -> ABANDONED
If incomplete package explicitly removed. (Possibly by
timeout from arrival in INCOMPLETE.)
SUBMITTED -> PUBLISHED
If commitment had no deferred inconsistencies, then the package
is considered ready for publication.
INCOMPLETE -> PUBLISHED
If the deferred inconsistencies, upon reevaluation, are now held
to no longer be inconsistent, then the package is considered
ready for publication.
PUBLISHED:
The package with the specific version string is present in the
catalog. Its contents remain in the repository.
PUBLISHED -> ABANDONED
On manual request for package decommissioning, the package will
be moved to the abandoned state.
ABANDONED:
A package with a specific version string is no longer in the
catalog or on the incomplete list. Its contents, if they were
in the repository, should be removed from the repository.
XXX ARCHIVED might be a special state connected to PUBLISHED, or
merely a substate. An archived package has its manifest and
contents in the repository, but cannot be installed on a client.
The point of including ARCHIVED packages is to allow client
deduction on a system not installed by the pkg system.

221
doc/pkg5_docs/history.txt Normal file
View file

@ -0,0 +1,221 @@
pkg
HISTORY
1. Summary
The intent of history is to enable pkg clients to maintain a record of all
image-modifying operations that they perform.
2. Discussion
2.1 Operations
The following operations are currently recorded by the clients that are
part of pkg(5):
add-publisher
image-create
image-set-attributes
image-update
install
purge-history
rebuild-index
refresh-publisher
remove-publisher
set-publisher
set-preferred-publisher
uninstall
update-publisher
These operations were chosen because they have the potential to alter the
behavior of pkg(5) operations or because they modify an image.
3. History Entries
3.1 Information Recorded
Each history entry contains a record of basic information about the client
performing the operation, along with a mix of required and optional
information about the operation performed.
The following information about the client is always recorded:
client_name - The name of the client performing the operation
(e.g. pkg, packagemanager)
client_version - The value of pkg.VERSION at the time the client
performed the operation (e.g. 2e5300c4f0a4+)
client_args - The command-line arguments used when executing the
client (e.g. /usr/bin/pkg install foo)
The following information about the operation performed is always recorded:
operation_name - The name of the operation performed (refer to 2.1)
start_time - When the operation started (e.g. 20080916T214726Z)
end_time - When the operation ended (e.g. 20080916T214800Z)
userid - The id of the user that performed the operation
(e.g. 0)
username - The username of the user that performed the operation
(e.g. root)
result - The outcome of the operation and the reason for it
(e.g. Failed, Transport) This maps to the pkg history
output columns of 'outcome' and 'reason'.
The following information about the operation will be recorded if provided
by the client or api:
start_state - Information about the operation requested before any
evaluation of that request is performed (e.g. an
image plan before evaluation)
end_state - Information about the operation requested after
evaluation of that request has been performed (e.g.
image plan after evaluation)
errors - Any errors that were encountered while performing the
operation (i.e. tracebacks, exceptions, etc.)
be - The name of the boot environment on which the
operation was performed
be_uuid - The uuid corresponding to the boot environment
new_be - The name of any new boot environment that was created
while performing the operation.
new_be_uuid - The uuid corresponding to the new boot environment
snapshot - The name of the snapshot that was taken as a result of
this operation. If the operation completed
successfully and the snapshot was destroyed, this
this information is not stored.
3.2 Storage
Each history entry is recorded in a XML file located in the metadata
directory of an image (e.g. /var/pkg) in a directory named "history".
Each XML file is named after the pattern %Y%m%dT%H%M%SZ-sequence.xml.
Where sequence is a numeric value appended so that when multiple
operations are performed on an image within the same second (e.g. -02,
-03, etc.) entries are still written correctly.
3.3 File Format
It should be noted that this format description is that of a private
interface and is subject to change.
History entries are recorded in a XML file with a fairly simplistic
structure. The format is as follows:
The first line of each file is a standard xml header along with the
encoding used to save the information. This allows the pkg history
command to correctly display this information if the client changes
locales later.
<?xml version="1.0" encoding="ascii"?>
The second element of every history XML file is a root element used to
contain the client and operation information. Only one per file ever
occurs.
<history>
The client element has a name and version attribute used to record
client_name and client_version. Only one per file ever occurs.
<client name="pkg" version="2e5300c4f0a4+">
The args element contains one or more "arg" (argument) elements
that match each element of the system argument list used to
execute the client. It has no attributes. Only one per file
ever occurs.
<args>
Each arg element contains a CDATA element to encapsulate
the raw information for the argument. It has no attributes.
<arg>
<![CDATA[/usr/bin/pkg]]>
</arg>
<arg>
<![CDATA[-R]]>
</arg>
<arg>
<![CDATA[/tmp/test-image]]>
</arg>
<arg>
<![CDATA[install]]>
</arg>
<arg>
<![CDATA[SUNWvim]]>
</arg>
</args>
</client>
The operation element has the following attributes: name,
start_time, end_time, userid, and username. Optional attributes
are be, snapshot and new_be, indicating the boot environment the
operation was applied to, the snapshot taken during this operation,
and any new boot environment created as a result.
It can also contain the following, optional elements: start_state,
end_state, and errors. It only occurs once per file.
<operation end_time="20080912T225513Z" name="install" result="Succeeded"
start_time="20080912T225327Z" userid="101" be="solaris" username="username">
The start_state element is used to store information about the
operation performed before a request is evaluated (e.g. image
plan before evaluation). It always contains a CDATA element with
the related information and only occurs once per file. It has no
attributes.
<start_state>
<![CDATA[UNEVALUATED:
+pkg:/SUNWvim@7.1.284,5.11-0.96:20080825T192756Z
]]>
</start_state>
The end_state element is used to store information about the
operation performed after a request is evaluated (e.g. image
plan after evaluation). It always contains a CDATA element with
the related information and only occurs once per file. It has no
attributes.
<end_state>
<![CDATA[None -> pkg:/SUNWvim@7.1.284,5.11-0.96:20080825T192756Z
None -> pkg:/SUNWcsl@0.5.11,5.11-0.96:20080825T183047Z
None -> pkg:/SUNWpool@0.5.11,5.11-0.96:20080825T191747Z
None -> pkg:/SUNWlxml@2.6.31,5.11-0.96:20080825T191518Z
None -> pkg:/SUNWzlib@1.2.3,5.11-0.96:20080825T193230Z
None -> pkg:/SUNWlibms@0.5.11,5.11-0.96:20080825T191411Z
None -> pkg:/SUNWopenssl@0.9.8,5.11-0.96:20080825T194948Z
None -> pkg:/SUNWlibsasl@0.5.11,5.11-0.96:20080825T191417Z
None -> pkg:/SUNWpr@0.5.11,5.11-0.96:20080825T192030Z
None -> pkg:/SUNWtls@0.5.11,5.11-0.96:20080825T192639Z
]]>
The errors element contains one or more "error" elements
that encapsulate the information about each error that was
recorded during the operation. It has no attributes, and
only occurs once per file.
<errors>
Each error element contains a CDATA element to encapsulate
the raw information about the error. It has no attributes.
<error>
<![CDATA[]]>
</error>
</errors>
</operation>
</history>

198
doc/pkg5_docs/image.rst Normal file
View file

@ -0,0 +1,198 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
.. :vim:set expandtab:
IMAGES
------
Image types
~~~~~~~~~~~
Each collection of installed package instances is some kind of
image. We envision three kinds:
- entire images
An entire image contains (and can contain) all appropriate
packages from a repository. It has a machine type and release.
- partial images
A partial image contains (and can contain) all appropriate
packages from a repository. An partial image is tied to an
entire image and has an identified list of non-writeable
directory paths. It inherits its machine type and release.
- user images
A user image consists entirely of relocatable packages. It is
tied to an entire image for dependency satisfaction of
non-relocatable packages. It inherits its machine type and
release.
Image configuration
~~~~~~~~~~~~~~~~~~~
Configuration inheritance
`````````````````````````
Some aspects of configuration can be shared between all images. For
instance, a user image may specify publishers beyond those encoded
into the system defaults. So, a user image must have authoritative
configuration, but be able to draw on the parent image's state for
default settings.
Operations on partial images and non-live entire images may need to
access image configuration data when smf(5) for that image is
unavailable. So images of these kinds must have cached
configuration state.
Roughly, these aspects are sufficient to constrain our configuration
behaviour for user and entire images:
[user image]
look up local configuration
if undefined, request properties from svc://application/pkg
if unavailable, examine parent's configuration cache
if undefined or unavailable, use hard-coded default
[entire image]
request properties from svc://application/pkg
if unavailable, examine configuration cache
if undefined or unavailable, use hard-coded default
Partial images could have differing behaviour depending on whether
the operation is invoked from within the partial image's "packaging
context" (as it would be for an operation issued from within the
zone), or outside it (operations on the entire image and its
associated partial images).
For the first case, the configuration strategy is the same as that
for an entire image. For the second case, we could do
[partial image, external context]
- examine partial image configuration cache
- if unavailable, request properties from svc://application/pkg
- if undefined or unavailable, examine entire image configuration
cache
- if undefined, use hard-coded default
For certain properties (or even certain packages), it may be
inappropriate to let the partial image configurations drift from
that of the entire image.
Configuration components
````````````````````````
List of publishers. For each publisher, we have a prefix, an
origin URL, a list of mirror URLs, and annotations.
publisher_[escaped_name]/ Property group of type "com.sun.pkg,publisher"
/prefix pkg: publisher
/origin http:, https:, or ftp: URL
/mirrors list of URLs
/disabled boolean indicating whether publisher should be used
Image properties. The image has a collection of simple properties,
like machine type and policies that control the behavior of the pkg(5)
software within that image. Policies are properties with a boolean value.
properties/
/pursue-latest (not currently used)
/require-optional (not currently used)
/display-copyrights (not currently used)
/flush-content-cache-on-success
should downloaded compressed files be removed
after a successful install
/preferred-publisher preferred publisher for unknown package lookups
/title title of image for use in GUIs and other UIs
/description longer description of the content of the image
/[various]
Entire images have a property group for each tied partial image.
partial_[escaped_name]/ Property group of type "com.sun.pkg,partial"
/path Filesystem path
/vdomainname Defined if this image is a virtual
domain controlled on this system.
/vdomaintype "xen", "zone", ...
(XXX Should we instead assume that each of Zones and Xen will
acquire service FMRIs per zone/domain?)
Image-filesystem interactions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The intent is to utilize snapshot-capable filesystems to provide a
rollback mechanism that includes both the pkg(1M)-driven
modifications as well as subsequent modifications from configuration
methods, etc. In most cases, we are considering a filesystem with
capabilities similar to ZFS.
XXX Is fssnap_ufs(1M) sufficient to build something meaningful for
UFS?
With appropriate policies set, the image plan, prior to execution
must snapshot the involved filesystems.
There seem to be two options:
1. The image can build its list of filesystems and types (plus
other attributes, like mutability.
2. A list of filesystems is given to each package plan, the package
plan then evaluates its proposed actions against that list, and
offers a method to return the affected subset of the list.
In this case, we must also determine whether or not we are
restricted to clones (because one or more packages in the image
plan require kernel-restart) or are potentially live.
XXX Either of these approaches is applicable in the image/substrate
model, where the substrate owns the low-level properties specific to
a particular "place to write data".
In the case that one or more filesystems in the image is or are not
capable of snapshots, we have two choices:
- take no snapshot, as image-revert not possible in any safe or
complete sense
- take a set of snapshots that lead to a revert point that requires
manual forcing
We must warn about images of this kind, unless policy settings allow
us otherwise. Since we want to allow and understand "zfs
split"-style operations [1], we also need to determine if a snapshot
set taken before a split can sensibly be restored after the split
operation.
[1] (The zfs split RFE.)

170
doc/pkg5_docs/image.txt Normal file
View file

@ -0,0 +1,170 @@
pkg
IMAGES
:vim:set expandtab:
1. Summary
2. Discussion
2.1. Image types
Each collection of installed package instances is some kind of
image. We envision three kinds:
- entire images
An entire image contains (and can contain) all appropriate
packages from a repository. It has a machine type and release.
- partial images
A partial image contains (and can contain) all appropriate
packages from a repository. An partial image is tied to an
entire image and has an identified list of non-writeable
directory paths. It inherits its machine type and release.
- user images
A user image consists entirely of relocatable packages. It is
tied to an entire image for dependency satisfaction of
non-relocatable packages. It inherits its machine type and
release.
2.2. Image configuration
2.2.1. Configuration inheritance
Some aspects of configuration can be shared between all images. For
instance, a user image may specify publishers beyond those encoded
into the system defaults. So, a user image must have authoritative
configuration, but be able to draw on the parent image's state for
default settings.
Operations on partial images and non-live entire images may need to
access image configuration data when smf(5) for that image is
unavailable. So images of these kinds must have cached
configuration state.
Roughly, these aspects are sufficient to constrain our configuration
behaviour for user and entire images:
[user image]
look up local configuration
if undefined, request properties from svc://application/pkg
if unavailable, examine parent's configuration cache
if undefined or unavailable, use hard-coded default
[entire image]
request properties from svc://application/pkg
if unavailable, examine configuration cache
if undefined or unavailable, use hard-coded default
Partial images could have differing behaviour depending on whether
the operation is invoked from within the partial image's "packaging
context" (as it would be for an operation issued from within the
zone), or outside it (operations on the entire image and its
associated partial images).
For the first case, the configuration strategy is the same as that
for an entire image. For the second case, we could do
[partial image, external context]
examine partial image configuration cache
if unavailable, request properties from svc://application/pkg
if undefined or unavailable, examine entire image configuration
cache
if undefined, use hard-coded default
For certain properties (or even certain packages), it may be
inappropriate to let the partial image configurations drift from
that of the entire image.
2.2.2. Configuration components
List of publishers. For each publisher, we have a prefix, an
origin URL, a list of mirror URLs, and annotations.
publisher_[escaped_name]/ Property group of type "com.sun.pkg,publisher"
/prefix pkg: publisher
/origin http:, https:, or ftp: URL
/mirrors list of URLs
/disabled boolean indicating whether publisher should be used
Image properties. The image has a collection of simple properties,
like machine type and policies that control the behavior of the pkg(5)
software within that image. Policies are properties with a boolean value.
properties/
/pursue-latest (not currently used)
/require-optional (not currently used)
/display-copyrights (not currently used)
/flush-content-cache-on-success
should downloaded compressed files be removed
after a successful install
/preferred-publisher preferred publisher for unknown package lookups
/title title of image for use in GUIs and other UIs
/description longer description of the content of the image
/[various]
Entire images have a property group for each tied partial image.
partial_[escaped_name]/ Property group of type "com.sun.pkg,partial"
/path Filesystem path
/vdomainname Defined if this image is a virtual
domain controlled on this system.
/vdomaintype "xen", "zone", ...
(XXX Should we instead assume that each of Zones and Xen will
acquire service FMRIs per zone/domain?)
2.3 Image-filesystem interactions
The intent is to utilize snapshot-capable filesystems to provide a
rollback mechanism that includes both the pkg(1M)-driven
modifications as well as subsequent modifications from configuration
methods, etc. In most cases, we are considering a filesystem with
capabilities similar to ZFS.
XXX Is fssnap_ufs(1M) sufficient to build something meaningful for
UFS?
With appropriate policies set, the image plan, prior to execution
must snapshot the involved filesystems.
There seem to be two options:
1. The image can build its list of filesystems and types (plus
other attributes, like mutability.
2. A list of filesystems is given to each package plan, the package
plan then evaluates its proposed actions against that list, and
offers a method to return the affected subset of the list.
In this case, we must also determine whether or not we are
restricted to clones (because one or more packages in the image
plan require kernel-restart) or are potentially live.
XXX Either of these approaches is applicable in the image/substrate
model, where the substrate owns the low-level properties specific to
a particular "place to write data".
In the case that one or more filesystems in the image is or are not
capable of snapshots, we have two choices:
- take no snapshot, as image-revert not possible in any safe or
complete sense
- take a set of snapshots that lead to a revert point that requires
manual forcing
We must warn about images of this kind, unless policy settings allow
us otherwise. Since we want to allow and understand "zfs
split"-style operations [1], we also need to determine if a snapshot
set taken before a split can sensibly be restored after the split
operation.
3. References
[1] (The zfs split RFE.)

183
doc/pkg5_docs/kvm.md Normal file
View file

@ -0,0 +1,183 @@
# KVM branded-zone support
KVM branded zones are configured mainly via custom attributes in the zone
configuration.
To get started, `pkg install system/zones/brand/kvm` and configure a zone with the
kvm brand and the appropriate attributes; see the example zone at the end of
this page.
To troubleshoot problems if the zone fails to start, review the log file
which will be created at `/path/to/zone/root/tmp/init.log`
### Attributes
| Attribute | Default | Syntax | Example
| --- | --- | --- | ---
| bootdisk<sup>1</sup> | | path[,serial=<serno>] | tank/hdd/kvm1
| bootorder | cd | \[c\]\[d\]\[n\]
| cdrom<sup>3</sup> | | path to ISO | /data/iso/FreeBSD-11.1-RELEASE-amd64-bootonly.iso
| cpu | qemu64 |
| console | pipe,id=console0,path=/dev/zconsole<sup>4</sup> | options |
| disk<sup>1</sup> | | path[,serial=<serno>] | tank/hdd/kvm2,serial=1234
| diskN<sup>2</sup> | | path[,serial=<serno>] | tank/hdd/kvm2,serial=1234
| diskif | virtio | virtio,ahci
| netif | virtio-net-pci | virtio-net-pci,e1000
| ram | 1G | n(G\|M) | 8G
| type | generic | generic
| vcpus | 1 | n | 16
| vnc<sup>4</sup> | off | off,on,options | unix:/tmp/vm.vnc
| extra | | extra arguments for hypervisor |
#### Notes
<ol>
<li>You will also need to pass the underlying disk device through to the zone via a <i>device</i> entry, see the example below;</li>
<li>Use diskN to specify the slot into which the disk will be placed. A plain <i>disk</i> tag will be put in the lowest available slot.</li>
<li>The ISO file needs passing through to the zone via a lofs mount, see the example below;</li>
<li>Setting vnc to <i>on</i> is the same as setting it to <i>unix=/tmp/vm.vnc</i>.</li>
<li>You can connect to the virtual machine console from the global zone with <i>zlogin -C zonename</i>;</li>
</ol>
### Example zone
The following example zone is shown twice, once in info format and once in
export (showing the necessary commands for creation). Note that the example
shows setting the `allowed-address` attribute for the network interface -
this does not configure the address within the virtual machine but rather
prevents the use of any other address (L3 protection).
```
bloody# zonecfg -z oi info
zonename: oi
zonepath: /data/zone/oi
brand: kvm
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
fs-allowed:
fs:
dir: /tank/iso/OI-hipster-minimal-20180427.iso
special: /tank/iso/OI-hipster-minimal-20180427.iso
raw not specified
type: lofs
options: [ro,nodevices]
net:
address not specified
allowed-address: 10.0.0.112/24
defrouter not specified
global-nic not specified
mac-addr not specified
physical: oi0
vlan-id not specified
device:
match: /dev/zvol/rdsk/tank/hdd/oi0
device:
match: /dev/zvol/rdsk/tank/hdd/oi1
device:
match: /dev/zvol/rdsk/tank/hdd/oi2
attr:
name: vcpus
type: string
value: 16
attr:
name: ram
type: string
value: 4G
attr:
name: cdrom
type: string
value: /tank/iso/OI-hipster-minimal-20180427.iso
attr:
name: vnc
type: string
value: on
attr:
name: bootdisk
type: string
value: tank/hdd/oi0
attr:
name: disk
type: string
value: tank/hdd/oi1
attr:
name: disk1
type: string
value: tank/hdd/oi2,serial=1234
```
```
bloody# zonecfg -z oi export
create -b
set zonepath=/data/zone/oi
set brand=kvm
set autoboot=false
set ip-type=exclusive
add fs
set dir=/tank/iso/OI-hipster-minimal-20180427.iso
set special=/tank/iso/OI-hipster-minimal-20180427.iso
set type=lofs
add options ro
add options nodevices
end
add net
set allowed-address=10.0.0.112/24
set physical=oi0
end
add device
set match=/dev/zvol/rdsk/tank/hdd/oi0
end
add device
set match=/dev/zvol/rdsk/tank/hdd/oi1
end
add device
set match=/dev/zvol/rdsk/tank/hdd/oi2
end
add attr
set name=vcpus
set type=string
set value=16
end
add attr
set name=ram
set type=string
set value=4G
end
add attr
set name=cdrom
set type=string
set value=/tank/iso/OI-hipster-minimal-20180427.iso
end
add attr
set name=vnc
set type=string
set value=on
end
add attr
set name=bootdisk
set type=string
set value=tank/hdd/oi0
end
add attr
set name=disk
set type=string
set value=tank/hdd/oi1
end
add attr
set name=disk1
set type=string
set value=tank/hdd/oi2,serial=1234
end
```
You can connect to kvm vga console of zone with vncviewer and socat. For example:
```
# socat TCP-LISTEN:5500 UNIX-CONNECT:/data/zone/oi/root/tmp/vm.vnc
$ vncviewer localhost:5500
```

523
doc/pkg5_docs/license.txt Normal file
View file

@ -0,0 +1,523 @@
pkg(5): image packaging system
LICENSE ACTIONS ACCEPTANCE PROPOSAL
1. Overview
License actions provide a way to deliver the textual data
contained within licenses, copyright notices, disclaimers, or
other legally-related documents that need to be associated with
the contents of a package. They also provide a way for packages
to provide guidance to pkg(5) clients as to how this information
should be presented or if it requires acceptance by the user
before the package can be delivered into an image.
This proposal has the following core goals for the implementation
of license acceptance functionality:
* Support for non-interactive and interactive package operations
* Enablement of package creators to provide guidance to pkg(5)
client API consumers as to how licensing or other informational
data should be presented and interacted with
* Enablement of administrators and users to query and report on
the licensing or other informational data related to packages
contained within an image
* Enablement of administrators and users to restrict what packages
can be delivered into an image based on the package's provided
licensing information.
To achieve these goals, changes must be made to the following
pkg(5) components:
* License Actions
* Image configuration
* Client API
* BUI: pkg.depotd(1m)
* CLI: pkg(1), pkgsend(1)
* GUI: packagemanager(1), updatemanager(1)
This proposal omits the GUI programs as a separate team will
develop and deliver the enhancements related to the changes
discussed here.
2. License Action attribute changes and additions
Currently, the license attribute of license actions is not
restrictive enough in what characters are allowed for the
name of the license. To ensure cross-platform compatibility
and consistent naming, it is proposed that the license
attribute's definition be amended as follows:
license This attribute provides a description for the license
to help describe the license in a meaningful way. Some
examples might include:
- "GPL v2 only"
- "GPL with Specific Exception"
A recommended list of descriptions for common open-
source licenses, ang guidelines for custom licenses
will be added to the license action documentation to
encourage consistency. This value must be unique
within a package.
To support license acceptance functionality, new attributes for
license actions are needed to allow packages to provide guidance
to pkg(5) API consumers:
class This optional attribute is primarily for use in
filter and query operations. It is intended that
only short, descriptive text be used here. As
such, the content of this value will be limited to
the characters [A-Za-z][A-Za-z0-9 _-.,]*. While
this value is optional, its usage is strongly
encouraged, and so a recommended list of classes
for common open-source licenses, and guidelines
for custom licenses will be added to the license
action documentation. This value must be unique
within a package.
must-accept A boolean value indicating whether this license
must be accepted by a user before the related
package can be installed or updated. Acceptable
values are "true" or "false". Omission of this
attribute must be considered equivalent to
"false". The reason for this is that many
licenses (especially copyleft ones) do not
require this and clients should not prompt for it
unless provided guidance to do so by the package
creator.
must-display A boolean value indicating whether the content of
the action's payload must be displayed by clients
during packaging operations. Omission of this
value must be considered equivalent to "false".
3. Image configuration additions
Note that changes proposed in this section implicitly depend on
on upcoming changes to the image configuration format for the
client. As such, how policies, etc. are stored is not discussed
here other than to note that they will be stored within the image
configuration.
To enable administrators and users to effectively manage the
packages contained within an image, users need to be able to
define, on a per-publisher basis, what the behaviour of the
packaging system and clients should be in the following key
areas related to licenses:
* filtering
Administrators and users need to be able to define a policy
that can be used to determine what packages are delivered into
an image based on licensing information defined in packages.
* acceptance
Administrators and users need to be able to define a policy
that can be used to determine what behaviour clients should
exhibit when encountering a license that requires explicit
acceptance.
It is believed that a clear delineation between image properties
(which describe the image itself) and policies (which provide
guidance to pkg(5) clients) will be helpful to end-users from a
usage and documentation standpoint. As such, a new policy system
has been devised to allow users to provide guidance to the pkg(5)
system and clients on a per-publisher (and/or global) level based
on the policy in question. Per-publisher values will override
the global value based on the publisher of the package that is
being evaluated by the client API.
The following new policies will be created, and can all be set at
a global or per-publisher level:
license-policy A keyword indicating what the behaviour of clients
should be when the license of a package requires
acceptance. The following keywords are supported:
accept Automatically accepts any license with
must-accept=true after license
filtering has been applied.
decline Automatically declines any license
with must-accept=false after license
filtering has been applied.
explicit Requires explicit acceptance of any
licenses with must-accept=true by the
user after licensing filtering has
been applied. This could be
implemented as an interactive prompt,
or by a failure of the client with a
requirement to pass an explicit
command-line option for acceptance.
This is the default value.
license-accept A list of values that will be used to mark any
license with a matching description or class as
accepted automatically if they require acceptance.
This value is undefined by default.
license-decline A list of values that will be used to mark any
license with a matching description or class as
declined automatically, regardless of whether they
require acceptance. This value is undefined by
default.
license-display A keyword indicating what the behaviour of the
client should be when displaying the text of
license actions. The following keywords are
supported:
all Suggests clients display the text of all
license actions, but must display the text
of license actions that have
must-display=true.
auto The default value for the image
configuration, which indicates that
clients must display the text of license
actions that have must-display=true.
The following, existing image properties will become policies and
can only be set on a global basis:
flush-content-cache-on-success
pursue-latest
require-optional
The following, existing image properties will become policies and
can be set on a global or per-publisher basis:
send-uuid
The following, existing image properties will be removed as they
were never implemented and are being replaced by functionality
discussed in this proposal:
display-copyrights
Finally, it should be noted that the existing 'property' subcommands
cannot be used to alter or view policies and the proposed 'policy'
subcommands likewise cannot be used to alter or view properties.
4. Client API
4.1 pkg.client.api
The client API, to enable license acceptance functionality
enforcement and control for clients, will need to change in
the following ways:
* Plan creation will be changed to analyze and determine license
acceptance and or build a list of licenses and their acceptance
status for packages being installed or updated.
* The ImagePlan object will have a new method named 'get_licenses'
added. After plan creation is finished, consumers may call this
method to retrieve a list of tuples containing a LicenseInfo
object, whether the license is allowed by image policy, and the
current acceptance status of the license as detailed later. The
default acceptance status of a license will be 'declined' unless
otherwise defined by image policy or operation policy.
* The ImagePlan object will have a new method named
'set_license_status' which will allow callers to mark the explicit
acceptance of a package's license by a user:
set_license_status(fmri, class=None, description=None, status)
Either 'class' or 'description' must be provided.
* To allow pkg.client.api consumers access to set_license_status
and get_licenses(), equivalent wrapper functions will be added
to the PlanDescription object provided by the API.
* Two new exceptions will be added to pkg.client.api_errors. The
first will be named 'PlanExecutionError' and be used as a base
exception class for all plan execution errors. The second
exception will be named 'PlanExecutionLicenseError' and will
inherit from the first exception, while being used to indicate
that a licensing related error occurred.
* Plan execution will be changed to verify that each license
that has must-accept=True has been marked as accepted. If
any license has not been marked as accepted, and requires
acceptance, a 'PlanExecutionLicenseError' will be raised.
* Plan execution will be changed to record each package that is
part of the operation using pkg.client.history. The full FMRI,
the classes and descriptions identifying the licenses contained
within the package, and acceptance status of each license within
a package will be recorded. The following statuses will be used
to indicate a package's license acceptance:
* accepted
Indicates that the license was accepted through the API,
presumably by the user.
* accepted-policy
Indicates that the license was automatically accepted
based on image policy.
* declined
Indicates that the license required acceptance and was not
marked as accepted.
* declined-policy
Indicates that the license was automatically rejected
based on image policy.
* not-applicable
Indicates that the license did not require acceptance and
was not declined by policy.
4.2 pkg.client.history
To ease logging of license acceptance information, and to
accurately reflect operation failure, the following changes will
be made to pkg.client.history:
* New operation result constants will be added:
RESULT_FAILED_LICENSE_DECLINED
Used to indicate that an operation failed because a
license that required acceptance was not marked as
accepted.
RESULT_FAILED_LICENSE_POLICY
Used to indicate that an operation failed because a
license was declined based on image policy.
* log_operation_error() will be changed to ensure that the new
'PlanExecutionLicenseError' triggers an appropriate failure
result as noted above.
LOG_PKG_OP_INSTALL
LOG_PKG_OP_UPDATE
LOG_PKG_OP_REMOVE
* A new method to log license status information about a package
will be added:
log_pkg_license(class=None, description=None, status)
Either 'class' or 'description' must be provided.
* A new method to log operation information about a package will
be added:
log_pkg_operation(operation, old_fmri, new_fmri)
5. BUI
5.1. pkg.depotd(1m)
The pkg.depotd(1m) server that pkg(5) provides will be changed
such that its output matches that of the CLI when listing licenses
as much as possible, such as showing the "license" attribute of
a license action.
6. CLI
6.1. pkg(1)
The pkg(1) client that pkg(5) provides needs to be enhanced to
provide the following functionality:
* A mechanism to temporarily override client behaviour, provide
intial policy values during image-creation, or to provide
explicit user intent (such as license acceptance).
A new command-line option as shown below will be added to the
pkg(1) client that will allow its usage with applicable sub-
commands. Policy names not applicable to the corresponding
subcommand will effectively be ignored (such as pursue-latest
with the 'fix' subcommand).
--policy <policy-name>=<policy-value>
The following subcommands will allow this option:
* contents
* fix
* history
* image-update
* info
* install
* list
* uninstall
* verify
image-create will also support the --policy option, but instead of
being used as a temporary override, it will be used to specify the
permanent defaults for the image being created.
* Graceful, informative failure due to license-related exceptions
If a license-related exception occurs during plan execution, a
failure message similar to the following will be displayed:
The following packages are not permitted by image policy
due to the current image's license policies:
--------------------------------------------------
License A
--------------------------------------------------
Foo
Bar
The following packages contain licenses that require
acceptance before they can be installed (use --accept
to accept these licenses):
--------------------------------------------------
License B
--------------------------------------------------
Baz
Quux
Note that it may be possible for the same license to be marked
as requiring acceptance in one package and not another, and so
the display above may indicate the same license more than
once.
* Display of licenses that require it during packaging operations
pkg(1) will be changed to retrieve and display the text of
any licenses that must be displayed or have been requested
for display after plan creation. The default value is based
on image policy as noted in section 3.
The display of the license text will be the same as the
proposed output for the 'info' subcommand as noted below.
* Improvement of license display by 'info' subcommand
Currently, the info subcommand will display license text for
one or more given FMRIs but does so without any visual
separation other than a newline between distinct license text
output, does not indicate which package the license belongs
to, and does not provide any clear indication of the identity
of a license (e.g. CDDL, BSD, etc.).
The output of this subcommand will be changed so that the
description and class of the license will be shown and a visual
separator will be placed between the output of each license.
The revised output is anticipated to be similar to this:
------------------------------------------------------------
Package: Foo
License: copyright
Copyright 2009 Example Company, Inc. All Rights Reserved.
------------------------------------------------------------
Package: Foo
License: Termsv1 "Example Company's License Description v1"
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Vivamus in risus sem, id blandit justo. Maecenas nulla massa,
mollis sed lobortis a, placerat vel elit. Quisque eleifend leo
ipsum. Donec feugiat gravida molestie. Nulla facilisi. Nullam
sit amet ligula sed mauris tempor fermentum quis at purus.
------------------------------------------------------------
Package: Bar
License: Termsv2 "Example Company's License Description v2"
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Vivamus in risus sem, id blandit justo. Maecenas nulla massa,
mollis sed lobortis a, placerat vel elit.
* A set-policy subcommand will be added to allow setting policy
values that functions as follows:
set-policy [-p publisher] policy-name policy-value ... ...
If -p is not provided, the global policy value will be changed.
One or more pairs of policy name and policy value may be provided
provided to this command to allow setting multiple values using a
single command invocation.
Some options may not be set at the publisher level and this will
result in an error message such as:
$ pkg set-policy -p opensolaris.org flush-content-cache-on-success
pkg: set-policy: flush-content-cache-on-success is a global policy
and cannot be set on a per-publisher basis.
* An unset-policy subcommand will be added to allow unsetting or
resetting policy values that functions as follows:
unset-policy [-p publisher] policy-name ...
If -p is not provided, the global policy value will be changed.
Some options may not be set at the publisher level and this will
result in an error message such as:
$ pkg unset-policy -p opensolaris.org \
flush-content-cache-on-success
pkg: unset-policy: flush-content-cache-on-success is a global
policy and cannot be unset on a per-publisher basis.
* A policy subcommand will be added for retrieving policy values
that functions as follows:
policy [-H] [-p publisher] policy-name ...
If -p is not provided, then all matching policy values will be
shown instead of those of a specific publisher. 'All' may be
specified to show global values that do not apply to a specific
publisher.
If -H is provided, headers will be omitted.
One or more policy names may be specified separated by whitespace.
If no policy-names are specified, all policies will be displayed.
Example output:
$ pkg policy
PUBLISHER POLICY VALUE
All license-accept
All license-decline
All license-display auto
All license-policy decline
opensolaris.org license-accept accept
$ pkg policy license-accept
PUBLISHER POLICY VALUE
All license-accept
opensolaris.org license-accept accept
$ pkg policy license-accept license-display
PUBLISHER POLICY VALUE
All license-accept
All license-display auto
opensolaris.org license-accept accept
$ pkg policy license-accept -H
All license-accept
opensolaris.org license-accept accept
$ pkg policy license-accept -p All
PUBLISHER POLICY VALUE
All license-accept decline
$ pkg policy license-accept -p opensolaris.org
PUBLISHER POLICY VALUE
opensolaris.org license-accept accept

1265
doc/pkg5_docs/linked-images.txt Normal file
View file

File diff suppressed because it is too large Load diff

53
doc/pkg5_docs/macros.rst Normal file
View file

@ -0,0 +1,53 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
.. Hyperlinks
.. _pkg-project: http://opensolaris.org/os/project/pkg/
.. _pkg-opensolaris-release: http://pkg.opensolaris.org/release/
.. _pkg-opensolaris-dev: http://pkg.opensolaris.org/dev/
.. Substitutions
.. |pkg5| replace:: ``pkg``\ (5)
.. |smf5| replace:: ``smf``\ (5)
.. |tar1| replace:: ``tar``\ (1)
.. |pkg1| replace:: ``pkg``\ (1)
.. |pkgsend1| replace:: ``pkgsend``\ (1)
.. |pkgrecv1| replace:: ``pkgrecv``\ (1)
.. |pkgdepend1| replace:: ``pkgdepend``\ (1)
.. |pkgdiff1| replace:: ``pkgdiff``\ (1)
.. |pkgfmt1| replace:: ``pkgfmt``\ (1)
.. |pkgmogrify1| replace:: ``pkgmogrify``\ (1)
.. |depotd1m| replace:: ``pkg.depotd``\ (1M)
.. |beadm1m| replace:: ``beadm``\ (1M)
.. |libbe3lib| replace:: ``libbe``\ (3LIB)
.. |OS_Name| replace:: OpenSolaris

119
doc/pkg5_docs/manifest_preprocessor.txt Normal file
View file

@ -0,0 +1,119 @@
A ON manifest pre-processor needs the following attributes:
1) Can selectively "comment out" parts of the
manifest.
2) Can perform string substitution in manifests.
3) Can include other files to be part of this manifest
using either inline include statements or command
line arguments. In either case, a search path
for these files can be specified via one or more
-I options ala cpp.
4) Can transform actions matching certain
criteria, using specifications delivered as part of the
manifest (or one of the included files).
Importer currently does the first two via simple recursive
string substitution:
Macros are of the form: $(MACRONAME) and can have
any string value, including another macro. Macros
are found in each line, repeatedly substituted
until no more are found. Expressions of macro
form w/o a value are passed through w/o modification.
Eliding lines from a manifest is done by using a
macro at the begining of the line that has a '#'
value, which is the manifest comment character.
As a practical matter, the macro names are assigned
in the importer Makefile to facilitate understanding;
eg
$(sparc_ONLY) has the value '#' unless the current
build is sparc; $(i386_ONLY) is similar. On sparc,
$(ARCH64) is sparcv9, etc.
The importer files then use these macros to allow
a single file to be used for both architectures.
For including files, the easiest mechanism would be
some unique (distinct from either a comment, macro
or action line) syntax such as:
<include "path">
The specified path is searched w/o substitution for
if it starts w/ "/"; otherwise any parameters specifed
via -I are preprended and then searched for in order.
Any files specified w/ -A would be searched for and appended
to the manifest as if they had be specified w/ <include "path">
at the end of the file.
Once the manifest preprocessor had performed macro substitution
and file inclusion, the last step would be executing any transformative
steps specified in the files. These transformative steps would
contain a set of matching criteria and the desired effect on the action:
<transform matching criteria -> operation>
matching criteria would be of the form
file|dir|signature|... any action types
path=var/svc/manifest/.*xml
mode=0?1[0-7][0-7][0-7]
when python regexp rules would be used to find matching attributes.
If multiple attributes are specified they all must be true for the
transform to take effect, with exception that specifying multiple
action types will match any specified.
operation specification looks like this:
drop # discard this action
edit attribute regexp [replace] # apply regexp to value of attribute;
# if match occurs, substitute replace for
# portion of attribute value that matched.
# If replace is omitted, remove the
# the attribute values that match regexp
set attribute value # set specified attribute to value
add attribute value # add value to attribute
Examples:
Add tags to smf manifest so they're properly imported:
<transform file path=var/svc/manifest/.*\.xml -> add refresh_fmri svc:/system/manifest-import:default>
Add tags to gnome icon files to rebuild icon cache:
<transform file path=usr/share/icons/.* -> add restart_fmri svc:/application/desktop-cache/icon-cache:default>
Change action location (including link targets) from usr/openwin to usr/X11:
<transform path=usr/openwin -> edit path usr/openwin usr/X11>
<transform target=.*openwin.* -> edit target openwin X11>
Command line
------------
pkgmog [-D macro=value] ... [-I includepath]... [-A filename] ... [inputfile [outputfile]]
Where:
inputfile is stdin if omitted
outputfile is stdout if omitted
-D defines the value of the specified macro; may be empty
-I adds the specified path to search for included files
-A appends the file to the end of the input as if the last line in the file
included it
(pkgmog is short for package transmogrify)

269
doc/pkg5_docs/mediated-links.txt Normal file
View file

@ -0,0 +1,269 @@
Problem Statement
=================
It's common to want multiple versions of a component installed on a system.
In order for this to work, each version must have no coincident pathnames.
For most pathnames, it's easy to put them in a versioned directory or give
them a versioned name. But for some frequently used pathnames (an
executable expected to be in $PATH, or a man page), it's highly convenient
to provide an unversioned path (possibly as a link to a versioned path) for
casual use.
This of course leads to a conflict when multiple component versions all
want to provide the same unversioned path. We need a mechanism by which we
can manage these paths.
The most typical desired behavior for these paths is that they refer to the
latest version available; that is, if component foo has versions 1.2, 1.3,
and 1.4 installed, then unversioned pathnames for foo should refer to the
1.4 version. However, there may be reasons that the package vendor, the
site administrator, or even the local system administrator may want to
override this choice. Since properly written code uses the versioned
pathnames, the reference should be allowed to switch at arbitrary times.
Another related problem is the delivery of different components which
provide similar functionality. This differs from the previously stated
problem in that the components don't share the same versioning space, and
so need a different namespace from which the desired component can be
chosen.
Finally, a versioned component may also have different implementations
(indeed, there may be multiple implementation axes), so we need to handle
both problems at once.
Proposal
========
We assert that this problem can be restricted to pathnames which can be
replaced by symbolic links. Although a solution could apply to other
filesystem entry types, the implementations would be significantly more
difficult, and likely wouldn't provide any real benefit.
To that, we introduce the notion of "mediated links", being symlinks or
hardlinks which are subject -- when necessary -- to mediation in the case of
conflict. Mediated links are link or hardlink actions with some extra metadata
that allows the packaging system to treat conflicts specially.
The one required attribute is "mediator", whose value denotes the entry in
the namespace shared by all the pathnames participating in a given
mediation group. For instance, the mediator attribute value for all
/usr/bin/python and /usr/share/man/man1/python.1 links might be "python".
The value of this attribute must be an alphanumeric string.
For a link mediated solely by version, the "mediator-version" attribute is
required. This is the version (expressed as a dot-separated sequence of
nonnegative integers) of the interface described by the "mediator"
attribute, and not necessarily related to the version of the implementation
delivered by the package, or by the version in the package FMRI. For
instance, links participating in the "python" mediation might have
"mediator-version" set to "2.4", "2.6", "3.2", etc.
When resolving the conflict in mediated links, pkg(5) will normally choose
the link with the greatest value of "mediator-version". The package vendor
may override this selection by adding the attribute "mediator-priority"
with the value "vendor" to its preferred link in a package it delivers.
Similarly, site administrators may override the selection by version or the
vendor override by adding the attribute "mediator-priority" with the value
"site" to its preferred link. The local system administrator may override
this selection with the commandline specified below.
Mediation may be performed by implementation in addition to or instead of
by version. In this case, the attribute is "mediator-implementation" and
the value must be a string containing only alphanumeric characters and '-'
(preferably short and descriptive). Implementation strings are not considered
to be ordered, and so one must be chosen explicitly, or the system will choose
arbitrarily (first in lexical order). As with versioned mediation,
"mediator-priority=vendor" and "mediator-priority=site" can be used to override
the default behavior, and the local administrator can set the preference
explicitly.
If the implementation itself can be or is versioned, then the value may
have a '@' and a version appended to it. If multiple versions of an
implementation exist, the default will be to choose the greatest as defined
by those versions, similarly to the purely version-mediated scenario.
By default, every time a package operation is performed, the "best" mediation
will be selected and then applied to all packages. However, the local
administrator may explicitly set both the version and implementation for a
mediation, or only one of them, in which case the system will select the "best"
mediation with the matching component.
All links delivered to a given pathname must participate in mediation or not
at all. If some links delivered to a given pathname are mediated, and some
are not, package operations will fail with a conflicting error message. Likewise,
all mediated links delivered to a given pathname must share the same mediator.
However, not all mediator versions and implementations need to provide a link at
a given path. If a version and/or implementation doesn't provide a link, then the
link is removed when that version is selected.
The commandline used for locally administering mediated links is as
follows:
- pkg set-mediator [-V <version>] [-I <implementation>] mediator ...
Sets the mediator <mediator> to a given version and/or implementation.
- pkg unset-mediator [-V] [-I] mediator ...
Reverts the mediator <mediator> to the system default. If the -V option is
supplied, revert the version-mediated aspect to the system default, but leave
the implementation mediation in place. If the -I option is supplied,
revert the implementation-mediated aspect to the system default, but
leave the version mediation in place. If neither -V or -I are specified,
both the version and implementation will be reverted to the system default.
- pkg mediator [<mediator> ...]
Display information about the mediators on the system. When one or
more mediators are specified, display information only about the specified
mediators. The information should include the mediator name, selected
version and implementation, and how each component was chosen by the system.
Examples
========
Mediation by version
--------------------
The canonical case for version mediation is a language platform. In this
case, we'll investigate Python. Two paths that might be mediated are the
python interpreter executable and its manpage. The runtime/python-26
package would deliver
file path=usr/bin/python2.6 ...
file path=usr/share/man/man1/python2.6.1 ...
link path=usr/bin/python target=python2.6 mediator=python \
mediator-version=2.6
link path=usr/share/man/man1/python.1 target=python2.6.1 mediator=python \
mediator-version=2.6
and the runtime/python-24 package would deliver
file path=usr/bin/python2.4 ...
file path=usr/share/man/man1/python2.4.1 ...
link path=usr/bin/python target=python2.4 mediator=python \
mediator-version=2.4
link path=usr/share/man/man1/python.1 target=python2.4.1 mediator=python \
mediator-version=2.4
If only runtime/python-26 were installed, then /usr/bin/python would be
linked to python2.6. Likewise, if only runtime/python-24 were installed,
then /usr/bin/python would be linked to python2.4. If both were installed,
then by default, 2.6 would be found to be greater than 2.4, and the link
would point to python2.6.
If the vendor of the two packages wished, they could deliver a package
"runtime/python" which would simply have a dependency on the desired
default version.
If the local administrator preferred /usr/bin/python to be version 2.4,
then she could run
pkg set-mediator -V 2.4 python
after which running "python" would run the 2.4 interpreter, and "man
python" would display the manual for the 2.4 interpreter.
Vendor override
---------------
If, in the above scenario, the vendor of runtime/python-26 and
runtime/python-24 wished to override the default behavior and have the 2.4
interpreter be the default, then the runtime/python-24 package would
deliver
file path=usr/bin/python2.4 ...
file path=usr/share/man/man1/python2.4.1 ...
link path=usr/bin/python target=python2.4 mediator=python \
mediator-version=2.4 mediator-priority=vendor
link path=usr/share/man/man1/python.1 target=python2.4.1 mediator=python \
mediator-version=2.4 mediator-priority=vendor
Thus in the case where both runtime/python-26 and runtime/python-24 were
installed, /usr/bin/python would point to python2.4, despite 2.6 being
greater than 2.4. The "runtime/python" package would likely then contain a
a dependency on "runtime/python-24". The local administrator could
override this choice to 2.6 by running
pkg set-mediator -V 2.6 python
Mediation by implementation
---------------------------
The canonical case for implementation mediation is the editor vi. There
are multiple implementations of vi, but there is no versioned specification
which each implements. We will examine vim, nvi, and the legacy Solaris
vi.
In the package editor/vim, we deliver
file path=usr/bin/vim
link path=usr/bin/vi target=vim mediator=vi mediator-implementation=vim
In the package editor/svr4-vi, we deliver
file path=usr/has/bin/vi
link path=usr/bin/vi target=../has/bin/vi mediator=vi mediator-implementation=svr4
And in the package editor/nvi, we deliver
file path=usr/bin/nvi
link path=usr/bin/vi target=nvi mediator=vi mediator-implementation=nvi
If editor/vim were installed on the system without either of the others,
then /usr/bin/vi would point to vim. If either of the other packages were
later added, the /usr/bin/vi would continue to run vim, until the local
administrator were to run
pkg set-mediator -I svr4 vi
which would switch the link to run legacy Solaris vi. If the packages were
all installed simultaneously, the link would be chosen arbitrarily. The
vendor ought to deliver a vendor priority tag:
link path=usr/bin/vi target=vim mediator=vi mediator-implementation=vim \
mediator-priority=vendor
so that the desired default is always chosen.
Mediation with multiple implementations
---------------------------------------
Vim can be configured with a handful of different compile-time options.
For this example, we'll focus on --with-features=tiny (the minimal
configuration) and --with-features=huge (the maximal configuration, sans
GUI and external languages).
As before, we deliver the package editor/svr4-vi:
file path=usr/has/bin/vi
link path=usr/bin/vi target=../has/bin/vi mediator=vi mediator-implementation=svr4
We deliver two vim packages -- one for tiny, one for huge -- with a new
"vim" implementation-based mediator, set to "tiny" for the tiny package and
"huge" for the huge package. In addition, an implementation-based mediated
link is delivered to /usr/bin/vi which targets vim if the vi mediator is
set to vim. Since these two actions are identical, there is no conflict.
Here is vim-tiny:
file path=usr/bin/vim-tiny
link path=usr/bin/vim target=vim-tiny mediator=vim mediator-implementation=tiny
link path=usr/bin/vi target=vim mediator=vi mediator-implementation=vim
and editor/vim-huge:
file path=usr/bin/vim-huge
link path=usr/bin/vim target=vim-huge mediator=vim mediator-implementation=huge
link path=usr/bin/vi target=vim mediator=vi mediator-implementation=vim
Thus to ensure that /usr/bin/vi is the huge vim, the administrator would
have to run
pkg set-mediator -I huge vim
pkg set-mediator -I vim vi

57
doc/pkg5_docs/multi-platform.rst Normal file
View file

@ -0,0 +1,57 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
pkg
MULTI-PLATFORM
The core pkg(5) technology is generic enough to be useful across multiple platforms
(e.g. Windows and Linux). The full range of supported platforms are listed here:
http://wikis.sun.com/display/IpsBestPractices/OS+Platform+Support
The following modules within the pkg(5) source base are multi-platform:
- the CLIs (client.py, publish.py, depot.py, pull.py)
- src/modules (the core of pkg(5))
- src/tests (except the CLI tests do not run on Windows)
- src/man
- src/web
- src/po (except for the GUI messages which are OpenSolaris-only)
The following modules are not multi-platform (only supported on OpenSolaris):
- src/brand
- src/gui, src/um and the start scripts (packagemanger.py, updatemanager.py,
and updatemanagernotifier.py)
- pkgdefs
- SMF support: src/svc-pkg-depot, src/pkg-server.xml, src/pkg-update.xml
- src/util
The following modules are only used for non-OpenSolaris support:
- src/scripts
Multi-platform support is focused on providing support for user images as the
operating system software is not delivered for other platforms using pkg(5).
Development best practices for writing multi-platform pkg(5) code are available
here: http://opensolaris.org/os/project/pkg/devinfo/bestpractices/.
Information about using multi-platform pkg(5) and pre-built binaries
are available here: http://wikis.sun.com/display/IpsBestPractices

34
doc/pkg5_docs/multi-platform.txt Normal file
View file

@ -0,0 +1,34 @@
pkg
MULTI-PLATFORM
The core pkg(5) technology is generic enough to be useful across multiple platforms
(e.g. Windows and Linux). The full range of supported platforms are listed here:
http://wikis.sun.com/display/IpsBestPractices/OS+Platform+Support
The following modules within the pkg(5) source base are multi-platform:
- the CLIs (client.py, publish.py, depot.py, pull.py)
- src/modules (the core of pkg(5))
- src/tests (except the CLI tests do not run on Windows)
- src/man
- src/web
- src/po (except for the GUI messages which are OpenSolaris-only)
The following modules are not multi-platform (only supported on OpenSolaris):
- src/brand
- src/gui, src/um and the start scripts (packagemanger.py, updatemanager.py,
and updatemanagernotifier.py)
- pkgdefs
- SMF support: src/svc-pkg-depot, src/pkg-server.xml, src/pkg-update.xml
- src/util
The following modules are only used for non-OpenSolaris support:
- src/scripts
Multi-platform support is focused on providing support for user images as the
operating system software is not delivered for other platforms using pkg(5).
Development best practices for writing multi-platform pkg(5) code are available
here: http://opensolaris.org/os/project/pkg/devinfo/bestpractices/.
Information about using multi-platform pkg(5) and pre-built binaries
are available here: http://wikis.sun.com/display/IpsBestPractices

1050
doc/pkg5_docs/on-disk-format.txt Normal file
View file

File diff suppressed because it is too large Load diff

427
doc/pkg5_docs/one-pager-main.txt Normal file
View file

@ -0,0 +1,427 @@
Template Version: @(#)onepager.txt 1.31 07/08/08 SMI
This information is Copyright 2008 Sun Microsystems
1. Introduction
1.1. Project/Component Working Name:
pkg(5): image packaging system
1.2. Name of Document Author/Supplier:
Stephen Hahn, Sun Microsystems,
on behalf of the pkg(5) project team
1.3. Date of This Document:
03/10/2008
1.4. Name of Major Document Customer(s)/Consumer(s):
1.4.1. The Community you expect to review your project:
Install and Packaging CG
1.4.2. The ARC(s) you expect to review your project:
PSARC
1.5. Email Aliases:
1.5.2. Responsible Engineer:
stephen.hahn@sun.com
1.5.4. Interest List:
pkg-discuss@opensolaris.org
2. Project Summary
2.1. Project Description:
The image packaging system, pkg(5), is a portable software
packaging and delivery system intended to allow efficient,
observable, and controllable transitions between known
configurations of software content. pkg(5) will subsume the
functionality of the of the packaging and patching utilities
included in historical Solaris releases. A primary goal for
this project is to improve and extend the usability and
functionality of our packaging system.
The project includes a set of recommended changes to the
existing software groupings--the package definitions--in an
attempt to produce a more rational and flexible organization of
the current components.
2.2. Risks and Assumptions:
We intend to preserve the legacy packaging system functionality
to support compatibility of existing packages. We believe that,
if migration and compatibility practices are made available, the
provision of a new packaging mechanism will be followed by
adoption.
We strongly believe that the refactoring and renaming of the
existing package graph is not achievable with reasonable cost
and duration with the existing packaging/patching/installation
software. We also believe compatibility with the existing graph
can be preserved, to support the earlier assumption about
preserving legacy package operations.
We also believe that, for the majority of the operating system's
development and deployment needs, binary software delivery is
preferred over source-based build delivery.
3. Business Summary
3.1. Problem Area:
Deficits in the current packaging, patching, and installation
tool set affect potentially all parties interacting with the
historical Solaris releases and successors. Such deficits
include:
- lack of support for dependency-based retrieval during package
installation, from one or more network repositories,
- coarse and incorrect dependencies, limiting use for
construction of appliances or other specific-purpose systems,
- lack of versioning and control over change,
- forced interactivity,
- integration with virtualized systems, particularly patching
performance,
- reliance of the installer on hidden information, limiting
participation in system upgrade scenarios,
- lack of safety, specifically around package completeness and
alternate package contexts,
- high developer costs around package and patch creation and
maintenance,
- lack of support for unprivileged package and patch
installation,
- lack of awareness of ZFS and smf(5),
- late or no correctness checking, and
- minimal ease of use.
Additionally, the absence of a portable and efficient
cross-platform software delivery system places additional costs
upon teams that must deliver software for multiple platforms,
such as enterprise middleware vendors.
3.2. Market/Requester:
Distribution providers and software content providers have
requested substantial changes to the legacy packaging system.
The requested changes focus on reducing maintenance costs and
increasing development efficiencies.
Various customers of the historical Solaris release have asked
for substantial capabilities not present in the legacy packaging
system.
Finally, multiplatform packaging capabilities are of interest to
a number of software content providers.
3.3. Business Justification:
See 3.1 and 3.2.
3.4. Competitive Analysis:
Every major operating system vendor--and most upcoming new
vendors--offers a form of networked software delivery and
updates. Well known companies with such technologies are
Microsoft, Red Hat, Apple, and Canonical; new companies include
rPath. Non-profit entities with equivalent technology include
the Debian Project.
3.5. Opportunity Window/Exposure:
In order for OpenSolaris-based systems to remain competitive in
software delivery functionality, Solaris 10 should be the last
Minor release with a packaging system that fails to meet the
needs stated in 3.1 and 3.2.
3.6. How will you know when you are done?:
In terms of basic capabilities, we can examine each component.
Project completion on the retrieval side can be measured by
achieving the capability of managing mixed content from a
variety of publishers, with potentially distinct entitlement
regimes. On the publication side, completion of the initial
project is reached once the goals around dependency and
correctness checking (and failure handling) are met for both the
server and the publication client. Finally, ease of use (or
familiarity) must match or exceed that of other leading
packaging systems.
In terms of the product as a whole, we must be able to upgrade,
with some statement about limitations on fidelity, a system
installed using the legacy packaging components such that it can
be further updated using the image packaging system.
4. Technical Description:
4.1. Details:
pkg(5) is a network-oriented binary packaging system. Although
it will have on-disk representations for versioned packages, the
primary expected use for installation of software will be
between an intelligent client and one or more relatively simple
servers.
The project defines a client-server publication mechanism. The
publication client offers up transactions on packages. The
server evaluates transactions from the publication client.
Transations that are deemed to be complete and/or safe by the
server are then made available to the retrieval client.
The initial transport will be HTTP and HTTPS, protocols around
which most sites have developed mature access policies. Support
for most common HTTP/HTTPS load-balancing, redirection, and
proxying techniques will be implemented, making the system easy
to deploy in a variety of scenarios. Additional transports may
be investigated during the course of the project or as future
work.
The project does not define a default mechanism for building
software as part of the packaging process. The project team
believes strongly that software builds are a separate function,
and probably also agrees that different kinds of software may
require different build techniques.
More controversially, the project, in an attempt to increase
system safety and to reduce developer burden, removes the notion
of arbitrary context scripting from packaging. (This removal
means that the legacy packaging system must remain on the system
for long-term compatibility.) Empirical evidence from the
prototype phase has so far borne out this decision.
4.2. Bug/RFE Number(s):
As an example of the kinds of defects and RFEs intended to be
resolved by this project, we present the following selection of
bug IDs from the past 15 years:
1105830 pkgadd and pkgrm should be able to handle dependency ordering
1149607 Package dependencies hidden within a cluster.
1165888 RFE: allow non-root users to install software using the package mechanis
1184238 patches should be fully managed by package utilites
1208431 pkgrm with no arguments defaults to all
1249015 pkgadd requires root access
4202113 pkginfo command is ridiculously slow
4240078 pkgadd should not allow an intel package to install on Sparc and visa-ve
4385316 RFE Support pkgadd of clusters
4480153 Improvements desired for pkg management
4762470 pkgadd: soft dependencies
4795539 pkgadd should check dependencies of all packages provided on the command
4847723 rem_drv in preremove scripts should have consistent usage model
4939605 grep-friendly pkgchk -l variant desired
5012345 request for tool to list package dependencies
6208580 pkgadd/pkgrm should be smarter about dependancies
6246595 Sun's package management needs improvement
6491381 Create audit log for packaging and patch commads
In contrast,
1181241 wants to split large binary across multiple floppies with pkgmk
will not be addressed by this proposal.
4.3. In Scope:
Package-service delivery and containment relationships.
Package installation behaviour in virtualized environments.
4.4. Out of Scope:
Specific operational scenarios for repositories operated by Sun
Microsystems.
Provision of a GUI/BUI for package management.
Specific package contents and manifests.
4.5. Interfaces:
pkg(5) will present a substantial set of new and modified interfaces
to the core system. In particular, documented definitions of
- retrieval client CLI,
- publication client CLI,
- administrative and server CLIs,
- client metadata representations,
- server metadata representations,
- retrieval and publication protocol operations,
- a dynamic language API to access packaging functions,
- an on-disk package format,
- package metadata conventions,
- available package constituents ("actions"), and
- package naming and versioning conventions,
will be presented as interfaces introduced by this project.
It is possible that some of the nominally private interfaces
associated with legacy packaging will be affected; at a minimum,
files previously delivered via legacy packaging will no longer
be tracked by the legacy system. This outcome could result in a
correctly functioning system that presents very differently in
terms of file-package membership when interrogated using the
legacy packaging API.
Various components of the project will be introduced at each
stability and/or commitment level. The components are being
engineered such that the public interfaces can be evolved
compatibly, once the initial development is complete. (In fact,
the prototype is expected to support this evolution during the
development phase.)
The components are currently implemented in Python
(PSARC/2005/532, PSARC/2005/555, PSARC/2006/666).
4.6. Doc Impact:
The project expects to provide reference manual pages for each
of the groups of interface identified above. Furthermore, the
project expects to provide a Developer's Guide to replace the
current Application Packager's Guide.
4.7. Admin/Config Impact:
Substantial new capabilities in software installation will
become available.
A related project to produce a package manipulation GUI is being
pursued.
4.8. HA Impact:
None known; dependent on specific impacts of legacy packaging on
these capabilities.
4.9. I18N/L10N Impact:
Commands will require localization, as will any publically
committed library or equivalent APIs.
4.10. Packaging & Delivery:
All package, cluster, and metacluster boundaries will be
examined in the course of the project.
The primary upgrade mechanism for operating systems using pkg(5)
will be achieved via pkg(5) components; this mechanism is
expected to replace the current standalone upgrade and
LiveUpgrade paths. The replacement is expected, in concert with
the SnapUpgrade project, to present capabilities that equal or
exceed those of LiveUpgrade.
4.11. Security Impact:
In the current implementation, the protocol is built atop access
to HTTP and/or HTTPS. Accordingly, the server side will
potentially listen on ports associated with those services.
The server and client side will require access to key and
certificate management interfaces.
A mechanism for signing repository catalogs and package
manifests will be a part of the publication interface. A
corresponding mechanism for verifying signed catalogs and
manifests will be implemented for the installation client.
These mechanisms will apply to both packages in a network
package repository and packages in their on-disk representation.
4.12. Dependencies:
The project is dependent on SnapUpgrade for coherent collection,
organization, and activation of filesystem snapshots.
5. Reference Documents:
Project site:
http://opensolaris.org/os/project/pkg/
Project team members have written a number of informal essays on
various goals--problems to solve, outcomes to avoid, hopes to
realize--on aspects of the project:
- General observations:
http://blogs.sun.com/sch/entry/observations_on_packaging
- On testability and complexity costs with the current patching
methods:
http://blogs.sun.com/barts/entry/rethinking_patching
- Eliminating scripting in a packaging system
http://blogs.sun.com/sch/entry/pkg_1_a_no_scripting
- Keep software builds separate from software delivery:
http://blogs.sun.com/sch/entry/pkg_leaving_the_build_system
- Keeping critical metadata back the packaging system, rather
than in the installer:
http://blogs.sun.com/sch/entry/pkg_no_more_installer_magic
Related efforts in the Caiman project:
- Snap Upgrade,
http://opensolaris.org/os/project/caiman/Snap_Upgrade/
- Distribution Constructor,
http://opensolaris.org/os/project/caiman/Constructor/
6. Resources and Schedule:
6.1. Projected Availability:
2008
6.2. Cost of Effort:
Unable to estimate at present time.
6.4. Product Approval Committee requested information:
6.4.1. Consolidation or Component Name:
ON
6.5. ARC review type:
Standard.
6.6. ARC Exposure: open
6.6.1. Rationale: Part of OpenSolaris
7. Prototype Availability:
7.1. Prototype Availability:
Prototype exit criteria are: ability to support multiple transports,
some access control capability, constrained dependency support,
bulk of OpenSolaris-specific actions.
7.2. Prototype Cost:
Unable to estimate.

205
doc/pkg5_docs/parallel-linked-images.txt Normal file
View file

@ -0,0 +1,205 @@
.. This document is formatted using reStructuredText, which is a Markup
Syntax and Parser Component of Docutils for Python. An html version
of this document can be generated using the following command:
rst2html.py doc/parallel-linked-images.txt >doc/parallel-linked-images.html
======================
Parallel Linked Images
======================
:Author: Edward Pilatowicz
:Version: 0.1
Problems
========
Currently linked image recursion is done serially and in stages. For
example, when we perform an "pkg update" on an image then for each child
image we will execute multiple pkg.1 cli operations. The multiple pkg.1
invocations on a single child image correspond with the following
sequential stages of pkg.1 execution:
1) publisher check: sanity check child publisher configuration against
parent publisher configuration.
2) planning: plan fmri and action changes.
3) preparation: download content needed to execute planned changes.
4) execution: execute planned changes.
So to update an image with children, we invoke pkg.1 four times for each
child image. This architecture is inefficient for multiple reasons:
- we don't do any operations on child images in parallel
- when executing multiple pkg.1 invocations to perform a single
operation on a child image, we are constantly throwing out and
re-initializing lots of pkg.1 state.
To make matters worse, when as we execute stages 3 and 4 on a child
image the pkg client also re-executes previous stages. For example,
when we start stage 4 (execution) we re-execute stages 2 and 3. So for
each child we update we end up invoking stage 2 three times, and stage 3
twice. This leads to bugs like 18393 (where it seems that we download
packages twice). It also means that we have caching code buried within
the packaging system that attempts to cache internal state to disk in an
effort to speed up subsequent re-runs of previous stages.
Solutions
=========
Eliminate duplicate work
------------------------
We want to eliminate a lot of the duplicate work done when executing
packaging operations on children in stages. To do this we will update
the pkg client api to allow callers to:
- Save an image plan to disk.
- Load an image plan from disk.
- Execute a loaded plan from disk without first "preparing" it. (This
assumes that the caller has already "prepared" the plan in a previous
invocation.)
In addition to eliminating duplicated work during staged execution, this
will also allow us to stop caching intermediate state internally within
the package system. Instead client.py will be enhanced to cache the
image plan and it will be the only component that knows about "staging".
To allow us to save and restore plans, all image plan data will be saved
within a PlanDescription object, and we will support serializing this
object into a json format. The json format for saved image plans is an
internal, unstable, and unversioned private interface. We will not
support saving an image plan to disk and then executing it later with a
different version of the packaging system on a different host. Also,
even though we will be adding data into the PlanDescription object we
will also not be exposing any new information about an image plan to via
the PlanDescription object to api consumers.
An added advantage of allowing api consumers to save an image plan to
disk is that it should help with our plans to have the api.gen_plan_*()
functions to be able to return PlanDescription object for child images.
A file descriptor (or path) associated with a saved image plan would be
one way for child images to pass image plans back to their parent (which
could then load them and yield them as results to api.gen_plan_*()).
Update children in parallel
---------------------------
We want to enhance the package client so that it can update child images
in parallel.
Due to potential resource constraints (cpu, memory, and disk io) we
cannot entirely remove the ability to operate on child images serially.
Instead, we plan to allow for a concurrency setting that specifies how
many child images we are willing to update in parallel. By default when
operating on child images we will use a concurrency setting of 1, this
maintains the current behavior of the packaging system. If a user wants
to specify a higher concurrency setting, they can use the "-C N" option
to subcommands that recurse (like "install", "update", etc) or they can
set the environment variable "PKG_CONCURRENCY=N". (In both cases N is
an integer which specifies the desired concurrency level.)
Currently, pkg.1 worker subprocesses are invoked via the pkg.1 cli
interfaces. When switching to parallel execution this will be changed
to use a json encoded rpc execution model. This richer interface is
needed to allow worker processes to pause and resume execution between
stages so that we can do multi-staged operations in a single process.
Unfortunately, the current implementation does not yet retain child
processes across different stages of execution. Instead, whenever we
start a new stage of execution, we spawn one process for each child
images, then we make a remote procedure call into N images at once
(where N is our concurrency level). When an RPC returns, that child
process exits and we start a call for the next available child.
Ultimately, we'd like to move to model where we have a pool of N worker
processes, and those processes can operate on different images as
necessary. These processes would be persistent across all stages of
execution, and ideally, when moving from one stage to another these
processes could cache in memory the state for at least N child images so
that the processes could simply resume execution where they last left
off.
The client side of this rpc interface will live in a new module called
PkgRemote. The linked image subsystem will use the PkgRemote module to
initiate operations on child images. One PkgRemote instance will be
allocated for each child that we are operating on. Currently, this
PkgRemote module will only support the sync and update operations used
within linked images, but in the future it could easily be expanded to
support other remote pkg.1 operations so that we can support recursive
linked image operations (see 7140357). When PkgRemote invokes an
operation on a child image it will fork off a new pkg.1 worker process
as follows:
pkg -R /path/to/linked/image remote --ctlfd=5
this new pkg.1 worker process will function as an rpc server which the
client will make requests to.
Rpc communication between the client and server will be done via json
encoded rpc. These requests will be sent between the client and server
via a pipe. The communication pipe is created by the client, and its
file descriptor is passed to the server via fork/exec. The server is
told about the pipe file descriptor via the --ctlfd parameter. To avoid
issues with blocking IO, all communication via this pipe will be done by
passing file descriptors. For example, if the client wants to send a
rpc request to the server, it will write that rpc request into a
temporary file and then send the fd associated with the temporary file
over the pipe. Any reply from the server will be similarly serialized
and then sent via a file descriptor over the pipe. This should ensure
that no matter the size of the request or the response, we will not
block when sending or receiving requests via the pipe. (Currently, the
limit of fds that can be queued in a pipe is around 700. Given that our
rpc model includes matched requests and responses, it seems unlikely
that we'd ever hit this limit.)
In the pkg.1 worker server process, we will have a simple json rpc
server that lives within client.py. This server will listen for
requests from the client and invoke client.py subcommand interfaces
(like update()). The client.py subcommand interfaces were chosen to be
the target for remote interfaces for rpc calls for the following
reasons:
- Least amount of encoding / decoding. Since these interfaces are
invoked just after parsing user arguments, they mostly involve simple
arguments (strings, integers, etc) which have a direct json encoding.
Additionally, the return values from these calls are simple return
code integers, not objects, which means the results are also easy to
encode. This means that we don't need lots of extra serialization /
de-serialization logic (for things like api exceptions, etc).
- Output and exception handling. The client.py interfaces already
handle exceptions and output for the client. This means that we don't
have to create new output classes and build our own output and
exception management handling code, instead we leverage the existing
code.
- Future recursion support. Currently when recursing into child images
we only execute "sync" and "update" operations. Eventually we want to
support pkg.1 subcommand recursion into linked images (see 7140357)
for many more operations. If we do this, the client.py interfaces
provide a nice boundary since there will be an almost 1:1 mapping
between parent and child subcommand operations.
Child process output and progress management
--------------------------------------------
Currently, since child execution happens serially, all child images have
direct access to standard out and display their progress directly there.
Once we start updating child images in parallel this will no longer be
possible. Instead, all output from children will be logged to temporary
files and displayed by the parent when a child completes a given stage
of execution.
Additionally, since child images will no longer have access to standard
out, we will need a new mechanism to indicate progress while operating
on child images. To do this we will have a progress pipe between each
parent and child image. The child image will write one byte to this
pipe whenever one of the ProgressTracker`*_progress() interfaces are
invoked. The parent process can read from this pipe to detect progress
within children and update its user visible progress tracker
accordingly.

67
doc/pkg5_docs/pkg-guide-web.css Normal file
View file

@ -0,0 +1,67 @@
/*
* This specification may be enough for adequate printing.
*/
@page {
margin: 2.5cm;
}
body {
font-family: serif;
margin: 5%;
}
h1, h2, h3, h4, h5, h6 {
font-family: sans-serif;
}
a {
text-decoration: none;
color: #4444aa;
}
a:hover {
color: #222288;
background-color: #eeeeee;
}
.subtitle {
font-style: italic;
}
th .docinfo-name {
text-align: right;
}
div .admonition {
border-color: #aaaaaa;
border-right-style: none;
border-left-style: none;
border-top-style: solid;
border-top-width: thick;
border-bottom-style: solid;
border-bottom-width: thin;
margin-right: 20em;
}
div .sidebar {
border-color: #ffaaaa;
border-width: thin;
float: right;
width: 20em;
background-color: #eeeecc;
}
p .sidebar-title {
font-family: sans-serif;
text-size: 24px;
}
p .topic-title {
font-family: sans-serif;
text-size: 24px;
}

195
doc/pkg5_docs/pkg-states.rst Normal file
View file

@ -0,0 +1,195 @@
.. CDDL HEADER START
.. The contents of this file are subject to the terms of the
Common Development and Distribution License (the "License").
You may not use this file except in compliance with the License.
.. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
or http://www.opensolaris.org/os/licensing.
See the License for the specific language governing permissions
and limitations under the License.
.. When distributing Covered Code, include this CDDL HEADER in each
file and include the License file at usr/src/OPENSOLARIS.LICENSE.
If applicable, add the following below this CDDL HEADER, with the
fields enclosed by brackets "[]" replaced with your own identifying
information: Portions Copyright [yyyy] [name of copyright owner]
.. CDDL HEADER END
.. Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
.. :vim set expandtab:
PACKAGE STATES
--------------
Server states
~~~~~~~~~~~~~
We phrase the state machine in terms of a single removal state,
ABANDONED, which covers both the never-created package instance
(even with a series of never-finished transaction events). It may
be more appropriate to separate the ABANDONED state into
TX_ABANDONED and PKG_DELETED.
This leaves us with a state transition diagram like::
0
|
|
v
+--> TRANSACTING --> ABANDONED <--+
| | ^ |
| | | |
| v | |
| SUBMITTED ----> INCOMPLETE |
| | | |
| | | |
+--- PUBLISHED <---------+ |
| |
| |
+------------------------+
0 -> TRANSACTING
On initial package creation.
TRANSACTING -> ABANDONED
If initial package transaction never committed, commitment
failed, or explicitly dropped.
TRANSACTING -> SUBMITTED
On successful package transaction commitment. Packages with
syntax errors or immediate inconsistencies would have failed in
commitment.
SUBMITTED:
The package modified by the transaction is known by a specific
version. Its contents are in the repository.
SUBMITTED -> INCOMPLETE
If commitment included a deferred inconsistency (package
dependency is the only expected form), then the package is left
in the incomplete state.
INCOMPLETE:
The package with the specific version string is on the
incomplete list. Its contents are in the repository.
INCOMPLETE -> ABANDONED
If incomplete package explicitly removed. (Possibly by
timeout from arrival in INCOMPLETE.)
SUBMITTED -> PUBLISHED
If commitment had no deferred inconsistencies, then the package
is considered ready for publication.
INCOMPLETE -> PUBLISHED
If the deferred inconsistencies, upon reevaluation, are now held
to no longer be inconsistent, then the package is considered
ready for publication.
PUBLISHED:
The package with the specific version string is present in the
catalog. Its contents remain in the repository.
PUBLISHED -> ABANDONED
On manual request for package decommissioning, the package will
be moved to the abandoned state.
ABANDONED:
A package with a specific version string is no longer in the
catalog or on the incomplete list. Its contents, if they were
in the repository, should be removed from the repository.
XXX ARCHIVED might be a special state connected to PUBLISHED, or
merely a substate. An archived package has its manifest and
contents in the repository, but cannot be installed on a client.
The point of including ARCHIVED packages is to allow client
deduction on a system not installed by the pkg system.
Client states
~~~~~~~~~~~~~
Within an image, each package version can have only one state, as
given in the following diagram::
0
|
v
IN_CATALOG ---------> OUT_CATALOG
| ^
+--->---+---<-------+ |
| | | |
| v | |
| INSTALLED --> FROZEN |
| | |
| | |
| v |
+-- PRESERVED |
| | |
| | |
| v |
+-- DELETED -------------------+
0 -> IN_CATALOG:
A catalog update with new entries.
IN_CATALOG:
An entry for this package is available in the locally installed
catalog.
IN_CATALOG -> OUT_CATALOG:
Entry formerly present on local catalog is no longer published by any
repository. (Package never locally installed.)
OUT_CATALOG:
Although a formerly known package, no entry for this package is
available in the locally installed catalog. An INSTALLED or
FROZEN package can never be OUT_CATALOG, as the system will
preserve the entry until the package is no longer in a locally
public state.
IN_CATALOG -> INSTALLED:
Transition takes place on package installation.
INSTALLED -> FROZEN:
Transition takes place if manually frozen or frozen by virtue of
reference from another package group.
FROZEN -> INSTALLED:
Manually unfrozen, or unfrozen by reference drop due to
change in formerly referring package group.
INSTALLED -> PRESERVED:
Old copies moved aside during upgrade of package components, but
not removed.
PRESERVED -> DELETED:
Old copies removed.
DELETED -> OUT_CATALOG:
Package has been removed from client catalog. Client software
would take a PRESERVED package through DELETED automatically to
reach OUT_CATALOG.
PRESERVED -> INSTALLED:
Package reinstalled or reversed.
DELETED -> INSTALLED:
Package reinstalled.
XXX How does the ZFS snapshot (that we might have taken prior to an
operation) get represented in the state? Is there an image state
machine model as well?
XXX Need a substate of INSTALLED for damaged packages.
XXX Need a substate of INSTALLED for packages where the global zone
portion is available, but local installation has not finished. Can
we generalize this state for all diskless installs?

165
doc/pkg5_docs/pkg-states.txt Normal file
View file

@ -0,0 +1,165 @@
pkg
PACKAGE STATES
:vim set expandtab:
1. Server states
We phrase the state machine in terms of a single removal state,
ABANDONED, which covers both the never-created package instance
(even with a series of never-finished transaction events). It may
be more appropriate to separate the ABANDONED state into
TX_ABANDONED and PKG_DELETED.
0
|
|
v
+--> TRANSACTING --> ABANDONED <--+
| | ^ |
| | | |
| v | |
| SUBMITTED ----> INCOMPLETE |
| | | |
| | | |
+--- PUBLISHED <---------+ |
| |
| |
+------------------------+
0 -> TRANSACTING
On initial package creation.
TRANSACTING -> ABANDONED
If initial package transaction never committed, commitment
failed, or explicitly dropped.
TRANSACTING -> SUBMITTED
On successful package transaction commitment. Packages with
syntax errors or immediate inconsistencies would have failed in
commitment.
SUBMITTED:
The package modified by the transaction is known by a specific
version. Its contents are in the repository.
SUBMITTED -> INCOMPLETE
If commitment included a deferred inconsistency (package
dependency is the only expected form), then the package is left
in the incomplete state.
INCOMPLETE:
The package with the specific version string is on the
incomplete list. Its contents are in the repository.
INCOMPLETE -> ABANDONED
If incomplete package explicitly removed. (Possibly by
timeout from arrival in INCOMPLETE.)
SUBMITTED -> PUBLISHED
If commitment had no deferred inconsistencies, then the package
is considered ready for publication.
INCOMPLETE -> PUBLISHED
If the deferred inconsistencies, upon reevaluation, are now held
to no longer be inconsistent, then the package is considered
ready for publication.
PUBLISHED:
The package with the specific version string is present in the
catalog. Its contents remain in the repository.
PUBLISHED -> ABANDONED
On manual request for package decommissioning, the package will
be moved to the abandoned state.
ABANDONED:
A package with a specific version string is no longer in the
catalog or on the incomplete list. Its contents, if they were
in the repository, should be removed from the repository.
XXX ARCHIVED might be a special state connected to PUBLISHED, or
merely a substate. An archived package has its manifest and
contents in the repository, but cannot be installed on a client.
The point of including ARCHIVED packages is to allow client
deduction on a system not installed by the pkg system.
2. Client states
0
|
v
IN_CATALOG ---------> OUT_CATALOG
| ^
+--->---+---<-------+ |
| | | |
| v | |
| INSTALLED --> FROZEN |
| | |
| | |
| v |
+-- PRESERVED |
| | |
| | |
| v |
+-- DELETED -------------------+
0 -> IN_CATALOG:
A catalog update with new entries.
IN_CATALOG:
An entry for this package is available in the locally installed
catalog.
IN_CATALOG -> OUT_CATALOG:
Entry formerly present on local catalog is no longer published by any
repository. (Package never locally installed.)
OUT_CATALOG:
Although a formerly known package, no entry for this package is
available in the locally installed catalog. An INSTALLED or
FROZEN package can never be OUT_CATALOG, as the system will
preserve the entry until the package is no longer in a locally
public state.
IN_CATALOG -> INSTALLED:
Transition takes place on package installation.
INSTALLED -> FROZEN:
Transition takes place if manually frozen or frozen by virtue of
reference from another package group.
FROZEN -> INSTALLED:
Manually unfrozen, or unfrozen by reference drop due to
change in formerly referring package group.
INSTALLED -> PRESERVED:
Old copies moved aside during upgrade of package components, but
not removed.
PRESERVED -> DELETED:
Old copies removed.
DELETED -> OUT_CATALOG:
Package has been removed from client catalog. Client software
would take a PRESERVED package through DELETED automatically to
reach OUT_CATALOG.
PRESERVED -> INSTALLED:
Package reinstalled or reversed.
DELETED -> INSTALLED:
Package reinstalled.
XXX How does the ZFS snapshot (that we might have taken prior to an
operation) get represented in the state? Is there an image state
machine model as well?
XXX Need a substate of INSTALLED for damaged packages.
XXX Need a substate of INSTALLED for packages where the global zone
portion is available, but local installation has not finished. Can
we generalize this state for all diskless installs?

138
doc/pkg5_docs/pkgs-and-groups.txt Normal file
View file

@ -0,0 +1,138 @@
PSARC/2008/190
pkg(5): image packaging system
PACKAGES AND GROUPS
1. Definitions
To be consistent with the system, following the introduction of the
fault management architecture, each package is named by an FMRI in
the "pkg:" scheme. That is, we have
pkg://publisher/pkg_name@version
The publisher is generally expected to be a forward or reverse
domain name identifying the publisher from which a package can be
retrieved. Publishers which cannot be determined to be a domain
name are legitimate, but optional functionality, like automatic
server discovery for a particular publisher, may fail to work.
In the examples that follow, we use "opensolaris.org" as a generic
publisher.
The pkg_name, like service names, can be split into a category,
subcategories, and a basename. This namespace might be populated
with "manifest" and other metadata endpoints, as well as the SHA-1
names of the package's included files. (Although the direct access
to properties of the svc FMRI scheme has been rarely used.)
A "group package" is a package that depends upon (minimum versions
of) other packages, as well as optionally delivering files or other
actions. An "incorporation" is a group package that places forward
constraints upon the versions of the packages it depends upon, which
restricts the interval of valid package versions to one the author
of the incorporation believes functional.
2. Namespace
2.1. Single namespace, separate publishers
The primary design point of the package namespace is to allow
multiple package producers to co-exist in a single namespace, so
that images can switch between equivalent components from different
producers.
2.2. Domain-name-based escape
At any point in the category hierarchy, a safe namespace can be
created by using the forward or reverse domain name, either as a
subcategory or as a comma-led prefix to a subcategory or package
base name. (This scheme is similar to FMRI namespace escapes in
smf(5), although we are eliminating use of stock symbol prefixes.)
For instance, when example.com wishes to publish the "whiskers"
package without reference to a larger namespace convention it can
use any of the following examples
pkg://opensolaris.org/.../example.com/whiskers
pkg://opensolaris.org/.../com.example/whiskers
pkg://opensolaris.org/.../example.com,whiskers
pkg://opensolaris.org/.../com.example,whiskers
pkg://opensolaris.org/.../example.com,software/whiskers
pkg://opensolaris.org/.../com.example,software/whiskers
and so forth.
2.2. Locally reserved namespace
The top-level "site" category is reserved for use by the entity that
administrates the server. Neither the organizations producing the
operating system nor those providing additional software components
may use the site category.
The top-level "vendor" category is reserved for use by organizations
providing additional. The leading subcategory must be a domain.
That is, if example.com wishes to publish the "whiskers" package in
the vendor category, it would use a package name like
pkg://opensolaris.org/vendor/example.com/whiskers
2.3. Additional reserved namespace
The top-level "cluster", "feature", "group", "metacluster", and
"service" categories are all reserved for future use.
Inception note: some or all of these reservations may be eliminated
or reduced when the single namespace convention reaches its final
form.
2.4. Single namespace conventions
2.4.1. Discussion
Packaging systems and distributions appear to have explicit
categories, subcategories, and potentially larger groups; some
distributions have explicit fields for these values, others use
tagging or multi-valued fields, which allows them to classify
certain packages multiply. For the FMRI namespace case, the system
is similar to a packaging system with multiple, single-valued,
category fields.
There appear to be two standard approaches to categorizing packages:
1. By what primary class of thing a package delivers.
2. By what area of functionality a package's contents address.
In the first approach, we get strict top-level categories like
"application", "driver", "library", and "system" or "kernel", as
well as potentially overlapping categories like "utility" and
"tool". Use of the leading subcategory is limited, and generally
given to the subsystem involved. A relatively detailed worked
example of the X11 subsystem under this scheme is given in
http://mail.opensolaris.org/pipermail/pkg-discuss/2008-February/001838.html
In the second, we would also see categories like these, but leading
subcategory is much more likely to classify according to
functionality, so that we would see "application/mail",
"application/web", "application/compiler", and so forth. Most
network packaging systems appear to classify in this fashion.
An appealing variation of the second form is to rotate all of the
non-"application" packages under a "system" mega-category, such that
all of the leaf packages (with the possible exception of device
drivers) are exposed at the category level. Table 1 shows some
example transformations under this scheme.
FROM TO
application/web/firefox web/firefox
application/compiler/gcc4 compiler/gcc4
library/c system/library/c
kernel/generic system/kernel/generic
Table 1: Rotating non-application categories under system.

22
doc/pkg5_docs/protocol-versioning.txt Normal file
View file

@ -0,0 +1,22 @@
For obvious compatibility reasons, the over-the-wire protocol needs to be
versioned. Since we're using HTTP, it makes sense to simply version the
URLs used.
The scheme is pretty simple:
http://host:port/operation/version(/extra)?
where "version" is a simple integer.
Open questions:
- Are there operations where we don't want to require a version?
- Is there a use for a "300 Multiple Choices" HTTP return code?
- If we always try the client-optimal version first and then back off
linearly, we'll want to cache the version that succeeded so we don't
spend all of our time trying versions that don't exist.
- Alternatively, we can call "versions" to find out what versions the
server supports for each operation, and cache that.

33
doc/pkg5_docs/publication.txt Normal file
View file

@ -0,0 +1,33 @@
pkg
PUBLICATION AND TRANSACTIONS
:vim set expandtab:
1. Problem
We prefer having "fat" packages, to having a plurality of speciated
nodes in our version tree. That is, we hope to have exception
packages have a single platform attribute. This means that we have
build files on platform A and B
commit fat package containing (A, B) files
as the gross ordering of operations. Taken strictly, this ordering
would be too restrictive, particularly once the set of platforms
which might participate in a package goes beyond two.
2. Ways to handle multi-platform transactions
Treat every transaction on v as an update to the version, v. A
platform-specific update would include both common files and the
platform-tagged or architecture-tagged files. If common files
differed between v:t_1 and v:t_2, then this transaction isn't
valid--it's branch modifying.
Otherwise, the v:t_2 package contains the common files and the set
of flavoured files submitted so far.
(This approach evades the problem of expecting a platform-specific
update to only replace files with alternate versions.)

98
doc/pkg5_docs/publisher_search_order.txt Normal file
View file

@ -0,0 +1,98 @@
In order to select packages from a variety of sources, pkg(5) supports
the configuration of multiple publishers. We've started with the
concept of a preferred publisher, but some ambiguity has arisen as to
what should happen when publishers are deleted, added, etc. In
addition, the single level of selection afforded by the preferred
designation appears to be somewhat limiting; at the same time, the
rules for selecting amongst multiple publishers of the same package
need clarification rather than additional complexity.
We propose the idea of publisher search order, starting with the
preferred publisher. Adding a publisher adds a new publisher at the
end of the search order; if a publisher is made preferred it is moved
to the front of the search order. Deleting or disabling a publisher
causes it to be removed from the search order. Re-enabling a publisher
causes it to added at the end of the search order.
When a package is nominated for installation without an explicit
publisher being specified, the first publisher found in the search
order to provide the package is selected. Once installed, subsequent
updates to that package by default always occur from that publisher,
even if a publisher earlier in the search order starts publishing a
package with the same name. This behavior of a publisher is
characterized as "sticky", and is the default. It can be disabled on
a per-publisher basis, and such disablement is useful mostly for
developers seeking to replace a portion of their packages w/
development versions. If a publisher is made "non-sticky", its
packages are searched for as on initial installation on every update -
no preference is afforded by the previous installation. Deleted/disabled
publishers are made non-sticky.
Each selection of the publisher for a package is made independently
according to the algorithms above; there is no implicit inheritance
of publisher across dependencies of any type.
The above suggests the following additions to the set-publisher
subcommand of pkg:
set-publisher [--search-before=publisher] [--search-after=publisher] publisher
set-publisher [--sticky] [--non-sticky] publisher
--search-before=publisherB publisherA causes publisher A to be moved
from its current place in the search order to be just ahead of publisher B.
--search-after=publisherB publisherA causes publisher A to be moved
from its current place in the search order to be just behind publisher B.
Specifying --non-sticky causes this publisher not to automatically
be selected for all updates to packages already installed from this
publisher; instead, publishers searched earlier are automatically preferred.
--sticky causes the original behavior to be restored for subsequent
updates.
Use cases
---------
Normal user getting packages from pkg.opensolaris.org and datahavens.org:
1) Installs system as per usual, points preferred to usual
best available publisher - say, pkg.opensolaris.org
2) Adds new publisher datahavens.org to acquire mplayer.
Without specifying search order, new publisher is appended
to the current order.
Project Developer:
1) Installs system as per usual, points preferred to usual
best available publisher - say, pkg.opensolaris.org
2) Adds new publisher "MyOwnRepo" pointing at his
latest and greatest bits. Note that his repo is lowest
in search order, but since his package names are unique no issues
arise.
Contrib User that prefers supported bits:
1) Installs system as per usual, points preferred to usual
best available publisher - say, pkg.opensolaris.org
2) Adds contrib repo after p.o.o, and makes it non-sticky.
3) Adds packages from contrib as desired.
4) When and if packages move to p.o.o, they're automatically
updated from p.o.o on the next image update or
pkg install ....
OpenSolaris developer:
1) Installs system as per usual, points preferred to usual
best available publisher - say, pkg.opensolaris.org/dev
2) Adds new preferred publisher "MyOwnRepo" pointing at his
latest and greatest bits; making it preferred places it
first in the search order.
3) Pkg.opensolaris.org is made non-sticky to cause pkgs from MyOwnRepo
to replace those from pkg.opensolaris.org on next update
4) If additional fresh bits are required, additional development
repos can be added and placed ahead of pkg.opensolaris.org
in the search order; in this way multiple consolidations
can be kept at the bleeding edge.

92
doc/pkg5_docs/razor.txt Normal file
View file

@ -0,0 +1,92 @@
Requirements gathering for new packaging system
-----------------------------------------------
Some of these requirements will be satisfied in part by the use of ZFS
as a root filesystem.
In no particular order:
A new packaging system must:
* replace existing patching/upgrade/live upgrade/Jumpstart/Jet/SUC/...
functionality; there should be one way of managing all software change on
Solaris 11.
* allow fine grain control of installation contents to support minimization
A customer should be able to select the desired functionality, and have the
system bring in the closure of the dependency graph.
* support the installation of packages to a directory for diskless and Xen
configs.
* be repository based to faciltate efficient software distribution.
* support the user's connection to multiple repositories to provide
different types of software, newer software, different vendors/suppliers,
etc.
* deal with zones:
* maintain global zone, whole root zones in sync
* cope w/ directory level split between global and local zones, or
eliminate them.
* allow a package to be installed in alternative (non default) locations.
* allow the installation of multiple instances/revisions of the same package
in different locations.
* manage package dependencies in multiple ways:
* allow a set of packages to be managed as a group; all packages
must transition together. Groups may include other groups.
* allow specification of a minimum version level
* dependency graphs need not be acyclic
* permit the selection of alternative software streams available from a single
repository.
* permit the "tagging" of packages with interesting information such as
external packaging version number, features provided (at least partially)
by this packages, etc.
* permit the creation of alternative package branches to represent either early
platform introduction or customer-specific fixes that are later merged into
the mainline.
* manage updates to client system in a transactional fashion; either we run the
old bits or the new bits, never some of each.
* support secure upgrading through firewalls, etc, w/o special handling,
ports opened, etc, on the client side. It must be possible to both allow
and disallow anonymous access to the repository, and offer fine grain access
controls.
* be robust in the face of filesystem damage on the client side. It must be
possible to identify where the system doesn't match the packaging information,
and to be able to repair any damage by restoring damaged components from the
repository.
* be open source, and included in OpenSolaris. All the tools necessary to build
and distribute OpenSolaris via a repository should be part of OpenSolaris.
* support interactive development. A developer should be readily able to
create a new repository, make changes, build, commit the changes to the
repository and update his machine with those changes and reboot.
* be of acceptable performance. Upgrade operations should not reacquire files
already in place on the system. As much as possible, packaging operations
should be order (1) rather then order(number of zones). Packaging operations
should not be significantly affected by the number of packages already
installed on the system
A new packaging system must not:
* require changing the build system of consolidations contributing to
software respository. Different parts of OpenSolaris build in many different
ways; forcing them all to be the same is unacceptable.
* require the use of non-local resources for clients; security conscious
companies must be able to run their own repositories completely disconnected
from any external networks.

105
doc/pkg5_docs/repository.txt Normal file
View file

@ -0,0 +1,105 @@
pkg
REPOSITORIES
1. Summary
2. Discussion
2.1. Repository configuration
2.2.1. Configuration inheritance
The pkg.depotd(5) server may need to access repository configuration data when
smf(5) is unavailable. As such, a repository must have a cached
configuration state.
Roughly, these aspects are sufficient to constrain our configuration
behavior:
request properties from svc://application/pkg
if unavailable, examine configuration cache
if undefined or unavailable, use hard-coded defaults
2.2.2. Configuration components
Repository attributes. The repository has a collection of simple
attributes for providing various bits of metadata and configuration
information. Note that changing these values requires a restart
of any pkg.depotd processes referencing the repository so that
changes will be reflected in operations and output.
repository/ Property group of type "application"
/name A short, descriptive name for the repository.
Examples: "opensolaris.org base repository"
"opensolaris.org contrib repository"
/description A descriptive paragraph for the repository.
/maintainer A human readable string describing the entity
maintaining the repository. For an individual,
this string is expected to be their name, or
name and email.
Examples: "Project Indiana"
"Project Indiana <indiana-discuss@
opensolaris.org>"
/maintainer_url A URL associated with the entity maintaining the
repository.
Example:
"http://www.opensolaris.org/os/project/indiana/"
/detailed_url One or more URLs to pages or sites with further
information about the repository.
Example: "http://www.opensolaris.org/"
feed/ Property group of type "application"
/id A Universally Unique Identifier (UUID) used to
permanently, uniquely identify the feed.
Changing this value can have unexpected effects
on feed consumers. In addition, when serving
multiple copies of a repository, each copy's
cfg_cache must have the same value for this
attribute.
/name A short, descriptive name for RSS/Atom feeds
generated by the depot serving the repository.
Example: "opensolaris.org packaging feed"
/description A descriptive paragraph for the feed.
/authority A fully-qualified domain name or email address
that will be used to generate a unique
identifier for each entry in the feed. Changing
this value can have unexpected effects on feed
consumers.
Examples: "opensolaris.org"
"indiana-discuss@opensolaris.org"
/icon A filename of a small image that will be used to
visually represent the feed (e.g. web
links to the repository, or the icon shown in a
user agent's address bar). This file must be
located in the depot server's inst_root
directory.
/logo A filename of a large image that will be used by
user agents to visually brand or identify the
feed. This file must be located in the depot's
inst_root directory.
/window A numeric value representing the number of
hours, before the feed for the repository was
last generated, to include when creating the feed
for the repository updatelog. The default value
is "24".
Example: "48"
3. References

38
doc/pkg5_docs/rest.txt Normal file
View file

@ -0,0 +1,38 @@
pkg
Use of REST
We use REST over HTTP as the primary networking API between client and
server. That choice means that
- the client can be reimplemented,
- the server can be reimplemented, and
- decorations on a transaction, like authentication, encryption, and
redirection, can be handled using the enormous technology set around
HTTP.
The first two are true of any well-defined protocol, but we benefit in
this case from the wide availability of HTTP client and server
implementation starting points.
Installer API:
GET /catalog
GET /version/pkg_name
GET /depend/pkg_name
GET /data/pkg_name[/from[/to]]
Packager API:
POST /trans/pkg_name
Returning a transaction ID
POST /add/trans_id/type
Plus metadata in submitted headers and contents as file body.
GET /summary/trans_id
POST /meta/trans_id/require
Dependency metadata in submitted headers.
POST /commit/trans_id
Returning package URL.

45
doc/pkg5_docs/rfes.txt Normal file
View file

@ -0,0 +1,45 @@
1. [sch] Put size in manifest so that we can do GET with offset to speed
retrieval. Example: reget on Windows.
2. [billm] ACLs and extended attributes. Are these both new action types?
3. [sch] Do we need other forms of signing, beyond publisher and depot SSL
certificates? Is it best to enforce a CA-based model, or should web
of trust also be allowed?
4. [dp] Package deletion should include a "subsumed-by" or "replaced-by". A
good example is when we stopped including Mozilla, its final
version--expressing the deletion--should have stated subsumption by
Firefox and Thunderbird.
5. [barts] Minimization boundaries based on setuid binaries.
6. [barts] Feature tagging. This should be offered via leaf packages
and grouping packages.
7. [dp] Present timestamp as UTC YYYYMMDDHHMMSS, as opposed to Unix
seconds.
8. [lianep] Be able to preserve specific files, even though the
package no longer provides them. This one's tricky: if you state,
"release file" in version 2, then an upgrade from v 1 to v 3 would
miss it (as v 2's manifest is not consulted). These kind of actions
could be treated as choking, or we could always examine intermediate
manifests.
9. [psa] Take a snapshot [of each affected filesystem] between every
package update operation in a larger image transaction, as opposed
to at the image transaction boundaries only.
10. [sch] Examine use of alternative, HTTP 1.1-friendly URL loading
modules. (Example: Duke's urlgrabber.)
11. [pelegri] Use of package-level metadata to provide additional
information, such as links to training/learning resources,
declarations of related packages, endorsements by certifying
publishers.
12. [sch] Support use of :timestamp field for "newer" and "older"
queries.

238
doc/pkg5_docs/sat.txt Normal file
View file

@ -0,0 +1,238 @@
It's become clear that our approach to evaluating
packaging dependencies and constraints needs to become
a lot more sophisticated as we've been trying to make
packaging metadata more accurately reflect the way
we build and test packages.
A significant part of the difficulty is dealing with
externally produced packages; if a variety of versions
are available we may need to iteratively test multiple
versions, evaluating their dependencies to find one that
is compatible w/ the constraints that may be active on
the current image.
One method of doing this sort of automated decision making
is to cast the problem as a series of boolean expressions,
and then apply a SAT solver to find a solution. These notes
describe the results of my experiments w/ the minisat solver
Stephen posted some time ago....
Notes:
--------
1) The presence of a particular package version is a
single boolean variable; True if it's present,
False if not.
The problem set handed to the SAT solver is a series
of clauses; each clause are boolean variables (or
their negation) or'd together. All clauses must be
true for the solution to exist.
The clauses need to encode the fact that only one version
of a package may be installed at a time, and also encode
all different package dependencies and constraints.
2) Each package has some number of versions, inherently ordered.
Only one version of a package may be installed at a time
pkg a -> a.1, a.2, a.3, a.4
pkg b -> b.1, b.2, b.3, b.4
Thus for "a":
!a.1 | !a.2
!a.1 | !a.3
!a.1 | !a.4
!a.2 | !a.3
!a.2 | !a.4
!a.3 | !a.4
where !a represents the negation of a.
This means that for N versions, we have N(N-1)/2 clauses;
pruning older non-accessible versions will be required to
bound memory consumption.
3) Each version of a package may have dependencies on other
packages, either w/ or w/o a version. The version specification
will likely not be fully specified (eg multiple versions
may satisfy this requirement).
4) dependencies may be of the following types:
required: fmri specifies minimum acceptable version
if a.1 requires b.2, b.3 or b.4:
!a.1 | b.2 | b.3 | b.4
optional: if present, fmri must be at this level or greater
if a.1 optionally requires b.3:
!a.1 | !b.1
!a.1 | !b.2
incorporate: if present, pkg must match fmri
if a.1 incorporates b.3:
!a.1 | !b.1
!a.1 | !b.2
!a.1 | !b.4
exclude: if present, pkg must be less that version in fmri:
if a.1 excludes b.3,
!a.1 | !b.3
!a.1 | !b.4
All of these are linear in the number of package versions
either meeting or failing to meet the dependency.
5) To express state, the presence of a package is encoded as a
clause. We compute the matching fmris and then construct
a clause that matches one of those fmris. Specifying a single
version requires that version to be present in the solution;
we can also specify current version or newer, or any version of
a package.
6) The SAT solver will find a particular solution to our packaging
problem, but there is no way of "preferring" newer packages, and
preventing the introduction of extraneous unneeded packages.
As a result, external optimization in the form of repeated
solution attempts w/ additional constraints is necessary.
The following algorithm has been implemented:
The packaging problem to be solved is expressed as a series of
boolean constraints, and a solution obtained. Then, for each
fmri appearing in the solution vector, all older versions are
excluded; in other words, if a.3 is part of the solution, then
subsequent solutions will not contain a.1 or a.2. Then a single
vector is added that is the negation of the just found vector,
and another solution is found. For example:
if solution is a.2, b.3, z.10, we add
# first negations of older versions
!a.1
!b.1
!b.2
!z.1
!z.2
...
!z.9
# now negate just found solution
!a.2 | !b.3 | !z.10
The latter vector requires that the new solution not contain
a.2 and b.3 and z.10; since we've excluded older versions we
will either get a vector that eliminates one of the packages
as unneeded (if dependencies allow) or one that has newer
versions of one of the needed pkgs.
We repeat the above process until a solution cannot be found;
the last found solution must therefore be the most optimal one.
The above technique may fail to find the overall optimal
solution if newer packages have incorporation dependencies
on earlier versions of their dependencies. This is expected
to be rare. Pruning the solution space to eliminate older
packages is necessary due to rapid solution space growth if
there are multiple versions that satisfy dependencies.
7) In order to prevent rapid growth of clause count as the
number of versions of packages increases, trimming the
solution space is essential. I'm currently using the
following techniques:
1) install a new package on existing system
identify any existing installed constraints,
and trim pkg catalog to eliminate versions
outside those constraints.
trim pkg catalog to exclude all pkg older than
those already installed
input to solver is trimmed catalog, and
vectors selecting any version of already installed
pkgs that meet constraints, plus a vector selected
any version of desired pkg.
2) upgrade to latest version of all available pkgs
identify any existing installed constraints,
and trim pkg catalog to eliminate versions
OLDER than those constraints.
trim pkg catalog to exclude all pkg older than
those already installed
input to solver is trimmed catalog, and
vectors selecting any version of already installed
pkgs
3) upgrade to specified version
identify any existing installed constraints,
and trim pkg catalog to eliminate versions
OLDER than those constraints.
trim pkg catalog to exclude all pkg older than
those already installed
input to solver is trimmed catalog, and
vectors selecting any version of already installed
pkgs, plus vector(s) selecting desired constraint(s).
8) One of the most difficult aspects of using a SAT solver
is providing a reasonable error message when no solution
can be found.
Some techniques that I'm experimenting with include:
Explicitly checking for obvious non-starters (pkg
version earlier than already installed, pkg version that
violates constraints on system) prior to passing to SAT
solver. This is needed to permit trimming in any case.
Using the pruned catalog to quickly evaluate the effect
of constraints.
Implementation details
-------------------------
combine catalog object w/ list of installed pkgs and proposed
changes:
class pkg_solver(object):
def __init__(self, catalog, existing_fmris):
def solve_install(existing_freezes, proposed_fmris):
"""tries to find solution that adds specified fmris
to existing set; any existing packages containing
incorporate dependencies which are at most only depended on
by name (no version) are frozen."""
def solve_reinstall(existing_freezes, proposed_fmris):
"""tries to find solution that replaces existing version
with specified version; this one allows stuff to go
backwards if specified on command line"""
def solve_uninstall(existing_freezes, proposed_fmris):
"""tries to remove specified package"""
def solve_update_all(existing_freezes):
"""find most recent version of all packages"""
solve* routines return a list of tuples (old_version, new_version)
for each fmri that is changing; new installs have None as old_version,
removals have None as new_version. A returned empty list indicates
that no action is needed.
A failure to find a solution throws an exception,
pkg_solver.No_Solution_Found.

121
doc/pkg5_docs/scripting.txt Normal file
View file

@ -0,0 +1,121 @@
Getting rid of install/upgrade scripting
The design of SVR4 packaging relies heavily on the use of scripting to
achieve common packaging operations. These scripts are delivered by
the package developers, and run in multiple installation contexts,
which may include install, alternate roots, zones, cross architecture
and cross OS versions. This has caused a host of complex and thorny
issues, and led directly to the IPS team's decision to eliminate the
use of scripting in IPS.
Instead, the IPS architecture requires software to be largely
self-assembling; changes in configuration are either detected at boot
time or the appropriate SMF services are restarted in the case of live
installation of packages. [In the few cases when this is not possible
due to assembly being needed for boot, the required support is being
built into IPS directly; such cases are actually rare].
Some concrete examples of how such "self-assembly" can be realized may
be worthwhile:
The recent hostid project moved the storage of the hostid information
on i386 architecture machines from the sysinit kernel module (which
was bpatch'd during install) into the /etc/hostid file. The initial
design for handling upgrade added code to both BFU and the installer,
which decoded the hostid from the sysinit module and created the
hostid file during upgrade. Freshly installed systems had no hostid
file, so the kernel generated one dyanmically and a SMF service
created the /etc/hostid file after boot. Since neither bfu nor
install code runs during IPS image-update operations, this strategy
did not work on OpenSolaris and the hostid was not preserved on
upgrade.
To fix this, the kernel was modified to search for a sysinit module
and read the hostid from that if /etc/hostid didn't already exist.
Only if that was not present did the kernel generate a new hostid. As
a result of this change, the upgrade code could be eliminated from
both BFU and upgrade, and the system upgrade process just works the
same in all cases.
The key design pattern here is to have the consumer of the
configuration file (here, genunix) handle the old file format/location
if the new file doeesn't exist; writing out the new file resolves the
upgrade process and completes the self-assembly.
Another common problem is the configuration file management problem.
A canonical example might be /etc/security/exec_attr. Here many
different packages contribute lines into this file. With SVR4
packages, a package's file fragments are merged in by the class action
script i.rbac during installation or upgrade. Removal is problematic
on uninstall and is not attempted.
For IPS, the proposed solution is to have each package deliver its
portion of this file into a separate directory, using the name of the
package (suitably escaped, of course) as the file name. A service
that runs at boot determines whether or not the file needs to be
re-composited from the file fragments. This cleanly handles both
install and uninstall.
The key design pattern here is to have packages deliver their
components as files, which is something the packaging system is
designed to do. Assembly of the composited file is left to boot time
and an SMF service, which deals with older package fragments, etc.
Another approach to the configuration file problem is to change the
format of the file. A classic problem is the management of
/etc/driver_aliases, which is "maintained" by add_drv and update_drv.
This file maintains a mapping between a PCI id and the driver that
supports that device. Note that rather than maintaining a text file
which would be read in by the kernel at boot, an alternate approach
would be to encode the same information as a directory of symbolic
links from the PCI id to the name of the driver. This would eliminate
the potential confusion of two different drivers trying to own the
same PCI id, the need for locking the file during updates, and the
need to run programs on install and uninstall to manage this file
since the links would be simply installed and uninstalled as part of
the normal packaging operations.
The design pattern here is also simple - the filesystem already
maintains a single namespace, locking, conflict detection, etc;
reusing these attributes saves time and effort.
Another common problem is moving files, links and directories around
the filesystem and between packages; this is often handled via
scripting in the case of SVR4 packages. In IPS, normal (not editable)
files can simply move between packages; this results in their removal
from the original location and then their re-installation in their new
location. To insure that editable files retain their customizations,
the receiving package must annotate the actions that deliver the new
file name/location with an attribute that defines the original owning
packages and location. This looks like this:
original_name=SUNWfoo:etc/foo.txt
indicating that the name of original owning package was SUNWfoo and
that the original path was etc/foo.txt. Note that no matter where
this file moves to subsequently, it should always maintain this
identifier.
Directories in IPS are automatically reference counted, and are
deleted when there are no longer any explicit declarations of
those directories or no files under packaging system control in
the directory. Left-over files & directories not under packaging
control are moved to a lost-and-found directory under var/pkg;
this is needed to support changing directories to symbolic links,
etc., w/o scripting.
There is not yet a simple method to move unpackaged files, etc
from an old location to a new one; developers facing this problem
should move those files as part of their self-assembly at first
boot and avoid removing the references to the original directories
to prevent their movement to lost-and-found during upgrade.
We are considering various solutions to this problem.
In the short term, those developers working on OpenSolaris have a
foot in both worlds - since OpenSolaris is built from Nevada SVR4
packages, their code needs to work in both environments. With
the exception of driver post-install scripts (which are converted
by the folks working on OpenSolaris to IPS driver actions) the goal
is to remove all post-install/class action processing other than
SMF service restarting on live systems using the patterns shown
here.

74
doc/pkg5_docs/search.txt Normal file
View file

@ -0,0 +1,74 @@
pkg
SEARCH
1. Goals
i. Provide relevant information
ii. Provide a consistently fast response
iii. Make responses consistent between local and remote search
iv. Provide the user with a good interface to the information
v. Allow seamless recovery when search fails
vi. Ensure the index is (almost) always in a consistent state
2. Approach
From a high level, there are two components to search: the
indexer, which maintains the information needed for search; the
query engine, which actually performs a search of the information
provided. The indexer is responsible for creating and updating the
indexes and ensuring they're always in a consistent state. It does this
by maintaining a set of inverted indexes as text files (details of which
can be found in the comments at the top of indexer.py). On the server
side, it's hooked into the publishing code so that the index is updated
each time a package is published. If indexing is already happening when
packages are published, they're queued and another update to the indexes
happens once the current run is finished. On the client side, it's
hooked into the install, image-update, and uninstall code so that each
of those actions are reflected in the index.
The query engine is responsible for processing the text from the user,
searching for that token in its information, and giving the client code
the information needed for a reasonable response to the user. It must
ensure that the information it uses is in a consistent state. On the
server, an engine is created during the server initialization. It reads
in the files it needs and stores the data internally. When the server gets
a search request from a client, it hands the search token to the query
engine. The query engine ensures that it has the most recent information
(locking and rereading the files from disk if necessary) and then searches
for the token in its dictionaries. On the client, the process is the same
except that the indexes are read from disk each time instead of being stored
because a new instance of pkg is started for each search.
3. Details
Search reserves the $ROOT/index directory for its use on both the client
and the server. It also creates a TMP directory inside index which it stores
indexes in until it's ready to migrate them to the the proper directory.
indexer.py contains detailed information about the files used to store the
index and their formats.
3.1 Locking
The indexes use a version locking protocol. The requirements for the
protocol are:
the writer never blocks on readers
any number of readers are allowed
readers must always have consistent data regardless the
writer's actions
To implement these features, several conventions must be observed. The
writer is responsible for updating these files in another location,
then moving them on top of existing files so that from a reader's
perspective, file updates are always atomic. Each file in the index has
a version in the first line. The writer is responsible for ensuring that
each time it updates the index, the files all have the same version
number and that version number has not been previously used. The writer
is not responsible for moving multiple files atomically, but it should
make an effort to have files in $ROOT/index be out of sync for as short
a time as is possible.
The readers are responsible for ensuring that the files their reading
the indexes from are a consistent set (have identical version
numbers). consistent_open in search_storage takes care of this
functionality.

252
doc/pkg5_docs/server_api_versions.txt Normal file
View file

@ -0,0 +1,252 @@
Version 12:
Incompatible with clients using versions 0-11.
pkg.server.api changed as follows:
* New constants 'PKG_STATE_RENAMED' and 'PKG_STATE_OBSOLETE' were
added to assist in determining package state information returned
from functions to the CatalogInterface class.
* The fmris() function of CatalogInterface now has an 'ordered'
parameter to control whether FMRIs are returned in ascending name,
descending version order.
* A new function named 'gen_allowed_packages' was added to
CatalogInterface to determine a list of packages allowed based on
the FMRI of a package that incorporates other packages.
* A new function named 'gen_packages' was added to
CatalogInterface as a replacement for the removed functions
get_matching_version_fmris() and get_matching_pattern_fmris(). It
provides superior filtering capabilities and includes package state
and attribute information.
Version 11:
Incompatible with clients using versions 0-10.
pkg.server.api changed as follows:
* A new property named 'version' was added to the CatalogInterface
class to expose the version format of the catalog.
* The 'rename_requests' property was removed from the ConfigInterface
class as the stat it represented was for functionality that was
never implemented.
* A new property named 'publisher' that returns the Publisher
object for the publisher related to the request was added to the
RequestInterface class.
Version 10:
Incompatible with clients using versions 0-9.
pkg.server.api changed as follows:
* The PackageInfo.PREF_PUBLISHER property and data was removed.
* The CatalogInterface.INFO_MULTI_MATCH property was removed and
the info() method no longer detects and considers multiple
matches for a single pattern an error.
Version 9:
Incompatible with clients using versions 0-8.
pkg.server.api changed as follows:
* All feed related properties retrieved or set using the
(get|set)_repo_* methods are no longer valid. Instead,
all feed-related configuration information must be
retrieved using the new depot configuration methods
found in the ConfigInterface class.
Version 8:
Incompatible with clients using versions 0-7.
pkg.server.api changed as follows:
* The return type of CatalogInterface.search changed so that instead
of returning the fmri string in the result, it returns a PkgFmri
object.
Version 7:
Compatible with clients using version 6.
CatalogInterface has changed as follows:
* A new function named 'info' was added to provide a way to
retrieved package information. It returns a PackageInfo
object representing the set of available package information.
See pydoc 'pkg.client.api.PackageInfo' for more information.
* A new function named 'get_entry_all_variants' was added to
provide access to the list of variants for a given package.
pkg.server.api changed as follows:
* New classes representing package metadata and license
information were added to pkg.api_common.
pkg.server.api_errors changed as follows:
* A new exception named 'UnrecognizedOptionsToInfo' was added.
Version 6:
Incompatible with clients using versions 0-5.
CatalogInterface:
* get_matching_pattern_fmris() and get_matching_version_fmris()
now return a tuple of (fmris, unmatched). Where 'fmris' is
a list of matching FMRIs, and 'unmatched' is a dict of
unmatched patterns or versions indexed by match criteria.
* package_count now returns the number of unique packages in the
catalog instead of the unique number of package versions.
* package_version_count, a new property, was added that contains
the number of unique package versions in the catalog.
ConfigInterface:
* get_repo_attrs was renamed to get_repo_properties
* get_repo_attr_value was renamed to get_repo_property_value
Version 5:
Compatible with clients using Versions 3-4.
ConfigInterface.get_repo_attr_value() has changed as follows:
* Section 'feed' attribute 'authority' has been removed. It has been
replaced by section 'publisher' attribute 'prefix'.
* Section 'publisher' with attributes 'alias' and 'prefix' was added.
* New attributes were added for section 'repository': 'collection_type',
'legal_uris', 'mirrors', 'origins', 'refresh_seconds',
'registration_uri', and 'related_uris'. See the pydoc for
pkg.server.api.ConfigInterface for details.
Version 4:
Compatible with clients using Version 3.
Changes:
CatalogInterface.search() has changed as follows:
* A docstring has been added; see pydoc pkg.server.api for details.
* Added optional keyword 'matching_version' that allows consumers to
filter search results based on version.
* Added optional, boolean keyword 'return_latest' that causes only the
the newest versions of packages to be returned when 'return_type'
is Query.RETURN_PACKAGES.
Version 3:
Incompatible with clients using Versions 0-2.
Changes:
CatalogInterface.search() has changed as follows:
* Added optional, boolean keyword 'case_sensitive'. This indicates
whether a case-sensitive search should be performed.
* Added optional keyword argument 'return_type'. This determines
whether results should be returned as Query.RETURN_ACTIONS or
Query.RETURN_PACKAGES.
* Added optional, integer keyword argument 'start_point'. This
specifies how many matching results should be skipped before
returning anything.
* Added optional, integer keyword argument 'num_to_return'. This
indicates how many results should be returned before the search
is aborted.
* search_done() was removed. Previously, after calling search(), api
consumers would have to call search_done(). This is no longer
necessary.
Version 2:
Incompatible with clients using Versions 0-1.
Changes:
CatalogInterface.get_matching_version_fmris() no longer accepts the constraint
parameter. However, as before, it expects a list of version strings for the
'versions' parameter. These version strings may now contain the wildcard
characters '*' and '?'.
Version 1:
Incompatible with clients using Version 0.
Changes:
CatalogInterface.search_done() was added to perform cleanup after all results
have been retrieved from search().
CatalogInterface.search() now returns a generator object, instead of a list
object, that requires that CatalogInterface.search_done() is called after all
of the desired results have been retrieved for proper cleanup.
Version 0:
Initial api version, containing the following:
class BaseInterface
-- used to instantiate other interface objects
class CatalogInterface
def __init__(self, version_id, base):
def fmris(self):
def get_matching_pattern_fmris(self, patterns):
def get_matching_version_fmris(self, versions,
constraint=pkg.version.CONSTRAINT_AUTO):
@property
last_modified
@property
package_count
def search(self, token):
@property
search_available
class ConfigInterface
def __init__(self, version_id, base):
@property
catalog_requests
@property
content_root
@property
file_requests
@property
filelist_requests
@property
filelist_file_requests
@property
in_flight_transactions
@property
manifest_requests
@property
mirror
@property
readonly
@property
rename_requests
@property
web_root
def get_repo_attrs(self):
def get_repo_attr_value(self, section, attr):
class RequestInterface
def __init__(self, version_id, base):
def get_accepted_languages(self):
def get_rel_path(self, uri):
def log(self, msg):
@property
params
@property
path_info
def url(self, path='', qs='', script_name=None, relative=None):

239
doc/pkg5_docs/signed_manifests.txt Normal file
View file

@ -0,0 +1,239 @@
Manifest signing
----------------
Manifests in IPS contain all the packaging metadata - file
permissions, ownership, content hashes, etc, and are stored and
transmitted as a simple text file with one line per action. During
download or when a system is checked for compliance, the manifest
contents are compared to the files to determine whether or not a
package is correctly received/installed. Given their importance,
verifying that all manifests are correct and reflect the original
publisher's intent is an important part of system validation.
Cryptographic signatures protecting the integrity of all actions form
a Merkle hash tree that includes the delivered binaries such that
complete verification of the installed software is possible. There
are other uses for manifest signing beyond validation; signatures can
also be used to indicate approval by other organizations or parties.
For example, the internal QA organization could sign manifests of
Sun-delivered packages once they had be determined to be qualified for
production use; policy settings could mandate such approvals prior to
installation.
As a result, it is a useful characteristic for signatures to be
independent of other signatures in a manifest; it should be possible
to add (or remove) signatures (but not other actions) in a manifest
without invalidating the other signatures that are present. This
feature also facilitates production handoffs, with signatures used
along the path to indicate completion along the way; subsequent steps
can optionally remove previous signatures at any time without ill-effect.
The need to treat signatures differently during hash computation
suggests that the signature itself should be easily distinguished from
other sorts of package metadata; this leads us to a new signature
action, of the form:
signature <hash of certificate> algorithm=<signature algorithm> \
value=<signature value> \
chain="<hashes of certs needed to validate primary certificate>" \
version=<pkg version of signature>
The payload and pkg.chain_certs attributes represent the packaging hash of the
pem file(s) containing the x.509 certificate(s) downloadable from the
originating repository; the value is the signed hash of the manifest's message
text, prepared as discussed below. The payload certificate is the certificate
which verifies the value in pkg.sigval. The other certificates presented need
to form a certificate path that leads from the payload certificate to the trust
anchor(s) that was established as part of the publisher configuration.
To compute the signature of a manifest, first a canonical representation of the
manifest is created. (See "Computing the manifest's message text" below.) The
message text is then given to the signature algorithm along with the private key
and the result is the value of the signature.
Two types of signature algorithms are currently supported. The first is rsa
group of signature algorithms. An example is "rsa-sha256". The bit after the
dash specifies the hash algorithm to use to change the message text into a
single value the rsa algorithm can use.
The second type of signature algorithm is compute the hash only. This type of
algorithm exists primarily for testing and process verification purposes and
presents the hash as the signature value. A signature action of this type is
indicated by the lack of a payload certificate hash. This type of signature
action is verified if the image is configured to check signatures. Its
presence however does not count as a signature if signatures are required.
signature algorithm=<hash algorithm> value=<hash> \
version=<pkg version of signature>
Additional signature types (pgp, for example) may be added in the future.
Additional metadata can be added to a signature if desired, as with any
other action.
Policies may be set for the image or for specific publishers. The policies
include ignoring signatures, verifying existing signatures, requiring
signatures, and requiring that specific common names must be seen in the chain
of trust. Other policies may be added in the future.
Computing the manifest's message text:
--------------------------------------
Manifests have an interesting property: the lines in a manifest may be
reordered without affecting the meaning of the manifest. As a result,
manifest order is not preserved and subject to change during package
publication processing. It is thus necessary for our manifest signing
to be independent of presented line order, or the action ordering
algorithm used for installation, as that may change over time.
Straightforward C-locale alphabetical sorting of attributes within
actions, the multiple values within those attributes, and across actions
can be used to enforce a consistent ordering for signature purposes and
is not subject to change for a given manifest.
In order to allow other signatures to be added or removed from a
manifest, computation of the manifest message text does not include
other signatures; in order to protect metadata on the signature
itself, the signature being produced or verified is included in the
message text at the end, aside of course from the actual signature
value itself.
For example, take the following manifest:
set name=fmri value=foo@1.0
dir path=foo/bar group=sys
signature cert1 algorithm=rsa-sha256 value=val1 random_attr=baz
signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee
The text used to compute val1 is:
dir group=sys path=foo/bar
set name=fmri value=foo@1.0
signature cert1 algorithm=rsa-sha256 value= random_attr=baz
The text used to compute val2 is:
dir group=sys path=foo/bar
set name=fmri value=foo@1.0
signature cert2 another_attr=whee algorithm=rsa-sha256 value=
Including the text of the signature action itself prevents the signature action
from being modified. Ensuring that the text used for compute val1 contains no
mention of the second signature (or any other signature) allows signatures to be
added and removed freely.
Verification of merged signatures:
----------------------------------
We produce "fat" packages (containing variants such as different
architectures, debug vs non-debug kernel, etc) by producing manifests
for each variant, and then merging them. Actions that are the same
between variants being merge are left unmodified; those that are
different receive a variant tag (see facets.txt). If it is considered
useful to have signatures persist and be useful across such merges,
some additional steps are required to verify such signatures after
merging.
Generally, signatures will be unique to their variant, thus they will
be tagged with variant tags after merging. To verify signatures
post-merge, the evaluation process has three steps after other signature actions
have been removed from consideration.
1) Any variant tags present on the signature are assumed to have been added
after the merge. Thus, all actions whose variants do not match the signature's
variants are removed from inclusion in the message text.
2) Since the variant tags were not present when the manifest was signed, they
need to be removed from the attributes of each of the remaining actions as
well. This includes removing the set attributes which define the variants for
the package.
3) Restore any set actions in the existing_pkg_vars attribute which
describe variants which have been removed from the text.
For example, consider the following manifest:
set name=fmri value=foo@1.0
set name=variant.arch value=sparc value=i386
set name=variant.debug value=true value=false
dir path=foo1 group=sys
dir path=foo2 variant.arch=sparc
dir path=foo2/d variant.arch=sparc variant.debug=true
dir path=foo2/nd variant.arch=sparc variant.debug=false
dir path=foo3 variant.arch=i386
signature cert1 algorithm=rsa-sha256 value=val1 random_attr=baz \
variant.arch=sparc variant.debug=true
signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee \
variant.arch=i386 existing_pkg_vars="variant.arch=i386"
To compute the message text for the first signature, the first step is applied
producing the following text:
set name=fmri value=foo@1.0
set name=variant.arch value=sparc value=i386
set name=variant.debug value=true value=false
dir path=foo1 group=sys
dir path=foo2 variant.arch=sparc
dir path=foo2/d variant.arch=sparc variant.debug=true
signature cert1 algorithm=rsa-sha256 value=val1 random_attr=baz \
variant.arch=sparc variant.debug=true
After the second step is applied, the text becomes:
set name=fmri value=foo@1.0
dir path=foo1 group=sys
dir path=foo2
dir path=foo2/d
signature cert1 algorithm=rsa-sha256 value=val1 random_attr=baz
Since the third step doesn't apply to this signature, the above text becomes the
canonical message text.
Computing the message text for the second signature proceeds like this:
After the first step the text is:
set name=fmri value=foo@1.0
set name=variant.arch value=sparc value=i386
set name=variant.debug value=true value=false
dir path=foo1 group=sys
dir path=foo3 variant.arch=i386
signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee \
variant.arch=i386 existing_pkg_vars="variant.arch=i386"
After the second step, the text is:
set name=fmri value=foo@1.0
dir path=foo1 group=sys
dir path=foo3
signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee \
existing_pkg_vars="variant.arch=i386"
In the third step, we restore the set action which was present in the original
manifest. The text becomes:
set name=fmri value=foo@1.0
set name=variant.arch value=i386
dir path=foo1 group=sys
dir path=foo3
signature cert2 algorithm=rsa-sha256 value=val2 another_attr=whee \
existing_pkg_vars="variant.arch=i386"
The existing_pkg_vars attribute allows whether a package variant set
action was present or not at the time of signing to be determined
deterministically.
Publication of signed manifests:
--------------------------------
Publishing a signed manifest is a two step process. First the package is
published, unsigned, to a repository. The package is then updated in place,
using pkgsign, appending a signature action to the manifest in the repository
but leaving the package, including its timestamp, intact. This process allows a
signature action to be added by someone other than the publisher without
invalidating the publisher's signature. For example, the QA department of a
company may want to sign all packages that are installed internally to indicate
they have been approved for use, but not republish the packages which would
create a new timestamp and invalidate the signature of the original publisher.
The disadvantage of this approach is that a fmri no longer represents a single
manifest eternally. Eventually, this problem is solved by having the client
ensure that the hash of the manifest it loads for the fmri matches the hash in
the catalog. Until that feature is implemented, it is imperative that
publishers ensure that no client has access to an unsigned manifest which they
plan to sign in the future, or to return to the QA example, that no client
inside the organization sees a manifest for a fmri without a signature which the
QA plans to sign in the future.
pkgsign is able to update a package in place through the use of a new publishing
operation called append. This operation opens a transaction to modify the
manifest of a fmri which is already in the repository. When the transaction is
closed, the signed manifest replaces the existing manifest and the catalog is
updated to reflect the new hash of the manifest.

106
doc/pkg5_docs/system_repository.txt Normal file
View file

@ -0,0 +1,106 @@
System Repository and Publishers
Introduction:
Linked images, and zones in particular, must keep certain packages
in sync with the global zone in order to be functional. The global zone will
constrain packages within the non-global zones and configure special publishers
in the non-global zone (NGZ). These publishers (henceforth called system
publishers) are special because the non-global zone cannot make certain kinds of
modifications to them. Among the forbidden operations for the non-global zone on
the system publishers are deleting, disabling, removing or replacing origins
provided by the system repository, and any other operations which might prevent
the solver from meeting the constraints imposed by the constraint package. The
global zone must provide the means for the non-global zone to configure itself
with system publishers by providing information like origins. The global zone
also has to provide a connection to the system publishers' repositories which is
available even in a scratch zone.
The Data path:
The pkg client in the NGZ uses the system repository in the global zone as a
proxy to the system publishers. To ensure that a communication path between the
pkg client in the NGZ and the system repository in the global zone always
exists, the zone proxy client and the zone proxy daemon were created.
The zone proxy client runs in the NGZ. When started, it creates a socket which
listens on an inet port on 127.0.0.1 in the NGZ. It passes the file descriptor
for this socket to the zone proxy daemon in the global zone via a door call. The
zone proxy daemon listens for connections on the file descriptor. When zone
proxy daemon receives a connection, it proxies the connection to the system
depot. The system depot is an Apache instance running in the global zone which
provides connectivity to and configuration of publishers.
The system depot acts as a proxy for the http and https repositories for
the publishers it provides. When proxying to https repositories, it uses the
keys and certificates in the global zone to identify itself and verify the
server's identity. It also provides a http interface to the file repositories
for the publishers it provides as well as serving publisher and image
configuration via the syspub/0 response.
Configuration:
The syspub/0 response is a p5s file. The p5s file contains publisher
configuration and image configuration. Currently, the only image configuration
it contains is the publisher search order for the provided publishers, but other
information may be added to the response as needed. In addition to the basic
collection of publisher information, the p5s file also contains a list of URIs
which the pkg client should proxy to via the system depot instead of contacting
them directly. When creating a p5s file, the URI for origins and mirrors can
be transformed. HTTPS URIs are transformed to HTTP URIs since the system depot
will be doing the SSL communication, not the pkg client. File URIs are
transformed into HTTP URIs with a special format. The URIs contain the special
token "<sysrepo>" which the p5s parser knows to replace with the URI of the zone
proxy client. The rest of the URI contains the prefix of the publisher, then
the sha1 hash of the global zone path to the file repository.
The information for the syspub/0 response comes from the global zone's image's
configuration. The application/pkg/system-repository service is responsible for
transforming the image configuration into an Apache configuration file and
causing the system depot to reread its configuration. The global zone
pkg client restarts the application/pkg/system-repository service whenever the
image's publisher configuration changes.
The Apache configuration file, written by sysrepo.py using the Mako template
sysrepo_httpd.conf.mako does two things:
* enables an Apache proxy accepting only requests to the configured
publisher http URIs (and utilizes a file-based proxy cache) only listening
on the loopback interface by default.
* adds a series of mod_rewrite RewriteRule and Alias directives to allow
the Apache instance to gain access to configured file:// publishers.
The Apache URIs are accessible from the system repository configured at
'<sysrepo>' for a given publisher '<publisher>' are below.
We serve static responses to the following URIs, the content being either
a known versions file, or the p5s file mentioned above:
http://<sysrepo>/versions/0/
http://<sysrepo>/syspub/0
For access to file:// repositories, we use rewrite rules and Alias directives to
access the file repository contents. In order to allow repositories with
different paths, yet prevent exposing those paths to system repository clients,
we use an SHA-1 hash of the path, and include that in the URI (below, this hash
is represented by "HASH") The URIs we accept for file:// repositories are:
http://<sysrepo>/<publisher>/HASH/file/1/<file hash>
http://<sysrepo>/<publisher>/HASH/manifest/0/<package name@version>
http://<sysrepo>/<publisher>/HASH/publisher/0
http://<sysrepo>/<publisher>/HASH/versions/0
The system publisher also responds to 'OPTIONS * HTTP/1.0' requests.
When we detect that a request to a given <publisher>/HASH refers to a GZ
publisher within a .p5p archive, that request is rewritten so that a custom WSGI
application, sysrepo_p5p.py handles the request. That application accepts URIs
of the form:
http://<sysrepo>/wsgi_p5p?pub=<publisher>&hash=HASH&path=<path>
where path is the remainder of the original system repository file:// URI, after
the <publisher> and HASH components have been removed.

307
doc/pkg5_docs/tags-and-attributes.txt Normal file
View file

@ -0,0 +1,307 @@
PSARC/2008/190
pkg(5): image packaging system
TAGS AND ATTRIBUTES
1. Definitions
Both packages and actions within a package can carry metadata, which
we informally refer to as attributes and tags. Both attributes and
tags have a name and one or more values.
Attributes: settings that apply to an entire package. Introduction
of an attribute that causes different deliveries by the client could
cause a conflict with the versioning algebra, so we attempt to avoid
them.
Tags: settings that affect individual files within a package.
2. Attribute and tag syntax and namespace
2.1. Syntax
2.1.1 Naming
The syntax for attributes and tags is similar to that used for
pkg(5) and smf(5) FMRIs.
[org_prefix,][name][:locale]
The organizational prefix is a forward-ordered or reverse-ordered
domain name, followed by a comma. The name field is an identifier
which may have a prefix ending in a period to allocate the namespace.
If the locale field is omitted, the default locale is "C", a 7-bit
ASCII locale.
Each of these fields is [A-Za-z][A-Za-z0-9_-.]*.
2.1.2 Manifests
In package manifests, the syntax for an attribute is:
set name=<attribute name> value=<value> [value=<value2> ...]
In package manifests, tags are included in the action line
for the action they apply to:
<action> [...] <tag name>=<tag value> [<tag name>=<tag value2> ...]
2.2. Unprefixed attributes and tags.
All unprefixed attributes and tags are reserved for use by the
framework.
Generally, unprefixed tags are introduced in the definition of an
action.
2.3. Attributes and tags common to all packages
Attributes and tags starting with "pkg." or "info." are for attributes
common to all packages, regardless of which particular OS platforms that
a specific package may target. "pkg" attributes are used by the
packaging system itself, while "info" attributes are purely informational,
possibly for use by other software.
2.3.1. Common attributes
pkg.summary
A short, descriptive name for the package. In accordance with
2.1.1 above, pkg.summary:fr would be the descriptive name in French.
Exact numerical version strings are discouraged in the
descriptive name.
Example: "Apache HTTPD Web Server 2.x"
pkg.description
A descriptive paragraph for the package. Exact numerical version
strings can be embedded in the paragraph.
pkg.detailed-url
One or more URLs to pages or sites with further information about
the package. pkg.detailed-url:fr would be the URL to a page with
information in French.
pkg.renamed
A value of "true" indicates the package has been renamed or split
into the packages listed in the depend actions.
pkg.obsolete
A value of "true" indicates the package is obsolete and should be
removed on upgrade.
pkg.human-version
For components whose upstream version isn't a dot-separated sequence
of nonnegative integers (OpenSSL's 0.9.8r, for example), this
attribute can be set to that string, and will be displayed when
appropriate. It cannot be used in an FMRI to install a particular
version; package authors must still convert the version into a
sequence of integers.
variant.*
See facets.txt
2.3.2. Common tags
disable_fmri
See "Actuators" section of pkg(5)
facet.*
See facets.txt
reboot-needed
See "Actuators" section of pkg(5)
refresh_fmri
See "Actuators" section of pkg(5)
restart_fmri
See "Actuators" section of pkg(5)
suspend_fmri
See "Actuators" section of pkg(5)
variant.*
See facets.txt
2.3.3 Informational attributes
The following attributes are not necessary for correct package installation,
but having a shared convention lowers confusion between publishers and
users.
info.classification
A list of labels classifying the package into the categories
shared among pkg(5) graphical clients.
Values currently used for OpenSolaris are prefixed with
"org.opensolaris.category.2008:" and must match one of the
categories listed in src/gui/data/opensolaris.org.sections
info.keyword
A list of additional terms that should cause this package to be
returned by a search.
info.maintainer
A human readable string describing the entity providing the
package. For an individual, this string is expected to be their
name, or name and email.
info.maintainer-url
A URL associated with the entity providing the package.
info.upstream
A human readable string describing the entity that creates the
software. For an individual, this string is expected to be
their name, or name and email.
info.upstream-url
A URL associated with the entity that creates the
software delivered within the package.
info.source-url
A URL to the source code bundle, if appropriate, for the package.
info.repository-url
A URL to the source code repository, if appropriate, for the
package.
info.repository-changeset
A changeset ID for the version of the source code contained in
info.repository-url.
2.4. Attributes & tags common to all packages for an OS platform
Each OS platform is expected to define a string representing that
platform. For example, the OpenSolaris platform is represented by
the string "opensolaris".
2.4.1. OpenSolaris attributes
org.opensolaris.arc-caseid
One or more case identifiers (e.g., PSARC/2008/190) associated with
the ARC case or cases associated with the component(s) delivered by
the package.
org.opensolaris.smf.fmri
One or more FMRI's representing SMF services delivered by this
package. Automatically generated by pkgdepend(1) for packages
containing SMF service manifests.
opensolaris.zone
Obsolete - replaced by variant.opensolaris.zone.
variant.opensolaris.zone
See facets.txt
2.4.1. OpenSolaris tags
opensolaris.zone
Obsolete - replaced by variant.opensolaris.zone.
variant.opensolaris.zone
See facets.txt
2.5. Organization specific attributes
Organizations wishing to provide a package with additional metadata
or to amend an existing package's metadata (in a repository that
they have control over) must use an organization-specific prefix.
For example, a service organization might introduce
"service.example.com,support-level" or
"com.example.service,support-level" to describe a level of support
for a package and its contents.
2.5.1 Sun/Oracle attributes
These are listed simply to record the currently used attributes in
the OpenSolaris /support repository and are subject to change as
Sun integrates into Oracle.
com.sun.service.incorporated-changes
A space-separated list of Change Requests ids in the Sun bugtraq
database for the changes incorporated in this revision of the
package that were not in the previous revision on the same branch.
com.sun.service.keywords
Keywords for the type of change included, such as "security"
for security fixes.
2.6 Attributes specific to certain types of actions
Each type of action also has specific attributes covered in the
documentation of those actions. These are generally documented
in the section of the pkg(5) manual page for that action.
2.7 Attributes specific to certain types of file
These would generally appear on file actions for files in a specific
format.
elfarch, elfbits, elfhash
Data about ELF format binary files (may be renamed in the future
to info.file.elf.*). Automatically generated during package
publication. See the "File Actions" section of pkg(5).
info.file.font.name
The name of a font contained in a given file. There may be multiple
values per file for formats which collect multiple typefaces into a
single file, such as .ttc (TrueType Collections), or for font aliases.
May also be provided in localized variants, such as a Chinese font
providing both info.file.font.name:en and info.file.font.name:zh for
the English and Chinese names for the font.
info.file.font.xlfd
An X Logical Font Description (XLFD) for a font contained in a
given file. Should match an XLFD listed in fonts.dir or fonts.alias
for the file. There may be multiple values per file due to font
aliases.
3.3. Attributes best avoided
built-on release
One problem we may run into is packages that have been built on a
release newer than that on the image. These packages should be
evaluated as unsuitable for the image, and not offered in the graph.
There are a few ways to handle this case:
1. Separate repository. All packages offered by a repository were
built on a known system configuration. This change requires
negotiation between client and server for a built-on match
condition. It also means that multiple repositories are needed
for a long lifecycle distribution.
2. Attributes. Each package comes with a built-on attribute. This
means that clients move from one built-on release to another
triggered by conditions set by the base package on the client.
Another drawback is that it becomes impossible to request a
specific package by an FMRI, without additional knowledge.
3. Additional version specifier. We could extend
release,branch:timestamp to release,built,branch:timestamp--or
fold the built and branch version together. Since the built
portion must reserve major.minor.micro, that means we move to a
package FMRI that looks like
coreutils@6.7,5.11.0.1:timestamp
This choice looks awkward. We could instead treat the built
portion as having a default value on a particular client. Then
the common specifier would be
name@release[,build]-branch:timestamp
build would be the highest available valid release for the
image.
The meaning of the built-on version could be controversial. A
simple approach would be to base it on base/minimal's release,
rather than uname(1) output.

17
doc/pkg5_docs/transaction-order.txt Normal file
View file

@ -0,0 +1,17 @@
pkg
PACKAGE TRANSACTION ORDER
- [zfs snapshot]
- pull bits to same filesystem
- test transaction
- begin operations
- cp old to SHA-old
- mv SHA-new to new
- perform actions
- complete
- when are old versions discarded
- when is snapshot discarded
- snapshot namespace for pkg(1M)

85
doc/pkg5_docs/usr-setuid-bins.txt Normal file
View file

@ -0,0 +1,85 @@
/usr/bin/atrm
/usr/bin/crontab
/usr/bin/fdformat
/usr/bin/pfexec
/usr/bin/newgrp
/usr/bin/passwd
/usr/bin/amd64/newtask
/usr/bin/amd64/uptime
/usr/bin/amd64/w
/usr/bin/rcp
/usr/bin/rdist
/usr/bin/rlogin
/usr/bin/rsh
/usr/bin/rmformat
/usr/bin/chkey
/usr/bin/ct
/usr/bin/mail
/usr/bin/mailx
/usr/bin/cu
/usr/bin/uucp
/usr/bin/uuglist
/usr/bin/uuname
/usr/bin/uustat
/usr/bin/uux
/usr/bin/cdrw
/usr/bin/pppd
/usr/bin/mailq
/usr/bin/lpset
/usr/bin/su
/usr/bin/login
/usr/bin/tip
/usr/bin/write
/usr/bin/at
/usr/bin/atq
/usr/bin/i86/newtask
/usr/bin/i86/uptime
/usr/bin/i86/w
/usr/lib/sendmail
/usr/lib/utmp_update
/usr/lib/uucp/remote.unknown
/usr/lib/uucp/uucico
/usr/lib/uucp/uusched
/usr/lib/uucp/uuxqt
/usr/lib/lp/bin/netpr
/usr/lib/fs/ufs/quota
/usr/lib/fs/ufs/ufsdump
/usr/lib/fs/ufs/ufsrestore
/usr/lib/cacao/lib/tools/cacaocsc
/usr/lib/webconsole/adminverifier
/usr/lib/webconsole/pamverifier
/usr/lib/ssh/ssh-keysign
/usr/lib/print/lpd-port
/usr/lib/acct/accton
/usr/lib/gnome-suspend
/usr/openwin/bin/lbxproxy
/usr/openwin/bin/xlock
/usr/openwin/bin/Xsun
/usr/openwin/bin/sys-suspend
/usr/openwin/bin/Xprt
/usr/sbin/allocate
/usr/sbin/eeprom
/usr/sbin/prtdiag
/usr/sbin/sacadm
/usr/sbin/traceroute
/usr/sbin/wall
/usr/sbin/deallocate
/usr/sbin/list_devices
/usr/sbin/ping
/usr/sbin/pmconfig
/usr/sbin/smpatch
/usr/sbin/i86/prtconf
/usr/sbin/i86/swap
/usr/sbin/i86/sysdef
/usr/sbin/i86/whodo
/usr/sbin/amd64/prtconf
/usr/sbin/amd64/swap
/usr/sbin/amd64/sysdef
/usr/sbin/amd64/whodo
/usr/SUNWale/bin/mailx
/usr/X11/bin/xscreensaver
/usr/X11/bin/amd64/Xorg
/usr/X11/bin/i386/Xorg
/usr/xpg4/bin/at
/usr/xpg4/bin/crontab
/usr/xpg6/bin/crontab

147
doc/pkg5_docs/versions.txt Normal file
View file

@ -0,0 +1,147 @@
pkg
Expressing versions
A package is a labelled time series of collections of objects.
That is, over time, we expect to see something like
pkg:///sunos/coreutils@5.11,1:[hex-timestamp-1]
|
| on-branch transition
V
pkg:///sunos/coreutils@5.11,1:[hex-timestamp-2] --> additional on-branch transitions
|
| branch upgrade
V
pkg:///sunos/coreutils@5.11,2:[hex-timestamp-3] --> additional branch upgrades
|
| micro release upgrade
V
pkg:///sunos/coreutils@5.11.1,1:[hex-timestamp-4] --> additional micro release upgrades
|
| minor release upgrade
V
pkg:///sunos/coreutils@5.12:[hex-timestamp-5]
In this sequence, we've assumed that the client's requirement to
remain with binaries associated with a particular build is implicit
with the client. As discussed in the tags and attributes note, we
need to handle the distinction between a package built on one release
and the same package's binaries built on another release.
XXX Need a full example with the build_version in place.
Each transition is constrained by the local client's decision to "stay
on branch", "move to a newer branch", "move to a newer release".
Both the release, the required build, and the branch can be
"dot-separated vectors". The timestamp is not. That is, a full
version is
version -> dot_vector(,dot_vector)?-dot_vector:timestamp
dot_vector -> digit+ ( "." digit+ )*
timestamp -> hexdigit+
Rollback is expected to be handled by image management. Rollback is
expected to be made convenient through use of ZFS.
If we had
sunos/coreutils@5.11,1:[hex-timestamp]
and wanted to go to the latest revision on this branch, we would
invoke
pkg get coreutils
which could upgrade other components.
If we wanted to go from 5.11,1.x to 5.11,2 we would invoke
pkg get coreutils@5.11,2
(which might be the result of displaying a cosmetic version string,
like "GNU coreutils 6.8" or something). This operation might cause
other components to be updated.
If we instead did
pkg get coreutils@5.11.1
or
pkg get coreutils@5.12
we would get a release constraint, which should tell us that we need
to request an update to base/minimal@5.11.1. This release constraint
comes from the fact that release ownership is held by a restricted set
of packages.
If coreutils had been frozen by another package, we would get, in
response a message like
pkg: sunos/coreutils frozen at 5.11,1 by site/workstation-cfg@5.11,1.12
The administrator can then pkg delete site/workstation-cfg (or pull
down an updated version lacking the "incorporate coreutils@5.11,1"
statement, with its implied freeze).
pkg delete on groups removes leaf packages in the group (included via
"pkg" statements) but leaves package dependencies untouched.
The "pkg freeze" subcommand can place an administrative freeze on a
specific package. "pkg unfreeze" (pkg thaw?) can remove a freeze,
either an administrative freeze or a freeze placed by another package.
Co-requisite packages
We need to support a@1 <-> b@2. This is handled as two transactions,
so we need to allow unresolved dependencies to exist in the
repository, _but_ the R = (a@1, ...) repository cannot offer a@1 until
it also has b@2. And also G (a@1, b@2) group package cannot be
submitted.
This requirement becomes a hint for our order of operations:
individual package transactions, group (base and stack) transactions.
When to increment the branch number?
On incompatible change to a private interface.
On addition of new private interfaces.
On addition of new public interfaces where the release version is
constrained.
Potentially on the addition of a newly delivered platform or ISA?
Using the (hex) timestamp as the sequence ID
We can source a package, category/pkg, from any repository we choose
to trust. That is, we can do
pkg update pkg://rosseau.sfbay/base/developer
on a system that had a default base FMRI of pkg://opensolaris.org
As long as the two repositories agree on the forward use of the
release and branch version components, then the timestamp approach
allows us to allow a system to move back and forth between multiple
repositories.
On-build release freezes
We may need an expression format that allows us to pin the on-build
portion of the version space, in addition to specifying the release
and/or branch portion of the FMRI as frozen. Otherwise, we could jump
from a binary compiled from one environment to that from a completely
different compilation environment, based on only a timestamp change.

57
doc/pkg5_docs/wos-filetype-stats.txt Normal file
View file

@ -0,0 +1,57 @@
From danek.duvall@sun.com Fri Jun 8 13:37:41 2007
Received: from sfbaymail2sca.sfbay.sun.com (sfbaymail2sca.SFBay.Sun.COM [129.145.155.42])
by zion.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id l58KbeOv016939
for <sch@zion.eng.Sun.COM>; Fri, 8 Jun 2007 13:37:40 -0700 (PDT)
Received: from zruty.sfbay.sun.com (zruty.SFBay.Sun.COM [129.146.168.40])
by sfbaymail2sca.sfbay.sun.com (8.13.6+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id l58Kbd8p023821;
Fri, 8 Jun 2007 13:37:39 -0700 (PDT)
Received: from zruty.sfbay.sun.com (localhost [127.0.0.1])
by zruty.sfbay.sun.com (8.14.0+Sun/8.14.0) with ESMTP id l58Kb7qa007119;
Fri, 8 Jun 2007 13:37:07 -0700 (PDT)
Received: (from dduvall@localhost)
by zruty.sfbay.sun.com (8.14.0+Sun/8.14.0/Submit) id l58Kb7oo007118;
Fri, 8 Jun 2007 13:37:07 -0700 (PDT)
Date: Fri, 8 Jun 2007 13:37:07 -0700
From: Danek Duvall <danek.duvall@sun.com>
To: Stephen Hahn <stephen.hahn@sun.com>
Subject: simple WOS filetype stats
Message-ID: <20070608203707.GI25472@zruty.sfbay.sun.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: mutt-ng/devel-r535 (SunOS)
Status: RO
Content-Length: 625
Lines: 30
264423 package objects (including type "i")
Breakdown by type:
f 193967
d 37374
s 22533
i 7015
l 2399
v 673
e 462
We're publishing files of type "[fevd]", which means we're publishing
232475 files, or 91%. Symlinks and hardlinks are all that's left from that
perspective.
Breakdown by class:
none 256435 (97%)
preserve 245
manifest 167
j.link 134
initd 77
renamenew 45
rbac 33
renameold 10
as well as a bunch of others which are either unimportant or in the noise.
Danek

31
doc/pkg5_docs/xml.txt Normal file
View file

@ -0,0 +1,31 @@
pkg
USE OF XML
It might be helpful, for tools development, to offer an XML description
of package transactions and manifests.
That is, if we're backing into a full fledged package dialect, beyond
something single line-based, then we should consider whether an XML
document is a suitable choice. Usual pro-XML, con-XML position follows.
1. Specific cases in support.
Multiline text properties and localized text properties argues for use
of XML.
Sympathy with <property>, <value>, and <propval> in the smf(5) DTD might
be of benefit.
2. Specific cases against.
Server side assembly of the transaction into a package version means
that the manifest might be difficult to construct. Or maybe not:
open -> <manifest package="...">
add -> <action ...>
close -> </manifest>
(But accept and publish will require that manifest to be imported and
updated.)