IPS Packaging - Part II - OmniOS

Share on:

OmniOS Community Edition is a free and open-source Unix operating system derived from OpenSolaris and based on Illumos. Developers forked OpenSolaris after Oracle Corporation discontinued it, in order to continue development and distribution of the source code. The OmiOSCE project is stewarded by the OmniOS Community Edition Association. OmniOSCE's goal is to produce a self-hosting, minimalist Illumos-based release suitable for production deployment.

The Image Packaging System, also known as IPS, is a cross-platform (written in Python) package management system created by the OpenSolaris community in coordination with Sun Microsystems. IPS is designed to eliminate some long-standing issues with previous software distribution, installation, and maintenance mechanisms that have caused significant problems for Solaris users, developers, and maintainers.

Today I will set up an IPS repository and make it available on the the local network. After this I will package the Apache httpd server and its dependecies, including the SMF Manifest and Method, & also the Apache 2.0 license. This will later be published to the local IPS Repository. The post will cover the various pkg tools, such as pkgdepend, pkglint, pkgmogrify, pkgrepo & pkgsend. These tools allow us to build a package manifest automatically and apply transforms to its contents. Further, pkglint runs a series of checks on the package manifest.

Creating and Running an IPS Repository

First I will create and configure an IPS repository that will host the Apache httpd Server package. I will create a local IPS repository, so that other hosts in my network may take advantage of the IPS packages.

The first thing to do is to create a new directory that can host the repository.

# mkdir /export/ips-repo

Next I create an IPS repository by using the pkgrepo create command. The pkgrepo command is verified by listng the contents of the new repo.

# pkgrepo create /export/ips-repo
# ls /export/ips-repo

After we have created the initial repository, we need to set a few different properties to reflect our environment. I start with defining a publisher, pbdigital, which is an entity (in this case, myself) that develops and constructs packages. A publisher name, or prefix, identifies this source in a unique manner.

We use the pkgrepo set command to set the prefix of the repository:

# pkgrepo set -s /export/ips-repo publisher/prefix=pbdigital

I will be creating a SMF service to control the repository. This is achieved with the svccfg command. Here we set the root directory, the port number the service will be available on and making the repository writable so that I can publish packages to the repository.

# svccfg -s application/pkg/server setprop pkg/inst_root=/export/ips-repo
# svccfg -s application/pkg/server setprop pkg/port=9001
# svccfg -s application/pkg/server setprop pkg/readonly=false

It is just a simple matter of starting the SMF service using the svcadm command:

# svcadm enable application/pkg/server
# svcs application/pkg/server
online        18:50:21  svc:/application/pkg/server:default

If you find that the STATE is in maintenance, it is likely that file ownership is not set correctly. You can check this in the /var/svc/log/application-pkg-server:default.log. If this is the case, set the file ownership as follows:

# chown -R pkg5srv:pkg5srv /export/ips-repo/publisher

You will then need to clear the maintenance state and again start the service.

# svcadm clear /application/pkg/server
# svcadm enable /application/pkg/server

Now that we have a running service, we can use the pkgrepo info command to get some information about the repository:

# pkgrepo info -s http://localhost:9001
pbdigital  0         online        2011-10-10T13:18:51.193453Z

We can see the repository contains no packages currently. Further we can visit the webpage http://localhost:9001 and we will presented with the following page.

IPS Repository

Creating the Apache Portable Runtime Package

I will outline here the various stages of the packaging process. This can be used as a reference later when I package apr-util-1.6.1 httpd-2.4.41.

Create proto Directory Structure

First create a directory to store the package files in. I will call it proto under the directory where I originally downloaded the apr-1.7.0 tarball. I will also create the directory structure that will be used when the package is installed.

mkdir -p proto/opt/pbdigital

Next move over all the files from the installed version of apr-1.7.0

mv /opt/pbdigital/apr-1.7.0 proto/opt/pbdigital/

I need to add the Apache 2.0 license if I am to distribute the software, so I add the license into the proto area.

mkdir -p proto/pbdigital/apr-1.7.1/share/licenses
cp APACHE20 proto/pbdigital/apr-1.7.1/share/licenses/

You can find a copy of the Apache 2.0 license here.

With our files now all in the proto area, we can now move onto packaging the appllication.

Create Manifest of proto Directory

From here I can generate a manifest for this proto directory. The proto directory should be in the current working directory.

1pkgsend generate proto | pkgfmt > apr-1.7.0.p5m.1

{: .language-shell}

This generates a list of all the files in the proto directory, in the file named apr-1.7.0.p5m.1

Add Metadata to the manifest

We need to add metadata to the manifest. We will create a file, apr-1.7.9.mog, and add the metadata to this.

set name=pkg.fmri value=pkg://pbdigital/library/apr@1.7.0,5.11-151032.0
set name=pkg.description value="The Apache Portable Runtime is a library \
of C data structures and routines, forming a system portability \
layer that covers as many operating systems as possible, including \
Unices, Win32, BeOS, and OS/2."
set name=pkg.summary value="The Apache Portable Runtime"
set name=variant.arch value=$(ARCH)
set name=info.classification \
license opt/pbdigital/apr-1.7.0/share/licenses/APACHE20 license="Apache v2.0"

This metadata file sets various declarations.

  • pkg.fmri: this is the name of what our package will be listed as under the IPS repository. The format is as follows:

    • starts with the publisher, category and application name. You can view the types of catogories available by looking at the Omnios IPS Repositories.
    • This is followed by the @ symbol and the package version.
    • Next is the build version, which must be preceded with a comma. The build bersion should match the case under uname, in this case 5.11.
    • Follows is the branch version, which must be preceded with a hyphen. I have chosen the current release of omnios that I am using to package the software. If a branch version is not set, there will later be problems when you pkglint the manifest.
  • pkg.description: A detailed description of the contents and functionality of the package, typically a paragraph or so in length.

  • pkg.summary: A brief synopsis of the description.

  • variant.arch: The arhchitecture of the package. This contains a variable now, but the architecture will be set when we run the next command.

  • info.classification: One or more tokens that a pkg client can use to classify the package. The value should have a scheme (such as org.opensolaris.category.2008 or org.acm.class.1998) and the actual classification, such as Applications/Games. A set of info.classification values is included in Appendix A of the OpenIndiana IPS Packaging Guide.

  • license: license action represents a license or other informational file associated with the package contents. The format is as follows:

    • license file-location license="description"

This metadata can be merged with the manifest with the following command:

1pkgmogrify -DARCH=`uname -p` apr-1.7.0.p5m.1 apr-1.7.0.mog | pkgfmt > apr-1.7.0.p5m.2

{: .language-shell}

Add Dependecies to the Manifest

The pkgdepend command will scan the manifest for the dependencies that are needed to create the package.

1pkgdepend generate -md proto apr-1.7.0.p5m.2 | pkgfmt > apr-1.7.0.p5m.3

{: .language-shell}

Resolve Dependecies in the Manifest

Now we need to resolve these dependencies. The pkgdepend resolve command searches the current boot environment for the dependencies that need to be resolved. When these dependencies are found they will be output to the file which defaults to the manifest file with the extenstion .res.

pkgdepend resolve -m apr-1.7.0.p5m.3

Add Further Data to the Manifest via "Transforms"

Transforms are a way to alter the package manifests in repeatable manner. These are issued with the pkgmogrify command, which we used when we added the metadata to the initial manifest. The detail of the file apr-1.7.0-transform.mog will be discussed below:

<transform dir path=opt$->drop>

<transform file path=opt/pbdigital/apr-1.7.0/(s?bin)/(.*) -> emit \
    link path=opt/pbdigital/%<1>/%<2> target=/opt/pbdigital/apr-1.7.0/%<1>/%<2> >

<transform file path=opt/pbdigital/apr-1.7.0/man/(man\d)/(.*) -> emit \
    link path=opt/pbdigital/share/man/%<1>/%<2> \
    target=/opt/pbdigital/apr-1.7.0/man/%<1>/%<2> >

<transform dir file link hardlink path=opt/pbdigital/.+/man(/.+)? -> \
    default facet.doc.man true>

<transform file link path=opt/pbdigital/share/man/.*\.\d[a-z]*$ -> \
    add restart_fmri svc:/system/update-man-index:default>
  • drop: this transform states the /opt directory will be dropped from the manifest. We do this as it is a system directory that does not belong to the package.

  • linking: here the binaries are linked to the /opt/pbdigital/bin & /opt/pbdigital/sbin directory so they will be available to endusers, that have /opt/pbdigital in their path.

  • linking: here the manpages are linked to the /opt/pbdigital/share/man directory so they will be available to endusers, that have /opt/pbdigital in their path.

  • restart_fmri: indicates to the installer that the sevice update-man-index needs to be re-indexed to include its results for the apropos & whatis commands.

Below is the command to create the manifest with the transforms file.

1pkgmogrify apr-1.7.0.p5m.3.res apr-1.7.0-transform.mog | pkgfmt > apr-1.7.0.p5m.4

{: .language-shell}

Check the Manifest with pkglint

The last thing we need to do before publication is run pkglint on our manifest. This catches any errors that have been made in the manifest.

pkglint -c repo-cache -r https://pkg.omniosce.org/r151032/core  apr-1.7.0.p5m.4

Publish the Package to local IPS Repository

With a clean output from pkglint, that has not reported any errors, we are ready to publish the package to the local IPS Repository with the following command:

pkgsend publish -s /export/ips-repo -d proto apr-1.7.0.p5m.4

Install the package

Now the easy part, install the package.

pkg install apr

Tada! ... and now is time to rinse and repeat for the following packages.

Creating the Apache Portable Runtime Utilities Package

The files that differ from above to create the apr-util-1.6.1 package are availble here as apr-util-1.6.1.mog & apr-util-1.6.1-transform.mog. With these file, packaging the Apache Portable Runtime Utilities follows the same process as above. However as we have an extra dependency on Berkeley-DB, we have to also include the omnios-extra repository as below.

pkglint -c repo-cache -r https://pkg.omniosce.org/r151032/core -r https://pkg.omniosce.org/r151032/extra apr-util-1.6.1.p5m.4

Once this has been published, we can move onto building the httpd package.

Creating the Apache httpd Package

As above we follow the same pattern to install the httpd server files into the proto directory. However we have files in 3 locations of the root flie system, so we create the appropriate directories from the httpd working directory as follows:

mkdir -p proto/opt/pbdigital /etc/opt/pbdigital /var/opt/pbdigital
mv /opt/pbdigital/httpd-2.4.41 proto/opt/pbdigital
mv /etc/opt/pbdigital/httpd-2.4.41 proto/etc/opt/pbdigital
mv /var/opt/pbdigital/httpd-2.4.41 proto/var/opt/pbdigital

Further we will be adding a SMF Manifest & Method to contol the server. We need to create the following directories and copy the Manifest (apache24.xml) & Method (apache24) files into place.

mkdir -p proto/lib/svc/{manifest/network,method} 
cp apache24.xml proto/lib/svc/manifest/network/
cp apache24 proto/lib/svc/method/

I will not discuss how Manifests and Methods are created. You should be able to get a good understanding of how these files work by viewing the above Manifest & Method links. However, if you would like to know more you can download this Solaris Service Management Facility White Paper.

We can now gererate the initial manifest and use the httpd-2.4.41.mog file to add the metadata.

Additional Transforms for the httpd Package

We need to add additional transforms which are listed in httpd-2.4.41-transform.mog. The file is below so this can be discussed:

<transform dir path=opt$->drop>
<transform dir path=etc$->drop>
<transform dir path=etc/opt$->drop>
<transform dir path=var$->drop>
<transform dir path=var/opt$->drop>
<transform dir path=lib$->drop>
<transform dir path=lib/svc$->drop>
<transform dir path=lib/svc/manifest$->drop>
<transform dir path=lib/svc/method$->drop>
<transform dir path=lib/svc/manifest/network$->drop>

<transform file path=opt/pbdigital/httpd-2.4.41/(s?bin)/(.*) -> emit \
    link path=opt/pbdigital/%<1>/%<2> target=/opt/pbdigital/httpd-2.4.41/%<1>/%<2> >

<transform file path=opt/pbdigital/httpd-2.4.41/man/(man\d)/(.*) -> emit \
    link path=opt/pbdigital/share/man/%<1>/%<2> \
    target=/opt/pbdigital/httpd-2.4.41/man/%<1>/%<2> >

<transform dir file link hardlink path=opt/pbdigital/.+/man(/.+)? -> \
    default facet.doc.man true>

<transform file link path=opt/pbdigital/share/man/.*\.\d[a-z]*$ -> \
    add restart_fmri svc:/system/update-man-index:default>

<transform dir  path=var/opt/pbdigital/httpd-2.4.41 -> set owner webservd>

<transform dir  path=var/opt/pbdigital/httpd-2.4.41 -> set group webservd>

<transform file path=lib/svc/manifest/network/ \
    -> add restart_fmri svc:/system/manifest-import:default> 

The new and interesting transforms we have made are as follows:

  • set owner: We set the ownership of /var/opt/pbdigital to the user webservd, which is a built in user that is already available.

  • set group: We set the group ownership of /var/opt/pbdigital to the group webservd, which is a built in group that is already available.

  • restart_fmri: indicates to the installer that the sevice manifest-import needs to be restarted in order to make the httpd server available via the svccadm command.

And finally, like the apr-util-1.6.1 package, we have to add a further repository for pkglint. This time we need to add our own local repository (http://localhost:9001) as the package httpd-2.4.41 depends on our previous local packages.

pkglint -c repo-cache -r https://pkg.omniosce.org/r151032/core -r https://pkg.omniosce.org/r151032/extra -r http://localhost:9001 httpd-2.4.41.p5m.4

Once published, we can restart the IPS Repository, and we will be presented with the available packages:

svcadm restart server/pkg

IPS Package Catalog

Installing and Starting the httpd Server

As always we use the pkg install command to install the package, and once installed, the httpd server can be started via the svcadm command.

pkg install httpd
svcadm enable apache24

Wrapping Up

Thank you for getting all the way to here! I hope it has saved you some time and if you would like to make any comments please email me.