FreeBSD has been long due a better package management system, pkg_add, pkg_info, etc just doesn’t cut it any more. For a long time GNU/linux users have always used this as a reason not to use FreeBSD and instead favour some GNU/linux combination with an all encompassing easy to use package manager, such as Debian’s apt-get. FreeBSD’s response has always been, (not actual quote), “We have the ports collection, which is cooler and more flexible than just having some easy to use package manager. You can always do it yourself by writing scripts around the pkg_* tools or using portmaster’s –packages-only option”. While this is all true, there is still a gap for a good package manager that needs filling. So here comes pkgng (pkg: (next|new) generation), this is FreeBSD’s next generation package manager and although still in beta, with a few features missing, it is very nice.

pkg (aka pkgng) 1.0 released http://lists.freebsd.org/pipermail/freebsd-ports/2012-August/077909.html

Here is a quick overview of pkgng, how to use it and some of the new features that will be available. You might find the official wiki page interesting reading if you haven’t seen it already, pkgng – FreeBSD Wiki.

Getting started: Install pkgng

First let’s install pkgng, (should we be calling it simply ‘pkg’?), it is in the FreeBSD ports tree and you can find it under ‘ports-mgmt/pkg‘. If you are running HEAD you will find that pkg is now part of the base system.

From ports

# portsnap fetch update
# cd /usr/ports/ports-mgmt/pkg
# make install clean

From packages

# pkg_add -r pkg
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/pkg.tbz... Done.

Getting familiar with pkgng

Now that we have pkgng (pkg) installed we can get used to the new CLI and man pages.

The main CLI utility for pkgng is ‘pkg‘, let’s see the usage brief for it:

# pkg
usage: pkg [-v] [-d] [-j |-c ]  []

Global options supported:
        -d             Increment debug level
        -j             Execute pkg(1) inside a jail(8)
        -c             Execute pkg(1) inside a chroot(8)
        -v             Display pkg(1) version

Commands supported:
        add            Registers a package and installs it on the system
        audit          Reports vulnerable packages
        autoremove     Removes orphan packages
        backup         Backs-up and restores the local package database
        check          Checks for missing dependencies and database consistency
        clean          Cleans old packages from the cache
        create         Creates software package distributions
        delete         Deletes packages from the database and the system
        fetch          Fetches packages from a remote repository
        help           Displays help information
        info           Displays information about installed packages
        install        Installs packages from remote package repositories
        query          Queries information about installed packages
        register       Registers a package into the local database
        remove         Deletes packages from the database and the system
        repo           Creates a package repository catalogue
        rquery         Queries information in repository catalogues
        search         Performs a search of package repository catalogues
        set            Modifies information about packages in the local database
        shell          Opens a debug shell
        shlib          Displays which packages link against a specific shared library
        stats          Displays package database statistics
        update         Updates package repository catalogues
        updating       Displays UPDATING information for a package
        upgrade        Performs upgrades of packaged software distributions
        version        Displays the versions of installed packages
        which          Displays which package installed a specific file

For more information on the different commands see 'pkg help '.

First thing we notice is the global options, other than the basic debug and display version flags we can see some jail/chroot options. One of the very cool features of pkgng, you can manage packages in side a jail or chroot from the host OS by providing the Jail ID (JID) or chroot path. The next thing we see are all the subcommands that we can use, including a useful ‘help’ subcommand that will quickly display the man page (if available for that command). Here’s an exmaple of the help command and the available man pages.

# pkg help
usage: pkg help 

Where  can be:
        add
        audit
        autoremove
        backup
        check
        clean
        create
        delete
        fetch
        help
        info
        install
        query
        register
        remove
        repo
        rquery
        search
        set
        shell
        shlib
        stats
        update
        updating
        upgrade
        version
        which

Example usage of the pkgng help command

# pkg help add
PKG-ADD(8)              FreeBSD System Manager's Manual             PKG-ADD(8)

NAME
     pkg add -- Registers a package and installs it on the system

SYNOPSIS
     pkg add 
     pkg add :///

DESCRIPTION
     pkg add installs a package from either a local source or a remote one.

     When installing from a remote source you need to specify the protocol to
     use when fetching the package.

     Currently supported protocols are FTP, HTTP and HTTPS.

OPTIONS
     The following options are supported by pkg add:

ENVIRONMENT
     The following environment variables affect the execution of pkg add.  See
     pkg.conf(5) for further description.

     ASSUME_ALWAYS_YES

     HANDLE_RC_SCRIPTS

     PKG_DBDIR

FILES
     See pkg.conf(5).

SEE ALSO
     pkg(8), pkg-audit(8), pkg-autoremove(8), pkg-backup(8), pkg-check(8),
     pkg-clean(8), pkg-create(8), pkg-delete(8), pkg-fetch(8), pkg-info(8),
     pkg-install(8), pkg-query(8), pkg-register(8), pkg-repo(8),
     pkg-rquery(8), pkg-search(8), pkg-set(8), pkg-shell(8), pkg-shlib(8),
     pkg-stats(8), pkg-update(8), pkg-updating(8), pkg-upgrade(8),
     pkg-version(8), pkg-which(8), pkg.conf(5)

FreeBSD 10.0                     June 12, 2012                    FreeBSD 10.0

The man pages currently installed by pkgng are:

Edit: In beta 7 all man pages in section 1 were moved to section 8.

• man 8 pkg
• man 8 pkg-add
• man 8 pkg-autoremove
• man 8 pkg-backup
• man 8 pkg-create
• man 8 pkg-delete
• man 8 pkg-info
• man 8 pkg-install
• man 8 pkg-query
• man 8 pkg-register
• man 8 pkg-repo
• man 8 pkg-search
• man 8 pkg-set (pkg set added in beta8)
• man 8 pkg-update
• man 8 pkg-updating (added in beta6)
• man 8 pkg-upgrade
• man 8 pkg-version
• man 8 pkg-which
• man 5 pkg.conf

Pkgng’s configuration file: pkg.conf

The last man page listed above is for pkg.conf, this is the system-wide confutation file for pkgng’s pkg tools. The configuration file is in YAML format. For more information on the syntax of YAML, please visit the official YAML website – http://www.yaml.org.

If you want to try pkgng for yourself now you won’t be able to use the official FreeBSD package mirrors, but you can use the beta package server at http://pkgbeta.freebsd.org. This beta server is not kept as upto date as the live repos but it will start you on your way. You could also setup your own package repository which can be used by pkgng, for this you could use poudrier http://unix-heaven.org/continuous-package-building-with-poudriere-and-jenkins or tinderbox http://www.glenbarber.us/2012/06/11/Maintaining-Your-Own-pkgng-Repository.html.

To get pkgng working you must first configure a PACKAGESITE in /usr/local/etc/pkg.conf, or you will get an error like this:

# pkg update
pkg: Invalid configuration format, ignoring the configuration file
Updating repository catalogue
pkg: PACKAGESITE is not defined.

file: /usr/local/etc/pkg.conf

# cat /usr/local/etc/pkg.conf
PACKAGESITE: http://pkgbeta.freebsd.org/freebsd-9-amd64/latest

In the near future this value will be set automatically.

If you want to test upgrading packages you might want to start with PACKAGESITE defined to http://pkgbeta.freebsd.org/freebsd-9-amd64/2012-08-12 and then change it to the above when you want to test upgrades.

pkg2ng – migrating to pkgng

Pkgng has it’s own way of keeping track of installed packages. This means to start using pkgng with all your existing packages you need to import them first. To do this pkgng provides a nice shell script which is installed in /usr/local/sbin/pkg2ng. Migrating to pkgng is a simple as this.

# pkg2ng

When the script finishes all of your packages will have been registered with pkgng and you can start using it as your new package manager. You will also notice the old pkg_* tools are now obsolete, running pkg_info will show no installed packages. The original /var/db/pkg directory is backed up by pkg2ng to /var/db/pkg.bak.

Linking ports to pkgng

When you make install a new port, the FreeBSD ports system will register the newly installed port with pkg_*. To make the ports instead register with pkg you need to make an add WITH_PKGNG=yes to make.conf.

# cat /etc/make.conf
WITH_PKGNG=yes

pkg update – update from remote package repository databases

Pkgng uses an sqlite database to store all the information it needs about a repo and the packages it contains. You can create your own for your own database repo by using the pkg repo <path_to_repo> command. This is one of the reasons why you cannot use pkgng with the official mirrors yet, this package ‘repo.txz’ and therefore the file ‘repo.sqlite’ does not exist yet.

Assuming you have PACKAGESITE setup as described in the pkg.conf section or set to a similar package site, maybe your own? You can now update pkgng from the remote repository database.

You should always do this before doing a remote install or upgrades.

# pkg update
Updating repository catalogue
repo.txz                         100% 9055KB   8.8MB/s   6.1MB/s   00:01

pkg search – Finding a package to install with pkgng

Now that we have our repository databases up to date we can start searching the remote repository for packages to install. For most of the examples here I will be using the mysql-server 5.5 package.

Search for MySQL server package to install

# pkg search mysql-server
mysql-server-5.1.65            Multithreaded SQL database (server)
mysql-server-5.0.95            Multithreaded SQL database (server)
mysql-server-4.1.25            Multithreaded SQL database (server)
mysql-server-5.5.27            Multithreaded SQL database (server)

If you get an error like below you need to configure your PACKAGESITE and run pkg update first (see above).

# pkg search mysql-server
Unable to open remote database "repo". Try running 'pkg update' first.

pkg install – Installing a package

When you have identified the package and version you wish to install you can easily install it using the pkg install command like this.

# pkg install mysql-server-5.5.15
Updating repository catalogue
Repository catalogue is up-to-date, no need to fetch fresh copy
The following packages will be installed:

        Installing mysql-client: 5.5.27
        Installing mysql-server: 5.5.27

The installation will require 116 MB more space

10 MB to be downloaded

Proceed with installing packages [y/N]:
mysql-client-5.5.27.txz                                                  100% 1493KB   1.5MB/s   1.5MB/s   00:01
mysql-server-5.5.27.txz                                                  100% 8691KB   8.5MB/s   6.5MB/s   00:01
Checking integrity... done
Installing mysql-client-5.5.27... done
Installing mysql-server-5.5.27...===> Creating users and/or groups.
Using existing group 'mysql'.
Using existing user 'mysql'.
 done
************************************************************************

Remember to run mysql_upgrade (with the optional --datadir= flag)
the first time you start the MySQL server after an upgrade from an
earlier version.

************************************************************************

Note: pkg install runs pkg update before attempting to install anything.

As you can see pkgng will find any dependencies it needs and install them. Before it actually does anything it will show you what it is going to do as well as how much disk space will be used and how much data will be downloaded. You can then proceed or cancel the install.

pkg add – Installs a package from a local or remote source

The pkg add command is a little bit like pkg install but don’t get them confused, they are quite different. pkg add requires a full path to the package file to install whether that is local or remote. It is also unable to handle dependencies automatically, as in this example.

#  pkg add http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/misc/hello-2.8.txz
hello-2.8.txz                                       100%   46KB  46.0KB/s  46.0KB/s   00:00
Installing hello-2.8...missing dependency libiconv-1.14
Failed to install the following 1 package(s): http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/misc/hello-2.8.txz.

With pkg add you must resolve all dependencies manually and tell it where to get the packages from.

# fetch http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/All/libiconv-1.14.txz
libiconv-1.14.txz                             100% of  590 kB 1511 kBps
# pkg add libiconv-1.14.txz
Installing libiconv-1.14... done
# pkg add http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/All/gettext-0.18.1.1.txz
gettext-0.18.1.1.txz                          100% of 4999 kB 5013 kBps
Installing gettext-0.18.1.1... done
# pkg add http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/misc/hello-2.8.txz
hello-2.8.txz                                       100%   46KB  46.0KB/s  46.0KB/s   00:00
Installing hello-2.8... done
# hello
Hello, world!

pkg info – Displaying information about installed packages

Now we have some packages installed with pkgng we can use pkg info to display some information about them. There are many options you can give to pkg info to customize the output for your needs. Such as ‘Display the list of packages which depend on .’ (-d) and ‘Display all files installed by .’ (-l), for the full list you should check the man page by using pkg help info or man 1 pkg-info.

Display all installed packages with versions and descriptions

# pkg info
mysql-client-5.5.15: Multithreaded SQL database (client)
mysql-server-5.5.15: Multithreaded SQL database (server)
nInvaders-0.1.1: The nIvaders game is a Space Invaders clone for ncurses

Display package dependencies

# pkg info -d mysql-server-5.5.19
mysql-server-5.5.19 depends on:
mysql-client-5.5.19

Display all files install by package

# pkg info -l nInvaders-0.1.1
nInvaders-0.1.1 owns the following files:
/usr/local/bin/nInvaders
/usr/local/share/doc/nInvaders/README

pkg query – Display information about packages how you want it

The pkg query command will print out information that you choose, in a format you choose, about all or selected installed packages. Very cool and useful feature for power users or for scripting. Here are some examples.

Query for package mysql-server and display name, version and size.

# pkg query "Package name = %n, Version = %v, Size = %sh" mysql-server
Package name = mysql-server, Version = 5.5.15, Size = 74 MB

Query all packages for name, version and size.

# pkg query -a "Package name = %n, Version = %v, Size = %sh"
Package name = mysql-client, Version = 5.5.15, Size = 36 MB
Package name = mysql-server, Version = 5.5.15, Size = 74 MB
Package name = nInvaders, Version = 0.1.1, Size = 53 kB

Get more information for mysql-server

If you want to copy and paste this command

pkg query "
	package[%n]n
	version[%v]n
	origin[%o]n
	prefix[%p]n
	maintainer[%m]n
	comment[%c]n
	www[%w]n
	licenselogic[%l]n
	flatsize[%sh]n
	flatsizebytes[%sb]n
	orphan[%a]n
	message[%M]n
" mysql-server

Which will output something like this

package[mysql-server]
version[5.5.15]
origin[databases/mysql55-server]
prefix[/usr/local]
maintainer[ale@FreeBSD.org]
comment[Multithreaded SQL database (server)]
www[http://www.mysql.com/]
licenselogic[single]
flatsize[74 MB]
flatsizebytes[78348693]
orphan[0]
message[************************************************************************

Remember to run mysql_upgrade (with the optional --datadir= flag)
the first time you start the MySQL server after an upgrade from an
earlier version.

************************************************************************]

There is more you can do with pkg query so dive into the man page and get playing.

pkg which – Pkgng can tell you which package created that file

When packages are installed they can scatter files all over the place. You might be told which file went were when installing the packages, but the chances are you wont know which files originated from which package. Unless you look at the plist or use pkg info -la, but this is messy just to grep for some file. Pkgng provides pkg which, you can run it against any file to see if it came from a package and if so which one.

Where did /usr/local/lib/mysql/libmysqlclient.a come from?

# pkg which /usr/local/lib/mysql/libmysqlclient.a
/usr/local/lib/mysql/libmysqlclient.a was installed by package mysql-client-5.5.15

pkg backup – Backing up the local package database

As mentioned before pkgng uses a database to keep track of the packages and to store information about them. As you can imagine if this file gets removed or corrupted you could find yourself in a mess. Pkgng gives you a way of backing up and restoring this database.

Backing up the local package database to a file.

# pkg backup -d /tmp/out
Dumping database...done
# file /tmp/out.txz
/tmp/out.txz: xz compressed data
# tar -tvf /tmp/out.txz
-rw-r--r--  0 root   wheel   12584 Jan  1  1970 mysql-client-5.5.15.yaml
-rw-r--r--  0 root   wheel   17557 Jan  1  1970 mysql-client-5.5.15.mtree
-rw-r--r--  0 root   wheel   24186 Jan  1  1970 mysql-server-5.5.15.yaml
-rw-r--r--  0 root   wheel       0 Jan  1  1970 mysql-server-5.5.15.mtree
-rw-r--r--  0 root   wheel     799 Jan  1  1970 nInvaders-0.1.1.yaml
-rw-r--r--  0 root   wheel       0 Jan  1  1970 nInvaders-0.1.1.mtree

Restoring the local package database from backup

# pkg backup -r /tmp/out.txz
Restoring database...done

pkg create – Creating packages using pkgng

You can create packages using pkg create from your local binaries, these packages can then be distrusted to other machines and installed using the pkg add command.

# pkg create -a
Creating package for mysql-client-5.5.15
Creating package for mysql-server-5.5.15
Creating package for nInvaders-0.1.1
# ls
mysql-client-5.5.15.txz
mysql-server-5.5.15.txz
nInvaders-0.1.1.txz

pkg version – Installed versions of packages

You can compare the versions of packages you have installed against the remote version using the pkg version command. If you and playing along at home this is the time to update your pkg.conf to use the newer repository, (see above).

First you should update your local copy of the remote repository database

Edit: You can use pkg version to see and compare the versions of packages you have installed. According to the pkgng wiki the pkg version command is the same as pkg_version, this means that package versions are compared with your local ports Makefiles or index file.

Then run the pkg version command to see packages that need upgrading

# pkg version -v
mysql-client                       <   needs updating (port has 5.5.20)
mysql-server                       <   needs updating (port has 5.5.20)
nInvaders                          =   up-to-date with port

pkg updating - Search /usr/ports/UPDATING for upgrading notes

It is important to check the /usr/ports/UPDATING file for any changes to a package that might require extra actions or things to be aware of before you upgrade it. pkg updating provides a way of searching this file using the pkg CLI. Running pkg updating with no options will print out the entire file, but you can add some basic filters to help you find what you're looking for.

For more details on pkg updating usage see the man page

# pkg help updating

Show messages after 01/01/2011

# pkg updating -d 20110101
20110319:
  AFFECTS: users of databases/mysql55-client
  AUTHOR: ale@FreeBSD.org

  The shared library version of the client library was increased to reflect
  ABI changes, and avoid compatibility problems with the client library
  in MySQL 5.1, so client programs that use the 5.5 client library should
  be recompiled against the 5.5.10 client library.
  This can be achieved with:

  # portmaster -r mysql-client-5.5
  or
  # portupgrade -fr mysql-client-5.5

pkg upgrade - Upgrading packages with pkgng

Pkgng can take advantages of packages that are upgrade aware, by executing the provided PRE-UPGRADE and POST-UPGRADE scripts. If the package is not upgrade aware it will simply just do a pkg delete and pkg install.

Upgrade all outdated packages

# pkg upgrade
The following packages will be upgraded:
	Upgrading mysql-client: 5.5.15 -> 5.5.19
	Upgrading mysql-server: 5.5.15 -> 5.5.19

the upgrade will require 4 MB more space
9 MB to be downloaded

Proceed with upgrading packages [y/N]: y
http://repos.etoilebsd.net/9-amd64-20111222/All/mysql-client-5.5.19.txz       100% 1518KB   1.5MB/s   1.5MB/s   00:00
http://repos.etoilebsd.net/9-amd64-20111222/All/mysql-server-5.5.19.txz       100% 8395KB   8.2MB/s   6.8MB/s   00:01
Checking integrity... done
Upgrading mysql-client from 5.5.15 to 5.5.19... done
Upgrading mysql-server from 5.5.15 to 5.5.19...==> You should manually remove the "mysql" user.
===> Creating users and/or groups.
Using existing group 'mysql'.
Using existing user 'mysql'.
 done

As with pkg install you will be shown the actions that will be taken, how much additional disk space will be used and how much data needs to be downloaded. You can then continue or abort if you see something you don't like.

pkg delete - Removing an installed package

Deleting a package with pkgng is pretty self explanatory, but as always do check the man page as it does't have some cool options, such as regular expressions.

Removing MySQL server

# pkg delete mysql-server
The following packages will be deinstalled:
	mysql-server-5.5.19

The deinstallation will save 79 MB

Proceed with deinstalling packages [y/N]: y
Deinstalling mysql-server-5.5.19...==> You should manually remove the "mysql" user.
 done

You will have a chance to abort if you change your mind and the usual pre-operation information is displayed. Any additional manual operations that you need to do will be printed at the end. Such as here, manually removing the "mysql" user.

pkg audit - built in portaudit

Security vulnerabilities are always being found and reported, so it is important to have a clean way of checking your installed packages against a currently list of known problems. Normally for this we would install portaudit, have it email us problems from periodic and warn us when trying to installing a port with known problems.

Guest what, pkgng has this feature built it. Lets look at a basic example of it in use.

Running pkg audit for the first time.

# pkg audit
pkg: unable to open audit file, try running pkg audit -F first

Ok so we need to download the audit database file first.

# pkg audit -F
http://portaudit.FreeBSD.org/auditfile.tbz               100%   76KB  37.8KB/s  55.6KB/s   00:02
openssl-1.0.0_9 is vulnerable
OpenSSL -- CMS and S/MIME Bleichenbacher attack

WWW: http://portaudit.FreeBSD.org/60eb344e-6eb1-11e1-8ad7-00e0815b8da8.html


1 problem(s) in your installed packages found.

If your familiar with portaudit then this command will make you feel right at home.

Pkg 1.0 is out now! The betas and RCs are over, say hello to the first release!

For more information see the pkgng - FreeBSD Wiki.