pkgng: First look at FreeBSD’s new package manager

FreeBSD has been long due a better package management system, pkg_add, pkg_info, etc just doesn’t cut it any more. For a long time GNU/linux users have always used this as a reason not to use FreeBSD and instead favour some GNU/linux combination with an all encompassing easy to use package manager, such as Debian’s apt-get. FreeBSD’s response has always been, (not actual quote), “We have the ports collection, which is cooler and more flexible than just having some easy to use package manager. You can always do it yourself by writing scripts around the pkg_* tools or using portmaster’s –packages-only option”. While this is all true, there is still a gap for a good package manager that needs filling. So here comes pkgng (pkg: (next|new) generation), this is FreeBSD’s next generation package manager and although still in beta, with a few features missing, it is very nice.

pkg (aka pkgng) 1.0 released http://lists.freebsd.org/pipermail/freebsd-ports/2012-August/077909.html

Here is a quick overview of pkgng, how to use it and some of the new features that will be available. You might find the official wiki page interesting reading if you haven’t seen it already, pkgng – FreeBSD Wiki.

Getting started: Install pkgng

First let’s install pkgng, (should we be calling it simply ‘pkg’?), it is in the FreeBSD ports tree and you can find it under ‘ports-mgmt/pkg‘. If you are running HEAD you will find that pkg is now part of the base system.

From ports

# portsnap fetch update
# cd /usr/ports/ports-mgmt/pkg
# make install clean

From packages

# pkg_add -r pkg
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/pkg.tbz... Done.

Getting familiar with pkgng

Now that we have pkgng (pkg) installed we can get used to the new CLI and man pages.

The main CLI utility for pkgng is ‘pkg‘, let’s see the usage brief for it:

# pkg
usage: pkg [-v] [-d] [-j |-c ]  []

Global options supported:
        -d             Increment debug level
        -j             Execute pkg(1) inside a jail(8)
        -c             Execute pkg(1) inside a chroot(8)
        -v             Display pkg(1) version

Commands supported:
        add            Registers a package and installs it on the system
        audit          Reports vulnerable packages
        autoremove     Removes orphan packages
        backup         Backs-up and restores the local package database
        check          Checks for missing dependencies and database consistency
        clean          Cleans old packages from the cache
        create         Creates software package distributions
        delete         Deletes packages from the database and the system
        fetch          Fetches packages from a remote repository
        help           Displays help information
        info           Displays information about installed packages
        install        Installs packages from remote package repositories
        query          Queries information about installed packages
        register       Registers a package into the local database
        remove         Deletes packages from the database and the system
        repo           Creates a package repository catalogue
        rquery         Queries information in repository catalogues
        search         Performs a search of package repository catalogues
        set            Modifies information about packages in the local database
        shell          Opens a debug shell
        shlib          Displays which packages link against a specific shared library
        stats          Displays package database statistics
        update         Updates package repository catalogues
        updating       Displays UPDATING information for a package
        upgrade        Performs upgrades of packaged software distributions
        version        Displays the versions of installed packages
        which          Displays which package installed a specific file

For more information on the different commands see 'pkg help '.

First thing we notice is the global options, other than the basic debug and display version flags we can see some jail/chroot options. One of the very cool features of pkgng, you can manage packages in side a jail or chroot from the host OS by providing the Jail ID (JID) or chroot path. The next thing we see are all the subcommands that we can use, including a useful ‘help’ subcommand that will quickly display the man page (if available for that command). Here’s an exmaple of the help command and the available man pages.

# pkg help
usage: pkg help 

Where  can be:
        add
        audit
        autoremove
        backup
        check
        clean
        create
        delete
        fetch
        help
        info
        install
        query
        register
        remove
        repo
        rquery
        search
        set
        shell
        shlib
        stats
        update
        updating
        upgrade
        version
        which

Example usage of the pkgng help command

# pkg help add
PKG-ADD(8)              FreeBSD System Manager's Manual             PKG-ADD(8)

NAME
     pkg add -- Registers a package and installs it on the system

SYNOPSIS
     pkg add 
     pkg add :///

DESCRIPTION
     pkg add installs a package from either a local source or a remote one.

     When installing from a remote source you need to specify the protocol to
     use when fetching the package.

     Currently supported protocols are FTP, HTTP and HTTPS.

OPTIONS
     The following options are supported by pkg add:

ENVIRONMENT
     The following environment variables affect the execution of pkg add.  See
     pkg.conf(5) for further description.

     ASSUME_ALWAYS_YES

     HANDLE_RC_SCRIPTS

     PKG_DBDIR

FILES
     See pkg.conf(5).

SEE ALSO
     pkg(8), pkg-audit(8), pkg-autoremove(8), pkg-backup(8), pkg-check(8),
     pkg-clean(8), pkg-create(8), pkg-delete(8), pkg-fetch(8), pkg-info(8),
     pkg-install(8), pkg-query(8), pkg-register(8), pkg-repo(8),
     pkg-rquery(8), pkg-search(8), pkg-set(8), pkg-shell(8), pkg-shlib(8),
     pkg-stats(8), pkg-update(8), pkg-updating(8), pkg-upgrade(8),
     pkg-version(8), pkg-which(8), pkg.conf(5)

FreeBSD 10.0                     June 12, 2012                    FreeBSD 10.0

The man pages currently installed by pkgng are:

Edit: In beta 7 all man pages in section 1 were moved to section 8.

Pkgng’s configuration file: pkg.conf

The last man page listed above is for pkg.conf, this is the system-wide confutation file for pkgng’s pkg tools. The configuration file is in YAML format. For more information on the syntax of YAML, please visit the official YAML website – http://www.yaml.org.

If you want to try pkgng for yourself now you won’t be able to use the official FreeBSD package mirrors, but you can use the beta package server at http://pkgbeta.freebsd.org. This beta server is not kept as upto date as the live repos but it will start you on your way. You could also setup your own package repository which can be used by pkgng, for this you could use poudrier http://unix-heaven.org/continuous-package-building-with-poudriere-and-jenkins or tinderbox http://www.glenbarber.us/2012/06/11/Maintaining-Your-Own-pkgng-Repository.html.

To get pkgng working you must first configure a PACKAGESITE in /usr/local/etc/pkg.conf, or you will get an error like this:

# pkg update
pkg: Invalid configuration format, ignoring the configuration file
Updating repository catalogue
pkg: PACKAGESITE is not defined.

file: /usr/local/etc/pkg.conf

# cat /usr/local/etc/pkg.conf
PACKAGESITE: http://pkgbeta.freebsd.org/freebsd-9-amd64/latest

In the near future this value will be set automatically.

If you want to test upgrading packages you might want to start with PACKAGESITE defined to http://pkgbeta.freebsd.org/freebsd-9-amd64/2012-08-12 and then change it to the above when you want to test upgrades.

pkg2ng – migrating to pkgng

Pkgng has it’s own way of keeping track of installed packages. This means to start using pkgng with all your existing packages you need to import them first. To do this pkgng provides a nice shell script which is installed in /usr/local/sbin/pkg2ng. Migrating to pkgng is a simple as this.

# pkg2ng

When the script finishes all of your packages will have been registered with pkgng and you can start using it as your new package manager. You will also notice the old pkg_* tools are now obsolete, running pkg_info will show no installed packages. The original /var/db/pkg directory is backed up by pkg2ng to /var/db/pkg.bak.

Linking ports to pkgng

When you make install a new port, the FreeBSD ports system will register the newly installed port with pkg_*. To make the ports instead register with pkg you need to make an add WITH_PKGNG=yes to make.conf.

# cat /etc/make.conf
WITH_PKGNG=yes

pkg update – update from remote package repository databases

Pkgng uses an sqlite database to store all the information it needs about a repo and the packages it contains. You can create your own for your own database repo by using the pkg repo <path_to_repo> command. This is one of the reasons why you cannot use pkgng with the official mirrors yet, this package ‘repo.txz’ and therefore the file ‘repo.sqlite’ does not exist yet.

Assuming you have PACKAGESITE setup as described in the pkg.conf section or set to a similar package site, maybe your own? You can now update pkgng from the remote repository database.

You should always do this before doing a remote install or upgrades.

# pkg update
Updating repository catalogue
repo.txz                         100% 9055KB   8.8MB/s   6.1MB/s   00:01

pkg search – Finding a package to install with pkgng

Now that we have our repository databases up to date we can start searching the remote repository for packages to install. For most of the examples here I will be using the mysql-server 5.5 package.

Search for MySQL server package to install

# pkg search mysql-server
mysql-server-5.1.65            Multithreaded SQL database (server)
mysql-server-5.0.95            Multithreaded SQL database (server)
mysql-server-4.1.25            Multithreaded SQL database (server)
mysql-server-5.5.27            Multithreaded SQL database (server)

If you get an error like below you need to configure your PACKAGESITE and run pkg update first (see above).

# pkg search mysql-server
Unable to open remote database "repo". Try running 'pkg update' first.

pkg install – Installing a package

When you have identified the package and version you wish to install you can easily install it using the pkg install command like this.

# pkg install mysql-server-5.5.15
Updating repository catalogue
Repository catalogue is up-to-date, no need to fetch fresh copy
The following packages will be installed:

        Installing mysql-client: 5.5.27
        Installing mysql-server: 5.5.27

The installation will require 116 MB more space

10 MB to be downloaded

Proceed with installing packages [y/N]:
mysql-client-5.5.27.txz                                                  100% 1493KB   1.5MB/s   1.5MB/s   00:01
mysql-server-5.5.27.txz                                                  100% 8691KB   8.5MB/s   6.5MB/s   00:01
Checking integrity... done
Installing mysql-client-5.5.27... done
Installing mysql-server-5.5.27...===> Creating users and/or groups.
Using existing group 'mysql'.
Using existing user 'mysql'.
 done
************************************************************************

Remember to run mysql_upgrade (with the optional --datadir= flag)
the first time you start the MySQL server after an upgrade from an
earlier version.

************************************************************************

Note: pkg install runs pkg update before attempting to install anything.

As you can see pkgng will find any dependencies it needs and install them. Before it actually does anything it will show you what it is going to do as well as how much disk space will be used and how much data will be downloaded. You can then proceed or cancel the install.

pkg add – Installs a package from a local or remote source

The pkg add command is a little bit like pkg install but don’t get them confused, they are quite different. pkg add requires a full path to the package file to install whether that is local or remote. It is also unable to handle dependencies automatically, as in this example.

#  pkg add http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/misc/hello-2.8.txz
hello-2.8.txz                                       100%   46KB  46.0KB/s  46.0KB/s   00:00
Installing hello-2.8...missing dependency libiconv-1.14
Failed to install the following 1 package(s): http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/misc/hello-2.8.txz.

With pkg add you must resolve all dependencies manually and tell it where to get the packages from.

# fetch http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/All/libiconv-1.14.txz
libiconv-1.14.txz                             100% of  590 kB 1511 kBps
# pkg add libiconv-1.14.txz
Installing libiconv-1.14... done
# pkg add http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/All/gettext-0.18.1.1.txz
gettext-0.18.1.1.txz                          100% of 4999 kB 5013 kBps
Installing gettext-0.18.1.1... done
# pkg add http://pkgbeta.freebsd.org/freebsd-10-amd64/latest/misc/hello-2.8.txz
hello-2.8.txz                                       100%   46KB  46.0KB/s  46.0KB/s   00:00
Installing hello-2.8... done
# hello
Hello, world!

pkg info – Displaying information about installed packages

Now we have some packages installed with pkgng we can use pkg info to display some information about them. There are many options you can give to pkg info to customize the output for your needs. Such as ‘Display the list of packages which depend on .’ (-d) and ‘Display all files installed by .’ (-l), for the full list you should check the man page by using pkg help info or man 1 pkg-info.

Display all installed packages with versions and descriptions

# pkg info
mysql-client-5.5.15: Multithreaded SQL database (client)
mysql-server-5.5.15: Multithreaded SQL database (server)
nInvaders-0.1.1: The nIvaders game is a Space Invaders clone for ncurses

Display package dependencies

# pkg info -d mysql-server-5.5.19
mysql-server-5.5.19 depends on:
mysql-client-5.5.19

Display all files install by package

# pkg info -l nInvaders-0.1.1
nInvaders-0.1.1 owns the following files:
/usr/local/bin/nInvaders
/usr/local/share/doc/nInvaders/README

pkg query – Display information about packages how you want it

The pkg query command will print out information that you choose, in a format you choose, about all or selected installed packages. Very cool and useful feature for power users or for scripting. Here are some examples.

Query for package mysql-server and display name, version and size.

# pkg query "Package name = %n, Version = %v, Size = %sh" mysql-server
Package name = mysql-server, Version = 5.5.15, Size = 74 MB

Query all packages for name, version and size.

# pkg query -a "Package name = %n, Version = %v, Size = %sh"
Package name = mysql-client, Version = 5.5.15, Size = 36 MB
Package name = mysql-server, Version = 5.5.15, Size = 74 MB
Package name = nInvaders, Version = 0.1.1, Size = 53 kB

Get more information for mysql-server

If you want to copy and paste this command

pkg query "
	package[%n]n
	version[%v]n
	origin[%o]n
	prefix[%p]n
	maintainer[%m]n
	comment[%c]n
	www[%w]n
	licenselogic[%l]n
	flatsize[%sh]n
	flatsizebytes[%sb]n
	orphan[%a]n
	message[%M]n
" mysql-server

Which will output something like this

package[mysql-server]
version[5.5.15]
origin[databases/mysql55-server]
prefix[/usr/local]
maintainer[ale@FreeBSD.org]
comment[Multithreaded SQL database (server)]
www[http://www.mysql.com/]
licenselogic[single]
flatsize[74 MB]
flatsizebytes[78348693]
orphan[0]
message[************************************************************************

Remember to run mysql_upgrade (with the optional --datadir= flag)
the first time you start the MySQL server after an upgrade from an
earlier version.

************************************************************************]

There is more you can do with pkg query so dive into the man page and get playing.

pkg which – Pkgng can tell you which package created that file

When packages are installed they can scatter files all over the place. You might be told which file went were when installing the packages, but the chances are you wont know which files originated from which package. Unless you look at the plist or use pkg info -la, but this is messy just to grep for some file. Pkgng provides pkg which, you can run it against any file to see if it came from a package and if so which one.

Where did /usr/local/lib/mysql/libmysqlclient.a come from?

# pkg which /usr/local/lib/mysql/libmysqlclient.a
/usr/local/lib/mysql/libmysqlclient.a was installed by package mysql-client-5.5.15

pkg backup – Backing up the local package database

As mentioned before pkgng uses a database to keep track of the packages and to store information about them. As you can imagine if this file gets removed or corrupted you could find yourself in a mess. Pkgng gives you a way of backing up and restoring this database.

Backing up the local package database to a file.

# pkg backup -d /tmp/out
Dumping database...done
# file /tmp/out.txz
/tmp/out.txz: xz compressed data
# tar -tvf /tmp/out.txz
-rw-r--r--  0 root   wheel   12584 Jan  1  1970 mysql-client-5.5.15.yaml
-rw-r--r--  0 root   wheel   17557 Jan  1  1970 mysql-client-5.5.15.mtree
-rw-r--r--  0 root   wheel   24186 Jan  1  1970 mysql-server-5.5.15.yaml
-rw-r--r--  0 root   wheel       0 Jan  1  1970 mysql-server-5.5.15.mtree
-rw-r--r--  0 root   wheel     799 Jan  1  1970 nInvaders-0.1.1.yaml
-rw-r--r--  0 root   wheel       0 Jan  1  1970 nInvaders-0.1.1.mtree

Restoring the local package database from backup

# pkg backup -r /tmp/out.txz
Restoring database...done

pkg create – Creating packages using pkgng

You can create packages using pkg create from your local binaries, these packages can then be distrusted to other machines and installed using the pkg add command.

# pkg create -a
Creating package for mysql-client-5.5.15
Creating package for mysql-server-5.5.15
Creating package for nInvaders-0.1.1
# ls
mysql-client-5.5.15.txz
mysql-server-5.5.15.txz
nInvaders-0.1.1.txz

pkg version – Installed versions of packages

You can compare the versions of packages you have installed against the remote version using the pkg version command. If you and playing along at home this is the time to update your pkg.conf to use the newer repository, (see above).

First you should update your local copy of the remote repository database

Edit: You can use pkg version to see and compare the versions of packages you have installed. According to the pkgng wiki the pkg version command is the same as pkg_version, this means that package versions are compared with your local ports Makefiles or index file.

Then run the pkg version command to see packages that need upgrading

# pkg version -v
mysql-client                       <   needs updating (port has 5.5.20)
mysql-server                       <   needs updating (port has 5.5.20)
nInvaders                          =   up-to-date with port

pkg updating - Search /usr/ports/UPDATING for upgrading notes

It is important to check the /usr/ports/UPDATING file for any changes to a package that might require extra actions or things to be aware of before you upgrade it. pkg updating provides a way of searching this file using the pkg CLI. Running pkg updating with no options will print out the entire file, but you can add some basic filters to help you find what you're looking for.

For more details on pkg updating usage see the man page

# pkg help updating

Show messages after 01/01/2011

# pkg updating -d 20110101
20110319:
  AFFECTS: users of databases/mysql55-client
  AUTHOR: ale@FreeBSD.org

  The shared library version of the client library was increased to reflect
  ABI changes, and avoid compatibility problems with the client library
  in MySQL 5.1, so client programs that use the 5.5 client library should
  be recompiled against the 5.5.10 client library.
  This can be achieved with:

  # portmaster -r mysql-client-5.5
  or
  # portupgrade -fr mysql-client-5.5

pkg upgrade - Upgrading packages with pkgng

Pkgng can take advantages of packages that are upgrade aware, by executing the provided PRE-UPGRADE and POST-UPGRADE scripts. If the package is not upgrade aware it will simply just do a pkg delete and pkg install.

Upgrade all outdated packages

# pkg upgrade
The following packages will be upgraded:
	Upgrading mysql-client: 5.5.15 -> 5.5.19
	Upgrading mysql-server: 5.5.15 -> 5.5.19

the upgrade will require 4 MB more space
9 MB to be downloaded

Proceed with upgrading packages [y/N]: y
http://repos.etoilebsd.net/9-amd64-20111222/All/mysql-client-5.5.19.txz       100% 1518KB   1.5MB/s   1.5MB/s   00:00
http://repos.etoilebsd.net/9-amd64-20111222/All/mysql-server-5.5.19.txz       100% 8395KB   8.2MB/s   6.8MB/s   00:01
Checking integrity... done
Upgrading mysql-client from 5.5.15 to 5.5.19... done
Upgrading mysql-server from 5.5.15 to 5.5.19...==> You should manually remove the "mysql" user.
===> Creating users and/or groups.
Using existing group 'mysql'.
Using existing user 'mysql'.
 done

As with pkg install you will be shown the actions that will be taken, how much additional disk space will be used and how much data needs to be downloaded. You can then continue or abort if you see something you don't like.

pkg delete - Removing an installed package

Deleting a package with pkgng is pretty self explanatory, but as always do check the man page as it does't have some cool options, such as regular expressions.

Removing MySQL server

# pkg delete mysql-server
The following packages will be deinstalled:
	mysql-server-5.5.19

The deinstallation will save 79 MB

Proceed with deinstalling packages [y/N]: y
Deinstalling mysql-server-5.5.19...==> You should manually remove the "mysql" user.
 done

You will have a chance to abort if you change your mind and the usual pre-operation information is displayed. Any additional manual operations that you need to do will be printed at the end. Such as here, manually removing the "mysql" user.

pkg audit - built in portaudit

Security vulnerabilities are always being found and reported, so it is important to have a clean way of checking your installed packages against a currently list of known problems. Normally for this we would install portaudit, have it email us problems from periodic and warn us when trying to installing a port with known problems.

Guest what, pkgng has this feature built it. Lets look at a basic example of it in use.

Running pkg audit for the first time.

# pkg audit
pkg: unable to open audit file, try running pkg audit -F first

Ok so we need to download the audit database file first.

# pkg audit -F
http://portaudit.FreeBSD.org/auditfile.tbz               100%   76KB  37.8KB/s  55.6KB/s   00:02
openssl-1.0.0_9 is vulnerable
OpenSSL -- CMS and S/MIME Bleichenbacher attack

WWW: http://portaudit.FreeBSD.org/60eb344e-6eb1-11e1-8ad7-00e0815b8da8.html


1 problem(s) in your installed packages found.

If your familiar with portaudit then this command will make you feel right at home.

Pkg 1.0 is out now! The betas and RCs are over, say hello to the first release!

For more information see the pkgng - FreeBSD Wiki.

You may also like...

  • Pingback: pkgng: First look at FreeBSD’s new package manager | FreeBSD News()

  • Bapt

    Very nice thank you, very well written. The only mistake is that for pkg version you don’t need a remote repository so you don’t need pkg update first.

    • Thanks for pointing that out, it seems that pkg version is no different to the pkg_version command. I have updated the post accordingly.

  • Bapt

    Very nice thank you, very well written. The only mistake is that for pkg version you don’t need a remote repository so you don’t need pkg update first.

    • Thanks for pointing that out, it seems that pkg version is no different to the pkg_version command. I have updated the post accordingly.

  • Warner Losh

    Very nice… does the chroot need to be one for the architecture of the hose? Or can one create an ARM chroot on an amd64 box and then use pkg to populate it with arm packages?

    • I believe pkgng should be able manage packages for any architecture. As it does not need to compile any of the packages, just extract and register them. You will need to set PACKAGESITE to the correct location for the required arch.

      How you actually run your ARM chroot on a amd64 machine is another mater, I suppose if you’re exporting the filesystem or running some kind of full ARM virtualization then it makes sense to do this.

    • Bapt

      technically the chroot can be any architecture: UNAME_m=arm pkg -c /mychroot ins -x nginx should work.

      Two problem might occur then: the script, pkgng will fail at executing arm scripts, and the users/groups management: currently add user is a script calling pw(8), we are switching to pw_utils/gr_utils but still pw_mkdb is executing pwd_mkdb which will fail on different arch, (I’m working on modifying our pw_mkdb function) and last it will depends on you endianness it will only work if the host and chroot both have the same endianness (because of pw.db)

  • Warner Losh

    Very nice… does the chroot need to be one for the architecture of the hose? Or can one create an ARM chroot on an amd64 box and then use pkg to populate it with arm packages?

    • I believe pkgng should be able manage packages for any architecture. As it does not need to compile any of the packages, just extract and register them. You will need to set PACKAGESITE to the correct location for the required arch.

      How you actually run your ARM chroot on a amd64 machine is another mater, I suppose if you’re exporting the filesystem or running some kind of full ARM virtualization then it makes sense to do this.

    • Bapt

      technically the chroot can be any architecture: UNAME_m=arm pkg -c /mychroot ins -x nginx should work.

      Two problem might occur then: the script, pkgng will fail at executing arm scripts, and the users/groups management: currently add user is a script calling pw(8), we are switching to pw_utils/gr_utils but still pw_mkdb is executing pwd_mkdb which will fail on different arch, (I’m working on modifying our pw_mkdb function) and last it will depends on you endianness it will only work if the host and chroot both have the same endianness (because of pw.db)

  • Guest

    What is the point of this? FreeBSD ports already manages the prereqs…

    • Dan McGee

      Not to burst your bubble, but FreeBSD ports and pkg_add are a complete disaster when anyone that has used a Linux distro in the last 10 years tries to jump in and use BSD. I’ve been turned off and simply given up multiple times because things feel archaic. It feels like something written in 1992 or earlier, FTP is extremely slow, there is no obvious way to search or query, etc.

      • Ciny Marcinko

        yes let’s build layers on top of layers of abstraction to obfuscate what’s in the core. You don’t just “jump in” to server operating systems. that’s how half-competent sysadmins come into existence. this “archaic” way actually forces you to dig deeper into the system and actually understand how it works – not just memorize a few commands. FreeBSD is a network operating system. it’s not meant for your desktop (PC-BSD is the other way), it’s not meant to be easy or user friendly…

        • Neal Clark

          looks like the tide is turning the other way

          • Andy Kosela

            Ciny,

            I think you got a very idyllic and yeah..archaic way of looking at “server operating systems”. FreeBSD is not considered a “server operating system” anymore in the industry. We are not back in 1999 anymore. I come from the Fortune 500 corporate environment where we have literally hundreds of boxes running mission cricital business applications like huge Oracle databases, SAP etc. When was the last time Oracle supported FreeBSD? :)

            FreeBSD is right now just a hobbyist system and in the industry mainly known now in the embedded market only (Juniper, Blue Coat etc.). It is not used or is used with minimal global impact as a “server operating system”. These are the cold facts. And I have been involved with FreeBSD since the ’90, so I’m not happy about that situation either, but it is what it is.

            I’m looking forward to pkgng(8) though. If it will be as good as NetBSD pkgin then I will be happy and bapt@ and imil@ are definetly sharing some ideas together.

  • Guest

    What is the point of this? FreeBSD ports already manages the prereqs…

    • Dan McGee

      Not to burst your bubble, but FreeBSD ports and pkg_add are a complete disaster when anyone that has used a Linux distro in the last 10 years tries to jump in and use BSD. I’ve been turned off and simply given up multiple times because things feel archaic. It feels like something written in 1992 or earlier, FTP is extremely slow, there is no obvious way to search or query, etc.

      • Ciny Kokociny

        yes let’s build layers on top of layers of abstraction to obfuscate what’s in the core. You don’t just “jump in” to server operating systems. that’s how half-competent sysadmins come into existence. this “archaic” way actually forces you to dig deeper into the system and actually understand how it works – not just memorize a few commands. FreeBSD is a network operating system. it’s not meant for your desktop (PC-BSD is the other way), it’s not meant to be easy or user friendly…

        • nclark

          looks like the tide is turning the other way

          • Andy Kosela

            Ciny,

            I think you got a very idyllic and yeah..archaic way of looking at “server operating systems”. FreeBSD is not considered a “server operating system” anymore in the industry. We are not back in 1999 anymore. I come from the Fortune 500 corporate environment where we have literally hundreds of boxes running mission cricital business applications like huge Oracle databases, SAP etc. When was the last time Oracle supported FreeBSD? :)

            FreeBSD is right now just a hobbyist system and in the industry mainly known now in the embedded market only (Juniper, Blue Coat etc.). It is not used or is used with minimal global impact as a “server operating system”. These are the cold facts. And I have been involved with FreeBSD since the ’90, so I’m not happy about that situation either, but it is what it is.

            I’m looking forward to pkgng(8) though. If it will be as good as NetBSD pkgin then I will be happy and bapt@ and imil@ are definetly sharing some ideas together.

  • Justin Yang

    Really promising and fascinating. I hope I can use pkgng right now!

  • Justin Yang

    Really promising and fascinating. I hope I can use pkgng right now!

  • Info

    This is excellent!

    I want to be able to configure multiple remote repos (a la git remotes) and explicitly specify which one to use. Can I do that?

  • Info

    This is excellent!

    I want to be able to configure multiple remote repos (a la git remotes) and explicitly specify which one to use. Can I do that?

  • Oliver Crow

    I read this article and I read the wiki page. It’s completely unclear to me what problem this solves that the pkg_* tools don’t solve. I’m not saying there aren’t problems … I know that there are. But part of the issue with the BSD package and ports system is that there are now so many tools available that it’s really easy to get confused about which tool to use when and which combinations of tools play nicely together.

    So … if you’re introducing a new tool, I think that the first things to explain to the community of users are: When should I use this tool? What problems does it solve? How is it better than the tools that it replaces? Which tools will work well in concert with it? What functionality to I need to still use other tools for, and which are the best options for those use cases.

    • Thanks Oliver, I think you have a good point, there does need to be more explanation as to why you would want to use this over the existing package tools or even the ports. I don’t think I can fit most of the main use cases for pkgng on this reply but I will outline why it is the missing tool in my toolbox.

      When managing ~300+ FreeBSD servers keeping all of the packages up to date and with the correct compile time options is a very big job and time consuming. This is why many people turn to some sort of separate build farm using Tinderbox or similar. Using it to build their own ports with the version, arch and compile time options they require for their many systems. The resulting packages are then distributed to or become the package source for the servers allowing upgrades etc. Pkgng can integrate with tinderbox. It is not only aware of the currently installed version of a package but also that of the remote package so you can compare and upgrade very easily with pkgng without maintaining an up to date ports tree on every server. There is no pkg_upgrade, pkg_search, pkg_which, etc in the base. I personally believe production servers should be as clean as possible and building, for example, apache directly from ports on a production server is not a good idea.

      This method currently possible by building your own port INDEX and hacking together a script which uses pkg_add and postmaster to do package only upgrades but it is very messy. If you try plugging package management into something like cfengine3 you will see why something like pkgng is required. So it benefits the power users as well as the users that are coming from linux that just want a more familiar package system.

      Ports are cool, creating packages from them makes them very portable and easy to manage. When you have many custom packages across many systems you need a good packaging system. For me pkgng is the missing piece to this puzzle and I hope to see pkng in the base system one day.

    • Canadian Human

      well … one problem it solves is that it is orders of magnitude faster:

      $ time pkg info -a
      pkg info -a 0.00s user 0.04s system 8% cpu 0.419 total

      $ time pkg_info
      pkg_info 0.41s user 0.30s system 3% cpu 20.460 total

  • Oliver Crow

    I read this article and I read the wiki page. It’s completely unclear to me what problem this solves that the pkg_* tools don’t solve. I’m not saying there aren’t problems … I know that there are. But part of the issue with the BSD package and ports system is that there are now so many tools available that it’s really easy to get confused about which tool to use when and which combinations of tools play nicely together.

    So … if you’re introducing a new tool, I think that the first things to explain to the community of users are: When should I use this tool? What problems does it solve? How is it better than the tools that it replaces? Which tools will work well in concert with it? What functionality to I need to still use other tools for, and which are the best options for those use cases.

    • Thanks Oliver, I think you have a good point, there does need to be more explanation as to why you would want to use this over the existing package tools or even the ports. I don’t think I can fit most of the main use cases for pkgng on this reply but I will outline why it is the missing tool in my toolbox.

      When managing ~300+ FreeBSD servers keeping all of the packages up to date and with the correct compile time options is a very big job and time consuming. This is why many people turn to some sort of separate build farm using Tinderbox or similar. Using it to build their own ports with the version, arch and compile time options they require for their many systems. The resulting packages are then distributed to or become the package source for the servers allowing upgrades etc. Pkgng can integrate with tinderbox. It is not only aware of the currently installed version of a package but also that of the remote package so you can compare and upgrade very easily with pkgng without maintaining an up to date ports tree on every server. There is no pkg_upgrade, pkg_search, pkg_which, etc in the base. I personally believe production servers should be as clean as possible and building, for example, apache directly from ports on a production server is not a good idea.

      This method currently possible by building your own port INDEX and hacking together a script which uses pkg_add and postmaster to do package only upgrades but it is very messy. If you try plugging package management into something like cfengine3 you will see why something like pkgng is required. So it benefits the power users as well as the users that are coming from linux that just want a more familiar package system.

      Ports are cool, creating packages from them makes them very portable and easy to manage. When you have many custom packages across many systems you need a good packaging system. For me pkgng is the missing piece to this puzzle and I hope to see pkng in the base system one day.

    • Canadian Human

      well … one problem it solves is that it is orders of magnitude faster:

      $ time pkg info -a
      pkg info -a 0.00s user 0.04s system 8% cpu 0.419 total

      $ time pkg_info
      pkg_info 0.41s user 0.30s system 3% cpu 20.460 total

  • BSDr

    There’s been other comparisons of freebsd ports/packages with ubuntu/fedora repos that suggest pkgng is going to close a gap or missing functionality in FreeBSD (or BSD in general I suppose since it’s portable). But what gap? Using ports (and portmaster! excellent!!) is a bit easier and slicker than using srpms with fedora/centos but on the binary pacakge front the problem for FreeBSD is the infrequency of builds/updates between OS releases. pkgng will make it easier (but does it make massive sysadmin, tracking, ITIL compliance things easier who knows) and faster to manage binary packages (e.g. the ability to remotely “server” a repos like solaris pkg does) but it is not going to make FreeBSD suddenly like ubuntu or fedora. For something closer to that that stick with PC-BSD …

    BTW any plans for pkgng and PC-BSD to play nicely together?

    • OSX != BSD

      Well with pkgng it is *much* easier to manage a host for building binaries from source so this is going to make a difference. Go ahead and use portmaster … I agree it’s a great tool for making binaries from the ports system and will still be needed/useful for handling all the dependencies that go into the more frequent package creation we’ll be doing on our machines ;-) You can create and export a repo of well tested “gold” versions of your working binaries (haha) e.g.:

      pkg repo ~/public_html/repo-`date “+%Y-%m-%d”` /usr/local/etc/pkg/pkg_repo-private.key
      pkg create -a -o ~/public_html/repo-`date “+%Y-%m-%d”`
      ln -s ~/public_html/repo-`date “+%Y-%m-%d”` ~/public_html/repo-LATEST

      (or something like that)

      and install them madly everywhere with puppet/salt/cfengine or whatever. Of course for end user “FreeBSD as Unix workstation” installs that use freebsd.org and the internet instead of local IT staff for support things might not become “like ubuntu” overnight. In fact we could end up like fedora/centos based linux distros which have crazy numbers of alternate and bleeding edge repos (with combinations of kernel mods the repo names added to the package filename etc etc argh!). That and usually one or more golden (“certified?”) set of signed binaries/mirrors thereof and users usually can figure things out. In any case pkgng is a key piece of making it easier to have more frequent or even continuous binary package updates (either from freebsd.org or some other service) per release/arch and during the whole life of each release instead of the one shot way things are done now. I can see binary installs still being hard to manage – it’s the nature of the beast with all the deps disappearing inside the packages experienced sysadmins still will prefer source builds for a lot of things. But will pkgng be the default for 10R ? I hope so.

      I’m wondering if there would ever be a need for a “pkgsnap” tool that could deal with binary diffs or snapshots of packages from a repo (like portsnap and freebsd-update do) will come along for keeping a whole bunch of binary repos synced.

  • BSDr

    There’s been other comparisons of freebsd ports/packages with ubuntu/fedora repos that suggest pkgng is going to close a gap or missing functionality in FreeBSD (or BSD in general I suppose since it’s portable). But what gap? Using ports (and portmaster! excellent!!) is a bit easier and slicker than using srpms with fedora/centos but on the binary pacakge front the problem for FreeBSD is the infrequency of builds/updates between OS releases. pkgng will make it easier (but does it make massive sysadmin, tracking, ITIL compliance things easier who knows) and faster to manage binary packages (e.g. the ability to remotely “server” a repos like solaris pkg does) but it is not going to make FreeBSD suddenly like ubuntu or fedora. For something closer to that that stick with PC-BSD …

    BTW any plans for pkgng and PC-BSD to play nicely together?

    • OSX != BSD

      Well with pkgng it is *much* easier to manage a host for building binaries from source so this is going to make a difference. Go ahead and use portmaster … I agree it’s a great tool for making binaries from the ports system and will still be needed/useful for handling all the dependencies that go into the more frequent package creation we’ll be doing on our machines ;-) You can create and export a repo of well tested “gold” versions of your working binaries (haha) e.g.:

      pkg repo ~/public_html/repo-`date “+%Y-%m-%d”` /usr/local/etc/pkg/pkg_repo-private.key
      pkg create -a -o ~/public_html/repo-`date “+%Y-%m-%d”`
      ln -s ~/public_html/repo-`date “+%Y-%m-%d”` ~/public_html/repo-LATEST

      (or something like that)

      and install them madly everywhere with puppet/salt/cfengine or whatever. Of course for end user “FreeBSD as Unix workstation” installs that use freebsd.org and the internet instead of local IT staff for support things might not become “like ubuntu” overnight. In fact we could end up like fedora/centos based linux distros which have crazy numbers of alternate and bleeding edge repos (with combinations of kernel mods the repo names added to the package filename etc etc argh!). That and usually one or more golden (“certified?”) set of signed binaries/mirrors thereof and users usually can figure things out. In any case pkgng is a key piece of making it easier to have more frequent or even continuous binary package updates (either from freebsd.org or some other service) per release/arch and during the whole life of each release instead of the one shot way things are done now. I can see binary installs still being hard to manage – it’s the nature of the beast with all the deps disappearing inside the packages experienced sysadmins still will prefer source builds for a lot of things. But will pkgng be the default for 10R ? I hope so.

      I’m wondering if there would ever be a need for a “pkgsnap” tool that could deal with binary diffs or snapshots of packages from a repo (like portsnap and freebsd-update do) will come along for keeping a whole bunch of binary repos synced.

  • Jesus

    That’s the most urgent thing that freebsd needs. A decent package manager.

  • Jesus

    That’s the most urgent thing that freebsd needs. A decent package manager.

    • Florian Heigl

      pity it didnt work out.
      it’s a package manager, but definitely not a decent one.

  • Peter

    I’m a big FreeBSD fan since a long time, but I have to admit that It’s not possible these days to maintain a bunch of servers using ports or doing a make buildworld.
    FreeBSD is a great OS, with great features but will die if FreeBSD developers don’t change these methods. We need something simple like Debian “apt-get” for installing software and maintain the system up to date! Can you imagine how many users FreeBSD has already lost because of this???? Please, you have to pass this message.

  • Peter

    I’m a big FreeBSD fan since a long time, but I have to admit that It’s not possible these days to maintain a bunch of servers using ports or doing a make buildworld.
    FreeBSD is a great OS, with great features but will die if FreeBSD developers don’t change these methods. We need something simple like Debian “apt-get” for installing software and maintain the system up to date! Can you imagine how many users FreeBSD has already lost because of this???? Please, you have to pass this message.

  • Tim

    We need something like “apt-get” from Debian for installing software and update the system. End of the story.
    Every day it passes FreeBSD looses more and more users because of this!

    • Have you had chance to try out pkgng yet, I feel it comes pretty close to “apt-get”.

      Have you noticed any features in “apt-get” that are currently missing from pkgng? The development seems to be in full swing at the moment, so now is as good a time to put some suggestions forward.

    • zhushazang

      WTF!! Who cares about dummy-lummy-debian-ubuntu users? Don’t say nonsenseless man. We must keep this kind of users far, far, faaaaaaaaar away from FreeBSD Community.

  • Tim

    We need something like “apt-get” from Debian for installing software and update the system. End of the story.
    Every day it passes FreeBSD looses more and more users because of this!

    • Have you had chance to try out pkgng yet, I feel it comes pretty close to “apt-get”.

      Have you noticed any features in “apt-get” that are currently missing from pkgng? The development seems to be in full swing at the moment, so now is as good a time to put some suggestions forward.

    • zhushazang

      WTF!! Who cares about dummy-lummy-debian-ubuntu users? Don’t say nonsenseless man. We must keep this kind of users far, far, faaaaaaaaar away from FreeBSD Community.

  • Pingback: New to FreeBSD and I have several questions.()

  • Pingback: pkg 1.0: FreeBSD's new package manager released()

  • Pingback: Pkgng and Freebsd 9 - VirtuallyHyper()

  • Alex Storn

    I Like this, MAC OS X Support ?

    but only freebsd bad idea :(

    i need this on linux, darwin, bsd

  • Alex Storn

    I Like this, MAC OS X Support ?

    but only freebsd bad idea :(

    i need this on linux, darwin, bsd

  • Alex Storn

    But pkgng+SQLite dabase very slow work on ZFS Comress SSD+trim :)

  • Alex Storn

    But pkgng+SQLite dabase very slow work on ZFS Comress SSD+trim :)

  • Pingback: FreeBSD and pkgng | In Nomine – The Lotus Land()

  • noname

    Anyway… I don’t think ubuntu users would ever use apt-get, they use the ubuntu software center, and ubuntu “power” users use synaptic. (OMG! Synaptic! Linusk guru!). As for the Debian lusers, they use aptitude whenever they use a CLI tool.

    My two cents.

    • Dan

      Debian users use apt-get and Ubuntu users are mostly beginners. What’s wrong with that ?

  • Alex Storn

    need update list

    pkg version -v | grep “needs updating”

  • Pingback: it | Pearltrees()

  • Pingback: linux-bsd | Pearltrees()

  • rfduarte

    little envy….

  • Francisco Reyes

    For anyone bumping into this old thread.

    Someone must have forgotten to tell Netflix that FreeBSD is not a serious “corporate” OS as they plan to build their CDN on it. https://signup.netflix.com/openconnect/software

    They are going to move most of the CDN serving from the amazon cloud to their own and it will be powered by FreeBSD.

    • batcaverna

      Someone must also have forgotten to tell Apple about that, since OSX is based on FreeBSD.

  • M.S. Babaei

    Thanks for the full-feature article. I found it very useful.

  • andrey

    “Who cares about users” seems to be the motto of FreeBSD Community.

    # ls –help
    ls: illegal option — e
    usage: ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwx1] [-D format] [file …]

    Who that is for? Why was that done like that? What did developers expect from the user to do next?

    # help
    help: not found

    • Florian Heigl

      Well it’s generally considered helpful to have the most basic understanding of unix.
      something like –help is not helping you.
      It seems to, but all it does is break convention and keep people helpless when dealing with multiple unix flavours. In the end they’ll never grasp the basic ideas fully and end up doing bullshit like a half dozen init replacements to be put in place of the worst clone of init ever made.

      To answer:
      — non-posix arguments were not expected at all by anyone.
      The question, for someone using many more Unix-derivated OS is:
      What were the GNU devs thinking adding a syntax that will leave users confused and trapped on any non-GNU toolset.

      To others, this is irresponsible design and they try to avoid it. The sad fact of life is that Ubuntu has dumbed down people so much they need to reinstall from VirtualBox to KVM to test the Ansible playbook that installs their Apache node.

      At that point some people as closed-minded as the OP might not see the why (bugs plus people not grasping Unix due to foam-wrapped Ubuntu) but only the result (too stupid – should stay away from FreeBSD)

  • FreeBSD

    looks good.

  • Dan

    Hey dud, I tried to put FreeBSD on some server and all I got was errors over errors, trying to install a LAMP with pkg_add. If you try doing that with apt, yum or pacman, it will just work. You won’t have to spend hours over hours just to make some apache and mail server work properly, cause of dependency issues, outdated packages and simply said bugs of all sizes and colors. A package manager, well written can turn FreeBSD into something usable again. Your attitude is that of a 5 year old who’s happy he has a toy that nobody wants. FreeBSD at this moment, is a mess to handle. A mess and this move is the best they made in several years. Speed of management and stability, support means lot more than satisfying some pumped up egos like yours. If FreeBSD wants to stay in the market and be relevant, it needs to update. Update it’s attitude and get more users, as with those like you, simply I don;t think will succeed on the long run. People use more *nix these days, than before, because of Debian, Ubuntu, Fedora, Arch and many others, but FreeBSD, which is like a relic these days, stuck with users like you, who think they’re smart cause they use something that;s simply outdated for these days. You should say thank you, to Debian, Ubuntu, Red Hat and others for making the *nix world a better and better known place, instead of acting like a monkey with a borrowed banana.

  • Dan

    Dude, whoever wants to understand how the system works, it can go through that. But mostly, people WANT THE JOB DONE. HAVE idea what that is ? AND IN TIME ! How about that ? How many car drivers know how a diesel engine works ? Hmm…quite few, I suppose. But if they feel curious, they can always brake their engine into pieces and see what’s inside. Or, better buy some books or watch some movies. Same can apply here…. this looks like a community of lazy developers.

  • Martin

    How pkgng solves a problem with specific package options? Does that mean that there is the only version/variant of the package and if it does not suite to me I need to deliver it from ports. How to achieve ‘pkg updgrade’ would not try to upgrade this package?

  • lopez

    pkgng is great! As a long time FreeBSD admin, managing many hosts and particularly jails, I highly welcome the new streamlined pkgng. There’s ONE only thing I miss, though – but this does not seem to be a structural problem, more a future feature: some thing like the ‘cascading’ pkg sources in dpkg. In pkgng talk, it would be nice to be able to specify more than one repo, in a particular order. This would make it easier to have some ports with custom build options, while keeping all others from the official FreeBSD repo. Kudos!

  • Pingback: Unix:Install a package and all its dependencies without a confirmation prompt with FreeBSD pkg – Unix Questions()