Chapter 31. Crosscompiling NetBSD with

Table of Contents

31.1. Building the crosscompiler
31.2. Configuring the kernel manually
31.3. Crosscompiling the kernel manually
31.4. Crosscompiling the kernel with
31.5. Crosscompiling the userland
31.6. Crosscompiling the X Window System
31.7. Changing build behaviour
31.7.1. Changing the Destination Directory
31.7.2. Static Builds
31.7.3. Using options
31.7.4. make(1) variables used during build

When targeting a product for an embedded platform, it's not feasible to have all the development tools available on that same platform. Instead, some method of crosscompiling is usually used today. NetBSD comes with a framework to build both the operating system's kernel and the whole userland for either the same platform that the compiler runs on, or for a different platform, using crosscompiling. Crosscompiling requires assembler, linker, compiler etc. to be available and built for the target platform. The new build scheme will take care of creating these tools for a given platform, and make them available ready to use to do development work.

In this chapter, we will show how to use to first create a crosscompiling toolchain, including cross-compiler, cross-assembler, cross-linker and so on. While native kernel builds are covered in Chapter 32, Compiling the kernel, these tools are then used to manually configure and crosscompile a kernel for a different platform, and then show how to use as a convenient alternative. After that works, the whole NetBSD userland will be compiled and packed up in the format of a NetBSD release. In the examples, we will use the Sun UltraSPARC ("sparc64") 64-bit platform as target platform, any other platform supported by NetBSD can be targetted as well specifying its name (see /usr/src/sys/arch).

Before starting, take note that it is assumed that the NetBSD sources are available in /usr/src, as described in Chapter 30, Obtaining the sources.

A more detailed description of the framework can be found in Luke Mewburn and Matthew Green's paper and their presentation from BSDCon 2003 as well as in /usr/src/BUILDING.

31.1. Building the crosscompiler

The first step to do cross-development is to get all the necessary tools available. In NetBSD terminology, this is called the "toolchain", and it includes BSD-compatible make(1), C/C++ compilers, linker, assembler, config(8), as well as a fair number of tools that are only required when crosscompiling a full NetBSD release, which we won't cover here.

The command to create the crosscompiler is quite simple, using NetBSD's new src/ script. Please note that all the commands here can be run as normal (non-root) user:

$ cd /usr/src
$ ./ -m sparc64 tools

Make sure that the directory /usr/obj does exist, or add a "-O" option to the call, redirecting the object directory someplace else.

If the tools have been built previously and they only need updated, then the update option "-u" can be used to only rebuild tools that have changed:

$ ./ -u -m sparc64 tools

When the tools are built, information about them and several environment variables is printed out:

===> started: Thu Dec  2 22:18:11 CET 2007
===> ended:   Thu Dec  2 22:28:22 CET 2007
===> Summary of results: command: ./ -m sparc64 tools started: Thu Dec  2 22:18:11 CET 2007
         No nonexistent/bin/nbmake, needs building.
         Bootstrapping nbmake
         MACHINE:          sparc64
         MACHINE_ARCH:     sparc64
         TOOLDIR path:     /usr/src/tooldir.NetBSD-4.0-i386
         DESTDIR path:     /usr/src/destdir.sparc64
         RELEASEDIR path:  /usr/src/releasedir
         Created /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake
         makewrapper:      /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
         Updated /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
         Tools built to /usr/src/tooldir.NetBSD-4.0-i386 started: Thu Dec  2 22:18:11 CET 2007 ended:   Thu Dec  2 22:28:22 CET 2007
===> . 

During the build, object directories are used consistently, i.e. special directories are kept that keep the platform-specific object files and compile results. In our example, they will be kept in directories named "obj.sparc64" as we build for UltraSPARC as target platform.

The toolchain itself is part of this, but as it's hosted and compiled for a i386 system, it will get placed in its own directory indicating where to cross-build from. Here's where our crosscompiler tools are located:

$ pwd
$ ls -d tooldir.*

So the general rule of thumb is for a given "host" and "target" system combination, the crosscompiler will be placed in the "src/" directory by default. A full list of all tools created for crosscompiling the whole NetBSD operating system includes:

$ ls tooldir.NetBSD-4.0-i386/bin/
nbasn1_compile          nbmakefs                nbzic
nbcap_mkdb              nbmakeinfo              sparc64--netbsd-addr2li
nbcat                   nbmakewhatis            sparc64--netbsd-ar
nbcksum                 nbmenuc                 sparc64--netbsd-as
nbcompile_et            nbmkcsmapper            sparc64--netbsd-c++
nbconfig                nbmkdep                 sparc64--netbsd-c++filt
nbcrunchgen             nbmkesdb                sparc64--netbsd-cpp
nbctags                 nbmklocale              sparc64--netbsd-dbsym
nbdb                    nbmknod                 sparc64--netbsd-g++
nbeqn                   nbmktemp                sparc64--netbsd-g77
nbfgen                  nbmsgc                  sparc64--netbsd-gcc
nbfile                  nbmtree                 sparc64--netbsd-gcc-3.3
nbgencat                nbnroff                 sparc64--netbsd-gccbug
nbgroff                 nbpax                   sparc64--netbsd-gcov
nbhexdump               nbpic                   sparc64--netbsd-ld
nbhost-mkdep            nbpwd_mkdb              sparc64--netbsd-lint
nbindxbib               nbrefer                 sparc64--netbsd-mdsetim
nbinfo                  nbrpcgen                sparc64--netbsd-nm
nbinfokey               nbsoelim                sparc64--netbsd-objcopy
nbinstall               nbstat                  sparc64--netbsd-objdump
nbinstall-info          nbsunlabel              sparc64--netbsd-ranlib
nbinstallboot           nbtbl                   sparc64--netbsd-readelf
nblex                   nbtexi2dvi              sparc64--netbsd-size
nblorder                nbtexindex              sparc64--netbsd-strings
nbm4                    nbtsort                 sparc64--netbsd-strip
nbmake                  nbuudecode
nbmake-sparc64          nbyacc 

As you can see, most of the tools that are available native on NetBSD are present with some program prefix to identify the target platform for tools that are specific to a certain target platform.

One important tool that should be pointed out here is "nbmake-sparc64". This is a shell wrapper for a BSD compatible make(1) command that's setup to use all the right commands from the crosscompiler toolchain. Using this wrapper instead of /usr/bin/make allows crosscompiling programs that were written using the NetBSD Makefile infrastructure (see src/share/mk). We will use this make(1) wrapper in a second to cross compile the kernel!

31.2. Configuring the kernel manually

Now that we have a working crosscompiler available, the "usual" steps for building a kernel are needed - create a kernel config file, run config(1), then build. As the config(1) program used to create header files and Makefile for a kernel build is platform specific, we need to use the "nbconfig" program that's part of our new toolchain. That aside, the procedure is just as like compiling a "native" NetBSD kernel. Commands involved here are:

$ cd /usr/src/sys/arch/sparc64/conf
$ /usr/src/tooldir.NetBSD-4.0-i386/bin/nbconfig MYKERNEL

That's all. This command has created a directory ../compile/MYKERNEL with a number of header files defining information about devices to compile into the kernel, a Makefile that is setup to build all the needed files for the kernel, and link them together.

31.3. Crosscompiling the kernel manually

We have all the files and tools available to crosscompile our UltraSPARC-based kernel from our Intel-based host system, so let's get to it! After changing in the directory created in the previous step, we need to use the crosscompiler toolchain's nbmake-sparc64 shell wrapper, which just calls make(1) with all the necessary settings for crosscompiling for a sparc64 platform:

$ cd ../compile/MYKERNEL/
$ /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64 depend
$ /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64

This will churn away a bit, then spit out a kernel:

   text    data     bss     dec     hex filename
5016899  163728  628752 5809379  58a4e3 netbsd
$ ls -l netbsd
-rwxr-xr-x  1 feyrer  666  5874663 Dec  2 23:17 netbsd
$ file netbsd
netbsd: ELF 64-bit MSB executable, SPARC V9, version 1 (SYSV), statically linked, not stripped 

Now the kernel in the file netbsd can either be transferred to a UltraSPARC machine (via NFS, FTP, scp, etc.) and booted from a possible harddisk, or directly from our cross-development machine using NFS.

After configuring and crosscompiling the kernel, the next logical step is to crosscompile the whole system, and bring it into a distribution-ready format. Before doing so, an alternative approach to crosscompiling a kernel will be shown in the next section, using the script to do configuration and crosscompilation of the kernel in one step.

31.4. Crosscompiling the kernel with

A cross compiled kernel can be done manually as described in the previous sections, or by the easier method of using, which will be shown here.

Preparation of the kernel config file is the same as described above:

$ cd /usr/src/sys/arch/sparc64/conf

Then edit MYKERNEL and once finished, all that needs to be done is to use to build the kernel (it will also configure it, running the steps shown above):

$ cd /usr/src
$ ./ -u -m sparc64 kernel=MYKERNEL

Notice that update ("-u") was specified, the tools are already built, there is no reason to rebuild all of the tools. Once the kernel is built, will print out the location of it along with other information:

===> Summary of results: command: ./ -u -m sparc64 kernel=MYKERNEL started: Thu Dec  2 23:30:02 CET 2007
         No nonexistent/bin/nbmake, needs building.
         Bootstrapping nbmake
         MACHINE:          sparc64
         MACHINE_ARCH:     sparc64
         TOOLDIR path:     /usr/src/tooldir.NetBSD-4.0-i386
         DESTDIR path:     /usr/src/destdir.sparc64
         RELEASEDIR path:  /usr/src/releasedir
         Created /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake
         makewrapper:      /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
         Updated /usr/src/tooldir.NetBSD-4.0-i386/bin/nbmake-sparc64
         Building kernel without building new tools
         Building kernel:  MYKERNEL
         Build directory:  /usr/src/sys/arch/sparc64/compile/obj.sparc64/GENERIC
         Kernels built from MYKERNEL:
          /usr/src/sys/arch/sparc64/compile/obj.sparc64/MYKERNEL/netbsd started: Thu Dec  2 23:30:02 CET 2007 ended:   Thu Dec  2 23:38:22 CET 2007
===> . 

The path to the kernel built is of interest here: /usr/src/sys/arch/sparc64/compile/obj.sparc64/MYKERNEL/netbsd, it can be used the same way as described above.

31.5. Crosscompiling the userland

By now it is probably becoming clear that the toolchain actually works in stages. First the crosscompiler is built, then a kernel. Since will attempt to rebuild the tools at every invocation, using update saves time. It is probably also clear that outside of a few options, the semantics are basically command. So, it stands to reason that building the whole userland and/or a release is a matter of using the right commands.

It should be no surprise that building and creating a release would look like the following:

$ ./ -U -u -m sparc64 release

These commands will compile the full NetBSD userland and put it into a destination directory, and then build a release from it in a release directory. The -U switch is added here for an unprivileged build, i.e. one that's running as normal user and not as root. As no further switches to were given nor any environment variables were set, the defaults of DESTDIR=/usr/src/destdir.sparc64 and RELEASEDIR=/usr/src/releasedir are used, as shown in the above.

31.6. Crosscompiling the X Window System

The NetBSD project has its own copy of the X Window System's which contains changes to make X going on as many of the platforms supported by NetBSD as possible. Due to this, it is desirable to use the X Window System version available from and for NetBSD, which can also be crosscompiled much like the kernel and base system. To do so, the "xsrc" sources must be checked out from CVS into /usr/xsrc just as "src" and "pkgsrc" were as described in Chapter 30, Obtaining the sources.

After this, X can be crosscompiled for the target platform by adding the -x switch to, e.g. when creating a full release:

$ ./ -U -x -u -m sparc64 release

The -U flag for doing unprivileged (non-root) builds and the -u flag for not removing old files before building as well as the -m arch option to define the target architecture have already been introduced, and the -x option to also (cross)compile "xsrc" is another option.

31.7. Changing build behaviour

Similar to the old, manual building method, the new toolchain has a lot of variables that can be used to direct things like where certain files go, what (if any) tools are used and so on. A look in src/BUILDING covers most of them. In this section some examples of changing default settings are given, each following its own ways.

31.7.1. Changing the Destination Directory

Many people like to track NetBSD-current and perform cross compiles of architectures that they use. The logic for this is simple, sometimes a new feature or device becomes available and someone may wish to use it. By keeping track of changes and building every now and again, one can be assured that these architectures can build their own release.

It is reasonable to assume that if one is tracking and building for more than one architecture, they might want to keep the builds in a different location than the default. There are two ways to go about this, either use a script to set the new DESTDIR, or simply do so interactively. In any case, it can be set the same way as any other variable (depending on your shell of course).

For bash, the Bourne or Korn shell, this is:

$ export DESTDIR=/usr/builds/sparc64

For tcsh and the C shell, the command is:

$ setenv DESTDIR /usr/builds/sparc64

Simple enough. When the build is run, the binaries and files will be sent to /usr/builds.

31.7.2. Static Builds

The NetBSD toolchain builds and links against shared libraries by default. Many users still prefer to be able to link statically. Sometimes a small system can be created without having shared libraries, which is a good example of doing a full static build. If a particular build machine will always need one environment variable set in a particular way, then it is easiest to simply add the changed setting to /etc/mk.conf.

To make sure a build box always builds statically, simply add the following line to /etc/mk.conf:


31.7.3. Using options

Besides variables in environment and /etc/mk.conf, the build process can be influenced by a number of switches to the script itself, as we have already seen when forcing unprivileged (non-root) builds, selecting the target architecture or preventing deletion of old files before the build. All these options can be listed by running -h:

$ cd /usr/src
$ -h
Usage: [-EnorUux] [-a arch] [-B buildid] [-D dest] [-j njob]
		[-M obj] [-m mach] [-N noisy] [-O obj] [-R release] [-T tools]
		[-V var=[value]] [-w wrapper] [-X x11src] [-Z var]
		operation [...]

 Build operations (all imply "obj" and "tools"):
    build               Run "make build".
    distribution        Run "make distribution" (includes DESTDIR/etc/ files).
    release             Run "make release" (includes kernels and distrib media).

 Other operations:
    help                Show this message and exit.
    makewrapper         Create nbmake-${MACHINE} wrapper and nbmake.
                        Always performed.
    obj                 Run "make obj".  [Default unless -o is used]
    tools               Build and install tools.
    install=idir        Run "make installworld" to `idir' to install all sets
			except `etc'.  Useful after "distribution" or "release"
    kernel=conf         Build kernel with config file `conf'
    releasekernel=conf  Install kernel built by kernel=conf to RELEASEDIR.
    sets                Create binary sets in RELEASEDIR/MACHINE/binary/sets.
			DESTDIR should be populated beforehand.
    sourcesets          Create source sets in RELEASEDIR/source/sets.
    params              Display various make(1) parameters.

    -a arch     Set MACHINE_ARCH to arch.  [Default: deduced from MACHINE]
    -B buildId  Set BUILDID to buildId.
    -D dest     Set DESTDIR to dest.  [Default: destdir.MACHINE]
    -E          Set "expert" mode; disables various safety checks.
                Should not be used without expert knowledge of the build system.
    -j njob     Run up to njob jobs in parallel; see make(1) -j.
    -M obj      Set obj root directory to obj; sets MAKEOBJDIRPREFIX.
                Unsets MAKEOBJDIR.
    -m mach     Set MACHINE to mach; not required if NetBSD native.
    -N noisy	Set the noisyness (MAKEVERBOSE) level of the build:
		    0	Quiet
		    1	Operations are described, commands are suppressed
		    2	Full output
		[Default: 2]
    -n          Show commands that would be executed, but do not execute them.
    -O obj      Set obj root directory to obj; sets a MAKEOBJDIR pattern.
                Unsets MAKEOBJDIRPREFIX.
    -o          Set MKOBJDIRS=no; do not create objdirs at start of build.
    -R release  Set RELEASEDIR to release.  [Default: releasedir]
    -r          Remove contents of TOOLDIR and DESTDIR before building.
    -T tools    Set TOOLDIR to tools.  If unset, and TOOLDIR is not set in
                the environment, nbmake will be (re)built unconditionally.
    -U          Set MKUNPRIVED=yes; build without requiring root privileges,
    		install from an UNPRIVED build with proper file permissions.
    -u          Set MKUPDATE=yes; do not run "make clean" first.
		Without this, everything is rebuilt, including the tools.
    -V v=[val]  Set variable `v' to `val'.
    -w wrapper  Create nbmake script as wrapper.
                [Default: ${TOOLDIR}/bin/nbmake-${MACHINE}]
    -X x11src   Set X11SRCDIR to x11src.  [Default: /usr/xsrc]
    -x          Set MKX11=yes; build X11 from X11SRCDIR
    -Z v        Unset ("zap") variable `v'. 

As can be seen, a number of switches can be set to change the standard build behaviour. A number of them has already been introduced, others can be set as appropriate.

31.7.4. make(1) variables used during build

Several variables control the behaviour of NetBSD builds. Unless otherwise specified, these variables may be set in either the process environment or in the make(1) configuration file specified by MAKECONF. For a definitive list of these options, see BUILDING and share/mk/bsd.README files in the toplevel source directory.


Identifier for the build. The identifier will be appended to object directory names, and can be consulted in the make(1) configuration file in order to set additional build parameters, such as compiler flags.


Directory to contain the built NetBSD system. If set, special options are passed to the compilation tools to prevent their default use of the host system's /usr/include, /usr/lib, and so forth. This pathname should not end with a slash (/) character (For installation into the system's root directory, set DESTDIR to an empty string). The directory must reside on a filesystem which supports long filenames and hard links.

Defaults to an empty string if USETOOLS is yes; unset otherwise. Note: will provide a default (destdir.MACHINE in the top-level .OBJDIR) unless run in expert mode.


If defined by the user, points to the root of an external toolchain (e.g. /usr/local/gnu). This enables the cross-build framework even when default toolchain is not available (see TOOLCHAIN_MISSING below).

Default: Unset


The verbosity of build messages. Supported values:

0 No descriptive messages are shown.
1 Descriptive messages are shown.
2 Descriptive messages are shown (prefixed with a '#') and command output is not suppressed.

Default: 2


Can be set to yes or no. Indicates whether preformatted plaintext manual pages will be created during a build.

Default: yes


Can be set to yes or no. Indicates whether cryptographic code will be included in a build; provided for the benefit of countries that do not allow strong cryptography. Will not affect the standard low-security password encryption system, crypt(3).

Default: yes


Can be set to yes or no. Indicates whether system documentation destined for DESTDIR/usr/share/doc will be installed during a build.

Default: yes


Can be set to yes or no. If set to yes, then for programs intended to be run on the compile host, the name, release and architecture of the host operating system will be suffixed to the name of the object directory created by make obj. This allows for multiple host systems to compile NetBSD for a single target. If set to no, then programs built to be run on the compile host will use the same object directory names as programs built to be run on the target.

Default: no


Can be set to yes or no. Indicates whether GNU info files, used for the documentation of most of the compilation tools, will be created and installed during a build.

Default: yes


Can be set to yes or no. Indicates whether lint(1) will be run against portions of the NetBSD source code during the build, and whether lint libraries will be installed into DESTDIR/usr/libdata/lint

Default: yes


Can be set to yes or no. Indicates whether manual pages will be installed during a build.

Default: yes


Can be set to yes or no. Indicates whether Native Language System locale zone files will be compiled and installed during a build.

Default: yes


Can be set to yes or no. Indicates whether object directories will be created when running make obj. If set to no, then all built files will be located inside the regular source tree.

Default: yes


Can be set to yes or no. Indicates whether shared objects and libraries will be created and installed during a build. If set to no, the entire build will be statically linked.

Default: Platform dependent. As of this writing, all platforms except sh3 default to yes


Can be set to yes or no. Indicates whether the ar(1) format libraries (lib*_pic.a), used to generate shared libraries, are installed during a build.

Default: yes


Can be set to yes or no. Indicates whether profiled libraries (lib*_p.a) will be built and installed during a build.

Default: yes; however, some platforms turn off MKPROFILE by default at times due to toolchain problems with profiled code.


Can be set to yes or no. Indicates whether files destined to reside in DESTDIR/usr/share will be built and installed during a build. If set to no, then all of MKCATPAGES, MKDOC, MKINFO, MKMAN and MKNLS will be set to no unconditionally.

Default: yes


Can be set to yes or no. For X builds, decides if the TrueType bytecode interpreter is turned on. See for details.

Default: no


Can be set to yes or no. Indicates whether an unprivileged install will occur. The user, group, permissions and file flags will not be set on the installed items; instead the information will be appended to a file called METALOG in DESTDIR. The contents of METALOG are used during the generation of the distribution tar files to ensure that the appropriate file ownership is stored.

Default: no


Can be set to yes or no. Indicates whether all install operations intended to write to DESTDIR will compare file timestamps before installing, and skip the install phase if the destination files are up-to-date. This also has implications on full builds (See below).

Default: no


Can be set to yes or no. Indicates whether X11 is built from X11SRCDIR.

Default: yes


Directory to hold the host tools, once built. This directory should be unique to a given host system and NetBSD source tree. (However, multiple targets may share the same TOOLDIR; the target-dependent files have unique names). If unset, a default based on the uname(1) information of the host platform will be created in the .OBJDIR of src.

Default: Unset.


Indicates whether the tools specified by TOOLDIR should be used as part of a build in progress. Must be set to yes if cross-compiling.

yes Use the tools from TOOLDIR.
no Do not use the tools from TOOLNAME, but refuse to build native compilation tool components that are version-specific for that tool.
never Do not use the tools from TOOLNAME, even when building native tool components. This is similar to the traditional NetBSD build method, but does not verify that the compilation tools in use are up-to-date enough in order to build the tree successfully. This may cause build or runtime problems when building the whole NetBSD source tree.

Default: yes if building all or part of a whole NetBSD source tree (detected automatically); no otherwise (to preserve traditional semantics of the bsd.*.mk make(1) include files).


Directory containing the X11 source.

Default: usr/xsrc

The following variables only affect the top level Makefile and do not affect manually building subtrees of the NetBSD source code.


Location for the make installworld target to install to.

Default: /


Can be set to yes or no. Indicates whether object directories will be created automatically (via a make obj pass) at the start of a build.

Default: no


Can be set to yes or no. If set, then addition to the effects described for MKUPDATE=yes above, this implies the effect of NOCLEANDIR (i.e., make cleandir is avoided).

Default: no


If set, avoids the make cleandir phase of a full build. This has the effect of allowing only changed files in a source tree to recompiled. This can speed up builds when updating only a few files in the tree.

Default: Unset


If set, avoids the make distrib-dirs of a full build. This skips running mtree(8) on DESTDIR, useful on systems where building as an unprivileged user, or where it is known that the system wide mtree files have not changed.

Default: Unset


If set, avoids the make includes phase of a full build. This has the effect of preventing make(1) from thinking that some programs are out-of-date simply because system include files have changed. However, this option should not be trusted when updating the entire NetBSD source tree arbitrarily; it is suggested to use MKUPDATE=yes in that case.

Default: Unset


If set, specifies the directory to which a release(7) layout will be written at the end of a make release.

Default: Unset


Set to yes on platforms for which there is no working in-tree toolchain, or if you need/wish using native system toolchain (i.e. non-cross tools available via your shell search path).

Default: depends on target platform; on platforms with in-tree toolchain is set to no.