O

1
OReilly Open Source Con October 19, 2005
FreeBSD Release Engineering Murray
Stokely murray_at_FreeBSD.org
2
Outline

• Introduction
• FreeBSD in a Nutshell
• FreeBSD Development Model
• Source Code Revision Control
• Project Management / Hats
• Release Process Overview
• Code Review
• Release Checklist
• CVS Operations
• Release Building

3
Outline

• Extensibility
• FreeBSD 6.0 and beyond
• Summer of Code - FreeSBIE, BSD Installer,
  launchd, etc..
• Future Directions

4
FreeBSD in a Nutshell

• Freely available Unix-like operating system
• Runs on x86, Alpha, Sparc64, IA-64, AMD64
  architectures
• Over 13,000 software packages available
• Many commercial users
• Thousands of developers around the world
  contributing to it
• Used as an operating systems research platform
• So far, so Linux...

5
Development Organization

• Two layers of FreeBSD organization
• The committers, and everyone else
• Committers have write access to the source tree
• Everyone else submits patches or bug reports
  using FreeBSD's problem reporting system (GNATS),
  and waits for a committer to commit the change
• Individuals who submit many patches (that work)
  are invited to
• become committers
• 9 committers form the elected "core team", for
  dispute resolution

6
Development Organization

Thousands of contributors
9 core team
Sourcecode
350 committers
7
FreeBSD Development Model

• Thousands of developers around the world
  contribute code to FreeBSD.
• Only the "committers" have write access to the
  CVS repository
• 353 total committers in FreeBSD (includes
  source, documentation, and ports committers).
• 212 active committers to src/ in the last 12
  months.
• 149 active committers to src/sys (kernel) in the
  last 12 months.
• 80 of major src/ commits in CVS made by roughly
  45 senior developers.

(active is loosely defined as making a single
commit.)
8
FreeBSD Development Model

• FreeBSD development continues along two parallel
  branches FreeBSD-CURRENT and FreeBSD-STABLE.
• FreeBSD-CURRENT is the main trunk of our CVS
  repository. All new development should happen
  here.
• FreeBSD-STABLE is the branch from which major
  releases are made. Changes enter this branch at
  a different pace, and with the general assumption
  that they have been well tested by our user
  community running -CURRENT.

9
Source Code Control

• The entire source code for FreeBSD is stored in
  a CVS repository.
• The logs, and individual changes for each file
  can be traced back to 1994.
• The source tree can be checked out at any state,
  or corresponding to any release tag.
• CDs are available taking the history back a
  further 20 years.

10
Source Code Control Perforce

• Limitations of CVS (vendor branches, poor three
  way merges, lack of atomic changelists, etc.) can
  really hamper large development projects.
• As an open source project, we strongly believe
  we should support and use open source tools.
• CVS is our official system, but next-generation
  development is often done in Perforce first
  before being merged to our -CURRENT CVS branch.

11
Source Code Distribution

• FreeBSD Source Code
• Available on CD (freebsdmall.com, others)
• Can be downloaded from ftp.FreeBSD.org
• Changes to the source code
• Can be updated using CVS
• Can be updated using CVSup (faster CVS)
• Changes can be automatically e-mailed in, and
  integrated with your local source tree
• Can be browsed, with history, on the web, at
  http//www.FreeBSD.org/cgi/cvsweb.cgi

12
FreeBSD Branch Model

• Head of the tree is now FreeBSD 7.0-CURRENT.
• Multiple STABLE branches for more mature
  releases. RELENG_4 branch from which 4.x
  releases are made, RELENG_5, RELENG_6, etc..
• Point releases are made at 4 month intervals off
  of STABLE release branches until the release
  branch becomes obsolete.
• 4.11 was last release on RELENG_4. Probably one
  more 5.x release.
• 6.0 real soon now -- we expect a long and
  successful future for 6.x releases.

13
March 2000, FreeBSD 4.0
14
FreeBSD-Current (became 5.0)
FreeBSD-Stable
March 2000, FreeBSD 4.0
15
FreeBSD-Current (became 5.0)
FreeBSD-Stable
4.3, April 2001
4.2, November 2000
4.1.1, September 2000 (crypto)
March 2000, FreeBSD 4.0
4.1, July 2000
16
Hats in FreeBSD

• Originally, large percentage of FreeBSD commits
  made by set of core developers.
• As the project has grown, the number of
  significant contributors has also grown.
• The core team model needed to be modified with
  the creation of additional teams to delegate
  project management responsibilities.

17
Hats in FreeBSD

• Release Engineering Team
• Security Officer Team.
• Port Manager Team.
• Documentation Engineering Team.
• Donations Liaison Team.
• Marketing / Press Team.
• Architecture Review Board.
• CVS / Perforce Meisters, Admins, postmaster,
  etc..

18
FreeBSD Build System

• The complete system is available in CVS
  kernel, compilers, libraries, headers, userland,
  third-party packages, etc.
• The entire system is built with a single command
  in /usr/src make world .
• Some GNU software included (e.g. gcc, gdb).
  These are imported on CVS vendor branches with
  BSD Makefiles instead of default configure based
  Gmake build (for parallel makes, make -j N,
  etc..)
• Over 13,000 third party applications in FreeBSD
  Ports Collection.

19
Ports Tree Growth
20
Release Process Overview

• Releases are made from the -STABLE development
  branch at approximately 4 month intervals.
• Reminder / Announcement email sent to developers
  lt45 daysgt
• MFC Sweeps ("Merge from -CURRENT")
• Code slush / freeze lt30 daysgt
• Release Candidates lt15 daysgt
• Eventually a final release
• Upcoming release schedule maintained at
  http//www.FreeBSD.org/releases/6.0R/schedule.html

21
Release Engineering Team

• A principal release engineer drives the process
  for each release.
• Other team members have different
  responsibilities to help principal release
  engineer
• help evaluate MFC merge requests to evaluate
  risk / reward tradeoff.
• Run builds for release candidates and releases
  on different architectures.
• Translate release notes, work on DocBook based
  documentation infrastructure.
• Etc.

22
Building Releases

• One can build a complete release of FreeBSD,
  including FTP install directories, floppy images,
  and ISO images for CDROMs with one command.
• make release is used by many large companies
  to produce special versions of FreeBSD with
  special patches or additional software installed
  by default.
• It is also the well documented way in which the
  release engineering team makes all official
  releases of FreeBSD.
• Schedules, checklists, process documentation
  available at http//www.FreeBSD.org/releng.

23
Release Building

• Tools necessary for building a FreeBSD release
  are available in src/release/
• These tools aim to provide a consistent way to
  build FreeBSD releases.
• A complete release can be built with only a
  single command, including the creation of ISO
  images suitable for burning onto CDROM,
  installation floppies, and an FTP install
  directory. This command is aptly named "make
  release"

24
Release Building

• To successfully build a release, you must first
  populate /usr/obj by running "make world" or
  simply "make buildworld". The release target
  requires that several variables be set properly
• CHROOTDIR - The directory to be used as the
  chroot environment for the entire captive release
  build.
• BUILDNAME - The name of the release to be built.
• CVSROOT - The location of a CVS repository.
• RELEASETAG - The tag corresponding to the sources
  to build.

25
"make release" (1/4)
There are many other variables that can be used
to customize the release build process. Most of
these are documented at the top of
src/release/Makefile. The exact command used to
build FreeBSD 4.6 (x86) was make release
CHROOTDIR/local3/release \
BUILDNAME4.6-RELEASE CVSROOT/host/cvs/usr/home/n
cvs RELEASETAGRELENG_4_6_0_RELEASE
26
"make release" (2/4)

• The release makefile can be broken down into
  several distinct steps.
• Creation of a sanitized system environment in a
  separate directory hierarchy with "make
  installworld".
• Checkout from CVS of a clean version of the
  system source, documentation, and ports into the
  release build hierarchy.
• Population of /etc and /dev in the chrooted
  environment.
• chroot into the release build environment, to
  make it harder for the outside world to taint the
  build.

27
"make release" (3/4)

• Steps of "make release" (continued)
• "make world" in the chrooted environment.
• Build Kerberos-related binaries.
• Build 'GENERIC' kernel.
• Creation of a staging directory tree where the
  binary distributions will be built and packaged.
• Build and installation of the documentation
  toolchain needed to convert the documentation
  source (SGML) into the HTML and text documents
  that will accompany the release.

28
"make release" (4/4)

• Steps of "make release" (continued)
• Build and install of the actual documentation
  (user manuals, release notes, tutorials,
  hardware compatibility lists, etc..)
• Build of the "crunched" binaries to be used on
  the installation floppies
• Creation of the distribution tarballs of the
  binaries and sources.
• Create the boot media and a fixit floppy.
• Create the FTP installation hierarchy.
• (optionally) Create ISO images for CDROM media.

29
Ports and Packages

• The FreeBSD Ports Collection is a collection of
  over 13,000 third-party software packages
  available for FreeBSD.
• The ports team (portmgr_at_FreeBSD.org) is
  responsible for maintaining a consistent ports
  tree that can be used to create the binary
  packages that accompany a given FreeBSD release.

30
Ports and Packages

• In order to provide a consistent set of
  third-party packages, every port is built in a
  separate chroot environment, starting with an
  empty /usr/local and /usr/X11R6.
• The requisite dependencies are installed as
  packages before the build proceeds.
• By starting the package build in a pristine
  environment, we can assure that the package
  metadata (such as required dependencies) are
  accurate, and so we will never generate packages
  that might work on some systems and not on others
  depending on what software was previously
  installed.

31
Build Clusters

• Several large companies have donated machines,
  rack space, power, and network bandwidth for
  build clusters for all supported architectures to
  allow release engineers and port masters to build
  releases and packages quickly.
• Current ports cluster can build 13,000 third
  party packages in just over 24 hours for i386
  (somewhat longer for most other architectures).

32
Extensibility

• A release can be customized in many different
  ways.
• If you have access to the staging directory of
  an existing "make release" build hierarchy then
  you may find it easiest to apply patches or
  additional files as necessary to the chroot build
  directory.
• rm CHROOTDIR/usr/obj/usr/src/release/release.
  48
• Rebuild sysinstall, the kernel, or whatever
  parts of the system your change affected.
• chroot CHROOTDIR ./mk release.4 release.8

33
Summer of Code

• Googles Summer of Code Program sponsored 19
  students to spend the summer working on FreeBSD
  (nearly 400 total with other open source
  projects)
• Many interesting projects in security, power
  management, networking, file systems, and release
  engineering.
• Specifically
• Dario Freni - FreeSBIE (FreeBSD LiveCD)
• Andrew Turner - BSD Installer (new installation
  tool)
• R. Tyler Balance - launchd (porting from MacOS X
  10.4 to speed up Unix boot process.)
• All student code in Perforce (perforce.FreeBSD.or
  g), some being merged into main CVS repository.

34
Future Work

• More process documentation
• More automated performance testing
• Better Installation Tools / Graphical User
  Interface
• Better integration with FreeSBIE livecd

35
Additional Information

• Detailed documentation on "make release" is
  available in release(7).
• A schedule of upcoming releases is maintained at
  http//www.FreeBSD.org/releng
• The release engineering paper is now maintained
  in CVS and the most recent version is available
  from http//www.FreeBSD.org/docs.html
• If all else fails, just email re_at_FreeBSD.org and
  we will be happy to answer your questions.
• Excellent Case Study on FreeBSD Development in
  September IEEE Transactions on Software
  Engineering.
• http//wikitest.FreeBSD.org

36
Questions?