BU Changelog

03/10/2005 Version 3.4 beta

Ported to NetBSD / OpenBSD. As a byproduct, it also now runs on MacOS X 10.2. However, it would not backup any files on MacOS X 10.1.

The CDRW dump code is still untested and assumed to not work on any platform other than FreeBSD-4.x.

Changed the license from the GPL back to the BSD copyright.

Bu now uses pax instead of tar for NFS backups.

Why I chose pax:

  • Tar's command line options are not consistent across platforms.
  • Pax is part of the base system on BSD and seems to be available for at least the mainstream Linux distributions.
  • Pax handles all the cpio and tar formats. After testing with both pax and cpio, I chose pax because GNU cpio (version 2.5) on FreeBSD-5.2.1 had a strange bug. When run from a pipe in Perl it truncated the destination file names on symbolic links (Linux may have the same problem since Linux distributions also use GNU cpio).

    For example, after backing up /dev, a long listing of part of the backup directory showed

     lrwxrwxrwx  1 vince  wheel   -       4 Dec 30 23:31 bktr -> bktr
 cr--r--r--  1 root   wheel   -  92,  0 Oct 27 19:37 bktr0
 lrwxrwxrwx  1 vince  wheel   -       3 Dec 30 23:31 log -> /va
 lrwxrwxrwx  1 vince  wheel   -       4 Dec 30 23:31 net1 -> net/
 lrwxrwxrwx  1 vince  wheel   -       4 Dec 30 23:31 net2 -> net/
 lrwxrwxrwx  1 vince  wheel   -       4 Dec 30 23:31 net3 -> net/

 A long list of the original directory showed

 lrwxr-xr-x  1 root   wheel   -       4 Jan  1 13:15 bktr -> bktr0
 cr--r--r--  1 root   wheel   -  92,  0 Jan  1 13:14 bktr0
 lrwxr-xr-x  1 root   wheel   -       3 Jan  1 13:15 log -> /var/run/log
 lrwxr-xr-x  1 root   wheel   -       4 Jan  1 07:15 net1 -> net/fwe0
 lrwxr-xr-x  1 root   wheel   -       4 Jan  1 07:15 net2 -> net/sk0
 lrwxr-xr-x  1 root   wheel   -       4 Jan  1 07:15 net3 -> net/plip0

    GNU cpio did not have this problem when run directly from the command line, so it has something to do with running it through a pipe from Perl. I also ran into bugs in tar when reading through pipes back when I originally wrote the multi-volume CD dump code. That is why I ended up writing perl code to break up the volumes. Pax comes from the BSD world and seems to be the most robust.

The rc file variable names have changed. They are now lower case which is more similar to the format of /etc/rc.conf in BSD, and they are all prepended with bu_ so that they are less likely to conflict with any other variables if set in the environment. For example $BACKUP_DIR is now $bu_backup_dir.

Added the following new configuration variables to the rc file:

    bu_etc_dir
    bu_mail_addr

Most of the settings may now be set as environment variables that override the configuration file.

To turn an option off in the rc file, you can no longer do it by unsetting the variable. If you do, it will fall back to whatever the default setting is, which could be either on or off. You must use one of the off settings mentioned in the rc file comments, such as off, no, or 0.

The Include and Exclude files are no longer installed into the log directory. They now go in $bu_etc_dir, which is set to /usr/local/etc/bu by default. If you set prefix=/usr during installation, it will be set to /etc/bu.

The default log directory has changed from /var/backups/bu to /var/log/bu.

The default log directory, when running as non-root, now stems off of the users home directory rather than the system backup log directory. This is so that bu will work out of the box without permission problems for regular users.

Added new command line option:

    --newrc* | --new-rc* | --newconf* | --new-conf* 
        Create a new rc file using the current settings.

The rc file is no longer automatically created by just using the help (-h) switch. It is still automatically created if you do a backup, are using the default rc file, and it does not already exist. Otherwise, you can now use --newrc to create a new rc file.

Added PS_PERSONALITY=bsd variable so that the newer GNU ps utility on Linux systems will work when bu checks for other bu processes.

The ``specified files'' portion of the logs now show full pathnames of specified files that do not have wildcards in them. That way if a relative path such as ``.'' is specified, it shows the real path.

Changed the default backup directory in the Include file from / to the current working directory.

Replaced the realpath script with the better realpath utility ported from FreeBSD.

Installation and de-installation enhancements.

  • You can now install and de-install as a regular non-root user without any special settings.
  • Changed $BASE to $destdir for installing or un-installing from directories other than root. The commonly used uppercase name, $DESTDIR, will also work.
  • The default $prefix is no longer set in the Makefile. Now Both $destdir and $prefix are automatically set in the install script, if not specified, based on whether it is being installed as root or a regular user. By default, as a regular user, it is installed with destdir=~/pkg and prefix unset. This creates an application directory structure like /usr/pkg on NetBSD only in the users home directory. If installing as root, the default is prefix=/usr/local, and destdir is unset.
  • Changed the code in the installation script that modifies the prefix setting in bu to include $destdir if installing as a non-root user so that it can find it's libexec files when installed to a user owned directory such as the users home directory.

Documentation is generated from the POD sources (see the man for perlpod(1)), which is now included in the package, during make. Before, only the final text version was included in the package.

A file list is no longer generated and installed in the doc directory during installation. This was more work to maintain and didn't seem to have much benefit.

Fixed minor bug that caused an error message when processing command line switches.

Enhanced the online usage (-h option).

Removed the white paper. I wrote it back in 1998. Much of the information in it was too out of date and I don't have time to re-write it right now.

Other miscellaneous minor bug fixes and code cleanup.

 

07/20/02 Version 3.3 beta

Minor fix to bu_dump to put quotes around the file names to accommodate file names with `` - '' strings (and hopefully any other possible strings) in them in --dump mode.

Minor fix to bu_lib.pm for --dump mode. It was running mkfifo with the --mode switch which only works with mkfifo from the GNU fileutils package. It now uses the '-m' switch so that it works on FreeBSD without GNU fileutils installed.

Modified --dump mode to use flock()file locking to arbitrate read and write access to the file that the archiver()process uses to communicate the amount of data written. This was because of a possible race case potentially causing it to not break up the volumes properly.

Added file, VERSION, in the libexec directory for bu to read it's version from when showing the help. Now all dependency on ident is eliminated since some Linux distributions do not install it by default, even with the development environment.

Fixed minor bug. Some error log entries were getting duplicated.

Bug fix and improvements to the code that checks available space on the media for each file in CDRW dump mode.

 

06/13/02 Version 3.2.2 beta

This release fixes several minor installation and de-installation bugs.

The ident utility is no longer required during installation, since you cannot depend on it being installed on some distributions even when the development environment is installed. It now includes a RELEASE file that contains the package release version.

Modified the installation script to show the file list during installation and to store a file list in the doc directory.

A Linux binary of GNU tar is no longer included because the included binary did not run on Redhat, apparently because of some library difference. Tar is new enough on modern Redhat releases anyway, and hopefully, so is the included tar on most other distributions. GNU tar version 1.12 or higher is required. The FreeBSD tar binary is still included and automatically installed in bu's private libexec directory when installing on FreeBSD.

Fixed the 'q' or 'a' option in the Un-install script for aborting. It was being ignored and un-installing anyway.

 

06/04/02 Version 3.2.1 beta

Fixed a bug in bu_lib.pm when using Perl versions before 5.6.0. The older versions of Perl require a MASK argument to mkdir().

 

06/02/02 Version 3.2 beta

This release primarily affects the NFS backup features. The CDRW features are unchanged from version 3.0.

Bu will appear to function the same as before on the surface (except much faster) from a user standpoint but this version is very different under the covers.

I completely re-wrote the backup()routine in Perl utilizing GNU tar and code that was written for the CDRW dump features. This resulted in the following benefits.

  • It now copies empty directories and preserves hard file links so that the backup filesystem is a closer replica of the original filesystem. Before, it used ``cp --archive --update'' which did not preserve hard links, so the backup filesystem used more space than the original.
  • The performance has been vastly improved. In my tests, bu-3.1 has been ranging from 2 to over 7 times faster than bu-3.0 and below. You should notice a substantial speed improvement across the board, but the largest performance increase is with incremental backups when there are not many files that need backed up. There are a lot of performance considerations in the new code, but the largest portion of the improvement is because of the following.
    • Bu now uses a single tar stream for the entire backup, so it no longer has the overhead of calling ``cp'' for every file. It runs two copies of tar, one that tars the original files piped to another which untars at the same time to the backup filesystem, similar to the way cpio is often used to copy files while preserving all file attributes. I did not use GNU cpio because it did not handle the large device numbers that FreeBSD uses for device files.
    • Before, cp was called for every file with the --update switch to handle incremental whether the file needed backed up or not. Now, incremental mode is handled directly with Perl code with only the overhead to stat the files and check the dates on files that do not need backed up.
    • It no longer has to generate the entire list of files before it begins backing them up. It spawns a background process, using the same code that the CDRW dumps use, to find the files, filter them through the Exclude filter, and stream the file names back to immediately begin backing them up. This way the file list generation process runs parallel to the process that backs up the files.
    • Using Perl, I was able to write tighter more efficient code due to it's powerful regular expression features for handling the file exclusion patterns, and the direct access to C language system calls among other things. The primary program is still in shell script but the main backup routine is now in Perl.
  • Since it now does not use GNU cp, the GNU fileutils package is no longer required under FreeBSD.

GNU tar binaries for FreeBSD and Linux are now included. Tar gets installed in bu's private lib directory. This is because FreeBSD, at least as of version 4.4, still uses an old version of GNU tar (version 1.11.2) which has different command line options. Most modern Linux distributions probably have a new enough version of tar, but bu still installs it's own copy just to be safe. I believe tar-1.12 is the oldest that will work with bu.

 

05/14/02 Version 3.0 beta

Added the ability to do compressed multi-volume dumps to CDRW's. This involved substantial enhancements including over 900 lines of new, changed, and rearranged shell script code, over 1300 lines of new code written in perl, plus a small routine written in C to provide a usleep function without creating dependencies on extra modules that are not distributed with perl5. There are no built in features yet for restoring from CD's but it is easily done manually with standard unix tools.

In this release, the CDRW dump capability only works on FreeBSD. I will get the Linux port of that part of the code done as soon as possible.

Options in the form of ``--option value'' may now also be specified as ``--option=value''.

All command line options now have a long form (ie. --word). Most, but not all, have a short form equivalent (ie. -x).

Changed -f to -I or --include. -f will still work for now but is deprecated.

Changed variable names LOGDIR and TMPDIR in the rc file to LOG_DIR and TMP_DIR to be more consistant with the other variables.

Made TMP_DIR configurable in .burc.

Added the following new options:

  --full     Do a full backup (turning off incremental mode)
             -i now always turns on incremental (no longer toggles)
  --xdev     Stay in the local filesystem (do not cross mount points).
  --exclude=file   Specify the exclude file

The following new options are for dumps to CDRW's:

  --dump    --restore   --differential   --image   --compress
  --eject   --blank     --test           --prompt
  --device=device  --label="label"  --speed=speed

New variables have also been added to the configuration file to set defaults for the new options.

Any command line switches that used to toggle (set a feature opposite to the setting in the rc file, now always turn on or off the feature. To gain command line control, the feature must be set in the rc file to the opposite of what the command line switch does. This way the switches have consistent behavior.

Added /var/run and */.opera/cache* to the default Exclude list.

Added lots of new documentation describing bu's new features as well as old features that were not previously documented.

Added a simple Makefile for installing and de-installing since bu now has a C program, usleep.c, that needs to be compiled.

Improved the Install script to be more gracefull about upgrading from a previous version of bu.

Changed the copying policy back to the GNU copyright.

 

9/30/01 Version 2.8

Fixed to properly detect GNU cp is installed with the newer fileutils-4.x package. They changed the output of ``cp --version'' to just output ``fileutils'' rather than ``GNU fileutils'' which broke bu's detection since it was looking for the string, ``GNU''.

 

8/27/01 Version 2.7

Fixed bug that caused warnings to be printed from the expr command when running under Linux.

Fixed a portability bug that prevented using non-GNU awk.

Modified to work with GNU cp installed as gcp as well as cp.

Improved the Install script to preserve existing custom Include/Exclude files when upgrading from a previous version.

Improved the Un-install script to offer a choice whether to do a completely clean un-install or preserve the configuration and log files.

Miscellaneous minor code clean up and improvements.

 

8/22/01 Version 2.6

Fixed bug with using wild cards in the Include list. It now works with wild cards specified in the directory paths.

Security fix: Logs are now created with 640 perms (no world read access) by default. Before, users could read the backup log files to see what files existed in directories that they had no 'x' permission to. The log permissions are also now configurable in ~/.burc via LOG_ACCESS_PERMS.

Enhanced the handling of file names. Before, names with spaces in them could not be specified directly, even though they could exist in specified directories. Now, file names with spaces can be specified directly on the command line as well as in the Include/Exclude lists.

Since almost any valid Unix file name may now be specified, bu no longer errors on unknown switches. It just assumes it is a file name that starts with a dash.

Improved the file exclusion filtering code to be more modular and more robust.

All files are now filtered through the Exclude file. Before, individual files specified were not filtered. The Exclude filter was only used when directories were specified.

Modified log generation to show files exactly as specified, with wild cards un-expanded, in the ``Specified files or directories'' section of the log.

Moved the temp files into their own subdirectory in /var/tmp for each instance of bu, rather than using the log directory.

Enhanced to automatically clean up stale temp files left behind by a system crash or bu getting ungracefully killed via SIGKILL (signal 9).

Enhanced to check to see if your cp command is the GNU one since bu depends on it. If not, it prints and error message and exits.

Other miscellaneous code improvements.

 

7/30/01 Version 2.5

Fixed bug generated in version 2.4 where the file list did not get created in quite mode causing the log to say there were no changed files to back up, even though the backup was successful.

Enhanced the log to show the time it was interrupted if bu gets interrupted or killed.

 

7/29/01 Version 2.4.1

Somehow, the updated default Exclude file got left out of 2.4. It now has it. No other changes.

 

7/24/01 Version 2.4

Ported to FreeBSD's sh shell. It no longer requires GNU bash. I expect it will now run on most any modern Bourne compatable shell.

Added signal handling and made improvements to the handling of the log files. Most importantly, it now cleans up it's tmp files and completes the log file before exiting when killed by a SIGTERM or SIGINT signal.

Enhanced to create unique log filenames for multiple instances of bu running at the same time so that there are no conflicts running parallel bu's, even when backing up the same file system. Each instance keeps it's own unique temporary file list of what it is backing up.

Simplified the default log filename by removing the hour and minute, because it now adds -# to the name, where # is a unique number if a log for the same date already exists.

Changed the default log directory from /var/backups to /var/backups/bu.

Added -m option to mail the log to a specified email address after the backup is finished.

Changed the cp option, -P, to --parents. Apparently, the meaning of -P is being changed in GNU cp to conform to POSIX standards.

Dropped the -v switch for verbose mode and added -q for quiet. Verbose mode is now the default.

Improvements to the default Exclude file.

Changed the copying policy from GNU style to BSD.

 

9/06/99 Version 2.2

Fixed bug when using wild cards in an Exclude list. File name globbing had to be turned off when constructing the regular expressions for the sed filter so that wild cards would not be expanded during that time. It was causing some files that matched the exclude expressions to be backed up anyway.

Fixed bug. When files rather than directories were specified, it did not expand the symbolic links to the real absolute path of the files.

Updated comments in the default .burc file for the INCREMENTAL setting. It still said that incremental only applied when directories were specified, which is incorrect.

Changed the default log directory from /usr/local/backups to /var/backups.

 

02/17/99 Version 2.1

Changed the default LOGDIR from /usr/local/backups to /var/backups.

 

10/15/98 Version 2.0

Minor clean up.

 

10/15/98 Version 1.9

In version 1.8 I accidently broke the -d and -f switches so that they didn't override the configuration file settings. This fixes that.

 

10/1/98 Version 1.8

Rewrote the backup()function so that it no longer uses cpio. Although cpio preserved the ownership of regular files, it sets the owner and group to root for directories it creates. It now uses cp to backup all files (utilizing the -u and -a options). It actually calls on cp for each and every file being backed up. This may sound slow, but it was reasonably fast in my tests. Apparently a tribute to Linux's lightning fast filesystem. Here is a description of the problems I ran into that prompted me to take this approach for those who are interested. If anybody has any better ideas, I am open to suggestions.

In order to fix the directory ownerships that cpio incorrectly set, I would have to extract the directory name from every file being backed up and find what the ownership is on the source directory and then do a chown on the destination directory. I don't want it to have that much overhead. When directories are specified, it creates the list of files before backing them up so they can be filtered through the exclude expressions. Then it needs to feed this list to one of the tools to do the copy. Tar handles permissions correctly but it won't read a list of files from stdin, like cpio will, and I can't put the whole list on the command line because of command line length limitations. I can't use it with xargs because I need to insert the args before the last argument (the destination) with cp. I experimented with going through a loop with a counter to break up the input file list into chuncks of x number of files and call on cp with each group until all the files were backed up. It turned out that the shell script overhead of creating the list of files for each copy was great enough that it bench marked considerably faster to just call on cp for each separate file.

Now, since it is going through a loop for each file, I went ahead and left the counter in it for the next feature.

  • Added the variables, GROUP_SIZE and DELAY (which are configurable in .burc), allowing a configurable number of files to copy at a time and a sleep time between each group of files so that the bandwith on a network can be spread out if you have to back up over NFS during high traffic hours. I don't know how many people need such a feature but it seemed nifty :-). I would appreciate email from anybody who thinks it is useful. If nobody ever uses it, I could eventually take it back out (unless I end up using it myself of course :-).
  • Fixed to work with filenames that have spaces in them.
  • Added ``-c'' option to specify an alternate runtime configuration file.
  • When Incremental mode is on, it now applies to all specified files, not just if directories are specified.
  • The log file that was specified by $LOG is no longer kept. Now there is only one log file for each backup. The filename is specified by the variable, $LOG, but it is the same as what used to be $FILELIST. Before, $LOG just contained the list of each top level directory with the number of bytes backed up. Since bu is no longer using cpio to get the number of bytes, and there was already $FILELIST, the old $LOG file didn't seem that useful. However, the new logfile is nicer with more information about the backup, including start and finish times.

 

9/19/98

Modified the README file to describe some of bu's features better, especially the way it handles symbolic links.