How To Troubleshoot Backup Failure Using AMANDA Backup System?

What Is Amanda Backup System?

AMANDA is backup system that enables the admin to establish an individual master backup server in order to back up multiple hosts over the network. The Amanda backup system will tape drives/changers or disks or optical media too. The backup algorithm of Amanda is highly impressive with which it can perform complete backup operations, including both full and incremental backups. Thus the backup decreases the network overload of a complete full backup.

How To Install Amanda?

The Amanda needs to be installed on the central server of the network along with the individual client machines. The central server is the major location in which all the data (whether disk or tape ) is saved. The significant files like amanda.conf, disklist and tapelist are also collected in the central server. Moreover,  Amanda.conf is the location that consists of the entire configuration of the system. On the hand, the other two files, namely disklist and tapelist, lists the resources that need to be backed up and tapes what data need to be written to.

How To Check Amanda Configuration?

A utility called ‘amcheck’ can check Amanda configuration. The command to verify the Tape Host Server configurations is given below:

# amcheck MyConfig

Run Backup Test

The tool to run backups is ‘amdump’. Run as the Amanda user as shown below.

# amdump MyConfig

To execute the above command will take a few seconds. If you get an output as ‘0’, it means you have backed up successfully. On the other hand, if you receive other than ‘0’, it implies that your backup has failed.

In this article, let’s have a look at the common Problem/Bug that occurs in the Amanda backup system that causes the backup system to fail. The daily backup is important so no backup fails to occur in this system.

Disk Space Issue

Holding disk is important for the Amanda backup system.  The holding disk will enable data to be streamed to media at maximum speed. This prevents tape buffer under-runs and shoe-shining of tapes.

The disk space issue may cause due to this intermediate storage area for caching backup data and the backup process fail.

Solution :

We can use Amflush command to flush Amanda backup files from holding disk to tape :

#amflush MyConfig

Amflush will search for holding areas associated with the daily configuration. When you choose your required holding area to be flushed, amflush composes the data to tape, updates the databases and sends a mail report similar to amdump.

The example noted below indicates how to flush everything to tape using amflush. Then ejects tape, using ‘at’ to run the task in the background.

% echo 'amflush -b -f MyConfig && mt offline' | at now

Issue Related To Network Services (x/inetd) 

Check if network services (x/inetd) and .amandahosts are configured correctly.

Correct xinetd and .amandahosts configuration are available at How To: Configure bsdtcp authentication or for older auths, How To: Configure Backward-compatible Authentication Methods.

Here is a checklist once you have verified the correct configuration:

Make sure you have added the Amanda services to /etc/services (or the NIS services map).

Make sure you signalled x/inetd to reread its configuration (some systems may need rebooting), for example

#/etc/init.d/xinetd reload

Pay special attention to typos in inetd.conf; error messages will probably appear in /var/adm/messages or /var/log/messages if you have typed the amandad program name incorrectly.
If you are building Amanda binaries on your own, make sure the dump user that has been specified at configure-time (–with-user=USERNAME) is listed in the (x)inetd config file.

Check whether the dump user has permission to run amandad, as well as any shared libraries amandad depends upon, by running the specified amandad command by hand, as the Amanda user. It should just time-out after 30 seconds waiting for a UDP packet. If you type anything, it will abort immediately, because it can’t read a UDP packet from the keyboard.

Verify the xinetd or inetd is running such as by executing

ps -ef | grep inetd

If not, start it manually, for example

/etc/init.d/xinetd start

Once network services are running

netstat -a | grep amanda

This can be used to verify that there is some program listening on the amanda/udp or /tcp port. Another tool that can be used for verifying that amandad is listening on the udp or tcp port is lsof, for example

lsof -c xinetd

Issue Related To Firewall/TCP-Wrapper Settings

Firewall between the backup server and the client can cause self-check to timeout if the firewall is not configured correctly.

Like most services started from x/inetd, the firewall or TCP-wrapper on the client has to be configured to allow the server to come in.

If you are using tcpd wrapper for Amanda inetd entries, hosts.allow have to modify to allow Amanda connections.

Example: inetd configuration entry using tcpd:

amanda dgram udp wait amandabackup /usr/sbin/tcpd /usr/lib/amanda/amandad

hosts.allow file:

amandad: ALL : ALLOW
amindexd: ALL : ALLOW
amidxtape: ALL : ALLOW

Permission/Ownership Issue

Access to Amanda processes should be restricted to only Amanda clients.

Wrong permissions or ownership for Amanda user home or log directories.

Incorrect permissions or ownership for log directories (/var/log/amanda) and/or the home directory of the Amanda user (/var/lib/amanda)on a client can produce the following amcheck error.

WARNING: clientname: selfcheck request failed: tcpm_recv_token: invalid size: amandad:

Client check: 3 hosts checked in 1.121 seconds. 1 problem found.

Such incorrect permissions such as Amanda’s home directory being owned by a different user can often occur if Amanda has been installed more than once on the same system.

Example of permissions for Amanda’s home directory on Linux

drwxr-xr-x 11 amandabackup disk 4096 2008-12-02 17:11 /var/lib/amanda

Amanda Version Releases And Bugs On Different Versions

There is a bug on version 2.6 that if one host fails then all other hosts can also fail. This bug is already fixed in the source and will be in 3.3.6.

The most recent stable release is version 3.5.1, released on December 1, 2017.

The latest release is the 3.4.x series is 3.4,5, released on June 8, 2017. This is a bugfix release for 3.4.4.

The latest release is the 3.3.x series is 3.3.9, released on February 10, 2016. It is a security fix. The Amanda user was allowed to run any code as root, the upgrade is not required if you trust the amanda user.

The latest release in the 3.2.x series is 3.2.3, released on May 9, 2011. It is a bug fix release for version 3.2.2.

The latest release in the 3.1.x series is 3.1.3, released on October 5, 2010. It is a security release for version 3.1.2.

Amanda-3.1.2 has a known security vulnerability, and all users should upgrade to Amanda-3.1.3 as soon as possible.

The Amanda development team has discovered a security vulnerability introduced in Amanda-3.1.2. The vulnerability affects both Amanda servers and clients and could lead to remote execution of code as the Amanda user.

The problem is fixed in Amanda-3.1.3, which is released as of October 5, 2010. It is a patch release in the 3.1.x series and contains the fix for this security vulnerability as well as fixes for several other significant bugs in 3.1.2.

This upgrade should be considered a high priority upgrade.

So we need to ensure the proper upgrade to the latest Amanda release.