Add Me!Close Menu Navigation

My technical corner about Linux, Perl, programming, computer networks and network security

Add Me!Open Categories Menu

Basics of writing bash scripts, an advanced backup

More inteligent and advanced backup

Sometimes something can go wrong and then a simple script may not help us. There is nothing worse than the admin realising hat their own script has crashed his professional career because it didn’t work correctly (corrupted spare files, low disc space, etc). Usually the disaster is hidden because when you know about the situation it’s too late (for example: A script has copied corrupted files for 3 months).

The script which i explained in the article was simple. It’s enough for many people (admins) but sometimes we really need a more advanced script. For example the last script doesn’t check ifof disk usage and in which disk it will copy files and valididate backup files.

Now, we need more advanced tool. Our new script should be able to cope with common threats in the future. The script must:

  • check free space on the destination backup place, if it’s too little, it must use an alternate location automatically.
  • check the integrity and validate files which are copying – corrupted files aren’t usable of course

Additionaly, if the script finds the above problems then it can send information by an e-mail to the admin. Many administrators who use scripts prefer that scripts send an e-mail after the work and there is no concern that any problem exists. In my opinion, this is a bad idea, because probably we can become too much accustomed to many e-mails and someday, an important e-mail may be accidentally ignored. I prefer to be informed by an e-mail only when it’s necessary.

 

Time to work. First, we check commands by hand. As you remember, the script will be doing the same as you, so you should know what do you do. :-)

We asumme that we have separate partition (/dev/sdb1 in this case) for backup. Now I use the df command to check free of space in the partition with the grep to clear the results:

$ df -h | grep /dev/sdb1
/dev/sdb1             184G   46G  129G  27% /backup

 

So as you can look, I have the separate partition /backup for backup. The df command lets me check the size and the usage of the partition. Usage its displayed as percentage, so I will try to use percents in my new script. To get the percent number from the df’s output I use the awk command:

$ df -h | grep /dev/sdb1 | awk {'print $5'}
27%

 

I won’t explain how the awk works – you shoud be able to use it because it’s very usefull tool in bash scripts. I use the awk with the „-F” switch – in the manual of awk you can find descriptions about not only this switch.

The command displays to me only percent number from the  entire data. It’s quite usefull now, but i need only the raw number, without the % char. To remove this char i use the tr command:

$ df -h | grep /dev/sdb1 | awk {'print $5'} | tr -d %
27

 

Now, I have that I want. We can see that the /backup partition has  only used 27%. I decided to be informed by the script when the usage of the /backup partition exceds 90%. This is a safe limit for me.

When the usage exceed 90%, a few backups still can be done on the /backup partition even if I will ignore first warnings.

If I ignore a few warnings and the /backup partition is full the script will use the second backup partition – /backup2, but this problem shoudn’t occur if I am a good admin. ;-)

$ df -h | grep backup2
/dev/sdb2              47G   33G   15G  70% /backup2

 

 

This partition is smaller than the primary /backup partition, but it will be only be used in emergency scenario. Information about percentage of the usage this partition I get the same way as for /backup.

An e-mail will have been sent by script only if a some problem will occur during the backup.

 

Advanced backup of a Firebird database

Ok, we have our basic guidelines. Now we can write the script. For example, I will write the advanced script which will be do a backup a small Firebird’s database (1 GB). Firebird databases is in common use in some companies so it will be a good example.

Before the backup of any databases we must check that databases files/dumps are valid. Why? It’s obvious: the backup of a currupted database is completely useless. It’s the same situation as if backup never existed.

 

For checking of the validity of Firebird’s databases we use the gfix program which is the part of the Firebird package. The gfix syntax:

$ gfix -v -f database.fdb

 

The command is very simple to use in the script, because if the database is valid, the gfix returns nothing, otherwise it shows some information about corruption. This behaviour is enough to use the bash condition ‘-z’.

To save of disk space we will create an archive by the 7zip program, because the 7zip is one of the most effective archive programs. We must check that the created archive is valid also. The reason is the same as the reason for checking the backup valid. To do this, we can use the command `7z t` which is simple mean as ‘only test archive’. If the command return the exit code of the last command ($?) different than 0 – something is wrong.

#!/bin/bash
TEST=`7z t somearchive.7z`
echo "Exit code: $?"

 

Time to conclude what do our script will do:

  • It will check the free disk space on the backup partition
  • If the disk space is full then it will use an alternate location
  • It will check the validity of the backup database
  • It will check the validity of the created archives
  • If any problems occure it sends information by e-mail
  • It will complete a log about the backup process (some companies require this)

 

Here you are, the entire script:

 

#!/bin/bash

### Path to database to backup dir
BACKUP_FROM="/database"

### Path to backup's dir/partition
BACKUP_PARTITION=/backup

### Path to emergency backup's dir/partition
BACKUP_PART_SEC=/backup2

### Percent of usage main partition backup when it exceeds then script
### will provide warning information
PERCENT=70

### Percent of usage main partition backup when it exceeds then script
### will backup on emergency partition
EMERGENCY=95

### Log dir about backup process
LOG_DIR=/var/log

### The CP command
CP=/bin/cp

### The RM command
RM=/bin/rm

### Path to the gfix to checking of validity
### Firebird database
GFIX=/usr/bin/gfix

### Path to the p7zip program to create archives
ZIP7=/usr/bin/7z

### Path to the mail program
MAIL=/usr/bin/mail

### An e-mail address where the script will send information about
### detected problems during backup process
EMAIL="adres@email.com"

#####################################################################
#####################################################################
#####################################################################

### Because it's more inteligent script, so it checks that some
### require to correctly working programs exists. 

if [ ! -e "$GFIX" ];
then
 echo "ERROR: $GFIX not found"
 exit 1
fi

if [ ! -e "$ZIP7" ];
then
 echo "ERROR: $ZIP7 not found"
 exit 1
fi

if [ ! -e "$MAIL" ];
then
 echo "ERROR: $MAIL not found"
 exit 1
fi

#####################################################################
#####################################################################
#####################################################################

### Suffiks of date
DATE=`date +%Y-%m-%d`

### Creating the log filename
LOGFILE=$LOG_DIR/backup.$DATE
:>$LOGFILE
echo "$DATE: Starting backup..." >> $LOGFILE

### 1. Check the free disk space on the main partition of backup
ILE=`df -h | grep /dev/sdb1 | awk {'print $5'} | tr -d %`

###
### Sending the information when first limit was reached
###
if [ "$PERCENT" -ge "$ILE" ];
then
 ### Warning
 OSTRZEZENIE="$DATE: warning - little the free disk space on $BACKUP_PARTITION "

 ### Writting the log
 echo "$OSTRZEZENIE" >> $LOGFILE
fi

###
### Change the backup partition if the second limit was reached
###
if [ "$EMERGENCY" -ge "$ILE" ];
then
 ### Writting the log
 echo "$DATE: !!! WARNING: There is no free space on the $BACKUP_PARTITION - emergency backup." >> $LOGFILE

 ### More danger warning
 UWAGA="$DATE: !!! WARNING: There is no free space on the $BACKUP_PARTITION"

 ### Changing of the backup partition to an alternate location
 BACKUP_PARTITION=$BACKUP_PART_SEC
fi

### Doing the backup

### Entering to the working directory
cd $BACKUP_FROM

###
### Checking of all databases in a directory first
###
for i in *.fdb;
do
 TEST=`$GFIX -v $i`
 if [ ! -z "$TEST" ];
 then
   ### Corrupted database - write the log and go further
   echo "$DATE: *** ERROR: $GFIX claims that the database $i is corrupted." >> $LOGFILE 
   UWAGA="corrupted database $i "
 else
   ### Valid database - write the log
   echo "$DATE: Copying $i.$DATE to $BACKUP_PARTITION" >> $LOGFILE 

   ### Copying the correctly database
   $CP $i $BACKUP_PARTITION/$i.$DATE
 fi
done

###
### Creating archives now
###
cd $BACKUP_PARTITION
for i in *.$DATE;
do
 ### Packing
 $ZIP7 a $i.7z $i

 ### Report
 echo "$DATE: compression $i as $i.7z." >> $LOGFILE

 ### Remove the original file
 $RM -f $i
done

cd $BACKUP_PARTITION

###
### Time to tests
###
for i in *.$DATE;
do
 ### Testing created archives
 $ZIP7 t $i
 if [ "$0" -gt "0" ];
 then
   echo "$DATE: * ! * ERROR: the archive $i is recognized as corrupted." >> $LOGFILE
   OSTRZEZENIE="the archive $i is corrupted"
 fi
done

###
### Checking if any problems exist - if yes, we should
### send warning information to an administrator
###
if [ "$OSTRZEZENIE" ] || [ "$UWAGA" ];
then
 SUBJECT="$DATE: $OSTRZEZENIE"

 if [ "$UWAGA" ]; 
 then
  SUBJECT="$DATE: $OSTRZEZENIE $UWAGA"
 fi
### Sent and e-mail with backup's log
INFO=`cat $LOGFILE`
 echo $INFO | mail -s "$SUBJECT" $EMAIL
fi

 

Leave a Reply

You must be logged in to post a comment.