KickStart Network Customization

One of the biggest problems I have found with the KickStart process is fine tuning the network values of a server. I couldn't find anything useful through the standard process so I decided to write my own. If you have seen my last post I reference some customized configuration scripts in the %post section of my KickStart file. In this post I will outline my first customization which I simply call general.cfg.

Basically this sets up the NTP daemon by rewriting the /etc/ntp.conf file and turns off services I don't need. But first it calls a special script called network_config.sh.

# cat general.cfg
# Clean up the network either based on existing DHCP or on configuration file

# network_config.sh requires an argument to tell it where the csv file is and where to output logs
/post_scripts/KickStart/net_config/network_config.sh /post_scripts/KickStart/net_config

# setup NTP
echo 'restrict default kod nomodify notrap nopeer noquery' > /etc/ntp.conf
echo 'restrict -6 default kod nomodify notrap nopeer noquery' >> /etc/ntp.conf
echo 'restrict 127.0.0.1' >> /etc/ntp.conf
echo 'restrict -6 ::1' >> /etc/ntp.conf
echo 'server 192.168.0.1' >> /etc/ntp.conf
echo 'server 127.127.1.0' >> /etc/ntp.conf
echo 'fudge 127.127.1.0 stratum 10' >> /etc/ntp.conf
echo 'driftfile /var/lib/ntp/drift' >> /etc/ntp.conf
chmod 644 /etc/ntp.conf

echo '192.168.0.1' >> /etc/ntp/ntpservers
echo '192.168.0.1' >> /etc/ntp/step-tickers

# modify the /etc/sysconfig/ntp file to add the -x startup option 
# required for Oracle 11gR2
echo 'OPTIONS="-u ntp:ntp -x -p /var/run/ntpd.pid"' > /etc/sysconfig/ntpd
echo 'SYNC_HWCLOCK=no' >> /etc/sysconfig/ntpd
echo 'NTPDATE_OPTIONS=""' >> /etc/sysconfig/ntpd

/usr/sbin/ntpdate 192.168.0.1

chkconfig ntpd on

# remove unnecessary services
chkconfig sendmail off

# printer
chkconfig cups off
chkconfig hplip off

network_config.sh is a bit long but I will post it in it's entirety here. Its primary job is to take input from a file called hostfile.csv and configure DNS, host name, and configure all of the interfaces. Network interfaces can be specified by adapter name (e.g. eth0, eth1, etc) or by MAC address just in case the enumeration isn't quite what you expect. It can also configure bonded interfaces which I am particularly happy with as this can be of significant annoyance getting production servers ready. Host names are defined as part of the dhcp options which I showed in this post. If no match is found, for example if there is no matching entry in hostfile.csv, the script will try to grab whatever IP has been assigned for the install and hard code that to the server. Logs are kept at a location specified on the command line which also happens to be the location of hostfile.csv.

Here is an example of a hostfile.csv entry for your reference when looking through the script.

# cat hostfile.csv
DOMAINSEARCH=example.com example2.com example3.com
# Format -- server_name,[bond|nic] eth# [eth#] IP MASK Primary,gw={gateway},dns={dns1 dns2 etc}
# as long as the server name comes first, the order of the rest doesn't really matter
# Primary is used to determine which interface should be placed in the host file
# an example with multiple bonded interfaces.
server1,bond=bond0 eth0 eth3 192.168.0.10 255.255.255.0 1,gw=192.168.0.1,bond=bond1 eth1 eth2 10.1.1.1 255.255.255.0,dns=192.168.0.254 192.168.1.254
# an example with one bond using MAC addresses
server2,bond=bond0 0050569c25e5 0050569c6cbd 192.168.0.11 255.255.255.0 1,gw=192.168.0.1,dns=192.168.0.254 192.168.1.254
# an example with a single nic
server3,nic=eth0 192.168.0.12 255.255.255.0 1,gw=192.168.0.1,dns=192.168.0.254

And here is the network_config.sh script itself

# cat network_config.sh
#!/bin/bash
DEBUG=off
IFCONFIG=/sbin/ifconfig

NIC_FILE_DIR=/etc/sysconfig/network-scripts/

GW_FILE=/etc/sysconfig/network

HOST_FILE=/etc/hosts

DNS_FILE=/etc/resolv.conf

DOMAIN_LIST="domain.com domain2.com"

####
## function readHostFile
## reads $HOST_MAP_FILE for specific network information about this host
## return 1 on error, 0 on success
## options can be in any order (nic, gw, or bond), broadcast and network address are calculated based on ip and mask
## calls functions to generate Gateway ($GW_FILE), Hosts ($HOST_FILE), and ifcfg ($NIC_FILE_DIR/ifcfg-{nic})
##
## host_map_file format
## {host},nic={eth#} ip mask [?primary],gw={gw_ip},bond={bond#} {nic1} {nic2} {ip} {mask},domain={dns_server},{dns_server}
## e.g. server1,nic=eth0 192.168.1.1 255.255.255.0,gw=192.168.0.1,nic=bond0 eth1 eth2 192.168.0.10 255.255.255.0 1,dns=192.168.0.254 192.168.1.254
####
readHostFile() {
        if [ -e $HOST_MAP_FILE ]
        then
                # override default DOMAIN_LIST if it exists
                DOMAIN_TMP=$(cat $HOST_MAP_FILE | grep -wi "DOMAINSEARCH" | cut -f2 -d =)
                if [ ! -z "$DOMAIN_TMP" ]
                then
                        log info "Domain search list found -- $DOMAIN_TMP"
                        DOMAIN_LIST="$DOMAIN_TMP"
                else
                        log info "Domain search not found, using defaults -- $DOMAIN_LIST"
                fi

                # parse the file for this host exactly (-w) and case insensitive
                HOST_INFO=$(cat $HOST_MAP_FILE | grep -wi `hostname`)
                # check to see there was an entry for this host
                if [ -z "$HOST_INFO" ]
                then
                        log warning "Host information for `hostname` was not found in HOST_MAP_FILE"
                        return 1
                fi
                log notify "Host information found for `hostname` in $HOST_MAP_FILE"
                log notify "Host info is $HOST_INFO"
                # parse HOST_INFO
                IFS=$','
                for entry in $HOST_INFO
                do
                        log debug "Working on entry $entry"
                        KEY=`echo $entry | cut -f1 -d =`
                        VALUE=`echo $entry | cut -f2 -d =`
                        case "$KEY" in
                        nic)
                                log debug "nic is specified -- $VALUE"
                                NIC=`echo $VALUE | cut -f1 -d " "`
                                if [ ${#NIC} -eq 12 ]
                                then
                                        # we are working with a MAC address
                                        NIC=$(getNIC $NIC)
                                fi
                                IPADDR=`echo $VALUE | cut -f2 -d " "`
                                MASK=`echo $VALUE | cut -f3 -d " "`
                                PRIMARY=`echo $VALUE | cut -f4 -d " "`
                                BROADCAST=$(getBroadcastAddress $IPADDR $MASK)
                                NETWORK=$(getNetworkAddress $IPADDR $MASK)
                                # MAC address for this card
                                MAC=$(getMAC $NIC)

                                if [ -z $NIC ]
                                then
                                        log error "Missing NIC information aborting file creation"
                                else
                                        log info "Values for NIC $NIC - MAC $MAC - IP $IPADDR - NetMask $MASK - Broadcast $BROADCAST - Network $NETWORK"
                                        genIPFile $NIC $MAC $IPADDR $MASK $BROADCAST $NETWORK
                                fi

                                if [ "$PRIMARY" == 1 ]
                                then
                                        genHostFile $IPADDR
                                fi
                                ;;
                        bond)
                        #nic=bond0 eth1 eth2 192.168.0.10 255.255.255.0 1
                                log debug "bond is specified -- $VALUE"
                                BOND=`echo $VALUE | cut -f1 -d " "`
                                NIC1=`echo $VALUE | cut -f2 -d " "`
                                if [ ${#NIC1} -gt 12 ]
                                then
                                        # we are working with a MAC address
                                        NIC1=$(getNIC $NIC1)
                                fi
                                NIC2=`echo $VALUE | cut -f3 -d " "`
                                if [ ${#NIC2} -gt 12 ]
                                then
                                        # we are working with a MAC address
                                        NIC2=$(getNIC $NIC2)
                                fi
                                IPADDR=`echo $VALUE | cut -f4 -d " "`
                                MASK=`echo $VALUE | cut -f5 -d " "`
                                BROADCAST=$(getBroadcastAddress $IPADDR $MASK)
                                NETWORK=$(getNetworkAddress $IPADDR $MASK)

                                log info "Values for BOND $BOND - NIC1 $NIC1 - NIC2 $NIC2 - IP $IPADDR - NetMask $MASK - Broadcast $BROADCAST - Network $NETWORK"
                                genBondFile $BOND $NIC1 $NIC2 $IPADDR $MASK $BROADCAST $NETWORK

                                if [ "$PRIMARY" == 1 ]
                                then
                                        genHostFile $IPADDR
                                fi
                                ;;
                        gw)
                                log debug "Gateway value - $VALUE"
                                genGWFile $VALUE
                                ;;
                        dns)
                                log debug "DNS is specified -- $VALUE"
                                genDNSFile "$VALUE"
                        esac
                done
        else
                log warning "Hostfile $HOST_MAP_FILE does not exist"
                return 1
                # configure eth0 as static based on the current DHCP address
        fi
}

####
## function getNIC {mac_addr}
## returns eth# based on MAC address
####
getNIC() {
        local RAW_MAC=$1
        # a properly formatted MAC address is 00:10:20:30:40:50 (17 characters)
        if [ ${#RAW_MAC} -ne 17 ]
        then
                # assume the user didn't put in : marks
                COUNT=0
                # in case this is IPv6 loop for the entire raw mac length
                while [ $COUNT -lt ${#RAW_MAC} ]
                do
                        if [ $COUNT -eq 0 ]
                        then
                                SEARCH_MAC=${RAW_MAC:$COUNT:2}
                        else
                                SEARCH_MAC="$SEARCH_MAC:${RAW_MAC:$COUNT:2}"
                        fi
                        COUNT=$(($COUNT + 2))
                done
        else
                SEARCH_MAC=$RAW_MAC
        fi

        # return eth# for a specific MAC
        local NIC=`$IFCONFIG -a | grep -i $SEARCH_MAC | awk '{print $1}'`
        if [ -z $NIC ]
        then
                log error "Network interface was not found for nic $SEARCH_MAC, this interface will not be configured correctly"
                log error "ifconfig output is \n`$IFCONFIG -a`"
        else
                log info "NIC $SEARCH_MAC found as $NIC"
        fi
        echo $NIC
}

####
## function genBondFile {bond#} {nic1} {nic2} {ip} {mask} {broadcast} {network}
## nic=bond0 eth0 eth1 192.168.0.10 255.255.255.0 
## nic=eth0 192.168.0.10 255.255.255.0 192.168.0.254 192.168.0.0
####
genBondFile() {
        local BOND=$1
        local NIC1=$2
        local NIC2=$3
        local IP=$4
        local MASK=$5
        local BROADCAST=$6
        local NETWORK=$7
        local BOND_FILE=${NIC_FILE_DIR}ifcfg-$BOND
        local NIC1_FILE=${NIC_FILE_DIR}ifcfg-$NIC1
        local NIC2_FILE=${NIC_FILE_DIR}ifcfg-$NIC2

        log info "Creating Bond file $BOND_FILE"
        echo "DEVICE=$BOND" > $BOND_FILE
        echo "BOOTPROTO=none" >> $BOND_FILE
        echo "ONBOOT=yes" >> $BOND_FILE
        echo "NETWORK=$NETWORK" >> $BOND_FILE
        echo "NETMASK=$MASK" >> $BOND_FILE
        echo "IPADDR=$IP" >> $BOND_FILE
        echo "BROADCAST=$BROADCAST" >> $BOND_FILE
        echo "USERCTL=no" >> $BOND_FILE
        echo "BONDING_OPTS=\"mode=active-backup miimon=100 primary=$NIC1\"" >> $BOND_FILE

        log info "Creating network file $NIC1_FILE"
        echo "DEVICE=$NIC1" > $NIC1_FILE
        echo "BOOTPROTO=none" >> $NIC1_FILE
        echo "HWADDR=$(getMAC $NIC1)" >> $NIC1_FILE
        echo "ONBOOT=yes" >> $NIC1_FILE
        echo "MASTER=$BOND" >> $NIC1_FILE
        echo "SLAVE=yes" >> $NIC1_FILE
        echo "USERCTL=no" >> $NIC1_FILE

        log info "Creating network file $NIC2_FILE"
        echo "DEVICE=$NIC2" > $NIC2_FILE
        echo "BOOTPROTO=none" >> $NIC2_FILE
        echo "HWADDR=$(getMAC $NIC2)" >> $NIC2_FILE
        echo "ONBOOT=yes" >> $NIC2_FILE
        echo "MASTER=$BOND" >> $NIC2_FILE
        echo "SLAVE=yes" >> $NIC2_FILE
        echo "USERCTL=no" >> $NIC2_FILE

        log info "Modifying modprobe.conf file /etc/modprobe.conf"
        echo "alias $BOND bonding" >> /etc/modprobe.conf
}

####
## function getMAC {nic}
## gets the MAC address for a given interface using ifconfig
####
getMAC() {
        HWINFO=`$IFCONFIG $1 | grep HWaddr` # eth0      Link encap:Ethernet     HWaddr 00:50:56:9C:1B:00
        if [ $? -ne 0 ]
        then
                log error "Cannot find MAC address for interface $1"
                # return nothing to the calling process
                echo " "
        else
                # return the MAC address 
                echo $HWINFO | awk '{print $5}'
        fi
}

####
## function genDomainFile {nameserver} {nameserver} {etc}
## creates a basic DNS file for nameserver entries
####
genDNSFile() {
        log info "Creating DNS file $DNS_FILE"
        OldIFS=$IFS
        IFS=" "
        > $DNS_FILE
        # create search entries
        echo "search $DOMAIN_LIST" >> $DNS_FILE
        # create server entries
        for dnsEntry in $1
        do
                echo "nameserver $dnsEntry" >> $DNS_FILE
        done
        IFS=$OldIFS
}
####
## function genHostFile {local_ip}
## creates a basic hosts file with loopback and this host
####
genHostFile() {
        local IP=$1
        log info "Creating host file $HOST_FILE"
        echo "127.0.0.1         localhost.localdomain localhost" > $HOST_FILE
        echo "$IP               `hostname`" >> $HOST_FILE

}

####
## function genGWFile {gateway_ip}
## create the default route file including default RedHat values
####
genGWFile() {
        local GW=$1
        log info "Creating gateway file $GW_FILE"
        echo "NETWORKING=yes" > $GW_FILE
        echo "NETWORKING_IPV6=no" >> $GW_FILE
        echo "HOSTNAME=`hostname`" >> $GW_FILE
        echo "GATEWAY=$GW" >> $GW_FILE
}

####
## function genIPFile {nic} {mac} {ip} {mask} {broadcast} {network}
## create the IP Address file (ifcfg-eth{x})
## e.g. nic=eth0 00:50:56:9C:1B:00 192.168.0.10 255.255.255.0 192.168.0.254 192.168.0.0
####
genIPFile() {
        local NIC=$1
        local MAC=$2
        local IP=$3
        local MASK=$4
        local BROADCAST=$5
        local NETWORK=$6
        local IP_FILE=${NIC_FILE_DIR}ifcfg-${NIC}

        log info "Creating network file $IP_FILE"
        echo "DEVICE=$NIC" > $IP_FILE
        echo "BOOTPROTO=static" >> $IP_FILE
        echo "BROADCAST=$BROADCAST" >> $IP_FILE
        echo "HWADDR=$MAC" >> $IP_FILE
        echo "IPADDR=$IP" >> $IP_FILE
        echo "NETMASK=$MASK" >> $IP_FILE
        echo "NETWORK=$NETWORK" >> $IP_FILE
        log debug "----------- ifcfg-$NIC file -----------"
        log debug "\n`cat $IP_FILE`"
        log debug "----------------------"
}

####
## function getNetworkAddress
## calculates the network address given an ip and subnet mask
## converts the ip and mask into an array and does a bitwise and for each element
####
getNetworkAddress() {
        OldIFS=$IFS
        IFS=.
        typeset -a IP_Array=($1)
        typeset -a MASK_Array=($2)
        IFS=$OldIFS
        echo $((${IP_Array[0]} & ${MASK_Array[0]})).$((${IP_Array[1]} & ${MASK_Array[1]})).$((${IP_Array[2]} & ${MASK_Array[2]})).$((${IP_Array[3]} & ${MASK_Array[3]}))
}

####
## function getBroadcastAddress
## calculates the broadcast address given an ip and subnet mask
## converts the ip and mask into an array and does a bitwise or (|) against an XOR (^)
####
getBroadcastAddress() {
        OldIFS=$IFS
        IFS=.
        typeset -a IP_Array=($1)
        typeset -a MASK_Array=($2)
        IFS=$OldIFS
        echo $((${IP_Array[0]} | (255 ^ ${MASK_Array[0]}))).$((${IP_Array[1]} | (255 ^ ${MASK_Array[1]}))).$((${IP_Array[2]} | (255 ^ ${MASK_Array[2]}))).$((${IP_Array[3]} | (255 ^ ${MASK_Array[3]})))
}

####
## function readDHCPAddress
## reads information currently running and writes it out as a static IP entry
####
readDHCPAddress() {
        log info "Host information was not found for this server, copying information from running configuration (DHCP)"
        # the grep will grab two lines of output and merge them together
        # eth0      Link encap:Ethernet  HWaddr 00:50:56:9C:1B:00
        # inet addr:192.168.0.10  Bcast:192.168.0.254  Mask:255.255.255.0
        HWINFO=`$IFCONFIG | grep -A 1 -i hwaddr`
        NIC=`echo $HWINFO | cut -f1 -d " "`
        MAC=`echo $HWINFO | cut -f5 -d " "`
        for i in $HWINFO
        do
                case "$i" in
                addr:*)
                        IP=`echo $i | cut -f2 -d :`
                        ;;
                Bcast:*)
                        BROADCAST=`echo $i | cut -f2 -d :`
                        ;;
                Mask:*)
                        MASK=`echo $i | cut -f2 -d :`
                        ;;
                esac
        done
        NETWORK=$(getNetworkAddress $IP $MASK)
        log debug "DHCP information is NIC $NIC - MAC $MAC - IP $IP - MASK $MASK - BROADCAST $BROADCAST - NETWORK $NETWORK"
        genIPFile $NIC $MAC $IP $MASK $BROADCAST $NETWORK
        genHostFile $IP
        GATEWAY=`netstat -rn | grep -w UG | awk '{print $2}'`
        genGWFile $GATEWAY
}

####
## function log
## logs activities to the screen, a file, or both
####
log() {
        LOG_TYPE="$1"
        LOG_MSG="$2"
        TIME=`date +'%H:%M:%S %Z'`
        # specify the log file only once
        if [ ! -d $SOURCE_DIR/logs ]
        then
                mkdir ${SOURCE_DIR}/logs
        fi
        if [ -z $LOG_FILE ]
        then
                LOG_FILE="$SOURCE_DIR/logs/network_config-`hostname`-`date +%Y%m%d-%H%M%S`"
        fi
        if [ $LOG_TYPE == "error" ]
        then
                echo -e "$TIME - **ERROR** - $LOG_MSG" >> $LOG_FILE
        elif [ $LOG_TYPE == "debug" ]
        then
                if [ $DEBUG == "on" ]
                then
                        echo -e "DEBUG - $LOG_MSG" >> "$LOG_FILE"
                fi
        elif [ $LOG_TYPE == "warning" ]
        then
                echo -e "$TIME - **WARNING** - $LOG_MSG" >> $LOG_FILE
        else
                echo -e "$TIME - $LOG_MSG" >> "$LOG_FILE"
        fi
}

# read source directory from command line.  This is where we will read the hostfile.csv and output logs to
SOURCE_DIR=$1
HOST_MAP_FILE=$SOURCE_DIR/hostfile.csv

readHostFile
if [ $? -ne 0 ]
then
        readDHCPAddress
fi

Modular KickStart

I am a big fan of the modular kickstart setup. This allows easier administration of multiple configurations and tends to keep things clean and consistent. KickStart allows this through the %include directive but in order to reference an external file, a small trick is required. You see, the kickstart configuration file is in fact read twice. Once looking for any %pre directives and another to actually parse the file. So, it's in this %pre section that you can setup your config file location.

I like to use NFS for my configuration and OS files. It is relatively easy to setup and can allow multiple kickstart servers to reference it if required. Here is a sample pre section:

%pre
mkdir /post_scripts
mount -t nfs -o nolock NFS_SERVER:/tftpboot/kickstart /post_scripts

Basically it says to mount a share to /post_scripts so I can reference the files. On the server I place my configuration files in /tftpboot/kickstart/ks_cfg/ and call them like this:

%include /post_scripts/ks_cfg/CONFIG_FILE

Here are the basic sections of my kickstart file:

# cat /tftpboot/ks_cfg/oracle_5.5.ks
install
nfs --server=NFS_SERVER --dir=/tftpboot/kickstart/OS/RHEL5.5/
key --skip
lang en_US.UTF-8
keyboard us
xconfig --startxonboot
network --bootproto dhcp 
rootpw --iscrypted YOUR_PASSWORD
firewall --disabled
firstboot --disable
authconfig --enableshadow --enablemd5
selinux --disabled
timezone --utc America/Vancouver
bootloader --location=mbr --append="rhgb quiet"
clearpart --all --initlabel
part /boot --fstype="ext3" --size=100
part pv.2 --size=0 --grow
volgroup rootvg --pesize=32768 pv.2
logvol swap --fstype="swap" --name=swap --vgname=rootvg --size=2048
logvol /var --fstype="ext3" --name=var --vgname=rootvg --size=2048
logvol / --fstype="ext3" --name=root --vgname=rootvg --size=1 --grow
reboot

%packages
%include /post_scripts/ks_cfg/packages.cfg

%post
chvt 3
%include /post_scripts/ks_cfg/generic.cfg
%include /post_scripts/ks_cfg/oracle.cfg
%include /post_scripts/ks_cfg/multipath.cfg

%pre
mkdir /post_scripts
mount -t nfs -o nolock NFS_SERVER:/vol/KICKSTART /post_scripts

# cat /tftpboot/ks_cfg/packages.cfg
@editors
@gnome-desktop
@core
@base
@ftp-server
@network-server
@java
@legacy-software-support
@base-x
@server-cfg
@admin-tools
@graphical-internet
emacs
kexec-tools
fipscheck
device-mapper-multipath
dnsmasq
xorg-x11-utils
system-config-boot
# Oracle Required Packages
elfutils-libelf-devel
gcc
gcc-c++
glibc-devel
libaio-devel
libstdc++-devel
sysstat
# these are for Oracle 10G
libXp
openmotif
# Oracle 11GR2
unixODBC
unixODBC-devel

I'll cover off some of the more 'advanced' features in a later post.