Reclaiming VMDK Space

Unless you are running Microsoft Cluster services, and with VMware HA I'm not sure why you would, I can't really see a reason not to thin provision. However, as capacity within the guest is consumed, the size of the VMDK files increases. Even once data is cleaned up, that VMDK space is never reclaimed. Virtual Center shows this as "Provisioned Storage" vs "Used Storage" as shown in the screen shot below.

In this case the VMDK is 16GB and I'm currently using 13.18GB on disk. When we take a look at the host we can see it has only a couple of GB consumed.

# df -h
Filesystem   Size Used Avail Use% Mounted on
/dev/mapper/rootvg-root  12G 1.9G 9.3G 17% /
tmpfs    499M 0 499M 0% /dev/shm
/dev/sda1   95M 43M 48M 47% /boot
/dev/mapper/rootvg-var  2.0G 117M 1.8G 7% /var

The easiest way I know to fix the problem is to storage vmotion the guest to another datastore and after, if you like, to move it back. The trick is, you have to zero out the extra space in the file system first in order for vmware to thin provision it.

This is pretty easy, although it can take a few minutes depending on how much space you need to 'fill'. The following will create a 9GB zero filled file, flush changes to disk, and then remove it. You could of course fill the entire file system but this could impact running applications, so I'll leave that up to you to decide.

# dd if=/dev/zero of=/fill_file bs=1024k count=9216; sync; rm /fill_file

Now your free space in the virtual disk is filled with zeros. All that's left is to storage vmotion the VMDK to another datastore. I only have experience with NFS, in which case I can selece "Same format as source", if you are using VMFS you should probably select "Thin provisioned format".

Once completed you'll see the used capacity back in line with what the host is actually using.

Misaligned I/O reporting

In the last post I showed a script for logging misaligned I/Os on a NetApp storage array. However, it is nice to produce some graphs of the data, so here is my quick parse script to crank out a CSV file for a given filer. If you recall the output from the collection script was as follows:

22:10:01 UTC - Zeroing stats from filer np00003
22:10:02 UTC - sleeping for 300 seconds
22:15:02 UTC - Capturing data from filer np00003
22:15:03 UTC - Collected Values:
22:15:03 UTC - interval = 300.399491 seconds, pw.over_limit = 249 and WAFL_WRITE = 540822
22:15:03 UTC - Percentage Misaligned Writes = .0460%
22:15:03 UTC - Successfully Completed!

My log structure is kept under ./NetApp_Logs/ followed by a file for each log like so, NetApp-align-{filer}-{yyyymmdd}-{hhmmss}. This little script will walk through all of the logs for a given filer and output a CSV as shown here:

2011-10-10,22:15:03,.0460%
2011-10-11,10:15:03,.0670%
2011-10-11,22:15:03,.1500%
2011-10-12,20:15:03,.4500%

I have found this works well for spread sheet programs as it is able to parse out both the date and time appropriately, saving me a lot of time. And here is the script:

$ cat na_parse_alignment
if [ "$#" == 0 ]
then
        echo "Usage: `basename $0` "
        echo "e.g."
        echo "`basename $0` {filer}"
        exit 0
fi
FILER=$1
grep -H Percent NetApp_Logs/NetApp-align-${FILER}-*  | sed -e 's/\(2011\)\([0-9].\)\([0-9].\)\(-[0-9].....:\)/ \1-\2-\3 /g' | awk '{print $2" "$3" "$NF}' | tr " " , 

Tracking Misaligned I/Os

For those that don't know about misaligned I/Os, I provided a brief introduction to them in my last post. In this post I'll show you how to track and quantify how many of your I/Os exhibit the problem. We currently run our VMware infrastructure from a NetApp array, so this script is tailored to that environment.

The only method I've found to track is by using the pw.over_limit counter. Unfortunately it is only available with advanced privileges on the command line as it isn't exported via SNMP. You can manually obtain the data with the following:

# priv set advanced
# wafl_susp -w
# priv set

This will produce lots of data, most of which you can ignore. You'll quickly notice a problem; pw.over_limit is an absolute number, like 29300. So what? Is that good, is it bad? That depends on how many I/Os you are generating in the first place. Since I love writing little programs, here is the output of my script:

$ cat NetApp-align-filer1-20111010-221001
22:10:01 UTC - Zeroing stats from filer filer1
22:10:02 UTC - sleeping for 300 seconds
22:15:02 UTC - Capturing data from filer filer1
22:15:03 UTC - Collected Values:
22:15:03 UTC - interval = 300.399491 seconds, pw.over_limit = 249 and WAFL_WRITE = 540822
22:15:03 UTC - Percentage Misaligned Writes = .0460%
22:15:03 UTC - Successfully Completed! 

As you can see, there are a few misaligned writes going on, but overall in pretty good health with just under 0.05% of the total. When should you panic? That depends on way to many variables to list here but your write latency is a key indicator to pay attention to. Your application mix (and users) will scream when you've crossed the line.

The code I use is listed below. Read the comments in the parse_wafl function to figure out what it's doing.

#!/bin/bash
# Script Information
# na_alignment_check Version: 1.0
# Last Modified: Sept 18/2010
# Created By: Michael England

# to run this script you will need an account on the filer with administrative privileges as it has to be run with 'priv set advanced'
# ssh prep work
# ssh-keygen -t dsa -b 1024
# cat id_dsa.pub >> {filer}/etc/sshd/{user}/.ssh/authorized_keys

# default to local user
FILER_USER=$(whoami)

######
### function log
### logs activities to the screen, a file, or both
######
function log {
 LOG_TYPE="$1"
 LOG_MSG="$2"
 TIME=`date +'%H:%M:%S %Z'`
 # specify the log file only once
 if [ -z $LOG_FILE ]
 then
  LOG_FILE="/work/Scripts/NetApp_Logs/NetApp-align-$FILER-`date +%Y%m%d-%H%M%S`"
 fi
 if [ $LOG_TYPE == "error" ]
 then
  echo -e "$TIME - **ERROR** - $LOG_MSG"
  echo -e "$TIME - **ERROR** - $LOG_MSG" >> $LOG_FILE
 elif [ $LOG_TYPE == "debug" ]
 then
  if [ $DEBUG == "on" ]
  then
   echo -e "DEBUG - $LOG_MSG"
  fi
 else
  echo -e "$TIME - $LOG_MSG"
  echo -e "$TIME - $LOG_MSG" >> "$LOG_FILE"
 fi
}

######
### check_ssh
### check the return code of an ssh command
######
function check_ssh {
 CHECK=$1
 ERROR_DATA="$2"
 if [ $CHECK != 0 ]
 then
  log error "ssh failed to filer"
  log error "return is $ERROR_DATA"
  exit 1
 fi
}

######
### capture_wafl_susp
### ssh to the filer specified and collect the output from wafl_susp
######

function capture_wafl_susp {
 log notify "Capturing data from filer $FILER"
 SSH_RET=`ssh $FILER -l $FILER_USER "priv set advanced;wafl_susp -w;priv set" 2>&1`
 RETVAL=$?
 check_ssh $RETVAL "$SSH_RET"

 parse_wafl $SSH_RET
}

######
### parse_wafl
### capture values for pw.over_limit and WAFL_WRITE and the overall scan interval
### e.g.
### WAFL statistics over 577455.189585 second(s) ...
### pw.over_limit = 29300
### New messages, restarts, suspends, and waffinity completions (by message-type):
### WAFL_WRITE           = 10568010   122597   122597        0
### 
### There are many other WAFL_WRITE lines so we need to find the New messages line first then the WAFL_WRITE in that section
######
function parse_wafl {
 oldIFS=$IFS
 IFS=$'\n'
 NEW_FLAG=0
 for line in echo $SSH_RET
 do
  if [[ $line =~ second* ]]
  then
   STATS_INTERVAL=`echo $line | awk '{print $4}'`
  fi

  if [[ $line =~ pw.over_limit* ]]
  then
   OVER_LIMIT=`echo $line | awk '{print $3}'`
  fi

  if [[ $line =~ New\ messages.* ]]
  then
   NEW_FLAG=1
  fi
  if [[ $NEW_FLAG == 1 ]] && [[ $line =~ WAFL_WRITE* ]]
  then
   WAFL_WRITE=`echo $line | awk '{print $3}'`
   NEW_FLAG=0
  fi
 done
 IFS=$oldIFS
 if [[ -n $OVER_LIMIT ]] && [[ $WAFL_WRITE -gt 0 ]]
 then 
  # by multiplying by 100 first we don't loose any precision
  MISALIGNED_PERCENT=`echo "scale=4;100*$OVER_LIMIT/$WAFL_WRITE" | bc -l`
 else
  log error "Error collecting values, pw.over_limit = $OVER_LIMIT and WAFL_WRITE = $WAFL_WRITE"
 fi
 
 log notify "Collected Values:"
 log notify "interval = $STATS_INTERVAL seconds, pw.over_limit = $OVER_LIMIT and WAFL_WRITE = $WAFL_WRITE"
 log notify "Percentage Misaligned Writes = ${MISALIGNED_PERCENT}%"
}

######
### function zero_values
### zeroes out existing wafl stats on the filer
######
function zero_values {
 log notify "Zeroing stats from filer $FILER"
 SSH_RET=`ssh $FILER -l $FILER_USER "priv set advanced;wafl_susp -z;priv set" 2>&1`
 RETVAL=$?
 check_ssh $RETVAL "$SSH_RET"
}

######
### function usage
### simple user information for running the script
######
function usage {
 echo -e ""
 echo "Usage:"
 echo "`basename $0` -filer {filer_name} [-username {user_name}] [-poll_interval ]"
 echo -e "\t-filer {file_name} is a fully qualified domain name or IP of a filer to poll"
 echo -e "\t-username {user_name} is a user to attach to the filer, if omitted will use current user"
 echo -e "\t-poll_interval {x} will zero out the filer stats and sleep for {x} seconds then return.  If omitted will read stats since last zeroed"
 echo -e ""
 exit 0
}

# parse command line options
if [ $# == 0 ]
then
 usage
fi
until [ -z "$1" ]
do
 case "$1" in
 -filer)
  shift
  FILER="$1"
  ;;
 -username)
  shift
  FILER_USER="$1"
  ;;
 -poll_interval)
  shift
  POLL_INTERVAL="$1"
  ;;
 *)
  usage
  ;;
 esac
 shift
done
# do the work

if [[ -n $POLL_INTERVAL ]]
then
 zero_values
 log notify "sleeping for $POLL_INTERVAL seconds"
 sleep $POLL_INTERVAL
fi
capture_wafl_susp

log notify "Successfully Completed!"

Misaligned I/Os

While not unique to virtualization, it generally doesn't cause much of a problem until you consolidate a whole bunch of poorly setup partitions onto one array that things tend to go from all right to a really bad day. The fundamental problem is a mismatch between where the OS places data and where the storage array ultimately keeps it. Both work in logical chunks of data and both present a virtual view of this to the higher layers. There are two specific cases that I'd like to address, one that you can fix, and one that you can only manage.

Storage Alignment
Lets start with a simple illustration of the problem.

Most legacy operating systems (Linux included) like to include a 63 sector offset at the beginning of a drive. This is a real problem as now every read and write overlaps the block boundaries of a physical array. It doesn't matter if you are using VMFS or NFS to host a data store, it's the same problem. Yes an NFS repository will always be aligned, but remember this is a virtual representation to the OS, which happily messes everything up by offsetting its first partition.

Alignment is bad enough when we read. The storage array will pull two blocks when one is read, but it is of particular importance when we write data. Most arrays use some sort of parity to manage redundancy, and if you need to deal with two blocks for every one write request, the system overhead can be enormous. It's also important to keep in mind that every storage vendor has this issue. Even a raw, single drive can benefit from aligned partitions, especially when we consider most new drives ship with a 4KB sector size called Advanced Format.

The impact to each vendor will be slightly different. For example, EMC uses a 64KB block size, so not every write will be unaligned. NetApp uses a 4KB block, which means every write will be unaligned but they handle writes quite a bit differently as the block doesn't have to go back to the same place it came from. Pick your poison.

As you can see, when the OS blocks are aligned, everything through the stack can run at its optimal rate, where one OS request translates to one storage request.

Correctable Block Alignment
Fortunately most modern operating systems have recognized this problem and there is little to do. For example Windows 2008 now uses a 2048 cylinder offset (1MB) as do most current Linux distributions. For Linux, it is easy enough to check.

# fdisk -lu /dev/sdb

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x77cbefef

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3907029167  1953513560   83  Linux

As you can see from this example, my starting cylinder is 2048. After that everything will align, including subsequent partitions. The -u option tells fdisk to display cylinders. I generally recommend this option when creating the partition as well although this seems to be the default for fdisk 2.19. You can also check your file system block size to better visualize how this relates through the stack. The following shows that my ext4 partition is using a 4KB block size:

# dumpe2fs /dev/sdb1 | grep -i 'block size'
dumpe2fs 1.41.14 (22-Dec-2010)
Block size:               4096

Uncorrectable Block Alignment
VMware has come out with a nifty utility I first saw in VDI (now called View) and later placed into their cloud offering called a linked clone. Basically it allows you to create a copy of a machine using very little disk space quickly because it reads common data from the original source and writes data to a new location. Sounds a lot like snap shots doesn't it?

Well the problem with this approach is that every block written requires a little header to tell VMware where this new block belongs in the grand scheme of things. This is similar to our 63 cylinder offset but now for every block, nice. It's a good idea to start your master image off with an aligned file system as it will help with reading data but doesn't amount to much when you write. And does a windows desktop ever like to write. Just to exist (no workload), our testing has shown windows does 1-2 write IOs per second. Linux isn't completely off the hook, but generally does 1/3 to 1/2 of that and isn't that common with linked clones yet as it isn't supported in VDI but will get kicked around in vCloud.

Managing Bad Block Alignment

When using linked clones, there are a few steps you can take to minimize the impact:

• Refresh your images as often as you can. This will keep the journal file to a minimum and corresponding system overhead. If you can't refresh images, you probably shouldn't be using linked clones.
• Don't turn on extra features for those volumes like array based snapshots or de-duplication. The resources needed to track changes for both of these features can cause significant overhead. Use linked clones or de-dupe, not both.
• Monitor your progress on a periodic basis. I do this 3 times a day so we can track changes over time. If you can't measure it, you can't fix it.

In the future VAAI is promised to save us by both VMware and every storage vendor that can spell. It's intent is to perform the same linked clone api call but let the storage array figure out the best method of managing the problem. I've yet to see it work in practice, it's still "in the next release", but I have hope.

Firefox hang with FilerView

Lately I've been having lots of problems using NetApp's FilerView with current versions of Firefox (I'm using version 7 now). I thought it was something specific to Linux, but when I upgraded Firefox on my windows machine it started happening there too. The basic symptom is the browser just hangs. When it hangs is a bit random. Invariably it would be at some step in one of the wizards or if there are pages with lots of check boxes. In windows you can just switch to IE, but I don't use windows on a day to day basis. So, the work around I have in place now is a configuration setting in Firefox.

1. In the address bar, type 'about:config'
2. Type 'html5' in the Filter bar
3. Set 'html5.parser.enable' to false

That's it, FilerView should now work properly again. I don't run into too many html5 based sites so I'm not entirely sure how much this breaks. Hopefully NetApp and Firefox can get along at some point soon.

Left Over .nfs (dot nfs) Files

I recently had a situation where an NFS file system was constantly filling with these strange .nfs files. If a file is removed while a running process still has it open, that file is renamed as .nfs and a long hex string. The symptoms are fairly easy to reproduce:

# touch testfile
# tail -f testfile
From another session:
# rm testfile
# ls -la
-rw-r--r-- 1 root    root    0 Jul 23 13:31 .nfs000000000033468f00000003

To find the offending process run an lsof .nfsxxxxx and kill it, however, because NFS mounts can be spread across several clients, it may take a bit of searching to find the right one. Once the process is terminated the client should automatically clean the file.

I should also point out that in my testing, both the process and the delete operation have to come from the same client. NFS doesn't enforce any file locking which means if a file is deleted from another machine the system doesn't know to rename it first. It is left up to the application to sort this out. Most do nothing about it which means you will get a stale NFS handle message on the source process.

Linux Logical Volume Management

There are a few reasons for using Logical Volume Management; extending the capacity of a file system beyond the available physical spindles by spanning disks, using it to have more dynamic control over disk capacity for example by adding or removing a drive, or to create backups in the form of snapshots. LVM can be applied against any block device such as a physical drive, software raid, or external hardware raid device. The file system is still separate, however, it must be managed in conjunction with LVM to make use of the available block appropriately.

In general there are three basic components:

Physical Disk

• Initially, each drive is simply marked as available for use in a volume group. This writes a Universally Unique Identifier (UUID) to the initial sectors of the disk and prepares it to receive a volume group
  

Volume Group

• A collection of physical disks (or partitions if desired). When created this will designate physical extents to all of its member disks, the default being 4MB. It will also record information about all other physical disks in the group and any logical volumes present.

Logical Volume

• Most of the work happens at this layer. A logical volume is a mapping between a set of physical extents (PE) from the disk to a set of logical extents (LE). The size of these are always the same and generally the quantity matches one to one. However, it is possible to have two PEs mapping to one LE if mirroring is used.

In the example shown, there is one volume group with two physical drives and two logical volumes mapped. Physical blocks that are not assigned to a logical drive are free and can be used to expand either logical drive at a later time.



Creating a Logical Drive
I a not going to bother with mirrored or stripped volumes. You could make a case for a stripe to increase performance, however, in general I believe it is better to use either the hardware or software raid functions available as they are better suited for that purpose. The steps are fairly simple, mark the device with pvcreate, create a volume group and then assign a logical volume. Depending on how big your volume group is, you may want to consider altering the default physical extent size. The man page for vgcreate states, if the volume group metadata uses lvm2 format those restrictions [65534 extents in each logical volume] do not apply, but having a large number of extents will slow down the tools but have no impact on I/O performance to the logical volume. So if I was creating a terabyte or larger volume, its probably a good idea to increase this to 64MB or even 128MB.

# pvcreate /dev/sdb
 No physical volume label read from /dev/sdb
 Physical volume "/dev/sdb" successfully created
# pvcreate /dev/sdc
 No physical volume label read from /dev/sdc
 Physical volume "/dev/sdc" successfully created
# vgcreate -s 16M datavg /dev/sdb /dev/sdc
 Volume group "datavg" successfully created
# pvdisplay /dev/sdb
 --- Physical volume ---
 PV Name  /dev/sdb
 VG Name  datavg
 PV Size  10.00GB / not Usable 16.00MB
 Allocatable         Yes
 PE Size (KByte) 16384
 Total PE  639
 Free PE  639
 Allocated PE         0
 PV UUID  apk7wQ-V9B2-vHVo-L5Yz-81U0-orx7-F8J0MI
# vgdisplay datavg
 --- Volume group ---
 VG Name         datavg
 System ID
 Format                 lvm2
 Metadata Areas         2
 Metadata Sequence No 1
 VG Access  read/write
 VG Status  resizable
 MAX LV          0
 Cur LV          0
 Open LV  0
 Max PV          0
 Cur PV          2
 Act PV          2
 VG Size  19.97GB
 PE Size  16.00MB
 Total PE  1278
 Alloc PE / Size 0 / 0
 Free PE / Size         1278 / 19.97GB
 VG UUID  Glyv9C-qRog-YZVk-08nR-csMe-quMp-A3Ksby

As you can see in this example, the volume group named datavg has two member disks each 10GB in size. I selected a different physical extent size not because I had to, just to show how it is done. You will also notice that the available PE size is one less than the total drive space. This is to accommodate the volume group metadata mentioned earlier. You can actually read this data yourself if you like.

# dd if=/dev/sdb of=vg_metadata bs=16M count=1
# strings vg_metadata

The last step is to create the Logical Volume itself. There are a myriad of options available depending on what you want to accomplish, the important ones are:

-L size[KMGTPE]

• Specifies a size in kilobytes, megabytes, gigabytes, terabytes, petabytes, or exabytes. Let me know if you actually use the last two.

-l size

• Specifies the size in extents. In this case 16MB each. You can also specify as a percentage of either the Volume Group, free space in the volume group, or free space for the physical volumes with %VG, %FREE, or %PVS respectively.

-n string

• Gives a name to your logical volume

-i Stripes

• Number of stripes to use. As I mentioned earlier, you should probably use raid to perform this functionality, but if you must, this should be equal to the number of spindles present in the volume group

-I stripeSize

• The stripe depth in KB to use for each disk

Here is an example for a simple volume, and then a striped volume

# lvcreate -L 5G -n datalv datavg
 Logical volume "datalv" created
# lvdisplay
 --- Logical volume ---
 LV Name  /dev/datavg/datalv
 VG Name  datavg
 LV UUID  fCoaFl-7aQY-CX5U-zDwO-at52-udkI-ke6CZn
 LV Write Access read/write
 # open   0
 LV size  5.00GB
 Current LE  320
 Segments  1
 Allocation  inherit
 Read ahead sectors auto
 - currently set to 1024
 Block device  253:0
# lvcreate -L 10G -i 2 -I 64 -n stripedlv datavg

If you are going to use striped volumes you should probably only use striped as it requires the proper number of blocks free on each physical volume. Once we have a volume we need a file system. For this exercise I am going to use ext4, but you can use what you like.

# mkfs.ext4 /dev/datavg/datalv
# mkdir /data
# mount /dev/datavg/datalv /data

Expanding a Logical Volume

# pvcreate /dev/sdd
# vgextend datavg /dev/sdd
 Volume group "datavg" successfully extended
# lvresize -L 15G /dev/datavg/datalv
 Extending logical volume datalv to 15.00 GB
 Logical volume datalv successfully resized
# resize2fs /dev/datavg/datalv
 resize2fs 1.41.9 (22-Aug-2009)
 Resizing the filesystem on /dev/datavg/datalv to 3932160 (4k) blocks.
 The filesystem on /dev/datavg/datalv is now 3932160 blocks long.

Depending on the state of your file system, you may not be able to expand online. You can check the output of tune2fs to ensure GDT blocks have been set aside, without those you will for sure have to be offline. For example, tune2fs -l /dev/datavg/datalv. You may also get a warning to run e2fsck first. The man page warns of running this on-line, so again you are probably best served by unmounting the file system first. If this was a system disk that generally means dropping back down to single user mode.

Reducing a Logical Volume
Before embarking on this journey, ensure you manage the file system first, which for the ext series anyway, means you have to have it unmounted. Once that is done you can go ahead and shrink the logical volume as shown here.

# umount /data
# resize2fs /dev/datavg/datalv 10g
 resize2fs 1.41.9 (22-Aug-2009)
 Resizing the filesystem on /dev/datavg/datalv to 2621440 (4k) blocks.
 The filesystem on /dev/datavg/datalv is now 2621440 blocks long.
# lvreduce -L 10g /dev/datavg/datalv
 WARNING: Reducing active and open logical volume to 10.00 GB
 THIS MAY DESTROY YOUR DATA (filesystem etc.)                
Do you really want to reduce datalv? [y/n]: y                 
 Reducing logical volume datalv to 10.00 GB                  
 Logical volume datalv successfully resized

Again you may be prompted to check your file system but it's unmounted anyway, so it shouldn't be a problem. If the file system is highly fragmented the resize process can take quite a while so be prepared.

Snap shots
Another benefit of lvm is the ability to take point in time images of your file system. Snaps use a copy of write technology where a block that is about to be overwritten or changed is first copied to a new location and then allowed to be altered. This can cause a performance problem on writes which can compound as more snaps are added so bear that in mind. You will also have to set aside some space within the volume group for this purpose. The amount really depends on how many changes you are making, but 10-20% is probably a good starting point. For this example I am going to use 1G as I don't expect many changes.

# lvcreate -L 1g -s -n datasnap1 /dev/datavg/datalv 
 Logical volume "datasnap" created

Notice the -s entry for snapshot and that the target isn't the volume group but rather the logical volume desired. It appears there is a bug in OpenSuSE that may be present in other distributions. It prevents the snap from being registered with the event monitor, to alert when full or reaching capacity. If you get this message you will have to upgrade both lvm2 and device-mapper packages as it was compiled against the wrong library versions.

OpenSuSE error:
datavg-datasnap: event registration failed: 10529:3 libdevmapper-event-lvm2snapshot.so.2.02 dlopen failed: /lib64/libdevmapper-event-lvm2snapshot.so.2.02: undefined symbol: lvm2_run
datavg/snapshot0: snapshot segment monitoring function failed.

To use your new snap, simply mount it like you would any other file system with mount /dev/datavg/datasnap1 /datasnap. You can view the snap useage through lvdisplay from the allocated to snapshot field.

# lvdisplay /dev/data/datasnap1
 --- Logical volume ---
 LV Name                /dev/datavg/datasnap1
 VG Name                datavg
 LV UUID                82IA4M-Md6s-MEI6-iNPW-6wFb-8pzD-eCQqmS
 LV Write Access        read/write
 LV snapshot status     active destination for /dev/datavg/datalv
 LV Status              available
 # open                 0
 LV Size                20.00 GB
 Current LE             5120
 COW-table size         1.00 GB
 COW-table LE           256
 Allocated to snapshot  68.68%
 Snapshot chunk size    4.00 KB
 Segments               1
 Allocation             inherit
 Read ahead sectors     auto
 - currently set to     256
 Block device           253:0

If the snap reserve space fills completely it will not be deleted but marked invalid and cannot be read from, even if it is currently mounted. Snaps aren't good forever but as a point in time image they can be invaluable for providing specific backup scenarios like quick reference points for database backups. Instead of moving the active file system to tape you can quiesce the database, snap it, and return it to normal operations and then perform a backup from the snapshot.

Moving Volume Groups
A handy utility that I have used many times under AIX is also available under Linux; the ability to move a volume group from one system to the next.

# umount /data
# vgchange -an datavg
 0 logical volume(s) in volume group "datavg" now active
# vgexport datavg
 Volume group "datavg" successfully exported

Shutdown the machine before removing and assigning to another machine.

# pvscan
PV /dev/sdb    is in exported VG datavg [10.00 GB / 0    free]
PV /dev/sdc    is in exported VG datavg [10.00 GB / 1.99 GB free]
Total: 2 [19.99 GB] / in use: 2 [19.99 GB] / in no VG: 0 [0   ]
# vgscan
 Reading all physical volumes.  This may take a while...
 Found exported volume group "datavg" using metadata type lvm2
# vgimport datavg
 Volume group "datavg" successfully imported

You should now be able to mount your file system on the new machine.

Other Commands
Some other important commands for volume management

# lvremove logical_volume_path
 e.g. lvremove /dev/datavg/datasnap1
# pvremove device
 e.g. pvremove /dev/sdd
# pvmove device
 moves data from an existing drive to free extents on other disks in the volume group
 e.g. pvmove /dev/sdc
# vgreduce volume_group device
 removes a device from a volume group
 e.g. vgreduce datavg /dev/sdc
# pvremove device
 removes a physical device from lvm
 e.g. pvremove /dev/sdc

Dynamic Linux Disk

To start with, I am going to assume udev and multipath are setup as per my last post. Udev isn't required for scanning or device naming but it is responsible for permissions and device location (directory). Device naming is actually controlled by the multipath driver, which in a modern Linux distribution is conveniently included in the kernel.

The second assumption is that multipath has the basic setup for your particular storage frame. Now, lets ensure multipath is running and set to start on every boot:

# service multipathd status
  multipathd is stopped

# chkconfig --list multipathd
  multipathd                0:off  1:off  2:off  3:off  4:off  5:off  6:off

# chkconfig multipathd on

# chkconfig --list multipathd
  multipathd                0:off  1:off  2:off  3:on   4:off  5:on   6:off

# service multipathd start

As in the last post, I am dealing with RedHat 5.4 and an EMC CLARiiON array. Without any LUNs allocated multipath -ll should look something like this:

# multipath -ll
sdb: checker msg is "emc_clariion_checker: Logical Unit is umbound or LUNZ"
sdc: checker msg is "emc_clariion_checker: Logical Unit is umbound or LUNZ"
sdd: checker msg is "emc_clariion_checker: Logical Unit is umbound or LUNZ"
sde: checker msg is "emc_clariion_checker: Logical Unit is umbound or LUNZ"

These entries are the four paths available to CX controllers.

Adding Devices

No Existing Devices
I have so far been unable to use the simple scan method on an HBA without any devices at all, so this process will unload and reload the adapter driver. It's a disruptive process on the fibre channel bus but there aren't any devices anyway, so it shouldn't matter.

First find the driver you are using, it is likely either an Emulex (lpfc) or Qlogic (qla). In this example I am using an Emulex card.

# lsmod | grep lpfc
lpfc                     352909    0
scsi_transport_fc        73801     1    lpfc
scsi_mod                 196569    10   scsi_dh,sr_mod,sg,usb_storage,lpfc,scsi_transport_fc,mptsas,mptscsih,scsi_transport_sas,sd_mod

remove the module
# rmmod lpfc

insert the module
# modprobe lpfc

Instead of modprobe, you can also use insmod. The difference being insmod will only load the specified driver and modprobe will load the driver and any dependent drivers.

This will allow the device(s) to show under /dev/ora_rdsk but won't create any multipath entries. To do that we simply run multipath.

# multipath
reload: 36006016015a01900796464949a36df11  DGC,RAID 5
[size=50G][features=1 queue_if_no_path|features=1
queue_if_no_path][hwhandler=1 emc][n/a]
\_ round-robin 0 [prio=2][undef]
 \_ 4:0:1:0 sdc 8:32 [active][ready]
 \_ 5:0:1:0 sde 8:64 [undef][ready]
\_ round-robin 0 [prio=0][undef]
 \_ 4:0:0:0 sdb 8:16 [undef][ready]
 \_ 5:0:0:0 sdd 8:48 [undef][ready]

Existing Devices
If you have at least one Fibre device existing, you can simply rescan the bus. This will not take down the existing devices and is able to operate one path at a time ensuring I/O can continue to flow. You will need to know which host devices are your fibre HBAs. To find that, we can list the know fibre adapters as follows and then issue a scan for each.

# ls -l /sys/class/fc_host
drwxr-xr-x    3    root    root    0    Apr 17 08:50 host4
drwxr-xr-x    3    root    root    0    Apr 17 08:50 host5

# echo "- - -" > /sys/class/scsi_host/host4/scan
# multipath -ll
36006016015a01900e2acd0d4a549df11 dm-3 DGC,RAID 5
[size=8.0G][features=1 queue_if_no_path|features=1
queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=1][active]
 \_ 1:0:0:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:1 sdi 8:128 [active][ready]

# echo "- - -" > /sys/class/scsi_host/host5/scan
# multipath -ll
36006016015a01900e2acd0d4a549df11 dm-3 DGC,RAID 5
[size=8.0G][features=1 queue_if_no_path|features=1
queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=2][enabled]
 \_ 1:0:0:1 sdh 8:112 [active][ready]
 \_ 2:0:0:1 sdj 8:144 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:1 sdi 8:128 [active][ready]
 \_ 2:0:1:1 sdk 8:160 [active][ready]

# ls -lL /dev/ora_rdsk
brw-rw---- 1 root   root 253, 3 Apr 17 15:57   36006016015a01900e2acd0d4a549df11

Renaming Devices
The newly scanned device has a WWID name which isn't terribly useful for something like Oracle as we want udev to apply appropriate permissions. To do this, cut and paste the ID into /etc/multipath.conf so it looks something like this:

multipath {
        wwid        36006016015a01900796464949a36df11
        alias        ora_test
      }

And then remove the old device name and re-import it into multipath

# multipath -f 36006016015a01900796464949a36df11

# multipath
create: ora_test (36006016015a01900796464949a36df11)  DGC,RAID 5
[size=50G][features=1 queue_if_no_path|features=1
queue_if_no_path][hwhandler=1 emc][n/a]
\_ round-robin 0 [prio=2][undef]
 \_ 1:0:1:0 sdc 8:32 [undef][ready]
 \_ 2:0:1:0 sde 8:64 [undef][ready]
\_ round-robin 0 [prio=0][undef]
 \_ 1:0:0:0 sdb 8:16 [undef][ready]
 \_ 2:0:0:0 sdd 8:48 [undef][ready]

# ls \-lL /dev/ora_rdsk
brw-rw---- 1 oracle dba 253, 2 Apr 17 09:44 ora_test

If you get an error "must provide a map name to remove" when running multipath -f, make sure you don't have a shell inside /dev/ora_rdsk directory. Also, be careful not to use multipath -F as that will remove all devices, probably not what you want.

Removing Devices
Before doing the actual removal you will need to note several pieces of information; the multipath device name and all block devices assigned to it. All of which can be obtained from multipath -ll.

# multipath -ll
*ora_test2* (36006016015a01900e2acd0d4a549df11) dm-3 DGC,RAID 5
[size=8.0G][features=1 queue_if_no_path|features=1
queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=2][active]
 \_ 1:0:0:1 *sdh* 8:112 [active][ready]
 \_ 2:0:0:1 *sdj* 8:144 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:1 *sdi* 8:128 [active][ready]
 \_ 2:0:1:1 *sdk* 8:160 [active][ready]

To remove the multipath device

# multipath -f ora_test2

Then remove the appropriate block devices from the system with 'echo 1 >
/sys/block/*dev*/device/delete'

# echo 1 > /sys/block/sdh/device/delete
# echo 1 > /sys/block/sdj/device/delete
# echo 1 > /sys/block/sdi/device/delete
# echo 1 > /sys/block/sdk/device/delete