One of the most compelling features of virtualization for me is the speed and flexibility in provisioning systems. There are a number of logical steps you can perform quite easily in order to get systems up really quickly. Just this week on an average KVM host with 10k RPM disks I brought up a minimal Fedora 13 VM ready to go in DNS on the network in just over 30 seconds: 14 to build, 18 to boot. It still brings a smile to my face. :) So, let's get started.
There are two basic components to a quick provisioning process:
1. Source preparation - This is the piece that takes an existing system and packages it up (what I'll call bundling) such that it can be quickly placed onto the disk of new VMs. As you'll see, I use trusty old tar to do this.
2. VM Build Process - This is the big piece that does "the rest." Everything that a new VM needs will be done here in a completely automated "one click" process. At a high level, MACs and IPs will be assigned, disk device(s) will be allocated, filesystem(s) will be made, the bundle will be extracted onto the new filesystems, local system files will be modified, DNS will be updated, and the VM will be booted.
Source Preparation
This section is going to be short. Once I have a machine in a state where it's ready to be used as a template for future VMs, I run a bundle process on it. The meat of the script to perform this is:
bundle=/tmp/$name.tar.gz
cd /
tar czpf $bundle --exclude "./tmp/*" \
--exclude ./lost+found \
--one-file-system . 2>&1 | grep -v 'socket ignored'
I can get away with this because I build my template systems on a single filesystem to make this process a snap. The operating system goes in one filesystem and data will be mounted separately (likely with logical volume management) so the --one-file-system option will ignore it. Simply adjust the tar command if you have broken out /var, /boot, and others so that you don't miss any required files.
The actual script that I use to perform this process does more than just this, but not all that much. Besides spending time getting options, printing usage information, and providing a means to update itself, it transfers the resulting tar file up to a web service that I have running on a host serving as a bundle repository. That way, my bundles get "published" and can be used across the hosting environment. Note that one large side benefit of the script's simplicity is that bundles can be used for both Xen and KVM virtual machines...it will be up to the provisioning process to make the minor adjustments to account for hypervisor differences.
VM Build Process
As I just mentioned, the bundles I create can be used with both Xen and KVM. When using libvirt as a middleman between configuration and hypervisor there are very few differences in the provisioning processes. I'm pretty anal so I do have separate programs to do both Xen and KVM, but that's by no means required and I probably will just combine the two at some point.
One of the differences between Xen and KVM is that Xen has the ability to assign logical volumes as individual disk partitions in the guest VM. For example, /dev/vg_xen/testvm-root can be presented to the VM named testvm as /dev/xvda1. This is great, because a filesystem can be created on that volume on the Xen host, quickly mounted up, and then the bundle can be untarred directly into it. When the VM boots, it'll see that filesystem as /dev/xvda1, its root filesystem.
From my research and testing, KVM does not have this ability. Instead, logical volumes can only be presented as whole disks, so /dev/vg_kvm/testvm-vda could only be mounted as /dev/vda in the VM. This presents the (minor) annoyance of having to partition the disk from the KVM host first and then playing the kpartx game before unbundling. I'll do that in the example below since this method could also be used for Xen.
Ok, with that out of the way, let's get down to it. I find that it's easiest to start with a list of all the information I'll need for a fully functional VM and then go from there:
- VM name (obviously)
- MAC address: Not absolutely required but I like to assign these programmatically.
- IP address: Again, not absolutely required since you could just use DHCP, but I like to assign these and possibly push the assignments to DHCP.
- Bridge: You'll need to know what bridge to use for your VM's network.
- Disk Device: This piece can be huge ... it can be something as simple as a file path or it can become a big process to generate an iscsi lun on a remote server and connect to it from the provisioning host. In this example, I'll use a local logical volume group. In future blog entries I'll probably talk a lot about iSCSI.
- CPU count
- Memory
For the purposes of this blog entry, I'm going to chop the stuff that isn't necessarily all that interesting: mac/ip/bridge assignment and cpu/memory definition. So for simplicity I'll take these items as variables used by a bash function named vm_build. I'll just spit out the function (from the KVM build process) in chunks and discuss along the way. Where my "real" scripts do more I'll try to make a comment.
vm_build()
{
vm=$1
### do name validation in real script
echo
echo "=== Creating new linux domain: $vm ==="
echo Bundle: $bundle
echo Memory: $mem
echo CPUs: $cpu
echo MAC: $mac
echo IP: $ip
echo Bridge: $bridge
echo Volume Group: $vg
echo Root Filesystem Size: $size_root
echo Swap Size: $size_swap
echo Filesystem Type: $fs
echo Boot on Completion: $boot
xml=/etc/libvirt/qemu/${vm}.xml
sed "s/\$NAME/$vm/" $BASE_XML | \
sed "s/\$MEM/$mem/" | \
sed "s/\$CPU/$cpu/" | \
sed "s/\$IP/$ip/" | \
sed "s/\$BRIDGE/$bridge/" | \
sed "s/\$MAC/$mac/" > $xml
This is just displaying the variables that I'll have defined by the time it's ready to build a VM. You could take this as command-line options or have functions to pick them or, like I do, use a mixture of both. I generally take bundle, memory, cpu, size_root, size_swap, and fs as command-line options since they're a unique decision for each VM. For the network info and disk volume, that gets programmatically generated since it would be annoying to manually create unique MACs and manually pick open IP addresses. :)
Once those variables are in place, start feeding them into a template libvirt xml file (defined as $BASE_XML) which we'll use to define the VM. Here's a link to this template:
base_kvm_linux.xml Now to the dirty part ... the actual disk provisioning.
echo creating logical volumes...
lv_size=$((size_root + size_swap))
lv_root=/dev/$vg/$vm
lvcreate -L ${lv_size}m -n $lv_root
if [ $? -ne 0 ]; then
echo "error: could not create root volume for $vm...aborting!"
return 3
fi
sed -i "s@\$ROOT@$lv_root@" $xml
parted -s $lv_root mktable msdos
parted -s -- $lv_root mkpart primary ext2 0 $size_root
parted -s -- $lv_root mkpart primary linux-swap $size_root -1
parted -s -- $lv_root set 1 boot on
# brutal hack workaround since parted creates weird mapper entries
mapper_name=`echo $vm | sed 's/-/--/g'`
dmsetup remove /dev/mapper/*${mapper_name}*p1
dmsetup remove /dev/mapper/*${mapper_name}*p2
kpartx -a -p P $lv_root
part_root=/dev/mapper/${vm}P1
part_swap=/dev/mapper/${vm}P2
mkswap $part_swap
echo building root filesystem...
mkfs.$fs -q $part_root
mkdir -p /mnt/${vm}
mount $part_root /mnt/${vm}
echo extracting operating system...
tar -C /mnt/$vm -xzf $bundle
sync # gotta sync otherwise the grub will occasionally fail???
The above allocates a logical volume, partitions it, plays device mapper games, mounts it locally on the KVM host, creates a filesystem (type specified by $fs - I use ext3 and ext4 as needed) on one partition and mounts it, makes swap on the other, and finally extracts the bundle. It's a lot of work for really not all that much activity. As I mentioned before, with Xen this *can* be simplified to create two logical volumes: one for root and one for swap. The mkfs will run on the root logical volume and that volume will be mounted. It eliminates all the partitioning steps and device mapper complexity.
From here, we go to the local modifications. For simplicity's sake again, I'll just assume that we only build Red Hat based systems. Debian would have its configuration files in other locations.
# update the hostname
sed -i "s/HOSTNAME=.*/HOSTNAME=${vm}.${dns_domain}/" \
/mnt/$vm/etc/sysconfig/network
# change the grub boot device
sed -i 's@root=[^ ]\{1,\} @root=/dev/vda1 @' \
/mnt/$vm/boot/grub/grub.conf
# create a base fstab using the virtio vda devices
cat >/mnt/$vm/etc/fstab <<+++
/dev/vda1 / $fs defaults,noatime 1 1
/dev/vda2 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
+++
# clear out the MAC address
sed -i '/^HWADDR/d' \
/mnt/$vm/etc/sysconfig/network-scripts/ifcfg-eth0
sed -i "s/^IPADDR=.*/IPADDR=$ip/" \
/mnt/$vm/etc/sysconfig/network-scripts/ifcfg-eth0
# for now, just remove persistent rules files
rm -f /mnt/$vm/etc/udev/rules.d/70-persistent-net.rules
# disable selinux
if [ -f /mnt/$vm/etc/sysconfig/selinux ]; then
echo 'SELINUX=disabled' > /mnt/$vm/etc/sysconfig/selinux
fi
Mostly it's just a matter of making sure that the system boots up on the device partitions that KVM is presenting (Xen will use xvda rather than vda) and that our eth0 network interface comes up with its own MAC address and IP. SELinux is disabled since it will hose stuff up in the event that our template VM had it enabled.
Now we need to make sure that the new VM will actually boot with grub. In Xen this is not at all required since it uses the beautiful concept of pygrub, but as far as I am aware KVM has no such ability and it requires that a boot loader be installed on each VM's root volume.
mount --bind /dev /mnt/$vm/dev
mount -t proc none /mnt/$vm/proc
cat >/mnt/$vm/vm-grub <<EOF
#!/bin/bash
ln -s $part_root ${lv_root}1
grub <<+++
device (hd0) $lv_root
root (hd0,0)
setup (hd0)
+++
rm -f ${lv_root}1
EOF
chmod 755 /mnt/$vm/vm-grub
echo chroot /mnt/$vm /vm-grub
chroot /mnt/$vm /vm-grub
umount /mnt/$vm/proc
umount /mnt/$vm/dev
umount /mnt/$vm
rm -rf /mnt/$vm
kpartx -d -p P $lv_root
If there is a better simpler way to do this, please, I am all ears. I struggled through this piece the longest in the conversion of my Xen provisioning scripts to KVM and would really love to clean this up. I mean, I love taking any chance I can to use chroot and all, but one command sure would be nice. :)
Anyway, finally, we get to the definition of the VM and its registration with DNS.
echo -n Defining VM...
virsh define $xml &>/dev/null
if [ $? -eq 0 ]; then
echo "success!"
echo Configuring VM to autostart...
virsh autostart $vm
if [ "$boot" -eq 1 ]; then
virsh start $vm
fi
else
echo "failure!"
fi
### call DNS registration in the real script
echo
echo "=== Creation of $vm complete ==="
}
And that's it! Not so bad, eh? :) The VM is built and started up if desired. When using ext4 and a compressed bundle of 400MB or so, the fastest I've provisioned a machine on 10k RPM disks was 14 seconds. I'd love to bring this number down to single digits so perhaps a nice RAID of 15k RPM disks is in my future.
I hope this proves useful and I'd absolutely love to receive any and all feedback. Until next time!
