SUN Delivers De-duplication on ZFS

Today marks yet another great milestone for OpenSolaris and OpenStorage. SUN has as promised, delivered a much anticipated de-duplication feature for us to explore and use.  I must say that I am very excited about it and with no doubt this is a very cool feature indeed The ideas for how to use it are running around in my head like neurons do and your sure to see some of those ideas surface in a blog or two.

Now before we get too excited we need to keep in mind that this is the first release of this feature to the public space and we are sure to find the odd bump or two along the road while seeing this new very great file system feature mature. I’m sure that we will be more than pleased with the new feature and the many other capabilities that are sure to come.

If your interested in experimenting with the development releases you should be able to get your hands on the feature in about 3-4 weeks through IPS or SXCE. Or if your an advanced kernel type IT pro you could build it using the source code now….right…so then, for the rest of us.

To try it out the easy way when it becomes available just download and install OpenSolaris with the LiveCD (I recommend an x64 CPU with 4G of ram)

http://dlc.sun.com/osol/opensolaris/2009/06/osol-0906-x86.iso

Then set your repository publisher to the dev IPS image server and issue the pkg image-update command

e.g.

$ pfexec pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org

$ pfexec pkg image-update

And explore!


Jeff Bonwick, Bill Moore and company are definitely thinking up some brilliant technical and practical applications of their knowledge bringing us a powerful new storage direction that has changed the game.

Thanks go to the ZFS team.

You rock!

Regards,

Mike

Site Contents: © 2009  Mike La Spina

Controlling Snapshot Noise

The ability to perform file system, database and volume snapshots grants us many data protection benefits. However there are some serious problems that can occur if we do not carefully architect snapshot based storage infrastructures. This blog entry will discuss some of the issues with data noise induction and data integrity when using point in time data snapshot activities  and how we can reduce the negative aspects of these data protection methods.

With the emergence of snapshot technology in the data center data noise induction is an unwanted by product and needs to addressed. Active data within a file store or raw volume will have significant amounts of temporary data like  memory swaps and application temp files. This type of data is required for system operations but unfortunately it is completely useless within a point in time snapshot and simply consumes valuable storage space with no permanent value within the scope of system data protection. There are many sources of this undesirable data noise that we need to consider and define strategies to isolate and eliminate them where possible.

In some cases using raw stores eg. iSCSI, FC etc. we will have duplicate snapshot functionality points in the storage stream and this further complicates how we approach a solution to noise induction issues. One common example of snapshot functionality duplication is Microsoft Windows 2003 Volume Snapshot Service aka. VSS. If we enable VSS and an external snapshot service is employed then we are now provisioning snapshots of snapshots which of course is less than optimal because much of the delta between points in time are just redundant encapsulated data. There are some higher level advantages to allowing this to occur like provisioning self service end user restores and VSS aware file system quiescence but for the most part it is not optimal from a space consumption or performance and efficiency  perspective.

If we perform snapshots at multiple points in the data storage stream using VSS we will have three points of data delta. The changed data elements on the source store of the primary files, a copy-on-write set of  changed blocks of the same primary store including it’s meta data and finally the external snapshot delta and it’s meta data. As well if the two snapshot events were to occur at the same time it creates a non-integral copy of a snapshot and meta data which is just pure wasted space since it is inconsistent.

With the co-existing use of VSS we need to define what functionality is important to us. For example VSS is limited to 512 snapshots and 64 versions so if we need to exceed these limits we have to employ an external snapshot facility. Or perhaps we need to allow a user self service file restore functionality from a shared folder. In many cases this functionality is provided by the external snapshot provisioning point. OpenSolaris, EMC and NetApp are some examples of storage products that can provide such functionality. Of course my preference is Custom OpenSolaris storage servers or the S7000 series of storage product which is based on OpenSolaris and is well suited for the formally supported side of things.

Solely provisioning the snapshots externally verses native MS VSS can significantly reduce induced data noise if the external provider supports VSS features or provides tools to control VSS. VSS copy on write snapshot capability should not be active when using this strategy so as to eliminate the undesirable snapshot echo noise. Most environments will find that they require the use of snapshot services that exceed the native MS VSS capabilities.  Provisioning the snapshot function directly on a shared storage system is a significantly better strategy verses allowing a distributed deployment of storage management points across your infrastructure.

OpenSolaris and ZFS provides superior depth in snapshot provisioning than Microsoft shared folder snapshot copy services. Implementing ZFS dramatically reduces space consumption and allows snapshots up to the maximum capacity of the storage system and OpenSolaris provides MS SMB client access to the snapshots which users can manage recovery as a self service. By employing ZFS as a backing store, snapshot management is simplified and snapshots are  available for export to alternate share points by cloning and provisioning the point in time to data consumers performing a multitude of desirable tasks such as audits, validation, analysis and testing.

If we need to employ MS VSS snapshot services provisioned on a storage server that uses snapshot based data protection strategies  then we will need prevent  continuous snapshots on the storage server. We can use features like snap mirror and zfs replication to provision a replica of the data however this would need to be strictly limited in count e.g. 2 or 3 versions and timed to avoid a multiple system snapshot time collision. Older snapshots should be purged and we only allow the MS VSS snapshot provisioning to keep the data deltas.

Another common source of snapshot noise is temporary file data or memory swaps. Fortunately with this type of noise the solution is relatively easy to solve as we simply isolate this type of storage onto storage volumes or shares that are explicitly excluded from a snapshot service function. For example if we are using VMFS stores we can place vswp files on a designated VMFS volume and conversely within an operating system we can create a separate vmdk disk that maps to a VMFS volume which we also exclude from the snapshot function. This strategy requires that we ensure that any replication scheme incorporates the creation or one time replication of these volumes. Unfortunately this methodology does not play well with storage vmotion so one must ensure that the a relocation does not move the noisy vmdk’s back into the snapshot service provisioned stores.

VMware VMFS volume VM snapshots is a significant source of data noise. When a snapshot is initiated from within VMware all data writes are placed on delta file instances. These delta files will be captured on the external storage systems snapshot points and will remain there after the VM snapshot is removed. Significant amount of data delta are produced by VM based snapshots and sometimes mulitple deltas can exceed original vmdk size. An easy way to prevent this undesirable impact is to clone the VM to a store outside the snapshot provisioned stores rather than invoking snapshots.

Databases are probably the most challenging source of snapshot noise data and requires a different strategy than isolation because the data within a specific snapshot is all required to provide system integrity. For example we cannot isolate SQL log data because it is required to do a crash recovery or to roll forward etc.  We can isolate temp database stores since any data in those date stores would not be valid in a crash recovery state.

One strategy that I use as both a blanket method when we are not are able to use other methods and in concert with the previously discussed isolation methods is a snapshot roll-up function. This strategy simply reduces the number of long term snapshot copies that are kept. The format is based on a Grand Father, Father and Son (GFS) retention chain of the snapshot copies and is well suited for a variety of data types. The effect is to provide a reasonable amount of data protection points to satisfy most computing environments and keep the captured noise to a manageable value. For example if we were to snapshot without any management cycle every 15 minutes we would accumulate ~35,000 delta points of data over the period of 1 year. Conversely if we employ the GFS method we will accumulate 294 delta points of data over the period of 1 year. Obviously the consumption of storage resource is so greatly reduced that  we could keep many additional key points in time if we wished and still maintain a balance of recovery point verses consumption rate.

Let’s take a look at a simple real example of how snapshot noise can impact our storage system using VMware, OpenSolaris and ZFS block based iSCSI volume snapshots. In this example we have a simple Windows Vista VM that is sitting idle, in other words only the OS is loaded and it is power on and running.

First we take a ZFS snapshot of the VMFS ZFS iSCSI volume.

zfs snapshot sp1/ss1-vol0@beforevmsnap

Now we invoke a VMware based snapshot and have a look at the result.

root@ss1:~# zfs list -t snapshot
NAME                               USED  AVAIL  REFER  MOUNTPOINT
sp1/ss1-vol0@beforevmsnap          228M      -  79.4G  -

Keep in mind that we are not modifying any data files in this VM if we were to change data files the deltas would be much larger and in many cases with multiple VMware snapshots could exceed the VMs original size if it is allowed to remain for long periods of weeks and longer. The backing store snapshot initially consumes 228MB which will continue to grow as changes occurs on the volume. A significant part of the 228MB is the VMs memory image in this case and of course it has no permanent storage value.

sp1/ss1-vol0@after1stvmsnap          1.44M      -  79.5G  -

After the initial VMware snapshot occurs we create a new point in time ZFS snapshot and here we observe some noise in the next snapshot and again we have not changed any data files in the last minute or so.

sp1/ss1-vol0@after2ndvmsnap       1.78M      -  79.5G  -

And yet another ZFS snapshot a couple of minutes later shows more snapshot noise accumulation. This is one of the many issues that are present when we allow non-discretionary placement of files and temporary storage on snapshot based systems.

Now lets see the impact of destroying a snapshot that was created after we delete the VMware based snapshot.

root@ss1:~# zfs destroy sp1/ss1-vol0@beforevmsnap
root@ss1:~# zfs list -t snapshot
NAME                               USED  AVAIL  REFER  MOUNTPOINT
sp1/ss1-vol0@after1stvmsnap       19.6M      -  79.2G  -
sp1/ss1-vol0@after2ndvmsnap       1.78M      -  79.2G  -

Here we observe the reclamation of more than 200MB of storage. And this is why GFS based snapshot rollups can provide some level of noise control.

Well I hope you found this entry to be useful.

Til next time..

Regards,

Mike

 



Site Contents: © 2009  Mike La Spina

Securing COMSTAR and VMware iSCSI connections

Connecting VMware iSCSI sessions to COMSTAR or any iSCSI target provider securely is required to maintain a reliable system. Without some level of initiator to target connection gate keeping we will eventually encounter a security event. This can happen from a variety of sources, for example a non-cluster aware OS can connect to an unsecured VMware shared storage LUN and cause severe damage to it since the OS has no shared LUN access knowledge.  All to often we make assumptions that security is about confidentiality when it is actually more commonly about data availability and integrity which will both be compromised if an unintentional connection were to write on a shared LUN.

At the very minimum security level we should apply non-authenticated named initiator access grants to our targets. This low security method defines initiator to target connection states for lower security tolerant environments. This security method is applicable when confidentiality is not as important and security is maintained with the physical access control realm. As well it should also coincide with SAN fabric isolation and be strictly managed by the Virtual System or Storage Administrators. Additionally we can increase access security control by enabling CHAP authentication which is a serious improvement over named initiators. I will demonstrate both of these security methods using COMSTAR iSCSI Providers and VMware within this blog entry.

Before we dive into the configuration details lets examine how LU’s are exposed. COMSTAR controls iSCSI target access using several combined elements. One of these elements is within the COMSTAR STMF facility where we can assign membership of host and target groups. By default if we do not define a host or target group any created target will belong to an implied ALL group. This group as we would expect grants any connecting initiator membership to the ALL group assigned LUN’s. These assignments are called views in the STMF state machine and are a mapping function of the Storage Block Driver service (SBD) to the STMF IT_nexus state tables.

This means that if we were to create an initiator without assigning a host group or host/target group combination, an initiator would be allowed unrestricted connectivity to any ALL group LUN views and possibly without any authentication at all. Allowing this to occur would of course be very undesirable from a security perspective in almost all cases. Conversely if we use a target group definition then only the initiators that connect to the respective target will see the LUN views which are mapped on that target definition instance.

While target groups do not significantly improve access security it does provide a means controlling accessibility based on the definition of interface connectivity classes which in turn can be mapped out on respective VLAN priority groups, bandwidth availability and applicable path fault tolerance capabilities which are all important aspects of availability and unfortunately are seldom considered security concepts in many architectures.

Generally on most simple storage configurations the use of target groups is not a requirement. However they do provide a level of access control with LUN views. For example we can assign LUN views to a target group which in turn frees us from having to add the LUN view to each host group within shared LUN configurations like VMware stores. With combination’s of host and target groups we can create more flexible methods in respect to shared LUN visibility. With the addition of simple CHAP authentication we can more effectively insulate target groups. This is primarily due to the ability to assign separate CHAP user and password values for each target.

Lets look at this visual depiction to help see the effect of using target and host groups.

COMSTAR host and target view depiction

In this depiction any initiator that connects to the target group prod-tg1 will by default see the views that are mapped to that target groups interfaces. Additionally if the initiator is also a member of the host group prod-esx1 those view mapping will also be visible.

One major difference with target groups verses the all group is that you can define LU views on mass to an entire class of initiator connections e.g. a production class. This becomes an important control element in a unified media environment where the use of VLANs separates visibility. Virtual interfaces can be created at the storage server and attached to VLANs respectively. Target groups become a very desirable as a control within a unified computing context.

Named Initiator Access

Enabling named initiator to target using unauthenticated access with COMSTAR and VMware iSCSI services is a relatively simple operation. Let’s examine how this method controls initiator access.

We will define two host groups, one for production esx hosts and one for test esx hosts.

# stmfadm create-hg prod-esx1

# stmfadm create-hg test-esx1

With these host groups defined we individually assign LU’s views to the host groups and then we define any initiator to be a member of one of the host groups to which it would only see the views which belong to the host group and additionally any views assigned to the default all group.

To add a host initiator to a host group, we must first create it in the port provider of choice which in this case is the iSCSI port provider.

# itadm create-initiator iqn.1998-01.com.vmware:vh1.1

Once created the defined initiator can be added to a host group.

# stmfadm add-hg-member -g prod-esx1 iqn.1998-01.com.vmware:vh1.1

An ESX host initiator with this iqn name can now attach to our COMSTAR targets and will see any LU views that are added to the prod-esx1 host group. But there are still some issues here, for example any ESX host with this initiator name will be able to connect to our targets and see the LUs. This is where CHAP can help to improve access control.

Adding CHAP Authentication on the iSCSI Target

Adding CHAP authentication is very easy to accomplish, we simply need to set a chap user name and secret on the respective iSCSI target. Here is an example of its application.

# itadm modify-target -s -u tcuid1 iqn.2009-06.target.ss1.1

Enter CHAP secret:
Re-enter secret:

The CHAP secret must be between 12 and 255 characters long. The secret is sent from the initiator to the target in clear text and this is one of the reasons why separate iSCSI networks are a best practice. The addition of CHAP allows us to further reduce any risks of a potential storage security event. We can define an additional target and they can have a different chap user names and or secrets.

CHAP is more secure when used in a mutual authentication back to the source initiator which is my preferred way to implement it on ESX 4 (ESX 3 does not support mutual chap). This mode does not stop a successful one-way authentication from an initiator to the target, it allows the initiator to request that the target host system iSCSI services must authenticate back to the initiator which provides validation that the target is indeed the correct one. Here is an example of the target side initiator definition that would provide this capability.

# itadm modify-initiator -s -u icuid1 iqn.1998-01.com.vmware:vh1.1

Enter CHAP secret:
Re-enter secret:

Configuring the ESX 4 Software iSCSI Initiator

On the ESX 4 host side we need to enter our initiator side CHAP values.

ESX 4 iSCSI Mutual CHAP

 

Be careful here, there are three places we can configure CHAP elements. The general tab allows a global point of admin where any target will inherit those entered values by default where applicable e.g. target chap settings. The the dynamic tab can override the global settings and as well the static tab overrides the global and dynamic ones. In this example we are configuring a dynamically discovered target to use mutual (aka bidirectional) authentication.

In closing CHAP is a reasonable method to ensure that we correctly grant initiator to target connectivity assignments in an effort to promote better integrity and availability. It does not however provide much on the side of confidentially for that we need more complex solutions like IPSec.

Hope you found this blog interesting.

Regards,

Mike

Site Contents: © 2009  Mike La Spina

Creating USB based boot media for ESX 4 installs

As a follow on to my Automating vSphere ESX4 Host Installations blog I have detailed a howto create USB based boot media using syslinux 3.82 and the ESX 4 installation source files. The process is actually quite simple as we can create the bootable USB from a Windows system.  You can also do the same with extlinux but most people will have a Windows based management system so lets only focus on this Windows based method within this blog.

The first step is to ofcourse obtain a copy of the Syslinux 3.82 or higher zip package from  http://syslinux.zytor.com/ and extract to a  file store of your choice.

Prepare the media:

To prepare a USB memory stick we need to format it with a FAT32 file system. Windows explorer provides that functionallity with a simple right click on your USB device.

Format USB device

Generate a bootable media device:

Once formated we will need to open a cmd prompt and go to our syslinux file store and execute the following example.

Syslinux cmd prompt

In this example the syslinux win32 tool creates a grub based loader and boot sector on the USB memory device mapped to drive G: the tool also defines the syslinux directory using the -d option as the root path and this is where we will copy the ESX 4 initial ramdisk image file and some additional syslinux text menu files.  If your planning to use the usb device as a source for the ESX 4 packages then those files  e.g. the VMware directory etc. would need to be placed in the root directory of the usb device and not the syslinux directory. In this blog the usb device is only used to launch a remote source file install.

Copy menu and ESX 4 install files:

From the ESX 4 ISO or CD copy the isolinux directory to G:\ and rename it to syslinux also copy the build_numbler file to G:\  additionally explore the downloaded syslinux file store and locate ..\syslinux\com32\menu\menu.c32, copy this file to the G:\syslinux location, you may also want to copy vesamenu.c32 if you wish to checkout a GUI based menu. That’s really just eye candy on the requirements side but it can provide some cool background display capabilities.

Create your selectable boot time menu:

Now we are ready to create the syslinux.cfg configuration file in the syslinux directory.  Here is an example I created for this blog.

default menu.c32
prompt 0
timeout 9000
menu title ESX 4 Automated Install VC1 HTTP Repo

label Default
kernel vmlinuz
append initrd=initrd.img vmkopts=debugLogToSerial:1 mem=512M quiet ks=http://vc1.laspina.ca:8088/esx/4.0/default.cfg

label vh0
kernel vmlinuz
append initrd=initrd.img vmkopts=debugLogToSerial:1 mem=512M quiet ks=http://vc1.laspina.ca:8088/esx/4.0/vh0.cfg

label vh1
kernel vmlinuz
append initrd=initrd.img vmkopts=debugLogToSerial:1 mem=512M quiet ks=http://vc1.laspina.ca:8088/esx/4.0/vh1.cfg

Once your cfg file is created your ready to boot the USB device either on your server or over RDAC/ILOM interfaces,  select a server target from the menu and walk away.

Yes it’s that simple and easy to create USB bootable media for your ESX 4 installs.

Regards,

Mike





Site Contents: © 2009  Mike La Spina

Automating vSphere ESX4 host installations

Automating ESX 4 installations is a great way to save time and to provide a method of server recovery in the event of hardware or software failure. It creates consistent high quality repeatable installations that can be quickly modified to handle new and changing hardware. The process can also provide some detailed levels of VMware ESX server instance documentation. This blog will discuss how the process works and how-to create the required elements for you to implement your own automated process.

The vSphere ESX 4 install process uses an updated linux boot release commonly referred to as Syslinux. This Syslinux release version 3.63 supports a variety of popular protocols to facilitate a remote central install repository. FTP, HTTP, NFS and gPXE are all available options for provisioning network attachment to a remote install repository.

From the Syslinux boot process vSphere launches it’s initrd.img kernel instance which is a custom VMware/Linux kernel containing a multitude of VMware ESX 4 drivers and components. The custom drivers allow for a more closely integrated VMware ESX 4 install process that targets an improved ESX 4 server configuration result.

The custom VMware kernel incorporates Linux kickstart scripting functionality to invoke automated installations. The script location is defined as part of the Syslinux functionality and is available as a menu at boot time. A control file located on the boot media provides these variable control elements. Depending on the media type Syslinux uses a respective cfg file to implement this function. The various available Syslinux boot methods that I am aware of are USB, CD, DVD, PXE and gPXE. In this blog I will demonstrate an ISO CDROM method to perform the automated boot cycle. Any of the boot methods mentioned will all work and have varying levels of complexity to achieve.

The ISO CD and DVD based Syslinux configuration uses a config file named isolinux.cfg, USB boot images would use syslinux.cfg as well as gPXE based boot services can use either depending on the final gPXE target image.
Here is a example and description of the boot time menu functional elements for the isolinux.cfg ISO based file in this demonstration.

default Default
gfxboot bootlogo
prompt 1
timeout 3000

label Default
menu default
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/default.cfg

label vh0
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/vh0.cfg

label vh1
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/vh1.cfg

This cfg file provides three menu choices of default, vh0 and vh1. It will invoke the default after 300 seconds (timeout 3000 is not a typo) or you can manually select the other menu items. The “ks=” append option entry can also be one of file://… cdrom://… ftp://… nfs://…  usb and UUID:ID/… The initrd.img element is an ESX initial ram disk image and it needs to version match your repository for a successful install process.

This isolinux.cfg file relies on DHCP to provide IP services. If DHCP is not an option you can use static methods to provision the same by passing the IP specific info into the initial ram disk image.

Here is an example of static IP parameters within the isolinux.cfg file.

label vh1
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ksdevice=eth0 ip=10.10.0.1 netmask=255.255.255.0 gateway=10.10.0.254 dns=10.10.0.253 ks=http://vc.laspina.ca:8088/esx/4.0/vh1.cfg

Static IP parameters can also be defined in the kickstart ks file but then the ks file would need to be locally available in order for it to work.

Let’s now look at how we can create a remote repository based, automated ESX 4 ISO CD installation boot image. My tool of choice for this process was ISOMagic of which you can make images 300MB or less in size for free. Of course you can use others like PowerISO which is also one of my favourites. The first step in the process is to open the vSphere installation ISO and delete all but the highlighted files show in the graphic. While you could leave them intact I prefer to remove elements that are not required as this can be a template for USB or other boot images.

ISOMagic SS

The ISO boot method is quite simple, all we need to do is create a text isolinux.cfg file based on the example show previously and drag it to the MagicISO window onto the isolinux folder. Once the file is replaced we can use SaveAs an ISO to the name of your choice then burn it or mount it on your server’s ILOM interface. I plan on making a USB based image later so stay tuned for that in another blog entry.

Be wary of using a windows based text editors as they do work well with Unix based text processing operations since it adds invisible characters to the edited files that will cause some of the unix processes to fail. I normally use Ultraedit to edit the Unix targeted files since it has a function that allows you to convert and save in Unix file format.  
To provision an install repository is a matter of choice, you can use any of the many different hosts that can serve one of the supported protocols. If your going to use an http repo you will need to take note of Mime types that may not be defined on the web service of choice. In this blog example we are going to define an http based repo on a Virtual Center Server (VC).

Create a base directory on the VC to host the install repo and extract the ESX 4 ISO to an appropriately named subfolder.
e.g. My base is D:\VMwareRepo\ and the subfolder is esx\4.0

We need to setup an IIS service instance on the VC and create a WEB site on port 8088. Do not use the default port 80 as it will conflict with other VC services.

IIS Repo Config 1

We assign our base repo directory to this site and allow directory browsing.

IIS Repo Config 2

The addition of MIME type pkl is required, right mouse -> Properties on the IIS instance within your Computer Manager MSC

IIS Mine Type SS

That’s all you need for provisioning an http repo with IIS on your VC. Once you have a repo defined and running make sure you can browse it using your favourite browser.
As an added layer of security I only allow ESX console IP interfaces of a specific subnet on the repo site. Here is an example screen shot. Remember to check browsing availability before you enable any subnet restrictions.

IIS Restricted Range

The next step is the most involved and interesting part of the process. Let’s use an example script named vh1.cfg to examine and discuss one of my scripted processes. The script is normally stored in your repo e.g. D:\VMwareRepo\esx\4.0\vh1.cfg.

##########################################################
# ESX 4 Kickstart installation script
# © Mike La Spina – Ubiquitous Talk
# File name: vh1.cfg

##########################################################
# Install or Upgrade
install url
http://vc.laspina.ca:8088/esx/4.0

“We first define a source for our ESX 4 install files, the possible methods are file://, ftp:// nfs:// cdrom://, take note that you can use a custom port like 8088.”

##########################################################
#Network install type
network –bootproto=static –ip=10.20.0.1 –gateway=10.20.0.254 –netmask=255.255.255.0 –hostname=vh1.laspina.ca –nameserver=10.20.0.200 –device=vmnic0 –addvmportgroup=0

“This defines our final static IP on the vswif0 management interface of vmnic0 (aka the Service Console) and addvmportgroup=0 disables the default VM network creation.”

##########################################################
# root Password
rootpw changeme

“Obviously this sets a root password, however I do not recommend you use an encrypted password method as it can be reversed with simple tools. It is better to just immediately change it to a secured one.”

##########################################################
# Authconfig
authconfig –enableshadow –enablemd5

“Enables a local password shadow file and stores the passwords as MD5 hashes.”

##########################################################
# Regional Settings
keyboard us
timezone America/Winnipeg

“Obvious”

##########################################################
# Firewall settings
firewall –allowOutgoing

“Obvious”

##########################################################
# Enable reboot after script
reboot

“Obvious”

##########################################################
# Boot Config
bootloader –location=mbr

“Installs a master boot record on the firstdisk by default”

##########################################################
# Disk Partitioning
clearpart –firstdisk –overwritevmfs
part /boot       –fstype=ext3    –size=250   –onfirstdisk  –asprimary
part vh1-local0  –fstype=vmfs3   –size=16384 –grow         –onfirstdisk
part None        –fstype=vmkcore –size=100   –onfirstdisk
# Create the vmdk on the cos vmfs partition.
virtualdisk cos –size=8192 –onvmfs=vh1-local0
# Partition the virtual disk.
part / –fstype=ext3 –size=4096 –grow –onvirtualdisk=cos
part swap –fstype=swap –size=256 –onvirtualdisk=cos

“Creates the ESX boot, core dump and VMFS partitions as we would expect. We have new partition function available, we can now create our Console Operating System on a vmdk. Here we are defining a virtual disk vmdk named cos on VMFS volume vh1-local0. Very cool, the ESX kernel can now snapshot itself. Take note of the –overwritevmfs option, this can wipe out any perfectly healthy production VMFS volume, I recommend that you remove this option once your testing cycle is complete and only add it to destroy a confirmed existing targeted VMFS volume.”

##########################################################
# Accept the EULA
vmaccepteula

“Obvious”

##########################################################
#
%post –interpreter=bash

“In pre VMware ESX 4 releases it was not possible to directly configure most of the ESX config elements. ESX 4’s initrd.img contains all most everything to need to configure the host without creating any special first time startup scripts on the systems reboot cycle. Now if we can just use vimsh directly … it’s still very cool!”

##########################################################
# Enable Kerberos Auth
/usr/sbin/esxcfg-auth –enablead –addomain=domain.local –addc=domain.local

“Sets up the Linux Plugable Authentication Module (PAM) to autheticate users against a Window Domain over Kerberos”

##########################################################
# Add Groups and Users
/usr/sbin/groupadd -g 5000 lg-esxsu
/usr/sbin/useradd -u 501 -G lg-esxsu super1
/usr/sbin/useradd -u 502 -G lg-esxsu super2
/usr/sbin/useradd -u 503 -G lg-esxsu super3

“Create a local group which will allow members full admin rights to the ESX ha-folder-root and create three user id’s that are members of the group. These user id’s will be authenticated against the domain.local Windows Domain”

##########################################################
# NTP time config
esxcfg-firewall -e ntpClient
echo restrict default kod nomodify notrap noquerynopeer > /etc/ntp.conf
echo restrict 127.0.0.1 >> /etc/ntp.conf
echo server 10.20.0.200 >> /etc/ntp.conf
echo driftfile /var/lib/ntp/drift >> /etc/ntp.conf
/sbin/chkconfig –level 345 ntpd on
/etc/init.d/ntpd start

“Enable outgoing NTP client port access and build the ntp.conf file to use a Windows Domain time service, this is important for kerberos authentication. Start the NTP client daemon.”

# Create vSwitch0, VMMotion1 and the Service Console port group
/usr/sbin/esxcfg-vswitch -a vSwitch0:64
/usr/sbin/esxcfg-vswitch -A “Service Console” vSwitch0
/usr/sbin/esxcfg-vswitch -A Network-00 vSwitch0
/usr/sbin/esxcfg-vswitch -A VMMotion1 vSwitch0
/usr/sbin/esxcfg-vswitch -p VMMotion1 -v 600 vSwitch0
/usr/sbin/esxcfg-vswitch -p Network-00 -v 700 vSwitch0
/usr/sbin/esxcfg-vswitch -L vmnic1 vSwitch0
/usr/sbin/esxcfg-vswitch -L vmnic0 vSwitch0
/usr/sbin/esxcfg-vswitch -M vmnic0 vSwitch0 -p “Service Console”
/usr/sbin/esxcfg-vswitch -M vmnic0 vSwitch0 -p VMMotion1
/usr/sbin/esxcfg-vswitch -M vmnic1 vSwitch0 -p Network-00

“Defines vSwitch0 for the Service Console and the default gateway IP, vMotion on VLAN 600 with a vMotion IP and create a VM Network-00. This vSwitch will be further configured for a custom standby adapter during the initial reboot with a post config script”

# Create vSwitch1 for iSCSI traffic
/usr/sbin/esxcfg-vswitch -a vSwitch1:64
/usr/sbin/esxcfg-vswitch -A iSCSI_Initiator vSwitch1
/usr/sbin/esxcfg-vswitch -p iSCSI_Initiator -v 500 vSwitch1
/usr/sbin/esxcfg-vswitch -L vmnic3 vSwitch1
/usr/sbin/esxcfg-vswitch -L vmnic2 vSwitch1
/usr/sbin/esxcfg-vswitch -M vmnic3 vSwitch1 -p iSCSI_Initiator
/usr/sbin/esxcfg-vswitch -M vmnic2 vSwitch1 -p iSCSI_Initiator
/usr/sbin/esxcfg-vmknic -a -i 10.20.10.64 -n 255.255.255.0 iSCSI_Initiator

“Defines vSwitch1 for iSCSI on VLAN 500″

##########################################################
#
%post –interpreter=bash

# Create post config script
cat << \EOF > /etc/rc3.d/S99postconf
#!/bin/bash

“Some configuration elements still require a post run this statement sends the follow on text to the s99postconf file until the EOF marker is met.”

# Enable TCP outgoing kerberos, there are issues with udp and enable blockOutgoing
/usr/sbin/esxcfg-firewall -–openport 88,tcp,out,KerberosClientTCP
/usr/sbin/esxcfg-firewall –blockOutgoing

“Seriously, this is important, udp kerberos port 88 is defaulted with ‘esxcfg-auth –enablead ..’, bad default! Also the VWware Kerberos client default uses tcp first and this needs to be fixed as it does not comply with RFC 4120. Even if this works why would we risk dropping an auth packet since any fragged udp packet would be dropped over VPN’s etc. Also turn on the outgoing firewall rules that were disabled previously.”

# Enable VMotion on the VMKernel Interface
/usr/bin/vmware-vim-cmd hostsvc/vmotion/vnic_set vmk1

“Enables vMotion on vmkernel interface 1″

# Define Active and Standby failover for shared vSwitche0
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 ‘Service Console’ –nicorderpolicy-active vmnic0 –nicorderpolicy-standby vmnic1
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 VMMotion1 –nicorderpolicy-active vmnic0 –nicorderpolicy-standby vmnic1
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 Network-00 –nicorderpolicy-active vmnic1 –nicorderpolicy-standby vmnic0

“Here we are overriding our adapter team for vSwitch0 so that we can separate our active traffic on the two adapters while maintaining failover capability.”

# Grant the group named lg-esxsu admin permission to ha-folder-root
/usr/bin/vmware-vim-cmd vimsvc/auth/entity_permission_add vim.Folder:ha-folder-root lg-esxsu true Admin true

“Enables any member of the local group lg-esxsu Administrator permissions to the VMware host”

# Reset system to normal boot mode
echo “Removing automated post script.”
rm /etc/rc3.d/S99postconf
EOF
chmod +x /etc/rc3.d/S99postconf

“Obvious”

As you can see the process is quite involved, however the benefits are outstanding. I can build or recover an ESX 4 host in 10 minutes or less and I can reconfigure it to a different target with ease.

Hope you found the entry usefull and interesting.
Regards,
Mike


Site Contents: © 2009  Mike La Spina

Next Page »