Hi,

Just a quick piece of advice regarding Dell R610s and Dell R910s in vSphere – when setting up the BIOS of the server, make sure to leave Node Interleaving disabled.

This messes with ESX’s own NUMA settings (Configuration -> Advanced Settings -> Numa) and significantly reduces the speed of the ESX server.

Reference

Cheers,

Leo

I had meant to post this sooner than now but unfortunately work is keeping me occupied 28×7 (that’s not a typo!)

This is a partial update to this post

Attila seems to be a scripting whiz and he’s made a change to my dd script to not utilize bc and added better tunable parameters – ddsleep, ddbs and ddsize. Then he emailed me to share with you! He has tested the default values on some systems and they seemed good: the progress was acceptable and it didn’t generate a high load, the system remained very responsive (in fact it was nearly not noticeable at all):

#!/bin/bash

# by default we skip /
doroot=0

# we want to keep 10 percent free
freepercent=5

# we want to keep at least 100 megabytes free
freespace=100m

# low priority
ddnice=19

# the file to create
file=.vtools-zerovmdk.bin

# the amount of time to sleep between two dd
ddsleep=0.4

# the block size to use with dd
ddbs=64k

# the size by which to increment the file in each turn
ddsize=10m

# if the file is already there, we continue to grow it
resume=0

fss=

while [ $# -gt 0 ]; do

	case $1 in
		--do-root)
			doroot=1
			shift 1
			;;

		--resume)
			resume=1
			shift 1
			;;

		--free-percent)
			freepercent=$2
			shift 2
			;;

		--free-space)
			freespace=$2
			shift 2
			;;

		--dd-nice)
			ddnice=$2
			shift 2
			;;

		--file)
			file=$2
			shift 2
			;;

		--dd-sleep)
			ddsleep=$2
			shift 2
			;;

		--dd-bs)
			ddbs=$2
			shift 2
			;;

		--dd-size)
			ddsize=$2
			shift 2
			;;

		--fss)
			fss=$2
			shift 2
			;;

		*)
			echo "Invalid parameter: $1"
			exit 1
			;;

	esac

done

function resolve_size
{
	local size=$1
	local m=0

	if ! (echo "$size" | grep -q "[0-9]*"); then
		echo "Invalid size: $size" >&2
		exit 1
	fi

	case $size in
		*g)
			m=3
			;;
		*m)
			m=2
			;;
		*k)
			m=1
			;;
	esac

	if [ $m -gt 0 ]; then
		local l=${#size}
		l=$(( $l - 1 ))
		size=${size:0:$l}
		local i=0
		while [ $i -lt $m ]; do
			size=$(( $size * 1024 ))
			i=$(( $i + 1 ))
		done
	fi
	echo $size
}

function calc_percent
{
	local v=$(($1 * $2))
	local l=${#v}
	l=$(($l - 2))
	echo ${v:0:$l}
}

function calc_fsfree
{
	df -B 1 $fs | tail -n 1 | awk -F" " '{print $4}'
}

function calc_fsmax
{
	df -B 1 $fs | tail -n 1 | awk -F" " '{print $2}'
}

freespace=`resolve_size $freespace`
ddbs=`resolve_size $ddbs`
ddsize=`resolve_size $ddsize`

test -z "$ddnice" && ddnice=`nice`

if [ ! -z "$fss" ]; then
	doroot=1
else
	fss=`mount | egrep "type (ext3|ext2|xfs)" | awk -F" " '{ print $3 }'`
fi

for fs in $fss; do

	if [ ! -d "$fs" ]; then
		echo "No such directory: $fs" >&2
		exit 1
	fi

	if [ "$fs" = "/" ]; then
		test "$doroot" = "0" && continue
	fi

	if [ $resume -eq 0 -a -f "$fs/$file" ]; then
		echo "File exists: $fs/$file" >&2
		exit 1
	fi

	fsmax=`calc_fsmax $fs`
	fskeep=`calc_percent $fsmax $freepercent`
	if [ $freespace -gt $fskeep ]; then
		fskeep=$freespace
	fi

	fsfree=`calc_fsfree $fs`
	fszero=$(($fsfree - $fskeep))

	#
	# If the last chunk will not reach
	# the space amount we need to free then
	# then we may enter an endless loop
	last=0

	while [ $fszero -ge 0 ]; do

		test $last -eq 1 && break

		ddcnt=$ddsize

		if [ $fszero -lt $ddsize ]; then
			ddcnt=$fszero
			last=1
		fi

		# this is the step where we may loose precision (see 'last' variable)
		ddcnt=$(( $ddcnt / $ddbs ))

		nice -n $ddnice dd count=$ddcnt bs=$ddbs if=/dev/zero of="$fs/$file" oflag=append \
conv=notrunc status=noxfer 2>&1 | sed '/^[0-9]\++[0-9]\+.*\(records\|rekord\)/d' >&2

		test ! -z "$ddsleep" && sleep $ddsleep

		fsfree=`calc_fsfree $fs`
		fszero=$(($fsfree - $fskeep))
	done

	nice -n $ddnice rm -f "$fs/$file"
done

The script is also downloadable from here.

Attilla, many thanks.

Cheers,

Leo

Just to let committed blog readers know, I’ve just passed the first stage of the VCDX exam.

I’ve also passed my VCP on vSphere.

• Do not go into the VCDX unprepared – your arse will be handed back to you on a platter. It is not an easy exam
• Anyone who is a VCP on VI3.5 only needs to read up the vSphere Configuration Maximums and you’ll be able to pass with your hands tied behind your back – there are some questions on VMDirectPath I/O and vShield/VMSafe but not too many.

Cheers,

Leo

First of all, apologies for not posting for a very long time. I’ve been busy setting up my Cloud Computing service. I’ll write about that some other time.

For now, I have some recent experiences to share with vCenter:

• vCenter 4.0 works on Windows 2008 R2 but the vSphere client does not unless a workaround is used
• If you plan on using the Cisco Nexus 1000V as part of your vSphere Enterprise Plus package, vCenter must be able to run on port 80 – this means, no IIS on the vCenter box.
• If upgrading from VirtualCenter 2.5.x and moving servers, make sure to keep a backup of the SSL certs in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL – if they are not copied over the ones newly installed by the vCenter installer, the vCenter Hardware Status and the vCenter Service Status will fail to enable.
• If installing on an x64 Windows box, you still need a 32-bit connection via ODBC to your SQL DB. Start, Run, type odbcad32.exe and enter your details

Cheers,

Leo

If you own Dell Servers and run your ESX farms on them, you’ll know what a pain in the arse it is to update the firmware.

What if you could do it as part of your install script, or even deploy it in the future as a script to be run on all boxes?

I’ve got the solution for that, because frankly, the Dell TechCenter one didn’t suit my purposes (mounting/lwp-downloading a 1.9GB ISO to each server to upgrade 30MB worth of updates is a serious waste of time and bandwidth across sites)

The only requirement is that Dell’s OMSA is installed and running on the ESX server.

So here’s my script:

#!/bin/sh

# Allowing outbound access through the ESX firewall
esxcfg-firewall --allowOutgoing

# Downloading Dell firmware/BIOS updates
lwp-download http://149.171.186.10/Dell/Toolkit/Systems/PE6950/DRAC.BIN /tmp/DRAC.BIN
lwp-download http://149.171.186.10/Dell/Toolkit/Systems/PE6950/BIOS.BIN /tmp/BIOS.BIN
lwp-download http://149.171.186.10/Dell/Toolkit/Systems/PE6950/RAID.BIN /tmp/RAID.BIN

# Disabling outbound access through the ESX firewall
esxcfg-firewall --blockOutgoing

# Make sure firmware/BIOS scripts are executable
chmod a+x /tmp/*.BIN

# Attach virtual media for DRAC update
racadm config -g cfgRacVirtual -o cfgVirMediaAttached 1
export rawdevice=`dmesg | tail -n30 | grep "32768 512-byte hdwr" | awk -F" " '{print $3}' | sed "s/://g"`
raw /dev/raw/raw1 /dev/$rawdevice

# Run the updates
for i in /tmp/*.BIN ; do $i -q ; done

# Delete the updates
rm -f /tmp/*.BIN

# Detach the virtual media for DRAC update
racadm config -g cfgRacVirtual -o cfgVirMediaAttached 0

The file is likewise attached.

Obviously, in the lwp-download section, put in your own URL for the .BIN files that comprise the Dell updates.

Cheers,
Leo

We have a customer with two ESX farms – one with AMD 8354 Opterons and one with AMD 8220 Opterons.

We use a manual sysprep – not the VC-integrated one, but a custom sysprep.inf file. And we found that all Windows 32-bit machines came up with Unknown device instead of the processor and we couldn’t roll back to the old driver. The only way to fix it was to run the setup.exe -s option on the AMD Processor driver. Now this is all well and good but that doesn’t fix the problem of why vanilla Windows installs screw up after imaging.

This is what the problem looks like:

So we examined our custom sysprep file – and we found it:

UpdateInstalledDrivers = Yes

That entry was in the sysprep.inf custom file. Removing it, caused the Unknown device issue to disappear. Having it there guaranteed the Unknown device issue appearing.

Here’s what we determined:

• There appears to be no performance impact from this issue
• This does not prevent any functionality in terms of software (Windows based) or VMware (such as VMotion/DRS/HA)

Just a heads up for you.

Cheers,

Leo