VMware ESXi 5.5 Purple Screen (PSOD) w/ Server 2012 & 2012 R2

I’ve had the displeasure of being greeted first thing in the morning by VMWare’s purple screen of death (PSOD) both in my home lab and in production. I seemed to have noticed a trend that once we had more than 2-3 2012 R2 guests on a single ESXi 5.5 host at somewhere round 40 days of uptime the system would purple screen with the message shown below:


After lots of digging and frustration I was able to find a VMware KB article (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2059053) that identified the issue being associated with the E1000E Virtual NIC adapter. The thing that struck me as odd about this is that the E1000E Virtual NIC is default for the 2012 and 2012 R2 servers in the desktop vSphere client, however the vCenter web client gives you a VMXNet3 adapter by default.

So long story short resolution involves capturing your IP configuration info, shutdown the VM (scheduling downtime like a responsible admin), remove the E1000E adapter and add a VMXNET3 Adapter. Please note that the VMXNET3 does require VMware tools to be installed on Windows Server in order to recognize the adapter. Below are the steps to complete this operation:


Right click your VM and choose edit settings:

Screen Shot 2015-08-30 at 11.56.56 AM

Once you’ve selected Edit settings click on your virtual NIC and look at the right hand side of the window to see what type of adapter it is:

Screen Shot 2015-08-30 at 11.58.13 AM


To remove the adapter click the remove button at the top of the windows with the Network Adapter Selected. Then click the Add button to add a new virtual NIC:

Screen Shot 2015-08-30 at 11.58.28 AM

Make sure to select the VMXNet3 from the dropdown list:

Screen Shot 2015-08-30 at 11.58.37 AM

Follow the wizard to victory, then reboot, install VMware tools if not already installed, and reconfigure your IPs. Total downtime for this operation should be 5-10 minutes per 2012 and 2012 R2 VM.

After implementing these steps I have successfully avoided another PSOD in both lab and production. Several other colleagues with similar issues report the fix above resolved the problem for them as well.

Attaching CentOS to iSCSI Target

In my last post we went over the steps to setup an iSCSI target server host iSCSI LUNs on a CentOS box. This tutorial includes steps for setting up an iSCSI initiator on CentOS box and connect to your iSCSI targets. We’ll start by isntalling the packages, discovering and connecting to the target, finish up by persistently mounting the LUN in /etc/fstab.

Install Packages

To setup and configure iSCSI initiator connections to targets we’ll want to install the following:

yum install iscsi-initiator-utils -y
It’s normal for the iscsid service to show as stopped when it’s not in use

Discovering & Connecting to Targets

To discover targets and the target IQN use the following command (note is used as an example, use the correct IP for your iSCSI target):

iscsiadm --mode discovery --type sendtargets --portal --discover

This should discover the iSCSI target and it’s available LUNs. To get more info run the following command:

iscsiadm --mode node --op show | more

To connect to the iSCSI LUN run the following substituting your target IQN path and IP address:

iscsiadm --mode node --targetname iqn.2015.com.tgsrv1:tgt1 --portal --login

To verify connection to the ISCSI LUN use the following command:

iscsiadm --mode session --op show

Once you are connected the LUN will show up as another block device on your system and will appear just like a local hard drive and will need to be formatted and have a filesystem written to it. After writing a filesystem we can mount the new iSCSI storage. In my case I used ext4 as my filesystem, if you used a different filesystem you’ll need to specify it instead of ext4.

mkdir -p /mnt/san
mount -t ext4 /dev/sdb /mnt/san

Setting Up Persistent Mounts

The ability to add iSCSI targets to the initiator system is great, but in situations where the iSCSI needs to reattach after reboot the following instructions will apply. To start with we will need to obtain the UUID of the iSCSI device:


Obtain the UUID of the device you’re working with (you may want to copy this to your clipboard for the time being).

vim /etc/fstab

Add the following line (using your UUID and correct filesystem (ext4 in my case) mount point).

UUID=d9275aa1-ab41-44d1-9f22-81ef1bf325e3 /mnt/san ext4 _netdev		0 0


CentOS ISCSI Target Server Setup

Requirements & Best Practices

ISCSI should have a dedicated disk or a dedicated LV to use, ensure you have sufficient space to create a new LV, otherwise storage should be added. If you are using SELInux please make appropriate changes or do the following to disable SELinux:

setenforce 0
sed -i 's/=enforcing/=disabled/g' /etc/sysconfig/selinux


If you are using iptables firewall be sure to add a rule to iptables to allow communication on port 3260 for iSCSI.

vim /etc/sysconfig/iptables

Above the explicit deny line in the iptables config enter the following line item:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 3260 -j ACCEPT

Restart IP tables:

service iptables restart


Install and Enable Target Server

install the required packages

yum install scsi-target-utils -y

Enable for auto-start

chkconfig tgtd on

Start the Daemon

/etc/init.d/tgtd start


Configure Targets

Edit the contents of of the targets configuration file

vim /etc/tgt/targets.conf

Edit the following block:

# Set the driver. If not specified, defaults to "iscsi".
# This can be iscsi or iser. To override a specific target set the
# "driver" setting in the target's config.
default-driver iscsi
#<target iqn.2008-09.com.example:iser>
#       Example: the next line would override default driver type.
#       driver iser

You’ll need to include the target iqn info as well as the backing store information. When you are finished it should look something like:

# Set the driver. If not specified, defaults to "iscsi".
# This can be iscsi or iser. To override a specific target set the
# "driver" setting in the target's config.
default-driver iscsi
<target iqn.2015-08.com.tgsrv1:tgt1>
        backing-store /dev/vg_centos6/iscsi
#       Example: the next line would override default driver type.
#       driver iser

Once you have completed this step you will need to restart the tgtd servie

service tgtd restart

At this point we can verify that the ISCSI target LUN we configured is now visible:

tgtadm --mode target --op show

If you completed the configuration correctly you should see the LUN0 the control LUN and LUN1 which will the LUN we just created. It will look something like the example below:

[root@localhost ~]# tgtadm --mode target --op show
Target 1: iqn.2015-08.com.tgsrv1:tgt1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 20401 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/vg_centos6/iscsi
            Backing store flags:
    Account information:
    ACL information: