VMware ESXi 5.5 Purple Screen (PSOD) w/ Server 2012 & 2012 R2

I’ve had the displeasure of being greeted first thing in the morning by VMWare’s purple screen of death (PSOD) both in my home lab and in production. I seemed to have noticed a trend that once we had more than 2-3 2012 R2 guests on a single ESXi 5.5 host at somewhere round 40 days of uptime the system would purple screen with the message shown below:

PSOD

After lots of digging and frustration I was able to find a VMware KB article (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2059053) that identified the issue being associated with the E1000E Virtual NIC adapter. The thing that struck me as odd about this is that the E1000E Virtual NIC is default for the 2012 and 2012 R2 servers in the desktop vSphere client, however the vCenter web client gives you a VMXNet3 adapter by default.

So long story short resolution involves capturing your IP configuration info, shutdown the VM (scheduling downtime like a responsible admin), remove the E1000E adapter and add a VMXNET3 Adapter. Please note that the VMXNET3 does require VMware tools to be installed on Windows Server in order to recognize the adapter. Below are the steps to complete this operation:

 

Right click your VM and choose edit settings:

Screen Shot 2015-08-30 at 11.56.56 AM

Once you’ve selected Edit settings click on your virtual NIC and look at the right hand side of the window to see what type of adapter it is:

Screen Shot 2015-08-30 at 11.58.13 AM

 

To remove the adapter click the remove button at the top of the windows with the Network Adapter Selected. Then click the Add button to add a new virtual NIC:

Screen Shot 2015-08-30 at 11.58.28 AM

Make sure to select the VMXNet3 from the dropdown list:

Screen Shot 2015-08-30 at 11.58.37 AM

Follow the wizard to victory, then reboot, install VMware tools if not already installed, and reconfigure your IPs. Total downtime for this operation should be 5-10 minutes per 2012 and 2012 R2 VM.

After implementing these steps I have successfully avoided another PSOD in both lab and production. Several other colleagues with similar issues report the fix above resolved the problem for them as well.

Resetting Forgotten Windows Admin Password

One of the “fun” experiences I have run into in my IT career is the situation where domain trust is broken or the network is not configured, but you absolutely have to get into the box in question. The trouble is the password was set by someone who was in the position before you and there is no documentation. I have come across 2 ways to do this, one of which I consider to the conventional bare metal method and the Hyper-V guest method.

 

Method 1 (Bare Metal Method):

If you have a physical server or a virtual machine that is in another format other than VHD to VHDX use this method. Essentially what we will be doing is booting from the windows disk into a command prompt repair option, backing up and renaming utilman.exe and making a copy of cmd.exe name utilman.exe. This will allow us to reboot to login prompt, press windows + U (which ordinarily brings up utilman.exe) which will launch a command prompt, from where we can reset the administrator password using the net user commands.

The first step is to insert the windows disk, reboot and choose to boot from disk. Once you are on the the screen that present the option to install windows or repair your computer

page1

 

repair-your-computer

Choose the troubleshoot option, and then select to command prompt.

Once you are booted into the command prompt you will need to type the following commands (note D is the normal drive for C: in this prompt, if you primary drive is labeled something other than C you can use diskpart to identify the proper disk)

  • D;
  • cd windowssystem32
  • ren utilman.exe utilman.bkp
  • copy cmd.exe utilman.exe

Close the command prompt, eject the disk and restart. Once you boot to the sign in screen press windows key + U then type the following:

  • net user administrator Password123!

Close the command prompt and login as administrator with the password you just set. Once you have completed this successfully you will want to remove the utilman.exe application and rename utilman.bkp to utilman.exe.

 

 

Method 2 (Mounting VHD Method):

If you are utilizing Hyper-V or another virtualization platform that utilizes VHD or VHDX files you have another option, which is to mound the VHD and perform similar steps to the above. If the VM in question is running shut it down. Once the virtual machine is shut down on the Hyper-V host machine open disk management (diskmgmt.msc) and choose the “action” menu, then select “attach VHD”.

attach vhd

Once you select attach VHD, you’ll need to navigate to the directory where your Hyper-V VHDs are stored (by default this is C:UsersPublicDocumentsHyper-VVirtual Hard Disks). Select the VHD and mount it. Once the VHD is mounted you’ll see it show up as a drive letter, navigate to this drive letter, choose the windowssystem32 directory. You will need to right click on utilman.exe and choose properties, select the security tab, choose advanced security and change ownership to your user account.

utilbak

Once you have taken ownership and applied changes you will be able to add your account on the security tab and grant full control. Once this is applied you will then have the permissions needed to change the filename to utilman.back. Once this is done locate cmd.exe in same directory, right click and copy, then paste and rename to utilman.exe. Unmount the VHD  by opening disk managment (diskmgmt.msc), highlight the disk for the VHD you mounted, then choose “action>all tasks>detach VHD”

detach

Once you have unmounted the VHD you can boot the Hyper-V virtual machine and press ctrl+U at the login prompt to reset the password via command prompt with the following command:

  • net user administrator Passwor123!

Once you are successfully in the server and you need to perform the same cleanup tasks as in method 1. If you run into issues moving utilman.bak to utilman.exe you can try shutting down the VM and re-mounting the VHD to the Hyper-V host and change these options, unmount the VHD and reboot the Hyper-V system.

 

Thank you all for reading and I hope this article is helpful. This article is intended to assist administrators to access systems they have rights to access but lack the password. This article is in no way mean to help or endorse illicit access to systems, please use this information carefully and please act ethically and only use this to access systems you own and/or are authorized to access.

Practical Powershell Part 2 of 2

It has been almost a month since the first installment of practical powershell. It’s been an incredibly busy month of finishing my Linux + and LPIC 1 certifications. Now that those are all out of the way it’s time to get back to blogging! For this post I’ll cover several powershell topics that range from the everyday to the not so everyday including adding MAC addresses to DHCP allow lists on DHCP servers configured using the integrated DHCP failover architecture in Server 2012 and later, Adding DNS records, Backing up GPOs, and pulling some important info from Exchange 2010 such as Mailbox size listing and listing whether or not distribution lists are hidden from the global address list.

 

DHCP Whitelisting

While you can manually log into your DHCP server, open the MMC and add the second DHCP server into the MMC, this results in redundant data entry and can be prone to human error. As illustrated in the previous post we will again use the invoke-command and -scriptblock cmdlets to accomplish this task. For the same of demonstration we will refer to the DHCP servers and Server1 and Server 2. Below is an example Powershell one-liner to add this:

Invoke-Command -Computername Server1,Server2 -ScriptBlock {Add-DhcpServerv4Filter -List Allow -MacAddress 00-00-00-00-00-00 

Note that the MAC address must be written in a hyphen separated format, if you attempt to enter this with colons it will fail. Additionally if you wish to keep your DHCP allow list organized you can optionally add a description (personally I use the computer’s hostname) by adding -Description “somehostname” at the end of the above Powershell statement.

 

Adding DNS Records

While not exactly a common everyday scenario, you may find yourself in a place where you need to rapidly create numerous DNS records, perhaps this is for a new zone or you are rolling out a series of servers that will need to be added to DNS. While it is arguably easier to use the dnsmgmt.msc GUI tools, Powershell wins if you decide to script this out. The  basic powershell cmdlet for all DNS records is Add-DnsServerResourceRecord(A,CNAME,MX,etc). This most common usage will be A records since these are often the most common record types entered into DNS. The below example shows the creation of an A record for a server named DAL-MEM27 in the zone named bigcompany.com.

Add-DnsServerResourceRecordA -Name “DAL-MEM27” -ZoneName “bigcompany.com” -AllowUpdateAny -IPv4Address “172.16.2.26” -TimeToLive 01:00:00

Additional resource record types can be created in powershell as well. For more information on this, refer to any of the Microsoft technet articles associated with the topic.

 

Backing up GPOs

Let’s face it, if you’ve been a Windows administrator in an environment with more than 10 client machines, you’re inevitably using group policy to some extent. Whether it’s a handful of IE settings, mapping network drivers, running scripts, installing software, etc, your group policies have take some time, thought, and finessing. Whether it’s to protect yourself from accidental changes or deletions or GPOs or part of your DR strategy, it’s a good idea to periodically take backups of your GPOs. The first way I will illustrate below is taking a backup of all GPOs and dumping them on a share via UNC path or by referencing a local destination. If you want to automate this process, you could turn this into a powershell script and leverage task scheduler to run this backup periodically for you.

Backup-Gpo -All -Path \Fileservermygpobackups -Comment “Scheduled GPO Backup”

If you’re interested in backing up a single GPO, this can be done one either by explicitly calling the GPO by its name or by referencing it’s GUID. To find the GUID of the GPO in question, you can go to \domain.localsysvolpolicies to view the GPOs in their folder titles by the GUID name. In the example below we’ll look at backing up the GPO by GUID to a local location, followed by an example of backing up a GPO by name to file share.

Backup-Gpo -GUID yourguidhere -Domain “bigcompany.com” -Server DC-01 -Path C:BackupRepoGPOBackup

Backup-Gpo -Name DriveMapScript -Path \Fileserverdrivemapscriptgpobackup 

To restore your backed up GPOs, you can either import them or use the Restore-GPO or Import-GPO cmdlets.

 

Exchange Mailbox Size Report

For the past year and half I have worked for an organization that has ridiculously large volumes of email. This has lead to the need to audit mailbox sizes about once per quarter to identify mailboxes with extreme space usage to see if we can reduce some space either through archival or deletion of mail. The below Powershell cmldet needs to be run on the Exchange Shell (please note I have only used this in Exchange 2010 but it should also work for Exchange 2013):

Get-Mailbox | Get-MailboxStatistics | Sort-Object TotalItemSize -descending | ft displayname,totalitemsize | Export-Csv -Path C:MailboxSizeAudit.csv

If you don’t want a CSV file to be created you an simply leave off the final pipe and everything after it.

 

Exchange Distribution List Hidden From GAL True/False

In many organizations, it’s commonplace to create distribution lists for former employees and direct their mail to a supervisor and possibly anyone else who will be taking on that individuals work. However most of the time we don’t want these ex-employees cluttering up the global address list. However auditing whether or not the option to hide from exchange address book is checked or not for each distribution list is a very tedious manual process involving a great deal of clicking, cursing, and drinking coffee. Another common attribute in my environment is to ensure the require sender authentication option is disabled for ex-employee distrubtion lists. However we do want this enabled or large internal DLs to prevent spam. This simple Powershell command string below will export a CSV list that you can import into Excel and filter using table formatting to quickly and effectively audit your distribution lists and associated attributes.

Get-DistributionGroup -ID “*” | Select Name,OrganizationalUnit,HiddenFromAddressListEnabled,RequireSenderAuthenticationEnabled | Export-Csv -Path C:ReportsDL_Hidden_Status.csv

 

I’m sure this will not be the last Powershell blog post, as Powershell is a topic near and dear to my heart. Thanks for sticking with me through my almost month long absence from blogging. I look forward to dedicating more time to more frequent blogging.