Unable to Manage VMs in Hyper-V

Recently I came across a situation where a set of VMs running on cluster shared storage were all in an un-manageable state. They were not listing as being present on the host and exhibited the error shown in symptoms:

Symptoms

Virtual Machines show in a running or paused state but have the following error when attempting to access console, settings, or take any other management actions:

Screen Shot 2015-12-26 at 12.45.36 PM

Additionally try listing the VMs in Powershell

Get-VM

If the VM does not show in the list it is definitely in this state.

Problem

Check to see if any other VMs exhibit this issue and if they are on cluster shared storage. If you are using it make an exclusion for the Cluster Volume or disable AV as this is usually the culprit.

Remediation

Install Sysinternals process explorer here: https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx

Next we will locate the GUID of the VM in question

Screen Shot 2015-12-26 at 12.10.36 AM

Once you have located the GUID open process explorer as administrator and locate right click the VWP.exe processes and check the commandline entry under the “image” tab. This will display the GUID of the VMs. Select the VWP.exe that matches your trouble VM and kill the process.

DO NOT kill any other processes or you will potentially bring down other VMs

Screen Shot 2015-12-26 at 12.10.23 AM

Once you have done this you will need to restart the Hyper-V management stack by restarting the VMMS process. I have found some of these will not work properly with the restart-service cmdlet, so you will need to stop and then start the service. This does not effect the running VMs, it only effects your ability to manage them while this service is stopped.

Get-Service vmms
Stop-Service vmms
Start-Service vmms
Get-Service vmms

Now you should be up and running!

Hyper-V Paused Critical

As many of you reading my blog have realized I am a huge proponent of datacenter virtualization. So here we are with a nicely virtualized Hyper-V failover cluster, we’ve got excellent guest to host density and everything is going well….until it isn’t. I was taking some time off when I received a call in the middle of the night that one of my customer’s entire infrastructure was down. Upon logging in it was apparent something was quite wrong, the failover cluster had failed convergence and the local Hyper-V MMC showed that most over 75% of the VMs were in a paused critical state (see the screenshot below as an example that was reproduced in a lab).

 

pausedcritical

As it turns out one of the cluster shared volumes had reached a point where > 200MB was remaining on the volume. Hyper-V will begin writing to the event log when the volume drops below 2GB with an event log entry for each VM. Once the space drops below 200MB all VMs on that volume will enter a paused critical state. This only applies to thin provisioned disks however at the time of writing this there is a bug that this will occur even with thick provisioned disks. The fun part in my scenario was that both domain controllers were residing on this volume. To remedy the issue I was able to forcibly power-off the domain controllers from their paused state and storage migrate them to another cluster volume. Upon bringing them back up cluster convergence was able to be restored and everything was up and running again. The other thing of note is that the clocks were all paused during this time resulting in major time drift that had to be manually reconciled.

 

Lessons Learned

  • ALWAYS ensure your storage admin has monitoring on the SAN to send alerts when the LUNs reach a critical threshold and warn before that at a lower threshold
  • If you have multiple LUNs keep your Domain Controllers spread across them

Resetting Forgotten Windows Admin Password

One of the “fun” experiences I have run into in my IT career is the situation where domain trust is broken or the network is not configured, but you absolutely have to get into the box in question. The trouble is the password was set by someone who was in the position before you and there is no documentation. I have come across 2 ways to do this, one of which I consider to the conventional bare metal method and the Hyper-V guest method.

 

Method 1 (Bare Metal Method):

If you have a physical server or a virtual machine that is in another format other than VHD to VHDX use this method. Essentially what we will be doing is booting from the windows disk into a command prompt repair option, backing up and renaming utilman.exe and making a copy of cmd.exe name utilman.exe. This will allow us to reboot to login prompt, press windows + U (which ordinarily brings up utilman.exe) which will launch a command prompt, from where we can reset the administrator password using the net user commands.

The first step is to insert the windows disk, reboot and choose to boot from disk. Once you are on the the screen that present the option to install windows or repair your computer

page1

 

repair-your-computer

Choose the troubleshoot option, and then select to command prompt.

Once you are booted into the command prompt you will need to type the following commands (note D is the normal drive for C: in this prompt, if you primary drive is labeled something other than C you can use diskpart to identify the proper disk)

  • D;
  • cd windowssystem32
  • ren utilman.exe utilman.bkp
  • copy cmd.exe utilman.exe

Close the command prompt, eject the disk and restart. Once you boot to the sign in screen press windows key + U then type the following:

  • net user administrator Password123!

Close the command prompt and login as administrator with the password you just set. Once you have completed this successfully you will want to remove the utilman.exe application and rename utilman.bkp to utilman.exe.

 

 

Method 2 (Mounting VHD Method):

If you are utilizing Hyper-V or another virtualization platform that utilizes VHD or VHDX files you have another option, which is to mound the VHD and perform similar steps to the above. If the VM in question is running shut it down. Once the virtual machine is shut down on the Hyper-V host machine open disk management (diskmgmt.msc) and choose the “action” menu, then select “attach VHD”.

attach vhd

Once you select attach VHD, you’ll need to navigate to the directory where your Hyper-V VHDs are stored (by default this is C:UsersPublicDocumentsHyper-VVirtual Hard Disks). Select the VHD and mount it. Once the VHD is mounted you’ll see it show up as a drive letter, navigate to this drive letter, choose the windowssystem32 directory. You will need to right click on utilman.exe and choose properties, select the security tab, choose advanced security and change ownership to your user account.

utilbak

Once you have taken ownership and applied changes you will be able to add your account on the security tab and grant full control. Once this is applied you will then have the permissions needed to change the filename to utilman.back. Once this is done locate cmd.exe in same directory, right click and copy, then paste and rename to utilman.exe. Unmount the VHD  by opening disk managment (diskmgmt.msc), highlight the disk for the VHD you mounted, then choose “action>all tasks>detach VHD”

detach

Once you have unmounted the VHD you can boot the Hyper-V virtual machine and press ctrl+U at the login prompt to reset the password via command prompt with the following command:

  • net user administrator Passwor123!

Once you are successfully in the server and you need to perform the same cleanup tasks as in method 1. If you run into issues moving utilman.bak to utilman.exe you can try shutting down the VM and re-mounting the VHD to the Hyper-V host and change these options, unmount the VHD and reboot the Hyper-V system.

 

Thank you all for reading and I hope this article is helpful. This article is intended to assist administrators to access systems they have rights to access but lack the password. This article is in no way mean to help or endorse illicit access to systems, please use this information carefully and please act ethically and only use this to access systems you own and/or are authorized to access.