Month: November 2015

Hyper-V Paused Critical

rstaatsLeave a comment

As many of you reading my blog have realized I am a huge proponent of datacenter virtualization. So here we are with a nicely virtualized Hyper-V failover cluster, we’ve got excellent guest to host density and everything is going well….until it isn’t. I was taking some time off when I received a call in the middle of the night that one of my customer’s entire infrastructure was down. Upon logging in it was apparent something was quite wrong, the failover cluster had failed convergence and the local Hyper-V MMC showed that most over 75% of the VMs were in a paused critical state (see the screenshot below as an example that was reproduced in a lab).

 

pausedcritical

As it turns out one of the cluster shared volumes had reached a point where > 200MB was remaining on the volume. Hyper-V will begin writing to the event log when the volume drops below 2GB with an event log entry for each VM. Once the space drops below 200MB all VMs on that volume will enter a paused critical state. This only applies to thin provisioned disks however at the time of writing this there is a bug that this will occur even with thick provisioned disks. The fun part in my scenario was that both domain controllers were residing on this volume. To remedy the issue I was able to forcibly power-off the domain controllers from their paused state and storage migrate them to another cluster volume. Upon bringing them back up cluster convergence was able to be restored and everything was up and running again. The other thing of note is that the clocks were all paused during this time resulting in major time drift that had to be manually reconciled.

 

Lessons Learned

  • ALWAYS ensure your storage admin has monitoring on the SAN to send alerts when the LUNs reach a critical threshold and warn before that at a lower threshold
  • If you have multiple LUNs keep your Domain Controllers spread across them

Removing Passphrase From SSH Key

rstaatsLeave a comment

So generally speaking it’s always preferable to use a passphrase with your SSH Keys, however there are times when a passphrase may get in the way, particularly with service accounts that run cron jobs or connect out to other servers in your environment. Rotating keys semi-frequently would be a wise security precaution to prevent your key from becoming compromised. That said lets go through the process of stripping out the passphrase from an existing key.

Remove Passphrase From Key

openssl rsa -in ~/.ssh/id_rsa -out ~/.ssh/id_rsa_nopass

This will create a new copy of your key without a password

 

Modify Permissions Remove Old Key

First we need to remove the old key and rename the new one

rm id_rsa
mv id_rsa_nopass id_rsa

Now we need to fix permissions:

chmod 600 id_rsa

Increasing Screen Resolution of VMs in Citrix XenServer 6.x

rstaatsLeave a comment

In my home lab I have a whole host of servers, the one I run all my home production off of is a Citrix XenServer 6.5 box running on a Dell PowerEdge R415. Recently I had a friend ask me to setup a Windows 7 VM for them to use with TeamViewer for their Quickbooks accounting and tie it into my offsite backups. This is ordinarily a non-issue for me as most of my VMs are Linux and the handful of Windows VMs I run are typically used via RDP. Upon setting up this VM with TeamViewer I discovered that TeamViewer would not allow the resolution to go beyond the limitations of the supported VM resolution. While the default 800×600 or 1024×768 has never been a problem, it did prove problematic in this case. I was able to find a workable solution however, which i was the change several of the video related VM parameters in the CLI on the Citrix host.

Below is a screenshot of the resolution options before making the change.
Screen Shot 2015-11-07 at 2.55.44 PM

To Make Changes

[root@intrepid ~]# xe vm-shutdown uuid=
[root@intrepid ~]# xe vm-param-set uuid= platform:vga=std
[root@intrepid ~]# xe vm-param-set uuid= platform:videoram=16
[root@intrepid ~]# xe vm-start uuid=

 

To Roll Back

[root@intrepid ~]# xe vm-param-remove uuid= param-name=platform param-key=vga
[root@intrepid ~]# xe vm-param-remove uuid= param-name=platform param-key=videoram

 

Below are the results resolution options after making the changes
Screen Shot 2015-11-07 at 2.56.39 PM