Lambda and Infrastructure as Code

Amazon Lambda for those not familiar with it is defined as an event driven serverless platform for compute resources. Rather than paying for an EC2 instance or running a docker instances on ECS that runs executable code, Lambda offers an alternative that is for many much less expensive. For instance if you need to routinely resize photos that are provided to you from various distributed sources, you can create an S3 bucket for images to be loaded to, which can act as a trigger for a Lambda job to resize them and output the resized photos into a different bucket.

There are a plethora of useful things you can leverage Lambda for. You can use it as a backend service for mobile applications that need to call databases, post data, etc. Lambda can also be used on a cron to run an EBS snapshot backup routinely as well as snapshotting RDS and replicating snapshots to another region to facilitate a disaster recovery scenario. Lambda can also be built to run repeated jobs such as resizing images, reacting to cloudwatch metrics and actioning accordingly (rebooting systems with pegged CPU, etc), or hitting an analytics API and loading the parsed data into a data warehouse.

I’m going to cover supported languages, some of the ways Lambda can be invoked, as well as a discussion of different framework options available to incorporate Lambda into your Infrastructure as code strategy, setting up a cron based Lambda to snapshot EBS volumes, and finally some gotchas of using Lambda.

Supported Languages

First and foremost the languages supported in AWS Lambda at the time of this writing include the following Node.js, Java, C# and Python(2.x and 3.x). To keep up to speed with supported languages for Lambda you can refer to their FAQ page located here.

Lambda Invocation Methods

Methods for invoking Lambas currently includes CloudWatch alerts, Cron style invocation, S3 trigger, API Gateway triggers, SES and others. A full list of AWS services that can act as Lambda triggers can be found here.

Lambda/Serverless Frameworks

Over the past few years a number of frameworks have emerged for building and maintaining serverless resources in AWS. I will break these down into broadly 2 catagories, the first category is a more traditional approach to infrastructure as code in which you define each of the peices needed for your lambda operation and use the framework to generate CloudFormation templates and apply them. The second category is frameworks specifically built to update and maintain your Lambda and serverless resources such as API Gateway and Dynamo DBs throughout their full life cycle.

Infrastructure as Code Frameworks:

Serverless Frameworks:

Your mileage may vary with each of these choices. Tools like Claudia, Node Lambda, and Serverless are built on node and will require npm and node locally to run. Tools such as Zappa and Chalice are Python based. Each of these frameworks supports more than their native language in Lambda, however depending on your workflow, development environment, and any CI processes you have this may sway your decision.

Lambda Gotchas

As with any cloud service offering there are configuration maximums and limitations to be aware of. It is best to be aware of these limitaitons and maximums ahead of time so there are no suprises down the road as demand increases for your application and your scaling needs change. The information outlined here is based off of current limitations, please ensure you are using the most up to date information by checking here.

Service Limits:

  • Invocations Per Second: 100 (this limit can be increased by requesting a limit increase from AWS support, please note this can take more than a day)
  • Concurrent Executions: 1000 per region
  • Max Execution Duration: 300 seconds
  • Memory Allocation: Min:128mb Max:1536MB
  • Deploy Package Size(compressed): 50MB
  • Total Size of All Deployment Packages Per Region:75GB

Some other things worth mentioning are that if you are running a cron based Lambda function for things such as snapshotting EBS volumes it is important to monitor these jobs, as the size of the backup set grows it will take longer to execute the lambda function. You can setup monitoring on an invididualized bassis in CloudWatch. While these individualized monitors are a fantastic idea it is also worth setting up some generalized blanket monitoring such as lambda invocation rate per minute and Lambda error rate. This way you are alerted when you are approaching your service limites for Lambda invocations and can request service increases, or be aware if there is a spike in Lambda error rates.

Setting up EBS Snapshotting Using Lambda and Terraform

Rather than endlessley rambling about all the cool features and things you can do with AWS I wanted to create an actual get your hands dirty dive into Lambda. For this lab we’ll setup a Lambda running on cron trigger to snapshot our EBS volumes and another lambda to delete snapshots that are more than 90 days old. For this example we will set this up using terraform. To follow along visit the Github project here, follow the instructions in the readme and profit! The code for the snapshot and snapshot management is originally from this repo.

Final Thoughts

Hopefully this post as has been helpful in seeing ways you can use Lambda and what tools and frameworks are available to use with your Lambda projects. Thank you for taking the time to read this post, I look forward to creating future blogs around serverless resources and other infrastructure as code topics.

Installing pyautogui on Ubuntu Server

The Issue

I was recently assisting a QA engineer with some #Fun™ problems on Jenkins where tests running pyautogui (running under Python 2.7) were failing because pyautogui was not installed. Every time we attempted to pip install pyautogui it would give an error similar to the following:

Downloading/unpacking pyautogui
Downloading PyAutoGUI-0.9.36.tar.gz (46kB): 46kB downloaded
Running setup.py (path:/tmp/pip-build-tbxln81q/pyautogui/setup.py) egg_info for package pyautogui
Traceback (most recent call last):
File "", line 17, in
File "/tmp/pip-build-tbxln81q/pyautogui/setup.py", line 6, in
version=import('pyautogui').version,
File "/tmp/pip-build-tbxln81q/pyautogui/pyautogui/init.py", line 115, in
from . import _pyautogui_x11 as platformModule
File "/tmp/pip-build-tbxln81q/pyautogui/pyautogui/_pyautogui_x11.py", line 160, in
_display = Display(os.environ['DISPLAY'])
File "/usr/lib/python3.4/os.py", line 633, in getitem
raise KeyError(key) from None
KeyError: 'DISPLAY'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "", line 17, in

File "/tmp/pip-build-tbxln81q/pyautogui/setup.py", line 6, in

The Fix

Ensure you have the following packaged installed: xserver-xorg, python-pip, python-xlib, xfce4 (you can use a different desktop env such as Gnome or KDE), and xauth.
sudo apt-get install -y xserver-xorg python-pip python-xlib xfce4 xauth
Also ensure that once you’ve install pip that you upgrade pip by running the following:
pip install --upgrade pip
Open a GUI session either through xrdp (if installed) or console. Once you are here run the following command:
echo $DISPLAY
Open an SSH connection to the server and set the $DISPLAY environment variable to match the output of the command above, in my case it was :10.0
export DISPLAY=:10.0
You can confirm this value is set by running the echo command again. Once this is in place run the following to install pyautogui
pip install pyautogui

Other Errors
If you receive any other errors around xauth or xserver try purging and reinstalling the packages above. You may also need to install a few other pip packages such as pillow and xlib.

Managing IIS With Puppet via Powershell DSC

Use Case:

Many of us in DevOps/Cloud/Site Reliability/System Engineering have long been using Puppet to manage our Apache and Nginx configurations. However it seems like every Windows deployment I come across using IIS is either not in any form of config management, or is managed through a series of VBScript and PowerShell scripts or System Center. If your organization is big enough to afford System Center this isn’t really much of an issue for you; however for those of us managing IIS web farms without SCCM, Microsoft has extended a nice olive branch with the delivery of PowerShell Desired State Configuration.

Given that most of us work with such broad toolsets, the idea of adding yet another configuration management platform into the mix seems less than ideal. However, there is an official Puppet Forge module that allows you to leverage PowerShell DSC within your Puppet code. What I found in my journey of doing this is that, while there is some decent documentation, there are pieces that clearly seem not to be well documented that required figuring it out the hard way. In this blog post I’ll cover and distill what I’ve learned in configuration of IIS through PowerShell DSC via Puppet.

Prerequisites:

This blog post will assume that you have already installed the PowerShell DSC forge module. I will link the module and its dependencies below; if you are unsure how to install Forge modules, see the documentation on the link to the module below.

This blog post will also assume that you have already installed IIS either in your AMI/Template or via Puppet. Great, now that that’s all out of the way, let’s get started.

 

Overview Example Class:

Before diving into the weeds in each section, I wanted to provide a code sample of a single application pool, website, virtual directory, and web application, complete with the DSC dependencies:

 

class webhosting::clientname::sites::dev {
 
dsc_windowsfeature{'iis':
  dsc_ensure => 'Present',
  dsc_name => 'Web-Server',
}
dsc_windowsfeature{'aspnet45':
  dsc_ensure => 'Present',
  dsc_name => 'Web-Asp-Net45',
}
dsc_windowsfeature{'iisscriptingtools':
  dsc_ensure => 'Present',
  dsc_name => 'Web-Scripting-Tools',
}
 
#App Pool Creation
dsc_xwebapppool{'dev.example.com':
  dsc_name => 'dev.example.com',
  dsc_ensure => 'Present',
  dsc_enable32bitapponwin64 => true,
  dsc_managedruntimeversion => 'v4.0',
  dsc_managedpipelinemode => 'Integrated',
  dsc_identitytype => 'ApplicationPoolIdentity',
  dsc_state => 'Started',
  require => Dsc_windowsfeature['iis'],
}
 
#Website Creation
dsc_xwebsite{'dev.example.com':
  dsc_ensure => 'Present',
  dsc_name => 'dev.example.com',
  dsc_state => 'Started',
  dsc_physicalpath => 'D:\\www\\dev.example.com\\webroot',
  dsc_applicationpool => 'dev.example.com',
  dsc_bindinginfo => [{
  ipaddress => '*',
    protocol => 'HTTP',
    port => 80,
    hostname => 'dev.example.com',
    },
    {
    ipaddress => '*',
    protocol => 'HTTPS',
    port => 443,
    hostname => 'dev.example.com',
    certificatethumbprint => '<certificatethumbprintgoeshere>',
    certificatestorename => 'WebHosting',
    }],
 require => Dsc_xwebapppool['dev.example.com'],
  }
 
#Web Applications
dsc_xwebapplication{'AwesomeApp':
  dsc_ensure => 'Present',
  dsc_name => '/AwesomeApp',
  dsc_website => 'dev.example.com',
  dsc_webapppool => 'dev.example.com',
  dsc_physicalpath => 'D:\\www\\dev.example.com\\webroot',
  require => Dsc_xwebsite['dev.example.com'],
  }
 
#Virtual directories
dsc_xwebvirtualdirectory{'AwesomeApp':
  dsc_ensure => 'Present',
  dsc_name => 'AwesomeApp',
  dsc_website => 'dev.example.com',
  dsc_webapplication => 'AwesomeApp',
  dsc_physicalpath => 'D:\\webapplicatons\\AwesomeApp',
  require => Dsc_xwebapplication['AwesomeApp'],
  }
}

DSC Windows Feature

Starting at the top and taking this a chunk at a time, we see after the opening Puppet class statement that there are several dsc_windows_feature statements. These statements install the PowerShell DSC packages on the endpoint nodes. PowerShell DSC wrapped in Puppet assumes that you are using the pull model style of using DSC. What essentially happens on the Puppet run is that your Puppet DSC code gets converted into PowerShell DSC code and run locally on the machine. DSC module can be found at https://gallery.technet.microsoft.com/scriptcenter. Note that modules beginning with an X are considered experimental. That said, most of the experimental modules in my experience have been stable.

 

Configuring The Rest

Beyond this point, the rest of the Puppet code you will see will be prefixed with dsc_. Essentially, to write DSC code in Puppet you will use the same values outlined in the DSC documentation, just prefixed with the dsc_ and using the Ruby hash rockets => instead of =. See the example below:

DSC Example:

    xWebsite DefaultSite
        {
            Ensure          = "Present"
            Name            = "Default Web Site"
            State           = "Stopped"
            PhysicalPath    = "C:\inetpub\wwwroot"
        }

Puppet Example:

     dsc_xWebsite{'DefaultSite':
            dsc_ensure       => 'Present',
            dsc_name         => 'Default Web Site',
            dsc_state        => 'Stopped',
            dsc_physicalpath => 'C:\\inetpub\\wwwroot',
        }

At this point, you can configure your applications specific to your environments. Much like any other Puppet code there are obvious dependencies (can’t create a website that depends on an application pool unless the pool already exists). For this you can use the normal Puppet meta parameters such as before and require statements to resolve dependencies. For IIS-specific DSC parameters see the readme for Microsoft’s XWebAdministration module here:  https://github.com/PowerShell/xWebAdministration

 

Don’t forget to prefix the parameters in the readme with dsc_ when converting to Puppet code. Happy configuring and Puppeting! Until next time, may your servers always be up and your coffee mugs never empty!