End of Year Gotchas

Mac OS Sierra Git Broken After Upgrade:

Every year when a new Mac OS is released I am excited to see what new features have come down the pipe but also fearful of what is going to break. Without fail ever release since Mavericks has broken something in my environment whether it be DevOps related or Audio Engineering related. This time around it was xcode which impacted my ability to use git bash. Fortunately the fix was easy and quick.

Below is the error I received the first time I used git bash after updating.

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun


xcode-select --install

If this doesn’t resolve for you try using the –reset flag instead of –install

AWS Lambda Invocation Limits:

I’ve had the privilege of doing DevOps work for a few fairly popular mobile games over the past 6 months. About 3-4 months ago we made the switch from Tomcat backend servers and SQL to using Amazon RDS and Lambda with API Gateway. This has proven to drastically reduce total cost of ownership (TCO) and make scaling a breeze. The challenge came when we started to grow the number of Lambda functions and encountered a not so widely known limitation. Lambda has a limit of 100 requests per second, which is honestly quite a bit, however this is GLOBAL not per Lambda. The more Lambda functions you have in your account the more this will grow. I have since requested 2 increases from Amazon to prevent throttling and setup some global CloudWatch dashboards and alarms, but this is valuable information I wish I knew before limitations were encountered.

Happy Holidays and 2nd Blog-Anniversary

Well, its that time of year again where snow is upon us and houses are filled with families celebrating the Holidays. 2 years ago I started this blog while working as a systems admin at a small company finding my path in the IT world and posting lessons I’ve learned along the way. Over the past 2 years my career has grown from a virtualization and Windows admin to working as an Operations Engineer for a hosting company primarily working on Linux systems and various hypervisors. I’ve now found myself in a DevOps role as a cloud engineer working on high availability cloud architecture for household name entertainment companies. Blogging has been a passion of mine throughout my career as i have relied on the blogs of others as well as documentation to find my way out of difficult issues. For things I’ve run into that weren’t terribly intuitive I felt it my responsibility to contribute back to the community in hopes that I can save someone else the same headache I went through to fix an issue. As I come to the 2nd anniversary of the blog I just want to say thank you to all who have come here and read posts and wish all of you happy holidays and happy new year. Also if you’re an IT professional I highly encourage you to blog and contribute back to the community, even if you feel like you don’t have much to contribute, I guarantee you do.

What TechOps Can Learn From Manufacturing

When I first embarked on my journey to complete my MBA in Business IT Management I had largely worked technical support jobs and had just recently landed my first Systems Administrator job in what was largely a Windows shop. Shortly into my experience there I found myself taking over an AWS infrastructure on Linux with Puppet for configuration management. During my MBA schooling I found myself not understanding the purpose of learning so much about manufacturing operations when my degree was on a technical path. If I could go back in time I think I’d smack myself and relate what I know now to my past self. In many ways schools with MBAs that have a technical focus are sort of missing the boat when it comes to relating the importance of studying manufacturing to future IT managers, CTOs and CIOs.

I recently finished reading the book “The Phoenix Project” and it opened my eyes to some things. In the book the main character takes over managing what is an all too typical IT Ops department, constant fires, outages, tribal knowledge, lack of communication between development teams, lack of consistency in environments from dev to prod. Without giving away too much of the read, the book underscores the importance for embracing DevOps methodologies. 

So what can technical operations learn by studying manufacturing and embracing DevOps methodologies. The answer is quite simply all the things. When we think back to the invention of the assembly line, Henry Ford was able to churn out more cars at a quicker pace by having assembly lines where jobs were done with repeatable processes and a limited number of options. The modern computer is made up of countless parts made by countless different manufacturers and tech stacks, but the establishment of things like IEEE , W3C, and others is a common unifying theme of a body of standards and interoperability requirements. The first step to embracing DevOps methodologies is to begin to look at your work flows and learn what can be standardized into repeatable and reusable processes. One of the best places to start this is to standardize an imaging platform, whether that be using a customized AMI in AWS, virtual machine templates in your hypervisor platform, or using a PXE service like WDS for Windows or Cobbler for Linux. By setting up a customized image that gets regularly updated you can install some baseline utilities, have the most current updates installed to minimize initial patching time and bootstrap your configuration management agent on the image.

One of the most amazing manufacturing capabilities that’s come about in the past half century is going beyond mass production of a single item to what is called mass customization. Where certain customized components can be mass produced as interchangeable parts or production lines can easily switch between colors and styles without tremendous effort being expended. The tech equivalent of that is configuration management. Time and time again we’ve heard the mantra of infrastructure as code and this is where configuration management truly shines. By creating Puppet modules, Chef recipes, Salt states, and Ansible playbooks we are essentially codifying what our configuration should look like on a system. This allows you to create modular desired states for various components and applications that can be added and removed with great flexibility. Always start with what you already know how to do manually before building automation for it, otherwise you could be creating an automated wrecking ball, and nobody wants that. Over time your configuration management can mature to a place of using variables and manage every aspect of your applications up to the point of deploying code. By bootstrapping your images with configuration management and setting up mature config management implementation the process of building out a new server can become completely automated or nearly automated.

In the world of manufacturing a great deal of time and money is spent to coordinate the most efficient workflows, to identify bottlenecks, and learn what processes can run concurrently and which processes act as blockers for other processes further down the line. This concept becomes important when we move to the world of code deployment. Yes we’re talking about Continuous Delivery and Continuous Integration (CI/CD). Once you understand what it takes to get code from a repository into a live running system you can begin to build your deployment pipeline. Several great tools exist for this including Jenkins, Atlassian Bamboo, Travis CI, and others. In the dark ages of tech developers would build out code on their local environments which almost never matched production, flung the code to Ops who in some cases used sketchy methods such as FTP or RDP file copy to get code on machines and the results were often chaotic and lots of blaming happened. In a well developed deployment strategy developers will work on code in feature branches, those branches will be merged using pull requests into environment branches which are then automatically run through unit testing and deployed by the CI/CD tool if they pass all required tests. Of course another key element is to make sure that dev, staging, and production environments are all configured on the same tech stacks and have consistency so that code behaves in the same way across the board. Configuration Management is also a great way to ensure the consistency between environments.

One thing that is fundamental to lean manufacturing is the need for constant improvement, monitoring the system for efficiency, looking for ways to further automate and improve, creating tight feedback loops. When applied to tech this looks like dedicating time for R&D, constantly looking for new tools and new ideas to better streamline what you have built and ensure that you don’t grow complacent in the face of ever changing tech stacks and tools. This also looks like setting up deployment and configuration management logging, automating patch management, creating automated reports and alerts around this logging so that you know the moment there is a problem, when the problem begins, where the problem exists, and mobilize to resolve it. Additionally the tighter integration between dev teams and operations teams creates an atmosphere of mutual trust and collaboration and allows for tight feedback loops at earlier stages within the development process.

Wrapping up this post I can only hope that as IT becomes a fundamental part of every company, and the tech must be a core competency for modern companies to out maneuver competition, slash costs, improve quality while improving agility, and that more is done to teach these important concepts to MBA students, not only in the context of manufacturing but in the context of technology. We have entered an era where a company’s technical aptitude can make or break their success, every company is a tech company now, whether you are a retailer with complex ERP, inventory, and logistics systems, or a biopharmaceutical company crunching mass amounts of data into human usable scientific data the game has shifted dramatically in recent years. Happy automating all the things!