Atlassian Bamboo Server Github Issues

As any of you who have been using Atlassian’s cloud offering of Bamboo know, it is going EOL at the end of January 2017. The migration has reasonably decent instructions to follow, but my migration ran into a big road block when linked repositories failed when choosing Github as the repository host with a 403 forbidden. The 403 forbidden is actually in relation to a javascript error and not the actual reason that this is failing. Tailing the output of the catalina.out revealed the issue was actually related to cross site request forgery issues. The reason this was happening is that I had setup nginx as an https proxy on 443 that routes traffic to 8085 on Tomcat.


I found the following link that indicates the resolution is to insert the following into your server.xml file


What was not terribly obvious in this documentation what what section this should go into. The section this should be placed in is the <Connector section within server.xml, see example below:


<Connector port="8085"

Last but not least don’t forget to restart the service for this to take effect. Hopefully this saves you hours of headache in your migration!

End of Year Gotchas

Mac OS Sierra Git Broken After Upgrade:

Every year when a new Mac OS is released I am excited to see what new features have come down the pipe but also fearful of what is going to break. Without fail ever release since Mavericks has broken something in my environment whether it be DevOps related or Audio Engineering related. This time around it was xcode which impacted my ability to use git bash. Fortunately the fix was easy and quick.

Below is the error I received the first time I used git bash after updating.

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun


xcode-select --install

If this doesn’t resolve for you try using the –reset flag instead of –install

AWS Lambda Invocation Limits:

I’ve had the privilege of doing DevOps work for a few fairly popular mobile games over the past 6 months. About 3-4 months ago we made the switch from Tomcat backend servers and SQL to using Amazon RDS and Lambda with API Gateway. This has proven to drastically reduce total cost of ownership (TCO) and make scaling a breeze. The challenge came when we started to grow the number of Lambda functions and encountered a not so widely known limitation. Lambda has a limit of 100 requests per second, which is honestly quite a bit, however this is GLOBAL not per Lambda. The more Lambda functions you have in your account the more this will grow. I have since requested 2 increases from Amazon to prevent throttling and setup some global CloudWatch dashboards and alarms, but this is valuable information I wish I knew before limitations were encountered.

What TechOps Can Learn From Manufacturing

When I first embarked on my journey to complete my MBA in Business IT Management I had largely worked technical support jobs and had just recently landed my first Systems Administrator job in what was largely a Windows shop. Shortly into my experience there I found myself taking over an AWS infrastructure on Linux with Puppet for configuration management. During my MBA schooling I found myself not understanding the purpose of learning so much about manufacturing operations when my degree was on a technical path. If I could go back in time I think I’d smack myself and relate what I know now to my past self. In many ways schools with MBAs that have a technical focus are sort of missing the boat when it comes to relating the importance of studying manufacturing to future IT managers, CTOs and CIOs.

I recently finished reading the book “The Phoenix Project” and it opened my eyes to some things. In the book the main character takes over managing what is an all too typical IT Ops department, constant fires, outages, tribal knowledge, lack of communication between development teams, lack of consistency in environments from dev to prod. Without giving away too much of the read, the book underscores the importance for embracing DevOps methodologies. 

So what can technical operations learn by studying manufacturing and embracing DevOps methodologies. The answer is quite simply all the things. When we think back to the invention of the assembly line, Henry Ford was able to churn out more cars at a quicker pace by having assembly lines where jobs were done with repeatable processes and a limited number of options. The modern computer is made up of countless parts made by countless different manufacturers and tech stacks, but the establishment of things like IEEE , W3C, and others is a common unifying theme of a body of standards and interoperability requirements. The first step to embracing DevOps methodologies is to begin to look at your work flows and learn what can be standardized into repeatable and reusable processes. One of the best places to start this is to standardize an imaging platform, whether that be using a customized AMI in AWS, virtual machine templates in your hypervisor platform, or using a PXE service like WDS for Windows or Cobbler for Linux. By setting up a customized image that gets regularly updated you can install some baseline utilities, have the most current updates installed to minimize initial patching time and bootstrap your configuration management agent on the image.

One of the most amazing manufacturing capabilities that’s come about in the past half century is going beyond mass production of a single item to what is called mass customization. Where certain customized components can be mass produced as interchangeable parts or production lines can easily switch between colors and styles without tremendous effort being expended. The tech equivalent of that is configuration management. Time and time again we’ve heard the mantra of infrastructure as code and this is where configuration management truly shines. By creating Puppet modules, Chef recipes, Salt states, and Ansible playbooks we are essentially codifying what our configuration should look like on a system. This allows you to create modular desired states for various components and applications that can be added and removed with great flexibility. Always start with what you already know how to do manually before building automation for it, otherwise you could be creating an automated wrecking ball, and nobody wants that. Over time your configuration management can mature to a place of using variables and manage every aspect of your applications up to the point of deploying code. By bootstrapping your images with configuration management and setting up mature config management implementation the process of building out a new server can become completely automated or nearly automated.

In the world of manufacturing a great deal of time and money is spent to coordinate the most efficient workflows, to identify bottlenecks, and learn what processes can run concurrently and which processes act as blockers for other processes further down the line. This concept becomes important when we move to the world of code deployment. Yes we’re talking about Continuous Delivery and Continuous Integration (CI/CD). Once you understand what it takes to get code from a repository into a live running system you can begin to build your deployment pipeline. Several great tools exist for this including Jenkins, Atlassian Bamboo, Travis CI, and others. In the dark ages of tech developers would build out code on their local environments which almost never matched production, flung the code to Ops who in some cases used sketchy methods such as FTP or RDP file copy to get code on machines and the results were often chaotic and lots of blaming happened. In a well developed deployment strategy developers will work on code in feature branches, those branches will be merged using pull requests into environment branches which are then automatically run through unit testing and deployed by the CI/CD tool if they pass all required tests. Of course another key element is to make sure that dev, staging, and production environments are all configured on the same tech stacks and have consistency so that code behaves in the same way across the board. Configuration Management is also a great way to ensure the consistency between environments.

One thing that is fundamental to lean manufacturing is the need for constant improvement, monitoring the system for efficiency, looking for ways to further automate and improve, creating tight feedback loops. When applied to tech this looks like dedicating time for R&D, constantly looking for new tools and new ideas to better streamline what you have built and ensure that you don’t grow complacent in the face of ever changing tech stacks and tools. This also looks like setting up deployment and configuration management logging, automating patch management, creating automated reports and alerts around this logging so that you know the moment there is a problem, when the problem begins, where the problem exists, and mobilize to resolve it. Additionally the tighter integration between dev teams and operations teams creates an atmosphere of mutual trust and collaboration and allows for tight feedback loops at earlier stages within the development process.

Wrapping up this post I can only hope that as IT becomes a fundamental part of every company, and the tech must be a core competency for modern companies to out maneuver competition, slash costs, improve quality while improving agility, and that more is done to teach these important concepts to MBA students, not only in the context of manufacturing but in the context of technology. We have entered an era where a company’s technical aptitude can make or break their success, every company is a tech company now, whether you are a retailer with complex ERP, inventory, and logistics systems, or a biopharmaceutical company crunching mass amounts of data into human usable scientific data the game has shifted dramatically in recent years. Happy automating all the things!