Python httpstat To Troubleshoot Connectivity

rstaatsUncategorized0 Comment

I’ve recently been working to diagnose intermittent latency issues from some HTTP calls that go through an outbound proxy. In an effort to determine whether the latency is at the destination server itself or is coming from the proxy I needed to so some digging.

After spending some time crafting a script in Python I found that I wanted more granular output than what I was getting from the options available using Python requests. This is when I stumbled upon this great github project called HTTPStat that leverages some fancy curl options and displays latency of the request in a waterfall style right in your terminal. https://github.com/reorx/httpstat

The process to install is a simple pip install httpstat. To use simply follow the readme on the github repo. For my case I needed to repeat a request several times in rapid succession and needed to include a few hundred lines of SOAP xml in my request. To do that I did the following:
– Create a soap.xml file containing just the SOAP xml
– Create a python script such as the example below

from subprocess import call

call("httpstat https://example.com -X POST --data-binary @/absolute/path/to/soap.xml --header 'Content-type: text/xml'", shell=True)

Assuming that you’re updated the above python snippet with a proper URI, absolute path, and the headers required you can now call your python script from the cli and you will see output similar to this (note the IPs have been obscured on purpose)

Connected to 111.222.333.444:443 from 444.555.666.777:37436

HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: application/soap+xml; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Tue, 21 May 2019 21:15:28 GMT
Content-Length: 1254

Body stored in: /tmp/tmpHm_Ts0

  DNS Lookup   TCP Connection   TLS Handshake   Server Processing   Content Transfer
[     4ms    |       9ms      |     83ms      |       215ms       |       377ms      ]
             |                |               |                   |                  |
    namelookup:4ms            |               |                   |                  |
                        connect:13ms          |                   |                  |
                                    pretransfer:96ms              |                  |
                                                      starttransfer:311ms            |
                                                                                 total:688ms

For bonus points if you want to repeatedly call the request to generate multiple samples you can simply run a bash one liner for loop such as this:

 for i in `seq 1 20` ; do python myhttptest.py ; done

Hopefully this has been helpful.

Using Dockerized Apache as a Reverse Proxy for Java

rstaatsUncategorized0 Comment

I recently was working on a project in an effort to replace a legacy development environment. Our dev environment had simply grown to require a multitude of items including ElasticSearch, DynamoDB, Apache, and MySQL. For Apache and MySQL the old environment was using a XAMPP stack which I’ve commonly used in other development environments in the past. The other components were previously installed using homebrew and required at least half a day for a dev to get their environment ready to run the application. Docker with Docker Compose seemed liked an easy enough solution for this as there are supported containers for each of the services listed above that make this easy to setup with minimal effort. Creating scripts within repos to load development data would also greatly decrease the setup time for a developer to be setup and productive on a project.

One of the biggest sticking points though was the Apache reverse proxy to Tomcat. During application development the Java application is run from within IntelliJ so that the code can be live edited and there are other services and reasons outside the scope of this blog for needing to use Apache to proxy calls on 80 and 443 back to the application on 8080. The challenge is you can map localhost 80 and localhost 443 to the container no problem, but getting back from the Apache container to Java proved to be somewhat of a challenge as the localhost or 127.0.0.1 referred to the container itself not the host.

Please note the solution below may not be optimal for everyone, relies on the users being on MacOS and using the interface en0 (this should be whatever your default network connection is).

To get around this my teammate and I discovered that you can map back using your host machine’s private IPv4 address however this is problematic as it would require modifying the container and it can change from time to time. After some experimenting we found that the simplest way to to get this to work was to add the following environemnt variable to your docker-compose.yml
environment:
- MYIP=${MYIP}

Next you will need to make sure your Apache container’s vhost file has a ProxyPass directive that includes this environment variable, this may look something like this:
ProxyPass / http://${MYIP}:8080/
ProxyPassReverse / http://${MYIP}:8080/

Next you will need to create a build script that sets the value to something. Then you’ll need a startup script that starts the docker-compose with a valid IP address.
The build script looks something like this:
MYIP=x docker-compose build --no-cache

The startup script looks something like this:
MYIP=$(ifconfig | grep -C5 en0 | grep inet | grep -v inet6 | awk ‘{print $2}’) docker-compose up -d ${1}

Squeezing More Life From Older Apple Hardware

rstaatsUncategorized0 Comment

Last year with the release of OSx Mojave support for many older models was dropped. While I do have a newer Macbook Pro I also have an old 2011 that’s still going strong thanks to its upgradability. As a DevOps professional that’s always on call I take a laptop with me nearly everywhere I go. This means dinner, concerts, band practice, bars, out of town trips etc. Some of these places I am concerned about theft and do not feel like risking having a $3000+ laptop stolen. So for some time now I’ve been using the trusty old 2011 Macbook Pro as my on call go everywhere machine. The only bummer is that Mojave is not supported on this hardware and that’s a shame as it is still plenty powerful.

Why would you want to use old hardware you ask? Put simply because I can and it fits the use case of what I’m doing. In fact I’m typing this blog right now on the old 2011 Macbook Pro. One of the biggest downfalls of the recent Apple hardware trends after 2012 is the lack of replaceable parts and the disposability of high dollar machines. For example if you don’t opt for a RAM upgrade at the time of purchasing a new Mac portable you’re stuck with that RAM permanently. Sure you can choose the RAM upgrade when you purchase but the cost of that RAM upgrade is many times more than what it would cost to do it yourself when those parts were user replaceable. The biggest problem here comes down waste, and less usable years out of an otherwise usable machine.

To further prove my point I upgraded the old 2011 MacBook Pro with a SATA SSD and 8gb of RAM. Comparing this to the function key late 2016 MacBook Pro the GeekBench CPU benchmarks were astoundingly not as earth shatteringly different as one might expect. Both machines feature and Intel i5 dual core processor with 8gb of RAM. Here are the results:

2011 13” MacBook Pro (i5 8gb)
– Single core: 2948 
-Multi core 5673

2016 13” MacBook Pro (i5 8gb)
– Single core: 3888 
– Multi core: 7721

So how to get Mojave on this old unsupported MacBook Pro… Well fortunately DosDude1 has created a patcher tool that essentially helps you to create a bootable MacOS USB stick along with a patching utility you run at the end of the install process that adds in kexts and other patches to extend functionality to older devices. The tool can be downloaded from his site here: http://dosdude1.com/mojave/#downloads

To complete the process you will need at least 10gb of space on a flash drive, to install the utility and follow the directions here: http://dosdude1.com/mojave/#instructions

I’ve been running Mojave on this machine without any issues for the past 2 months and all has been well. The only gotcha is to make sure you check his site before installing minor release updates as some newer MacOS patches require an updated version of the tool to be installed. If you forget to this and find your machine unbootable all hope is not lost, simply create a new bootable flash drive from another Mac and proceed to install the OS again. This will not overwrite your files or home directory unless you erase the partition using disk utility.