November 13, 2015
As a user on this homebrew issue utter's like a savant "package managers managing package managers rarely works out well."
October 21, 2015
This is an FYI and warning, be very careful with haswell processors with RHEL/CentOS 6.6. There is a futex wait() bug that can cause processes which wait to never resume agian. A good description is on InfoQ.
August 6, 2015
Something not often mentioned and tested is the impact of latency in the wild on the operation and scalability of a website. The vast majority of load tests conducted are ran from a local load source, jmeter in the same availability zone. In this case the latency is incredibly low, probably sub millisecond. In the real world your application will never see this kind of latency again, it will be anywhere from 50 to 500ms depending on the global mix of traffic you receive. This can kill the performance of your application in surprising ways.
July 15, 2015
I just setup my blog on google's webmasters tool and saw that they wanted a sitemap. This led to the question of "how do I make one of those!" Luckily I found David Singer's blog post on building a sitemap!. This is a direct paste of his code which I found to work exactly as required, be sure to check out his blog.
June 19, 2015
With puppet, sometimes it is necessary to make case statements around what version of a package is installed rather than having puppet dictate what version is installed. Say "add x line to sshd config for x version." To do this we need to have fact with what package version is already installed on the system, here's a quick script to do that in bulk. To deploy, add this as a plugin in a module.
April 28, 2015
For capacity planning and stability reasons it is important to be able to estimate how Cassandra will act in a multi-dc environment during adverse networking conditions. This includes capacity planning for WAN bandwidth and ensuring that after a network partition you can stream the hints off both fast enough that they don't expire and TTL off or stream so fast that it saturates your WAN and crashes the cluster. In this blog we'll go into how to compute the streaming speeds. Then in the next blog we will look at fine-tuning the number of threads needed to stream data safely after a network partition.
January 29, 2015
This is just a quick FYI post as I don't see this documented on the web elsewhere. As of now in all versions of Cassandra a gc_grace_seconds setting of 0 will disable hinted handoff. Basically to avoid an edge case that could cause data to reappear in a cluster (Detailed in Jira CASSANDRA-5314) hints are stored with a TTL of gc_grace_seconds for the keyspace in question. A gc_grace_seconds setting of 0 will cause hints to TTL instantly and they will never be streamed off when a node comes back up.
December 30, 2014
Sometimes its handy to be able to unpackage and rebuild an RPM to remove specific files. In this example I'll rebuild the RPM for Datastax Opscenter's agent without the init files. The tools used to do this are rpm2cpio and FPM. I manage services with daemontools so having stuff start under init is a real issue. I could just install from tarbal but I prefer to keep everything under rpm for simplicity. Plus there is a lot of useful stuff that the rpm does on install that I don't want to reproduce manually.
September 23, 2014
The documentation on getting rails server --debug working with the Software Collections (SCL) version of Ruby is a little weak. So here's how to do it. If you installed SCL ruby193 you'll probably get this error when you try to start the debugger:
September 22, 2014
I spent quite a while in the last few days trying to figure this error out and since there was no blogs or info on it online I felt compelled to write one. Basic setup is RHEL 6.5 with either Foreman 1.5 or Foreman 1.6. I'm using the Foreman rpm's and the Centos SCL repo for Ruby193.
July 17, 2014
TCP SYN Floods can wreak havoc on a network and at the node level they look quite weird. Since they are just SYN packets, from the normal monitoring point of view they looks like a decrease in traffic, as the kernel holds on to these non-existent connections waiting for the final ACK. So rather than looking at graphs and saying "wow we're getting hammered," it sounds like "wow, where'd our apache traffic go?" or "why does this server have less traffic than the rest". At the load balancer level though you'd still see all the connections, its just they don't make it into most OS level monitoring. Wikipedia's got some decent documentation on SYN Floods and you can increase resiliency with changing a few kernel settings.
May 23, 2014
I recently went through the exciting task of decommissioning a Cassandra data center. Although the documentation out on the web is pretty good for this I thought it was worth a post on a couple of the technicalities and "what to expect" questions I had prior to starting the process. The basic steps to remove a DC in Cassandra are outlined in the Cassandra 1.2 documentation. To recap the basic steps are to:
March 26, 2014
By default libvirt/kvm virtual machines created with Foreman will not be set to autostart. This is a slight irritation in that if the host ever crashes the virtual's won't start back up automatically. So here's the solution!
March 14, 2014
In the last blog I wrote I detailed how to send backups directly into S3. You might want to send backups to S3 rather than doing snapshots of your block devices in EC2 so that you can later download those backups and keep some form of your data in house. In this blog detail a basic script that does just that.
March 11, 2014
This blog post will go over the basics on getting automatic backups going from an AWS EC2 instance into an AWS S3 bucket. Storing your backups in S3 is a nice method because you get such good network performance keeping the data in AWS and then you can do a local backup to from the S3 data without effecting server performance or opening up any extra ports from their firewalls.
September 25, 2013
In an environment we often have files or configurations that we want to have on every node but then customize on specific nodes. For example a customized Ganglia gmond.conf file with differnt ports based on which cluster the machine belongs. The most basic method of doing this is with a giant case statement within the configuration file, but this gets unweildy at scale. A great solution for this is a resource collector. Resource collectors let you do a "find and replace" on an already defined resource. In the ganglia example it will let us define the gmond file once with default cluster settings, and then override it's attributes for any node with a more specific cluster.
August 29, 2013
I finally got some time to sit down and play the beta version of Crea by Siege Games and I must say its been great so far. The content seems to have been coming along nicely since the early versions I played and the mechanics are pretty great. It is really amazing what they've done considering the team size. I'm now just sitting around waiting for bosses. =D
July 15, 2013
Basic HAProxy monitoring for puppet masters is pretty easy, just have HAProxy check puppet's default port, 8140. This works great for normal usage but what if you want to upgrade the puppet masters or test large manifest changes slowly? Just stopping puppet works for upgrades, but makes it difficult to test the server after the upgrade is complete. Not to mention that there's a chance puppet could die but since HAProxy only tests opening an ssl connection, the node would stay in the VIP. We obviously need a better way to dynamically enable/disable nodes from HAProxy without having to manually edit HAProxy's config.
June 28, 2013
Although my profession is in the technology industry, I've always been interested in finance, Wall Street and investing. I've read a number of personal investment books and just finished reading A Random Walk Down Wall Street. I must say this is the best investment book I've read so far. Most books I've read have fit in either the "inside view into the world of finance" (i.e. The Big Short) or a more basic "follow this formula and you'll be rich" (i.e. You're Money Ratios). This book takes a stab at both categories by first picking apart the wider finance industry and its pitfalls and then looking at how you should invest to work around all the sharks out there.
June 25, 2013
I'm super excited about the 1.1 release of Foreman, as it has a chunk of my code in there! My old work needed a way to track historical trends in our infrastructure over time and gave me the task of coming up with a solution. The Trend pages in Foreman give a graph of the number of hosts with a puppet fact value over time, and the list of current hosts.
March 27, 2013
Saw this article on USA Today about early retirement. The article hits on a lot of really good points that I've felt for a few years now. I personally believe that all the retirement mumbo-jumbo out there is a little wrong, or at least non-optimal. My personal goal is not so much to retire early but to be finished saving for retirement early so I can maximize the value of compounding (and have a lot of fun after its off the list). For example finish saving by 35 then just topping up for fun as I get older from there.
March 11, 2013
I just found this tool recently and have found it extremely useful. It is called OneTab and it’s so simple that it's brilliant. I personally am an avid browser, but not a great reader, of the web. So I often open up tabs of interesting articles for later consumption, usually once I'm home and have a beer in hand. This often leads to 50+ tabs open in Chrome. Luckily my MacBook has gobs of ram so this isn't a pressing issue, but it is hard to find the tab I'm looking for. Not to mention that if my Chrome crashes I get this sinking feeling in my gut.
March 1, 2013
A few days ago I moved my hosting off of Amazon EC2 and Rackspace and onto my local Comcast connection. Sure, I've lost a lot of reliability but hey, can't beat free. I was running into space issues and needed more RAM and my local PC has and i7 with 24GB of RAM so it sounded like a better solution.
February 27, 2013
I just saw an amazing little video on the Crazyflie Nano Quadcopter. This thing is so cool! I am going to try to talk the wife into getting one. One of the neatest things is that they are open source and moddable so you can change it's mechanics and software to what you want.
January 25, 2013
Sometimes technology is totally frustrating. I spent a couple hours today fighting with my monitor and power at my desk. The external monitor, a Dell 2405, kept flashing "can not display this mode" as though my laptop was sending a bad signal. I searched around and found a lot of posts with others having the same issue.
January 18, 2013
I just finished adding Cloudinary as my host for images uploaded on a test site of mine. It ties in cleanly with CarrierWave and removes the need for RMagick to be installed on the VPS. This was a great solution as the RMagick gem is no longer being maintained on github and the Cloudinary free plan is quite reasonable for my usage.
January 15, 2013
Hey folks,
January 11, 2013
After going through making the tutorial for OpenStack CLI, I got a little irritated at how long the commands are; primarily the boot command with its dozen arguments. So I made a plugin for oh-my-zsh to autocomplete the madness. The plugin is located in my github repo oh-my-zsh-nova. To install just copy the nova directory into the oh-my-zsh plugins directory.
January 8, 2013
Starting and managing machines from the CLI is at times useful. It is nice managing your instances from your local machine without the need of a web browser. Here's the basics on how to start a host.
January 4, 2013
I've been working with OpenStack at work and there has been a little confusion with users on how to spin up instances. Some of the concepts are a little new, even if they are relativly simple after you see it once. And the OpenStack documentation is so huge that it takes months to process, which is not great for Joe User. This will guide you through getting a plain instance up on OpenStack and is geared to an average user who just wants this damn system started already.
December 28, 2012
Just finished my Jenkins setup for this Jenkins blog. Jenkins is a pretty sweet Continuous Integration server. Here's how it all went down:
December 22, 2012
There were quite a few articles around the web yesterday about a UN Report that the Syria conflict turns 'overtly sectarian'. I know this is a little different topic from what I normally would write about, but the report is just so dumb that I must comment.
December 18, 2012
I find different languages quite interesting in programming. It is mind boggling that there are so many and so many personal preferences to go along with them. Compounded with the fact that they are almost all constantly changing and evolving. I just finished an interesting glance at an article on a visual comparison of language complexity
December 11, 2012
Foreman has a script to update it's facts by going through the fact's in /var/lib/puppet/yaml/facts/*.yaml and posting them to the Foreman server. The only issue is that the script does this one fact at a time and waits for the server response before proceeding which was taking long enough that the hosts had all reported back in again before the script could loop through all the hosts once.
December 7, 2012
I recently had a battle to get Andchat and freenode IRC to play nicely. Freenode changed their policy sometime ago where people on mobile or coming from Tor need to connect with SASL (Simple Authentication and Security Layer). Its not too hard to setup but finding the resources on how to do it was a little troubling. Main issue is that AndChat doesn't just have a "turn on SASL" setting and no hints in its UI. So... here's the scoop:
December 6, 2012
I'm excited to get this blog thingie started. I'll be collecting a few of the tidbits that I've learned over time here. And migrating my little bit of content on Tumblr over here.