Sometimes that thing you built doesn't work

As we continue to move infrastructure toward more scalable solutions, our current project is a big one. And extremely important to our business and customer.

We’re currently running one of our key products on a pair of EC2’s within AWS OpsWorks—I know. Transitioning this Rails app to ECS is a major undertaking. From the DevOps side, we were able to framework the infrastructure within Terraform while our Developers worked in tandem to build our Docker elements.

Things were clicking well, and local testing was spot-on. So, it’s time to ship up to AWS, figure out how to slow roll and test it with real data. We needed to find a way to easily flip and flap between the current EC2 infrastructure and the new ECS tasks. Our first attempt was using a weighted ALB with aws_lb_listener_rule.

No need to deep dive this, it’s well-documented in the Terraform docs. We scoped this out, built the infra, and felt really good about it.

…until some things upstream we hadn’t considered made it clear that this would not work for us.

And it wasn’t just that we couldn’t use the listener rule to facilitate A-B testing, we found a fountain of things we hadn’t considered that required us to literally tear down our entire ECS stack, and rethink it.

This was weeks of development that momentarily felt like it was flushed down the drain, but the reality is we learned a lot. And our ability to adapt to this, update our project with a correct solution and get to production was accelerated from everything we learned.

Basically, this mostly pointless post is a simple reminder that any amount of effort toward a goal, even if completely scrapped, is progress. I feel like there’s this whole CEO culture built around a quote about failure, and I don’t even remember what the quote exactly is.

But, failure is fine. Failure is a phenomenal teacher.

I think development requires one to detach from their Ego. I could almost go into a Zen/Buddhism/Tao analog, but perhaps I’ll save that for another time.