Agile technologies
This is the first in a series of musings about how to best use technology to achieve discrete outcomes — often business applications or services.
he said to him, “There is still one thing lacking. Sell all that you own and distribute the money to the poor…” …when he heard this, he became sad; for he was very rich.
Achieving agility with technology reminds me of this story. Often what is needed is to give up the tools we want to use and to rely on something more basic, more fundamental. And we are so, so rich in tools and frameworks these days — the choice seems unending.
I worked with a team recently who were using Serverless (the framework, not just the architecture). Their rationale was good, and the framework itself is an awesome piece of kit. The team I was working with wanted to use it to deploy an API and an application into a VPC integrating with external JSON and SOAP-based APIs. Deployment used a typical CI/CD pipeline.
The Serverless framework leverages serverless technologies, like AWS Lambda or Azure Functions, to automagically deploy applications consisting of pay-per-execution functions into production. It builds on infrastructure-as-code concepts to ensure that you can repeatedly deploy your stack into production without manual configuration. I like it. It’s very handy. Single command deploy — nice.
The downside to this flexibility is… well you have to run a single command to deploy it.
The teams’ solution, designed by a very senior technical resource in the organisation, was to spin up an EC2 instance (a sort of bastion), download the code, run the single command, deploy the serverless app, then teardown the EC2 instance again. Bizarre but true — create an ephemeral virtual server to deploy a serverless application. This works well, is performant and cost effective.
If I recall correctly, the creation of this EC2 instance was orchestrated using a terraform script, which, again, takes the heavy lifting out of creating the necessary resources. That in turn was kicked off using a bash script running in a Circle CI container.
I’m glossing over some essential details here, like ensuring the correct IAM credentials exist and that provisioned resources are versioned in a way that can be tracked — built into the Serverless framework paradigm.
The thing is, just a few months earlier, I had worked with a different team on a similarly spec’d app deployed in a similar scenario which was missing almost all of these moving pieces. There was still a Circle CI container to kick off builds, but deployment was basically just:
- Create a ZIP
- Publish ZIP
- Archive ZIP
This seemed to work just fine — no terraform, no Serverless framework, no bastion EC2 instance. It left me wondering if I was missing a trick. Was there some benefit to the extra hops that I was missing? Was it just over-engineering?
When choosing the technology, this simpler solution was designed to avoid as many tools as possible in order to allow more junior engineers to wrap their heads around it.
I also have a friend who publishes Twitch analytics and is very successful at what he does. He uses very basic technologies, very effectively, to produce a product which is beneficial to a large community of users at almost no cost, and with a very small investment of time. I am in awe of his ability to be useful to the world. I’ve seen countless people (yup, me too) fail to produce working software despite having far more financial, technical and human resources at their disposal.
Why is it so hard to be effective sometimes? When we have so many tools at our disposal, why is it sometimes so hard to do good work?
What does it take to have truly agile technologies? What are the different types of agility we might be interested in achieving?
Why do we employ “Agile” processes to achieve agility but neglect to ensure we are building agile solutions?
This is part of a series of articles on agility and software which are linked below:
