Simply put, your hardware stack is created and maintained using the programming concept of immutability: once something is instantiated, it is immutable and does not change. If an update is needed (either from a scheduled upgrade or bug), a new instance is created to replace the existing one. Thus, once a component is launched it is never changed, only replaced.
Speed – The Koddi dev team strives to continually move fast and push new features for our clients. We have multiple autoscaled web server stacks (development, internal, production, white label, etc.) that all get updated multiple times per week. Using traditional infrastructure methods to maintain all of these servers (package/security updates, code changes, distro/kernel upgrades, config files) and ensuring each server is in line with the others would cause a drastic slowdown in productivity. Using II, every server is provisioned using the same base, so everything always matches.
Reliability – Before a new feature or platform update is launched at Koddi, it must undergo multiple rounds of testing/QA. This includes both automated unit tests, as well as human QA in our internal environment. However, (rarely) bugs can make it into production. II allows us to quickly roll back our entire stack to a previous version that we know is free from error and debug the issue in a dev environment. With traditional architecture, changes would have to be undone on production and then hope the issue is fixed. Being able to quickly switch back to a previous version ensures that our clients experience as little impact as possible.
As we’ve discussed before on our blog, Koddi uses Amazon Web Services as our cloud provider. Our process for server deployment consists of two parts: packing our release and deploying new servers.
Once a new feature or update is ready to be released to our production environment, a tag is created from our GitHub master branch. We then use Packer from HashiCorp to create an Amazon Machine Image (AMI) that will be deployed. Packer does a few awesome things behind the scenes to simplify this process.
One key benefit of a packed AMI is the significant reduction in the time it takes an instance to be ready when launched. Since the image has already made all the updates/code changes during packing, this isn’t required when the instance spins up. This is especially beneficial when scaling up an Auto Scaling Group (ASG). After the AMI is created, we can deploy the new image at any time.
For deploying our environment, we implement a blue-green deployment using Ansible. Ansible is extremely customizable, and the Koddi use case is just one of the many possibilities. Below is our process:
After all of our instances are deployed, we do some background cleanup like removing any LCs/ASGs/AMIs older than our last five deployments. Similar to Packer, Ansible allows us to automate the entire process from start to finish.
Now that we have our base II system in place, our plan is to continually make our platform faster and more fault tolerant. Here are some ideas for what we might do next:
If you have questions about our process or have comments on how we could do it better, please reach out in the comments below. We’d love to hear from you!