In July 2024, Marko Matotan presented at the WeAreDevelopers conference in Berlin. His talk focused on the Cloud vs. Bare Metal philosophy, detailing the transition from cloud solutions to deploying Kubernetes on bare metal in Kodius DevOps stages. The aim was not to advocate for a specific approach but to provide valuable insights that may prove useful in future decisions.
In case you missed the conference, here’s an overview of Mark’s talk!
Bare metal and cloud - comparison and basics
Let’s first start by defining bare metal and cloud. Simply put, running applications on bare metal involves using physical servers, configuring them, and running your applications on them. On the other hand, cloud computing can be viewed as renting servers from providers, sparing you the need to invest time in configuring them. I often think of it as buying a house versus renting one. You own the house, and there are no monthly charges (well, at least after you pay off the mortgage 😅). However, it’s not all ideal. You are responsible for maintaining it when things go wrong and for preventing accidents before they happen. Additionally, you need a deeper understanding of the technologies behind it. So why would anyone choose bare metal?
There’s always the option to consider AWS: AWS OpsWorks.
Be cautious, though; their sales tactics can be aggressive. They offer free services to startups initially but may lock you into their ecosystem. While scaling quickly is a benefit, if you reach that stage, investing in your DevOps team and bare metal might be a more sustainable option.
Here are some downsides to be aware of:
- Large initial discounts
- Significant costs after discounts expire
- Fees for transferring data out of their services
- Deprecation of older services, requiring migrations to newer ones.
Many companies, including X (formerly Twitter) and 37signals, have recently transitioned to bare metal. We’re proud to have recognized the advantages before it became a trend. While cost savings are a major benefit, it’s also important to consider other advantages of using bare metal.
Business benefits of bare metal
To understand our needs, it’s important to know that Kodius is primarily a software development agency with numerous projects requiring hosting. Each project typically includes multiple environments (UX/UI design, development, staging, production). Some clients prefer hosting everything on our servers due to cost savings, and once set up, maintaining these environments incurs minimal ongoing costs for us.
While I can’t provide specific data showing cost reductions due to the variable nature of our application hosting (different numbers of applications and changing requirements), here’s an illustrative example of the cost to maintain five projects on AWS (with several assumptions):
For EC2 instances alone (excluding tax and additional AWS features), the cost totals around $7,326. This amount nearly covers the cost of both our servers and datacenter expenses. Additionally, with our servers, we have 400 GB of RAM compared to the 40 GB provided by AWS, superior CPU performance, and more storage.
Furthermore, bare metal significantly increases our project limits, offering more flexibility and efficiency.
Workflow benefits
Ok, Bare Metal is a winner when comparing pricing and services that can be offered to our clients, but do developers get some benefits from it as well?
There are a lot of open-source applications available that work on the model of open-source code—you pay for hosting either on their infrastructure or yours. Over the years, we’ve used many of them. Some examples include:
- Zulip – Removes the need to pay for Slack/Teams.
- Yopass – Secure password sharing with clients.
Not to mention our development tools. We run our CI on the servers by using GitHub Actions runners (which, as anyone who uses them knows, can get pretty expensive). We also run a Docker registry to store our Docker images, and ArgoCD (formerly SwarmPit) to manage our CD. We are even considering switching to Gitea for our repositories. This setup allows for cost-effective scaling, and in the end, even if the software becomes obsolete, you are left with actual hardware that has inherent value. :)
1st DevOps stage: Clouds
So, just as the title says, we first started hosting applications on the cloud, specifically AWS.
This stage lasted until 2019. Key points are:
- A small number of projects
- Chef recipes managed through OpsWorks and RDS on AWS
- Some projects hosted on Microsoft Azure and DigitalOcean
2nd DevOps stage: Docker swarm stage
Then we acquired our first server and began hosting our applications on a Docker Swarm managed single-node cluster. The setup was straightforward but somewhat overcomplicated.
With only one server, high availability was not an option. The Nginx web server ran directly on the server rather than within Docker alongside the applications. Configuration files (stacks, essentially Docker Compose files) were stored on the server’s filesystem. For each application, we had to write a corresponding Nginx configuration file, which sometimes became overwhelming. Downtime occurred with every deployment, a problem mitigated by Kubernetes through readiness probes.
Our CI (which was more accurately auto-deployment, rather than CI) functioned as follows: A commit would trigger a build on Docker Hub, which would push the image to its registry. A webhook would then fire a request to our custom-built auto-deploy tool, Deployko, which was responsible for pulling the image.
3rd DevOps stage: Kubernetes stage
When we purchased our second server, we revisited the issues that had been bothering us, explored alternative solutions, and implemented the following changes:
- Adopted the GitOps model
- Completely revamped our CI/CD pipeline
- Implemented a self-hosted private Docker registry
- Reduced server-side configuration
Here’s how we set up our pipeline and a simple way you can try it too:
Future plans
We are still in the process of migrating everything to Kubernetes, and while having two servers is an improvement, we should really be aiming for three, which is what we are working towards next. It is a never-ending learning and improvement experience. We aim to make this simple enough to easily add new servers and deploy new projects so that everyone in Kodius can add new projects using GitHub templates.
Conclusion
If I’ve inspired some of you to start exploring bare metal as a cloud alternative, that’s fantastic! Go ahead and dive in! However, while it might seem like that was my main intention, I had a broader goal in mind.
When I first delved into DevOps and best practices, I had no idea what I was looking at (not that I’m an expert now). Exploring new solutions will teach you a lot and can be a lot of fun. You might get frustrated at times, but it’s all valuable experience.
If there’s one thing to take away from this talk, it’s that curiosity and tackling challenging problems pay off. Sure, it’s easier to ask ChatGPT to generate some Terraform code and apply it to AWS accounts, but there’s immense value in understanding the technologies behind it. Even if this isn’t your area of interest, adopting a curious mindset can lead to great discoveries, regardless of your work experience or age.
