Why I run Kubernetes without EKS

On one of my projects I run Kubernetes.

Most people who do that use EKS, which is Amazon’s managed version of it. That is the normal choice, and for most people it’s the right one. I went a different way.

This post is why. The reason turns out to have very little to do with Kubernetes itself - it’s mostly about a bill.

What those two words mean

Kubernetes is software for running other software across a group of machines. You tell it “keep four copies of this program running” and it decides which machines to put them on, restarts them when they crash, and moves them when a machine dies.

The machines it runs things on are called nodes. The part making the decisions is called the control plane - just a program, usually sitting on its own machine.

EKS is the managed version. You pay Amazon and they run the control plane for you. You still supply the nodes, which are ordinary AWS servers.

That’s the normal path. Here’s how I ended up off it.

Why I bought machines

The project does a lot of heavy background work. Not in bursts - a steady grind, all day, every day.

Renting servers to do that is expensive. It was the biggest single line on my cloud bill.

And renting has a particular sting to it. You pay the same amount next month. And the month after. Forever. You never stop paying for the same capacity.

A machine you buy works differently. You pay once. After a few months it has cost you less than renting would have, and from then on it’s nearly free apart from electricity.

So I bought some machines. Self-hosted boxes, not in the cloud, doing the heavy work.

There’s nothing clever about that part. It’s arithmetic. The interesting bit is what happened next.

The problem with having two of everything

The obvious way to run this is as two separate setups. Cloud things over here, my own machines over there, and some scripts holding hands between them.

I really didn’t want that.

Two setups means two ways to deploy. Two places where passwords live. Two sets of habits to remember. Every change costs double.

But the bigger problem isn’t technical, it’s human. When putting something in the cloud is easy and putting it on your own machine is fiddly, people choose the cloud. Every single time. Not out of laziness - it’s just the path with less friction in front of it. Six months later the machines you bought are sitting half idle, and you’re still paying the cloud bill you bought them to avoid.

So I wanted one cluster. One place to deploy to. Putting work on a machine I own should be exactly as easy as putting it in the cloud, or it simply won’t happen.

Where EKS stops helping

This is the wall I hit.

Managed Kubernetes really wants your machines to be its machines.

The control plane itself doesn’t care. As far as it’s concerned, a node is a node, wherever it happens to sit. But the product built around it assumes your nodes are cloud servers that it created for you. The tools assume it. The defaults assume it. The documentation assumes it.

You can push against all that. I didn’t want to spend my time pushing.

So the question changed. It stopped being “which Kubernetes should I use” and became “what will let a machine join from anywhere?”

The answer is k3s. It’s a small, complete version of Kubernetes that installs as a single file. You give a machine an address and a token, and it joins the cluster. That’s the whole process.

My control plane is one small cloud server running k3s. It costs about sixty dollars a month.

What it looks like in practice

Every machine - the cloud ones and my own - connects to the same private network. It’s an encrypted tunnel, so they can talk to each other safely across the public internet. Kubernetes runs on top of that and never knows the difference.

Each machine gets a label saying where it lives. Each piece of work says where it wants to run:

nodeSelector:
  site: self-hosted

That’s the entire scheduling system. A label, and a matching rule so nothing lands somewhere by accident.

Why this beats picking one or the other

The two halves are good at completely different things. It works because I stopped pretending they were the same.

The cloud half is the reliable half. The database, the cache, file storage, the CDN, the firewall, the password store, the audit logs. All managed by AWS. None of it mine to keep alive.

That’s worth real money. When the database has a problem at three in the morning, somebody whose actual job that is gets woken up, and it isn’t me. Their database is also better than the one I would run. I have no interest in becoming good at that.

The self-hosted half is cheap, fast, and mine. For the price of renting, I get considerably more machine. It’s already paid for. And there’s something I didn’t expect to enjoy as much as I do - it’s a real computer that belongs to me, doing real work, and I know exactly what’s on it.

The mistake would be treating those two halves as the same kind of thing. They aren’t. One of them I trust to be up. The other one is cheap and powerful and might, occasionally, not be there.

The rule that makes it safe

Everything follows one rule. Nothing important lives only on my own machines.

The control plane runs in the cloud. The website runs in the cloud. The scheduler runs in the cloud. There is always at least one background worker in the cloud.

My own machines do most of the actual work, because that’s where the cheap capacity is. But every job that runs on them also has a cloud version, already written and ready, sitting at zero copies. Configured, reviewed, just not switched on. And the part that routes requests looks for the application by name - it doesn’t care which machine answers.

So what happens when one of my machines drops off the internet? Power cut, dead switch, somebody trips over a cable.

The website stays up. People keep using it. Requests keep getting answered.

The only thing that changes is that background work moves through the queue more slowly. What I lost was speed, not the ability to run at all.

And if I need that speed back immediately, I switch the cloud copies on. One command. It costs more while it’s running, which is exactly the trade you want to have available in an emergency.

That’s the part I’m actually pleased about. Not that nothing breaks. That the cheap half is allowed to break.

Would I tell you to do this

It depends on your situation, and for plenty of people the answer is no.

If every machine you will ever run is a cloud server, use the managed option. You’re paying somebody to keep a control plane patched and alive, and that’s a fair deal. The money isn’t the interesting part of it.

But if renting compute is your biggest bill, and buying a machine would obviously fix that, then the only thing standing in your way is a control plane willing to accept the machine. That turns out to be a much smaller problem than it looks. Mine is one small server. I’ve rebuilt it from scratch and reconnected everything in an afternoon.

You get to keep cloud reliability for the things where reliability is what you’re actually paying for. And you get your own hardware for the things where you’re just buying raw compute.

I don’t think you have to pick one.