Hacker Newsnew | past | comments | ask | show | jobs | submit | senoff's commentslogin

thanks! mine hit $145. i'll have to check out your solution too.

Some context for why this hit us harder than most: we run a ~20-agent dev harness — coding agents that work in parallel, open PRs, and push commits more or less around the clock. Great for shipping. Rough on your CI meter. A team of humans pushes a few dozen times a day; 20 tireless agents push a lot more, and every push fires the full test matrix. Our GitHub Actions bill climbed accordingly — nobody made a bad decision, the machines just never stop typing.

So we did the obvious thing: moved CI to a single self-hosted box. Fixed monthly cost instead of a per-minute meter, and the builds actually got faster because the box keeps a warm Docker cache and doesn't cold-start every job.

It sounds like a five-minute job — spin up a VM, run config.sh, done. It mostly is. But three things bit us that nobody warns you about, and they're the reason we bothered to package this up:

1. OOM. A memory-constrained box running a couple of heavy jobs at once just gets its builds OOM-killed, silently, and you spend an afternoon blaming flaky tests. Fix: give the box swap — specifically one swapfile per runner — and set vm.swappiness=10 so it only leans on swap under real pressure. Swap isn't the hot path; it's the airbag.

2. "No space left on device" — but the disk looks half empty. The culprit was /tmp silt: dead job-workspace directories that never get cleaned up, slowly filling the disk until a big build tips over. Fix: give each runner its own /scratch (so they don't fight over one /tmp), and add a systemd-tmpfiles rule that ages /tmp + scratch every 6h. Set-and-forget.

3. The test database. Our suite needs Postgres. One shared instance every job writes into means parallel runs clobber each other's data and burn through connections; the other extreme, a fresh Postgres container per job, is heavier and slower than it needs to be. What worked: one small native Postgres on the box (128MB shared_buffers — deliberately tiny), shared by all runners, with each run isolated in its own schema — create it on the way in, point search_path at it via DATABASE_URL, drop it in an always-run cleanup step. Small-and-shared beats N heavy instances on a constrained box, and per-run schemas make concurrent runs safe without paying for a container spin-up on every job.

The operating rule we landed on for an 8 GB box: provision 3 runners, but cap concurrency at 2. The third is a warm spare and gives you headroom; running all three heavy jobs at once is how you meet the OOM killer again. Bigger box, bump both numbers — it's just policy.

We wrote it up as a few idempotent scripts + a README so it's reproducible instead of tribal knowledge:

https://github.com/senoff/self-hosted-ci-runner

- provision-box.sh — swap / per-runner scratch / reaper - install-runners.sh — download, register, systemd-install N runners - provision-postgres.sh — one small shared Postgres, with a per-run-schema example workflow - check-runners.sh — how many are actually online (a dropped runner silently slow-walks every PR)

Everything's parameterized through a config.env, and the README has the exact reference box config (a small Hetzner VM, Ubuntu 24.04, Docker) with copy-paste setup steps.

Fair warning / the tradeoffs: self-hosted runners are great for a private repo you control. Do not point them at a public repo that runs untrusted-PR workflows — that's remote code execution on your box by design. One box is also a single point of failure, and you own patching it now. For our case (private repo, trusted team + agents, predictable cost) it's been a clear win.


Does anyone write anything anymore themselves? This is insulting, even for an AI coding enthusiast like myself.

because you want to code and stand up ci runners yourself or because i just shared what i did for anyone to adopt including the code?

not sure what's insulting. I just shared why and how I did this, but hey.


I think it's about the comment you wrote, or rather that AI (appears to have) written entirely.

ah. it writes and i edit it. me and Stanley Duckenmiller. it's a way better writer than i am. so is my cousin who is a golf writer for 30 years. so you're correct. i am excellent at editing though :)

Your parent comment reads like AI slop, so you may not be as good at editing as you think. Worth thinking about the impression you’re conveying about yourself.

Yeah I completely gave up after a paragraph of overly wordy AI garbage.

There's no way just writing it out would have been worse


I’d suggest adding emdashes to your filter list.

Ramp Receipts was built by the team at RampVentures.com for the saas community. Our receipts service is free as a way to give back and solve an issue every startup saas company (including ours) has. So yes its free forever for folks signing up now.


When you want to do code email campaigns this is the service to use.


Yep. Robert's a crack dev and I found his insights on the issues with Angular spot on.


spam.


This is true but it also means you have to be prepared to finance yourself for quite some time. Not everyone can do that. Also, by the time these guys took money they did it to get experience, connections and sponsorship.


Excellent due diligence. If you just take the revenues and expenses and project them out a bit it's not hard to see $1B in value.


Could you tell me how you did 10 years of study for Oral Surgery and yet you're 25 yo?


It's not full 10 years, I started when I was 17. By 2012 I am on my 10th year. I was born in 22 july 1986. After few month I will be 26.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: