They don't even let you really run those yourself now either, We moved off a big GHE footprint because the support for it was getting abysmal and features were slowly become GH.com. Ive been dreading the move to GitHub.com and it was worse than I expected.
I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.
GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner.
More likely is that the log streamer takes up some amount of resources and thus running it requires a baseline level of compute. But I doubt they spin up a small runner per-self-hosted runner, most likely some ruby workers (on distributed azure instances) are dedicated purely to streaming self-hosted runner logs to log consumers.
I'm certainly not surprised that they tried to sneak in billing self-hosted runner minutes at some ratio to potentially recoup the costs here.
Oh that is pretty gross, thanks for sharing. I was not aware of that process, that explains the cost they attempted to roll out for runners too a few months back.
I think they kind of have to operate that way if you want a control plane/access to GitHub’s own layered services like caching and artifact storage. There are plenty of things to blame GitHub for, but this one doesn’t seem sinister at all.
Edit: to be clear, I mean the part where they maintain state on their end. I have no idea if they do that with another runner instance, that seems unlikely.
I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. '
If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.
That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)
The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).
I don't disagree that it's obvious they've got problems but I'm just saying it's obvious to me the part that falls over (the scheduling of jobs) and why that would impact self hosted runners, which do no scheduling but depend on it to function.
As for 'just a message queue with some database updates and sharding that's easy to reason about'... Here's a job scheduling problem as an example: imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?
A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
> The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).
Sure, but the part that actually schedules where a 'job' gets run is based on a relatively simplistic tag system. Reading the yaml and plopping some job metadata into a queue-like system isn't where I would expect their issues to be, but at their scale I'm sure everything becomes fragile and inscrutable.
> imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?
You don't need to. GitHub Actions runners, and most CI runners that I've interacted with appear to have a pull-based model where they ask for work that matches their declared tags/shape (usually platform/runtime/OS/etc.). This probably amounts to a database query, but who knows.
> A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
I highly doubt it's a simple message queue. My issue is git repos and their CI infrastructure have very low coupling to other repos or entities in most circumstances, at least conceptually, so parts of the system (ie. regions, shards, etc.) should be able to function even when others are down (ie. it shouldn't break for everyone). There's clearly centralization and coupling that isn't obvious from an outside perspective, which sorta tells me it's incidental, but that's a guess.
The only statement I'm making about their system is that it appears to be poorly designed, as evident by their poor uptime. The rest is just visible details about what data is involved in performing the task at hand, agnostic to the implementation underlying it.
The entire complexity and value of GHA is rent-seeking to keep the lights on for other things.
They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.
As someone who’s building a github actions control plane replacement, the scheduler is extremely simple. It basically the kost naive version of a FiFO queue.
They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see.
Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can.
They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again.
I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.
Microsoft in the 90-s and early 2000-s was a serious company. Windows NT and Office were engineering marvels.
Windows 95 was a brilliant hack that allowed 32-bit GUI programs to run on hardware with 2Mb of RAM (Windows had 4Mb of RAM as the official minimum, but you could run it on 2Mb) while preserving compatibility with the majority of DOS software.
They definitely managed to create a lot of reliable products.
Windows NT was one of them. Up to Windows XP the products were pretty solid and each had visible improvement against the previous one.
Their language products were/are still solid IMO. Maybe Visual Studio is sluggish, but we can still use an older version if we want. Plus they put a lot of effort optimizing VSCode, too.
Even back in the MS-DOS/16-bit Windows days, when things broke down quite easily, I think they still provide the best bang for individual users and developers. There was no competitors who could provide so much value back then.
Thanks for sharing. It was that bad back then? As a consumer user the stability brought by the NT kernel was great back then, but maybe enterprise wise it was a different story? I never worked as a sys admin so can't say.
For the last point what was the golden standard back in the mid-late 90s, if we don't include mainframe/minicomputers? Was it Solaris or BSD?
Now that I think about it, I feel sad that I do not have the technical prowess to compare operating systems :/
Beside the NT4 servers we had SunOS 5.x and Digital UNIX (later upgraded to Tru64 UNIX). Yes, from sysadmin point of view they were way more consistent performance wise and less problematic to operate in general. BSDs were not 'approved' but nonetheless we were running DNS and some other internet facing stuff on OpenBSD 2.x if I remember correctly.
reliable? Windows is the only operating system you have to constantly reinstall because it gets slower after a few months. To the point people think it's totally normal to do that!
At least they are reliable in/after the Windows 2000/XP/7 era? Even as I don't like Windows 10, I haven't reinstalled it for years.
Back then the other options for home computers are just too technical for normies. I tried to install Linux back in early 2000 and I couldn't even get through the install process. Skill issue for sure, but if I couldn't do it back then I'm sure none of my class mates could.
Wow this is really poor justification for its name, the folks at cloudflare really should not be overloading the term "Operating System" pointlessly like this.
They even put it in quotes, because they know what they did.
Maybe just take out the word "traditional" and be honest: "This is not an operating system. We're redefining the term because it is convenient for marketing."
> This obviously isn't entirely true: [...] it's the way I access my LLM.
It doesn't change my opinion that there's a very real possibility that we move to thin clients or microkernels intended primarily to run chat interfaces (or some other UI that organizes LLM sessions and their outputs) either locally on the hardware, or against cloud computing environments. An operating system is just the interface a user has with their computer.
More to the point - it is possible to run an operating system on top of another operating system: this is what VMs are. I'm obviously not claiming that LLMs replace the glue between hardware and software(?!)
Having an embedded scripting engine for large programs is nice for simplicity and keeping configuration modular - it would be messy to need to modify source constantly. It’s not an issue of it being difficult to compile something so much as it being pointlessly difficult to maintain.
I think you're missing the forest for the trees with this statement and it makes your comment seem a bit reductive. It's not so much a dismissal of writing on a particular product (iPad) as much as the article is about optimal writing by hand, the best materials, and some of the reasoning as to why.
While you may be convinced writing on a screen or tablet is objectively superior to doing things the "old way" there is literature on the topic indicating otherwise you might want to consider - that being said, everyone is different in their preferences and anatomy. I for one can't stand to write on a plastic/glass surface with a stylus, the ergonomics feel wrong and my retention for journaling is significantly worse.
Still, you are barely more protected than someone running this as non root at home. By your same logic you could “confine” the agent to any arbitrary directory on your filesystem and say it’s sufficient.
Yes, that is exactly the point I am making. The agent / cli is confined within the WSL2 environment. How is it "barley" protected? Neither you nor the other guy have yet answered this question.
I never claimed WSL2 is a security sandbox. I am saying running it in a container or WSL2 allows you to severly limit the blast radius. I am not expecting the agent to be malicious, but I am expecting it to do unexpected things.
That’s a fair approach, thanks for elaborating. I think the main point of contention is there is an expectation this software should be more reliable, that doesn’t take away from your approach, running close sourced software like a harness (or operating system) is asking for problems.
