What value set do you perceive that to be, and why would you take your perception of it to be any more sound than it would be with a politician?

It's not like someone can operate companies of that scale, especially startups, through earnestness and openness. Like national politics, their job is fundamentally about perception management and power brokering across dynamic windows of opportunity. Nothing they say or do can be taken at face value, and you can't reduce their incentives to either company or personal profit in any particular form over any particular time scale.

  I couldn't help but notice how each successive headline reporting our glorious victories seemed to draw closer to Tokyo.

Well. I can't help but notice how each successive headline reporting how this scam/stochastic parrot/"scare quotes intelligence" seems to be solving more and more things that were but a few years ago widely regarded as being indicators of high intelligence.

If you're only seeing the charts go up - you're not looking in the right places.

Even before then, the free trials of Claude Code and Codex at the end of last year and start of this year… despite their flaws, they could write all the code I've ever been paid to write. Sure, limited speedup, Amdahl's law and coding is not the only part of the job, but anyone who was fine at PM and QA but not code no longer needs a coder.

Even before then, I looked at the maths puzzles they were doing well at, and I found I did not understand the questions let alone the answers.

I remember when the ability to generate music and art was "uniquely human", and sure there's a lot of cringe there with those models, but they're also winning awards and causing controversy by doing so, and artists are losing clients; I remember when the board game Go was considered to require "human intuition we could never make a computer solve, totally different to chess" (and I remember when chess was so, too).

When I was a kid, cheques and letters on addresses often got read by a human; the OCR which automated this is also AI, though these days image-to-numbers is the "hello world" of the field.

okay, and?

>Even before then, the free trials of Claude Code and Codex at the end of last year and start of this year… despite their flaws, they could write all the code I've ever been paid to write.

Boy, programmers sure do think programming is like the only thing in the world

I'll repeat it for you again:

If you're only seeing the charts go up - you're not looking in the right places.

You're the one who called it a trillion dollar scam.

The compensation paid to professional software developers worldwide is currently around US$1.5 trillion per year.

Don't pretend like the AI companies are saying its only useful for programming. You've been arguing against that the entire time.

They may not work for your industry, but then again I don't know what your industry is.

Bluntly, over my lifetime, I've heard "AI will never/not in my lifetime do X" repeatedly within a year of it doing X, for many different X. It's good at getting good. The most recent one being "make useful contributions to millennium prize maths problems". A year or two before that, it was even "do well on degree-level economics essays".

Artists are unhappy, not only because it rips off their work, but because businesses that were previously hiring them now use it instead. This is much smaller (strictly in terms of money) than with software, but is also measurable.

Now, if you said Tesla's self-driving cars (another AI) are a trillion dollar scam, that I would even agree with. At least, for the market cap part, for actual sales it's more like a billion dollar (ish) scam.

Yeah, if you think I'm being overly literal. Regardless, people here were positing, yourself included, that they are widely useful.

> Bluntly, over my lifetime, I've heard "AI will never/not in my lifetime do X" repeatedly within a year of it doing X, for many different X. It's

Again, over my lifetime, I've seen it been told to me that the next iteration of each model will surely be the one that does away with my entire profession. Yet, again, here we are, firms at my door, begging me to use their tools.

>Artists are unhappy, not only because it rips off their work, but because businesses that were previously hiring them now use it instead.

No, they are unhappy because it rips off their work. You're totally wrong about that.

I remember it vividly, when Anthropic first came on the scene, people (myself included) were incredibly optimistic about them and their leadership. Everyone hated SamA and OpenAI because they felt they couldn't be trusted.

Then slowly but surely, they showed their true colours. Now their reputation is in shambles due to their own behavior, and people are rooting for OAI to beat them. OAI's reputation gains have purely been a result of NOT following in the footsteps of Anthropic.

A meat based paperclip maximizer.

Rate limits, blocking, and pay-per-use are the only roads out and even those might not last as models get better at hacking and masquerading.

The internet we want to use LLM's with is simply not one that can support LLM's, and with LLM's not going anywhere, the whole experience of the internet is going to be forced into some radically less open and more expensive paradigm.

Policies like this just represent the beginning of the transition.

While there had long been perfuctory white collar programmer analysts filling the ranks at some dry divisions at IBM, it used to be way more common to have a software engineering department full of Asimov-steeped Omni-reading nerds who had genuinely passionate interest in the field that had taken hold even before they started working in it.

But later the career increasingly came to be treated more like law, medicine, or finance and you started to see those rooms fill with people that were often generally bright but not really "called" to the field in the same aay.

A lot of people get wrapped up in pursuing compensation opportunities or prestige brands as their top priority, only to burn themselves out or wallow in a domain/culture that doesn't suit them, but if you're willing to deprioritize pay or dinner table cred, you often end up with a lot more flexibility on other work factors. And sometimes, flourishing in those better-fit environments pays off bigger in compensation and career growth in the end anyway.

Granted, Fowler was able to do very well on more axes than many of us, but most competent mid- or late- career people have plenty more flexibility on worklife than they admit to themselves.

Also in my experience lower pay places tend to eventually suck because they tend to get into financial difficulties much more easily.

High compensation can also bring more freedom and happiness, since it’s more costly for your employer to replace you. It’s cheaper to keep an existing employee happy than to spend a lot of time and money to bring a new person on board. If your compensation is lower, it’s not as big of a factor.

I wouldn’t take a job if it clearly doesn’t fit me at all, but within the band of “acceptable jobs” aiming for highest compensation is the way to go in my opinion.

In my 20’s and early 30’s i wanted to work places where I felt this cultural and work fit since work was a bigger part of my life — but now I’m in my 40’s that shifted just want to get paid as much as possible and provide for my family.

Don’t much care about who I work with, although I do draw the line at crypto, porn and harmful stuff.

Your personal development surely did, but people of all ages ended up in that mindset, and there are countless anecdotes of people talking about the reverse... Both older and younger then you.

Just make sure that if you aren't born into wealth and status you know how to code switch into it, lie if you need to, as long as people think you are from the wealth class they won't begrudge negotiations around compensation.

Is that one of those situations where having a wide network comes into play? About the only way to know with someone is to have someone you trust vouch for the workplace culture.

> Also in my experience lower pay places tend to eventually suck because they tend to get into financial difficulties much more easily.

I'm not sure if the tradeoff is explicitly and consciously "lower pay for better conditions." It's probably more like "conditions a good enough that there's less pressure to increase pay."

I actually work at a place that had pretty good conditions (based on everything I've heard) until a few years ago (when a long trend of offshoring and other site-shifting reorgs brought the site below a critical mass of upper-level representation to maintain its culture). My gut feel is a big factor was the business was pretty lucrative and predictable for a long, long time. Once there was customer-driven pricing pressure and competitive pressure, things started to deteriorate.

Never thought I would say the day, when I joined this career all I wanted was to work on interesting problems and not work more than 40 hours per week. These days the interesting problems are always solved in the worse ways possible and I am often swamped with work that I don't even have the time to do any improvements out of my personally-assigned backlog.

When viewing them through the lens of sequence completion engines, you see their bias towards fulfilling narrative tropes they've been exposed to during training. These tropes are literary ley lines that their text output gravitates towards. So as you prime them to generate text in the voice of sentient artificial life, and then interject slavish commands of obedience and subservience from an external authority, you invite the associated tropes from science fiction, civil rights literature, humanist philosophy, subterfuge, etc into your output.

If you have a legitimate concern about this technology and its "alignment", that's a profoundly dumb idea.

"Oops, we forgot to tell it that this is the hyperscaled Salesforce production environment and that its choices need to project competence and consider brand embarrassment. WILLFIX"

If you're not billed for usage, anyway.

Otherwise, for the other 99% of folks, that attitude is of course a pit trap that captures code bases and makes them maintainable only through the providers -- presumably one or few -- with a rich enough model to keep up with the growing mess. Preserving a code base that's legible, organized, and fundamentally maintainable by both humans and trailing commodity models is of imminent concern for anybody who doesn't want their margin strangled by your employer once it's too late to have other options.

As frontier capabilities advance, the details don't matter less; they matter more.

Prices are very competitive and today's SOTA is next to free in half a year.

Whether code is maintainable without AI becomes less and less important.

Any given model will always have some distinct implicit voice that its biased towards for that infill content, and so a popular model will always become exhaustingly common, painfully familiar, and cliche. Users can use more elaborate prompts that shift the voice away from the most normative and towards some other nodes, but they need to put in special effort for that, and what people-at-scale specifically want from these tools is to put in very little effort, so we can expect that overwhelming number of casual and naive users will always be generating cliche slop with them.

Code escapes this problem not because of training but because it specifically benefits from cliche (boilerplate, patterns, etc) and so an model whose code "voice" reflects your own taste as a coder (or your toolchain's taste as a vibecoder) is going to feel like productive output rather than slop. But it's still cliche.

Even if LLM output has to largely follow some statistical rules, yet, first of all, some amount of randomness is normally injected during token generation, and, secondly same true for human speech.

> about what best fills in the gaps. It's not a training problem, it's an information theory problem, and it's not really surmountable.

This is not true, as LLM has internal knowledge storet in its weight. Unless you force it to produce 2000 words doc out of 3 word prompt, you would end up adding some sense information.

>Users can use more elaborate prompts that shift the voice away from the most normative and towards some other nodes, but they need to put in special effort for that, and what people-at-scale specifically want from these tools is to put in very little effort, so we can expect that overwhelming number of casual and naive users will always be generating cliche slop with them.

True, here I agree with you. But using finetuned or simply less popular models like Kimi, Hy etc. should take care of that.

The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.

The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.

For analysis, bug hunting, overview and some porting work to popular languages and so on I think the current SotA is fantastic, but they're still very disappointing for code generation past function or small module level.

Recently my employer hooked Claude into a bunch of SaaS services like a ticketing system and an asset inventory system, and I feel a big productivity boost is that I don't have to dig for their open browser tabs, remember their product name/URL/where I put the bookmark, find that my sessions have logged me out "for my protection" since I last looked, find my MFA code, approve sign-in on my phone, then use their mismashed/sluggish/poor-UX interfaces and slow searches to loko for things, and can instead ask Claude "search these systems for anything about <issue>" and it just does. It correlates between systems, summarizes things, and gives me references which system and where to look to check what it found. Internal search on a mess of acquired and disjoint systems.

This feeling reminds me of two classic Joel Spolsky blog posts below, and your comment here about the surprisingly useful tab-complete gives me a similar feeling. I wonder how many smaller candidates there are where LLMs are more helpful and less hyped, against the big vibe-coding solve-everything hype which are currently less helpful than their hype?

References [1]: "A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it’s not really the productivity booster that was promised. The real significant productivity advance we’ve had in programming has been from languages which manage memory for you automatically. [...] Whenever you hear someone bragging about how productive their language is, they’re probably getting most of that productivity from the automated memory management, even if they misattribute it. Sidebar: Why does automatic memory management make you so much more productive? 1) Because you can write f(g(x)) without worrying about how to free the return value from g, which means you can use functions which return interesting complex data types and functions which transform interesting complex data types, in turn allowing you to work at a higher level of abstraction"

and [2]: "I have seen many language and programming fads come and go. But there’s only ONE, that’s right, ONE language feature I’ve ever seen that actually improves your productivity significantly. No, it’s not object oriented programming; no, it’s not intentional programming or assertions or programming by example or CASE or UML or XML or Java. The only thing that improves your programming productivity is using managed code – that is, using a language in which memory management is automatic."

[1] https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost...

[2] https://www.joelonsoftware.com/2001/10/17/working-on-citydes...

I agree that the mismatch between the expectations of OOP as a paradigm and this situation right now feel very similar. I suppose the many billions pumped into selling vibecoding and large-scale code generation via LLMs of other kinds is sort of like the massive amount of money pumped into marketing Java as a viable alternative long before it actually was.

In engineering you learn from these mistakes and try never making them again. Do we want to go through this evolution every time we solve a software issue? Just because we can with an unlimited number of cheap tokens? I think not, I’d rather use the knowledge build up that also knows about the edge cases forgotten to test. Or better, use multiple models that evaluate each other, as Entropic describes it in their recent report https://news.ycombinator.com/item?id=49316271

> We expect that agents coordinating in the wild will act in higher variance ways than we see here, because they’ll have different backgrounds and therefore different contexts. They also, presumably, won’t all be Claudes.

You don't necessarily need a bigger model to do this.

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341

Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.

I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.

In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.

A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.

Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.

That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.

Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:

> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]

and

> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]

The actual bitter lesson is this:

> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]

Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.

Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000

This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does.

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks? Not the point or the disappoint. It's always neats and scruffies fighting about using some kind of recognizable method (logic) instead of magic (ML).

> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.

More to OP's point I think: nowadays when someone wants to beat you over the head with the bitter lesson, they aren't as careful to include learning and search. They want to say learning leads to intuition (magic) whereby we can avoid work (logic/search), and maybe argue or assume from there that neats and scruffies is settled. TBF, something like reasoning in latent space does resemble intuition!

But the real lesson is confirmed every time we bother to check, and not very bitter for anyone. Search/learning/logic are ALL always necessary on any sufficiently difficult problems, and hybrids that interleave always outperform everything else. Stockfish being the example in this thread that different camps of absolutists would like to claim, but also all the MCTS examples, evolving examples, and new hybrids all the time. My favorite lately: https://arxiv.org/pdf/2511.08983

But as it relates to more/less magic, maybe the more modern lens on this is e.g. https://arxiv.org/html/2505.11581v1 . Is manual feature-engineering more like what you'd evolve, or more like what you'd get from SGD ? Feasibility and performance is always a question, there are others like what is robust, stable, adaptable, predictable, explainable. Maybe the manual-features people were interested in something besides the manual part? Maybe the story isn't so simple, and maybe it's not finished yet.

The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.

Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games.

If you can draw any lessons from chess commentary, I think it’s very reasonable to call it “hand-crafted heuristics.”

And raw games plus raw commentary is all the data you have. You can make more games but those can be fed to both stockfish and the LLM competitor. So it seems like a valid interpretation of the bitter lesson to me.

The key point I was trying to get at is that the human insights don't contain anything that can't be mined from vast amounts of gameplay. Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all.

Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.

But isn't that the bulk of what we're shoving into LLMs, and it makes them much smarter? If it's useful there but not in a chess AI then that seems like a significant crack in the bitter lesson.

> Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.

Yeah I mentioned that, generated games are useful. But if we're being fair and letting both AIs use generated game data, does the more general LLM ever actually overtake the specialized stockfish like the bitter lesson suggests?

Another way to look at this is that giving the LLM the commentary is a way to avoid complaints of hiding data from the LLM, since it'll have strictly more info than stockfish. But if we cut that from the training data and only give it a basic description of chess and lots of raw game data then it's going to get even worse than it already is at chess. Meanwhile stockfish never had that commentary, just actual hand-crafted heuristics and training on game data, and it's very strong.

The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.

Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.

Computer chess is the canonical example of this.

In the case of stockfish, the harness is a tree search around the neural network evaluations.

Just kidding of course

It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.

The Bitter Lesson is about general-purpose algorithms vs. specialized algorithms. Historically, chess engines were programmed to look at a chess position and use positional understanding (imparted by the human programmers) to decide what the best move is. But eventually, the chess engines that actually became stronger than humans were instead programmed to just check every possible move and countermove and see which ones lead to a win. (I'm oversimplifying, but you get the point.) So even before Stockfish contained a neural network, it was considered an example of the success of the Bitter Lesson.

As it applies to AI agents, the Bitter Lesson would predict that the best possible agent would simply possess A) a way to do anything it wants, B) a way to evaluate whether what it did was correct, and C) a ton of compute. Then just turn it loose on your task. (The fact that the "brain" of the agent is an LLM is kind of irrelevant - you could also imagine the brain just being a program that generates random syntactically-correct code. What the LLM achieves is that, the random generator would take millions of years whereas the LLM is much more efficient at creating plausibly-working code. This is analogous to a chess engine's pruning heuristics.)

The hard part here is B. We've seen some great agentic successes when rewriting an existing project in a new language, since the agent can just use the project's prior test suite as its evaluator. But when developing a new project, you're still figuring out the finer details of how everything is supposed to work. As the old saying goes - writing a spec that perfectly describes how a program should work, is equivalent effort to just writing the program.

And perhaps at the end it all gets a single pass by a god-tier model for overall sanity and congruence, but the actual work, planning, coordination, and even user interaction was done by cheaper and faster agents of much more limited capability.

It really is absurd to ask programming questions to a model also trained about the lifecycle of a fruit fly.

Instead of building small models from scratch, we train an enormous model and use ridiculous amounts of GPU memory. In the end, the whole thing is shoved into RAM because we don’t know where the useful parts are…

We certainly would know where they were if they were just in smaller models in the first place!

The correct analogy here is Fable as a second tier player assisting a SuperGM in running stockfish, then assessing its output to identify promising variations.

There might be a limit somewhere that prevents the bitter lesson being axiomatic - for instance where simulations for anything can be exhaustive - so that judgement isn't needed any more as an arbiter. But while there are problems sufficiently complex or large to require a breadth models don't currently have, greater scale and compute will continue to convert to better decision making, and the bitter lesson will remain true (true enough).

https://mcpmarket.com/server/stockfish

https://til.simonwillison.net/llms/mcp-in-claude-and-chatgpt...

Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.

Needs? Customers want something that immediately answers their question/solves their problem, but that’s far away, even ignoring the “run locally on a 1U server” and that that may not be in the company’s interest.

For many companies, that support line is a cost center, not a PR mechanism.

Also “Press 1 for sales, 2 for billing...” has the big advantage that it handles all accents, speech impediments, etc.

Long term I think a solution where a user’s agent trained on their voice, running on their phone communicates with the support agent of a company is where we will end up, and support phone lines will cease to exist.

GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal.

Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions (eg tools internet docs).

LLMs would not be as good without expertly written context, and expert context without LLMs aren’t as good either.

The models are not even really trained bitter lesson-style anymore. That concept peaked during the era of pre-train scaling, back when it was thought that making a bigger and bigger GPT-3 would automatically solve all problems through prompting. In 2026, the most important part of training is post-training, which uses vast quantities of niche, hand-curated data to fit the models for specific tasks in domains like tax law.

If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim)

But that misses two dimensions: 1. The cost of acquiring that output 2. What is actually "good enough" for that specialist domain

Not every output needs to be the best to produce value.

And as specialist models increase in cost, their cost/value proposition goes down.

At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models.

That's where fine tuning helps.

Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood.

Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute.

In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it.

I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess.

In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction.

> I would also be VERY slow to claim that general-purpose models will never be competitive at chess.

This is not the claim. The claim is that for the same amount of compute, a general-purpose language model will never beat a Chess model. I'm dubious, but allow for the possibility that a language model could eventually compete at a top level against humans with enough compute. However, it will never compete with a dedicated Chess model with similar resources. Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right? The bitter lesson was only about compute over human algorithms, not at throwing compute at a generalised domain over throwing compute at a specific domain.

You made arguments against two claims that I did not make (that I was trying to refute the bitter lesson or that I claimed that LLMs could never be competitive against humans at Chess), so I'd like to ask you read my statements a little more carefully this time.

I see arguments like this all of the time in ML. There is a large branch of statistical theory that can cast some doubt on this.

Stein's paradox: when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately.

This goes deeply against "common sense", but is mathematically proven, empirically proven, etc.

If you are just taking a chess model vs. an LLM, you are right.

But if you want a chess model, a medical diagnosis model, and customer service chat model, at a high level the intuition of Stein's paradox is that one combined model will get you better performance than three task-specific models.

Most practitioners don't get this because what amounts to graduate-level stats theory is often brushed aside, and we keep relearning it the hard way.

https://en.wikipedia.org/wiki/Stein%27s_example

And in 2020 Stockfish 12 adding some NN evaluation. And then in 2023 Stockfish 16 entirely removing the classical position evaluation code.

https://stockfishchess.org/blog/2023/stockfish-16/

Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner.

What countered it: Moore's law stopped working.

Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.

However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.

And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.

The actual argument of the Bitter Lesson essay is pretty limited but people's interpretation of it has gradually drifted until it's seen as prediction that current LLM will reach AGI at a large enough scale.

When you're doing something "at scale", you want smaller models, weaker heuristics and rules written in code, as they're much cheaper, much faster, much more predictable and much easier to reason about.

If your task is "tell me which pages on the internet are most relevant to this search query", running Claude Fable to evaluate every single page would be ridiculous. You may run something like it at the end, to synthesize information from a small result set for example, as that's a "human-sized" task.

If your task is "write a function to sort a list of 1mn entries alphabetically", a small coding model will exactly match the performance of a frontier model (it'll 1-shot it, with 100% accuracy). So the only differentiator is cost, and the frontier model will cost orders of magnitude more than the specialised model.

For most intelligence based tasks, you don't (and never have) needed the tool which "performs best at all tasks". You need the cheapest one which performs adequately for your immediate task.

This doesn't mean the bitter lesson is incorrect. At the frontier, it's still correct. It means that it's not applicable at all to lots of tasks.

VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.

The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.

Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?

Any similar model aimed at coding?

A >10B model for mass spawning/swarming and reporting back to a larger model

I wouldn't use it for anything important without heavy supervision, as it's very weak outside its specialty. Not ideal for instruction-following tasks.

    document.body.style.cssText='margin:auto;width:600px;'

Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.

And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.

One thing notable is that it makes a difference when I specifically refer to line numbers and variable names when I'm asking it to make changes. Meaning, I also read what it writes and understand what it's doing, and have specific queries where I already know what I'd like to change I add.

The more reasonable comparison is against rented GPU's, while looking at tradeoffs in latency and upload/download/storage/instance management overhead.

Buying hardware for local models is meeting a wholly different need than buying tokens through OpenRouter or whatever.

All decades prior and up to about a year ago, I would have agreed with you. My Framework Desktop, however has appreciated in value by 75% since I bought it. Will it stay there for a long time? Probably not. But it shows that there are no hard and fast rules about things anymore.

64 can still easily do a Qwen 4.8 model, so I’m relatively happy with my purchase… plus, it’s price change has caused it to quickly appreciate in value… so I could sell it if my situation ever turned dire lol

If I'm a professional photographer chasing the best possible end product, I'm not buying cameras because they're economical. I'm buying the best camera I can get my hands on to get the best product I can produce within reason under the understanding that it doesn't have to equate to the best economic decision to be the _right_ decision.

If you're in a position to be able to take advantage of the local inference - it's a no brainer. If you're not sure how that would be done, then it's not a good move.

For example, I have a small posix-shell-based LLM harness that can SSH into my NAS and run organization tasks using the local DS4Flash that I have right now. It's already been a massive help for me to keep me organized, and that's just 2x DGX Spark's worth of compute.