So the LLM would have done all of this on its own? Why is it ok to acknowledge the human was needed but it’s not a collaboration? Is there a defined percentage of ownership required to make the word collaboration valid?
If you get someone to build you a house and they do it on their own, does that not count because they wouldn't have done it if you didn't pay them to do it?
Technically you built it yourself and the builder was just a minor collaborator?
"However, the most astonishing thing about this break is that the GPT–6 Astra did it entirely on its own. Carter Leffer only directed GPT–6 Astra to see if it could break any of the unbroken Enigma messages published on the Crypto Cellar Research web page."
I mean... I'm all for collaboration but I think this case is pretty clear, no?
I mean, the LLM could do it even without all the HUMAN knowledge that was stealed during training about the Enigma machine?
We are fooling to me, there is no intelligence in these models, they just apply methods that were invented by humans without any consciousness on what they are doing.
You could say exactly the same about humans. Ex nihilo nihil fit. Every human depends on a vast corpus of prior human knowledge to be able to accomplish anything. This doesn't mean they have no intelligence.
It is much closer now. Just like how C/C++ devs used to say "JavaScript isn't real code" or "Python script kiddies". Code was literally designed to be just easy enough for people to understand, now it's just even easier to understand.
It is not yet the same as C++ or Python, for two reasons:
1) Ambiguity is still the default. Formal languages force you to resolve it up front.
English lets you paper over it until the model or the compiler (the human) notices.
2) The "compiler" (the LLM) is statistical and non-deterministic. Same prompt, different day, different bugs. A real language has a spec.
The practical move is to treat English as a high-level specification language, keep the generated artifacts inspectable, and "still know enough of the lower layers to notice when the translation went wrong."^1
[1] This is the key that is where humans can still be necessary, or at least another pass through the LLMs to decide on the best path, in the compiled code. Compilers for other languages do the same C -> Binary, etc.
A conventional compiler is bound by an as-if rule. It can take many internal routes, but the observable behavior has to match the language spec. Same source, same defined semantics. If two gcc runs emit different binaries, the program is still supposed to compute the same answers on the same inputs. That is why people treat the source as the artifact and the binary as disposable.
An LLM compiling English has no as-if rule unless you add one. "Sort the users by last active" can become a stable sort, an unstable sort, a SQL order by, an in-memory timsort, or a query that drops people with null timestamps. All of those can look like success. They are different programs. The model is not optimizing under a spec. It is filling in the parts you did not write.
A human who can read the destination language still notices when the chosen path is the wrong program.
A second model pass can compare paths, but only if you give it a way to score them: tests, types, invariant
So the historical analogy still holds, with one correction. JavaScript and Python were dismissed for being too easy, but they already had grammars and evaluators. English is easier still, and the evaluator is a statistical translator that will invent a dialect if you let it. The practical move stays the same: treat English as the spec language, pin the generated artifacts behind tests, and keep enough fluency in the lower layer to see when the translation chose a different program than the one you meant. The human is not required because the computer is weak. The human is required because the source language still leaves room for more than one destination.
The difference is that those C/C++ devs were wrong. Claiming Javascript isn't code is just wrong. Even claiming HTML isn't code is wrong. How is it much closer now? Did English evolve to be more like code, or did code become more like English? And I'm not talking about how Python is more English-like than 6502 Assembly, I mean in the last few years.
English is not code just because a technology was developed that could make educated guesses based on being trained with other code that people have written as to what the code generated should look like.
A cyclist pedaling up a mountain isn't a "collaboration" between a bicycle and a human. This is the same. You don't see feral bicycles roaming the land. All models are ultimately built and run by humans, with human-provided instructions. And as with any program, it's garbage in, garbage out.
More apt analogy here: a cyclist pushing a bicycle down the mountain and seeing it somehow get down the whole track without falling down, is not a collaboration between a bicycle and a human. The human was not involved beyond giving the initial push.
Cyclist still chooses the time, mountain and direction the bicycle gets pushed in. Bicycles have no agency and only go down because gravity. Bicycle will not "discover" tree or wall, any outcome solely the result of cyclist's decisions even if thrown bicycles don't have generally deterministic paths. Don't anthropomorphize the bicycle.
If I told you to pick an encrypted message from the web site and decrypt it, and you went and did it with no further input from me, would you okay with me calling it a collaboration to decrypt the message?
The bicycle -- a simple method of transport powered entirely by humans -- used analogically to prove a point about [clears throat] automation.
I think this sounds somewhat less silly in English because "automotive" and "automative" don't have the same hyper-visible affinity, but all the same; you may want to consider the car as a more viable analogand.
What do you even mean it wasn't a collaboration. At any meaningful level LLMs just plain out suck when left unguided.
The shortcomings should really be obvious by now to anyone honest. And the marketing distortion being oushed out is just tiresome and detrimental for all of us.
What's tiring is the constant snide dismissiveness of the advances in technology by people who don't even RTFA, but yet show up on every AI thread and spout the same nonsense.
A generic agreement to slow down is prisoners dilemma stuff (and maybe necessary too), but just taking a super hard line stance on enforcing laws on ai companies and going after them for negligence and breaches for eye watering amounts of money would have the dual beneficial effects of slowing them down and making them more careful without needing complex game theory.
No that's not what they did, they exposed jfrog raw. It would have been so extremely simple to gate services they need the llm to access... I mean, jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools
Right, you mean it didn't go through a gateway? But would that actually have helped? The requests all went through jfrog didn't they? I guess it depends on the level of filtering at the gateway?
Whilst it might not be JFrog's threat model, I wouldn't assume it can be used as a full internet proxy.
I don't really mean to defend OpenAI here, but they did make some attempts at sandboxing. Although it does seem that they didn't really know what they were doing.
It would have been a case of isolating exactly what functionality is needed and wiring that up with the actual requests. Not like a full pass-through proxy. This is what we've been doing in our company as well
> jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools
And we don't just magically know all the consequences of that.
Which is exactly why we do need full, physical air gapping. (Which, yes, would also include self-hosting a mirror of the package repo, if the point of the simulation is to see what's possible with the real package repo.)
Currently I don’t use any CDN. I just host everything on Scaleway on VMs and serverless. Works pretty well for my use, but I imagine latency is bad for South Americans. I’ve looked into Bunny and I think that is the closest you get to Cloudflare’s offerings.
That actually makes the "Drift" analogy perfect, because you have to be "Drift compatible", you can't just grab two people off the street and throw them in a Jäger together and have it work.
What are you talking about. You think service providers can backdoor aes-gcm? There will always be technology that they cannot get around. The only way to backdoor is to explicitly change the encryption.
This is not specific to 'backdooring aes-gcm implementations' at all. Read again what I wrote, namely this is the general status quo on 'information system security': they don't need your password or aes-gcm key to watch if they really want to. You would be a fool to presume anything else.
There are ways to do it correctly. You just end up spending a lot of time conceptualizing and refining abstractions.
To me the issue is more that conceptualizing requires a certain state of mind. Before llms it was 10% hard thinking 90% implementing. Implementation was actually sort of a reward, it felt so good just being in the zone and fleshing out ideas.
Post llms I find myself walking up and down quite a lot, only doing the thinking. Now it's more like 40% thinking 60% reviewing plans/code. I haven't experienced flow state since. The thinking is fun but exhausting, the reviewing is just kind of annoying, especially as llms get into these weird failure modes. Before I could look at a bad piece of code and instantly tell what the author was thinking and why the thing doesn't work. Now I need to be a lot more careful because there is little code smell, but a lot of badly chosen abstractions.
Somebody talking about researchers, I think it was Hamming, once said that there are people who just can't think without a bench full of equipment in front of them. So if you want to get good work out of them, your job as a lab director, then, is to give them that bench full of equipment and let 'em cook. I think the same thing is true of some programmers, and I think I might be one of them. We could sit around and conceptualize till we're blue in the face, but without an editor open with code in it we can't think through our conceptualizations effectively, and a chatbot is no substitute. A chatbot just adds another layer of abstraction to a process that's already thick with them, like a wall that got repainted so many times it's covered in a few millimeters of stratified goo that partially melts in the summer, and what's worse its behavior cannot be meaningfully predicted or reasoned about. Everything you think you know about how to correctly get results out of an LLM is either guesswork or folklore, and may be obsolete by Labor Day.
This also partially explains why I'm fond of Lisp. Paul Graham once said that while Lisp is a great language to work in, its real value comes as a language for thinking in.
I'm a junior and I probably spend a similar amount of time thinking vs reviewing. I rarely write code unless it's about <5 lines.
I find the instantaneous thinking easier now. I can have several ideas in mind, and have a concrete implementation made for each, making it easier to compare alternatives. Although, since each problem is alone easier to think about, I do end up handling a greater number of problems. But I expect that my total volume of thinking is likely the same as before.
Where I do certainly feel more tired is when I try to solve too many problems in parallel. If I try to do that, I end up constently dropping context. So I generally try to finish a big chunk of something before switching (usually that means getting it ready for another code-review cycle).
I do miss writing code myself. It's certainly satisfying. It's just significantly slower in most cases. I try to do it in my free time.
> I can have several ideas in mind, and have a concrete implementation made for each, making it easier to compare alternatives.
I would ask what exactly are you comparing. I don't think I've ever wrote 2 versions of code to compare between each.
I've written exploratory code. A few lines to quickly inspect the behavior of module/function because it's undocumented. If something needs tuning, I surface the parameter in the interface, hook it to an harness to plot and manually tune.
I've also written alternative implementation of some feature, that later was abandoned.
But I've never written multiple versions of the same feature at the same time. I either model it (algorithm) or sketch it (interfaces or some other flow). It's way easier to interate with those than some demo/prototype code. The latter is when we settled on a solution and wants to fine tune it.
The mega-rich clowns firing everyone and then hiring back just those few that they realize they actually needed to keep after-the-fact are actually driving themselves right off that same cliff along with everyone else, and they don't even realize it. I bet money all this "AI" nonsense don't end up leadin' to the "Rich Guy Utopia" they think they're workin' toward. It's much more likely gonna just lead to a shittier world for everyone on Earth, rich and mega-ultra-totally-too-rich folk included. Wait'll the "AI bubble" bursts and see how much fun they have losin' zillions while pretty much everyone else lands in the "poor house" and comes lookin' to them for retribution. I suspect the guillotine is makin' a comeback sometime in the future.
They will most certainly cut subscription access, same as Anthropic. It is inevitable that they will go down the same lockin+squeeze route. I unsubscribed from Claude a couple of weeks ago, on gpt now. However, Openai will have to make a similar move.
At that point however open weight model providers will start to shine. All eyes on China.
Agree on the knock-on effects. My prediction is deflation. Money will be worth more and more. As a consequence governments will have to step in to ensure inflation(with e.g. universal income), otherwise the economy stops.
But honestly I'm not sure this will be enough for people to spend on e.g. restaurants or activities or oh I don't know, children. I think this will imply a freezing or even stepping back on the Maslow pyramid, the majority of people consolidating in the middle.
What I'm mostly concerned about is not even economic, it's psychological. With nothing to do, people will not have purpose, and bored people are a gunpowder keg.
> What I'm mostly concerned about is not even economic, it's psychological. With nothing to do, people will not have purpose, and bored people are a gunpowder keg.
I'm not so sure about this one. The powerful in a society like the one you describe would surely know about that potential powderkeg and supply ample cheap entertainment to dull the edge. Then we'll have a society of mostly dull, idle, useless people with no purpose.
That's even more dystopian than your scenario, if you ask me.
Yeah that also sounds realistic, and actually there's evidence of this dulling effect from even before llms. The attention economy has been literally streamlining.. well, the road to death. And nobody is angry.
I can spin this in a weirdly positive light though. With fertility rates going down, life becoming less and less meaningful and simultaneously a small and decreasing group of people becoming extremely productive.. maybe humanity will finally stop exploiting the planet and start a sort of transition.
AI enhanced increased lifespan forest elves watching over nature. Mm I'd prefer that over soma. We are the heralds of The Great Ones
Nonsense. What the hell would you do with 1B? Give it to charities maybe. Maybe set up an investment where dividends are paid to charity. Running out of ideas
Set up a nice investment vehicle with maybe 400m so I can get 1.6m in dividends a year which would be better enough to comfortably travel the world, have a private chef, someone who organized travel so I don't have to..
A nice 12 person yacht on the Mediterranean is 400k eur for 2 weeks (with staff) so I'd realize it's not enough and invest the rest so I could get comfy.
Along the way help friends and family, pay off mortgages, usually good stuff.
What's the point of that? That sounds like the most boring life. You want to rot away on a yacht? Private chef? Are you kidding?
Help family? Sure, although you don't need that much money for that. Friends? Ehh not very smart, just think about the changes in the friendships' authenticity.
Private chef, absolutely. Like some people rot away managing Linux as a desktop or putting together 3D printers instead of buying one that works and using a Mac, I enjoy food.
You've run out of ideas already? Try harder!
What charities? Why? How much, to which ones? How involved with those charities are you going to be? What dent in history are you going to make with that billion? With or without your name attached. Build housing, cure cancer, feed the hungry, buy this simulator https://www.1940airterminal.org/news/liquidation-of-simulato...
I could be wrong but I think you could get started with all of that with a fraction of $1B.
Sure there is leisure and entertainment but if you want to use it to do something meaningful, with only 24 hours in a day you'll probably have much more money than time to use it well.
On the other hand 1B is really an arbitrary choice of number, so I think the reason he would choose this specific number definitely has more to do with arbitrary reasons (class, status), perhaps subconciously.
Personally I don't agree with the parent that everyone wants that much money. I think I can safely say not only am I content with much less but I also don't ever want to have the responsibility of having to manage that. Though I'm already saying that from a place of privilege where I don't need to worry about survival.
Furthermore, a lot of money almost certainly places you in an outlier group where normal laws and rights as formulated by humans don't apply the same. Assuming everyone has some empathy and sense of justice/righteousness, that should make them intrinsically not want to be in that group.
Completely missing the other costs associated with any of these things. If money was enough to “feed the hungry” Musk or Gates would have already done it. The real problem is systemic injustice, like governments stealing foreign aid that’s meant to go to the poor. Money can’t always solve these.
Time is more valuable than money and unless you have tons of time and space that simulator is just an expensive paperweight.
My point was that there isn't anything I could do with that money, and neither can the vast majority of people in the world. So I would immediately try to pass it on to people who have better use for it
Wishing for 1B is completely nonsensical if you understand what kind of money that is.
If anything less than $1B isn't enough then it is never enough. $1B is the new $100M thanks to ongoing currency debasement.
Also, there is something called "taxes" which is what makes anyone who has millions or billions to want even more money and the IRS will still come after you anywhere in the world.
Otherwise they have to renounce their citizenship and move to a tax haven.
reply