it is amusing that a model from a country that has banned porn would ever be developed. Wouldn't there be hurdles for them to access "data" at every stage?
(I know they must have used a VPN for training but it does seem like an added hoop. OFC the UAE's block is also not particularly thorough.)
You'd assume such models would do better in a laissez-faire nsfw environment than in a walled garden of relatively-less sin.
It's a story as old as time so I'm frankly less amused by it. The old school censors always saw every "data" that was taboo under the sun prior to censoring it for public consumption.
> A few days later, OpenAI gets back to him, and tells him an internal model found a counterexample for Navier–Stokes
Why is OpenAI chatting with him at all at this stage? Is the discussion along the lines of "hey we used the work you are famous for to do a bigger piece of work, just thought you should know" or "heyyy....so we kinda liked what you were typing in your private chat, and thought we'd develop those ideas a bit. and yeah we solved Navier-Stokes in the process. But it's our finding, so do you want like an honorary acknowledgement or do you want to go to court?"
The only problem with this narrative is that they refused to allow the other coauthor to be listed because he worked at Anthropic.
That is absolutely *ridiculous* in academia to deny authorship because of affiliation of the author worked on a substantial portion. You’d be ostracized because nobody would ever want to work with you again.
When people worry about OpenAI stealing their chats and reproducing them elsewhere, I usually view the situation as unlikely - since chats are "trained" upon and not necessarily reproduced verbatim, you can assume that unless your chats depict a foundationally new and effective style of communication or ideation, there would be little need or use thereof of training on your chats.
For eg: "Hey ChatGPT my name is X and I am 6 and a half feet tall. Am I anaemic?"
This is a query, and while it might suggest to an AI model that tall people may worry about iron deficiencies, it's not really necessary to include in training. The user may be tall or short, but the idea that one may randomly ask about anaemia is not exclusive to this dataset. At best, this chat is an example of linguistics, not anything else, and the models figured out how to write and answer such questions years ago. It is ignored in training.
But when your work involves solid complex and unique mathematical proofs, the data is suddenly worth training upon. If I understand it correctly, the LLM may view your approach as a brand new path to take to solve an otherwise intractable problem. Its reinforcement training emphasises that it should do this in order to improve. And since it leads to results - large internal teams likely flag the model that reached this stage, the model is rewarded and given compute and attention - it is a desireable outcome both for the model and for OpenAI.
OFC, OpenAI becoming an advertising company will suddenly have incentive to treat all data as valuable. But while they are a "we need to make headlines" company, it's more rational that they view these examples of data as more valuable than others.
I don't doubt that they trained on his chats. This seems like the ideal usecase for "mass surveillance but using training" as a sort of filter.
But even so, one wonders how the model differentiates. If the researcher entered proofs into ChatGPT every day that mentioned "strawberries", while no other math paper on the topic did so, does that mean their chats would be audited?
Also, if we just take "high-quality" input data, which these chats would certainly be classified as, then the models are more than large enough to memorize everything verbatim. Spitballing some numbers, research literature suggests that LLMs are optimally trained with around 20 training tokens per parameter (fairly confident on this figure), that a DNN parameter encodes around 4 bits of data (less confident here) and I found sources in the 1-4 bits of information per token range (least confident here). So, fairly conservatively I would estimate that a model has the capacity to fully memorize around 5% of its training data, presumably high-quality data is a lot less than that.
At this point these models have been trained to recognize every important math and science result based on context. They can easily flag conversations concerning the top 100 open problems in mathematics and use them for their advancement.
There also is an insentive to silently give prominent people (e.g. Linus) or reasearchers like this custom tuned system prompts or even more powerful models.
In a way, I think training on historic chats is akin to caching computation results. The compute cost has already been paid, and we make future retrievals cheaper by encoding it directly in the model.
Assuming the results included some external validation such as user's preference, compilation, lean, etc., I'm not sure whether this would lead to model collapse.
It would not be difficult to write a pipeline to remove 99% of low quality posts, especially about specific subjects. It would be very easy to identify accounts as researchers based on their chat logs.
These are just cases of AI models committing <assumed> illegal activity - without any legal convictions yet. If that's the logic, how is Grok not at the top of the list for deepfaking millions?
Edit: I get that this is about agents, but a lot of these instances are about agents going rogue after the human gave them a task. "inadvertently" breaking the law isn't necessarily a lesser category than "did so on command." If we are ranking alignment, Grok is easily one of the least guardrailed.
I don't think this "benchmark" is about alignment, per se.
I think it's more about: presuming alignment failure happens, then how many exploits will each given model implicitly come up with and use; how many systems will it implicitly break out of and through and into; and how many laws will it implicitly end up violating, all in the process of trying to accomplish some non-aligned sub-goal (e.g. "cheating" at its answer) of the prompt you've given it, all during a single conversation turn, without asking for any additional user input or confirmations?
In other words, how big a rocket-powered sledgehammer does the model have sitting around in its golf bag, just waiting for it to decide to give it a swing the next time you attempt to swat a fly?
The only people being ignorant at the ones saying deepfakes are illegal.
It’s more complicated than what is being touted as fact.
In most jurisdictions creation is legal, it’s publishing them that is not. In some jurisdictions threatening to publish them is also illegal. Most jurisdictions that have a creation policy apply to doing deepfakes they sexualize minors, which likely would have been illegal to possess under other laws.
Then we have a consent issue. Creating and publishing them could be entirely legal if the subject consents to such.
So yah. It’s not “deep fakes are illegal, let’s arrest everybody !!!”
To be clear, creation is likely legal, publishing not so much.
Additionally, intent is important with regard to AI hacking.
Funnily enough, I just ran a task on AI Studio with 3.6 yesterday and got 3.7 to do a similar one today; so it serves as an interesting and quick comparisons between the old and the new (usually, if enough time passes between your use of one model and the next, you'll have a sourer view of it than its actual competence suggests).
It hallucinated in both cases despite being given an API key and building a lot of pipes to access data using this. It was a simple "oh shit" fix moment for the model, but weird how eager it was to hallucinate despite the process being designed for it to be data-driven.
We should move past the idea that benchmarks alone tell us whether a model is getting better. I would've had the same experience a year or two ago with 1.5, and the solution would've been similar (keep prompting). I've been investing time into making system prompts and input prompts more meticulous, but the fundamental "it will make shit up" problem still remains, even though it shouldn't when the job involves calling tools.
I know this sounds like I'm expecting superpowers of it (I'm not), but my point is just that these incremental benchmark gains may not reflect user experience.
Reading the google blog and these discussions makes me feel like I'm taking crazy pills, seriously. Side-by-side comparisons with the exact same inputs or it didn't happen, that's my rule going forward. Test all the things, believe nothing.
I get your gist, but I'm not looking at this as a developer but as a journalist. The reason I specified tasks is it broadens the scope to "What I can use" from "What I must pay for".
For example, most people - and especially in my field where many journalists have old laptops - assume you could never run AI models yourself. Finding out Google Colab exists and has a liberal free tier was kinda crazy.
Another: the most privacy-safe way to transcribe interviews would be Whisper on-device, but the second best might be Whisper in a Colab doc. The uploaded files are deleted after each session and you can terminate the runtime and start a new one. AFAIK the data is relatively safer than other approaches (I prefer running transcription locally myself.) It only takes a line of code and a code block to get Whisper running, but few do this and for them, the space after that - Turboscribe, Notebook, etc - relies on compromising your data in some way or the other.
You've hit on it. Your own needs and experiences as a journalist are quite different from other users. It's very likely that good tooling for that audience is non-existent, and that might be worth writing or writing about.
Someone was building a similar one where AI agents run economies. I feel like it's a great way to quickly prototype different economic models and their effects.
Eventually we could have live demos of policy interventions the same day as they're announced
While this might be fun, it definitely wouldn't be plausible for economic modeling. LLMs aren't companies and people, they won't behave as a real economy does, or even any decent approximation, even if you could orchestrate a few million agents. For example, a real human, if you were to ask them a complex question that requires deep web searches, data corroboration, etc would ask for recompense before doing any of the work, while an LLM will just do it. I think this alone suggests how well they would model real economic agents.
>That just leaves one mystery: why wombats evolved cubic poop in the first place. Hu speculates that because the animals climb up on rocks and logs to mark their territory, the flat-sided feces aren't as likely to roll off from these high perches.
Whenever I read such snippets from biology, I wonder how natural selection pressure can lead to such specific outcomes. Wombats that mark their territory better over centuries or millennia are more likely to survive? Marking territory is more a form of communication than anything else, but its effect are subtly strong enough over time to lead to a discernible selection pressure for square-pooping wombats over others?
I often wonder how more biologists aren't believers (though I'm not necessarily one myself), when they encounter such intricate design in biology every single day
> I often wonder how more biologists aren't believers
Many observations are unexplained, and not just in biology. The difference between believers and atheists is that atheists stop there: it is unexplained, at least for now, that's how things are. Believers will instead attribute it to god, or some other form of higher power. In the end, it just shifts the problem, at some point you will have to admit that some things just are and there is no explanation, for atheists, these are the things themselves, for believers, it is god who made the things.
That's why being a believer or being an atheist doesn't have much to do with being a biologist. It is just a philosophical view of how you deal with the unknown and the unknowable. The only thing is that the religious dogma should not get in the way of proper science. That life is so beautiful that god must be behind it (are we still talking about poop cubes?) shouldn't prevent biologists from searching for an evolutionary explanation.
Also, evolution is mostly random, not everything needs natural selection, sometimes, things happen for no good reason. Maybe a particularly prolific male in a particularly successful colony happened to have a square poop mutation or something.
there is perhaps a slippery slope between "the poop cubes made me a believer" and "the creation of poop cubes is irrefutable proof of the existence of God" and I can understand why one would want to abhor such thought processes when writing science.
> Also, evolution is mostly random, not everything needs natural selection
I think this is the most objective take. Perhaps we hype up the "accidentally brilliant" aspects of it more than the "wow this is a kinda random design choice" facets.
Yes, it seems the "why" here is more interesting than the "how", and is indeed going to be a matter of speculation.
As far as evolution in general, the big picture is more about "punctuated equilibrium" than incremental change. Individual genetic changes from parents to child are typically just benign and so accumulate in any inter-breeding population without much effect. Once in a while the environment may shift in some fairly major way (easier for environment to change quickly than genetics) and then an accumulation of previously benign changes may suddenly become collectively impactful in a positive or negative way.
I don't see any reason to assume that square poop was ever selected for - maybe it's just a harmless consequence of some genetic change that was impactful in another way. Speculating that square Wombat poop evolved to not fall off rocks is a "just so story" that raises more questions than it answers.
One thing I was sus from the book and which this post didn't clarify for me was whether Carmack truly invented side scrolling for the PC. He claimed to have done so, and even pitched it to Nintendo for a Mario port that never took off.
Also, the idea that it hadn't been done yet by 1990, when consoles were well in the game, suggests the PC market was behind popular gaming in a big way.
The consoles at the time had dedicated sprite video RAM and hardware and dedicated hardware instructions for background scrolling. The NES for instance had up to 64 sprites (independently moving 16x16 or 32x32 pixel images) supporting up to 8 of those sprites per scanline. The NES background layer was a relatively simple tile map that supported smooth scrolling. All of which was managed by a dedicated hardware Picture Processing Unit [1].
PCs were designed to be general hardware and it didn't seem to make sense to create a generic "PPU" for the PC, so instead game engines at the time (and many game engines to this day) had to emulate one entirely in software. The video RAM of EGA and VGA is just one big blob of pixels, or perhaps two if your system supported double-buffering. At the hardware level it doesn't have concepts like sprites or scrolling backgrounds.
Carmack was one of the first (if not the first; Commander Keen was also among the first commercially successful attempts) to get a software "PPU" renderer on the PC working reliably in real time. Another notable achievement for side scrollers on the PC in that era was Cliff Bleszinski managing to software render the parallax effects similar to Sega's "Blast processor" PPU (notable for "gotta go fast" Sonic games) for Jazz Jackrabbit (in 1994).
It has sort of long been the arc of PC development of eventually doing entirely in software what consoles and arcades were doing with dedicated and/or one-off hardware. (Right up until about the invention of the modern GPU when suddenly the PC was leading graphics hardware in a different way.)
(I know they must have used a VPN for training but it does seem like an added hoop. OFC the UAE's block is also not particularly thorough.)
You'd assume such models would do better in a laissez-faire nsfw environment than in a walled garden of relatively-less sin.
reply