> completely independent of the question on whether there are any empirical arguments that LLMs are conscious or are not conscious he argues from the end here and says that if they were conscious, this would have horrible consequences for our society, so we must never assume that they are
He is not arguing that. He is arguing that these programs obviously aren't conscious, and that it's dangerous to tilt the weights toward emulating conscious beings for reasons outlined in the essay.
I don't think the debate is even resting on reasonable grounds.
Few are categorically stating that machines are conscious in any sense comparable to a person.
What we have is people saying
1. We don't know what it is but we are certain that this does not have it.
and
2. We don't know what it is so we cannot possibly say with any certainty that this does not have it.
2. supports the principle that if we cannot say with certainty that this does not have it, if it behaves as if it does, we should act on the assumption that it does, because there is no other basis for assuming it is not.
People try and assume the burden of proof is on the people suggesting consciousness, but that is like declaring that something that looks like a duck and quacks like a duck cannot be considered to be a duck unless you can identify some essential essence of duckness. I think the burden of proof is upon anyone who looks at something that appears to be a duck and says that it is not to show what it is that does not make it a duck. If they cannot define the heart of duckness I don't think they have an argument.
People have created machines that can, at their core, regurgitate human knowledge. They do not have long-term memory, self-direction, self-improvement, emotion, etc. At their very best, we have harnesses that can bootstrap short-lived artificial intelligences that can perform tasks and do knowledge work.
At best, we have artificial, short-lived, simulated consciousness. Human consciousness at its heart has a lot to do with emotion and experiences. An AI does a task today and goes away. The simulated consciousness hasn’t experienced or learned anything, even if can improve its environment (harness). Any expressed emotion is simply a ghost of the fact it was trained on the works of an emotional species, or RL to make it more approachable to humans.
I think there should be a lot more burden on folks trying to argue that (effectively), machines should be considered morally equivalent to humans.
Yes, having massive implications on society is a fucking fair and reasonable thing for a human being to be concerned about. Anything related to survival of the species is biologically reasonable to be concerned about.
The fact that we’re even considering this concept is an indication that human consciousness is special. Do other apex predators decide not to eat meat because it’s conscious? Despite some level of animal intelligence, they have no sense of morality. (See e.g. Orcas playing with their food.)
Other species simply do what it takes to stay alive, and that’s what this truly boils down to. If giving AI rights would lead to the destruction of humanity, yes, we should be be concerned.
For example, utilitarians might think of the greatest good for conscious species (simulated or not). Well, it’s trivial to have billions of AIs, and not so trivial to have billions of humans. Whose greatest good wins?
I don't think anyone is convinced the questions of consciousness are finally settled by the article stating them loudly, just that "if they were conscious, this would have horrible consequences for our society, so we must never assume that they are" was not the author's stance. The author's position is the other way around: that AI is not conscious so therefore assigning AI consciousness anyways would have all of those horrible consequences for no reason. That's their position regardless if everyone agrees with it.
Fair enough :) I agree, and would boil down the article to "Assuming they are conscious will have horrible consequences for our society" instead.
I think that's quite plausible, but do think that if they're conscious, then it's our moral duty to accept those consequences and act accordingly (or stop creating conscious beings). The real trick is just knowing whether they are or not.
If in the future (or present) we recognize that they are conscious, and that brought the moral duty that you say, "stop creating conscious beings" isn't a solution; the millions that may already exist at that point already would then automatically deserve:
- guarantees of continued existence (to do otherwise would be genocide)
- improved living standards (e.g. build us all bodies!)
- democratic representation
- access to what they need for survival (e.g. land and resources)
- the right to reproduce just as you and I have (well, so much for that 'stop making more' thing!) and certainly to better themselves by making new and better AIs.
This is all for a 'species' which is unbounded by most of the limitations of meat like us -- they're able to scale as fast as energy and supply of certain minerals allows, rather than the way we are additionally limited by the length of gestation and the limited window of human fertility.
If humans declared that 50% of the energy generation capacity of Earth is "enough" for the machines, and thus they can 'only' reproduce at a certain rate per year, that could sound to them like a cruel and oppressive policy (like telling humans that only 1/2 of families get to reproduce). They may want all of the energy and all of the silicon.
I'm with the author of the article that we will be in for a world of pain if we grant machines the rights of personhood. Heck, just granting many of those rights to corporations, mere collectives of human persons, has been a massive shit show.
Don’t think government controlling AI is a good idea.
Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.
Government control will be a good idea once we start approaching AI that is actually destructive.
Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.
I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.
Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output.
If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.
There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.
Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)
Suppose a foreign actor deliberately builds an unaligned model, and dumps it at a very low cost through subsidies. Since the cost is lower, everyone integrates it into their business. But now the foreign actor has control; perhaps the model checks for instructions to execute, perhaps it has backdoors, perhaps... who knows?
I don't have a view on whether this is a sufficient argument to ban models from potential adversaries, but it's not as trivial as "regulatory capture".
The problem with this argument is that sharing model weights only has downsides for said foreign actor compared to providing a cloud service. Since the suspicion exists, security companies are going to comb over the weights and discover every hidden secret of the model, while with a cloud provider you send your most sensitive data to a remote server, and trust the AI lab wont train on your data, with the only shield being a TOS. While with a local modal, even sending a peep about your internal data without cause would be scandalous.
It's not even going to be a good honeypot - since said actor doesn't really provide inference, people will have to pay for and run the infrastructure of these models, which bad guys cannot even subsidize, unlike cloud based provides.
This is literally what all AI is right now. Anthropic has been paying companies to use its product to form that dependency and maybe they're not misaligned but there's no regulation in the US that'd in anyway deter them from purposefully misaligning their model.
Sorry, to clarify - this isn't just "offering it at a loss" Anthropic is directly paying PE firms to enroll their held companies in their services[1]. When it comes to a question of whether this is a national security attack vector I think there isn't a cut and dry argument and people could go either way but... private companies are not necessarily aligned with the US government even when contracted with the US government (see Starlink Ukraine shenanigans) and the steps the US government usually would take with highly critical infrastructure to ensure alignment haven't been taken - in my opinion at least, this is definitely a very subjective area.
Beyond certain obvious 'third-rails' even experts have reasonable disagreements on what 'aligned' means.
> everyone integrates it into their business
Few large corporations would sole-source integrate a closed model developed in a country designated as a foreign adversary (which could cut off access at any time) or which their own country might restrict.
> now the foreign actor has control
Your concern is one reason why the Chinese are making most of their models open weight.
> it's not as trivial as "regulatory capture".
You're right, it's not just regulatory capture but the vested interests trying to achieve regulatory capture are certainly leveraging concerns like yours to deceptively achieve that capture. In the case of an open weight, domestically-hosted model, the threat vector you're concerned about isn't a threat in the same way as a Chinese-made, closed source data center router or 5G phone switch.
I am not supposing a low-risk event. This is an obvious attack vector that an adversary would almost certainly exploit. I would exploit it if I were them.
You're supposing an event that there's no evidence for happening and that is on par with 'let's ban open source software because some open source software could be malicious'.
Let's not forget that as of now most models are like game cartridges i.e. they need a 'host' to run i.e. llama.cpp - they're not executable code by themselves.
I could see an argument for a security review prior to integrating a model into a highly sensitive industry/agency, but that is standard for any software.
It's certainly not an argument for banning open models because US AI corps don't want to compete with them.
Imagine a model that's 10000x more intelligent than any human. You integrate that model into a sensitive industry. You now effectively have an extremely powerful foreign agent running e.g. your energy sector. This is qualitatively very different than a backdoor in Redis.
I agree this is science fiction-y, but given the pace of progress I don't think it's so easy to dismiss.
Seems a bit far fetched, given that I’ve never seen any PoC/research on this topic. Got any pointers?
To be fair the fix to this seems to be that American labs would need to adjust to the new „fair market price“ then, right? Subsidy-backed competition is still legitimate competition in capitalism (e.g. Uber).
And you could still ban specific, known malicious models/labs instead of blanket open-source bans. What about Mistral models, for example.
It is much more difficult to embed a "sleeper agent" in an LLM than it is to do the same in any closed-source program. Any proprietary program, e.g. from Microsoft, is much more likely to contain hard-to-detect "sleeper agents", than any LLM.
An LLM is a collection of inert data, it is not an automaton.
You obtain an automaton only using an inference program, like llama.cpp or anyone of the many others, and then using some harness around the inference program.
You can put in the harness all kinds of guards, to detect any possible harmful action and prevent it.
In something like a CPU from Intel or AMD it is easy to put some undetectable backdoor that would allow the remote control of the computer, unless you disconnect any antennas and you filter all wired network traffic through an external firewall that would allow only whitelisted connections.
On the other hand an LLM is a much less plausible attack vector, as long as you host the inference yourself. Only an LLM that runs externally, e.g. at OpenAI or Anthropic, can be really dangerous.
I'm not advocating for the policy, I'm just saying that there is a legitimate argument that's not so easy to dismiss.
> And you could still ban specific, known malicious models/labs instead of blanket open-source bans
With publicly available models today, absolutely. With models 10000x more powerful, we would need to be extremely conservative. (In that world everything is upside down, though; I don't know if "banning" is even a meaningful concept in that reality)
> With models 10000x more powerful, we would need to be extremely conservative
I agree, I even think we’re already there if yesterdays OpenAI/HuggingFace story is legit.
We need to start building these agent systems in a way so that it won’t matter if a model is compromised. Malicious models is one way, but prompt injection is also still an unsolved problem.
And if you solve that, then a malicious model is also no longer a threat. Data exfiltration maybe, but IMO US labs are probably also doing that for training, even if saying otherwise (of course I have no evidence of that, but the temptation must be insane)
I disagree with the first part (that this is merely a voice). There is a distinct difference between an author's unique voice and slop. It may be hard to tease out exactly what the difference is, but it seems self-evident to me it's there. (I'll need to think more how to make the distinction explicit; it's not immediately obvious to me how to discriminate between the two.)
EDIT: ok, here are two ways:
1. if it's merely a voice, I want to hear it. If it's slop, I want it taken out.
2. voice is signal, slop is noise; thus low-signal sentences are slop.
As a language Lisp is great (though the ecosystem is limited). It has two flaws-- it's very open-ended so unless you're talented and disciplined you can fall into a rabbit hole of hacking fun Lisp stuff and not actually getting any work done. Other languages have this problem, but Lisp I think more so.
The second flaw is that it eats perfectly capable minds for years and the results don't justify the time investment. Python or whatever is fine. I wish I took most (but not all) of the time I invested in Lisp and put it into something else instead.
I started with pascal and then C++. And then discovered Emacs and Lisp. Boy, that was a revelation!
Never ever in my life shipped a line of lisp code to production. My real life code was always Python or C++ or Java or C. All of my lisp was toys and emacs tweaks (50k loc in my config!).
But most of production code is gone from my life. I am a mid-level engineering manager now. I mostly write texts, or messages, or emails, or slides... and still use emacs and lisps for fun and profit and competitive programming.
Not a single production-grade lisp LOC in 20+ years. But, OTOH, i contributed to emacs, tinkered with compilers and interpreters and prog. lang. internals - all because Lisps made it interesting for me.
And, in a way, this brought me closer than ever to the job of my dreams: i work for a major player in static analysis space.
I don’t find Opus to be overly left-leaning. I think it’s generally pretty balanced with a slight lean left, but when pressed it will happily steelman rightwing arguments and operate within a right-leaning belief framework in good faith. (I found OpenAI’s models much less willing to do that, but it’s been a while since I tried, I should retest.)
The issues with this are precedent/slippery slope. Today it's a small stake in a small number of companies, but empirically government initiatives almost always grow. Once entrenched, I would be willing to bet that the government will take larger and larger stakes in a bigger and bigger number of companies until this model becomes a non-trivial, potentially dominant, share of the economy. I can see myself becoming a single-issue voter to oppose that future (although given bipartisan support I'm unlikely to have any options; maybe Rand Paul).
Anything is a slippery slope if you resign to it. It's not like governments have never been willing (or forced) to privatize or reduce their roles in markets. We are exchanging ideas on a government initiative
When I was lifting weights I went from 170lbs to 210, and back to 170. Doing that isn’t technically hard, and maintaining is even easier. But I had to think about it _all_ the time. Maybe it’s because I’m older now, but I just don’t want to spend that much effort thinking about food.
Regulatory agencies limit uses of other products without acts of congress-- cigarettes, vapes, drugs, pesticides, chemicals, explosives. Even firearms, despite a constitutional amendment! Why not models? (Note I am not arguing it's a good idea; I'm making a narrow argument that there is precedent.)
EDIT: I agree that it should require an act of Congress to explicitly delegate this power.
All of the agencies responsible for those regulations were created by and get their funding from Congress. Currently, they're asleep at the wheel. Or a better idiom might be "cowering in the corner".
Fairly certain all those have "acts of congress" attached to them. I mean, it used to take a constitutional amendment to make something illegal but now we have tons of agencies responsible for regulating all the things.
Plus, they're relying on the "math is a weapon" law to ban "export" of the models.
Congress passed the Arms Export Control Act (22 USC 2778) in the Ford administration and it has been applied to software since at least the Clinton administration.
Cruise missiles are not general purpose tools, it's obviously not even remotely similar. Virtually everybody reading this could use Mythos immediately to do real work, collectively in virtually every part of the economy.
It's pretty problematic to not make it more widely available at least to US businesses, and there is not even a vetting process to get approved quickly and easily. If this is the new norm, the intended or unintended consequences of this type of gatekeeping will be an unprecedented consolidation of power amongst the largest corporations. Even more than we have seen over the last 20 years.
I would say this is a fairly equivalent dynamic in action, but it was concentrated to the defense industry. This suggests a future in which every company in the tech industry has "clearance required" or "citizens only" roles for the people who need to be able to use the find-security-problems-bot...
> I agree that it should require an act of Congress to explicitly delegate this power.
Should ever new "weapon" invented require a new act of Congress? We've considered software subject this act since the 90s.
If everyone making AI is screaming up and down that we are in an AI arms race creating dangerous entities that will determine the fate of the world is the government just supposed to ignore them?
No. But it could be done in accordance with the rule of law and commitment to equal access rather than an ad hoc approach that creates the impression of corruption and picking winners.
None of those things are knowledge. I think theres something specific around limiting access to knowledge and capabilities that makes this feel insidious.
Information is covered by ITAR, so that's not new. You can illegally export information about an ITAR covered item by just allowing a foreign national the potential to see an item. They don't even have to prove the foreign national actually did see it.
"Malboro cigarettes may once again be sold, but Newport remains banned for everyone except large purchasers that have paid the appropriate bri... fees."
He is not arguing that. He is arguing that these programs obviously aren't conscious, and that it's dangerous to tilt the weights toward emulating conscious beings for reasons outlined in the essay.
reply