It's pretty interesting to watch an LLM drive ngspice. ... clearly an area that could use more training, but good enough to make it less annoying for me to use.
No it doesn't. TODAY you can run with good performance a local LLM entirely on CPU that can do translation, summaries, search assistance, and all the things mentioned here.
On a 8GB system with a SSD it might be a bit slow due to having to keep experts on disk, but perfectly usable. On fast systems it may well be faster than the network round trip for many tasks.
> No it doesn't. TODAY you can run with good performance a local LLM entirely on CPU that can do translation, summaries, search assistance, and all the things mentioned here.
What specific LLM and how much RAM does it take up to load? How does it run on somebody's 8GB RAM $600 laptop they bought from Best Buy? What pp/s and token/s rate on that hardware?
How much ram has a fair amount of flexibility since MoE can be kept on flash and swapped in at a performance cost, and depending on how small a context you can use.
Well the linked thread mention 14700KF which is a 400$ processor and RTX 2070.
Beside its mentioned 5 or 6 gb of ram.
People run Firefox on pentium with less than 8gb of ram, just 4 or 5 years ago 4gb computer were still extremely common. Even Microsoft sold its surface devices with 4gb of ram, I was dumb enough to buy one.
To give a counterpoint, you could disable it on those low end devices.
But then people would complain about the size, like what happened with Google Chrome embedded llm.
I think there wasn’t any bad choice, mistral was the safest.
My post was an earlier post in this thread. I give figures for pure cpu, not using any GPU. The RTX 2070 figure is a separate figure (with rather insane speeds for that model!). You could contribute to the discussion by grabbing some numbers on lower power hardware if you have some available.
Though low memory devices like 4-8gb probably does need specialized inference to give fair numbers.
> you could disable it on those low end devices
it's not like it does anything if you don't ask for it-- and good thing, because the privacy invasion would be all the worse if it did!
> mistral was the safest
Sending the user's confidential information to third parties while falsely suggesting that it is private can cause them irrecoverable harm.
An alternative is that a feature is slow for some users on slow hardware, and there is a setting to make it faster at the expense of privacy and security that they can switch on. The large body of users that find the default config fast enough have no reason to flip the switch.
> But then people would complain about the size, like what happened with Google Chrome embedded llm.
People or astroturf accounts? :P but having read some of the commentary on it there was some pretty weird takes, like generalized complaints about AI and people thinking google was using their computer to serve other people. Google's business model generally prevents correctly marketing the functionality in any case: it's not like they're going to make a proper pitch for how important it is for your privacy when the rest of their business is centered on hoovering everything up. Mozilla is not so constrained :P
Another element of these "claimed to be private but can't keep its promises" is how caustic it is to the trust we have for our friends, family, and business associates.
People here can be privacy aware and well informed and avoid these data slurps-- perhaps go dig up the hidden settings to completely disable it or add firewalls so an errant keypress won't upload your browsing history. That's good.
But anything I share with another person or put on some webpage for another-- perhaps highly trusted person like a doctor or lawyer-- is exposed to them uploading it perhaps completely unwittingly (e.g. they fell for the exaggerated privacy claims) or due to an innocent misclick.
Normalizing privacy fails like this undermines the ability of even the most informed and aware people to opt-out.
I have never felt anything about gambling. Most boring thing ever. I don't feel anything about investment, and wrote automation to track mine because it's all so boring.
I wouldn't tempt fate to try to expose myself more, but it doesn't seem like I contain whatever is required to form these addictions.
Has it been studied if some people are just much more vulnerable than others, and are there predictive factors?
That's because parkisons drugs can upregulate a certain protein that is implicated in all forms of addiction that feature that kind of "seeking" behavior and modification of the reward system!
We found that if you give an addicted rat a chemical which suppresses the expression of this gene, it will improve their addiction situation, reduce seeking behaviors, etc.
I find the "normal" forms of gambling very boring: going to a casino, sports betting, lottery tickets, prediction markets, etc. I just assume I'll lose money because that's what tends to happen, so I have no interest in playing. As a result I thought I was a higher life-form, immune to gambling.
Then I played some mobile JRPGs that had gacha (gambling) elements. New fancy unit (big damage numbers or skimpy clothing or both) comes out every week and you get them by spending scarce resources on slot machine-like animations. Somehow I fell for this, despite knowing all about how gachas work, spending hundreds of dollars over the course of a few years. Now I realize I'm not a superior life-form. I just needed to be gifted a few wins of something that mattered to me, and then I'm throwing away money on pull after pull just like a casino gambling addict.
I like alcohol and trying new things, whether that be food or languages. I get nothing from gambling. Not all stimuli affect everyone equally. You’d need something approaching a unified theory of consciousness to adequately explain this finely.
I feel the same way - it's just not fun for me. I like playing poker with my friends, but I like the social aspect of it - putting down $20 on the table is no different than buying a few beers at a bar, and I don't care if I come out of the game with more or less money.
Going to a casino, which I've done a few times, is just wildly disinteresting.
But I know I'm prone to other sorts of addictions, so I guess gambling is somehow different.
I think that I would get more enjoyment dropping $20 in the parking lot at a casino and seeing somebody excited to find $20 than going inside to gamble it away.
Have you tried Ling-3.0-tiny? It runs fine on CPU-- on a 14700KF gets 40tg/s and 250pp/s and on a ordinary gpu (RTX 4070) does over 200tg/s with no MTP and 7185pp/s. (my figures are Q8, though presumably a good Q4 would be faster)
It's certainly not as capable as something that needs a high memory gpu for quick performance, but I was quite impressed with it for what it is.
(and fwiw, I had it translate your last paragraph to German, then used google translate back to english: "I wish they had handled this clearly and transparently via an opt-in mechanism—not enabled by default—that explains what Mistral is (not a major American cloud company, but a relatively small French startup) and that your prompts and LLM activities are sent to their servers. I also wish there were documentation explaining how the data is handled and stored in a way that inspires trust.").
How much RAM does it take up in total? I'll have to give that a try on one of my test systems. Looking at a somewhat randomly chose GGUF quantization of it, looks like just under 5GB on disk in Q4, so RAM usage somewhere around 5-6GB?
it's an extremely sparse MOE, so there is some odds of acceptable performance using a smaller in-memory cache and the rest on flash. ... I don't have a setup to test that right now.
(Of course, if translation is all you want much smaller models will work. Ling-tiny can do summarization, dom manipulation, scripting, etc. too).
Cloud inferrance is not private: These providers, even if they don't get hacked, have bribed employees, or outright lie, cannot and will not resist a subpoena or similar.
Mozilla is acting unethically by obfuscating what's on offer here and overstating the privacy and security properties that can be achieved.
It's doubly unfortunate because simple translations are completely reasonable to do with cpu inference.
The breach is bad no doubt-- but this information was already readily available to bad actors e.g. via Lexis Nexis. Practically all states sell DL and registration information to information brokers, and the remaining ones require you to obtain auto insurance, and the insurers all sell the information.
Many people pretend this isn't happening because of the "The Drivers Privacy Protection Act" but the DPPA is paper thin protection at best as it has a long list of permitted uses which anyone can just lie about (and are you worried about threats from parties so honest they're unable to lie?). Not that they usually have to lie given that the permitted uses include "For use by licensed private investigation agencies" and "For the bulk distribution of surveys, marketing materials, or solicitations"... In practice this just means accessing the information costs a little money and requires someone check a "this is for a permitted purpose" checkbox. The biggest impact is that it causes abusers of the information to be circumspect about their sources, which helps maintain the data-harvesting status quo.
(Guess what: the same databases also have ALPR gathered pictures of your car at whatever locations its been in public view... stores, your home, your mistresses home... Makes flock (YC S17) look pretty mild by comparison. The fundamental sin is requiring ID without also making it a crime for anyone but the owner and issuer to posses someone elses ID information.)
In some sense the IDScan breach may (ultimately) improve our privacy and security because it will break people out of the FALSE belief that this information is private, or that it can be protected by anything short of restricting its collection in the first place.
Right on! It's always frustrating reading articles framed in terms of "security breaches" and "dark web", invariably doing hand waving at unspecified harm, when the real threat actor for pretty much everybody is the "above board" surveillance industry. Random people who buy this info outside the law can't really hurt me - it's not like I'm a witch and my DL# is my "true (system-given) name" and I disappear when they say it or something. Rather the parties who can hurt me are the ones who pretend knowledge of this semi-public information is an authentication system, and then hassle me with legal nastygrams when they get defrauded. Or who keep comprehensive dossiers on my behavior to unaccountably sort me into corporate-defined boxes so they can better extract my wealth and otherwise form anti-competitive arrangements against me. These actual attackers operate mostly according to the (very broken) law, and they are what needs fixing. Not just scaremongering when some bogeyman "wrong people" get a small taste of the exact same information.
Ia also wonder just how hard it is to figure out the info someone’s drivers license from scratch. My license has:
- license number
- license class
- license issue date
- license expiration date (birth day and month in my state)
- birth date
- eye color
- sex
- height
I think if I was targeting you specifically, probably not, most of these details would be apparent upon seeing you. But, would you be comfortable if I had these details on a card with my photograph? Highlights a risk around ease of credit applications etc.
"We know our product is an unlawful violation of federal wiretap law since it will make zero-party-consent recordings, not that we care about violating the law at Apple because we know we can out spend anyone who would litigate against us, but we like to keep up appearances."
Leaders in the MIRI/EA cult-o-sphere have advocated mass murder via nuclear weapons against towns that don't prevent people from performing too many multiplication operations. Why is anyone surprised that they'd engage in deceptive false flagging operations?
This is a lie, so you are either repeating a lie from someone else or lying yourself.
The people you are talking about have not advocated mass murder via nuclear weapons. The idea was precisely targeted non-nuclear strikes against only the servers themselves. (Which, would have plenty of forewarning to allow people to leave the building.)
The claim that they advocated for the use of nuclear weapons is a lie.
I am speaking from direct personal experience: Yud's cult is inherently violent. By having defined their priorities as essential to the survival of any life on earth they have pre-justified taking any action 'necessary', no matter how cruel or extreme... even actions that they admit have only a small change of helping according to their own assessments because they've defined their doom as so unimaginably bad, effectively infinitely bad. (much worse than mass-extinction, in fact)
But you don't have to take my word for what the mentally ill AI doomer cult believes, you can take it from their leader when he called on states to "Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."
Don't make excuses for omnicidal authoritarian cults and their sadist leadership.
reply