I conduct interviews pretty often for various levels. Interviews seems so broken from both sides. Hard to find jobs that aren’t ghost jobs. Hard to find candidates that won’t cheat using AI and actually know their stuff. Just feels like AI has made hiring in tech even more inefficient which says a lot.
if its remote initially, closed eye interviews where you just have a discussion might be easiest way. Pretty easy to tell if someone is bullshitting if they can't discuss topics that likely should excite them somewhat, or at the very least topics they've been around for years.
Feel like any test nowadays should INCLUDE ai, but you likely just want insight into how they are getting from A to B. I don't wanna assume what you mean by cheating with AI, could be as simple as answering live questions with cheating software or whatever, which is obviously a huge issue. I wanna respond and say it should be expected for any take home type stuff, and be rolled into the expectations if anything, but that could be way off of what you mean.
both sides of the entire process is just a nightmare now, one side has all this noise to sift through, and the other side is pressured to mass apply because of all the competing noise they can't be seen through. I was listening to something about AI generated responses, and they seemed to have no empathy for the people who gave human responses, but used AI to give them an overview of the company and what it does for the question "why do you want to work at x?", and sorta mocked the person because they can tell they asked for a generic description of what the company does.
There has to be some give and take, initial applications if you are spending hours researching a single company, you are setting yourself up to be absurdly let down if that same company sends you an automated rejection within a day. A reasonable person will only do that so many times, before just giving up and waiting for the other side to be the first mover instead.
I've been asked to do take homes for different companies recently. Each time I was very confused whether using AI is acceptable, desirable, required or unacceptable. One time I asked just in time, the other time I wasn't sure if even using Google was okay.
I mean in University there were tests where you were allowed to bring a cheat sheet, or even a bag full of books. So called bag tests. And people were still able to fail. Although I never had the latter in my curriculum.
I think it's necessary that teams for which people are hired for brainstorm up-to-date tests. Just recycling the same challenge from 3 years ago isn't good enough.
If I were in a meeting with a current employee, and discovered that they were reading an LLM script at me instead of considering and responding to my questions, I would definitely tell them and their manager that this is unacceptable.
Most technical interviews aren't designed to emulate what the job will be like, so what you would or wouldn't have access to on the job isn't really relevant. The goal is to evaluate the human being applying for the job. An interview that effectively emulates the job would really have to be done over at least a week, and most good candidates can't or won't make time for that.
Also heard about this! But the point of our benchmark was to evaluate frontier LLMs with vision out-of-the-box, which we wouldn't expect to have been specifically trained on driving real cars. The fact that they can do anything at all (even in an open lot cone course, at low speeds) is pretty impressive. I'm sure Qwen Drive and models specifically trained for driving would do even better.
All these new models are such tease for us folks with 128GB of shared memory. Buying another unit now to expand to 256GB is a mortgage payment but it’s getting tempting…
I can imagine someone building a device that uses NAND flash or similar tech but with a very different controller that is optimized for streaming the data out with a predetermined access pattern at very high speed.
Flash, like pretty much every solid state storage technology, can scale its output bandwidth to ridiculous levels limited pretty much only by the readout circuitry. There may be a price to pay in power consumption, though.
You could always stream from SSD storage. Especially effective if you get a cheap old-gen HEDT with lots of PCIe slots to add NVMe storage to and reasonable overall PCIe bandwidth.
That nearly certainly boots you to secs-per-tok land (as opposed to tok/s). Plausible if you are willing to wait hours to days for responses for simple testing, but not (debatably) "usable".
Demonstrated inference speed for the biggest open-weight models is indeed around 1 token per second, with weights read from SSDs.
However, this is for relatively low-end systems, with a couple of fast SSDs providing around 20 GB/s throughput (or with a few more, but connected through relatively slow Thunderbolt, for a similar total throughput).
If you use 16-lane PCIe add-on cards with 4 M.2 slots for SSDs and a total throughput of 50 to 60 GB/s, you can quadruple the previous speed in a desktop PC where you use the GPU PCIe slot for SSDs (a fast CPU, e.g. an AMD 9950X, would be alone fast enough for inference limited by SSD throughput, so a discrete GPU would not be required).
If you have a server/workstation motherboard, e.g. with 6 16-lane PCIe slots, you might gain another factor of 4 in the inference speed, so one might reach around 15 tokens per second for a very big model, but the cost would also be high, with so many SSDs, even if at that number of SSDs each SSD could be the smallest that can be found with a PCIe 5.0 interface.
You can definitely offload n-gram embeddings to storage; they're very sparsely used (only a few KB fetched per token) so this is quite effective. Loading to DRAM only becomes necessary if they are a bottleneck to overall performance (which might happen if you're doing very wide batches and everything else uses super fast VRAM/HBM).
I was looking at the qwen-next-flash, and the weights would fill my OEM Spark on their own, before the n-gram. I'm unclear if offloading to disk can work here, is that what you are implying is possible?!
I'd recommend pointing your agent at it (after installing sparkrun), and asking it to research the absolute latest in TP=1 Flash-Next - mine grabbed particular vLLM nightlies and mods to improve performance, and it was well worth it.
I have a quirky vLLM on k8s on 2x OEM sparks setup with 9 models available to me. I'm not keen to run nightly vLLM, too many issues with it in the past. Going the qwen-next path means displacing things I use daily :/
I have a watchful eye on the diffusion ~ Jev/Kev PR
For what it's worth, Flash Next outperforms every other model that is available to us on the GB10 in all of my testing; though if you have two sparks then the TP=2 version is even better and easier (I don't think you'll need the nightly for that at all, just use the recipe)
Nah, I’m streaming ngrams off NVMe on my Spark-alike right now. Works surprisingly well (except for when I accidentally bottlenecked it through my NAS)
GB10 boxes have way more compute than they have memory bandwidth, which nicely fits medium sized MoE models with speculative execution (MTP, DSpark/DFlash, etc)
Qwen 3.8 Flash Next (what I'm running basically entirely now) sees 30 / 35.0 / 45 tk/s for prose, analysis and code respectively for actual use (not short context benchmarking) with Pi. Thinking blocks are ~35tk/s or so.
The GB10 having so much compute is great for prefill too, 2000-3000/s for 14k to 64k token prompts (cold cache too) in the quick benchmark I did. 3500tk/s for warm cache which is nice :)
When I accidentally streamed my ngrams over the 2.5Gb/s network, it cut all the throughput down in half basically. Especially notable for the time-to-first-token, which is what clued me in that I'd messed up somehow!
For Qwen 3.8 27B, I got it up to a consistent 20tk-25tk/s but 27B thinks so much that it was honestly too painful: Flash Next is as smart, as useful, but much faster for real agentic dev usage IMO
Laguna S 2.1 saw similar numbers to Flash Next if I remember right, but their latest updates means it doesn't quite fit a GB10 128GB anymore at full context which is a shame.
Note: these are all NVFP4 quants (usually a dynamic one where some tensor layers are left at full precision though)
I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for me, like sub 10 TG.
I've found the opposite on my R9700 (n-max=7, no other speculative decoding like ngram-mod, which I found slows it down). I think it depends whether your workload and system are bandwidth limited or compute limited. I see draft acceptance around 0.55, so 0.55 * 7 = 3.8 tokens per pass, which on my bandwidth-limited card takes me from 30tps to a peak of 80tps on llama.cpp (MTP peaked at ~65tps). I'm also running a Qwen fine tune whose speculative execution is better than the base model.
Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan.
Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
I've also got a strix halo box. 30tok/s would we usable, but I wonder how it compares to the Qwen3.8-Flash-next - I get about 40tok/s running that on Halogen and it feels like using Claude 4.6.
Vulkan, I have never used ROCm on it but have been debating since the latest big update. How is your prefill? Do you hit over 1K? If it’s 1000K prefill, and 40 TG, I might have to try this over the weekend. Also, can you fit 128K without offload the ngram onto SSD?
I have not measured pre-fill, but it's said to be around 1000.
It feels very snappy and unlike my experience with running 27B models the performance stays pretty flat even as the context increases. Unfortunately, we don't know how Halogen is doing this because it's closed source, but I think AMD should offer that guy some $$$ because he's done a lot of good work getting more performance out of Strix Halo.
> audio-visual performance close to Gemini 3.8 Flash and overall audio performance that exceeds Gemini 3.8 Flash
Wow crazy if true. I think Gemini's audio capability and multi language was the "selling point" for a lot of people. Other capability also matches or exceeds 3.8 Flash.
They also made a new harness but github link seems to 404.
I tried adding my own endpoint, but the save button was still greyed out so it did not work. Maybe the feature is not fully implemented yet? In about:config, I could add the endpoint and api key, but not the model name.
Mark Gurman seems to get extraordinarily accurate and detailed leaks. I wonder how Apple got this leaky. Pretty much everything about the Duo was known months in advance.
Well, either corporate security got incompetent or they intentionally let select leaks exist to "test the market response". Established companies tend to get more risk averse overtime.
reply