Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases
It's one thing to be able to scale database compute/storage; it's another thing to be able to partition it. It's extremely common for bad queries/access patterns to cause noisy-neighbor effects on other simultaneous accesses to the database, to the extreme of knocking the whole database over with timeouts/OOMs/etc.
Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.
Iceberg & Delta Lake do partition the data. You can add as many DuckDB servers processing them as you want, each app can have as many as you need. You pay Amazon for all the I/O and let them worry about it scaling it.
I'm not saying there is no point of contention - depending upon your write patterns you have practical limits, but for reading data that is already stored you pretty much just pay more to do more and the rate of that cost is close to linear.
I'm sure there is a scale at which that pattern breaks, and I'm sure 99.9% of people reading this will never operate a system at that scale.
Iceberg addresses the hot s3 key prefixes. It places entropy in the prefix path per object so reads/writes are randomized over the bucket key prefix space.
It’s much less of an issue. Also, 99%+ of Aws customers likely don’t have enough QPS to hit the per prefix limits anyway
Most software developers now are absolutely ignorant of performance concerns. Just throw more compute at it until it works, and someone else will pay the AWS bill.
I have AI confirm the logic works as expected, but review for system design.
Often in both web/backend I’ve found AI to produce overly duplicative code, or have aspects that could be hard to maintain. Generally less due to the AI, and more because of the prompt itself.
That and even if you’re going to AI slop it up, I’d still demand it be broken up into 1-2k LOC chunks or per meaningful “thing”. This also lets us gradually ramp the change to confirm it actually works earlier on
And that is exactly why AI marketing touts ability to replace programmers, when in reality it is much better suited to replace middle management between client and programmers...
It’s faster than trino and spark for 99% of your use cases. Only once the datasets reach the 100GiB to TiB range is trino/spark faster. Compute is scaling vertically faster than Data Eng/Analytics dataset growth, over time DuckDB will be the better choice.
But greater turnover only occurs when people don’t want to stay.
It’s my observation a high-turnover business is often good for nobody. There’s more spent in retraining etc than if you just paid halfway-competent people properly instead of literally rolling the dice every year or more.
Yeah. Itd be ideal if companies could adjust their existing employees wages fairly. This alternative of people moving frequently creates a high turnover org which isnt efficient. But if employers dont compensate their existing employees fairly then high turnover is unfortunately the only way to go
I have zero respect for people >50 especially any in official policy roles. Zero fucks for anyone but themselves this whole time; ignored reality just like religious nutters and presumed political dogma would be on their side
Jokes on them; Millennials are even more convinced it all just goes black with death, fewer young people going into elder care jobs, population decline crushing those jobs... GenX can enjoy hobbling to their toilet unassisted with bed sores and gout. Fuck them too then
I dunno, the Ute and Navajo folks I live near don't agree. A lot of the indigenous folks I know think we are living in a literal apocalypse.
Historically the way white folks have squared that circle has been to say that those folks aren't really people and simply don't know the joys of Christian city dwelling.
And statistically worse in many ways. Statistics are just euphemism for history, prior observation; stats exist for both categories. We can write and carry on about wars we won and ignore Vietnam with stats just as we do English. It's just a language that lets us ignore physical reality like religious catechism; same biological ability to delude ourselves.
There's the whole ignoring externalities of capitalism and reality itself for toxic positivity thing.
Sure is great though watching where I grew up slowly become engulfed in wildfire smoke; something that never happened until 10 years ago.
I don't think I'd say rarely. Companies rarely allocate the design resources to produce that, but the companies that do are typically much larger, so the actual number of individual developers that get rich mocks is probably closer to 40-50%.
I’d say rarely in the sense that out of the 500 times (not really that much over 30 years) I’ve built similar things, I got UX mockups this detailed maybe 10% pf the time.
reply