I think the Rust feature you’re looking for regarding recompiling the standard library is called “build-std”, that should be enough for you to search for it. (For similar reasons you also need that flag if you are trying to use Rust to build multithreaded wasm binaries, so it might come up for you!)
In my own journey of discovery I found https://cheats.rs/ very helpful, and in particular its "memory layout" section has visualizations. (No affiliation with the site, just a happy reader!)
[Ninja author here] Nice post, cool to see the deep dive! I also appreciate the details on how they produced their numbers.
As they observe, Ninja gets to be fast mostly by cheating: it avoids a lot of work by saying many things are just out of scope for Ninja to do, and that means it is a useful a target to race against. (Funny thing: when I wrote Ninja I was misremembering how fast an earlier build system was so I kept trying to make it faster. So don't treat it as a lower bound, I just made it up!)
I comment here to say I find the explanation for 'why' in this post unsatisfying. They mention three design decisions.
The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why. I might have misunderstood?
The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely. It's a very small amount of work: the post mentions 300 compiles, so maybe parsing 300 small text files?
The third is that they run the compiler up front an additional time to gather headers, which is strictly more work than Ninja. There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches. They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
Maybe it's just my own curiosity, I think this post would be better if it had a better explanation for the reason. I'm not disputing the result, I just think the result should make you suspicious that something else is going on, and you might learn something from that! You could for example explore whether it's the header dependency thing by profiling the Ninja invocation and seeing if it's waiting for CPU or waiting for tasks to execute.
(If I had to guess without looking at any of the involved code, I would predict it's something about how CMake generates the build, like it introduces serialization in a place where build2 is parallel, or it adds some extra build steps like gathering the current git hash into a header file or something.)
[build2 author here] Thanks for the feedback! Some additional details:
> The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why.
Fair enough. The point I was making is that if you want to compete with Ninja, you cannot leave any potential performance gains on the table.
> The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely.
We are talking about ~2% performance difference here. Parallelizing even a small amount of work across 24 threads rather that doing it serially saving a percent or two feels plausible to me.
> There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches.
It fits into the system's file cache unless there is memory pressure, like one would expect from having 24 C++ compiler jobs running in parallel. We actually measured this in isolation (with more detailed results in the linked article) and it has a measurable effect.
> They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
I measured this, it costs 70ms or ~2% of the overall time.
> It fits into the system's file cache unless there is memory pressure, like one would expect from having 24 C++ compiler jobs running in parallel. We actually measured this in isolation (with more detailed results in the linked article) and it has a measurable effect.
300 TUs is not much. If they build in 3 seconds then they are trivial (small). If the machine is 24-thread (I assume some sort of heterogeneous 12-core), how little RAM does the machine have for the kernel to start evicting page cache during the build?
> > They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
I measured this, it costs 70ms or ~2% of the overall time.
Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?
> Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?
Yes, that would be nice, but the tricky question is can any of this information change without the compiler binary mtime changing? First off, GCC's gcc/g++ binaries are drivers and are not what does the actual compilation, it's private cc1/cc1plus binaries that do the job. Can one of these change but not the driver? I think it's plausible (some package manager optimization where the file is not touched if it hasn't changed). So at a minimum we would need to discover where those are located (probably by invoking gcc/g++) and checking them as well. Could there be something else? Who knows. We value speed very much but we value correctness even more.
I think a more fruitful direction to explore is to improve GCC itself to dump all this information in a single invocation and in a machine-readable format (JSON). I think if we go from 70ms to 14ms (and perhaps even lower because this special GCC mode could conceivably do things faster than how we do it now), it would be good enough.
Simple CMake projects using the Ninja generator are very efficient, unless you create generated files.
And if they are in their own targets, they are not really an issue (they would serialize everything that depends on them as you'd expect), but if you have them in a library grouped with other files to compile, then the whole library compilation is serialized.
And obviously worse if you also have to build the generator for the generated files, but that's not a big surprise, you can observe that in full builds of Chromium or its libraries too waiting for protoc if you crank the parallelization a lot.
Yes, I don’t remember the details but vaguely remember that CMake tends to group things together that could in principle be made more parallel, as you mention with generates files in a library. On the other hand if build2 makes it easier for authors to express these kinds of patterns without the serialization then I count that as a win for build2!
I feel like make/ninja suffer from the same output modstamp > input modstamps to rebuild. Really a build system should track modstmaps on all inputs (apps included), and the rebuild on not-equal.
Reading about manifests immediately reminded be of Clearcase/Clearmake. My memory may be not correct after all the years passed, but configuration records for derived objects contained the same information as proposed manifests + some more metadata about source (versions, at least).
If I remember correctly SCons allows to implement a custom "checker" (or whatever it is called there) to compare files based on any criteria that are useful, not just timestamps. But this is about the only visible advantage of SCons over Make, at least for our case.
Why not just embrace Content Based Addressing and the Bazel action cache? Not necessarily adopt the RBE protos, but you could have on-disk cache that are similar or even identical.
Nowadays, I think a build tool that doesn't natively support distributed caching (and possibly remote execution) is a weird choice. And I don't think that spawning layers of processes allows for good parallelization as you don't know if an action is going to be network bound or compute bound and the job slot is then spent. So you either oversubscribe or undersubscribe.
Sadly, I think things like distcc or ccache have mostly gone out of mode. Vertically scaling (multi core, multi thread, multi TB RAM build machines) and simple artifact caching have won for now. Artifact caching with dependency tracking in the midst of job scheduling was solved “good enough” by OS based mechanisms. Having used and managed the basics of a build farm, removing the need for networking, build farm management and build job coordination is a huge burden relief when your source tree doesn’t require it.
Part of the motivation to use processes is because their structure helps to keep the job generic, uni
I stumbled upon the very experimental nix-ninja [0] recently.
They seem to create a dynamic nix derivation per compilation unit, which would be very similar to what you describe as manifests in your post as it also creates a hash of all inputs.
Would be interesting to here your opinion on that approach
I really like the ideas you have in n2. Hashing the entire build string is a great idea. And it really is as simple as just having all the timestamps of the inputs in the hash. And you have the output timestamp as a quick test.
Always glad to see authors on here. Thanks for your insight!
Since they had to rewrite the build file for their program I also assume that something is missing. Didn’t see any mention of verifying that.
I also really didn’t like their denigrating tone. It totally turns me off trying build2, because it seems they don’t understand the point of separating build stages like environment setup (getting dependencies), configure, native build, cross build, packaging. I am a very happy ninja user instead of a batteries-included solution because it does its one job well and can be used very flexibly. I personally detest cmake, so I use nix + own configure script + ninja.
The blog is also wrong about ninja being unable to call configure, but I intentionally don’t want that (I want the build stages to communicate in one direction for sanity).
> I want the build stages to communicate in one direction for sanity
That's easy to do if you control the whole pipeline and can integrate all the features together, not so much with the CMake model unfortunately. I think it would be nice if CMake had Ninja integrated as a library, it could lead to some nice optimizations later.
If you imagine Google's job is to present useful information, these blogs that are maximizing cash while simulating usefulness are exactly the sorts of things I would hope Google to want to filter out.
(I don't think Google's often capricious ranking changes really succeed at this, but the outcomes in this post seems like something hypothetically good?)
I tried doing that at first. I kept running into edge cases that made the whole thing fall to ribbons. I gave up and am just falling back to what I know works: compiling the WASM to JS.
One thing I sometimes think about when I think about text layout problems is how the text we use also has a bunch of complexities that we can take for granted.
Think of variable width characters and kerning and ligatures and hyphenation and justification. Imagine computers had been won by a CJK language, which have none of these problems. You could imagine a similar article about how exotic and difficult English layout is.
Both Latin and Chinese have been modified by the technology used to write them.
When carved in stone the lines are much straighter. When written with brush or pen they became semi-cursive. When printing was introduced, they became grid-like and regular.
What westerners who are passingly familiar would think of as the standard Chinese typeface - the strict square grid with straight-line characters - arises in part from printing technology. Easy to carve that into wood blocks, and easy to line up the slots into a grid.
Latin was similarly morphed to fit into the realities of printing in the 1500s. And is still being morphed. Notice how numbers 123... are in-line and at the same height as the letters. That's a very modern convention, typewriter and computer influence on our orthography. Traditionally digits were more likely to appear as subscript, off-centre.
what selective pressures against oldstyle numerals with ascenders/descenders existed that wouldn't have equally applied to letterforms with those same features?
(aha i have found the answer to my own question: miniaturization for fractions in phototypesetting)
Not really. The selective pressure really comes well before that: Tabular presentation of numbers, whether that was log/trig tables or railroad time tables, there was a preference for uniform-width and regular height characters for those contexts (this is also why there is a number-width parameter in TT typography to enable a designer to let digits be variable-width in text but still allow tabular setting if desired).
the other part is that numbers and symbols were very much not the priority. The printing press was for books, magazines etc. math remained hand written until the computer
Nope, not at all. Monotype had a special system for doing math in hot metal typesetting. With handset type it was possible, but very time-consuming. You can find typeset mathematics going back centuries before the computer. There were also (somewhat impractical) systems for setting music with metal type although engraving was more common because of the interactions of lines and symbols.
Conversely, English has a joined form(cursive) that is nearly dead because mechanical text assistance devices (first typewriters, now computers) work much better with the block form. While sad in a cultural loss sort of way the joined form only really makes sense when the text is hand written.
I am not familiar with the history of Arabic typography, but I sort of assume there was an archaic block form and their current joined form is the result of many centuries of encoding hand writing practice. advanced enough that falling back to a block form is impossible with the side effect of making simple mechanical text formatting also impossible.
As for Chinese derived characters. we currently are able to jam them awkwardly into our alphabet optimized structures(one code per character) but I wonder if a Chinese native encoding would look different. Would it make sense to try and represent the sub-characters present in each Chinese character in the encoding? I suspect not, Chinese works, but it also does not appear amiable to simple mechanical assistance.
There's the https://en.wikipedia.org/wiki/Ideographic_Description_Charac... that kind of does that. The problem is that there's character divergence (see all the brouhaha about Unicode Han unification), so there needs to be something else to select variants too.
As a reference, I don't believe any of the pre-Unicode CJK&c encodings attempted that.
Another wrinkle with Arabic is linguistic conservatism. Due to Islamism and the idea that Arabic is the language of of God (the Quran was written in Arabic by the supposedly illiterate prophet), Arabic has lagged behind other languages in terms of innovation.
Hebrew is a closely related semitic language that simply adopted a block and cursive form. It has also been greatly simplified and friendlier towards loanwords, which has made it far easier to learn.
Muslims don’t believe Arabic is the language of God. They believe that the Quran was revealed in Arabic (true). Thinking the creator of the heavens and earth only speaks one language is absurd. It also kind of implies that Muslims believe in a superiority of Arabs which is also not true.
Weird to say Arabic hasn’t innovated or evolved considering the wild variety of dialects spoken in the modern world.
Conflating the language with the script is also bizarre. In terms of adapting Arabic to technology, look into romanized Arabic which was used before Unicode was common.
I didn't write "God only speaks Arabic" in Islam. That's your intepretation of my post. All I meant was that Arabic has special status in Islam.
> Weird to say Arabic hasn’t innovated or evolved considering the wild variety of dialects spoken in the modern world.
I didn't say Arabic has not innovated or evolved; only that it "has lagged behind other languages in terms of innovation". My belief is that that is due to linguistic conservatism, and linked to Islamism (or, at minimum, the centrality of Islam in Arab culture). Also related to this is the existence of Fusha, its place in Arab culture, and its branding as "modern standard Arabic".
I didn't conflate anything. While a script and a language are not the same, it's not a coincidence that Arabic is often written today in a script that is very close to Quranic script. And -- to really kick the hornet's nest -- it's also not a coincidence that there have been so few outstanding Arab writers (in Arabic) in the past 100 years. One novelist and a couple poets.
> And -- to really kick the hornet's nest -- it's also not a coincidence that there have been so few outstanding Arab writers (in Arabic) in the past 100 years. One novelist and a couple poets.
Now, reading that point one might ask the question if writing has been properly funded, or if the priority of cultural funding in the Arab world has been lower than, say, the funding of architecture and other forms of art. And on top of that, I'd also have a serious look at the market size, especially when compared with English-language writing.
With all due respect, your comment comes off as a bit ignorant and rude. A few points:
Firstly, the Qur'an wasn't written by the Prophet, he would dictate it and it would be written by his scribes.
Secondly, it's hard to argue that Islam has had a negative effect on Arabic or caused it to lag behind. In fact, it's easy to argue for the opposite. It's a historical fact that the Arabic language developed and proliferated rapidly due to the rise and spread of Islam. This is when its script and grammar were standardized, and when more and more works started being composed. And shortly thereafter the Islamic Golden Age began.
I don't have any issue with Hebrew, and maybe it is easier to learn. But this is because it was a dead language which was revived, resulting in a simplified language. Almost every other major language on Earth will have the same amount of "innovation" as Arabic. In fact, Arabic has many colloquial dialects which are used in day to day conversations, and these do consist of a simplified version with many loanwords. So I really don't know what you mean by a lack of innovation.
I don’t think anybody said that Arabic has suffered a complete standstill, and it has doubtlessly evolved significantly.
But if you compare it with basically any other major language, it’s clearly much, much more conservative. If you are a native English speaker, understanding English from 1,000 years ago is like learning a completely different language. If you are a native speaker of Italian, you cannot understand a text in Latin without significant training. This is true for all European languages other than Icelandic.
Chinese is pretty similar, even though the written language is slightly more stable.
So in comparison, Arabic is incredibly conservative.
There is no one "Arabic". Yes, formal modern Arabic (fusha) is based on (but not identical to) the classical Arabic of the Quran, but nobody speaks this in real life. The actual Arabics are the 20-odd spoken languages, many of which are effectively different languages at this point:
A rough equivalent in both time and space is how the Vatican continues to use Latin, but the rest of the Roman Empire has splintered into Italian, French, Spanish, Romanian, etc.
They speak it on tv and it's written in newpapers. They learn it in schools. Educated Arabs code switch into Fusha all the time. Islamist leaders (e.g. Nasrallah) speak Fusha in their broadcast speeches.
It's also pretty hard for foreigners to learn an ammiyya (outside of immersion). "Studying Arabic" almost always means Fusha.
I agree with you that "the actual Arabics are the 20-odd spoken languages". In a healhier culture, Fusha wouldn't exist or would have the same cultural place as Latin in the Western world.
There is no unjoined form of Arabic. The Arabic script became Arabic when Nabataean script started developing joined letter forms. Unjoined Nabatean is as foreign to Arabic as Phoenician is to Greek.
Looking at dictionaries and printing presses from China before the invention of computers reveals that they probably would have done something similar to ascii, just with more bits to encompass all the characters.
Chinese apparently was originally always written vertically top-to-bottom. (And then columns would be right-to-left.) Modern Chinese just rotates everything except the characters themselves 90 degrees to the Latin order.
I also read that a few Chinese texts only make sense in vertical order: one had a pun where the characters read one way as separated characters, but as stacked was also a single character pun for something like a "crumbly cookie".
It's a good tool! That said, I also recommend looking at European (and other) nations from a subnational lens as well.
The North-South divide in Italy, the FRG/GDR divide in Germany, Northeast and Southern versus Central France, and various other representations of spatial inequality exist within Europe as well.
The reality is a Parisian, Londonian, and New Yorker have much more overlap with each other than they do with their own compatriots, yet it is this class that is overrepresented in any discourse on social and traditional media.
Thanks, I'd love to add them! Do you have a good source for this data? I did a quick look at the site you linked above and I'm not sure whether it has numbers for GDP or landmass for these regions.
This is my second emulator, and in my first I picked a name more like that and regretted it. A thing I now appreciate about emulators is that it's common to increase scope -- like this one already supports non-wasm output, and I am tinkering with adding support for DOS executables as well, which means the name 'win2wasm' would already become obsolete!
reply