Very interesting! I posted a vector and quaternion lib here a few weeks ago, and got great feedback on the state of SIMD on these things. I since have went on a deep dive, and implemented wrapper types in a similar way to this library. Used macros to keep repetition down. Split into three main sections:

  - Floating point primitives. Like this lib. Basically, copied `core::simd`'s API. Will delete this part once core::simd is stable. `f32x8`, `f64x4` types etc, with standard operator overloads, and utility methods like `splat`, `to_array` etc.

  - Vec and Quaternion analogs. Same idea, similar API. Vec3x8, Quaternionx8 etc.

  - Code to convert slices of floating point values, or non-SIMD vectors and quaternions to SIMD ones, including (partial) handling of accessing valid lanes in the last chunk.

I've incorporated these `x8` types into a WIP molecular dynamics application; relatively painless after setting up the infra. Would love to try `Vec3x16` etc, but 512-bit types aren't stable yet. But from Github activity on Rust, it sounds like this is right around the corner!

Of note, as others pointed out in the thread here I mentioned, the other vector etc libs are using the AoS approach, where a single f32x4 value etc is used to represent a Vec3 etc. While with this SoA approach, a `Vec3x8` is for performing operations on 8 Vec3s at once.

The article had interesting and surprising points on AVX-512 (Needed for f32x16, Vec3x16 etc). Not sure of the implications of exposing this in a public library is, i.e. might be a trap if the user isn't on one of the AMD Zen CPUs mentioned.

From a few examples, I seem to get 2-4x speedup from using the x8 intrinsics, over scalar (non-SIMD) operations.

Why do the x4/x8 types seem to be the default in rust?

A portable SIMD feature should encurage portable SIMD and not a specific vector register size.

This sounds like a great idea. I went with this approach because I'm new to SIMD, so I aped the most promising API (core::simd), extending it naturally.

I need to think through the consequences. It might involve feature gates, and/or an enum. So, for example, instead of:

  pub struct f32x8(__m256);

It might be this internally, with some method to auto-choose variant based on system capability?:

  pub enum f32_simd {
    X8(__m256),
    X16(__m512), 
  }

etc. Thoughts?

I’ve said it before and I’ll say it again: Rust feels like a Python developer’s idea of a high-performance computing language. It’s a great language for many kinds of applications — just not when you need to squeeze out every bit of performance from advanced hardware.

Even before getting into SIMD, try using Rust for concurrent, succinct, or external-memory data structures. It quickly becomes clear where the friction is.

Cargo is fantastic — clean, ergonomic, and a joy compared to many toolchains. But it’s much easier to keep things simple when you don’t have to support dozens of AVX-512 variants, AMX, SME, different CUDA generations, ROCm, or any of the other modern hardware capabilities.

Standardising SIMD in the standard library — in Rust or C++ — has always been a questionable idea. Most of these APIs cater to operations that compilers already auto-vectorize reasonably well, and they barely touch the recent capabilities of SIMD. Just consider how hard it is to build any meaningful abstraction over the predicate/register models across AVX-512, SVE, and RVV.

RVV aside, this should illustrate the point: https://www.modular.com/blog/understanding-simd-infinite-com...

I don't think Rust is particularly problematic here. As long as you don't want to do funky things like use immutable argument memory as temporary scratch space (with you restoring the values afterwards of course), all it means is some `unsafe`ing at worst, compared to C/C++. And there are some safe abstractions you can make over loads/stores (everything else being safe, even if not yet marked as such).

Do agree that a standard SIMD type is rather pointless, if not immediately, then in like 5 years. (and, seemingly, both Rust and C++ are like over 10 years behind on SIMD, so they're already out-of-date)

Maybe somewhat useful if you just want the simple ~8x speedup, and not squeeze out the last 1.4x or whatever, but autovectorization should be capable of covering a significant amount of such.

… except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers. For me the solution was to implement those manually, and even at a scale of just 2 libraries I’ve eneded up with somewhat different project layouts & dispatch mechanisms: https://github.com/ashvardanian/SimSIMD , https://github.com/ashvardanian/StringZilla

One big family not covered there, is sparse data-strictures and related algorithms. I’ve only started integrating scatter/gather in AVX-512 and SVE, and on synthetic benchmarks both look promising: https://github.com/ashvardanian/less_slow.cpp/releases/tag/v...

Those should probably unlock a much wider set of applications for SIMD, but designing libraries for those may benefit from yet another project structure.

except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers.

Counterexamples: Chromium's byte-level HTML scanning, several var-len bit packing codecs, and Gemma.cpp's matmul is mixed-precision (fp8->bf16->fp32->fp64). All written with the Highway general-purpose SIMD wrapper. Please revise your post or expand upon the structure/dispatch concern.

Interesting references! I remember that Gemma.cpp used Highway, but I haven't checked the others much.

Here is a puzzle, then. Let's say we are checking a single register of bytes for element-wise equality with another register of the same size. In AVX2, the output is another YMM register of 0xFF or 0x00 values. In AVX-512, for full ZMM-wide comparisons, it's a 64-bit mask in the K register.

I struggle to see a good way to abstract such things, even for two consecutive SIMD generations on x86.

A simple enough abstraction is to have the mask type have inputs of both element size and count; so for a ≤256-bit product of those you do the homogeneous-bit elements, and for 512-bit you do the packed-bit values. Conversion methods on the mask can convert those to either an explicit full vector, or a packed bit integer, as you need. (and clang can optimize out unnecessary conversions between the two)

Sure, that would make sense, but in AVX-512, there is also a comparison variant for 2x256-bit inputs that outputs a 32-bit mask and another one for 2x128 inputs and a 16-bit output mask. I would use those in different ways, occasionally mixing with the old AVX2 variant, depending on what I’m doing. I have tried to create a generalizable SIMD framework several times in the past, mostly in 2015-17, and still don’t have a good abstraction even between AVX2 and AVX-512.

You started with this... take:

Rust feels like a Python developer’s idea of a high-performance computing language. It’s a great language for many kinds of applications — just not when you need to squeeze out every bit of performance from advanced hardware.

And went on to say that Rust in particular is problematic for:

byte-level processing

It's particularly odd for you to say this given that the memchr Rust crate is just as fast as stringzilla for substring search. And is generally faster in cases where the needle is invariant, because stringzilla doesn't have APIs for amortizing searcher construction.

We've had a discussion about this before where I provided receipts[1] and we have not had a meeting of the minds on this point. The thing I'm trying to achieve here is to point out that your claims are contested and there is evidence that you're wrong. And so I'd caution readers to also in turn question your higher level claims about Rust being a "Python developer's idea of a high-performance computing language."

[1]: https://old.reddit.com/r/rust/comments/1ayngf6/memchr_vs_str...

Don't worry his comment is obvious clickbait (complete with a plug for a random blogpost written by him). People that matter (people that are actually writing simd for their day job) can immediately spot a poseur (especially relative to you and Raph).

To be clear, I don't support this. StringZilla is a real and useful project, and its performance is competitive with memchr. They aren't a poser.

(There are reasons to use StringZilla over memchr beyond performance. StringZilla provides a number of interesting string operations beyond just substring search. The memchr crate is far more specialized.)

Do agree that a standard SIMD type is rather pointless, if not immediately, then in like 5 years. (and, seemingly, both Rust and C++ are like over 10 years behind on SIMD, so they're already out-of-date)

What language would you consider to have cutting-edge SIMD support?

I've dipped my toes into SIMD with Rust, [1] on stable with platform-specific intrinsics (SSE2, AVX2, NEON). I would have liked to use stable `std::simd`. I learned that (particularly on AVX2) getting things into the right lanes efficiently is a pain. I would have liked to just use `simd_swizzle!` for that part, and mix that with intrinsics calls. My approach of writing a small C++ or unstable Rust program that does the swizzling and then copying the intrinsics operations it chose into my program's "source" code worked, but I prefer to not have a manual copy'n'paste step between compilation and assembly.

If there's something much better out there in another language, well, I'd be very interested to see it.

[1] I wrote this: https://github.com/infiniteathlete/pixelfmt/blob/main/docs/s...

Don't think any language has standardized SIMD that's particularly nice; Highway is probably a quite nice library on C++, though I haven't used it enough to get comfortable.

The thing I use for my projects is Singeli[1], a DSL specifically made for SIMD stuff (though it's capable of generally sanely doing abstractions over types/operations/loops; it's just a fancy code generator). Obligatory disclaimer that I'm one of the two people working on its design. It's far from a nice experience starting from nothing, but it's pretty nice for what I do.

CBQN's the main place it's used, can click around its source: https://github.com/dzaima/CBQN/tree/develop/src/singeli/src

Its goal isn't necessarily to unify architectures, but rather make it as easy as possible to make abstractions that do; as such its built-in includes for x86 don't have arbitrary shuffling, but do provide a sane interface over the cases that are supported in a single instruction (not including constant creation/loading), and those can run on NEON unchanged (assuming they're ran on 128-bit vectors, of course, as NEON doesn't support larger ones); and, with Singeli just generating C/C++ currently, you can just map in __builtin_shufflevector if desired. e.g. here's your AVX2 `pre`:

    include 'skin/c' # defines infix a+b & a*b etc to run __add/__mul/... (yes, those aren't here by default, and you can define custom infix/prefix ops)
    include 'arch/c' # defines __add & __mul to do C ops
    include 'arch/iintrinsic/basic' # not necessary for a shuffle, but provides basic x86 arith ops
    include 'arch/iintrinsic/select' # x86 shuffles; there's similar 'arch/neon_intrin/basic' & 'arch/neon_intrin/select' for NEON

    fn pre(inp: [32]i8, out: *[32]i8) : void = {
      store{out, 0, vec_shuffle{16, inp, merge{ # 16 specifies to repeat per 16-elt lane
        range{8}*2+1,            # lower half: 8 Y components; compile-time index calculations
        range{4}*4, range{4}*4+2 # upper half: (4 * U), (4 * V).
      }}}
    }

As a more fancy thing, I've got this working (via bodging together the definitions in CBQN with some sugar to make this pretty; not including all that boilerplate here), compilable to SSE2/AVX2/NEON producing a 4x unrolled core loop, plus tail handling (via reading past the end and doing a load-blend-store if necessary because that's what CBQN's fine with; could easily define a fancy_loop such that it does a scalar tail though). (also can be compiled to RVV via currently-unpublished mappings; no need to unroll for RVV; can choose to do either a stripmined loop or one with a separate tail):

    fn sigmoid{E}(r:*E, x:*E, n:ux) : void = {
      def V = arch_preferred_vector{E}
      @fancy_loop{V,4}(r in tup{'dst',r}, x, M in 'mask' over n) {
        # this loop body is generated 3 times for x86 & ARM - with x being a 4-elt tuple (core unrolled loop); a 1-elt tuple and no masking; a 1-elt tuple and masking
        if (any_hom{M, ...(x!=x)}) {
          emit{void, 'abort'}
        }
        r{x / __sqrt{1 + x*x}}
      }
      # were it not for a bug in tuple loop var mutation in Singeli having undesired pervasion, this would be possible:
      # @fancy_loop{V,4}(r, x, M in 'mask' over n) {
      #   if (...) ...
      #   r = x / __sqrt{1 + x*x}
      # }
    }
    export{'sigmoid', sigmoid{f32}}

(e.g. generated C for AVX2: https://godbolt.org/z/KTeGsazKP)

(I'm not actually particularly expecting interest in Singeli; I just like writing stuff :) )

[1]: https://github.com/mlochbaum/Singeli

Rust feels like a Python developer’s idea of a high-performance computing language.

I might be wrong, but I think it sounds more like Rust doesn't move as far away from the C or C++ way of doing things as you want it to. At the very least, Rust is no worse than C or C++ at any of the things you mentioned.

In my (biased) experience, Rust is much harder to use for advanced projects, than C and C++. On the bright side, it’s also harder to misuse :)

I've found it much easier to use from writing a web service to writing a high performance DB that outperforms RocksDB and clearly people are using it for things like writing operating systems as well as game engines. I'm not sure what in your mind falls under "advanced projects" but I suspect it's something like number crunching (although you link StringZilla so not sure).

I'm still not seeing any description of specific challenges you feel are harder in Rust than in C/C++. In my mind Rust is completely equivalent in being able to accomplish the same tasks.

They wrote 2 whole paragraphs about SIMD.

easier to use from writing a web service to writing a high performance DB

Have you looked at what the actual implementation code of tokio / axum etc looks like?

I can't comment on DB because that is quite far out of my wheelhouse but regarding game engines, I've almost universally seen people revert to storing objects in arrays and passing around "handles" which is literally pointers with extra steps... But I guess at least you are protected from some issues there.

Rust forces absurd levels of abstraction onto code very early and if you are wrong you need to make sweeping refractors.

I really want to love the language, but building bottom up is extremely difficult with it, it may be Stockholm syndrome and familiarity but I can generally get much further into a project much quicker in C++.

In Rust it feels like I'm fighting the language and it's not the borrow checker.

There are so many great ideas in Rust and I keep trying it because of them, don't get me wrong I do think a safer language is the future but Rust isn't there yet. Every time I try it it's better though.

I have actually looked at the interior of Tokio. While there's a lot of complexity there, most of that complexity is in making a highly performing multi-threaded work-stealing runtime rather than anything Rust-specific. Indeed, I find it easier to reason about than in C++ because the ownership rules are enforced and where they're violated is clearly annotated with unsafe & lots of documentation explaining why it's safe. I suspect you'll see similar things in axum but I can't speak to that codebase specifically.

I don't think you'd be able to do something meaningfully easier in C+. I think it is Stockholm syndrome. It took me about 2 months to get familiar that it wasn't uncomfortable anymore and 8 months to reach the "I can generate code faster than I ever could in C++".

For context consider that I'd spent the prior ~11 years coding on and off in C++ professionally and the prior 10 years before that as a kid learning C++ so C++ was definitely a language I felt at home in. I spend almost no time compared to C++ trying to make the build system work and I can quickly pull in high quality components others have written vs in C++ where I either have to implement it myself, spend time on build integration, or figure out a way to do without.

Oh I completely agree Tokio is a complex piece of software, but my point was you need to implement a decent amount of that even for trivial cases where Tokio isn't needed and therefore the default response is to just pull in Tokio.

Part of the complexity for me at least in async Rust was that it is some weird mix between coroutines and completion handlers. It's a pretty nice abstraction with once you wrap your head around it but how many Rust devs would be able to do that?

Should I find a performance regression and need to dive in to the code it takes a lot of unknown unknown to be explored because what is actually happening is hidden from you by the abstraction and the Async book's response is "don't worry about that just use tokio". And the "advanced" section is incomplete.

I've almost always found the complexity of these systems to be reasonable when I've dug through it, but I had to manually mess with Rust code to understand the async model, the book failed to explain what it is and when it clicked it made a ton of sense.

Maybe my issue is that Rust just forces abstractions down your throat and says "trust me you need it" but I would much rather discover why and even find a more relevant abstraction.

The last example I made still stands, there is so much complexity caused by abstractions that the wgpu "book" recommends an older version of winit because the current version made API changes that requires a lot of code change. Winit 0.30 released almost a year ago, I understand that it is likely a volunteer maintainer there but if the language allowed for easy refactoring why has it been a year?

The fact that the webgpu standard is still in active development likely adds to that.

Async rust is purely a state machine equivalent to c++ coroutines. There’s not really any more magic than that within the language itself. In other words, async rust is syntactic sugar and nothing else.

Tokio is a work stealing runtime that then layers other requirements on it (eg futures must be send to support work stealing). This is similar to Apple’s libdispatch.

I’m not sure what performance problems or other magic you’re referring to but it’s hard to answer non-specific gripes with information. It sounds like you’re saying it should be easier to roll your own runtime and there’s slow progress on standardizing some of the pieces. But there’s a lot of runtimes you can pick from (async-std, snol, glommio, monoio, etc). It’s fine that implementing a runtime is complex because a lot of that complexity is inherent (Rust adds some extra boilerplate but that’s all it really is)

I haven’t really found Rust’s async to be all that hard to grok. It’s got some annoying boilerplate in some places where there’s impedance mismatches in very rare hyper specific cases but those don’t normally come up.

I can't comment on DB because that is quite far out of my wheelhouse but regarding game engines, I've almost universally seen people revert to storing objects in arrays and passing around "handles" which is literally pointers with extra steps... But I guess at least you are protected from some issues there.

Regarding game engines, the empirical evidence shows that Bevy has been moving at least as fast as, if not faster than†, comparable C++ engines. I listed all the features I landed in the past year here: https://news.ycombinator.com/item?id=42945730

We love to argue on message boards about theoretical or anecdotal productivity of programming languages, but at the end of the day the only thing that matters is what people have actually been doing with the language, and for Rust the answer is "quite a lot, actually".

†In my view, Bevy has been moving significantly faster than comparable C++ engines, but I don't need to argue that to make my point.

Don't take it as an attack on game engines in Rust. It really isn't, it's more an observation that people happily work around the fundamental language design and that should be an indicator to the language authors.

And they are definitely listening though, when I last took a dive into Rust I faught a lot of weird borrow checker edge cases that I could easily verify as correct. The borrow checker now no longer complains about some of those cases.

As I said I think it's a language maturity thing. It will get there.

The borrow checker changes you’re talking about is several years old. The next gen borrow checker should solve the remaining annoyances but that’s years away.

But I disagree that Rust code breaks down to emulating pointers by way of handles.

I'm still not seeing any description of specific challenges you feel are harder in Rust than in C/C++

It's harder to model domains in Rust. C has the same problem and it is a big reason why the industry standard for game dev is C++

Care to give an example? I think GUIs remain the softest domain modeling area, but that’s about how do you do next gen GUI toolkits that are super high performance, safe, and lower overhead vs what Rust has today. But that’s an underserved niche anyway vs c++ toolkits or electron.

The other weak parts might be ecosystem immaturity (eg Unreal vs Bevy) but that’s not a language modelling issue.

Rust is easy to understand as "a language by browser writers for writing browsers." That statement alone gives you most of Rust's design choices:

* Safety over everything else

* Very good single threaded performance

* Javascript-like syntax and a Javascript-like package manager

I have been working on some software in Rust recently that needs bit and byte manipulation, and we have "unsafe" everywhere and hugely complicated spaghetti compared to the equivalent code in C.

Most likely because many keep forgetting that bit and byte manipulation in C is a mix of implementation defined and UB, depending on how it is coded.

What's the problem of using toolchain-specific features instead of behavior defined by the standard? Isn't this the bread and butter of embedded development and the reason why some behavior is purposely left undefined in the standard?

[...] and the reason why some behavior is purposely left undefined in the standard?

It's one of the saner reasons why there's UB in C and C++. But there's lots of crazier reasons.

It's one of the saner reasons why there's UB in C and C++.

It is the reason why the standard purposely leaves some areas undefined.

C and C++ detractors talk a lot about UB but they always show their understanding on the subject is at best superficial. The parrot UB as if it was this major gotcha, when it is literally behavior the standardization committees make a call to not define behavior to purposely leave it open, so that any implementation can still be conforming even if they decide to implement behavior not defined in the standard. It is that simple, but somehow people parrot UB behavior as if it was this major gotcha. Baffling.

Baffling is the ignorance in the ways of WG14 and WG21, while pretending to be a know it all, maybe update yourself on the relevant papers for C2y and C++26, aimed at clearing up references to UB with erranous behaviour, or implementation defined.

This subthread is kinda strange with people claiming IB and UB are the same, considering that C/C++ have had clearly delineated definitions of undefined behavior versus implementation-defined behavior for decades in their term definitions.

You are upset we aren’t familiar with papers and proposals which are not yet agreed on as standard? And for unreleased C++ versions?

And then it was a decision on the part of compiler vendors to define it to do insane things like time travel. Not the standards committee. Compiler vendors could have just as well defined it to wrap.

Some compiler decided to define overflow as wrapping. Such as GCC/clang when passing the -fwrapv flag.

Most projects don't use that flag though, why not?

Nite that if you assume -fwrapv, you're not writing in C anymore, you're using a vendor specific dialect.

Exactly. The classic example is invalid pointer deref. It’s too costly to check every deref against every allocation (outside of special debug modes). So the system usually can only detect if there is a virtual memory page fault, in which case it can crash.

In embedded systems without virtual memory there is no validity checking at all. Or it could periodically check (random sampling).

So the standard makes the reasonable choice to leave it undefined. If you can detect it and crash, that’s great. If not and you accidentally overwrite your programs instructions, anything can happen.

Rust only avoids UB in so far as it relies on default clang behavior on modern hardware.

Re Rust, that’s simply not true. Rust relies on compile time validity checks that have nothing to do with virtual memory.

Rust only avoids UB in so far as it relies on default clang behavior on modern hardware.

That's not true.

Rust behaviour avoids UB by not compiling with invalid reference deref because it checks life time.

Clang's behaviour is likely a crash but anything can still happen. It is literally undefined.

Exactly. The classic example is invalid pointer deref. It’s too costly to check every deref against every allocation (outside of special debug modes). So the system usually can only detect if there is a virtual memory page fault, in which case it can crash.

You could still define what happens.

Eg you could define that whatever error might happen because of invalid pointer deref can't travel back in time. (UB famously can travel back in time: UB at any time during the execution makes the whole execution UB.)

Security exploits is the problem, because what common people that don't read WG14 mailings think the word undefined means, and what everyone else involved with creating compilers understand what they are allowed to do, is not the same.

Security exploits is the problem, because what common people that don't read WG14 mailings think the word undefined means, and what everyone else involved with creating compilers understand what they are allowed to do, is not the same.

Your comment lacks credibility. Your hypothetical scenario would only be conceivable if a) a team was well versed enough in C to adopt a specific toolchain to leverage implementetion-defined behavior that leveraged behavior left undefined by the C standard, b) somehow the same team decides on a whim to replace their toolset with some other random toolset without eve being aware of their toolset-specific code. This is far from a realistic scenario, and reads more like mindless complains about UB coming from a place of ignorance.

I have been working on some software in Rust recently that needs bit and byte manipulation, and we have "unsafe" everywhere and hugely complicated spaghetti compared to the equivalent code in C.

I'm curious what makes this so different from my experience. I rarely ever have to write "unsafe", and I'm writing quite low-level engine code that certainly uses bit manipulation. In fact, crates like bitflags and fixedbitset make it so easy that I tend to get dinged in code reviews for using bit flags when structs of booleans would be simpler :)

I'm curious what makes this so different from my experience. I rarely ever have to write "unsafe", and I'm writing quite low-level engine code that certainly uses bit manipulation.

Perhaps your usecase is similar enough to eg JavaScript engines? Because that's a usecase that browser writers would at least have in mind?

It's not. It's Bevy.

What effects this is separating your unsafe into unsafe abstractions. If you're careful you don't have to write too much unsafe. Granted it's not easy.

A game engine is much more similar to a browser engine than you think. The operations you have discussed, using bits as flags, are also the most basic forms of binary-level manipulation out there. Things like tagged pointers and bit flags fit nicely and neatly into an encapsulated unsafe abstraction (provided you want to add an extra 1000 lines of code for it).

As far as I can tell, any time you want to rely on the exact binary layout of something in memory, you need unsafe. As a corollary, any time you want to bit cast from one type to another, you need unsafe. This means that things like succinct data structures and building network protocols need quite a bit of unsafe everywhere. The former needs you to do things like "store 14 bits of X here and 12 bits there..." The latter needs control of bit and byte layout because you want to carefully eliminate implementation-defined compiler behavior.

And a jet engine and IC engine are engines, but putting your car engine into a Boeing and vice versa would be an unwise decision.

I'd argue you're over-abstracting the differences. They have different purposes. Game engines need high performance, while browsers need to enable wide selection of APIs.

Browsers need very high performance, too.

As far as I can tell, any time you want to rely on the exact binary layout of something in memory, you need unsafe. As a corollary, any time you want to bit cast from one type to another, you need unsafe.

No, that's what bytemuck is for. If bytemuck didn't exist, sure, I'd be using a lot of unsafe.

Bytemuck handles the latter, not the former. In the applications I am working on, both matter.

* Javascript-like syntax and a Javascript-like package manager

I think this is not serious criticism. The "javascript-like package manager" reference actually refers to fixing a major problem with the developer experience playing legacy programming languages such as C or C++. Java has those, .NET has those, every single mainstream programming language has those. Except C or C++.

Rust might be riddled with "the emperor has no clothes" aspects, but having a package manager is not it.

That's not a negative aspect of Rust, and I'm not picking out a list of things I dislike about Rust. Making a statement that isn't unequivocally positive does not equal criticizing something.

Cargo is, however, a similarity with JS. On the whole a good one. Also, cargo works much more like npm than maven, for example.

Even before getting into SIMD, try using Rust for concurrent, succinct, or external-memory data structures. It quickly becomes clear where the friction is.

It's the exact opposite for me: I use concurrent data structures more often in Rust than I do in C++ because I don't have to worry about dumb data race bugs. If one of my Bevy systems is slow, I slap par_iter() on the query and if it compiles it probably works, or at least fails for a not-stupid reason.

What do you think about 'par_iter' having to wait for work imbalance to return execution back? With 'when_all' like primitive one can continue execution on any thread without losing one for waiting. ps as someone who does not have rust job i would like to see an example how rust deals with task based systems if you have public one at hand ofc.

Concurrent data structures are rarely faster than "lock + non-concurrent data structure" and if you're putting constructs like par_iter() in a lot of places, there's a good chance you would be better off with the "dumb" pattern than the concurrent data structure.

The same goes with Arc - if you're using it a lot there's a good chance that code with a lot of Arcs is slower than equivalent GC-ed code.

While it's true that par_iter() uses a concurrent data structure under the hood, it's specifically designed to use work-stealing to avoid needing threads to communicate.

Why would putting a lock over a global workqueue be faster than per-thread workqueues that don't require inter-thread communication (except in the case where work-stealing is required)?

Atomics are very expensive operations. Lock/unlock is two atomics. Many concurrent data structures will end up doing many more atomic operations than you expect.

The general wins of concurrent data structures come when you really are accessing them truly concurrently - as in when many threads on many cores are heavily contending for access and you need to make global progress.

Sure! However, the work-stealing queue in rayon [1] uses three atomic operations instead of the two atomic operations for a mutex for a global lock. The difference, however, is the three atomic operations for the thread-local queue should be uncontended, whereas a global lock on a global work queue would experience contention from every thread trying to access it for jobs.

Between the choices of "single work sharing queue with a big mutex on it that all threads access for work" vs "per-thread work-stealing queue that's uncontended for the cost of one extra atomic," in what situations would the work-sharing queue with the global mutex outperform? Perhaps if there's a small number of jobs, and there's not enough time for the work-stealing algorithm to distribute jobs to the worker threads before the work-sharing algorithm has already finished.

[1]: https://github.com/crossbeam-rs/crossbeam/blob/423e46fe20471...

Run a benchmark. With low contention, the lock will outperform. Atomics are very expensive assembly instructions.

Fedor Pikus has a good talk on this at cppcon 2019.

I've seen the talk! The issue with using a global lock on a global work queue is that, unless the work items have drastically different compute times, there _will_ be high contention on the lock.

I ran a benchmark [1], which shows that this is correct:

Results on quad-core Intel Linux box:

$ hyperfine target/release/testit 'env USE_RAYON=1 target/release/testit' Benchmark 1: target/release/testit Time (mean ± σ): 2.526 s ± 0.139 s [User: 4.709 s, System: 11.425 s] Range (min … max): 2.391 s … 2.730 s 10 runs

Benchmark 2: env USE_RAYON=1 target/release/testit Time (mean ± σ): 174.1 ms ± 0.9 ms [User: 212.1 ms, System: 121.1 ms] Range (min … max): 173.1 ms … 175.4 ms 16 runs

Summary env USE_RAYON=1 target/release/testit ran 14.51 ± 0.80 times faster than target/release/testit

Results on M1 Pro:

$ hyperfine target/release/testit 'env USE_RAYON=1 target/release/testit' Benchmark 1: target/release/testit Time (mean ± σ): 692.2 ms ± 8.3 ms [User: 491.4 ms, System: 5693.6 ms] Range (min … max): 683.2 ms … 704.5 ms 10 runs

Benchmark 2: env USE_RAYON=1 target/release/testit Time (mean ± σ): 63.0 ms ± 2.1 ms [User: 97.7 ms, System: 47.0 ms] Range (min … max): 61.0 ms … 71.2 ms 44 runs

Summary env USE_RAYON=1 target/release/testit ran 10.99 ± 0.39 times faster than target/release/testit

[1]: https://play.rust-lang.org/?version=stable&mode=debug&editio... (I'm just using the rust playground as a pastebin; the actual benchmarks were run locally)

Atomics are very expensive operations.

Atomics are very expensive when contended - which is also the case where locks would introduce blocking and reduced performance. Uncontended atomics are relatively cheap.

I suggest you run a microbenchmark. Uncontended atomics are some of the most expensive assembly instructions out there. They also acquire global system locks on certain stages of execution.

One of the things I find interesting about SIMD is that a lot of behavior that is “undefined” for scalar types in C-derived languages is explicitly fully defined when you use SIMD intrinsics with the same integral types. UB exists to hide the fact that major CPU architectures give different results for basic ALU operations in some cases. SIMD makes no such pretense of abstraction. If I am using AVX-512 I explicitly get the full Intel architecture experience, the implementation details are not hidden behind UB. Same with ARM, etc.

For example, shift overflows are masked on x86, zero-filled on ARM, and undefined in C/C++. In SIMD-land, none of this is hidden and so you design your code to leverage the reality that those instructions behave differently, whereas in C/C++ only the behavior they have in common is “defined”.

The vector ISAs are sufficiently different from each other (and normal CPUs) that it is like trying to build a compiler that can automagically produce optimized code for both CPUs and GPUs from the same source tree. I am not optimistic that this will happen anytime soon. AVX-512 essentially started life as a GPU ISA, which probably explains the interesting fact that a modern x86 CPU core has more AVX-512 registers than x86 registers.

Rust feels like a Python developer’s idea of a high-performance computing language.

I'd say the language complexity (and it keeps piling more things) means it's pretty far from Python's ideas.

Julia is closer.

Rust is an extraordinary simple language. Compare it to the complexity of C++.

We can say that Mars is more similar to Earth than Jupiter even though Sagittarius A* exists.

If you only compare against C++, maybe. Otherwise, no. C++ and Rust are in a class of their own when it comes to compile times and complexity.

A problem for RISC-V is going to be that there's currently no way for user code to detect the presence of RVV. I have no idea how you can do multiversioning with that limitation.

The solution is to ask the OS to detect it for you. Linux offers a syscall for this (riscv_hwprobe). Has the drawback that it requires OS support, of course. But RVV requires OS support anyway (e.g. managing mstatus, saving vector registers on context switch), so that seems fine to me.

What happens there when you try to execute a missing RVV instruction ? On other archs you get a SIGILL which you can handle.

It's "reserved" which is basically the same as C's UB - anything can happen (nasal dragons etc.) so you can't rely on it.

Just because it's reserved doesn't mean anything can happen. On x86_64, you get a clearly defined error when you use reserved bits and the like. I don't know if RISC-V is the same, but if it isn't it's because they chose to be vague, not because that's what it means to be reserved.