See also: https://ispc.github.io/

ISPC is great software. It's a shame in many ways Intel didn't put more resources behind it.

This so-called GPU programming model has existed many decades before the appearance of the first GPUs, but at that time the compilers were not so good like the CUDA compilers, so the burden for a programmer was greater.

As another poster has already mentioned, there exists a compiler for CPUs which has been inspired by CUDA and which has been available for many years: ISPC (Implicit SPMD Program Compiler), at https://github.com/ispc/ispc .

NVIDIA has the very annoying habit of using a lot of terms that are different from those that have been previously used in computer science for decades. The worst is that NVIDIA has not invented new words, but they have frequently reused words that have been widely used with other meanings.

SIMT (Single-Instruction Multiple Thread) is not the worst term coined by NVIDIA, but there was no need for yet another acronym. For instance they could have used SPMD (Single Program, Multiple Data Stream), which dates from 1988, two decades before CUDA.

Moreover, SIMT is the same thing that was called "array of processes" by C.A.R. Hoare in August 1978 (in "Communicating Sequential Processes"), or "replicated parallel" by Occam in 1985 or "PARALLEL DO" by "OpenMP Fortran" in 1997-10 or "parallel for" by "OpenMP C and C++" in 1998-10.

Each so-called CUDA kernel is just the body of a "parallel for" (which is multi-dimensional, like in Fortran).

The only (but extremely important) innovation brought by CUDA is that the compiler is smart enough so that the programmer does not need to know the structure of the processor, i.e. how many cores it has and how many SIMD lanes each core has. The CUDA compiler distributes automatically the work over the available SIMD lanes and available cores and in most cases the programmer does not care whether two executions of the function that must be executed for each data item are done on two different cores or on two different SIMD lanes of the same core.

This distribution of the work over SIMD lanes and over cores is simple when the SIMD operations are maskable, like in GPUs or in AVX-512 a.k.a. AVX10 or in ARM SVE. When masking is not available, like in AVX2 or Armv8-A, the implementation of conditional statements and expressions is more complicated.

DirectX 8 added programmable shaders as we mostly know it today in 2000 with directx 9's shader model 2.0 really solidifying things in 2002.

CUDA didn't show up until 2007.

SIMT and SIMD are different things. It's fortunate that they have different names.

A GPU is a single instruction multiple data machine. That's what the predicated vector operations are. 32 floats at a time, each with a disable bit.

Cuda is a single instruction multiple thread language. You write code in terms of one float and branching on booleans, as if it was a CPU, with some awkward intrinsics for accessing the vector units in the GPU.

That is, the programming model of a GPU ISA and that of Cuda are not the same. The GPU gives you vector instructions. Cuda gives you (mostly) scalar instructions and a compiler that deals with this mismatch, lowering branches to changes in exec mask and so forth.

With my numerical library hat on, I hate this. Programming a simd machine through a simt language means trying to get the compiler to transform the control flow into the thing you could easily write using vector instructions.

With my compiler implementer hat on, I hate this. It gives you two control flow graphs intertwined and a really bad time in register allocation.

It's not totally clear to me why simt won out over writing the vector operations. I'm certainly in the minority opinion here.

I guess that a lot of people are uncomfortable thinking about vector instructions, and dealing with masks manually? And for vector instructions you need to align things properly, pad the arrays such that they are of the right size, that people are not used to I guess.

It's not totally clear to me why simt won out over writing the vector operations.

From the user side, it is probably simpler to write an algorithm once without vectors, and have a compiler translate it to every vector ISA it supports, rather than to deal with each ISA by hand.

Besides, in many situations, having the algorithm executed sequentially or in parallel is irrelevant to the algorithm itself, so why introduce that concern?

I'm certainly in the minority opinion here.

There are definitely more userland programmers than compiler/numerical library ones.

Subtle difference. A parallel_for can have asynchronous threads. They can all divergen and run independent instructions (due to if statements etc)

SIMT means multiple processors all executing the exact same instruction. One instruction decoder, but like 64 execution pipelines.

The same is true for CUDA/OpenCL kernels. They can include conditionals and they can execute independent instructions.

The difference is that because the software "kernels" (i.e. software threads) may be mapped by the compiler either on hardware threads or on hardware SIMD lanes, and this mapping is not controlled by the programmer, the divergent instructions will cause inefficient (serial) execution when they happen to be executed on SIMD lanes of the same core, so they must be avoided.

This however is only an optimization problem, if the speed would be irrelevant all the kernels could execute divergent instructions.

The reason for the existence of CUDA is to mask for the programmer the existence of the SIMD lanes and to allow programming as if the software threads would map only to hardware threads. Nevertheless, for optimum performance the programmer must be aware that this abstraction is not really true, so the programs should be written with awareness of the limitations introduced by SIMD.

Isn't this already implemented in QEMU?

In addition to ISPC, some of this is also done in software fallback implementations of GPU APIs. In the open source world we have SwiftShader and Lavapipe, and on Windows we have WARP[1].

It's sad to me that Larrabee didn't catch on, as that might have been a path to a good parallel computer, one that has efficient parallel throughput like a GPU, but also agility more like a CPU, so you don't need to batch things into huge dispatches and wait RPC-like latencies for them to complete. Apparently the main thing that sunk it was power consumption.

[1]: https://learn.microsoft.com/en-us/windows/win32/direct3darti...

The recent "AI chip" proposals (Tenstorrent, Esperanto Technologies etc.) seem to be quite similar to the old Larrabee design, except based on RISC-V as opposed to x86. So we might see that happen after all.

I helped make a really cursed RISC-V version of this for a class project last year! The idea was to first compile each program to WASM using clang, and lower the WASM back to C but this time with all opcodes implemented in terms of the RISC-V vector intrinsics. That was a hack to be sure, but a surprisingly elegant one since 1. WASM's structured control flow maps really well to lane masking 2. Stack and local values easily use "structure of arrays" layout 3. Heap values easily use "array of structures" layout

It never went anywhere but the code is still online if anyone wants to stare directly at the madness: https://gitlab.com/samsartor/wasm2simt

This is cool! If you wrote a blog post going detail about the insights, I'd read it

This is in contrast to SIMD, or "single instruction multiple data," where the programmer explicitly uses vector types and operations in their program. The SIMD approach is suited for when you have a single program that has to process a lot of data, whereas SIMT is suited for when you have many programs and each one operates on its own data

This statement is comparing the SIMT model to SIMD. Can anyone explain the last part about SIMT being better for many programs operating on its own data? Are they just saying you can have individual “threads” executing independently (via predication/masks and such)?

SIMT let's a scheduler get clever about memory accesses, SIMD can practically only access memory linearly (scatter gather can do better but it's still usually quite linear) whereas SIMT can be much smarter in terms of having lots of similar bits of work going on in ways that use the bandwidth maximally and don't overlap.

https://developer.nvidia.com/blog/how-access-global-memory-e...

SIMT still expects coalesced memory access that's close together otherwise performance falls off a cliff

Hey, AVX-512 again!

"Show HN: SimSIMD vs SciPy: How AVX-512 and SVE make SIMD nicer and ML 10x faster" (2023-10) https://news.ycombinator.com/item?id=37805810