ceci n'est pas une RDMA

I'm going to pick on RDMA here a bit- and then I'm going to show you an alternative that we made at Hyperion Gray. Am I selling this to you? Is this a marketing blog now? No. It's just unreleased for now and I haven't decided what's getting open sourced and what's getting sold to NVIDIA or AMD for 10 billion dollars (bidding starts now). Either way, some version of this that's better than what you'll see here will be released to the public in some way. We'll talk about the Hyperion Gray Open Source License in another entry, but it's pretty sweet, and that's where a lot of this work is going to be released under.

Anyway RDMA- what is it? It stands for Remote Direct Memory Access. One thing I learned is that to meet the definition of RDMA you really need more than just actuall DMA in a remote manner. You need to conform to the RDMA model- that means RDMA verbs and RDMA semantics- things like defining registered memory regions, work is typically done in queue pairs, where you need to emit to a completion que....you know what I'm just going to show you a picture of the RFC table of contents for RDMA because I'm starting to bore myself.

If I were to exaggerate it a bit I'd call this "a blight upon humanity" or perhaps "gods punishment to computer people". So I'll be fair, and i'll just call it a blight amongst polite society. If you think I'm exaggerating a bit, let me note that that screenshot is cut off because I couldn't zoom out farther (further? ferver? farthing? whatever). So in keeping with the theme of this oddball HPC work I've been doing for the last year and a half- let me show you what it looks like under our model called DMAR. DMAR stands for something totally different than RDMA, it's Distributed Remote Access Memory. We did this because RDMA people would yell at us and make us feel little if we called it RDMA, because it doesn't conform to the standards of doing a bunch of unnecessary stuff to make everyone's life difficult.

Lets talk about WHY people would subject themselves to 100 lashes RDMA despite it being a PITA: it's real fast. By real fast I mean it caps at 400Gbps theoretical limit, it requires special hardware and software, and must learn the ins and outs of RDMA proper, which is about as easy as learning ancient Aramaic. But once it does work! The magic starts to happen: a shitload of debugging. But after that: the real magic starts to happen, and your bytes move at the speed of 50 GB/sec theoretical high limit (400/8). That's pretty fast for something you can do over ethernet (which is a type of RDMA called RoCE- RDMA over converged ethernet). Why the smarmy name? Because this is fancy dammit, this is enterprise shit, you wouldn't understand.

So that's the little intro to RDMA, it basically provides the ability to go memory-memory over a LAN, without disturbing the CPU, and requiring no copies of the data to occur (so it's super fast zero copy networking from memory to memory). And that's why people in High Performance Compute (HPC) use it- sometimes you just need more than one machine, and this allows for that in an efficient-ish way and at high speeds. It's considered the gold standard for moving stuff from memory node a to memory node b and having it be immediately useful(ish).

Neat right? I still don't like it.

DMAR - Distributed Memory Access Remote

DMAR is our version of RDMA, and I won't say it's easier or more sane, but I will say it's a lot faster, puts a lot more thought into the end goal, and uses some neat concepts to achieve a much much faster (~10,000 times faster) remote memory distribution. So how does it work?

It sort of works like RDMA in that we use verbs, except those verbs are just HTTP verbs. Because "this is enterprise shit" as I said- it's super complicated. It's a REST API (websocket capable). It does however, require a proprietary driver that we made. I'll be open sourcing that one here soon, and i'll no longer be proprietary. Here's the general process of DMAR with smug comments about how RDMA would do things. In HPC- you're trying to eek out every little bit of performance from your machine. This is especially true if you're a researcher that has constrained themselves to commodity hardware (hi).

The idea behind memory arenas is they're large chunks of memory, preallocated. This can be either kernel, DMA access to unified memory of all assets - called a Unified Memory Arena (UMA), and personally I fall back to something like hugepages if what I wrote has failed for some reason (usually I have too many Chrome tabs open, like 2, eating my memory)*. If you've ever used Hugepages you know that there's a split there between THP (transparent hugepages) and the other kind which I just call normal people Hugepages but I'm sure has another name. DMA arenas are a lot like that as well, you can allocate a DMA arena and access it via a program only, OR you can mount it and work physically (logically? spiritually?) in memory- meaning you don't have disk overhead, cache misses become far more rare, and you can do a bunch of other complicated stuff that I won't get into.

Back to DMAR- so how does DMAR work? The real magic behind it and the "speed" it achieves, is, as the theme of this notebook is, doing less, moving as few bytes as possible, and moving them when it's convenient (i.e. not during an intense run of something, though we'll get to that). The technique is essentially this:

• We use our custom referential pseudo-compression PVRTC to create a small artifact that's filled with common byte patterns. We use a very quick "mining" process against any artifacts (and sometimes dumped memory) from the process we're going to use. This produces a small artifact that's essentially just references into the dumped memory and stuff on disk (that will soon be in memory). The size is roughly a few KiB per a few 10s of GiB of the file. In other words- it's real small, we've developed a ton of tooling to make these files automatically and very quickly.

• These files are now both squarely: small- the byte pack (a few KiB) and the artifact weighing in at ~0.01% - and yes that's percent as in the size is 0.0001x the original file- which gets compiled into the decoding/decompression program- note these two little files are all you need to reconstruct the original file. So now you can treat it all just like any normal zip or tar.gz or .lzma or .lz4 file- it's just a program that compresses/decompresses a tiny file. Why is the file so small? Because everything on a machine is patterns of bytes. The same 16 nibbles over and over again, so we don't need to write them all down in their entirety- we just have to point at them. And each time we point at them we represent that pointer cheaply, we greedily match patterns and runs of bytes, it's ridiculously effective when done right and I am always in awe of how many files have the same repeating patterns.

• OK neato trick huh? We have a few Kib to a few MiB representing potentially very very large files, we have a program that can give us the original file anytime we want, now what and how does this relate back to the totally amazing 400Gbps that RDMA achieves and that thing I called DMAR!? So now that everything is in tiny-land, the byte motion doesn't really matter much does it (no, the answer is no if you're wondering)? I could rsync that thing, reconstruct the file which takesa few microseconds per GiB, I could scp it on over, really anything and I'd effectively be moving the original bytes somewhere else really quickly. But since we care about efficiency and all that, really what we do is connect up the DMA arenas of two nodes, and we make a REST API call to the DMAR daemon, telling it to move from one arena to the other. If you're in VRAM (GPU RAM), we make it easy to put the references into kernel memory (DRAM, the cheaper stuff) via a DMA-BUF. It doesn't matter what that is exactly, it's basically just a place where VRAM and DRAM can share data efficiently.

• We can now move bytes from VRAM to kernel memory or userspace, we can move from DMA arena to DMA arena, and it's all super cheap. The one negative part of DMAR compared to RDMA is um - well, you don't achieve the actual throughput of RDMA, but also you don't have to to get your work from one place to another (the whole point of RDMA). So I'm not sure if that's a negative or more of a being smarter than RDMA. But anyway, we typically term this the "effective speed" of transfer so we don't accidentally lie to anyone - yes we are taking say 100GiB and moving it in a matter of a few microseconds (real timing) to wherever you'd like to go and expand it back to the original file in a few more microseconds, but we are not shoveling bytes faster than RDMA. I just want to make that clear so no one tries to say "that's breaking physics" when I say that we achieve an effective speed of 640 million times faster than NVIDIA's GPUDirect can shove bytes at their own GPUs from userspace with special drivers, special NICs, special hardware, special engineering, and a special hole in your wallet. We aren't shoving bytes faster, we're gently carressing the bytes so that they'll move to the other side much more easily. Also yes that's a real benchmark number, 640 million from kernel/userspace <-> VRAM.

So what'd we learn today? Well, I'm kind of a dick to large enterprises. But to be fair, they're profiting from inefficiency. And OK, if it's a high school project I don't care, but when it's something that is literally called "High Performance Compute" and your solution is "lets build a massive ecosystem of huge machines that get REALLY hot when you use them. That won't do anything bad like, kill the planet and stuff right?" Also words mean things, HPC should be about performing things better- not just shoving more hardware at stuff. But that's where we are right now and honestly it's kind of a sad place to be. In my "security research" (hacking, it's hacking), I see hackers doing really amazing things with literally whatever they have on them. And then I look at the enterprise side, and they're making a big deal out of releasing a thing that moves bytes from gpu to gpu 4x (theoretical max) faster IFF you have that H100.

Ahem, anyway, end rant and sincere apologies to everyone except NVIDIA. So DMAR, right, the DMAR thing. Because I like things to go a little bit extra - instead of just a faster than RDMA system, I figured why not first, see how fast this is in a remote context, which I'll ruin for you right now, it's 20,000x faster, but what I made an overlay mesh network of DMA arenas? So we went ahead and did that. Now all of my machines in my setup can RDMA, er sorry, DMAR the f*** out of things without breaking a sweat.

*Protip sudo sysctl vm.nr_hugepages=2048 will enable hugepages and allocate some. Then, for memory hungry applications, you can tell them they can only have so much with

sudo hugeadm --pool-pages-min 2MB:2048 --pool-pages-max 2MB:2048
sudo hugeadm --create-mount

That will allocate 2048 2MiB hugepages, create a mountpoint, where you can for example direct your Chromium cache to:

sudo mkdir -p /mnt/chromium-hugecache
sudo mount -t hugetlbfs -o uid=1000,gid=1000,mode=0700,pagesize=2M none /mnt/chromium-hugecache
chromium-browser --disk-cache-dir=/mnt/chromium-hugecache

Then at runtime also start the application with hugectl:

hugectl --text --data --bss --heap --shm /usr/bin/chromium 2>&1 >/dev/null &

Only quick security note there is that /dev/shm (shared memory) is 777, so if you're worried about someone being on your local machine, um, well really just turn off your computer, you shouldn't be using it or worrying about permissions. Reinstall your OS and go from there. Here's a nice shelly if you want it:

#!/usr/bin/env bash

sudo sysctl vm.nr_hugepages=20
sudo hugeadm --pool-pages-min 2MB:2048 --pool-pages-max 2MB:2048
sudo hugeadm --create-mount
sudo mkdir -p /mnt/chromium-hugecache
sudo mount -t hugetlbfs -o uid=1000,gid=1000,mode=0700,pagesize=2M none /mnt/chromium-hugecache
chromium-browser --disk-cache-dir=/mnt/chromium-hugecache
hugectl --text --data --bss --heap --shm /usr/bin/chromium 2>&1 >/dev/null &

Now Chromium will eat away at your hugepages allocation, but bounded, and a little bit quicker.