Training LLMs with stuff that processes other stuff

Lets talk about training LLMs a little bit. It's a semi-interesting process that involves... well it depends. OK lets talk a bout training LLMs using low rank adapters (LoRAs). Those little things allow you to essentially mess with the weighting of the big fat layered statistical model. They honestly don't get enough credit, they're thought of as sort of a lightweight training or tuning a lot, whereas they can actually be really powerful. Most of the time you hear about making an adapter (usually a LoRA) vs. fine-tuning where fine-tuning is the powerful one and the LoRA is the sissy one. First of all, we've gone and done it again, we've fked up important terminology. a LoRA is a kind of fine-tuning that allows you to weigh certain parts of your models existing layers (usually made of something like tensors, multidimensional matrices) more or less, as well as add "things" to those layers. What those things are are um, more tensors really, but not enough that they make up a full layer, they more work with a layer, and do some mathy stuff to let the LLM make somewhat different decisions. What those decisions are depends on where you put the adapter. A large language model is just a bunch of layers of things, when you hear about stuff like mixture of experts, it's really just talking about various layers that are statistically trained on specific topics. You can kind of picture it in your head as experts in topics deciding on things, but that's not the actual reality of it. The mixture of experts thing is just a way to visualize statistical layers that will favor specific tensors/vectors as a path forward to the next layer. Depending on the model the entire set of "expert" layers might be used, or they might be selectively used. A solid example of this is Kimi K3 by Moonshot AI- it has an absurd number of expert layers (892), but it only chooses a handful of them for any prompt to go through (I think it's 12ish or 16ish). It uses a thing called Kimi Delta Attention (KDA) to keep stability within the model. When you route through a metric f'ton of layers or rather in this case, have to decide which to use you can essentially do what is scientifically known as "fcking it all up" and everything just sort of looks like noise, losing any kind of coherence to an already complex system. I forget what my point was, so I'll just move on.

LoRAs are the kind of fine-tuning that you as a user probably want, and you just don't know it. The other thing that they call "fine-tuning" is really changing the whole-ass model by significantly changing layers of it. Unless you have an idea for how a model can work totally differently and not stuff like "learn a new topic or skill" then you want a LoRA. It's what I generally want, because I'm not an actual AI researcher, I just want to use it to do my bidding (world domination, etc). Generally, LoRAs aren't super intensive processing power, usually you'll see folks recommend a squarely solid GPU with something like in the 20s of GiB of VRAM. So here at Hyperion Gray, we kind of figured, that sounds low enough that you can get away with seriously optimizing a runtime and do it without a GPU. So we went ahead and tried.

The pTPU and GPUless, training LLMs without a GPU

So as you know, I screw around a lot. I don't take myself that seriously, I curse in research journals, and I make several questionable decions per day. But, I actually do take my work really seriously and I do a lot of it. If anyone asks me how I did x,y,z the answer is always the same- I did a LOT of times, I failed a lot of times, then I managed to do a thing after beating my head against the wall repeatedly. It's the same here.

A lot of the work that I've been doing lately involves the question of "are we doing this the right way?" overall. I sort of look at the way that AI is done and look at hyperscalers and GPU farms and massive datacenters and it all just seems... wrong. Or at least super inefficient. I've taken that work really far, and put a lot of effort into it, and this is the first of many many things I'll be releasing in that vein.

Long story short, I made a service that anyone can use to make an adapter from their data, and totally turnkey upload their data, upload their model or use an already-existing model to train it to do something new. The frontend is super simple, it's a little web page I call GPUless, you upload some data, you have an existing LLM format it for you on our servers (it's trained on doing so), you can choose some weights for the adapter and where to insert it from a drop-down menu, and then you hit train and out comes a LoRA after a little while. It's pretty cool and pretty useful, I've used it to train LLMs to learn about um, lets call it cybersecurity, those of you that know me probably know what that entails for me, but we don't use that word, or the government puts us on an ITAR list or some BS. Lets say I use it to train stuff on bounded, authorized offensive security testing. Yeah that's the spot. The front-end is almost boring really, but it's cool, you can actually just shove entire books at it or stuff you collect around, and train a model to do a new thing using a LoRA or series of LoRAs which usually works best. Small and specific, solid data, works way better than "everything you could find". We also charge a tiny fee (2 dollars) per LoRA training session so use VERY small data, like half a JSON line and run the thing like 100 times. I'm kidding of course, but really small and specific data, even if it's totally unformatted, as long as it's focused the new service that we are calling GPUless will train and give right back your model now with LoRA inserted and give the LoRA data itself so that you can apply it to any other models you have. It's pretty neat.

But what's really cool is the backend. This training uses exactly 0 GPU to train, and instead opts to use a specialized virtual TPU that we made. It's based off of work in making virtual GPUs that we've been doing to solve the exact thing I was talking about in paragraphs one and two- are we doing this right? "It" being using massive amounts of hardware and massive amounts of energy to train AI? Something about it seems off, so squarely within our current mission statement of "this round of Hyperion Gray" we're taking the problem and taking a totally fresh look at it. This is one of the first things we're releasing that show- sometimes you don't need all that bullsh**. Sometimes an efficient and specialized system can do the job. In this case the virtual TPU or pTPU (that stands for packet TPU, for historical reasons I'll talk about at some point), is a specialized system for tensor mathematics that uses among some things, a lot of reductions in work for tensors that you can do instead of brute forcing them. Exactly what those are is boring, but they came out of some work in cryptomining and optimizing the PRL (Pearl) algorithm (https://compute.pearlresearch.ai/), which is pretty awesome.

Along with a bunch of tensor math reductions, it was also a lot of computer engineering work. In particular, the pTPU works with an interesting execution shape that makes it ultra-L1-cache-friendly for a CPU. Paired with some pretty solid CPUs- in this case, the first box this is running on is a dual AMD EPYC setup with 96 physical cores/192 threads. The way it works is by using AVX2 and AVX-512 for their full advantage, and maintaining L1 cache residency to the absolute max. On the backend you can actually see the cache hits piling up with no misses, which is cool to see when that's your goal. With vectorized execution or single-instruction multiple execution (SIMD) and SIMT (multithreading) in batches, we managed to achieve an efficiency here, along with our 1337 math skillz, to rival a GPU's execution in terms of the amount of time it takes to train a LoRA over a sizable model. To increase efficiency we also work entirely in a DMA arena (that is a direct memory access arena), so we're effectively working entirely within direct kernel memory, 0 disk overhead, and little memory fragmentation. So it's all cache all day long on Linux baby, hell yeah. While this all fits in a single paragraph, it involved a ton of tuning, more than one kernel mod, like one we call memhub (it makes DMA arenas and useful structures within them), and a lot of testing, tuning, failing, repeat. Admittedly it was made easier by prior us'es in that it's based on a virtual CPU and virtual GPU architecture that we also made earlier in the year. I haven't actually talked about that yet, but it's coming your way to a computer near you, starring Ryan Gosling as "your GPU" and Rachel McAdams as the love interest "the Data". I won't claim it's faster than a GPU because it's not faster than an RTX 5090 the last time I measured, but it makes training on a CPU pretty lightweight, not memory heavy, and extremely affordable. That means we can pass that on to you, the people, and at least somewhat reduce the need for renting machines with GPUs for a low low $6,000 per day. And you can still train your model by shoving a bunch of data at us where we'll format it for you and give you a nice and neatly trained model-with-a-LoRA.

So in short, it's cool, it's a virtual TPU, it just does mathy sh**, and it's gong to be training models for you. Again it's literally just 2 dollars to make a LoRA and you have to do virtually no work for it except give it some trash formatted data that you want your model to either learn, focus on more, or some mix of both.

Despite this being a relatively small and, while not exactly simple, bounded system that does one thing and does it well, it's an important milestone for Hyperion Gray. Finally it's a tangible thing that's come out of the research we've done for the last year and a half or so, that reduces the reliance on massive GPUs and makes the tradeoff of efficiency and specialization over "shove some big hardware that way" engineering. So I proudly present to you GPUless. Oh right, it's not released yet, I'll post the link once it is in a couple of days.

Thanks for reading everyone.

• Alex aka _hyp3ri0n aka P4X aka The Dude