GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

WASTE — Weight-Aware Streaming Tensor Engine Kimi K3 — 2.78 trillion parameters — running on a consumer laptop. $ waste run ~/models/k3.waste 'What is the capital of Italy?' waste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB) The capital of Italy is **Rome**. [16 tokens, 31.09 s, 0.51 tok/s | experts 3357 hit / 20195 miss = 14%] WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache. Its current proof point is the complete open-weights Kimi K3 model: 2.78 trillion parameters, converted into a 982 GiB container and running on a 64 GB MacBook Pro at 0.49–0.54 tokens per second. This is not a distilled, pruned, or reduced variant. Model Container Minimum RAM Tested speed Kimi K3 2.78T 982 GiB 29.05 GiB 0.49–0.54 tok/s Kimi-Linear 48B 19 GiB 1.87 GiB 10.7 tok/s WASTE was written for that one model and that one constraint: K3 does not fit in the RAM of current mainstream consumer systems. It is 1.42 TB as published and 982 GB after conversion. But a mixture of experts activates about 4% of itself per token, so almost all of that weight is idle at any instant — and idle weight does not need to be in memory, it needs to be reachable in time. WASTE keeps it on disk in a layout where one expert costs exactly one read, streams what each token actually needs, and spends every remaining byte of RAM on the part that repeats. Where this stands The engine is correct: every layer is validated against a PyTorch reference, the final logits agree to 3.6e-06, and the vision tower matches its own oracle to 2.3e-06. It is also slow — half a token per second, thirty seconds for the sentence above. Both of those matter, and the second one should not be read as a disclaimer. We are not aware of another published demonstration of a model this size streaming from disk on a consumer machine: we found none for trillion-scale NVMe streaming, and the best-documented 671B-class recipes assume a server with a terabyte of DDR5. That is a report of what our search turned up rather than a survey — this repository carries no bibliography and no comparison table, so read it as an invitation to send a counter-example, not as a result. The interesting part is not the speed, it is that the whole thing is in the reachable range on a single consumer machine — and that from here the question is engineering rather than feasibility. Where the levers were is not where they are. Overlapping the expert reads with the arithmetic was worth ~1.6x and shipped; the two that looked bigger — reading fewer bytes per token, and keeping more of them in RAM — were both measured and both refused, one because this family's router has no tail to demote and one because a cache the machine will not leave resident cannot be bought at any price. Even with the reads overlapped they are still 55% of a decode step against the arithmetic's 27%, so what is left is a faster disk or a machine with more RAM, not another pass over the kernels. docs/EFFICIENCY.md is the account of how each of those was priced, including the two that were built before being measured. What that opens up, concretely: a frontier-scale model that answers with no network, no per-token invoice, and nothing leaving the machine — which is the difference between "you may not send that data to an API" and "run it here". The format and the engine are not K3-specific in any deep way; K3 is simply the hardest case that exists today, and a model that streams at 2.78T streams comfortably at 48B. Every number in this document was measured on the commit it is published with, and the ones that were wrong are recorded as wrong in docs/LEARNED.md rather than quietly corrected. Why the name Every token answered by a cloud service is paid for twice: once on the invoice, and once in the electricity of a datacenter running a model that would fit — barely, awkwardly, but genuinely — on hardware already sitting on a desk. WASTE means to be the first concrete step toward ending that waste of tokens. The acronym came second. What you need disk, for the model 982 GB for the converted container — plan a terabyte disk, to convert it another 1.42 TB of staging for the published shards, freed afterwards RAM 29.05 GB minimum to open K3 at 4K context; 64 GB for the numbers here storage speed the container must be on internal NVMe — see below build a C11 compiler and make. No BLAS, no CUDA, no Python at run time Sizes here are powers of two, the way df and the engine both report them: the container is 982 GiB, which a disk vendor would call 1.05 TB. The RAM floor is what the engine refuses to start below, and it is almost entirely the 27.28 GB resident trunk. Useful throughput starts higher: on a 64 GB machine the engine gives itself a 46 GB budget, of which 17.56 GB is expert cache, and that is the top of the measured curve. A 32 GB machine can technically open the model and will page badly; treat 64 GB as the real requirement. Storage speed is not a detail. A token reads 17 GB of experts. On the internal SSD that is 12.78 GB/s and the model streams; over a USB enclosure it is 0.94 GB/s and the same token takes thirteen seconds. Convert onto internal NVMe, and use the external disk for the download only. If a terabyte is not available, the same engine and the same format run Kimi-Linear-48B-A3B-Instruct from a 19 GB container with a 1.87 GB floor, at 10.7 tok/s. That is the good path for trying WASTE out before committing a disk to K3. What it is Self-contained. One libwaste.a, one waste binary, nothing at run time beyond libc and pthreads. Zero dependencies. No BLAS, no ONNX, no Python in the inference path, nothing to install. The Python under tools/ converts models and validates the engine; it never runs alongside it. Fully embeddable. Twenty-six public functions in src/waste.h: open a model under a RAM ceiling, generate, save the session, close. The CLI is a client of that API and touches nothing private — if the CLI can do it, so can an embedding host. waste_cfg cfg; waste_cfg_init(&cfg); cfg.ram_budget_bytes = 46ULL << 30; /* a hard ceiling, not a hint; 0 sizes it to this machine */ waste_ctx *ctx; if (waste_open("/path/to/k3.waste", &cfg, &ctx) != WASTE_OK) return 1; waste_generate(ctx, ids, n, &params, on_token, user); waste_close(ctx); The path is the container directory the converter wrote — no ~ expansion here, that is the shell's job. How it works Placement decides the speed A model is converted once into a .waste container: a JSON manifest, a resident trunk, and one expert bank per layer. Each expert record is 4 KiB-aligned with its gate, up and down matrices adjacent, so routing to an expert costs exactly one pread — not three, not a seek per matrix. The arithmetic was never the bottleneck. Reads bypass the page cache (F_NOCACHE on macOS, O_DIRECT on Linux, FILE_FLAG_NO_BUFFERING on Windows). That is deliberate: with a container smaller than RAM the kernel would cache everything, and the hit rates measured that way are a fiction that does not survive contact with a 982 GB model. Every record's header is checked on the way in — right magic, the expert the index asked for, offsets that fit — so a bank that has been truncated or spliced stops the generation and names the record instead of answering from the wrong bytes. That costs nothing measurable. The record also carries a crc32 over its payload, and checking that is --verify, off by default: it is a pass over every record on every cache miss, about 5% on Kimi-Linear and 1% on K3. Worth it for a container you copied or downloaded and have not read since; not worth it on every token of one you converted yourself. See docs/FORMAT.md. Three bits per expert weight Experts are stored as residual vector quantization — three stages of 256-entry codebooks over 8-dimensional vectors, 3.00 bits per weight — and the matrix is never materialized. For each token the engine builds a table of partial dot products, one per codebook entry per vector position, after which every expert row is three table reads and two adds. The trunk stays at 4 and 8 bits. The model was trained with quantization-aware training on the experts only, so it has no trained tolerance for a squeezed trunk: a 3-bit trunk was built and measured, the cache prediction held, the throughput did not, and the output collapsed. The cache floor is one token's working set The most predictive number in this project. K3 touches 16 experts in each of 92 layers per token: 17.0 GB. Below that, an expert cached for one token is evicted before the next token asks for it, and the hit rate is not low — it is zero. Above it the curve bends sharply. budget expert cache hit rate decode 32 GB 3.32 GB 0% 0.31 tok/s 46 GB 17.32 GB 13% 0.32 tok/s 52 GB 23.32 GB 27% 0.11–0.14 tok/s 58 GB 29.32 GB 37% 0.04 tok/s Measured in that order, on an otherwise idle machine. Order matters: re-run after the 52 and 58 GB rows have driven the machine into paging, 46 GB gives 0.22–0.25 rather than 0.32 — while reporting hit and miss counts identical to the digit. The engine is deterministic; the machine is not, and it does not fully recover between runs. Sweep upward. The decode column predates read-ahead and has not been re-swept: 46 GB now runs at 0.51 rather than 0.32. The shape is what the table is for, and read-ahead does not move it — it hides I/O behind arithmetic, which makes every row faster and none of them a different budget. Everything in the memory design exists to get above that line, which is why the engine works to free RAM rather than to save it. And there is a ceiling on the other side, closer than it looks. Read that table twice: the hit rate climbs all the way down. At 58 GB on a 64 GB machine the cache serves 37% of experts from RAM and the engine is eight times slower than at 46 GB, where it serves 13%. The engine is inside its budget; the