Training on the Edge: A Model Pretrained on an Inference-Only NPU
aimachinelearningedgeainpu
TL;DR
Edge AI has had a clean division of labor: NPUs serve, GPUs train. A new open model, VKORE-46M-NPU, breaks it. It was pretrained from scratch, with real backpropagation, on a first-generation inference-only NPU, no GPU involved. Trained under identical conditions on a GPU, the loss trajectories correlate at 0.99. For anyone building on-device or on-prem, this changes what a small, low-power box can do.
Why edge teams should care
The appeal of NPUs for edge AI has always been efficiency: low power, low cost, installable in an office or a device rather than a data center. The catch was that you could only run models on them, not build or adapt them. Training still meant renting or buying GPUs, which pulls data and work back to the cloud or a central cluster.
If an inference-only NPU can also train, the edge box stops being just a serving endpoint. The same chip you deployed for low-power inference can fine-tune or even pretrain during idle hours, on-site, without shipping sensitive data anywhere.
What was actually shown
- Model: VKORE-46M-NPU, a 46M decoder-only transformer (RoPE, SwiGLU, RMSNorm, GQA), character-level, trained on the public TinyStories dataset.
- Hardware: a 2025 first-generation inference-only NPU, 32 chips.
- Result: from-scratch pretraining with backprop on the NPU, loss trajectory correlating 0.99 with a matched GPU run, and comparable throughput.
- Also confirmed: inference acceleration (3.8x single-user, 4.1x server throughput) and quantized-model execution on general open models.
The capability is packaged as VKORE, a hardware-independent acceleration layer the team calls "CUDA for NPUs."
The edge implication
Put the pieces together for an edge or on-prem deployment:
- Serve by day on the NPU.
- Fine-tune or pretrain by night on the same NPU.
- Keep data in the building the whole time.
That is a different cost and privacy story than "serve locally, train in the cloud." The released model is deliberately small, a proof that the training path runs on this hardware, not a quality benchmark. Scaling to larger models and real data is the open question that matters next.
Honest limitations
46M, character-level, TinyStories is toy scale. The core training technique is patent-pending and the training code is private. What is public and reproducible is the model, the card, and the NPU-vs-GPU comparison.
Model: https://huggingface.co/FINAL-Bench/VKORE-46M-NPU