Where Rust actually fits in the Python AI stack
Rust is taking over the systems code around Python AI, not the GPU math. What's driving the shift, and where C++ and compilers still lead.
“Rust is replacing C++ in AI” is a headline that keeps coming back. Look at where Rust code is actually landing, though, and the picture is more specific: Rust is taking over the new systems code that sits right next to Python, while the GPU math underneath it barely moves. Here’s where Rust is winning and why, what memory safety does and doesn’t buy, and why the GPU layer is changing for a different reason.
Python decides, native code does the work
Almost nobody runs AI workloads in pure Python. Python is where you describe what should happen: load this dataset, tokenize it, run this model. The work itself happens in native libraries underneath. Tokenizing, decoding files, shuffling data and multiplying matrices all run in compiled code, and Python only coordinates.
So the useful question isn’t whether Rust replaces Python. It’s which native language each layer under Python should be written in. The answer differs by layer, because each layer carries a different kind of risk.
Where Rust is winning
Three widely used pieces of the Python AI toolchain have Rust at their core:
- Tokenizers. Hugging Face’s tokenizers library is written in Rust and ships bindings for Python, Node.js and WebAssembly.
- Model files. safetensors is a format for storing and sharing model weights, built with Rust. Its README sums up the problem with pickle in four words: “Unsafe, runs arbitrary code.” Loading a pickled model file can execute whatever code is inside it. Loading a safetensors file reads tensors and nothing else.
- Dataframes. Polars describes itself as “an analytical query engine written in Rust.”
They share four traits. Each is new code, so there’s no legacy C++ to protect. Each parses input that someone else produced, from text to model files from the internet to messy datasets. Each is expected to use every core on the machine. And each needs to feel native from Python, which PyO3 makes cheap: you write the core in Rust and expose it as an ordinary Python module.
That combination of untrusted input, parallelism and no legacy is exactly where Rust’s guarantees pay off. In a model loader, a memory bug is a way for a malicious file to take over the process.
| Layer | What’s winning | Why |
|---|---|---|
| Tokenizers, model formats, dataframes | Rust (tokenizers, safetensors, Polars) | New code that parses untrusted input and runs in parallel |
| New Python extensions | Rust via PyO3 is a strong fit | Thread safety is checked at compile time |
| GPU kernels | CUDA C++ libraries, increasingly generated by compilers such as Triton | Vendor libraries and compilers carry the performance work |
| Model code and experiments | Python | Iteration speed matters more than anything else |
What memory safety buys, and what it doesn’t
The strongest argument for Rust is a statistic that shows up in two very different codebases. In 2019, Microsoft reported that about 70% of the vulnerabilities it assigns a CVE each year are memory safety issues (MSRC). The Chromium team reports that “around 70% of our high severity security bugs are memory unsafety problems (that is, mistakes with C/C++ pointers)” (Chromium). These are codebases written and reviewed by experienced engineers. Care doesn’t make the problem go away.
In safe Rust, the compiler rejects the code patterns behind those bugs, such as reading memory after it has been freed, or two threads writing the same data without coordination. The program doesn’t build until the problem is fixed.
Two claims that often ride along with this argument don’t hold up:
- Rust doesn’t prevent memory leaks. The Rust Book says so plainly: “Preventing memory leaks entirely is not one of Rust’s guarantees, meaning memory leaks are memory safe in Rust” (The Rust Book). And modern C++ already handles most cleanup automatically through smart pointers and RAII, which ties freeing a resource to the end of the object that owns it.
- Rust isn’t inherently faster than C++. Both compile to native code through optimizing compilers. Well-written Rust and well-written C++ land in the same range.
The case for Rust, then, isn’t speed and isn’t leaks. It’s removing a whole class of bugs that careful engineers still ship.
Free-threaded Python raises the stakes
For most of Python’s history, the global interpreter lock (GIL) let only one thread run Python code at a time. C extensions could lean on that: if only one thread runs at a time, shared data can’t be corrupted by two at once.
That assumption is starting to change. Since Python 3.13, CPython offers a separate free-threaded build with the GIL disabled (Python docs). In Python 3.14 that build became officially supported (What’s New in Python 3.14), but it’s still optional: making it the default is a later step that PEP 779 leaves to a future decision. It also has a cost: single-threaded code runs roughly 5-10% slower in free-threaded mode. The GIL isn’t gone; it’s become a choice.
The migration is gentler than it sounds. When a free-threaded interpreter imports an extension that hasn’t declared support for free threading, it re-enables the GIL and prints a warning instead of letting the extension break. You can check what actually happened at runtime:
import sys
# False only when the GIL is actually disabled in this process
print(sys._is_gil_enabled())So an old extension falls back to the old behavior instead of crashing. But anyone writing a new extension today has to decide how it behaves under real parallel threads. Rust answers part of that question before the code ships: PyO3 has supported free-threaded builds since version 0.23, and builds on Rust’s Send and Sync traits to aim for what it calls “fearless concurrency,” where the compiler rejects data shared unsafely between threads.
That’s migration pressure rather than a cliff, and it favors Rust for new extension code.
The GPU layer is changing differently
The compute core of AI, the kernels that multiply matrices on GPUs, is a different story. Even Rust’s own machine learning frameworks lean on NVIDIA’s libraries here. Hugging Face’s Rust framework candle runs on GPUs through a CUDA backend, uses NVIDIA’s cuDNN library for extra speed, and uses NVIDIA’s NCCL library to spread work across GPUs. Compiling Rust straight to GPU code is real but early: rust-gpu compiles Rust to SPIR-V, the GPU code format used by Vulkan, and says it is “not yet production-ready.”
What’s actually reducing hand-written CUDA C++ is compilers. Triton is “a language and compiler for parallel programming” that gives you a Python-based way to write custom GPU kernels. PyTorch’s torch.compile uses TorchInductor as its default compiler, which generates Triton code for NVIDIA, AMD and Intel GPUs and C++ for CPUs (PyTorch). More and more kernels are written by a compiler rather than by hand.
Mojo is another contender for this layer, with Python-like syntax. It isn’t a way to avoid ownership rules, though: it has its own ownership model and a lifetime checker. It’s too early to say how much of the kernel layer it will claim.
Today, moving away from hand-written CUDA C++ mostly means adopting compilers that generate kernels, not rewriting kernels in Rust.
Key takeaways
- Rust is winning the new systems code at Python’s boundary: tokenizers, model formats, dataframes and extensions.
- The reasons are untrusted input, parallelism and no legacy to port, with PyO3 making the Python side cheap.
- Memory safety removes use-after-free bugs and data races at compile time. It doesn’t prevent leaks or make code faster than good C++.
- Free-threaded Python is optional and supported, not the default. Extensions that haven’t declared support fall back to the GIL instead of crashing.
- GPU kernels are moving from hand-written CUDA C++ to compilers like Triton, not to Rust.
Conclusion
The AI stack isn’t converging on one language. It’s sorting itself by risk. Python states intent. Rust takes the new boundary code that handles untrusted input and parallel work. C++, CUDA and kernel compilers keep the math. I expect that layering to hold for years.
One more reason I expect the Rust side to grow: a growing share of code is now written with AI assistance. A compiler that refuses to build code that shares memory unsafely is a cheap first reviewer for code no human typed.