Google has made the TPU Raiden library open source for transferring temporary KV memory between chips while running large language models, in a move that expands its TPU ecosystem tools beyond its closed scope.
Article contents
- What is the TPU Raiden library?
- How does Raiden manage prefill and decoding phases?
- Inter-chip transfer and memory offloading
- Why is Google opening these tools now?
- An open race for inference architecture
- Frequently asked questions
What is the TPU Raiden library?
Google published the TPU Raiden library on GitHub under the Apache 2.0 license. The library operates in an inference ecosystem layer similar to the one covered by Nvidia’s NIXL library, providing mechanisms to transfer KV Cache memory between processing units responsible for the prefill and decode phases, alongside tools for offloading temporary memory data.
The SemiAnalysis platform was among the first to draw attention to the release, considering it a clear signal of Google’s readiness to make available infrastructure components that had long been tied to its proprietary TPU units ecosystem.
How does Raiden manage prefill and decoding phases?
Large-scale inference is typically split into two phases. The Prefill phase processes the prompt or input text, while the Decode phase handles generating tokens one by one.
Running both phases on the same chip can be less efficient, which is why production systems separate them and distribute them across independent hardware clusters. After the prefill phase finishes, the KV Cache is formed, representing the operational memory of the context processed by the model, and it must be quickly transferred to the chips responsible for generation.
Raiden handles this transfer process to facilitate running disaggregated inference across TPU units.
Inter-chip transfer and memory offloading
According to the repository documentation, Raiden includes modules for direct inter-chip transfers within the same device, network-based transfers between virtual machines, and offloading KV memory blocks from TPU memory to host RAM.
The library also provides a shared memory mode that allows temporary data to remain in DRAM even when restarting the model server, which can be useful during routine operational updates.
However, Google emphasizes that the project is still under active development and is not currently intended for general use in production environments.
Why is Google opening these tools now?
Google does not sell TPUs as standalone products, but the user base for its custom chips has expanded. Anthropic gained access to one million TPUs, and OpenAI has also started using Google’s units for parts of its operations.
These customers need software tools that match the ecosystem built around Nvidia units. Frameworks like vLLM and SGLang are closely tied to the NIXL library, so similar integrations with Raiden in the future could help reduce the effort required to migrate certain inference workloads to TPUs.
An open race for inference architecture
The release of Raiden coincides with similar moves from Nvidia. On August 4, the company announced the opening of cuFile interfaces and its storage ecosystem through a new multi-vendor foundation on GitHub, listing Google, Intel, and Meta among the early supervisors.
These steps reflect a broader trend toward making inference architecture components previously kept by vendors within closed ecosystems available. For Google, open tools could lower the barriers to TPU adoption in production workloads, rather than choosing the Nvidia ecosystem simply because its tools are more widespread and documented in open sources.
In contrast, Nvidia asserts that its GPUs remain advanced and capable of running various artificial intelligence models across computing environments. The extent of the impact of Google’s ecosystem openness on this superiority will remain a key question for organizations choosing inference deployment architecture at scale.
Frequently asked questions
What is the function of the TPU Raiden library?
It transfers temporary KV memory between TPU units responsible for the prefill and decoding phases, and also supports offloading data to host memory.
Is Raiden a direct alternative to the NVIDIA NIXL library?
Both libraries operate in a similar layer of the inference ecosystem, but Raiden is targeted at TPUs and is still in active development.
Can Raiden be used in production systems right now?
Google does not currently recommend general use in production environments because the project is still under development.
What is the license for the Raiden library?
Google released the open-source library under the Apache 2.0 license.
