Amazon on Monday, Oct. 19, announced TorchNeuron, software that Amazon says lets AI code written in PyTorch run on Amazon's own Trainium AI chips by changing one setting, with no rewrite. PyTorch is the free software most AI researchers and developers use to write their models; Amazon says 85% of them use it. Amazon says it will make TorchNeuron open source on Oct. 20 at the PyTorch Conference North America. That means it will be free for anyone to use, study, change and share, and developers can install a public test version from that day.
Nvidia dominates the market for AI chips partly because much of the industry's AI software was built around Nvidia's own programming tools, and adapting that code to other hardware can take months of engineering work. Until now, Amazon says, developers who wanted to run PyTorch code on Trainium had to change it to call programming tools made only for Amazon's chips. TorchNeuron plugs Trainium into the standard slot PyTorch already has for new hardware, so in Amazon's basic example, the one change is a line telling the code to run on Trainium, which it calls "neuron". Code that uses PyTorch's speed-up feature needs one more line. Amazon says the tools developers use to test, fix and scale up their models work without changes.
TorchNeuron works on Amazon's Trainium2 and Trainium3 chips. More than 30 companies and Amazon's own teams used it in a private test that began late last year. Deepgram, a speech AI company, says that in its tests Trainium processed about 60% more audio per dollar than its existing setup on GPUs, the advanced AI chips that AI models are trained and run on. That result came after an Amazon team spent several months helping Deepgram move its model to Trainium. Hugging Face, whose free tools many developers use to download and run openly shared AI models, has added Trainium support to them with Amazon.
The tool does not yet cover everything PyTorch does. When Amazon ran about 154,000 of PyTorch's own tests in August, it set aside about 39,000 as not yet suited to Trainium. More than half of those, about 22,000, involved calculations the software can't yet run directly on the chip. Amazon also left some features out of the first release, such as handling data that is mostly zeros, which some models store in a compact form, and data whose size changes while a program runs; Amazon says customers had not asked for them. Amazon says it plans to add the missing calculations in coming releases.
So far, Trainium's biggest users are AI labs with close financial ties to Amazon: Anthropic, in which Amazon has invested billions, and OpenAI, which agreed in February to use Trainium as part of a deal in which Amazon is investing $50 billion in OpenAI. It is not yet clear whether a simpler switch will bring many other developers over, and apart from Deepgram's test against GPUs, Amazon's announcement gives no new figures comparing Trainium's speed or cost with Nvidia's chips.