Arm Brings Transformers to IoT Devices

Find the Latest SoC Solutions
for... Automotive... IoT... Security... Audio... Video

NUREMBERG, Germany–The next generation of Arm's Ethos micro-NPU, Ethos-U85, is designed to support transformer operations, bringing generative AI models to IoT devices. The IP giant is seeing demand for transformer workloads at the edge, according to Paul Williamson, senior VP and general manager for Arm's IoT line of business, though in much smaller forms than their bigger brothers, large language models (LLMs). For example, Arm has ported vision transformer ViT-Tiny and generative language model TinyLlama-1.1B to the Ethos-U85 so far.

www.eetimes.com/, May. 04, 2024 – 

"Most machine learning inferencing is already being done on Arm-powered devices today," Williamson said. "It may seem like the AI explosion came overnight, but the truth is Arm's been preparing for this moment for a long time. The benefits of edge AI cut across a whole host of segments within IoT...AI needs tight integration between the hardware and the software, and Arm has invested heavily in the last decade."

Ethos-U85 features a third-generation microarchitecture. Versus the second-generation U65, U85 in its biggest configuration is 4× more performant and 20% more power efficient. It can now be driven by either Cortex-A application processor cores or Cortex-M microcontroller cores (previous Ethos generations were paired only with Cortex-M).

U85 NPU IP is configurable between 128-2048 MACs for 256 GOPS to 4 TOPS performance at 1 GHz, using INT8 weights with INT16 activations. INT8 activations are also supported.

Some applications will require INT16 activations for better prediction accuracy, Parag Beeraka, senior director of segment marketing for IoT at Arm, told EE Times.

"Audio use cases are one of the unique end markets where they want higher precision–customers are asking us to support 32-bit," Beeraka said. "For the imaging side it's the opposite, they want INT4 on the weights, and even 2-bit if you can do it. So it's a balance that we are trying to achieve."

Support for trendy shared exponent formats in future versions of the NPU is a "tricky" decision, Beeraka said, adding that Arm is looking into it but has not made a decision yet.

click here to read more...

 Back