New Frontiers for AI applications

With the release of the new System-on-Module architecture Jetson™ AGX Orin, manufacturer NVIDIA® has significantly stirred the market for hardware solutions. But how much influence will this new platform for scalable embedded and edge computing exert on the future?

Artificial intelligence will be ubiquitous around us in the near future. Already today, fully autonomous drones deliver packages to end customers or inspect agricultural fields for pest infestation and potential harvest yields. Automation in industrial sectors is progressing steadily, and the excitement is further fueled by new technologies like 5G and 6G.

Hardware manufacturers and IT distributors are trying to reduce the network load on already strained clouds and are utilizing decentralized edge computing solutions in close proximity to the application. As the world's leading platform for autonomous machines, NVIDIA® Jetson™ taps into the full potential of AI-powered mobile robots, drones, network video recorders, or optical inspections. The announcement of the Jetson AGX Orin™ modules in September 2022 raises the question of how the new architecture will affect future hardware systems.

AI performance at server level

NVIDIA® Jetson™ comprises a series of System-on-Modules (SOM) including processor, graphics card, memory, power management, high-speed interfaces, and more. The systems vary in form factor, features, and energy efficiency to cover different industries. The newly introduced Jetson Orin™ modules offer up to 275 TOPS (Tera Operations per Second) in the highest model class, equivalent to 8 times the AI performance potential of the Xavier™ generation.

The integrated Ampere GPU consists of 2 Graphics Processor Clusters (GPC), up to 8 Texture Processor Clusters (TPC), and up to 16 Streaming Multiprocessors (SM) with 192KB of L1 cache and 4MB of L2 cache. Additionally, 128 CUDA cores per SM are supported, giving the 64GB version of the Orin™ module a total of 2048 CUDA cores and 64 Tensor cores.

The NVIDIA® Deep Learning Accelerator (DLA) is a fixed-function accelerator optimized for deep learning operations. It is designed for full hardware acceleration of convolutional neural network inference. The Orin™ SoC offers support for the next generation of NVDLA 2.0 with nine times the performance of NVDLA 1.0 and a more energy-efficient architecture.

The GPU and DLA optimizations elevate Orin™'s AI performance to server level. Meanwhile, the demands on computing power are increasing exponentially for functions such as multi-sensor perception, mapping and localization, planning and control, as well as situational awareness and security. Particularly, robotics and other edge AI applications require more and more resources for computer vision and conversational AI.

Functional block diagram of the Jetson AGX Orin™ series (Image source: NVIDIA Corporation)

The Jetson AGX Orin™ module delivers up to 3.3 times the performance of Jetson AGX Xavier™ in real AI applications (measured in inferences per second), as pre-trained models already demonstrate. Based on this, this performance could be increased by almost 5 times with future software updates.

Use cases and hardware in the pipeline

With Orin™ developer kits, companies can realize complex and future-oriented AI applications at the edge. These include areas such as Industrial IoT, logistics and transportation, agriculture, or healthcare. Through the compatibility of cloud-native workflows and AI software already used on predecessor platforms, the new modules enable companies to achieve additional performance and energy efficiency to develop mobile robots for various everyday tasks. Autonomous machines are already sorting goods in logistics halls or autonomously serving customers in retail. Compact Jetson™ hardware also enables real-time automated access controls for hazardous areas, where employees are only allowed to enter with helmets and safety vests.

Demand for the new modules exists and will continue to rise, considering the price-performance ratio. Manufacturers and system houses have recently set up a tight schedule, and in the coming months and years, embedded PCs and GPU computers based on Orin™ SoCs will reach market maturity. Hardware manufacturers for freely configurable edge intelligence solutions like AI-BLOX have been reporting on novel processor modules with Jetson AGX Orin™ since December 2022. Further models with Orin NX™ and Orin Nano™ are expected to follow soon. NASDAQ-listed One Stop Systems plans to introduce a compact dual-Orin system in a ruggedized chassis in collaboration with system house BRESSNER Technology in 2023. The high-performance computing experts are leveraging synergies to tap into the European market for industrial supercomputers.

The release of the new NVIDIA® Jetson™ AGX Orin platform will undoubtedly have a significant impact on the future of edge computing. The combination of performance and scalability meets the requirements that industrial companies place on modern AI applications: classification, object recognition, text-to-speech, speech translation, recommendation services, sentiment analysis, and much more. Artificial intelligence will increasingly become part of our everyday lives in the near future, and the Jetson AGX Orin™ platform will play a key role in this development.