Why Your Next Laptop Will Have an NPU, Just Like Your Phone

If you’ve been thinking about purchasing a new laptop, you’ll no doubt have noticed that they’re increasingly coming with NPU capabilities that look an awful lot like the hardware we’ve seen in the best smartphones for several years now . . The driving factor is the desire for laptops to catch up with mobile AI capabilities, integrating them with advanced AI features, like Microsoft’s Copilot, that can run securely on the device without the need for an Internet connection. So here’s everything you need to know about NPUs, why your next laptop might have one, and whether or not you should buy one.

What is an NPU?

NPU is an acronym for Neural Processing Unit. NPUs are dedicated to performing mathematical functions associated with neural network/machine learning/AI tasks. While they may be standalone chips, they are increasingly being integrated directly into a system-on-a-chip (SoC) alongside more familiar CPU and GPU components.

NPUs come in different shapes and sizes and are often called by a slightly different name depending on the chip designer. You will already find different models scattered across the smartphone landscape. Qualcomm has Hexagon in its Snapdragon processors, Google has its cloud TPUs and mobile Tensor chips, and Samsung has its own implementation for Exynos.

The idea is now also taking off in the field of laptops and PCs. For example, there’s the Neural Engine in the latest Apple M4, Qualcomm’s Hexagon features in the Snapdragon X Elite platform, and AMD and Intel have started integrating NPUs into their latest chipsets. Although they’re not quite the same, NVIDIA GPUs blur the lines, given their impressive computing capabilities. NPUs are becoming more and more ubiquitous.

As we mentioned, NPUs are specifically designed to handle machine learning workloads (as well as other math-intensive tasks). Simply put, an NPU is a very useful, perhaps even essential, component for running AI on the device rather than in the cloud. As you may have noticed, AI seems to be everywhere these days, and integrating support directly into products is a key step in this journey.

Today, much of AI processing is done in the cloud, but this is not ideal for several reasons. The first is latency and network requirements; You can’t access offline tools or you may have to wait for long processing times during peak hours. Sending data over the internet is also less secure, which is a very important factor when using AI that has access to your personal information, like Microsoft’s Recall.

Simply put, running on device is better. However, AI tasks are computationally intensive and do not perform well on traditional hardware. You may have noticed this if you’ve tried generating images via Stable Diffusion on your laptop. This can be extremely slow for more advanced tasks, although the processors can perform a number of “simpler” AI tasks quite well.

The solution is to adopt dedicated hardware to accelerate these advanced tasks. You can learn more about what NPUs do later in this article, but the TLDR is that they perform AI tasks faster and more efficiently than your CPU can do on its own. Their performance is often expressed in billions of operations per second (TOPS), but this is not a terribly useful metric because it doesn’t tell you exactly what each operation does. Instead, it’s often better to look for numbers that tell you how quickly to process tokens for large models.

Speaking of TOPS, NPUs for smartphones and early laptops are categorized by dozens of TOPS. Broadly speaking, this means they can speed up basic AI tasks, such as detecting camera objects to apply bokeh blur or summarize text. If you want to run a large language model or use generative AI to produce media quickly, you’ll need a more powerful accelerator/GPU in the TOPS line.

A neural processing unit is very different from a central processing unit because of the type of workload it is designed for. A typical processor in your laptop or smartphone is versatile enough to meet a wide range of applications, supporting large instruction sets (functions it can perform), various ways of caching and recall functions (to speed up repeating loops) and large out-of-order execution windows (so they can keep doing things instead of waiting).

However, machine learning workloads are different and don’t require as much flexibility. For starters, they are much more math-heavy, often requiring repetitive, computationally expensive instructions, like matrix multiplication, and very fast access to large stores of memory. They also often operate on unusual data formats, such as sixteen, eight, or even four-bit integers. In comparison, your typical processor is built around 64-bit integer and floating-point math calculations (often with additional instructions added).

Building an NPU dedicated to mass parallel computing of these specific functions allows for faster performance and less energy wasted on idle features that are not useful for the task at hand. However, not all NPUs are equal. Even aside from their computational capabilities, they can be designed to support different integer types and operations, meaning that certain NPUs perform better on certain models. Some smartphone NPUs, for example, run in INT8 or even INT4 formats to save on power consumption, but you’ll get better accuracy with a more advanced but power-hungry FP16 model. If you need truly advanced computing, dedicated GPUs and external accelerators are still more powerful and more form factor diverse than integrated NPUs.

As a backup, processors can perform machine learning tasks, but are often much slower. Modern processors from Arm, Apple, Intel, and AMD support the necessary math instructions and some of the smallest quantization levels. Their bottleneck often lies in how many of these functions they can perform in parallel and how quickly they can move data in and out of memory, which is what NPUs are specifically designed to do.

While far from essential, especially if you don’t care about the AI ​​trend, NPUs are required for some of the latest features you’ll find in the mobile and PC space.

Microsoft’s Copilot Plus, for example, specifies as a minimum requirement an NPU with 40TOPS of performance, which you’ll need to use Windows Recall. Unfortunately, Intel’s Meteor Lake and AMD’s Ryzen 8000 chips found in current laptops (at the time of writing) do not meet this requirement. However, AMD’s recently announced Stix Point Ryzen chips are compatible. You won’t have to wait long for an x64 alternative to the Arm-based Snapdragon X Elite laptops, as Stix Point-powered laptops are expected in the first half of 2024.

Popular PC tools like Audacity, DaVinci Resolve, Zoom and more are increasingly experimenting with more demanding on-device AI capabilities. While not essential for basic workloads, these features are becoming increasingly popular and AI capabilities should be considered in your next purchase if you use these tools regularly.

When it comes to smartphones, features and capabilities vary a little more by brand. For example, Samsung’s Galaxy AI only works on its powerful flagship Galaxy S handsets. It hasn’t brought features like chat assistance or interpreter to the affordable Galaxy A55, likely because it doesn’t does not have the necessary processing power. That said, some of Samsung’s features also work in the cloud, but they’re likely not funded by more affordable purchases. Speaking of which, Google is equally poor in terms of feature consistency. You’ll find the best of Google’s AI extras on the Pixel 8 Pro, such as Video Boost. Still, the Pixel 8 and even the affordable 8a run many of the same AI tools.

Ultimately, AI is here and NPUs are the key to taking advantage of on-device features that can’t run on older hardware. That said, we’re still in the early days of AI workloads, especially in the laptop space. Software requirements and hardware capabilities will only increase in the coming years. In this sense, waiting for the dust to settle before getting started won’t do any harm.