Microsoft wants to bring generative AI to the forefront of Windows and the PCs that run it.
In two talks at its annual Build developer conference this week, the company unveiled a new line of Windows machines it calls Copilot+ PC, as well as AI-driven generative features like Recall, which help users find apps, files and other content they have. seen in the past. Copilot, Microsoft’s generative AI brand, will soon be much more deeply integrated into the Windows 11 experience. And new Microsoft Surface devices are on the way.
We’ve rounded up all the major announcements from Monday and Tuesday here.
Volumetric applications
Microsoft offers Windows Volumetric applications – essentially interactive, spatially aware VR applications – to Meta Quest headsets. Through a partnership with Meta, Microsoft says it will deliver Windows 365 and local PC connectivity to Quest headsets, allowing developers to extend their applications into the 3D space.
During Tuesday’s keynote, Microsoft showed off a 3D digital exploded view of an Xbox controller from the perspective of a Meta Quest 3 headset – a digital object that the user could manipulate with their hands. “We’re deepening our partnership with Meta to make Windows a best-in-class experience on Quest devices,” Pavan Davuluri, vice president of Windows and devices at Microsoft, said during the demo.
Developers can sign up for a preview to gain access to Microsoft’s new volumetric API.
PC Copilot+
Copilot+ PCs represent Microsoft’s vision of flagship AI-driven Windows hardware. All include dedicated chips called NPUs to power AI experiments like Recall. And they come with a minimum of 16GB of RAM, paired with SSD storage.
The first Copilot+ PCs will feature Qualcomm’s Snapdragon X Elite and Plus chips, which Microsoft says offer up to 15 hours of web browsing and 20 hours of video battery life. Chipmakers Intel and AMD are also committed to creating processors for Copilot+ devices in partnership with a number of manufacturers, including Acer, Asus, Dell, HP, Lenovo and Samsung.
Copilot+ PCs start at $999 and some are available for pre-order today.
Surface Pro and Surface Laptop
Microsoft’s recently unveiled Surface devices, the Surface Laptop and Surface Pro, focus on performance and battery.
The latest Surface Laptop – available with a 13.8- or 15-inch screen – has been redesigned with “modern lines” and thinner screen bezels. It lasts up to 22 hours on a charge and is up to 86% faster than the Surface Laptop 5, according to the company. It also supports Wi-Fi 7 and has a touchpad with haptic feedback.
As for the new Surface Pro, Microsoft claims that it is up to 90% faster than the previous generation Surface Pro (the Surface Pro 9), and that it sports a new OLED with HDR display, Wi-Fi 7 (and optional 5G) and an improved ultra-wide front camera. Additionally, its detachable keyboard – which has been reinforced with additional carbon fiber – now features haptic feedback.
Reminder
Windows 11’s upcoming Reminder feature can “remember” which apps and content a user accessed on their PC weeks or even months ago, helping them, for example, find a Discord chat where they were discussing clothes he was planning to buy. Users can use Recall’s timeline to “scroll back” to see what they have worked on in the recent past and explore files such as PowerPoint presentations to surface information potentially relevant to their research.
Microsoft says Recall can create associations between colors, images and more to let users search for virtually anything on their PC in natural language (much like startup Rewind’s technology); Developers will be able to improve Recall by adding contextual information to their applications. And Microsoft says all user data associated with Recall remains private and on the device – and is not used to train AI models, which is important.
Here’s more from Microsoft: “Your snapshots belong to you; they stay locally on your PC. You can delete individual snapshots, adjust and delete time ranges in Settings, or pause at any time directly from the system tray icon on your taskbar. You can also filter apps and websites so that they are never recorded.
Image editing and live translations
There is now more AI than ever in Windows and some exclusively on the new Copilot+ PCs.
A new feature called Super Resolution can restore old photos by automatically scaling them. And Copilot can now analyze images to give users ideas for creative compositions. Through a feature called Cocreator, users can generate images and also have the AI model follow what they draw to modify or restyle the image.
Elsewhere, live captions with live translations translate any audio passing through a PC – whether from YouTube or a local file – into the user’s chosen language. Live translations will initially support around 40 languages, including English, Spanish, Mandarin, and Russian.
A separate but related new feature in Microsoft Edge offers real-time video translation on sites like LinkedIn, YouTube, Coursera, Reuters, CNBC, Bloomberg and more. Expected to be available in the near future, the feature – which supports translation from Spanish to English and from English to German, Hindi, Italian, Russian and Spanish – translates spoken content via live dubbing and subtitles.
Team Copilot and extensions
Team Copilot is the latest expansion of Microsoft’s growing Copilot suite of generative AI technologies. It integrates with Teams, the company’s video conferencing app, to help manage meeting agendas and take notes that anyone in a meeting can co-author. And that extends to Loop and Planner, Microsoft’s collaboration and planning platforms, for creating and assigning tasks, tracking deadlines, and notifying team members when their input is needed.
In somewhat related Copilot news, Microsoft has launched (in private preview) Copilot Extensions, which allows developers to extend GitHub’s code generation tool, GitHub Copilot, with third-party apps and skills . Launch partners include DataStax, Docker and LambdaTest; The extensions will live on the GitHub marketplace, but developers will also be able to create their own private extensions to integrate with their internal systems and APIs.
Windows Copilot Runtime
Features like recall and super resolution are powered by Windows Copilot Runtime, a collection of about 40 generative AI models that make up what Microsoft describes as “a new layer” of Windows. In tandem with the Semantic Index, a local vector system on an individual Copilot+ PC, the Windows Copilot Runtime allows AI-driven generative applications, including third-party applications, to run without necessarily requiring a connection Internet.
“(The runtime) consists of ready-to-use AI APIs such as Studio Effects, live subtitle translations, OCR, recall with user activity and (more) , which will be available to developers in June,” Davuluri said on Tuesday.
Microsoft says CapCut, the popular video editor from TikTok owner ByteDance, will use Windows Copilot Runtime and the new accompanying Windows Copilot Library, a set of APIs and AI development tools, to accelerate its AI features. And Meta will add the aforementioned Studio effects to WhatsApp to offer features like background blur and eye contact during video calls.
Improved Robot Builders
Azure AI Studio, the toolset from Microsoft’s Azure OpenAI service that allows customers to combine an AI model and create an application that “reasons about” that data, will soon allow developers to create applications using it. Using paid inference APIs – APIs through which developers can access and refine generative AI models hosted on Azure infrastructure. Microsoft calls this “model as a service” and it’s launching with models from Nixtla and Core42 to start.
In the adjacent Copilot Studio product suite, Microsoft is launching Copilot Agents, which the company describes as AI bots that can “independently orchestrate tasks tailored to specific roles and functions.” (Copilot Studio provides tools to connect Copilot for Microsoft 365, the AI-powered “copilot” in apps like Excel and Word, to third-party data.) Leveraging memory and context awareness, agents Copilot can navigate different types of business workflows, learn from user feedback, and ask for help when they encounter situations they don’t know how to handle.
Snapdragon SDK
There is a new SDK from Qualcomm aimed at developers creating applications for Copilot+ Arm-chip-packing PCs.
The $899.99 Snapdragon Dev Kit for Windows – which incidentally measures about the same width, height and length as Apple’s Mac Mini – houses Qualcomm’s Snapdragon X Elite chip paired with 32GB of RAM, 512 GB of storage and plenty of I/O. The Dev Kit supports Wi-Fi 7 and Bluetooth 5.4 and, via its various USB-C and HDMI ports, it can drive up to three 4K monitors at once.
Phi-3
Microsoft announced an addition to its Phi family of generative AI models, Phi-3-vision, capable of performing general visual analysis and reasoning tasks, such as answering questions about graphics and images. The model can read both text and images and is efficient enough to work on a mobile device.
Phi-3-vision is available in preview, while the previously announced text-only counterparts – Phi-3-mini, Phi-3-small and Phi-3-medium – are now generally available.
Partnership with Khan Academy
Microsoft is partnering with Khan Academy to provide access to cloud computing infrastructure, enabling Khan Academy to offer teachers in the United States free access to Khan Academy’s AI-powered tools. The two companies will also collaborate to explore opportunities to improve AI applications for mathematics education using generative AI, Microsoft announced Tuesday.
We are launching a newsletter on AI! Register here to start receiving it in your inboxes on June 5.