At a recent event celebrating both its Copilot product and the company's 50th anniversary, Microsoft introduced a new update to its Windows and mobile applications called Copilot Vision. This development is being described as a significant leap in the integration of artificial intelligence with visual capabilities, designed to assist users in navigating their digital environments more effectively.
During the unveiling, Microsoft highlighted the capabilities of Copilot Vision, which functions by identifying objects through a device's camera and providing contextual assistance. This update relies on a combination of Microsoft AI (MAI) and OpenAI's GPT generative models, enhancing the software's memory, search functionality, personalisation, and visual understanding.
Upon activating Copilot Vision via the dedicated eyeglasses icon within the Copilot interface on a Windows Desktop app, users can see a list of currently open applications. For instance, during a demonstration, users were working with two applications: Blender 3D and Clipchamp. Although the Copilot can discern which applications are in use, it does not continuously monitor user activities, preserving a degree of privacy.
One of the key features showcased was Copilot's ability to comprehend users' context based on the specific applications they were operating. For example, when the team opened Blender 3D with an ongoing project focused on designing a coffee table, they posed a question about how to infuse a more traditional aesthetic into their design. Despite minimal details provided, Copilot successfully generated a relevant response, demonstrating its contextual awareness.
Similarly, the team sought guidance on adding annotations within the same application. Copilot not only began to respond but also seamlessly redirected to the pertinent icon when the question was refined further. This targeted assistance serves to streamline workflows, enabling users to maintain their focus without needing to pause for extended searches or explanations.
As part of the ongoing development in Copilot Vision, the capabilities are set to expand with future updates. In another instance, when using the Clipchamp application, users queried how to enhance video transitions. Instead of a generic text response, Copilot Vision illustrated the location of the transitions tool with an animated arrow on the screen, guiding the user through the necessary steps. While this feature is still in the trial phase and did not consistently function as intended, it signals a potential shift in interaction paradigms with software applications in Windows.
Demonstrations also included sight of Copilot Vision delving deeper into Adobe Photoshop, highlighting relevant tools to assist users in their creative tasks. This evolution in AI support suggests that in the future, users could verbally or textually interact with their applications and receive guided assistance directly within the context of their projects.
Currently, users can access the preliminary version of Copilot Vision that identifies the apps and projects they are working on; however, more advanced features that will allow for direct interaction and guidance through tasks do not yet have a defined release schedule. Nevertheless, the anticipation surrounding these capabilities reflects a broader enthusiasm for the role of AI in enhancing productivity and user experience in digital workplaces.
Source: Noah Wire Services