Microsoft is about to add more capabilities to the consumber version of Copilot including a new way we interact with it Copilot through its latest feature, Copilot Vision.
Is this a privacy step to far? or is this simply leveraging the power of vision to bring new experience and assistance to users.
Designed to enhance the capabilities of Microsoft’s Copilot AI assistant by allowing it to see and understand the same content as the the user is seeing on scene is said to bring a new dimension to how we work with AI tools.
What is Copilot Vision?
Copilot Vision is an extension of Microsoft’s Copilot AI assistant that enables it to visually perceive the content on your screen. Whether you’re browsing websites, reading documents, or viewing images, Copilot Vision can interpret and respond to the visual data, making it a more intuitive and helpful assistant.
Examples of Copilot Vision in Action
- Travel Planning: Instead of manually searching for travel options, Copilot Vision can provide recommendations and answer questions based on the travel websites you’re viewing.
- Recipe Adjustments: If you’re looking at a recipe online and want to make substitutions, Copilot Vision can suggest alternatives and cooking tips without needing to switch tabs or open a new search.
- Document Assistance: While working on a document, Copilot Vision can offer suggestions, corrections, and additional information relevant to the content you’re editing.
All this happens without you having to copy and paste stuff to the Copilot chat interface so it’s really like having eyes on what you are doing (once turned on of course). It’s like having your assistant working and viewing with you so it can see exactly what you see on your screen, whether it’s a website or a document.
That’s what Copilot Vision does. It can read both typed and handwritten text displayed on your screen, and it responds to what it sees, offering answers to your questions and suggesting next steps.
Privacy Consideration
Microsoft has emphased that privacy is a top priority with Copilot Vision. The data processed by Copilot Vision is not saved or used beyond the current session (like clearing the cache). Microsoft Copilot Vision will initially also be limited to certain popular websites that meet Microsoft’s security standards, ensuring a safe and secure user experience. It also won’t work on sites that contain sensitive data such as banking sites.
Remember, this is something you can choose to use or not, so you can turn it off!
Rollout Timeline
Microsoft say that after a successful trial period with a select group of users in the Copilot Labs experiment hub, Copilot Vision is “now ready” for a broader rollout.
The feature will soon be integrated into the Microsoft Edge browser, accessible via a screen-like icon. This phased rollout will allow Microsoft to gather more feedback and ensure a smooth user experience.