Google's Gemini on macOS Now Features Intelligent Dictation and Screen Awareness
Google has rolled out two new updates for Gemini on macOS, introducing intelligent dictation that removes filler words and a screen-aware reasoning feature.
Stock photo for illustration only, not from the actual event
- Gemini's latest macOS update introduces two brand-new features.
- Intelligent dictation automatically removes filler words like 'ums' and 'ahs' from speech.
- Screen-aware reasoning allows Gemini to view and understand your Mac's display.
- Enabling screen access introduces potential privacy and security considerations.
On Wednesday, Google announced two new updates for Gemini on macOS, one of which carries potential privacy and security implications for users who choose to opt in. This update aims to push the AI further into the realm of a personal assistant rather than just acting as a standard conversational chatbot.
For those who utilize dictation on macOS, speaking words aloud to have them typed out, Gemini now offers a compelling alternative to Apple's built-in tools. Google has dubbed this feature 'intelligent dictation,' stating that it polishes text instantly at the cursor. This means Gemini will strip away stammers and verbal fillers like 'ums' and 'ahs' that you would never manually type out. The functionality operates across any window on your Mac simply by pressing the Fn key and speaking.
This capability is enabled by default for everyone using Gemini on Mac unless disabled within the settings. It represents a minor adjustment that streamlines drafting texts, emails, or documents without requiring total manual self-editing.
Stock photo for illustration only, not from the actual event
Google's second update is far more complex and is not enabled by default. Dubbed 'screen-aware reasoning,' this feature grants Gemini permission to understand what is occurring on your Mac's display. The core concept involves delegating tasks directly to the AI; for instance, with the Notes app open, you can press the Fn key and ask Gemini to turn those notes into an executive summary for an afternoon meeting, prompting the AI to analyze the text and configure it on your behalf.
Allowing an AI model to visually process desktop displays and execute chained commands marks a major evolution in desktop productivity assistants. However, granting continuous screen visibility also expands the attack surface for risks such as prompt injection, where malicious actors might embed hidden instructions in web content to trick autonomous assistants.
Users can stack multiple commands together, such as drafting a team dinner email while pulling specific file data and searching Google Maps for venues, with the ability to correct details on the fly. While undeniably useful for expanding Gemini's utility across the entire operating system, users must weigh these productivity gains against the inherent privacy and security risks of granting an AI full visibility into their personal computer activity.
Source: Lifehacker
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment