A number of weeks in the past, Google Gemini taught me some new graphic design abilities.
I used to be enhancing a screenshot in Photopea, a free on-line Photoshop different, and needed to put the picture over a colourful border with a drop shadow behind it. As a substitute of digging by way of documentation or on the lookout for YouTube tutorials, I simply shared a reside view of my net browser with Gemini and requested for steerage. Google’s AI assistant proceeded to stroll me by way of Photopea’s complicated menus step-by-step.
That is probably the most underrated function of Gemini’s Mac app, which launched in April. Whereas different desktop AI apps have more and more centered on taking direct management of your laptop, Gemini’s app nonetheless sees the worth of instructing you to do issues by yourself.
Trying over your shoulder
Whenever you click on the + button in Gemini’s Mac app, you’ll see a Share Window amongst Gemini’s listing of instruments. Dragging your cursor over this feature brings up a listing of open home windows to share with Google’s AI assistant.
(This function does require some further privateness permissions, enabled beneath Settings > Privateness & Safety > Display screen & System Audio Recording. From there you’ll be able to activate the Gemini toggle in order that the app can mechanically take screenshots.)
When you’ve shared a window with Gemini, it is going to take a screenshot of that window every time you publish a query. Meaning you should use Gemini alongside your different apps and get assist alongside the best way.
Whereas creating my picture border in Photopea, as an example, I bumped into some hassle making use of a gradient impact to my background. In response, Gemini checked out which menu was open in Photopea and advised me precisely which buttons to click on from there, citing Photopea’s on-line documentation.
I’ve since used Gemini for steerage in couple of different software program interactions. It helped me navigate the labyrinthine Fangraphs web site whereas wanting up some current baseball statistics, and after I vibe coded a few Raycast scripts for window administration, it led me by way of Raycast’s Settings menu to allow them.
Different desktop AI apps have their very own built-in methods to share your display screen, however the course of is clunkier. ChatGPT and Claude each require you to manually add new screenshots when one thing modifications in your display screen, and in Claude you should click on and drag to outline the seize space every time. Gemini’s Share Window mode feels extra like a trainer that appears over your shoulder and presents steerage as wanted.
Whereas there’s no desktop Gemini app for Home windows, Google presents a separate Google app for desktop on Home windows with an analogous share display screen function. The primary distinction is that the dialog flows by way of Google Search’s AI Mode somewhat than Gemini. (Microsoft’s Copilot app additionally has a screen-sharing function, although in my expertise its directions haven’t been as useful.)
What’s subsequent
As a substitute of instructing you to make use of your laptop extra successfully, Google’s rivals are focusing extra on controlling your laptop themselves.
Each Claude’s desktop app and OpenAI’s ChatGPT Codex app now provide Pc Use modes that may navigate by way of your desktop with digital cursors and keyboards, utilizing persistent screenshots to information them alongside. The hope is that you simply’ll have the ability to automate complicated computing duties even whenever you’re not on the laptop your self.
Google appears more likely to go down this path earlier than lengthy. Whereas the Gemini app can’t management your laptop at present, Google began previewing a Computer Use model for Gemini final fall.
However full laptop management has its downsides. Anthropic warns of safety dangers from malicious apps and net pages, which might ask Claude to override the user’s own instructions. It additionally cautions in opposition to letting AI make choices with “significant real-world penalties,” not less than not with out in search of human affirmation first. AI can be only a lot slower at clicking by way of buttons and menus, and letting these firms see every thing in your display screen is a potential privacy nightmare.
My hope, then, is that whilst laptop use turns into a much bigger focus, Google doesn’t surrender on permitting AI to play the position of software program tutor. Not each computing job must be automated away, and there’s at all times worth in studying to do it your self.
