ISSUE 26 · ME+MACHINE
Even with a good recipe, somebody has to do the cooking. When you ask AI for help, it can give you the steps, but you still have to carry them out… or do you?
With computer use, an AI assistant can take those steps for you. It looks at what’s on screen, then clicks, types, or navigates through an app or website. After each action, it checks the updated screen before deciding what to do next.
For example, you could ask the assistant to make a presentation slide easier to read, and it might make the text bigger, then check whether everything still fits on the slide. You describe what you want changed without needing to find the right settings yourself.
To try it, enable computer use in your AI’s desktop app:
For ChatGPT, go to Plugins → Computer Use and install or enable the plugin as prompted. Use it with Work or Codex. It is available on macOS and Windows in supported regions.
For Claude, go to Settings → General → Enable computer use. Use it with Cowork or Claude Code. At the time of writing, it is available on Pro and Max plans.
Then start a task, name the app or website you want it to use, and describe what you want done. Approve the requested screen and app access when prompted. Keep your computer awake and the AI’s desktop app open.
Some apps offer a connector, which creates a direct link between the AI assistant and the app so they can exchange information. Depending on what the connector supports, the assistant can also take actions, such as adding a calendar event or updating a document, without clicking through menus. That can be quicker than working through the screen. Computer use is useful when a suitable connector isn’t available or when the assistant needs to check or change something visually.
Testing computer use with a tough job
I wanted to try something difficult that readers could follow visually. So I asked Codex to build Gizmo standing next to a cottage using Blender, an app for making 3D models.
Blender does have a connector that lets AI work with it directly, and I’ve found it works very well. For this experiment, though, I asked Codex to work through the app’s on-screen controls only.
Codex started clicking through menus and shaping objects, and the cottage began to take shape. Watching it work made it easier to follow the progress. It did sometimes struggle to control the Blender window, and a front view of the saved model revealed a gap between the walls and the roof.
Codex operating Blender’s on-screen controls.
The saved image at the stopping point. The roof fit was still unfinished.
After watching Codex struggle with the roof for some time, I decided to pause its work to save my precious AI credits. I have no idea how to use Blender myself, so Gizmo and the cottage remain unfinished in the saved file, roof gap and all.
Not bad for a first attempt. Next time, I’d try giving Codex reference images of the cottage and Gizmo from several angles (front, back, side and three-quarter) to see whether that helps.
Try it yourself
Start with a small task whose result you can easily check. You could ask the assistant to:
Fill out a form using information from your notes.
Format a newsletter by adjusting headings, spacing and images.
Download a report from a dashboard using dates and filters you specify.
Check a website for broken buttons or confusing steps.
Tell it what you want saved and where to stop for your review. It is making real changes in your apps, so work on a copy where possible and ask for approval before anything is sent, published or deleted. Keep unrelated sensitive information out of the apps it uses. Remember that pausing a task stops further work, but it doesn’t undo changes that have already been made.
Here’s a sample prompt for creating a first version of a presentation in Codex:
/goal Use computer use in [presentation app] to turn these notes into three slides for [audience]. Save as a new presentation, show me the slides, and stop for my review. If you get stuck, save your progress and explain why.
Notes: [paste your notes here]Using /goal is optional. It tells your AI to keep working toward a result you define, even when the task takes several rounds.
THE 30-SECOND VERSION
Computer use lets an AI assistant click, type, and navigate apps or websites to carry out a task.
It can help you work in unfamiliar software, but you still need to check its work.
A connector can be quicker for tasks it supports. Computer use helps when there’s no suitable connector or the task needs a visual check.
Start small, work on a copy and tell the assistant when to stop for your review.
Pause if it gets stuck or goes off track. Stopping doesn’t undo changes already made.
You don’t have to do all the cooking yourself anymore. Just be sure to taste the food before you serve it.
me+machine.