-
Ask the window manager
The window manager lists every window with its title, app, position and focus: KWin, Sway, Hyprland or EWMH on Linux, Win32 on Windows, CGWindowList on macOS. The agent can bring one forward, minimize or close it, including apps that publish nothing else.
$ nexremote windows * win:{c31f…} org.kde.konsole 1100x780+0+0 api : claude — Konsole win:{8c1e…} systemsettings 760x540+1180+60 Appearance — System Settings win:{a41f…} firefox 880x560+1060+520 Player — Mozilla Firefox -
Read the accessibility tree, act by name
Buttons, switches and fields come with names and roles, through AT-SPI, UI Automation or the macOS accessibility API. The agent presses “Dark” by its name. The mouse does not move, and nothing depends on where the control is drawn.
$ nexremote find "Dark" --window win:{8c1e…} atspi:4:0/2/7 [checkbox] Dark <toggle,click> $ nexremote click atspi:4:0/2/7 --json { "ok": true, "ref": "atspi:4:0/2/7" } -
Read terminal text directly
Konsole tabs and tmux panes hand over their text along with the program running in each; Windows consoles, Terminal.app and iTerm2 hand over theirs too. No screenshot, no OCR, no focus change.
$ nexremote term-read konsole:7891:2 ● Running test suite… tests/test_api.py ........F.F ✗ 2 failed · test_login, test_refresh Should I fix them? (y/n, or tell me) -
Only then: screenshot and click pixels
For games, canvas apps and Chromium apps not started through
launch. The shot is cropped to one window and scaled for the model; a pixel in that image is mapped back to the screen and clicked through the system's own input: the RemoteDesktop portal, XTEST or Sway IPC, SendInput, CGEvent.$ nexremote screenshot --window win:{a41f…} --max-width 1568 $ nexremote click 440 190 --from-shot --json { "ok": true, "x": 1500, "y": 710, "button": "left" }
Desktop control for AI agents · Linux · Windows · macOS
Your agent can run commands. Now it can use the screen.
NexRemote turns the desktop into shell commands, the same ones on Linux, Windows and macOS: every window, a real mouse and keyboard, buttons pressed by name, and the terminals you already have open, other agents included. No root, no SDK, and a vision model only when an app gives it nothing else to read.
./install.sh && nexremote setup
Setups documented for
The problem
Agents live in the terminal. Your work lives on the screen.
Coding agents edit files and run builds. The moment a task needs a settings dialog, a running app or another terminal, they stop and hand it back to you. The usual ways around that each break somewhere.
-
Screenshot every step
The model looks at the whole screen, guesses coordinates, clicks, then takes another screenshot to see what happened.
Where it breaks: every step is a vision call, so it is slow and costly, and a few pixels off is a wrong click. Text already on screen is read back from an image.
-
An integration per app
APIs, SDKs and browser drivers are exact, but each one covers a single program and has to be built and maintained.
Where it breaks: desktop apps without an API, and the terminals already open on screen, stay out of reach.
-
X11 automation scripts
xdotool and similar tools were made for X11, where any client could read and type into any window.
Where it breaks: Wayland closed that door. The usual workaround, ydotool, writes to
/dev/uinput, which needs root or a udev rule.
NexRemote takes the most direct channel the desktop already offers, and falls back to pixels only when nothing else is there.
How it works
Structure first. Pixels last.
Before it looks at a single pixel, NexRemote asks the desktop what it already knows. Each step down is used only when the one above has nothing to give.
One core between any agent and the desktop
Adapters only know the core; the core only talks to channels the desktop already provides. If a backend is missing, the rest keep working and nexremote caps says what is off.
Replay
A session, replayed
Commands and output are in the exact format the CLI prints. The desktop is drawn, so you can see where each command lands.
The agent asks what is on screen: every window with its ref, every terminal with the program in it.
Reach
What it reaches
The same commands on Linux, Windows and macOS. Each surface takes the most direct channel the system offers (the platform table below names it), and nexremote caps says when one is missing.
-
Windows
Every window on screen, including apps that publish no accessibility tree.
- Commands
windowswindow REF activate
-
Mouse
Your pointer is put back where it was right after each click.
- Commands
click X Yscrolldrag
-
Keyboard
Any language. Text goes in as Unicode; where the channel can't carry a character, it is pasted and your clipboard restored.
- Commands
typekey ctrl+s
-
Controls by name
Presses buttons and toggles without moving the mouse at all.
- Commands
findclick REF
-
Electron by name
Starts Discord, VS Code or Slack with their tree switched on, so their buttons have names too.
- Commands
launch vesktop
-
Open terminals
Shows what runs in each one: claude, ssh, vim, a build.
- Commands
termsterm-readterm-send
-
Its own shell
Sessions survive between calls, so it can start something and check on it later.
- Commands
runscreenkeys
-
Screen
Crop and scale are remembered, so
click X Y --from-shotlands on the right pixel.- Commands
screenshot --window --max-width
-
Agent cursor
A second, visible arrow glides to where the agent works. Your mouse stays yours: clicks pass through it and focus never moves.
- Commands
pointcursorcursor-hide
-
Something missing?
One command reports which channels work on this machine and which packages the rest need.
- Commands
nexremote caps
Platforms
Three desktops, one set of commands.
Refs, output and flags are the same everywhere. Underneath, each platform gets its own channel, named in the table. How far each one is tested is stated per environment, not rounded up.
-
Linux
KDE Plasma 6, GNOME, X11, Sway, Hyprland
Tested
- KDE Plasma 6 · WaylandReal desktop
Run on a real desktop. Input tests run in a sandbox with its own D-Bus, a virtual KWin, portal and PipeWire.
- X11Xvfb sandbox
End-to-end suite on a headless X server: XTEST input, EWMH windows, screenshots, the CLI.
- SwayHeadless Sway
A real Sway 1.10, run headless: windows, clicks and scrolling through Sway IPC, typing with wtype, screenshots with grim.
- HyprlandFake IPC
hyprctl is replaced by a fake; not yet run on a real compositor.
- GNOMEUntested
The portal path should work; the window list falls back to AT-SPI.
- KDE Plasma 6 · WaylandReal desktop
-
Windows
Built for Windows 10 and 11
Tested
- Wine 1047/47 passed
Real Windows Python under Wine, against a real Notepad window and a cmd.exe console. Not yet run on Windows 10 or 11 hardware.
Click by name is untested: Wine's UI Automation has no control tree to walk.
- Wine 1047/47 passed
-
macOS Preview
Quartz events and the accessibility API
Tested
- Unit testsMock Quartz
The logic is unit-tested against a fake Quartz library. Not yet run on a real Mac.
Allow Accessibility and Screen Recording once, in System Settings.
- Unit testsMock Quartz
Solid: a real desktop. Outline: a real display server, run headless or under Wine. Amber: logic checked against fakes. Dashed: not tested.
9 surfaces × 3 platforms
| Capability | Linux | Windows | macOS Preview |
|---|---|---|---|
| Every window on screen | KWin · hyprctl · swaymsg · EWMHGNOME: AT-SPI only | EnumWindows | CGWindowList + AX |
| Real mouse | XTEST · RemoteDesktop portal · Sway IPC | SendInput | CGEvent |
| Real keyboard | XTEST · portal + clipboard · wtype | SendInput (Unicode) | CGEvent (Unicode) |
| Press a button by name | AT-SPI | UI Automationnot exercised under Wine | Accessibility (AX) |
| Electron and Chromium with their tree on | flag + environment | flag | flag |
| Open terminals | Konsole D-Bus, tmux | console buffer (AttachConsole) | Terminal.app, iTerm2, tmux |
| Its own terminal | tmux / PTY | cmd.exe | tmux / PTY |
| Visible agent cursor | X11/XWayland window · your cursor theme | layered window · system arrow | NSWindow · system arrow |
| Screenshots | spectacle · grim · Xlib | GDI | screencapture |
Channels are read from the project README when this page is built.
Terminals
Eight agents open? One of them can run the others.
Every Konsole tab and tmux pane is listed with the program in its foreground, and so are Windows consoles and Terminal.app and iTerm2 windows. An agent can read another agent's screen, answer its question, or stop it with --key C-c.
None of it needs window focus or your keyboard. You keep working while it does.
$ nexremote terms
konsole:7891:1 claude ~/projects/api api : claude
konsole:7891:2 claude ~/projects/web web : claude
konsole:7891:3 vim ~/projects/docs docs : vim
konsole:29769:1 ssh ~ deploy@staging
$ nexremote term-send konsole:63308:1 'echo NEXR_$((6*7))' --enter
$ nexremote term-read konsole:63308:1
bash-5.2$ echo NEXR_$((6*7))
NEXR_42
bash-5.2$Last resort
When an app has nothing to read, it looks.
Games, canvas apps and Chromium apps started without nexremote launch publish no controls. Take a screenshot cropped to one window and scaled for the model; it clicks on pixels in that image and NexRemote converts them back to the screen.
screen = origin + image ÷ scale
(3000, 0) + (412, 230) ÷ 0.8333 = (3494, 276)Compare
Where it stands
Compared with the kinds of tools in use today, not with named products. Each approach has its place; this is how they do on the work NexRemote is built for.
| NexRemote | Screenshot-only computer-use agents | Classic RPA and xdotool scripts | Per-app APIs and SDKs | |
|---|---|---|---|---|
| Presses controls by name, not by coordinates | YesAT-SPI · UI Automation · AX | Noguesses coordinates from the image | Partlysome RPA tools use selectors; xdotool clicks coordinates | Yesinside that one app |
| Reads on-screen text without a vision model | Yesaccessibility tree and terminal text | Noevery read is an image | Partlydepends on the tool | Yesinside that one app |
| Works with apps that publish nothing to read | Yesscreenshot fallback | Yesits main mode | Partlyblind coordinates | Noneeds an API |
| Reads and answers terminals already open | YesKonsole · tmux · Windows consoles · Terminal.app | Partlyfrom pixels, when the window is visible | Partlycan type, cannot read | Noout of scope |
| Runs without root or admin rights, Wayland included | YesWayland portal; user rights on Windows and macOS | Partlydepends on setup; often a VM | NoX11 only, or root for /dev/uinput | Yesdoes not touch input |
| Plugs into any agent through a shell command | Yesalso MCP, HTTP and a text loop | Partlyusually tied to one model's tool format | Yesit is a command too | Partlyone integration per app |
Categories, not products. “Yes” means the approach does it out of the box; “Partly” means it depends on the tool or the setup.
Numbers
By the numbers
Counted from the v0.5.0 source every time this page is built. No estimates, no projections.
- MCP tools
- 23
- CLI commands
- 28
- ways to plug in: shell, MCP, HTTP, text loop
- 4
- desktop surfaces it reaches
- 9
- harness setups documented
- 11
- test suites, 4 of them in a throwaway desktop
- 15
- lines of Python, core and adapters
- 11,258
- root or admin rights needed
- 0
Why now
The pieces are in place.
No forecast needed. This is where the tools already are.
-
Agents already work in terminals
Coding agents such as Claude Code, Codex, Aider and Gemini CLI run in a shell. A shell command is the one interface they all share, so that is the interface NexRemote speaks.
-
MCP became a common tool protocol
Editors and agent clients load tools over MCP. One NexRemote server gives all of them the same desktop tools, with screenshots returned as images.
-
Linux desktops moved to Wayland
Wayland removed the X11 shortcuts that old automation relied on. The sanctioned way in is the xdg-desktop-portal RemoteDesktop interface, which asks the user once. On Linux, NexRemote is built on it.
Roadmap
What exists, what comes next
The left column ships in the download today. The right column is planned, not built.
In v0.5.0 today
- KDE Plasma: every window through KWin scripting, mouse and keyboard through the RemoteDesktop portal, approved once
- X11 (XTEST, EWMH), Sway and Hyprland (their own IPC) behind the same commands; on Sway the mouse needs no extra package
- Reading and typing into open Konsole tabs and tmux panes
- 23 MCP tools, 28 CLI commands, an HTTP API and a text loop for models without tool calling
- Windows: Win32 windows, Unicode input through SendInput, GDI screenshots, UI Automation, reading and typing into open consoles
- macOS preview: Quartz input, CGWindowList windows, the accessibility tree, Terminal.app, iTerm2 and tmux
- Electron and Chromium apps by name:
nexremote launch vesktopstarts them with their tree switched on - A compact tree: hidden items and empty wrappers left out, and every control carries the ref to click it
- Windows, terminals and the accessibility tree as live MCP resources
- A user service for the HTTP adapter, with a token file and TLS, to drive another machine
- A visible agent cursor on Linux, Windows and macOS that never touches your mouse, from every adapter: MCP, HTTP, CLI
- A pip package:
pip install .[all]on Linux, Windows and macOS - Input tests in an isolated sandbox desktop, X11 in Xvfb, Sway on a headless compositor, Windows under Wine
Planned
- Real Windows and Mac hardware. Run the suites on Windows 10 and 11 and on a Mac, then drop the preview label.
- GNOME and a real Hyprland. Test the portal path and the window list on GNOME, and run Hyprland on a real compositor instead of fake IPC.
Connect
Plug it into your agent
Same core, four ways in. Pick whatever your harness already speaks.
Any harness that can run a command: Claude Code, Codex, Aider, opencode, a cron job. Give the agent the skill text and it knows the rest.
nexremote skill # paste into your agent's instructions
nexremote observe --json23 tools over stdio for Claude Code, Claude Desktop, Cursor and other MCP clients. Screenshots come back as images the model can look at.
claude mcp add nexremote -- ~/bin/nexremote-mcpFor agents on another machine. Listens on localhost by default and refuses any other address without a token.
nexremote-http --port 8788 --token "$SECRET"
curl -s -H "Authorization: Bearer $SECRET" -X POST localhost:8788/terminalsFor local models without tool calling. The model writes nexremote lines; the loop runs them and feeds the output back until it writes DONE.
nexremote-agent --model-cmd "ollama run qwen2.5" \
--goal "Open the calculator, compute 7×8, read the result"Safety
What it does to your machine
- Asks once
- On Wayland the desktop shows a remote-control dialog the first time; tick Allow restoring and it won't ask again (delete
~/.config/nexremote/portal.jsonto take it back). On macOS you allow Accessibility and Screen Recording once. Windows and X11 don't ask. - Gives your pointer back
- There is one pointer. A click borrows it and puts it back where it was.
- No root, no admin
- No
/dev/uinput, no driver, no service running as another user. On Windows it runs as you; windows running as administrator stay out of reach unless NexRemote is elevated too. - Closed to the network
- The HTTP adapter will not listen beyond localhost without a token of at least 24 characters. The token can live in a file instead of the command line, and traffic can run over TLS.
- Tested where it can't hurt you
- Input tests run in a throwaway desktop with its own D-Bus, a virtual KWin, its own portal and PipeWire. X11 runs in Xvfb, Windows in its own Wine prefix on a headless display.
Limits
- Linux: KDE Plasma 6 on Wayland is tested on a real desktop, X11 on a real X server (Xvfb), Sway on a real headless compositor, Hyprland with fake IPC. On GNOME the window list falls back to AT-SPI only.
- Windows: tested on Wine 10, not yet on real Windows 10 or 11. Wine's UI Automation can't walk the control tree, so click by name is untested there. Windows running as administrator are reachable only when NexRemote runs as administrator too (UIPI).
- macOS: the code is written and its logic tested against a fake Quartz, but it has not yet been verified on a real Mac. It ships as a preview.
- Konsole exposes the visible screen only. For scrollback, run the agent in tmux.
- Electron and Chromium apps publish their tree only when started with
--force-renderer-accessibility.nexremote launchadds it, but an app that is already running has to be restarted. Games and canvas apps are driven by screenshot and coordinates. - On Linux the visible agent cursor needs X11 or XWayland and does not show on Sway without XWayland. The Windows cursor is tested on Wine; the macOS cursor has not yet run on a real Mac.
Install
Install in three steps
Linux
-
Download and unpack
unzip nexremote.zip && cd nexremotenexremote.zip (237 KB) · nexremote.tar.gz (192 KB)
-
Install the commands
./install.shPuts
nexremote,nexremote-mcp,nexremote-httpandnexremote-agentin~/bin, then lists anything missing. -
Grant access once
nexremote setup && nexremote capsTurns on the accessibility tree and, on Wayland, opens the desktop's remote-control dialog: tick Allow restoring, press Share. X11 asks nothing.
Needs python3-dbus python3-gi xdg-desktop-portal-kde pipewire wl-clipboard. Optional: tmux python3-pil python3-xlib, pip install mcp. Rather use pip? pipx install --system-site-packages . keeps the system's D-Bus and GObject bindings in view.
Windows
-
Download and unpack
Expand-Archive nexremote.zip -DestinationPath .; cd nexremotenexremote.zip (237 KB)
-
Install with pip
py -m pip install .[all]Puts
nexremote,nexremote-mcp,nexremote-httpandnexremote-agenton your path.comtypes, for UI Automation, comes along. -
See what works
nexremote capsNo permission dialog. Windows running as administrator are reachable only if NexRemote runs as administrator too.
Needs Python 3.9 or newer, started with the py launcher. [all] adds pillow for cropping and scaling screenshots and mcp for the MCP server. Tested under Wine 10, not yet on real Windows 10 or 11.
macOS Preview
-
Download and unpack
unzip nexremote.zip && cd nexremotenexremote.zip (237 KB) · nexremote.tar.gz (192 KB)
-
Install with pip
python3 -m pip install '.[all]'Puts the four commands on your path.
[all]adds screenshot scaling and the MCP server. -
Grant access once
nexremote setup && nexremote capsOpens the Accessibility and Screen Recording prompts. Allow your terminal in System Settings → Privacy & Security; it is not asked again.
Needs Python 3.9 or newer. Optional: tmux for scrollback. Preview: tested against a fake Quartz, not yet on a real Mac.
Give your agent the desktop.
Unpack, install, approve once. For questions about the project, a partnership or an investment, write to us.
./install.sh && nexremote setup