Why AI Agents Shouldn't Click the Screen: Building a Local Control Plane for a Desktop App

The short answer

The tempting way to let an AI agent use a desktop app is to let it look at the screen and click, the way a person would. For a video app, that's the wrong design. Clicks are fragile, the agent can reach anything a mouse can, and "it clicked Export" doesn't prove the export is right.

• Letting an AI agent drive a desktop app by looking at screenshots and clicking works in demos but is fragile, slow, and grants the agent everything a mouse can do.
• KwaFlux exposes a structured control plane instead: typed MCP tools, a small set of parameters per call, and checks in the app before anything is queued.
• The agent doesn't write the full job. The KwaFlux worker rebuilds it with the same job builder the app uses when you click Export.
• Rules shared with the app are generated, not copied, and tests fail if the two drift apart.

KwaFlux takes the other route: a local control plane that accepts typed requests, checks them, and builds the real job inside the app.

In this guide, you'll learn why screen automation falls short for this job, how KwaFlux's layers fit together, why the agent only sends small parameters, and what this design deliberately can't do.

Contents

1. What is a control plane for AI agents?
2. Screen automation vs. a control plane
3. The layers, from chat to worker
4. Small parameters, rebuilt by the app
5. One source of rules for the app and the agent
6. What the control plane checks before queueing
7. Results the agent can rely on
8. What this design gives up
9. FAQ

What is a control plane for AI agents?

A control plane for AI agents is a narrow, programmatic interface that lets an agent ask an application to do specific things, while the application keeps the authority to check, refuse, and carry out each request.

In KwaFlux, the interface is a set of MCP tools, and the authority sits in the running desktop app. The agent asks; KwaFlux decides.

Screen automation vs. a control plane

Computer-use agents that work from screenshots are a real advance, and for apps with no API they're the only option. Anthropic's own documentation for its computer use tool is frank about the limits. It notes that the model "sometimes acts as if its actions succeeded without explicitly checking their results," that dropdowns and scrollbars "might be tricky" to operate with the mouse, and that instructions on web pages or in images "might override your instructions." It recommends running computer use in "a dedicated virtual machine or container with minimal privileges."

For a video app on your own computer, such as KwaFlux with its Quality Repair and conversion tools, those limits matter:

The last row isn't a contest of exact numbers. The point is that a structured call costs the same whether the export takes ten seconds or ten minutes, while a screen-watching agent pays for every look.

The layers, from chat to worker

A request passes through six layers. Each layer has one job, and each one can say no.

1. The AI host (Cursor, Claude, Codex, and others) starts the KwaFlux MCP server as a local process and talks to it over standard input and output.
2. The MCP process turns tool calls into small typed requests and checks paths in the same way the app does before touching any file.
3. The local channel connects the MCP process to the running app: a named pipe for your user account on Windows, a socket only your account can open on Mac. Every connection starts with a token handshake. The local channel post covers this layer.
4. The control plane in the app checks the request: is the path local and absolute, is the user signed in with the right plan, does the task belong to this agent, is the agent within its rate limit, does the download setting allow it?
5. The scheduler queues the job next to the ones you started in the window, so they share the same GPU and the same task list.
6. The worker builds the full job, verifies a signed lease for paid features, runs the engine, and writes the result next to the source file.

Small parameters, rebuilt by the app

Here's the decision that shapes everything else: the agent doesn't send a full job description.

A KwaFlux export job has many fields: decoder settings, model choices, tile sizes, encoder options, color handling, and more. If the agent could send that whole structure, two things would go wrong. The model would be inventing values it can't check, and anyone who could steer the model, through a poisoned file name or web page, could reach every engine parameter.

So each tool takes only the few fields a person would choose: the input file, the module, an output size from the app's list, a quality tier, a time range for a GIF. The worker then builds the complete job with the same job builder the app uses when you click Export, and trims it to the parameter contract the engine accepts at run time.

The result is that an agent-started job and a clicked job go through the same code from the builder onward. We describe this as "the same parameter rules as the app," not "identical output," because a few tools are narrower over MCP. The capabilities post lists those differences.

One source of rules for the app and the agent

Two copies of the same rules drift apart. KwaFlux avoids keeping two copies:

• The module catalog that MCP offers, including each module's route, settings, and allowed values, is generated from the same configuration file as the app's screens. Photo variants come from the image section of that same file.
• Five rule sets are translated directly from the app's source code for the MCP server: output resolution tiers, the Compressor's container and codec choice, the audio workspace's algorithm parameters, the voice list for AI Voice Change, and one detection rule.
• A drift test fails if the app's version of any of those rules changes and the MCP version wasn't regenerated.

The same idea keeps the tool descriptions small, which the tool design post covers.

What the control plane checks before queueing

The MCP specification says servers "MUST" validate all tool inputs, implement proper access controls, rate limit tool invocations, and sanitize tool outputs. In KwaFlux those checks sit in the app, not in the MCP script, so a different client can't skip them:

Results the agent can rely on

Every KwaFlux tool declares an output schema, so the result is structured data the host can validate, not a paragraph the model has to interpret. When the host passes a progress token, KwaFlux sends progress notifications while a task runs. When the user stops the request in the host, KwaFlux stops waiting and the agent can cancel the task.

Task results only say "output ready" when a task has actually succeeded, and they give the output file's name. A queued or failed task doesn't claim to have produced anything.

What this design gives up

A control plane is only as broad as its tools. Nine KwaFlux features need someone to draw on the frame, click a subject, or judge a preview, such as object removal and sky replacement. We could have let the agent guess coordinates, or accepted a raw job description. The first gives results nobody can predict, with a receipt that says it worked. The second removes the boundary that makes the design safe. So those features stay in the app, and the agent is told to send the user there.

That's a product choice, not a gap we plan to close by loosening the rules.

Final thoughts

If an agent has to look at pixels to use your app, you can't check what it's about to do. A control plane flips that: the agent asks in a narrow, typed language, and the app checks, builds, and runs the job with its own code. For KwaFlux, that's how an agent-started export goes through the same job builder as one you click.

KwaFlux's local MCP server lets Codex, Claude, Cursor, and other MCP apps queue real jobs in the desktop app. AI processing runs locally on your GPU — media files are not uploaded for processing. It runs on Windows 10 (64-bit, version 1809 or later), Windows 11, and macOS 12 or newer on Apple Silicon. Importing files and exploring the app need no account. Signing in with an active paid plan is required only to export, convert, repair a file, or download a file from a link.

Download KwaFlux · See plans