Meet Ace: The First AI Agent for Real-time Computer Autopilot

When we talk about artificial intelligence (AI), do you just think of a chatbot that just talks? Now is the right time to change your thinking because computers are no longer just responding to your commands and questions; AI can now plan, reason, act, and complete tasks for you autonomously without you needing to review them constantly. Companies are constantly trying to develop agents that can reason and act (ReAct) on your behalf; heard of ChatGPT Operator by OpenAI, Proxy by Convergence AI, or some other AI agent that can perform tasks on your behalf?

Let us introduce a new AI agent that is similar to ChatGPT Operator, named Ace, by a company called General Agents.

What is Ace by General Agents?

Ace by General Agents is the first AI agent real-time computer autopilot that can use your mouse and keyboard to get work done, potentially much faster than you can. The Ace system watches how tasks are done on a computer and then learns to do them itself, like an assistant who can actually click through menus, type into forms, and manage files on your desktop.

Ace is designed to work directly on your computer, using your mouse and keyboard to operate the applications you already have installed. Think about that for a moment: an AI that can click, type, and scroll through interfaces, essentially performing tasks you'd normally do yourself.

The team at General Agents has been demonstrating Ace, and its capabilities are certainly attention-grabbing. They claim it can complete tasks in milliseconds that might take a person minutes. The key difference, they say, lies in how Ace learns.

Here's a clearer look at what makes Ace different:

• Direct Action: Ace operates your computer's mouse and keyboard to perform tasks based on what it sees on your screen and the instructions it's given.

• Learning by Watching: Ace is trained on "behavior"—actual recordings of people completing tasks instead of being fed just text or images, meaning it understands the process of doing work.

• Impressive Speed: Early demonstrations show Ace completing certain actions very quickly. For example, it's reported to process some tasks in about 533-324 milliseconds, which is much faster than some other AI operator models.

• Works With Your Setup: A key point is that Ace is designed to function directly on your desktop, using the applications you already have installed without needing special sandboxes.

• Still Evolving: The creators note that Ace is in its early stages. While it shows promise, its intelligence and range of capabilities are expected to grow as it's exposed to more data and computing resources.

• Early Access: General Agents is currently making Ace models available to selected partners through Research Preview.

Why this matters

Every "AI agent" demo in the past year has lived inside a browser sandbox, which is handy for web forms, but most daily jobs still happen on spreadsheets, design suites, and line-of-business apps that live on the desktop. Ace skips the sandbox and goes straight to your screen, which means it can learn any workflow a human can demonstrate without needing an API, plug-in, or a long setup.

A Different Way to Learn

The way Ace learns is quite interesting. Most AI models you hear about are trained on vast quantities of text and images. However, General Agents didn't feed Ace mountains of text or images; it is trained on behavior.

This approach to training, focusing on behavior, is interesting because it learns from the actual steps someone takes to complete a task on a computer. General Agents fed Ace behavior: screen recordings plus the exact mouse and key events behind them.

• Engineers recorded more than a million real tasks, turning that footage into supervised lessons.

• The approach borrows the chain-of-thought or step-by-step reasoning idea that is now common in language models but applies it to cursor moves and shortcut combos.

The idea is that learning the "how-to" directly from human actions and better generalize its abilities to more practical skills and similar tasks.

This approach also makes it simpler for experts to contribute training data; they just record themselves doing their work with their usual tools without needing to learn new programming or complex annotation methods.

What Could This Mean for Your Workday?

The speed is a major talking point. If an AI can handle routine computer tasks many times faster than a person, it could change how certain work gets done. Think about repetitive data entry, filling out standard forms, or scrolling through complex software interfaces. If Ace can manage these kinds of activities quickly and accurately, it could free up considerable time for professionals to focus on more complex problem-solving and strategic thinking.

Limitations—and what's next

It's important to mention that General Agents is upfront about Ace being in its early stages. They acknowledge it's not perfect and has been developed with a fraction of the computing power and data typically used for large-scale AI language models.

In certain edge cases, unusual keyboard layouts, multi-monitor setups, or software that redraws its UI mid-task can stumble the agent. The team implies that Ace's intelligence and capabilities should grow and will smooth out weak spots with increased resources, training footage, and GPUs.

Conclusion

Ace is a reminder that progress in AI isn't only about larger language models; it's also about teaching machines how we work step by step. General Agents' vision, where a system learns from a large pool of human experience, could, technically, in turn, offer expertise to help us manage our digital tasks more effectively. If Ace develops and keeps scaling data and computing as expected, it could be the next step in how we use AI for practical, everyday computer-based work. It's less about conversation and more about real action on your screen.