Remote Labor Index: Measuring AI Automation of Remote Work

Mantas Mazeika, Alice Gatti, Cristina Menghini, Udari Madhushani Sehwag, Shivam Singhal, Yury Orlovskiy, Steven Basart, Manasi Sharma, Denis Peskoff, Elaine Lau, Jaehyuk Lim, Lachlan Carroll, Alice Blair, Vinaya Sivakumar, Sumana Basu, Brad Kenstler, Yuntao Ma, Julian Michael, Xiaoke Li, Oliver Ingebretsen, Aditya Mehta, Jean Mottola, John Teichmann, Kevin Yu, Zaina Shaik, Adam Khoja, Richard Ren, Jason Hausenloy, Long Phan, Ye Htet, Ankit Aich, Tahseen Rabbani, Vivswan Shah, Andriy Novykov, Felix Binder, Kirill Chugunov, Luis Ramirez, Matias Geralnik, Hernán Mesura, Dean Lee, Ed-Yeremai Hernandez Cardona, Annette Diamond, Summer Yue, Alexandr Wang, Bing Liu, Ernesto Hernandez, Dan Hendrycks

01 Nov 2025 3 min read

AI-generated image, based on the article abstract

Quick Insight

How AI Is Starting to Take Over Remote Jobs – The Remote Labor Index Reveals All

Ever wondered if the robots in movies will soon be answering your emails? Scientists have built a new test called the Remote Labor Index to see just how much AI can handle real‑world remote work. Think of it like a giant “speed‑run” where AI agents try to finish everyday tasks—writing reports, analyzing data, even scheduling meetings—just like a human would. The surprising result? Even the best AI managed to automate only about 2.5% of the work, barely scratching the surface. It’s like a rookie chef trying to bake a perfect cake; the ingredients are there, but the skill still needs polishing. This finding tells businesses and workers that, for now, AI is a helpful assistant rather than a job‑stealing monster. As the technology improves, the Remote Labor Index will keep tracking the progress, giving us a clear picture of when—and how—AI might truly reshape the way we work from home. Stay curious, because the future of work is just beginning to unfold. 🌐

Short Review

Unveiling AI's Real-World Automation Capabilities: A Critique of the Remote Labor Index

The article introduces the Remote Labor Index (RLI), a novel, multi-sector benchmark designed to empirically measure AI automation of real-world, economically valuable remote work. Comprising 240 end-to-end freelance projects sourced from professionals, RLI evaluates AI agents' practical performance beyond research-oriented tasks. The study aims to ground discussions of AI's economic value and automation capabilities in empirical evidence. A key finding reveals state-of-the-art AI agents achieve a remarkably low automation rate of just 2.5% on the RLI, indicating significant limitations in complex project completion.

Critical Evaluation

Strengths

A significant strength of this research lies in the development of the Remote Labor Index itself, offering an unprecedented, economically grounded benchmark for AI evaluation. Unlike prior benchmarks focused on specific tasks, RLI assesses end-to-end agent performance in practical, multi-sector settings, providing a more realistic measure of AI's economic utility. The methodology is robust, involving rigorous sourcing, cleaning, and PII protection for its 240 projects, alongside defining clear metrics like Automation rate and Elo scores for comprehensive AI agent evaluation.

Weaknesses

While the low automation rate of 2.5% is a key finding, it also highlights the current limitations of AI in complex interactive tasks, limiting immediate practical application. The evaluation process, relying on complex manual human assessment, while thorough, could be resource-intensive and potentially introduce subtle biases, despite efforts to standardize. Furthermore, the "near the floor" performance of AI agents on RLI suggests that significant advancements are still needed before widespread economic automation becomes a reality.

Implications

The findings from the RLI provide crucial empirical evidence, grounding often speculative discussions about AI's economic impact and labor automation. This benchmark establishes a common, objective basis for tracking AI impacts over time, enabling proactive navigation of AI-driven labor automation. By detailing common failure modes and linking them to cognitive skill deficits, the research also offers valuable insights for guiding future AI development towards more robust and general cognitive automation capabilities.

Conclusion

This article makes a substantial contribution by introducing the Remote Labor Index, a vital tool for realistically assessing AI's current economic value and automation potential. It effectively shifts the conversation from theoretical benchmarks to practical, real-world performance, setting a critical baseline. The research underscores the significant gap between current AI capabilities and the demands of complex economic work, providing essential data for informed policy-making and strategic investment in AI research and development.