In an effort to make the internet more accessible for people with disabilities, researchers at The Ohio State University have begun developing an artificial intelligence (AI) agent that could complete complex tasks on any website using simple language commands.
In the three decades since it was first released into the public domain, the internet has become an incredibly intricate, dynamic system.
However, because internet function is now so integral to society’s well-being, its complexity also makes it considerably harder to navigate.
Today there are billions of websites available to help access information or communicate with others, and many online tasks can take more than a dozen steps to complete.
That’s why Yu Su, co-author of the study and an assistant professor of computer science and engineering at Ohio State, said their work, which uses information taken from live sites to create web agents — online AI helpers — is a step towards making the digital world a less confusing place.
Su said: “For some people, especially those with disabilities, it’s not easy for them to browse the internet.
“We rely more and more on the computing world in our daily life and work, but there are increasingly a lot of barriers to that access, which, to some degree, widens the disparity.”
The study work was presented in December at the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), a flagship conference for AI and machine learning research.
By taking advantage of the power of large language models, the agent works similarly to how humans behave when browsing the web, Su said.
The Ohio State researchers showed that their model was able to understand the layout and functionality of different websites using only its ability to process and predict language.
They started the process by creating Mind2Web, the first dataset for generalist web agents.
Though previous efforts to build web agents focused on toy simulated websites, Mind2Web fully embraces the complex and dynamic nature of real-world websites and emphasises an agent’s ability of generalising to entirely new websites it has never seen before.
Much of their success is due to their agent’s ability to handle the internet’s ever-evolving learning curve, Su said.
The researchers lifted over 2,000 open-ended tasks from 137 different real-world websites, which they then used to train the agent.
Some of the tasks included booking one-way and round-trip international flights, following celebrity accounts on Twitter, browsing comedy films from 1992 to 2017 streaming on Netflix, and even scheduling car knowledge tests at the US Department of Motor Vehicles (DMV).
Many of the tasks were very complex, such as booking one of the international flights used in the model would take 14 actions.
Such effortless versatility allows for diverse coverage on a number of websites, opening up a new landscape for future models to explore and learn in an autonomous fashion, said Su.
Su said: “It’s only become possible to do something like this because of the recent development of large language models like ChatGPT.”
Since the chatbot became public in November 2022, millions of users have used it to automatically generate content, ranging from poetry and jokes to cooking advice and medical diagnoses.
But because one website could contain thousands of raw HTML elements, it would be too costly to feed so much information to a single large language model.
To address the gap, the study also introduces a framework called MindAct, a two-pronged agent that uses both small and large language models to carry out these tasks.
The researchers found that by using this strategy, MindAct significantly outperforms other common modeling strategies and is able to understand various concepts at a decent level.
With more fine-tuning, the study said, the model could likely be used in tandem with both open-and closed-source large language models such as Flan-T5 or GPT-4.
However, their research does highlight an increasingly relevant ethical problem in creating flexible artificial intelligence, said Su.
While it could certainly serve as a helpful agent to humans surfing the web, the model could also be used to enhance systems like ChatGPT and turn the entire internet into an unprecedentedly powerful tool, Su said.
He said: “On the one hand, we have great potential to improve our efficiency and to allow us to focus on the most creative part of our work.
“But on the other hand, there’s tremendous potential for harm.”
For example, autonomous agents able to translate online steps into the real world could influence society by taking potentially dangerous actions, such as misusing financial information or spreading misinformation.
Su said: “We should be extremely cautious about these factors and make a concerted effort to try to mitigate them.”
But as AI research continues to evolve, he said that it’s likely society will experience major growth in the commercial use and performance of generalist web agents in the years to come, especially as the technology has already gained so much popularity in the public eye.
Su added: “Throughout my career, my goal has always been trying to bridge the gap between human users and the computing world.
“That said, the real value of this tool is that it will really save people time and make the impossible possible.”
