A breakthrough framework developed by researchers from MIT and NVIDIA may soon allow people to correct a robot’s actions in real-time using simple, intuitive feedback—similar to how they would guide another person.

Imagine you’re doing the dishes and a robot grabs a soapy bowl from the sink—but its gripper misses the mark. Instead of having to retrain the robot from scratch, a new method could enable you to fix its behaviour in real time. Through basic interactions like pointing to the object, tracing a path on a screen, or physically nudging the robot’s arm, you could guide it to complete the task more accurately.

This new approach eliminates the need for users to collect data and retrain the robot’s machine-learning model, unlike other traditional methods. Instead, it allows the robot to immediately adjust its actions based on user feedback to get as close as possible to fulfilling the user’s intent.

In tests, the framework’s success rate was 21 percent higher than an alternative method that did not leverage human corrections.

“This approach is designed to let robots perform tasks effectively right out of the box,” says Felix Yanwei Wang, an MIT graduate student in electrical engineering and computer science (EECS) and the lead author of a paper on the framework. “We can’t expect laypeople to gather data and fine-tune models. If a robot doesn’t work as expected, users should have an intuitive way to fix it.”

Wang’s co-authors include Lirui Wang PhD ’24, Yilun Du PhD ’24, senior author Julie Shah, MIT professor of aeronautics and astronautics and director of the Interactive Robotics Group at CSAIL, along with Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Perez-D’Arpino PhD ’19, and Dieter Fox from NVIDIA. The research will be presented at the upcoming International Conference on Robots and Automation.

A New Approach to Robot Correction

Currently, many robots use generative AI models trained on vast amounts of data to perform tasks. These models can solve complex tasks but often struggle to adapt to real-world situations that differ from their training environment. For example, a robot might fail to pick up a box from a shelf if the shelf in the user’s home is arranged differently than in its training environment.

To address this, engineers often collect new data and retrain the model—a time-consuming and costly process. However, the new MIT-NVIDIA framework allows users to interact with the robot during deployment, correcting its behavior in real time without the need for retraining.

“We want users to guide the robot without causing mistakes that could misalign with their intent,” says Wang. “The goal is to provide feedback that adjusts the robot’s behavior in a way that is both valid and aligned with the user’s goals.”

The system offers three ways for users to provide feedback: they can point to the object they want the robot to interact with, trace a desired trajectory on a screen, or physically nudge the robot’s arm. Wang explains, “Physically nudging the robot is the most direct way to specify user intent without losing any of the information.”

Ensuring Valid Actions

To avoid the robot making invalid moves—like colliding with nearby objects—the researchers developed a sampling procedure. This technique ensures that the robot chooses actions that are both feasible and aligned with the user’s request.

“Rather than just imposing the user’s will, we allow the robot to take the user’s intent into account while ensuring the actions remain valid,” Wang says.

The researchers’ framework outperformed other methods during tests with a real robot arm in a toy kitchen. While the robot might not always complete tasks immediately, the system allows users to correct it on the spot, without waiting for it to finish and then provide new instructions.

The framework also has the potential to learn from user corrections. For instance, if a user nudges the robot to pick up the correct bowl, the robot could log this action and incorporate it into its future behavior, gradually improving over time.

“The key to continuous improvement is having a way for users to interact with the robot,” says Wang. “This method makes that possible.”

Looking ahead, the researchers aim to improve the speed of the sampling procedure and test the framework in new, more complex environments, paving the way for robots that are more adaptable to real-world scenarios.

Last year on June 5th, NASA astronauts Sunita Williams and Barry Wilmore were on a flight testing mission to dock a Boeing Starliner spaceraft to the International Space Station (ISS). Set to return just eight-days later, their mission met with an ill-fated death. A few thrusters failed, in addition to a helium leak onboard, rendered the Boeing Starliner spacecraft too unsafe for NASA’s liking. The agency’s stubborn refusal to let their astronauts be under harm’s way, meant the Starliner returned to earth later in September without its crew.

In the months passing since then, Williams and Wilmore never left the public gaze. Media headlines and TV news anchors have taken to report the event as a major predicament. This is despite the fact, that the astronauts were neither stranded, nor left alone. Williams and Wilmore weigh in on the issue recently during a live interaction with the media.

“Butch (Barry Wilmore) and I knew this was a test flight,” Sunita Williams said to CBS News. “We knew that we would probably find some things (wrong with Starliner) and we found some stuff, and so that was not a surprise.”

Musk made a statement there that sparked controversy. “They were left up there for political reasons.”

This is not to say the situation the duo found themselves in is unprecedented; for it is indeed unprecedented. When NASA had Boeing Starliner‘s software reconfigured and return to earth in one shape. NASA had the benefit of doubt, given their original assessment was made with the best possible evidence available at the time; and not to compromise upon crew safety. As of latest plans, Williams and Wilmore will return to earth by late-March 2025 at the earliest.

But the rhetoric has reinforced calls to put together a “rescue mission.” SpaceX CEO Elon Musk, who advises incumbent US President Donald Trump, claimed at a Fox News interview that his proposal to bring the astronauts back in September was rejected by the previous administration led by President Joe Biden. Musk made a statement there that sparked controversy. “They were left up there for political reasons.”

Narratives draw ire from the space community

Musk’s comments drew ire from other veteran astronauts. Andreas Mogensen, a former ISS commander during Expedition 70, reacted to Musk’s comment on X to say, “What a lie. And from someone who complains about lack of honesty from the mainstream media.” Musk responded in kind soon there after, aggressively standing his ground. However, the astronauts themselves found the claims unsubstantiated.

According to WCVB Boston, Barry Wilmore himself said, “I have not heard that … I’m not sure that could be the case based on what I know. We came up here with a plan to return, and the plan changed.” NASA themselves had issued a clarification in the aftermath of Musk’s own comments, claiming it had never received a direct proposal from SpaceX for any mission. Nor did they warrant such a “rescue mission”, as now President Trump has called on for.

Political considerations are not a factor in changing the timelines in the ISS expeditions. “The White House was very good about letting us make safety decisions and leaving that to the experts at NASA,” Bloomberg reported Pam Melroy, an ex-NASA administrator involved in the mission, as having said.

Long-exposure photograph taken on July 3, 2024, of the Boeing Starliner docked to the ISS, with the earth in the background | Credit: Matthew Dominick/NASA

“Help us change the rhetoric …”

Risks and derailed plans are part and parcel of space travel, and something space agencies draw backup plans for. Much of the public angst and concern for the astronauts is the loneliness arising from prolonged isolation in space, and fears of mishap with the ISS.

“That is what the human space flight program is; it prepares for any and all contingencies that we can conceive of, and we prepare for those,” Newsweek reports Sunita Williams as having said. Health professionals on ground have helped monitor and manage their physical and psychological fitness. Inadvertently, they contribute to research studying the human body’s ability to adapt in the micro-gravity conditions; as well as psychological resilience and the astronauts’ ability to handle stress. But this is nothing astronauts cannot handle. In fact, Williams compared her situation with that of a tourist. “I call it a little vacation from earth.”

“So if you’ll help us change the rhetoric, help us change the narrative…let’s change it to ‘prepared and committed’ rather than what you’ve been hearing,” WCVB Boston reported Williams as having said.

They have had astronauts from the Crew 8 expedition give them company during the arrival in June, assisting them with their microgravity-based scientific experiments. In September, they were joined by a new party of astronauts of the Crew 9 mission – Roscosmos’ Alexander Gubnov, and NASA’s Nick Hague – replacing the astronauts from Crew 8.

In addition to extra clothing and stockpile of food, NASA had left two extra seats were left empty for Williams and Wilmore to return along with Gubnov and Hague on their return later this March or April, when astronauts from upcoming Crew 10 dock later this month. Given there is a spacecraft docked to the ISS at all time, they have all what it takes to evacuate during an emergency.

“So if you’ll help us change the rhetoric, help us change the narrative…let’s change it to ‘prepared and committed’ rather than what you’ve been hearing,” WCVB Boston reported Williams as having said.

Researchers at MIT have unveiled an autonomous, programmable computer integrated into elastic fibers that can monitor health conditions and physical activity, offering real-time alerts for potential health risks. The fiber, which is nearly invisible to the wearer, is comfortable, machine washable, and can be embedded in clothing such as shirts or leggings.

Unlike traditional “wearables” that monitor health from a single location, such as the wrist or chest, this fiber-based technology offers a unique advantage. It is woven into fabrics, allowing it to stay in contact with large areas of the body, including those close to vital organs, thus enabling a more comprehensive understanding of human physiology.

The fiber computer incorporates a range of microdevices—sensors, microcontrollers, memory, Bluetooth modules, optical communication, and a battery—into a single elastic fiber. MIT researchers attached four fiber computers to a top and a pair of leggings, with each fiber running along a limb. These computers were programmed to use machine learning to autonomously recognize different exercises, achieving an average accuracy rate of about 70%. Remarkably, when the individual fibers communicated with each other, their collective accuracy increased to nearly 95%.

Yoel Fink, Professor of Materials Science and Engineering at MIT and senior author of the study, shared his vision for the future of this technology: “Our bodies broadcast gigabytes of data through the skin every second in the form of heat, sound, biochemicals, electrical potentials, and light, all of which carry information about our activities, emotions, and health. Unfortunately, most if not all of it gets absorbed and then lost in the clothes we wear. Wouldn’t it be great if we could teach clothes to capture, analyze, store, and communicate this important information in the form of valuable health and activity insights?”

In a real-world test, U.S. Army and Navy service members will wear fiber-computer-equipped base layer shirts during a month-long winter mission to the Arctic. The mission, dubbed Musk Ox II, will cover 1,000 kilometers in temperatures averaging -40°F. These fiber computers will provide valuable health data, helping to ensure the safety of participants in extreme conditions.

“In the not-too-distant future, fiber computers will allow us to run apps and get valuable health care and safety services from simple everyday apparel,” said Fink. “We are excited to see glimpses of this future in the upcoming Arctic mission through our partners in the U.S. Army, Navy, and DARPA.”

This research builds upon more than a decade of work at MIT’s Fibers@MIT lab and was supported by various military and academic institutions. The breakthrough comes from overcoming a major engineering challenge: integrating complex microdevices into a fiber that retains flexibility and durability. The researchers achieved this by using a flexible circuit board design and an advanced thermoplastic elastomer that allows the fibers to stretch more than 60% without breaking.

The fiber computers, which can communicate via Bluetooth to a smartphone or other devices, enable the creation of a textile network within garments. When multiple fibers are embedded in a garment, they form a network that shares data and enhances functionality, as seen in their exercise-recognition model. This breakthrough could revolutionize health monitoring and injury prevention.

As the team looks ahead, their next steps include enhancing the interposer technique to incorporate additional microelectronic devices. The team is also preparing for the Arctic mission, where the fibers will be used to monitor the physiological effects of extreme cold on the human body.

U.S. Army Major Hefner, who will lead the Musk Ox II mission, highlighted the potential of this technology: “One of my main concerns is how to keep my team safe from debilitating cold weather injuries—something conventional systems just don’t provide. These computing fabrics will help us understand the body’s response to extreme cold and ultimately predict and prevent injury.”

Karl Friedl, Senior Research Scientist at the U.S. Army, emphasized the transformative potential of the technology: “Imagine near-term fiber computers in fabrics and apparel that sense and respond to the environment and to the physiological status of the individual, increasing comfort and performance while providing real-time health monitoring and protection.”