Imagine you have a jigsaw puzzle with 10,000 pieces but no picture on a box. In fact, you don’t even have the box—it was destroyed nearly 2,000 years ago. These puzzle pieces are fragments of frescoes in the ancient Roman city of Pompeii that were leveled or buried by the eruption of Mount Vesuvius in A.D. 79. Some pieces are missing, others are broken. And instead of being precisely-cut shapes designed to neatly interlock, they’re damaged, irregular fragments. How would you solve this puzzle?
Scientists at the Italian Institute of Technology (IIT) have a plan: Send in the robot. Their project—dubbed RePAIR (Reconstructing the Past: Artificial Intelligence and Robotics meet Cultural Heritage)—melds robotics, AI and archaeology in an attempt to reconstruct architectural features of Pompeii that would otherwise remain incomplete, because they’re either too complex or would require impossible amounts of human labor and time.
Over the next several months, the IIT researchers and their collaborators at a handful of international universities will build, train, test and deploy a robot to piece together ruined wall frescoes in two buildings. In the first building, they know what the frescoes should look like because they were intact until recently. The investigators have no idea what the frescoes in the second building depict. Its fragments have lain in a storage room for decades waiting for someone—or something—to put them back together.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
RePAIR is experimental and might fail, admits Arianna Traviglia, director of the Venice-based IIT Center for Cultural Heritage and Technology and a principal investigator of the project. It’s funded by a 3.5-million-euro grant from a European Commission fund that supports risky projects aimed at “radically new future technologies.” If the project succeeds, the technology can be used to reconstruct a variety of fragmented cultural artifacts whose restoration has remained “out of human reach,” says project coordinator Marcello Pelillo, a professor of computer science and artificial intelligence at the University of Venice.
“Let’s hope we are successful, because it’s never been tried before,” Traviglia laughs. “It’s really craziness at the highest level.”
THE BRAINS
RePAIR’s as-yet-unnamed robot must have a combination of brains, strength and a sensitive touch. The brains will come from a mix of high-tech computer vision to scan the fresco puzzle pieces, machine-learning algorithms to solve the puzzles and human expertise to guide the AI.
“We think that using an AI-based system will not be enough,” says Pelillo, whose group is developing the puzzle-solving AI in conjunction with a team at Ben Gurion University of the Negev. That’s why they’ll get guidance from archaeologists and art historians about the different styles of frescoes found in Pompeii.
In Pompeii, the researchers are in the process of manually digitizing every fresco fragment in the two test sites to create a digital database for the RePAIR robot. Once the whole system is fully operational, though, “we’ll have the robot doing the scanning process itself,” Pelillo says.
When robots can handle routine digitization, humans are free to handle more complex tasks. Graduate students can’t scan 24/7 without food, water or sleep—but a robot can. “The idea is to work towards automating as much as possible this quite time-consuming and also boring activity of digitizing cultural heritage,” Traviglia says.
Researchers are training soft robotic hands to carefully manipulate ancient fresco fragments found at Pompeii. Credit: IT-Istituto Italiano di Tecnologia.
The humanoid robot is composed of a torso and arms being developed by Nikolaos Tsagarakis and his colleagues at the Humanoid & Human Centered Mechatronics lab at IIT in Genoa. With arms 80 to 100 centimeters long and a weight of 25 to 30 kilograms, the robot will be about the same size as the upper body of an average person. Its design is based in part on earlier “compliant humanoid” robots Tsargarakis’s lab developed for use in disaster sites, including WALK-MAN, which in 2016 explored a damaged building in the Italian village of Amatrice, where an earthquake had killed nearly 300 people.
Hands will attach to the arms through sockets at the wrist. They’re being designed in the lab of Antonio Bicchi, a senior scientist at IIT in Genoa and the chair of robotics at the University of Pisa, who has developed robotic hands for use in industry and as prosthetics. At Pompeii, the soft robotic hands will need to grasp, move and orient fragments of varying sizes and weights with extreme care—and gather information about them in the process. These hands are like soft but smart gloves that are embedded with tactile, kinesthetic and position sensors. Ideally, Bicchi says, people will be able to wear them over their own hands one day to gather additional data that human hands can’t obtain on their own.
The researchers are still working on the final design of the robot (an alternate design forgoes the torso and simply suspends the arms from a metal frame), but it will probably be mounted on a slider on a workbench so that it can easily move from one end of the workspace to the other. Once installed, the soft hands will grasp the pieces, 3-D-scan them, and send the data to the puzzling-solving AI, which will attempt to find virtual matches. Once it has a solution, the AI sends the data to the hands, which assemble the pieces.
THE PUZZLE SITES
The first fresco puzzle the RePAIR scientists will attempt to solve with the new robot is a part of the Schola Armaturarum, the headquarters of a military-style association located on Via dell’Abbondanza, Pompeii’s main street, where members planned military activities and gladiator battles. Unearthed in 1915, its large hall was decorated with representations of trophies and weapons and lined with wooden cabinets that may have displayed armor and trophies. It survived both the volcanic eruption and a direct hit from an Allied bomb in 1943, but parts of it collapsed in 2010 after heavy rains poured on Pompeii, including the wall frescoes the researchers will attempt to reconstruct. Because they know what the frescoes originally looked like, it’s a good place to test out the robot’s capabilities.
If the robot succeeds with the Schola Armaturarum frescoes, the team will next attempt to solve the fresco puzzles found in two collapsed rooms in the House of the Painters at Work. This building is so named because artisans were in the process of painting frescoes there when Vesuvius erupted; archaeologists unearthed their buckets and brushes, as well as the faint outlines of frescoes awaiting color, shading and detail that never came. Parts of the building were destroyed then, and it was also badly damaged by World War II bombings.
Because each crumbled wall represents one fresco puzzle, and there are no reference images to work from, this site poses a far larger challenge. “Imagine you have five different puzzles—one with the Eiffel Tower, the other one with the Coliseum, etcetera,” Pelillo says. “You open the boxes, and you just mix all the pieces. And then you are asked to solve the five puzzles. This is actually what we are going to do.”
They RePAIR scientists aim to install the robot in Pompeii (the exact location is still being decided) by spring or summer 2022. They have some time to experiment; their grant supports the project through 2025.
The researchers see a world of possibility ahead for the robot. Museums and universities across the globe hold fragmented remains in their collections, many of them ignored for decades. “We hope that if we succeed with this project, we will be able to offer a technology which will allow many museums around the world to reconstruct large-scale broken frescoes or similar artifacts,” Pelillo says. “We are focusing mainly on frescoes, but that’s just the case study. If we succeed, we think we can export this technology to other artifacts or even, for example, to papyri.”