Preserving heritage sites and objects is a race against time and elements. But thanks to scanning technologies, experts are now more confident humanity’s cultural legacy can be saved.
One of the most iconic and visited landmarks in the world is the Notre-Dame Cathedral in Paris. So, it came as a shock on the evening of April 15, 2019, when fire razed the 850-year-old Gothic structure, causing its roof and spire to collapse. As firefighters battled to stop the flames, the world watched nervously for 15 hours to see if the cathedral could be saved.
Still, even if most of the religious relics and artworks were successfully saved, the cathedral’s wooden structures were severely damaged. Clearly, restoration work must be done. And with Paris playing host to the 2024 Summer Olympics, the French president, Emmanuel Macron, pledged to rebuild Notre-Dame within five years.
But how does one start to reconstruct a 12th century edifice that was built over the course of 200 years? Conservationists were initially afraid that they would have to rely on surviving sketches made during the cathedral’s last major restoration in 1864. But by a stroke of luck, rebuilding Notre-Dame will be slightly less challenging thanks to the scanning work started a decade ago by an art historian.
In 2011, art professor and historian Andrew Tallon realized that he could use laser scanning technology to deconstruct the way medieval workers built Notre Dame. Collaborating with his students and Art Graphique & Patrimoine (AGP), a Paris company that specializes in architectural and archaeological surveys, they ditched their pencils, steel tapes, and other old tools of trade, and substituted them for tripod-mounted laser scanners. These light-emitting machines swept the cathedral’s structure, collecting data points to recreate the centuries-old edifice in three-dimension.
With the original archived scans collected by Tallon’s team, and additional data from more recent scanning projects, experts were able to complete the highly-detailed digital twin of the cathedral in March 2021, almost two years after the fire. This 3D model, made up of 50 billion points, is now the foundation for the Notre-Dame’s reconstruction.
Scan them while you can
Laser scanners collected digital data of the interior of the Notre-Dame Cathedral after the fire. The resulting 3D model is now being used to assess the damages and help in the cathedral’s reconstruction. Credit: AGP
“The Notre-Dame fire is a case study which sets a precedent for the acquisition, processing, and sharing of 3D replicas to the wider audience as a form of public good,” says Jonathan Chemla, an engineer who specializes in Computer Vision and Image Processing. He is also the chief technology officer at Iconem, a French company dedicated to the digitalization of the world’s endangered cultural sites.
Established in May 2013, Iconem has been creating 3D replicas of cultural heritage sites in and out of Europe, more notably in places where a different set of threats affect mankind’s built legacy.
“Our founder, Yves Ubelmann, an architect, has witnessed the disappearance of many heritage sites and artifacts in Afghanistan, Syria, and Pakistan due to urban expansion, floods, earthquakes, massive tourism, looting, and conflicts,” says Chemla. “Yves wanted to find a way to precisely and quickly record the state of a site at one point in time and in 3D.”
And Iconem is doing just that. For example, they were the first team to build realistic 3D models of the ancient city of Palmyra, a UNESCO World Heritage Site in Syria, despite the wave of violent occupations that rocked the country. “In Palmyra, we acquired scans in 2016 and 2017, a period marked with heavy destruction on heritage perpetrated by ISIS,” says Chemla. “Our aim in Palmyra was to use the huge visual data that we have collected to save what remains of Syria’s cultural heritage.”
Iconem has also scanned similar UNESCO World Heritage Sites in other conflict zones, such as the minaret of Jam in a region under Taliban control in Afghanistan, as well as the ruins of the old city of Mosul in Iraq. “Our 3D replica of the minaret in Jam was used to evaluate and assess the damages suffered by the structure due to recent floods, while our scanning work in Mosul has helped UNESCO prioritize their restoration work,” he adds.
Chemla recognizes that working in these areas can be risky, thus making the best use of time spent on the ground vital. “We’ve had projects where timing was critical and being able to quickly acquire the scanned data was key,” he says. “Reducing the duration of scanning missions in the context of emergency or conflict is very useful.”
Saiful Bakhri (left) with team members Sophie Russel (right) and Anna Carlos (bottom) working in Guiuan and capturing the image of a wooden carved door using Reflectance Transmittance Imaging. Photo: Kim Vernon
Apart from timing, another important factor when conducting scanning missions in unstable regions is trust. Chemla says that they “rely heavily on security rules enforced by trusted partners like UNESCO.”
UNESCO, the main agency under the United Nations for heritage protection, helps countries in protecting and preserving their natural and cultural heritage so that future generations can enjoy their irreplaceable values.
By inscribing important places to the “World Heritage Site” list, UNESCO not only acknowledges the outstanding universal value of these locations, but also bestows a broad set of legal framework and policies for their protection.
UNESCO has also has been promoting the use of digital technologies for documenting the status of heritage sites. Among these tools include terrestrial laser scanning and photogrammetry, as well as aerial-based data capture techniques using drones. These cutting-edge gadgets can be used to create 3D representations of important cultural sites and objects, often in very high definition.
“In the field we mostly use aerial and terrestrial photogrammetry, or sometimes aerial lidar for particular contexts,” says Chemla as he describes their 3D modeling workflow. To capture images from high altitudes, Chemla says they rely on fixed-wing drones, especially to acquire data from natural or urban landscapes at medium resolution. Conversely, multi-rotor drones are used to fly closer to monuments and other objects of interest.
“These aerial data are combined with images taken from the surface-level, and with the use of ground control points for georeferencing, we can immediately process some initial 3D scans in the field,” he says. “Then back in the office, we launch the full processing workflow resulting in 3D point clouds that combine multiple resolutions of the heritage site, its landscape, and its surroundings.”
The resulting 3D models of heritage sites are useful for many reasons. In extreme cases, for instance, the 3D documentation may be the only record left of these valuable sites. Archaeologists will rely on them for visual inspections and analysis, while local officials will use them for planning future restorations.
Another reason for scanning heritage sites and creating their 3D avatars is to relive their value, albeit virtually.
“These 3D scans are now being used more and more to produce immersive exhibitions that teleport the visitor to these distant and sometimes vanished places,” says Chemla.
In 2019, I got to see just that when I went to the Federal Museum in Bonn, Germany, where Iconem presented an exhibition of the company’s heritage projects in 3D. Walking inside the museum halls, while donning a virtual reality-headset, I was transported to the ravaged cities of Mosul, Aleppo, Palmyra, and Leptis Magna within an immersive VR environment. It was a very moving experience.
Yves Ubelmann, founder of Iconem, scans ruins in Palmyra, Syria.
Those who scan, do
In addition to man-made conflicts, the Earth’s changing climate also poses a pronounced threat to heritage sites. One recent case is the flood of 2019 in Venice that submerged more than 80 percent of the historically-important Italian city. This was the turning point that made local officials realize how rising sea levels can damage their canals and other cultural heritage sites, risking the city’s main touristic draws.
Iconem collaborated with several organizations spearheaded by Factum Foundation, a non-profit group from Spain, to aid in Venice’s preservation by using laser scanning technology. In 2020, they turned to lidar and photogrammetry to create a three-dimensional digital model of the entire island of San Giorgio Maggiore, recording every inch of its surface.
“Using the right technology and software, one can quickly visualize, manipulate, and extract meaningful data from the 3D scans,” says Chemla.
But according to Saiful Bakhri, an archaeologist and conservator, there is still a gap in the familiarity of the general public about these technologies despite their availability. Bakhri works for the Bali Cultural Heritage Preservation Office in Indonesia, and he also does scanning and photogrammetry in his job.
“I am not sure how to address this technological gap,” he says. “So, I guess it goes back to awareness-raising of the cultural stakeholders on the importance of these new technologies to increase their use in the field.”
In 2018, he did just that when he joined a group of conservators from the University of Melbourne during their field work in the island of Guiuan in the Philippines. The island’s 16th century church was damaged by Haiyan, one of the strongest typhoons in recent years, and Bakhri ‘s job was to demonstrate to the local people how imaging technology can be used to create a digital record of cultural objects, providing a point of reference for their future reconstruction.
“Cultural heritage is at risk of natural disasters and climate change, so recording them is becoming more important than ever,” says Bakhri. “Imagine a century-old statue that has been destroyed by an earthquake. If we have a digital record of it, we can ask local artists and craftsmen to recreate the same object to replace it. That scan is intended to provide the most basic yet fundamental method to safeguard cultural heritage which is to preserve by record.”
In Guiuan, Bakhri’s group captured images of the church’s intricately carved wooden doors and the embedded shell decoration in the interior walls using Reflectance Transformation Imaging (RTI). RTI is a computational photographic method developed by Cultural Heritage Imaging (CHI), a non-profit organization based in San Francisco, California.
“RTI provided amazing records of the door and shells due to its versatile lighting angles,” says Bakhri. “I am so grateful that there is an open-source technology from CHI that we could use for our work.
“Our organization is focused on scientific applications of photography-based imaging,” says Carla Schroer, founder and director of CHI. “And that includes photogrammetry to produce 3D, spectral imaging, and Reflectance Transformation Imaging.”
In 2002, Schroer co-founded CHI with her husband Mark Mudge, while she was still working on the Java team for Sun Microsystems. “Mark and I shared a love of art and history and thought we could bring something useful to that space,” says Shroer. “At that time, digital cameras were just moving out of the research labs when we started discussing what later became CHI.”
Schroer emphasizes they did not invent RTI “but along with the original inventors, we adopted it and then fostered a community around it, raised grant funds to develop open-source tools, developed a workflow, began training classes and workshops along with a free user forum to help people use the technology.”
RTI’s usefulness for planning and documenting the treatment of cultural objects comes from its ability to see fine surface details, thus it’s not surprising that this imaging technology has a lot of users in the art conservation community. “Folks working with petroglyphs and other forms of inscriptions and tool marks use the technique for studying and visualizing those surfaces,” says Shroer.
“RTI is complementary to photogrammetry and we often use both techniques on the same project,” she says. “For example, we tend to use RTI for imaging smaller areas to help us see things that are difficult or impossible to discern with the naked eye, while we use photogrammetry where we need measurable 3D data of a whole subject with a larger area.”
Their expertise in imaging can be seen on how they captured every detail of the Pan American Unity mural, the last work that Diego Rivera painted in the U.S. The large-scale mural, hanging at the City College of San Francisco (CCSF) since 1961, was moved in 2021 to the San Francisco Museum of Modern Art where it can be viewed until 2023.
“This 1940 fresco is 22 feet tall by 74 feet long, and we were able to produce an eight-billion-point model of the surface of the mural using photogrammetry,” says Shroer.
Through their work that began in December 2015, conservators were able to assess the condition of the mural and aid in planning for its move. Scans made from photogrammetry were used to build the mural’s 3D model, thereby allowing CHI to make orthomosaics and Digital Elevation Models (DEM), revealing the surface color and shape of the mural in high definition.
“Conservators used the orthomosaics and DEMs to assess whether surface issues in the mural such as abrasions, were the result of the move or happened prior to the move,” she says. “Even though they already had a detailed condition assessment, it could never have the same level of visual detail, especially for an artwork of this size.”
Sharing the scanned goods
According to the UNESCO, mankind’s cultural heritage must not only be preserved, but digital records should be made accessible to anyone who appreciates their universal value.
With this in mind, Iconem has opened the depository of scanned 3D models from their heritage projects and made them available online. “For us, it is important to document heritage in order to share it to a wider audience,” says Chemla.
He believes that sharing and having open access to this digital information “will allow new experiences to be built on top of it, permitting us to reach more people all over the world who are working in the sectors of heritage restoration, education, and tourism.” Chemla also hopes that making their 3D models readily available online “will raise awareness on the threats facing our cultural heritage to help us come up with possible solutions.”
Iconem’s online portal (https://app.iconem.com/) is based on open-source technology standards from the Open Geospatial Consortium so that anyone can easily view, manipulate, and navigate through their massive 3D datasets “without the need to have an ultra-powerful machine,” says Chemla.
“This makes it easier for the wider public to visualize heritage sites, all thanks to our local partners who gave us the authorization to make them accessible.”
Similar online initiatives have been employed by other organizations. CyArk, for example, partnered with Google in 2018 to make its library of laser-scanned 3D models of historical sites available via the Google Arts & Culture mobile app. And in 2020, the Smithsonian Institute and 20-plus cultural organizations around the world opened their digital collections of historic artifacts to a popular 3D web platform, Sketchfab, effectively reaching out to a big online 3D modeling community. During the pandemic-induced lockdowns last year, viewing these digitized artifacts became popular while museums were off-limits to visitors.
This triumph of tech in the cultural sector is laudable. Nevertheless, CHI’s Shroer thinks that caring for our cultural heritage should not end with technology. “One issue facing quality documentation of cultural subjects today is not so much a technological one, but rather one of power and control,” she says. “Local and indigenous people, and cultural caretakers of all stripes should be empowered to document their own material, to decide the priorities, and also to decide what should and shouldn’t be shared.”
Shroer adds that successful heritage documentation does not just mean putting technology in people’s hands. “We also need to have a clear methodology for collecting scientific data, keeping track of metadata, and making it easy for folks to deposit their digitization work with repositories for long term use and re-use.”
Putting action to where their heart is, CHI has created an open source software called the Digital Lab Notebook (DLN), which has been designed to specifically do that. “The DLN is a simple-to-use metadata knowledge management system for the capture, generation, and archiving of advanced computational photography-based digital representations,” says Shroer. CHI has recently completed a project funded by the U.S. National Endowment for the Humanities to develop this software.
“We envision a world where the documentary work of an uncredentialed local cultural caretaker, who learns a good practice-based imaging workflow and does it properly, will stand toe to toe with the work from the most respected and authoritative sources,” she says.
Scanning and other related technologies, by themselves, cannot stop the impact that time and elements have on mankind’s cultural legacies. But it is hard to imagine how the problem can be solved without them.