Ironing out the material challenge for the Metaverse and digital twins

Better file formats and standards for representing 3D structures such as USD and glTF have played a crucial role in the evolution of the metaverse and digital twins. On the other hand, there is far less agreement on the representation of materials. Vendors, artists, and companies today solve these problems by adhering to a specific ecosystem of design tools and rendering engines, or by creating multiple versions.

Now the 3D industry and industrial designers are looking for ways to encourage interoperability of materials across tools. This could allow designers or companies to create a virtual representation of new fabrics, upholstery, styles, shoes or colors and accurately render them anywhere via different tools and 3D worlds.

There are actually two complementary challenges in physical interoperability. First, each rendering engine has a different approach to capturing and representing the physical appearance of materials under different lighting conditions. Second, there are several ways to represent the physical properties of materials, e.g. B. how they fold, drape, feel, blow in the wind or resist fire.

It might take a while for the industry to converge on one format. Various file formats have emerged to support the exchange of materials across tools and virtual worlds, including U3M, AXF, MDB, MTL, KMP, and SBS, each with their strengths and weaknesses. Industry-specific formats may be dominant within their respective domains, while others are used across domains.

A realistic look

Companies that create 3D assets for games and entertainment are exploring how better material processing techniques like Physically Based Rendering (PBR) can improve the look of virtual worlds. “People usually think of a material as cloth, but the 3D industry is talking about materials as something visual,” Elliott Round, co-founder and CTO of M-XR, a 3D production platform, told VentureBeat.

Most people are familiar with the way primary colors like red, yellow, and blue combine to create a variety of colors. Materials take it a step further with additional texture maps representing other properties such as albedo, metal content, roughness, opacity, and scatter. This is where it gets complicated. “Different render engines have different amounts of material properties,” Round explained. “Some will have five parameters while others may have ten, so they can all work a little differently. We hope to solve that with other companies to unify 3D a bit more.”

Industry has traditionally faced computational and storage limitations to accurately represent materials. But now, with better computers and algorithms, these limitations are gradually disappearing. “I hope to get to a position where we no longer have to cut any corners and chop materials because there are fewer restrictions,” Round said. “It could be unified like in the real world.”

His company has developed tools and techniques to quickly transfer the visual properties of real objects into virtual worlds. They started using tools like photogrammetry and structured light scanning to capture 3D objects. “All of these approaches give you really good 3D geometry, but none of them gives you material information. And that’s arguably the key to photorealism,” Round explained. This includes aspects of how light is reflected from an object and whether it is scattered, absorbed or transmitted.

His team also studied different types of pattern scanners commonly used in the textile industry. These types of scanners from companies like Vizoo and X-Rite can capture visual material properties by scanning fabric swatches or pieces of paper. Artists and enterprise apps can later apply these to 3D objects. Round said these scans are really good, but don’t work particularly well for scanning a whole object, prompting research into better techniques for capturing whole objects. Epic recently invested in M-XR to scale these tools for 3D developers.

A realistic feeling

Companies that produce physical materials such as textiles, upholstery, and apparel face additional material challenges. You also need to capture the physical feel of things using different tools and approaches. For example, Bru Textiles, a Belgian textile giant, has spent four years developing a workflow to capture visually and physically correct textile digital twins for its new Twinbru service. Twinbru Partnership Development Manager Jo De Ridder told VentureBeat: “[The digital twin] is a 100% replica of the physical fabric, both physically and in terms of specification.”

This helps design firms create realistic prototypes, e.g. B. a new hotel lobby, and quickly explore variations for customers. In the past, they had to approximate the look through a swatch book and create a mockup that didn’t always look like the finished product. “Digital twins shorten the supply chain, reduce complexity and increase accuracy,” said De Ridder.

However, it is a complex process. It took the Twinbru team years to develop and streamline the workflow to capture the visual and physical properties and convert them into digital twins. They used a combination of X-Rite and Vizoo scanners to capture AXF and U3M files representing visual aspects of the fabrics. In addition, they worked with Labotex to capture the physical properties of the textile in a SAP database, which is converted to the appropriate physics engine format. They have created digital twins of the stuff available for Nvidia Omniverse, Chaos Cosmos, ArchiUp and Swatchbook.

Improved industry collaboration could help streamline similar workflows for other companies that manufacture and work with textiles, paints, fabrics and other materials. A 2020 Digital Fabric Physics Interoperability Survey by the 3D Retail Coalition concluded that it is now possible to measure five physical fabric attributes once and accurately translate them into the equivalent physical values ​​for multiple 3D apparel software solutions. These include flexure, strain/elongation, shear/diagonal strain, weight, and thickness.

Industry leaders are also beginning to collaborate on open standards. For example, Browzwear, a maker of 3D fashion design software, is working with Vizoo to advance the adoption of the Unified 3D Material (U3M) standard in the fashion industry. A big plus compared to other formats is that it can capture both the visual information and the physical properties of the fabric.

“I firmly believe that advancing the metaverse to mass adoption requires materials and textures to be accurately represented,” Browzwear CEO Avihay Feld told VentureBeat. “Synthetic visions with digital twins as frozen snapshots of the physical world are a good place to start. Digital twins are even better as an evolving image of reality synchronized with reality.”

He argues that it’s not clear where the Metaverse is going, but it’s easy to imagine two possibilities. One is a metaverse that represents a departure from reality in which virtual worlds defy the laws of physics. The other is a metaverse that mimics reality so users have experiences that match those in the real world.

He believes that a true-to-life representation of both visual and physical properties will be essential in this second case. Having realistic things in the virtual world will make it more immersive and compelling, but it will also allow the metaverse to support a variety of use cases. An important one is trading, not with purely digital items, but with real objects. In this second case, from the point of view of visualizing textures and simulating the physics of an object, it is essential to have real digital twins.

“It’s possible for these two possibilities to coexist, but without the real-world experiences, the metaverse will likely remain a fantasy world for techies, rather than the transformative new universe it could be,” Feld said.