Lightweight neural network enables realistic rendering of woven fabrics in real-time

Recent advances in the field of artificial intelligence (AI) and computing have enabled the development of new tools for creating highly realistic media, virtual reality (VR) environments and video games. Many of these tools are now widely used by graphics designers, animated film creators and videogame developers worldwide.

One aspect of virtual and digitally created environments that can be difficult to realistically reproduce is fabrics. While there are already various computational tools for digitally designing realistic fabric-based items (e.g., scarves, blankets, pillows, clothes, etc.), creating and editing realistic renderings of these fabrics in real-time can be challenging.

Researchers at Shandong University and Nanjing University recently introduced a new lightweight artificial neural network for the real-time rendering of woven fabrics. Their proposed network, introduced in a paper published as part of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers '24, works by encoding the patterns and parameters of fabrics as a small latent vector, which can later be interpreted by a decoder to produce realistic representations of various fabrics.

"Our paper comes from the [desire] to improve the realism of the real-time virtual world," Xiang Chen, co-author of the paper, told Tech Xplore. "Fabrics are widely used and influence the realism, so a method to render real fabrics in real time is a must. We found that the latest surface-based fabric models are lightweight but realistic, which lay a foundation for real-time fabric rendering, so we tried to propose a method to achieve it."

Woven fabric patterns (i.e., the patterns observed in textiles that are formed by woven threads) tend to be regular and repetitive. Chen and his colleagues set out to leverage their repetitive nature to enable their realistic rendering in real-time.

The algorithm they developed first encodes the regular patterns of woven fabrics into a small latent vector. This vector is then processed by a small decoder, which interprets it and utilizes the information encoded in it to produce realistic representations of a given fabric.

"We propose a neural network with encoder-decoder structure," Chen explained.

"By encoding woven fabric materials into latent vectors, our network can represent multiple woven fabric types once trained. Moreover, we found the distribution of our target is complex but can be separated. Thus, we separate that into four simpler parts, allowing a lightweight decoder to represent them."

While the network developed by the researchers is tiny and lightweight, it was found to effectively and rapidly create realistic reproductions of many types of fabrics. In contrast with other computational approaches introduced in the past, it also enables the rendering and subsequent editing of rendered fabrics in real-time.

In initial tests, Chen and his colleagues showed that their algorithm enables the rendering and editing of woven fabrics at a remarkable speed of 60 frames per second on an NVIDIA RTX 3090 graphics card. Notably, it can also produce high quality renderings that are free from visible noise and discernible defects.

In the future, the team's neural network could be integrated into graphics design platforms, allowing designers to further improve the realism of the videogames or animations they are creating. Meanwhile, Chen and his colleagues plan to extend the capabilities of their algorithm, for instance, allowing it to realistically reproduce other textiles beyond woven fabrics.

"By introducing fabric representation models into real-time rendering, our approach could further improve the realism of various real-world applications, such as video games," Chen added.

"Currently, our method only supports woven fabrics, but we soon want to extend it to other fabric types like knitted fabrics. A further direction we plan to explore is the representation of more complex fabrics."

© 2024 Science X Network