Tactile avatar system trained using the tactile decisions made by its human counterpart, capable of making tactile decisions similar to a human. Credit: Advanced Science, doi: 10.1002/advs.202002362

Tactile avatar is an artificial tactile perception and cognition system used as a surrogate for human tactile cognition with potential to produce smooth or soft and rough tactile sensations by its user. In a new report now published on Advanced Science, researchers used a piezoelectric tactile sensor to record varying physical information including pressure, temperature, hardness, sliding velocity and surface topography. In this work, Kyungsoo Kim and a team of scientists in information and communication engineering, neurology, brain and cognitive sciences in Korea engineered artificial tactile cognition by testing the tactile feelings of human participants to a variety of materials ranging from smooth or soft to rough. To account for the response variation among humans, Kim et al. designed a deep learning structure for personalization through training based on individualized histograms of tactile cognition while recording physical tactile information. The decision error of each avatar system was less than 2% for 42 different types of materials, where tactile data could be measured with 100 trials for each material. The tactile avatar machine categorized new experiences of materials based on the knowledge of the tactile training data to show a high correlation with the specific user's approach. The scientists intend to propose an advanced method with tactile emotional exchange capabilities for advanced digital experiences in electronic devices.

Digital tactility

Innovations in electrical devices and have advanced the digital experiences based on the five human senses. For instance, virtual reality (VR) can provide unusual and auditory sensations, while augmented reality (AR) can provide more personalized experiences with 3-D spatial images and stereo sound across disciplines of entertainment and internet marketing. These technologies are also evolving to exchange emotions between humans and machines with considerable attention placed on tactile sensor-based technologies. In this work, the artificial tactile system provided 'smooth/soft' and 'rough' tactile sensations based on the user's tactile feelings to develop a 'tactile avatar.' The method mimicked psychological tactile feelings based on a piezoelectric sensor system and a deep learning process. The human-like sensor and processing system provided an artificial tactile cognition system allowing the researchers to test the performance of the device for tactile -making and understand its performance with untrained or novel tactile materials.

Human tactile decisions for 42 tactile samples. a) Averaged histogram of the tactile decisions of 10 participants; (a)‐1 and (a)‐2 are the graphical representations in 2D and 3D, respectively. b) Tactile decision difference (mean RMSE), that is, difference between the average decisions of each subject. Dashed horizontal line, which indicates the average of tactile decision RMSE divides the S (lower) and D (upper) groups. c) Tactile decision standard deviation values for the S (red, n = 7) and D (blue, n = 3) groups. Dots and shadows correspond to the group mean and variance, respectively. d) Example tactile decisions of the four participants identified in (b). Credit: Advanced Science, doi: 10.1002/advs.202002362

Designing the tactile avatar

The human tactile system is complex and remains to be clarified in detail. To represent human tactile cognition, Kim et al. trained the system using individual tactile decision histograms. The setup obtained the baseline signal arising from touching and sliding processes in the parallel input layer. The tactile sensor generated signals relative to the hardness, temperature and surface features of the materials much like those made by humans. The slope and oscillation frequency of the touch signal contained information of the hardness and surface topography of the material. The team also built a decision processing system with combined neural network layers to emphasize specific features for sample classification. They processed two types of data on touching and sliding separately in the hidden layer. The neural network assigned multiple labels to different weights that reflected human tactile cognition. They expect the setup to have applications in online shopping and AR/VR environments for desirable tactile sensations. The process can also be included into an artificial skin system to resemble general human tactile feelings.

Information obtained from rubbing and touching the materials. a,c) Surface images of sample No. 2 and No. 38, respectively. b,d) Measured piezoelectric voltage by rubbing sample No. 2 and No. 38, respectively. e) Hardness level of 42 samples measured by commercialized durometer. f) Piezoelectric slope plotted using information obtained by touching hard or soft materials. g) Comparison of the first slope of piezoelectric voltage and measured hardness by durometer with increasing material hardness. Credit: Advanced Science, doi: 10.1002/advs.202002362

Understanding the human tactile system

To design a human-like tactile system, Kim et al. used 42 materials ranked from the smoothest to the softest and roughest, among 10 participants. The samples differed in surface structure, thickness and other characteristics selected from a large library of general materials used in clothing. The participants ranked the materials from 1 to 42, where 1 was smoothest and 42 the roughest. They averaged the test results and ordered the tactile materials accordingly. Human judgements were inconsistent during the ranking process and therefore the team accounted both tactile decision and tactile confusion to assess the sensitivity of humans during the sorting process. The researchers created a colored decision matrix to compare individual differences in tactile decision making in detail and included root-mean-square-error (RMSE) values. The results showed how a unique individuality supported tactile system will be required to mimic human tactile cognition.

Artificial sensor systems for deep learning and deep learning-based tactile discrimination

The team used multiarray tactile sensors made of piezoelectric materials to measure the surface information. They applied a machine-learning algorithm to produce artificial tactile feelings among 42 test tactile materials. The hardness of the material provided one of the main physical parameters to inform the deep learning process of tactile sensation in the study. The scientists used a durometer to measure this and determined the hardness based on piezoelectric signals. The researchers designed a tactile decision system based on the piezoelectric signals, contingent on a combination of neural network layers. They designed each parallel network to extract specific features and conduct complex classification processing in order to mimic human cognition for tactile classification based on tactile receptors. The researchers adopted labels to train neural networks and classified tactile materials with artificial sensors. During the training process, the team optimized the network to pick the correct tactile sample from among all untrained samples.

Deep learning for tactile decision‐making. a) Histogram of human tactile cognition. b) Tactile histogram of the machine trained using labels. c) Output layer of the proposed histogram‐based network for the 42 tactile materials with textures ranging from smooth/soft (bottom) to rough (top). d) Classification accuracy according to the tolerance bounds. Credit: Advanced Science, doi: 10.1002/advs.202002362

Optimized machine learning

The scientists then proposed a new approach to train the machine to make tactile decisions by using a human tactile decision histogram. To accomplish this, they mapped the human tactile decision histogram to the output nodes to train the machine, where the histogram contained information relative to the averaged tactile decision of the humans and decision confusion-based variance. Instead of focusing on higher classification accuracy, the researchers focused on mimicking human tactile decision making with the deep learning network. Kim et al. noted larger mean RMSE (root-mean-square-error) for human participants compared to the tactile decisions of the machine. The touching and sliding motions allowed the machine to compare a newly experienced material surface to multiple trained materials to classify new tactile materials based on pre-existing knowledge. Kim et al. formed the tactile avatar system on a neuromorphic system to reduce the time delay of computation and minimize the size of the machine to perform the realization.

Fully trained network developed to mimic the tactile decisions made by humans. a) human and b) artificial tactile decisions for the four participants represented earlier. Analysis of the tactile decision similarity between the machine and human participants, based on c) mean and d) kurtosis values. Data for trained are presented as the mean ± Standard deviation.

In this way, Kyungsoo Kim and colleagues developed a tactile avatar system with a multiarray tactile sensor fabricated from piezoelectric materials and a deep learning process based on human tactile cognition. They expect future research to improve the capabilities of the tactile avatar to process tactile information, which will allow machines to replace humans in virtual spaces such as online shopping malls.

More information: Kim K. et al. Tactile Avatar: Tactile Sensing System Mimicking Human Tactile Cognition, Advanced Science, doi.org/10.1002/advs.202002362

Chortos A. et al. Pursuing prosthetic electronic skin, Nature Materials, doi.org/10.1038/nmat4671

Rich S. I. et al. Untethered soft robotics, Nature Electronics, doi.org/10.1038/s41928-018-0024-1

Journal information: Nature Materials , Nature Electronics