Transmitting the sense of touch via the internet

What JPEG, MP3 and MPEG are for images, audio and video, haptic codecs are for transmitting the sense of touch via the internet.

After eight years of standardization work under the consortium leadership of the Technical University of Munich (TUM), a standard for the compression and transmission of the sense of touch has been published for the first time under the name "Haptic Codecs for the Tactile Internet" (HCTI). It lays the foundation for tele-surgery, tele-driving and new online gaming experiences.

When audio or video files are sent over the internet, the process is quite simple from today's perspective: every 20 milliseconds, a data packet is put together from which information irrelevant to human vision and hearing is already filtered out. This reduces the amount of data. Information is only sent in one direction, to the recipient.

When transmitting haptic information, both the sender and receiver play an equal role. To change the position of a robot arm remotely, for example, the user indicates this intention by moving. If the hand on the robot arm grips a tennis ball, the user senses this from a distance. Information must flow in both directions.

A global control loop is created, with the commands to the robot in the remote environment and the haptic feedback transmitted back to the user influencing each other. Ideally, the haptic information must now be transmitted in a millisecond, a speed usually used in physical interactions with robots.

To reduce the amount of data to be sent, programs known as codecs encode and decode data for transmission. This enables data to be transmitted efficiently. "The IEEE standard 1918.1.1, recently published for the first time, defines a codec as the standard for tactile data transfer," explains Prof. Eckehard Steinbach, Head of the Chair of Media Technology at TUM.

It records the sensations for movements, namely the positions of the limbs and forces acting on them, and for the sensitivity of the skin, to make it possible to feel paper or metal, for example. These two haptic codecs are supplemented by a standardized protocol for exchanging device properties, known as handshaking, when establishing a connection.

How HCTI works: Up to 4,000 data packets per second

In contrast to image, audio and video compression standards, it was previously common practice to send data packets in both directions up to 4,000 times per second for the transmission of tactile information.

"This places very high demands on the communication network that transports the data packets," explains Steinbach. The advantage of the high clock rate is that the teleoperation is realistic and the transmission is highly robust, even if individual data packets are lost. Nevertheless, the researchers want to reduce the clock rate to around 100 times per second. "That's close to the human perception threshold," says Steinbach.

In 2014, a working group within the IEEE Standardization Association was launched with researchers from Imperial College London, New York University of Abu Dhabi (NYU Abu Dhabi), Dalian University in China and the German Aerospace Center (DLR), among others, under the consortium leadership of TUM. The aim was to develop a standard for haptic communication.

"The new codec is something like JPEG or MPEG, only for haptics," explains Prof. Steinbach, who has led the standardization group for the last eight years.

For the new compression standard, called HCTI, the researchers have optimized both the control loop between the sender and receiver and the compression of the data. The crucial feature: Even if data packets are sent over long distances, this should not be noticeable at the other end of the line.

"The integrated control has a stabilizing effect. The forces exerted by a distant robot, for example, are slightly dampened. Hard surfaces feel softer," says Steinbach, describing the now standardized solution, in which Prof. Sandra Hirche and Prof. Martin Buss from TUM were also involved.

Possible applications for the tactile standard

The standard will be of interest for various future applications:

• Telesurgery: The new standard avoids oscillations over arbitrary distances. This means that an operating theater robot can be controlled just as well remotely as onsite. Experts from leading heart centers in Munich or New York, for example, can be called in for certain operations and perform the surgery themselves.
• Teledriving: Companies working on autonomous driving are already offering teleoperation services. Drivers do not sit at the wheel of the vehicle, but in "driving centers" from which they control vehicles remotely.
• Ultrasound in the ambulance: Paramedics can provide first aid but are not authorized to take ultrasound images. In critical situations, this can be done by a doctor while en route to hospital.
• Gaming and film industry: HCTI can bring computer games and films closer to reality and deliver convincing sensations. An exoskeleton can be used to transmit vibrations in a car or centrifugal forces in tight curves, for example.
• Shopping: Online clothing shoppers will not need to have items shipped to them to find out how they feel.

"In the case of JPEG, MP3 and MPEG, many applications emerged after the standards were made public," explains Prof Eckehard Steinbach from TUM. "I expect the same from our new haptic codecs."