The Winect system transmits WiFi signals and analyzes their reflections off human body to generate dynamic 3D human skeletons. Credit: Ren & Yang.

Wireless sensing devices, tools that allow users to sense movements and remotely monitor activities or changes in specific environments, have many applications. For instance, they could be used for surveillance purposes as well as to track the sleep or physical activities of medical patients and athletes. Some videogame developers have also used wireless sensing systems to create more engaging sports or dance-related games.

Researchers at Florida State University, Trinity University and Rutgers University have recently developed Winect, a new wireless sensing system that can track the poses of humans in 3D as they perform a wide range of free-form physical activities. This system was introduced in a paper pre-published on arXiv and is set to be presented at the ACM Conference on Interactive, Mobile, Wearables and Ubiquitous Technologies (Ubi Comp) 2021, one of the most renowned computer science events worldwide.

"Our research group has been conducting cutting-edge research in wireless sensing," Jie Yang, one of the researchers who carried out the study, told TechXplore. "In the past, we have proposed several systems to use Wi-Fi signals to sense various human activities and objects, ranging from large-scale human activities, to small-scale finger movements, sleep monitoring and daily objects For example, we proposed two systems dubbed E-eyes and WiFinger, which are among the first work to utilize Wi-Fi sensing to distinguish various types of daily activity and finger gestures."

While the wireless sensing systems developed by the researchers in their past studies achieved promising results, they mostly rely on models that were pre-trained on a fixed set of known activities, thus they can only classify a limited number of human poses or movements. In their new study, Yang and his colleagues explored whether they could also use Wi-Fi signals to sense free-form human activities (i.e., involving swift and more elaborate movements). The accurate tracking and estimation of free-form movements could enhance several real-world computing applications, including virtual reality (VR) implementations, tech-augmented fitness and videogame development.

Winect, the system created by the researchers, can examine movements in home environments by transmitting Wi-Fi signals and analyzing how these signals are reflected off the human body. This allows it to track free-fork movements and human poses in 3D.

A key advantage of Winect is that it uses Wi-Fi-transmitting devices that are already within an environment, such as laptops, , smart TVs or smart speakers, to send out the signals it needs to examine human activities. Subsequently, it employs deep-learning techniques to create a digital version of a user's 3D full-body movements, dividing the body into different parts separated by joints (e.g., head, spine, shoulders, elbows, wrists, hips, knees and ankles).

"Our system first emits the Wi-Fi signals to probe the home environment and then analyzes the signals reflected off the human body for free-form activity tracking," Yang explained. "More specifically, it extracts and analyzes the phase of the received Wi-Fi signals to detect the presence of human activities and the number of moving limbs. Next, our system uses signal processing techniques to separate the Wi-Fi signals reflected from each moving limb and track the trajectory of each limb. "

Winect creates a digital version of a human user's body poses in 3D using deep learning techniques. Essentially, it creates a 3D skeleton of a user's body by modeling the relationship between the movements of limbs and the corresponding joints.

The researchers' system has numerous advantages over other existing wireless sensing systems. Contrarily to computer-vision-based devices such as Kinect or Leap Motion, for instance, it can also sense through walls and is not affected by occlusions, as it does not rely on a camera but on Wi-Fi signals that can pass through physical obstacles.

"In contrast with intrusive wearable systems, such as Xsens, which require a user to wear or attach motion sensors or visible markers on the human body, Winect requires no sensors on the human body and thus is transparent to the users," Yang said. "In addition, as Winect could reuse commodity Wi-Fi devices at home (e.g., laptop, desktops, smartTV, smart speakers), it does not incur an additional cost, and thus is promising for mass adoption for end-users in smart homes."

Yang and his colleagues evaluated their wireless sensing system and found that it achieved remarkable results. In their tests, Winect could track free-form human activities with centimeter-level accuracy in a variety of challenging environments and scenarios. Overall, their findings suggest that Wi-Fi signals reflected from the carry rich information that can be used to extract fine-grained human motions and poses.

"Human-computer interaction applications, such fine-grained motions could be mined to understand human activities and behaviors for smart healthcare applications," Yang said. "For example, in aging-in-place settings, it is very helpful to understand the activities and the behavioral changes of elderly people to detect falls and other situations of need. By tracking the daily activities and generating statistics for a person, it is also possible to monitor the wellbeing and suggest interventions that improve health."

In the future, this 3D human pose estimation system could be used to create more engaging and better-performing applications that involve the tracking of free-form human movements. For instance, the 3D poses it predicts could enhance the performance of smart fitness assistants and VR platforms.

So far, the researchers have primarily focused on the detecting human activities without analyzing the context or environments they take place in. In their next studies, however, they plan to create systems that can track human activities and predict intentions using contextual information following an approach known as context sensing.

"For example, if a user is laying down on the bed without moving, he/she could be sleeping or simply listening to music, so it could be difficult for existing systems to fully understand what they are sensing without the context," Yang said. "Thus, it is important to analyze human activities together with the scene they take place in, by considering the semantic context of its contents and the intrinsic relationships between them."

More information: Yili Ren, Jie Yang, 3D human pose estimation for free-form activity using WiFi signals. arXiv:2110.08314v1 [cs.CV],