Stepping into the future

From facetime filters to smart glasses, mixed reality is seen as the technology of the future and is already changing how we interact with the world.

A leading-edge collaboration between departments in the School of Health Sciences and Elon College, the College of Arts and Sciences, is exploring the technology’s untapped potential to study human movement and improve the health and safety of older adults. It’s also advancing knowledge around how humans interact with these new technologies.

The project, led by Associate Professor of Physical Therapy Education Srikant Vallabhajosula, Director of Clinical Education Alys Giordano, and Assistant Professor of Computer Science Pratheep Paranthaman, compares physical navigation around obstacles in real-world, virtual-reality and mixed-reality settings.

They call it the “MVP Project” for mixed, virtual and physical realities. Beginning with tried-and-true physical therapy practices — training patients to avoid tripping-related falls — then asking whether technology can improve that training with virtual obstacles, their work could improve care for adults with a high risk of falling.

Navigating obstacles and maintaining balance are critical in physical therapy, especially when working with those with increased risk of falls. The ultimate goal is to be able to implement mixed reality and virtual reality obstacle clearance in the clinic.

Most people are familiar with virtual reality, where a user’s field of vision is covered by a head-mounted display unit showing an interactive, 3D space. Mixed reality combines real-world environments with computer-generated content, like how “Pokémon Go” characters materialized in our surroundings by harnessing smartphone cameras and screens. This mixed-reality technology uses head-mounted displays with advanced sensors to map the environment of the physical world, which can then be used to superimpose digital content.

“The technological world is ever-changing, and mixed reality and virtual reality are becoming more and more popular,” says Stacey Walton, a third-year student in Elon’s Doctor of Physical Therapy program completing her second year of research with the project. “Our research investigates those technologies to see if they are an effective way to simulate obstacle clearance — something people deal with every single day. Navigating obstacles and maintaining balance are critical in physical therapy, especially when working with those with increased risk of falls. The ultimate goal is to be able to implement mixed reality and virtual reality obstacle clearance in the clinic.”

Now in their second year, trials are underway in adults at high risk for falls. Elon faculty and students have presented the research at multiple conferences and plan to pursue funding for future phases.

Falls are the leading cause of injury-related deaths among adults over 65, according to the Centers for Disease Control and Prevention. More than one in four people over 65 experience a fall each year — about 36 million incidents annually. More than 13 million falls cause injuries requiring medical care, costing Americans more than $50 billion annually.

As with any skill, practice makes perfect. Seniors who train to clear everyday obstacles and tripping hazards — door thresholds, steps, curbs — are less likely to fall and be injured. That training is common for older adults with mobility issues.

“The hope is to see if we exhibit similar obstacle clearance strategies in mixed reality as compared to virtual and real environments,” Vallabhajosula says. “If that’s true, you can train someone in their own home environment using virtual obstacles.”

First steps

The MVP Project began with simple curiosity. In spring 2021 Vallabhajosula and Paranthaman connected around the technology in their departments. Vallabhajosula’s expertise is in biomechanical analysis of posture and gait. Paranthaman is the coordinator of Elon’s game design minor with expertise in user interface and user experience. He was curious about the motion-capture technology in the School of Health Sciences’ Glen Raven Inc. Biomechanics Lab.

That technology measures physical motion through 12 high-speed cameras that record the movement of small reflective balls worn by users. Game designers employ similar motion-capture technology embedded in suits to program character animation.

Intrigued, they began exploring projects that would advance knowledge in each of their fields. Obstacle-navigation training has been studied extensively in physical-world and virtual-reality scenarios. Not so much for emerging mixed-reality applications.

Paranthaman spent that summer and early fall prototyping virtual- and mixed-reality environments in 20 different iterations, first so School of Health Sciences researchers could better understand the technologies and then to refine their use for the biomechanics study. The initial prototyping of virtual- and mixed-reality applications involved brainstorming, requirement analysis, virtual-space planning, interface design and programming interactions. Paranthaman developed applications for each platform, which were deployed on the Microsoft HoloLens 2 mixed-reality headset and Oculus Quest virtual-reality headset. Paranthaman taught Walton and Oliver Tuisa, also a third-year student in the DPT program, how to operate the applications and hardware.

In October 2021 the study began with 12 young adult participants. It continued in 2022 with 12 older adults at low fall risk. The current team — consisting of Walton, Tuisa, Christopher Go, Alexander Japit, Matthew Lawler and Alicia Wilson — is recruiting older adults with mobility issues.

Study participants aren’t instructed how to walk or clear obstacles. Each is evaluated while walking normally and clearing obstacles at three heights — 2 inches, 6 inches and 15 inches — mimicking everyday activities. Among the 60 trials each participant performs, some include walking and navigating obstacles while counting backward to simulate walking while performing other tasks.

“I think it’s fascinating for students to see how some people will cross an obstacle without much effort, but when you give them an additional task, there are very different behaviors. People walk more slowly and take more time to cross,” Vallabhajosula says. “When we tested our first older adult with high risk for falling, they literally stopped before the obstacle and then crossed.”

Go, a second-year DPT student, applied the research to his clinical experience last winter.

“I’ve gained invaluable experience in evaluating an individual’s gait patterns through this project,” Go says. “We witnessed very different gait patterns in each individual, and I’ve been lucky to work in a clinic with [equipment that measures] a patient’s stride length, impact force and weight distribution. With that, I’m able to help cue and train patients for proper mechanics to decrease pain and improve functional gait.”

Moving forward

Results so far show that most partici­pants successfully clear virtual obstacles with their leading leg but underestimate the heights of obstacles with their trailing leg.

“If it was a real object, they would kick it,” Vallabhajosula says. “Past research has shown that people start planning how to traverse obstacles in the physical environment about two steps ahead. We want to see if that’s similar in mixed and virtual realities.”

Their collaboration has resulted in multiple conference presentations and published articles.

In 2022, Vallabhajosula presented at the North American Congress of Biomechanics in Ottawa, Canada. Paranthaman presented virtually for the World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4) in London.

Walton presented early findings from healthy young adults at the American College of Sports Medicine Annual Meeting in 2022. Walton, who plans a career working with veterans and Paralympic athletes, also presented at the American Physical Therapy Association Combined Sections Meeting earlier this spring. Wilson, a second-year DPT student, will present findings on older adults at low fall risk at the ACSM annual meeting this summer.

“Moving forward, I believe that this study will continue to influence my approaches in the clinic and working with patients,” Walton says. “Being part of this taught me how to build a research project from the ground up and showed me that one study may lead to different questions or to continue through phases to reach an ultimate conclusion.”

Both Vallabhajosula and Paranthaman have enjoyed and appreciated this inter-departmental collaboration and are planning additional future partnerships.

“This collaborative research allowed us to investigate and address research questions from our distinct fields, biomechanics and computer science,” Paranthaman says. “As a games researcher, this project contributed to more versatility, where I was able to apply design principles and development processes from human-computer interaction outside the field of video games and computer science.”

Paranthaman values better understanding user experiences in virtual and mixed realities. Users displayed higher emotional responses in virtual reality but were more comfortable moving in mixed-reality environments where they could see their surroundings. He continues to research those aspects of user experience, including observation of users’ brain activity in each medium, and apply them to his teaching.

“This research directly pertains to my teaching because it’s so centered on user experiences in different contexts,” Paranthaman says. “It’s helped me to design courses and give students a real-world example of the challenges and failures in research and development. Those are an essential part of research because of how you learn from them.”

Vallabhajosula has enjoyed watching students gain confidence in the research process and experience with emerging technology. He is excited about future studies that will illuminate mixed reality’s role in therapy. He plans to apply for external funding to continue trials beyond this academic year.

“We need more training studies. Multiple iterations of this kind of research would show if it will be effective, or if it’s possible but we need to perfect the process, or if it isn’t effective,” Vallabhajosula says. “We hope the outcome will be that we can train patients to avoid falls using this technology, but first we must understand how people behave in mixed-reality environments. You can train as much as you want, but when it comes to a person walking on carpet, will they navigate obstacles correctly?

“How will it translate to the physical environment? That’s the million-dollar question.”

Sidebar: Simulation and health care

The use of augmented-reality, virtual-reality and mixed-reality technology simulation is opening the door to a new world in the education of health care practitioners.

Thanks to a Golden LEAF Foundation grant, Elon’s School of Health Sciences has taken the first steps toward creating a mixed-reality lab as part of its Interprofessional Simulation Center. The center serves as the hub for experiential health care learning. Once it’s fully functional, the space will allow faculty to develop complete scenarios in low-stakes environments for students to practice and grow confidence and skill in high-pressure situations. The lab will also be a space for health care providers in the local community to continue their education, thus strengthening the quality of care in the region.

“I am so excited about the potential of this lab,” says Nita Skillman, the school’s director of interprofessional simulation who is overseeing the development of the space. She is currently exploring different software packages to determine the right combination that can best address the needs from different academic programs as well as the community.

The lab will be able to accommodate the planned growth of the School of Health Sciences in the coming years as Elon’s new nursing programs continue to expand. With that in mind, the implementation of the lab has several goals that include creating a library of augmented-reality, virtual-reality and mixed-reality simulations for nursing,
PA and PT students and increasing innovation opportunities within continuing education courses for health care professionals. But above all, it aims to produce more well-rounded health care professionals.

“Including mixed reality in the experiences offered through the Interprofessional Simulation Center expands our commitment to educating future health professionals and leaders who are prepared to address health care inequities and optimize patient care outcomes,” says Stephen Folger, program director for the Department of Physical Therapy Education.