VR Locomotion
“Teleportation” is an NSF funded project aimed at investigating the cognitive process surrounding navigating virtual environments. My role in this project was to design a completely online VR user study to collect teleportation data during the COVID-19 pandemic.
Problem Statement
Before the COVID-19 pandemic, the grand majority of VR research was conducted in controlled lab environments. In order to maintain health and safety, our team decided to design a study that could be conducted completely online using the participants’ own VR hardware. My job was to design an online study capable of replicating the results of a similar study previously conducted in a lab environment.
Research Questions
The main goal of this experiment was to see if an unmoderated online study could replicate the results from a full-controlled lab experiment. In order to do this, we needed to learn how to conduct a successful VR study online, despite very little previous research on the subject. Lastly, I was personally interested in learning how we could use adaptive feedback in thee virtual environment to help the participant complete all the research tasks successfully.
- How can a lab-based VR experiment be adapted into an unmoderated, online study?
- Can the online study replicate the results from a lab study?
- How can we replicate “in the moment” feedback, normally provided by a research moderator?
Online Study Considerations
I used existing literature about conducting unmoderated online research from the HCI and UX domains to develop the procedure and application for this online study. Based on this information, we decided to conduct our research on Amazon Mechanical Turk (MTurk) and Prolific. We also decided that all data should be automatically uploaded to a data server by the VR application to prevent any data loss. However, there are some additional challenges when designing an online study for VR. These include ensuring the application is device agnostic, the software is easy to install and launch, and the instructions and controls are easy to learn.
One notable difference between an in-person lab study, and an online VR study is that in the lab, the researcher can teach the participant how to use the controls to perform the task. This allows them to adapt their instruction to the needs of the user and repeat the instructions until an understanding is reached. The researcher can also intervene when the participant is performing the tasks incorrectly. To replicate this process in VR, I designed adaptive instructional feedback that was automatically displayed when the participant performed one of 5 common incorrect actions (as determined by pilot testing.)
Demonstration of triangle completion task
Task & Variables
The task set by the lab study was a triangle completion task which tested the participant’s ability to remember where they have been inside a virtual environment. First the participant was asked to teleport to three locations in the shape of a triangle. Then they were asked to point back towards the location they started from. The amount of error in their response was then measured. The study had three different independent variables, each with two levels: teleportation method, virtual room size, and path length. The experiment had a full factorial, within-subjects design with 6 trials of each treatment.
Data Analysis
Independent samples t-tests were used to compare error magnitude and angular error between the online and lab groups. An analysis of variance was conducted on the online group to determine if the effects found in the original lab study were replicated. Regression analysis was also used to test the effectiveness of the adaptive instructional feedback, with error magnitude and completion time as dependent variables and feedback types and quantity as predictors.
Results
The online study successfully replicated the key findings from the original lab experiment. This confirmed that unmoderated, online VR studies can produce reliable spatial cognition data comparable to controlled lab settings. The adaptive feedback system also proved effective at helping participants complete the task correctly without a live moderator present.
The results were published in IEEE Transactions on Visualization and Computer Graphics (Vol. 28, No. 5, May 2022): J. W. Kelly et al., “Remote research on locomotion interfaces for virtual reality: Replication of a lab-based study on teleporting interfaces.”
Lessons Learned
During the study, we faced several challenges with online data collection. Participants on MTurk often falsified their Head-Mounted Display ownership, while Prolific was far more reliable for finding genuine VR headset owners. Linking web-based Qualtrics survey data to VR application data required assigning each participant a unique ID used across both systems. We also found it difficult to verify whether participants completed the study as designed, since they could experience technical issues that made their data unusable. Adding self-report survey questions about hardware or software problems made data filtering much easier.