Gesture Elicitation as a Computational Optimization Problem
Theophanis Tsandilas
Université Paris-Saclay, CNRS, Inria, LISNPierre Dragicevic
Université de Bordeaux, CNRS, Inria, LaBRI
Fig 1. Three examples of simulated gesture elicitation studies. The top graphs show the evolution of population and sample guessability scores of optimal solutions. The bottom graphs show the evolution of the guessability error.
Abstract
Gesture elicitation studies are commonly used for designing novel gesture-based interfaces. There is a rich methodology literature on metrics and analysis methods that helps researchers understand and characterize data arising from such studies. However, deriving concrete gesture vocabularies from this data, which is often the ultimate goal, remains largely based on heuristics and ad hoc methods. In this paper, we treat the problem of deriving a gesture vocabulary from gesture elicitation data as a computational optimization problem. We show how to formalize it as an optimal assignment problem and discuss how to express objective functions and custom design constraints through integer programs. In addition, we introduce a set of tools for assessing the uncertainty of optimization outcomes due to random sampling, and for supporting researchers’ decisions on when to stop collecting data from a gesture elicitation study. We evaluate our methods on a large number of simulated studies.
Publication
Our methods are described in the following paper:
Theophanis Tsandilas and Pierre Dragicevic. Gesture Elicitation as a Computational Optimization Problem. CHI Conference on Human Factors in Computing Systems (CHI ’22), April 29-May 5, 2022
[doi] [author version] [bibtext]
Code and case studies
We provide R Code that implements our methods. We demonstrate its use with two case studies:
Optimizing the Métamorphe keyboard. This case study demonstrates the full potential of our approach for the most common case, when each referent (system command or action) can be only mapped with a single and unique sign.
Optimizing on-skin gestures. This case study demonstrates how to use our optimization approach to deal with custom design constraints, where multiple related signs can be mapped to the same referent.
Supplementary materials
Supplementary materials (Appendix and R code for reproducing the tables, graphs, and datasets of the paper can be downloaded from here: https://osf.io/5w6y2/
Further readings
Our analysis is complementary to agreement analysis, which assesses participants’ consensus on their gesture assignments. For a in-depth discussion about agreement analysis methods and statistics, please refer to our previous work.