This page demonstrates how to use our R code and apply our optimization methods (Tsandilas and Dragicevic 2022) to the gesture elicitation study of Weigel, Mehta, and Steimle (2014).

The study

Weigel, Mehta, and Steimle (2014) conducted a gesture elicitation study where 22 participants proposed on-skin gestures for 40 referents. We focus here on a subset of 33 referents that represent standard commands and variations, and disregard ones that represent emotions. The authors identified (i) the on-body location of gestures (e.g., upper arm, forearm, palm, and fingers) and (ii) their input modality (e.g., “slide” “tap”, and “twist”). The participants produced a total of 18 unique on-body locations, 56 unique input modalities, and 120 unique combinations of locations and modalities for these 33 referents. Our elicited sign set is the set of those 120 unique combinations (our signs).

We use the signs used by Tsandilas (2018) for this study, which were based on the textual tags provided by the authors in a spreadsheet. These signs do not coincide with the original signs considered by the investigators in their original analysis, as the latter cannot be fully reproduced.

The optimization problem

Our goal is to find an optimal set of mappings between signs (combinations of gesture locations and modalities) and referents. We denote this optimal sign mapping set as \(\pi_{opt}\). Now, to define the optimization problem, we require:

  1. An objective function. We aim to maximize guessability (Wobbrock et al. 2005), where guessability \(G(\pi)\) can be defined as the probability that the sign of a gesture performed by a random user for a random referent belongs to the sign mapping set \(\pi\).

  2. A set of design constraints. We will examine two alternative sets of constraints, where each requires a different solution. First, we will require each referent to be mapped to a single and unique sign. Second, we will relax this constraint by allowing larger groups of signs (in particular, the same modality appearing in related parts of the body) to be mapped to the same referent.

Ideally, we should find a sign mapping set that is optimal for the full population of users. In practice, however, we can only provide an estimate \(\widehat\pi_{opt}\) of the optimal mapping set based on a sample, i.e., the participants of the study. We will examine how to perform cross-validation to evaluate this uncertainty.

Read the dataset

We first read the data, split into two separate csv files: (i) for body locations and (ii) for input modalities.

# Reading input modalities
data = read.csv("datasets/weigel2014-modalities.csv", stringsAsFactors=F)
nparticipants <- ncol(data)-1
nreferents <- nrow(data)
# Remove the "Referent" column to analyze the data
modalities <- data[2:ncol(data)]
row.names(modalities) <- data$Referent

# Reading body locations
data = read.csv("datasets/weigel2014-locations.csv", stringsAsFactors=F)
# Remove the "Referent" column to analyze the data
locations <- data[2:ncol(data)]
row.names(locations) <- data$Referent

# Also build a table with location and modalities combined.
locations_modalities = data.frame(locations)
for (r in 1:nrow(locations_modalities)) {
  locations_modalities[r,] = paste0(locations_modalities[r,], "#", modalities[r,])
}
    
# We focos on the 33 first referents that represent standard commands and variations, 
# We disregard ones that represent emotions
locations_modalities <- locations_modalities[1:33,]
modalities <- modalities[1:33,]
locations <- locations[1:33,]

The resulting locations_modalities data frame is as follows (scroll to see details):

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 P18 P19 P20 P21 P22
Rotate handback#slide handback#slide handback#slide elbow#twist palm#slide palm#slide palm#slide palm#slide handback#twist palm#slide palm#slide handback#shear forearm#slide forearm#slide forearm#twist forearm#slide forearm#slide forearm#twist forearm#slide forearm#twist forearm#slide forearm#twist
Zoom-in handback#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide palm#slide forearm#slide palm#slide handback#push-press palm#slide handback#pull forearm#slide forearm#slide forearm#slide handback#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide
Zoom-out handback#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide palm#slide forearm#slide palm#slide fingers#push-press palm#slide handback#squeeze forearm#slide forearm#slide forearm#slide handback#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide
Help palm#slide handback#tap upperarm#grab+squeeze upperarm#grab+squeeze elbow#pull+twist fingers#slide palm#tap upperarm#tap fingers#grab handback#slide handback#push-press palm#tap forearm#slide handback#push-press handback#slide handback#tap shoulder#tap forearm#pull palm#tap upperarm#grab+squeeze forearm#scratch forearm#push+slide
Open forearm#tap forearm#slide palm#push-press forearm#grab+squeeze forearm#slide handback#slide palm#tap upperarm#slide palm#tap forearm#slide handback#slide-swipe palm#slide forearm#tap forearm#grab+squeeze forearm#tap handback#touch handback#tap upperarm+forearm#slide forearm#slide forearm#slide palm#tap forearm#tap
Switch task forearm#slide forearm#slide fingers#grab+push-press forearm#slide handback#twist forearm#slide palm#slide-swipe elbow#poke palm#slide forearm#shear fingers#slide fingers#slide forearm#slide handback#slide handback#slide-swipe forearm#slide forearm#twist forearm#slide forearm#tap forearm#touch+slide forearm#slide forearm#slide
Selection-Confirm forearm#slide forearm#grab+squeeze fingers#tap wrist#grab handback#slide handback#push palm#tap forearm#slide palm#push forearm#slide fingers#touch fingers#tap forearm#tap forearm#tap forearm#tap forearm#pull forearm#slide forearm#slide palm#slide forearm#push palm#tap forearm#tap
Selection-Numbered forearm#slide forearm#grab+squeeze fingers#tap wrist#grab fingers#grab handback#push fingers#tap forearm#slide palm#push forearm#slide fingers#tap fingers#tap forearm#tap forearm#slide fingers#tap handback#tap forearm#slide fingers#tap fingers#tap forearm#push palm#tap forearm#slide
Maximize forearm#slide forearm#tap handback#tap palm#slide handback#slide forearm#slide+tap palm#slide palm#slide palm#push handback#slide fingers#tap palm#slide forearm#tap wrist#slide+pressure handback#tap forearm#slide forearm#slide forearm#tap forearm#slide forearm#touch shoulder#tap-touch forearm#slide+tap
Enlarge forearm#slide forearm#tap handback#tap palm#slide handback#slide forearm#slide palm#slide palm#slide palm#slide handback#slide palm#tap palm#slide forearm#slide handback#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide
Shrink forearm#slide forearm#tap handback#tap palm#slide handback#slide forearm#slide palm#slide palm#slide palm#slide handback#slide palm#slide palm#slide forearm#slide handback#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide
Minimize forearm#slide forearm#tap handback#tap palm#slide handback#slide forearm#slide+tap palm#slide palm#slide palm#slide handback#slide palm+fingers#slide palm#slide forearm#tap wrist#slide+pressure handback#tap forearm#slide forearm#slide forearm#tap forearm#slide forearm#touch fingers#tap-touch forearm#slide+tap
Close-hide palm#slide handback#slide forearm#tap forearm#grab forearm#slide palm#slide-swipe palm#slide upperarm#slide palm#slide forearm#slide handback#slide-swipe handback#slide-swipe forearm#slide forearm#slide forearm#slide-swipe handback#touch forearm#pull forearm#slide-rub forearm#slide handback#slide palm#slide fingers#push+move
Close palm#slide palm#slide forearm#tap forearm#grab forearm#slide palm#slide-swipe palm#slide-swipe upperarm#slide palm#push forearm#slide handback#slide-swipe handback#tap forearm#slide forearm#slide handback#slide-swipe handback#tap forearm#twist forearm#push forearm#slide handback#slide palm#slide fingers#push+move
Force-close palm#push-press handback#tap forearm#push-press forearm#grab forearm#slide palm#tap+slide-swipe palm#slide-swipe+push upperarm#slide palm#push forearm#grab+twist fingers+handback#grab+squeeze handback#push-press forearm#slide forearm#push forearm#slide-rub handback#tap forearm#slap forearm#push forearm#slide handback#slide+push-press handback#slide fingers#push+move
Delete-Temp palm#twist handback#slide forearm#slide fingers#scratch forearm#slide forearm#slide palm#slide-swipe handback#slide-swipe handback#twist forearm#slide handback#scratch handback+fingers#slide-swipe forearm#push-press forearm#slide forearm#slide-swipe forearm#pull forearm#tap forearm#slide-rub forearm#push+slide forearm#slide palm#slide-swipe forearm#scratch
Delete-Perm palm#slide palm#slide forearm#slide fingers#scratch forearm#slide forearm#slide palm#slide-swipe handback#pull palm#twist forearm#slide handback#scratch+twist handback#slide-swipe forearm#push-press forearm#slide-rub forearm#slide-swipe forearm#slide forearm#pull forearm#push-press forearm#pinch-squeeze forearm#slide handback#slide-swipe forearm#scratch
Move-less palm#slide forearm#slide forearm#slide upperarm#slide handback#slide forearm#touch palm#slide-flip handback#slide palm#slide forearm#slide forearm#slide upperarm#slide forearm#slide handback#tap forearm#slide-swipe forearm#slide forearm#slide forearm#tap forearm#slide forearm#slide forearm#slide forearm#grab+slide
Move-more palm#slide forearm#slide forearm#slide upperarm#slide handback#slide forearm#slide palm#slide-flip handback#slide palm#slide forearm#slide forearm#slide-swipe forearm#slide forearm#slide handback#push-press+slide forearm#slide-swipe forearm#push-press+slide forearm#slide forearm#tap forearm#slide forearm#tap forearm#slide forearm#grab+slide
Previous any#slide forearm#twist wrist#squeeze forearm#scratch palm#push-press forearm#slide palm#slide handback#tap palm#slide forearm#slide fingers#slide handback#slide forearm#slide wrist#push-press palm#tap forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide
Next any#slide forearm#twist wrist#squeeze forearm#scratch palm#push-press forearm#slide palm#slide handback#tap palm#slide forearm#slide fingers#slide handback#slide forearm#slide wrist#push-press palm#tap forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide forearm#slide
Undo forearm+elbow#slide forearm#scratch forearm#slide elbow#slide forearm#slide handback#tap palm#slide upperarm#shear palm#slide-swipe forearm#slide wrist#tap handback#twist forearm#slap forearm#slide handback#slide-swipe forearm#slide palm#slide forearm#twist palm#slide-swipe forearm#twist forearm#slide handback#tap
Redo forearm+elbow#slide forearm#scratch forearm#slide elbow#grab forearm#slide handback#tap palm#tap upperarm#shear palm#slide-swipe forearm#slide palm+fingers#slide-swipe handback#twist forearm#slap forearm#slide handback#slide-swipe forearm#slide palm#clap forearm#twist palm#slide-swipe forearm#twist forearm#slide handback#slide
Shuffle fingers#slide upperarm#slide-tickle forearm#shear palm#slide handback#slide forearm#twist palm#slide forearm#shear handback#slide forearm#slide palm#push-press handback#slide forearm#squeeze forearm#slide forearm#slide-rub forearm#scratch forearm#grab+rotate forearm#squeeze+shake forearm#shear handback#slide forearm#slide-rub fingers#tap
Mute forearm+upperarm#tap forearm#tap forearm#tap forearm#pull fingers#tap fingers#push-press palm#slide-swipe forearm#tap forearm#push handback#slide handback#tap handback#slide wrist#twist handback#twist forearm#grab-cover forearm#tap forearm#slap forearm#push wrist#slide-swipe forearm#push upperarm-joint#tap handback#shear
IncrVolume forearm+upperarm#slide forearm#slide forearm#slide forearm#slide fingers#slide fingers#push-press palm#tap forearm#slide palm#slide handback#slide forearm#slide handback#slide wrist#pull wrist#tap forearm#slide forearm#slide forearm#slide forearm#slide wrist#slide forearm#slide forearm#slide handback#shear
DecrVolume forearm+upperarm#slide forearm#slide forearm#slide forearm#slide fingers#slide fingers#push-press palm#tap forearm#slide palm#slide handback#slide forearm#slide handback#slide wrist#pull wrist#tap forearm#slide forearm#slide forearm#slide forearm#slide wrist#slide forearm#slide upperarm#slide handback#shear
AcceptCall palm#slide handback#slide wrist#grab handback#slide palm#slide fingers#push-press palm#tap palm#clap palm#push-press wrist#slide palm#touch fingers#tap forearm#grab forearm#tap forearm#slide-swipe handback#tap palm#slide-swipe forearm#slide wrist#slide-swipe fingers#slide palm#tap fingers#touch+move
RejectCall palm#slide handback#slide forearm#push-press handback#slide palm#twist fingers#slide palm#slide-swipe handback#slide handback#push-press wrist#slide forearm#slide-swipe fingers#push-press forearm#grab forearm#slide-rub forearm#slide-swipe handback#twist handback#slide-swipe forearm#push-press wrist#slide-swipe fingers#slide handback#tap fingers#touch+move
CallSoon forearm#slide handback#tap forearm#push-press handback#slide palm#twist+push-press palm#slide palm#slide-swipe fingers#slide-rub elbow#twist wrist#slide handback#slide-swipe forearm#grab forearm#grab+slide forearm#slide-rub forearm#slide-swipe+push-press handback#slide-swipe fingers#slide-swipe forearm#push+slide-rub wrist#slide-swipe+tap fingers#slide+tap handback#tap+slide fingers#touch+move
CallLater forearm#slide handback#tap forearm#push-press handback#slide palm#twist+push-press palm#slide palm#slide-swipe upperarm#grab+squeeze forearm#slide wrist#slide handback#slide-swipe upperarm#grab forearm#grab+slide forearm#slide-rub forearm#slide-swipe+push-press handback#slide-swipe palm#slide-swipe forearm#push+slide wrist#slide-swipe+tap fingers#slide+push handback#tap+slide palm#push
SeekAttention forearm#poke-tap forearm#slap forearm#grab+squeeze palm#twist handback#tap+twist handback#pull palm#poke-tap handback#slide-stroke fingers#slide forearm#poke-tap palm#clap forearm+upperarm#slide-rub forearm#poke-tap upperarm#grab palm#squeeze upperarm#grab+shake palm#clap forearm#poke-tap forearm#poke-tap forearm#slide-scratch forearm#knock forearm#shake+rub
Emergency forearm#tap palm#scratch fullarm#tap+pinch elbow#pull forearm#slap palm#tap palm#tap forearm#twist+pull elbow#twist upperarm#grab+squeeze palm#clap palm#push-press forearm#slide+tap+slide forearm#tap elbow#tap upperarm#grab+squeeze+shear forearm#scratch forearm#tap fingers#grab+squeeze forearm#grab+squeeze handback+palm#tap forearm#slide-rub

Optimize for one-to-one mappings between signs and referents

We examine the first alternative of design constrains, where each referents is assigned a single and unique sign. We use a sign mapper that is based on the Hungarian algorithm to solve the optimization problem.

source("gelicopt/sign-mappers.R") # Implementation of various sign mappers

mappings <- hungarian_sign_mapper(locations_modalities)

The resulting mappings data frame is the following (scroll to see details):

refname ref sign
Rotate 1 forearm#twist
Zoom-in 2 forearm#slide
Zoom-out 3 fingers#push-press
Help 4 upperarm#grab+squeeze
Open 5 palm#tap
Switch task 6 fingers#slide
Selection-Confirm 7 forearm#tap
Selection-Numbered 8 fingers#tap
Maximize 9 palm#push
Enlarge 10 handback#slide
Shrink 11 palm#slide
Minimize 12 forearm#slide+tap
Close-hide 13 forearm#pull
Close 14 handback#slide-swipe
Force-close 15 handback#tap
Delete-Temp 16 handback#twist
Delete-Perm 17 handback#pull
Move-less 18 upperarm#slide
Move-more 19 forearm#slide-swipe
Previous 20 any#slide
Next 21 wrist#push-press
Undo 22 forearm#slap
Redo 23 upperarm#shear
Shuffle 24 forearm#shear
Mute 25 forearm#push
IncrVolume 26 handback#shear
DecrVolume 27 wrist#tap
AcceptCall 28 palm#clap
RejectCall 29 forearm#push-press
CallSoon 30 elbow#twist
CallLater 31 palm#slide-swipe
SeekAttention 32 forearm#poke-tap
Emergency 33 forearm#scratch


Let us evaluate the guessability of these mappings:

source("gelicopt/guessability.R") # Implementation of guessability functions

guess <- guessability(mappings, locations_modalities)
cat("Guessability =", guess, "\n")
## Guessability = 0.1184573

Unfortunately, this guessability score \(\widehat G(\widehat\pi_{opt}) = 11.8\%\) has been evaluated on the actual sample on which it was previously optimized (trained), so there is a risk of overfitting. This training guessability value generally overestimates the true guessability \(G(\widehat\pi_{opt})\) that we would like to assess and compare to \(G(\pi_{opt})\). To evaluate this overfitting problem, we perform cross-validation using the Leave-One-Out Cross Validation (LOOCV) method:

guess <- guessability.loocv(locations_modalities, hungarian_sign_mapper)
cat("LOOCV Guessability =", guess, "\n")
## LOOCV Guessability = 0.06198347

Both scores are low, and the target guessability \(G(\pi_{opt})\) is expected to be somewhere between the two. So we cannot hope to get any major improvements in guessability, even if we recruit a much larger number of participants. In machine learning practice, this is known as high bias and demonstrates underfitting: the model that we use to map signs to referents does not capture well the essential parameters of the data.

Optimize with relaxed constraints

A possible direction is to relax the way signs are mapped to referents through custom constraints, driven by our design experience.

Step 1. First, we simplify our signs by merging similar or equivalent body locations:

# Each line below, defines pairs of body locations that should be treated as equivalent
equivalent <- t(matrix(
      c("forearm+upperarm", "upperarm+forearm",
      "forearm+upperarm", "fullarm",
      "fingers+handback", "handback+fingers",
      "upperarm", "upperarm-joint"), nrow =2))

# This function replaces the second member of equivalent items by the first
replaceEquivalentSigns <- function(proposals, eq){
    df <- data.frame(lapply(proposals, function(x){gsub(eq[2],eq[1],x)}))
    rownames(df) <- rownames(proposals)

    df
}

### For each row for the matrix, I merge equivalent locations
for(r in 1:nrow(equivalent))
  locations_modalities <- replaceEquivalentSigns(locations_modalities, equivalent[r,])

Step 2. Second, we define legitimate pairs of signs that share the same modality but are executed in different yet related parts of the body. We consider three larger groups of body locations to infer legitimate pairs of signs: (i) upper parts of the arm and the shoulder, (ii) the palm and the fingers, and (iii) the wrist and the back of the hand, with or without fingers.

# Groups of body locations for inferring legitimate pairs of signs (modality must be the same)
group1 <- c("forearm+upperarm", "upperarm", "shoulder")
group2 <- c("fingers", "palm+fingers", "palm") 
group3 <- c("fingers+handback", "handback", "wrist")

To extract all legitimate pairs in our dataset, we need to write some code:

# helper function: extract only the modality from the sign strings
getModality <- function(signs){
    extractModality <- function(sign){
        unlist(strsplit(sign, "#"))[2]
    }

    mapply(extractModality, signs)
}

# helper function: extract only the body location from the sign strings
getLocation <- function(signs){
    extractModality <- function(sign){
        unlist(strsplit(sign, "#"))[1]
    }

    mapply(extractModality, signs)
}

# First, find all unique signs
signs <- unique(c(t(locations_modalities))) 
nsigns <- length(signs)

# Second, find all pairs of signs that have the same modality
pairs <- t(combn(1:nsigns,2)) # These are all possible pairs of signs (represented by their index)
sameModalityWhich <- getModality(signs[pairs[,1]]) == getModality(signs[pairs[,2]])
legitimatePairs <- pairs[sameModalityWhich,]

# Third, find the pairs of signs that have the same modality 
# and their body location is in group1, group2, or group3
legitimatePairs <- rbind(
  legitimatePairs[getLocation(signs[legitimatePairs[,1]]) %in% group1 
                  & getLocation(signs[legitimatePairs[,2]]) %in% group1,],
  legitimatePairs[getLocation(signs[legitimatePairs[,1]]) %in% group2 
                  & getLocation(signs[legitimatePairs[,2]]) %in% group2,],
  legitimatePairs[getLocation(signs[legitimatePairs[,1]]) %in% group3 
                  & getLocation(signs[legitimatePairs[,2]]) %in% group3,]
)

# Sort the matrix by the first then the second column
legitimatePairs <- legitimatePairs[order(legitimatePairs[,2],decreasing=FALSE),] 
legitimatePairs <- legitimatePairs[order(legitimatePairs[,1],decreasing=FALSE),] 
cat("Number of legitimate pairs =", nrow(legitimatePairs), "\n")
## Number of legitimate pairs = 21

The legitimatePairs matrix has 21 pairs of signs, where signs are identified by their index (position) in the signs vector. It has the following form (scroll to see details):

index1 index2
1 88
3 15
3 45
4 83
8 70
9 87
10 24
11 69
12 74
15 45
16 35
17 19
17 82
19 82
26 86
27 85
31 76
31 94
39 89
59 111
76 94


Step 3. Third, we specify the maximum number of referents to which a sign is mapped. By default, our optimization solver assumes that each individual sign can be mapped to no more than one referent. For this study, however, we want to capture the fact that the “slide” modality provides a richer set of possibilities. We focus here on binary sliding directions (e.g., upwards vs. downwards), so we allow for up to two referents to share the same “slide” sign.

 # Which signs (their index) have a slide modality?
selectedSigns <- which(getModality(signs) %in% c("slide"))

# Those slides can be mapped to maximum 2 referents
maxRefs <- rep(2, length(selectedSigns)) 

# We combine them into a matrix
maxRefConstraints <- matrix(c(selectedSigns, maxRefs), ncol=2, dimnames=list(NULL, c("index","maxRef")))

The maxRefConstraints now specifies which signs can be mapped to more than one referent and the maximum number of referents for each (maxRef = 2 in our case):

index maxRef
1 2
3 2
6 2
15 2
26 2
29 2
45 2
68 2
71 2
72 2
86 2
88 2


As we mentioned earlier, for all other signs, the maximum number of referents to which they can be mapped is by default 1.

Step 4. We can now solve the optimization problem. We will provide three arguments to our optimization mapper function:

  1. the locations_modalities data frame (from Step 1)

  2. the signs vector (from Step 2). If we omit it, the function will construct it as: signs <- unique(c(t(locations_modalities)))

  3. the legitimatePairs matrix (from Step 2), where indices correspond to sign positions in the signs vector. If we omit it, the default value is NULL, and the mapper will allow multiple signs to be mapped to each referent. If, instead, we assign it a NA value, then exactly one sign will be assigned to each referent (as with the Hungarian algorithm).

  4. the maxRefConstraints matrix (from Step 3), where indices again correspond to sign positions in the signs vector. If we omit it, the maximum number of referents will be one for all signs.

source("gelicopt/custom-constraints.R") # Implementation of optimization with custom constraints

# We record how long the optimization takes. It can take long.
time <- system.time({
  mappings <- custom_constraints_mapper(locations_modalities, signs, legitimatePairs, maxRefConstraints)
})
cat("Time to complete the optimization (in sec): ", time[["elapsed"]], "\n")
## Time to complete the optimization (in sec):  50.371

The time cost of optimization is considerable. It can take even longer if the number of signs becomes larger, and additional constraints are added. The mappings produced by this approach are as follows (scroll to see details):

refname ref sign
Rotate 1 forearm#twist
Zoom-in 2 forearm#slide
Zoom-out 3 forearm#slide
Help 4 upperarm#grab+squeeze
Open 5 palm#tap
Switch task 6 fingers#slide
Selection-Confirm 7 forearm#tap
Selection-Numbered 8 fingers#tap
Maximize 9 forearm#slide+tap
Enlarge 10 handback#slide
Shrink 11 palm#slide
Minimize 12 palm#slide
Minimize 12 palm+fingers#slide
Close-hide 13 forearm#pull
Close 14 palm#push
Force-close 15 handback#push-press
Delete-Temp 16 handback#twist
Delete-Perm 17 forearm#push-press
Move-less 18 upperarm#slide
Move-more 19 forearm#slide-swipe
Previous 20 handback#squeeze
Previous 20 wrist#squeeze
Next 21 forearm#scratch
Undo 22 handback#tap
Undo 22 wrist#tap
Redo 23 palm#slide-swipe
Redo 23 palm+fingers#slide-swipe
Shuffle 24 forearm#shear
Mute 25 forearm#push
IncrVolume 26 handback#shear
DecrVolume 27 upperarm#slide
DecrVolume 27 forearm+upperarm#slide
AcceptCall 28 fingers#push-press
AcceptCall 28 palm#push-press
RejectCall 29 handback#slide
RejectCall 29 wrist#slide
CallSoon 30 handback#slide-swipe
CallLater 31 forearm#push+slide
SeekAttention 32 forearm#poke-tap
Emergency 33 palm#scratch


If we compare these new mappings with the mappings produced by the Hungarian algorithm (see our earlier analysis), we observe that they are more consistent. For example, symmetrical signs like “Zoom-in” and “Zoom-out” are both assigned to the “forearm#slide” sign; “Undo” is a tap on the backhand or the wrist, and “Redo” is a slide or swipe on the palm and fingers. Such combinations make more sense from an interaction design perspective.

Step 5: Finally, we evaluate guessability.

guess <- guessability(mappings, locations_modalities)
cat("Guessability =", guess, "\n")
## Guessability = 0.15427

Our observed (training) guessability has increased. This increase, however, may be largely due to overfitting, since mapping constraints are now more flexible. We assess the effect of overfitting with cross-validation. Cross-validation is now very computational expensive, because optimization is repeated 22 times (which is the number of participants). To accelerate it, we can specify a number of CPU cores to use in parallel (we use four cores below):

time <- system.time({
  guess.loocv <- # The implementation of this function is within the custom-constraints.R file
    guessability.loocv(locations_modalities, signs, legitimatePairs, maxRefConstraints, CoresNum = 4)
})
cat("Time to complete the cross-validation (in sec): ", time[["elapsed"]], "\n")
## Time to complete the cross-validation (in sec):  440.545
cat("LOOCV Guessability =", guess.loocv, "\n")
## LOOCV Guessability = 0.09090909

The cross-validation score has also increased but is still low. If we look for mappings with a higher guessability, we definitely need to increase our sample of participants. But even with more participants, any improvements are expected to remain below the training guessability score of \(15.4\%\) that we calculated above.

The agreement analysis of Tsandilas (2018) for this study shows that there was no consensus among participants on the body location of on-skin gestures. In addition, consensus on the input modality was limited and largely due to the dominant use of slide gestures (“slide” alone was responsible for \(80\%\) of all agreements on the input modality of gestures). Therefore, creating a guessable set of mappings between the signs of on-skin gestures and the above 33 referents may not be feasible.

References

Tsandilas, Theophanis. 2018. “Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation.” ACM Transactions on Computer-Human Interaction. https://doi.org/10.1145/3182168.

Tsandilas, Theophanis, and Pierre Dragicevic. 2022. “Gesture Elicitation as a Computational Optimization Problem.” In CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3501942.

Weigel, Martin, Vikram Mehta, and Jürgen Steimle. 2014. “More Than Touch: Understanding How People Use Skin as an Input Surface for Mobile Computing.” In Proceedings of the Sigchi Conference on Human Factors in Computing Systems, 179–88. CHI ’14. New York, NY, USA: ACM. https://doi.org/10.1145/2556288.2557239.

Wobbrock, Jacob O., Htet Htet Aung, Brandon Rothrock, and Brad A. Myers. 2005. “Maximizing the Guessability of Symbolic Input.” In CHI ’05 Extended Abstracts on Human Factors in Computing Systems, 1869–72. CHI Ea ’05. New York, NY, USA: ACM. https://doi.org/10.1145/1056808.1057043.