Apologies and multimodal interfaces

Firstly, apologies for a delay in the next post. It has been a rather busy time in the Hunter Valley! 

Exhibit A - "rather busy" in the Hunter Valley

Today I'd like to talk about one subject which makes for interesting discussion, yet remains a fairly dry topic: learning to type.

 Now, hear me out...

Late last term I was in a staff room where a teacher lamented that a particular boy had terrible handwriting. Two things immediately come to mind - would said teacher be equally critical of a student's ability to use other forms of communication and, secondly, I was and remain that boy with terrible handwriting.

Yep, that's right ... my handwriting has the style and form of a GP with a broken thumb. It is barely intelligible to me, let alone others. This is as good as it gets:

Exhibit B - my terrible handwriting

 However, I can proudly say that I am pretty good at gesture interfaces. I am also fairly handy in voice commands and voice-based communication (shutting me up is the trick). And with a fairly solid ego, I think I do well in person and and over video.

And though not stylish, I can solidly function at 65 words per minute (wpm) and 95%+ accuracy when typing.

Here's the kicker though... I almost never use a real ink pen at all.

Alright, alright ... so I hear you dragging out the Mueller and Oppenheimer research saying that typing is bad. First of all, that's not what it says. It states that students who use laptops (and specifically the keyboards) to record notes tended to be more likely to record verbatim, and had reduced factual and conceptual recall.

Sharon Oviatt, however, paints a much richer picture. She draws a landscape that identifies that different modes of communication suit different activities and, there is specific benefits in a multimodal interface.

It makes sense doesn't it?

If I was undertaking a design project, typing wouldn't suit. The process of ideation requires one to use non-linguistic forms, such as drawings, diagrams, sketch and the like.

If I was undertaking a creative writing project, typing would suit. The process of linguistic construction is well suited to the keyboard interface for speed, accuracy and the secondary but more important benefits of creating modifiable product.

Further, Oviatt cites what is called the "Performance-Preference Paradox", which whereby there is mismatch between the interface people say they prefer (e.g., keyboard) and what best supports their performance (e.g., pen).

Sharon's work is well suited to start and form a discussion of BYO and 1:1 device programs and learning space design. That, however, is a discussion for another day.

I want to get back to the issue of learning to type.

The same teacher who had lamented the handwriting of her student, then proceeded to type an email with the style and form of a drunk heron. Her single index fingers dived one at a time, only to miss repeatedly (acknowledged by the irritated smacking of the backspace key).

See, once upon a time we did explicitly teach typing. But then it disappeared. Meanwhile my schools have free access to LUX Typing, an aged by serviceable product, and there are wonderful newer educational products like Ultimate Typing Tutor which are affordable (a few dollars per year) and powerful tools to improve typing.

That still doesn't address all the other forms of interface. For example, in many cases I hear a negative response to Windows 8 tablet devices. Yet I see dozens of touch based devices with mice attached to them. How many iPads have mice attached to them?! There are other ways to get around...

What about keyboard shortcuts? Windows 8 is arguably the best Windows for keyboard shortcuts ever made - just try smacking the ÿ key and a random letter and you'll find a whole lot of goodness. Meanwhile, if you have ever seen someone use something like Adobe Photoshop using keyboard shortcuts it looks like Clarke's Third Law in action.

Meanwhile gestures are increasingly commonplace for all mobile devices. Samsung's flagship phone includes multiple gesture points. The Kinect has for years used body movements as an input source. And there is no doubt that the next stages of augmented and virtual reality (tomorrow, not the future kids) will use advanced gesture interfaces.

And never forget the pure power of the voice. Civilisation started with the spoken world and is still, in many ways, ruled by it. Voice recognition was the wild frontier a decade ago filled with dorky headsets and US-centric voice algorithms. Today, Cortana and Siri are able to translate and communicate the world for us while the aforementioned Kinect and its Playstation rival, as well as a swatch of televisions, use voice controls.

And to be honest, we've always used different forms to communicate. Writing, in and of itself, is purely a linguistic communication. However, many areas of knowledge are not communicated very well in purely linguistic forms - especially the ever touted STEM subjects. Just as I might lament my skills in writing, I also lament my inability to sketch and draw. For ultimately, ideation is about the best of both linguistic and non-lingusitic forms. Even with a pen, the process of moving up Bloom's taxonomy requires more than just words (see Exhibit C).

Exhibit C - I cannot draw either

The simple fact is that most students use a range of interfaces for learning. Increasingly, students are using multimodal interfaces like smartphones where voice, typing and gesture interact. The stunning world being opened up by augmented and virtual worlds will tip the scales in favour of multimodal interfaces.

And while Oviatt and Cohen might state that “multimodal interfaces have eclipsed keyboard-based interfaces as the dominant computer interface worldwide (2015, p.169). Keyboards remain well suited for “relatively mechanical tasks, like email, text editing, and information search via the Web (Crowne, 2007; Oviatt et al., 2006; Oviatt and Cohen, 2010a)” (Oviatt, 2013, p. 39). That is, keyboards remain well suited to those tasks that each of us does on a regular day.

So I agree with the teacher who laments poor handwriting. But I suggest that poor handwriting is as equally problematic as poor sketching abilities, poor typing abilities, poor understanding of gesture based inputs, poor understanding of keyboard shortcuts, poor mouse function and use, poor understanding of voice commands and a general inability to interact with modern technology. Or, as a better litmus test, who is more employable - the person with fluency and accuracy when writing, the person with fluency and accuracy when typing, or the person with fluency and accuracy when operating in a multimodal environment?

References

Mueller, Pam A.; Oppenheimer, Daniel M.,  “The Pen Is Mightier Than the Keyboard: Advantages of Longhand Over Laptop Note Taking,” Psychological Science, June 2014, Vol. 25, No. 6. doi: 10.1177/0956797614524581.

Oviatt, S. L., & Cohen, P. R. (2015) The Paradigm Shift to Multimodality in Contemporary Computer Interfaces, Morgan and Claypool.

Oviatt, S.L. (2013) The Design of Future of Educational Interfaces, Routledge Press.