Voice User Interface Design VUI

Very interesting debate was triggered by the recent tests (Part 1 and Part 2) on Google Voice performed by readers of the Gadgetwise blog.

The overall premise was for readers to call the article′s writer phone number and leave a creative voicemail to gauge the effectiveness of Google′s voicemail transcription system.

Even though at first glance this seems to be another “why speech recognition isn’t ready for prime time” type of article (yes, I know the author claimed they wanted to test the boundaries of the technologies, but as many readers pointed out, individual items such as the president′s name aren′t that far fetched and should′ve worked), I think it also brings up some interesting issues often faced when testing speech recognition systems:

1) What are we testing? - This one very often depends on who you′re talking to, particularly on the business side of things. Some team members look care about containment rates (how many individuals stay in a system without having to talk to an agent), some others care about transfer rates (the increase or decrease in the volume of calls going into the call center), while some others care about customer experience (how long does it take for someone to solve accomplish their goal), and even a few (sorry to say upper management included) call systems to see how well they work when given odd statements or commands, or even worse, how close the system matches their particular expectations (without taking into consideration how the system was designed in the first place). So for me, this is one of the most important aspects of any system, which should be captured as part of the requirements phase – knowing what project owners want to test allows you to stir your design in the right direction (and push back when necessary as early as possible). For example, in the case of these Google Voice tests, there were some very interesting comments from readers because some felt they were testing the accuracy of the transcriptions, while others thought the test should only involve how well is the overall intent being captured, while some others (sadly) though they were testing how does speech recognition work nowadays.

2) How are we testing it? - This one depends a lot on what the answer to #1 might be. For example, in the case of Google Voice, I felt the test would′ve been much more valid if readers were asked to forward samples of their own voicemails into the writer′s voicemail (meaning real world examples) instead of having them come up with messages that seemed to have turned into a challenge to see who came up with the one that broke the system the most. Going back to some of the things business owners normally want to test, some of the methods in which we might need to test those items might vary significantly: for example, to test containment or transfer rates, one should not only look at raw numbers but at reasons behind those numbers – it′s very different if the numbers are driven by users exceeding a failure threshold than if they are due to users pressing 0 or if they are truly due to business requirements whose proper behavior is to retain/transfer the user.

3) What do these results mean? Particularly when dealing with numbers and percentages, the interpretation of results if very often tricky. For example, would you modify a menu if 50% of your users end up making the wrong selection? (I′m sure your gut reaction is “yes”, “of course”)… but what if that number is based on 2 out of 4 users that someone listened to during a morning′s test? Similarly, we sometimes run into situations where decisions are based solely on someone′s like or dislike (often C-level individuals) about how the system is performing (subjective analysis) without any consideration for the reasons behind the choice, the data of a much larger sample, or the fact that the system might still be on a pre-pilot phase that will eventually get tuned. I felt this was probably one of the main things lacking from the article.

The examples are definitively interesting (and funny sometimes), but I think it would′ve been worth doing some sort of analysis about the possible reasons behind some of those misrecognitions (line quality, odd pausing, user′s accents, etc.) as well as a more detailed explanation of what the transcription process really is. Some readers might think the results reflect the accuracy of an advanced speech recognition engine when in reality most transcription processes out there in the market involve a hybrid environment where the recognition engine might perform the first pass, and then human beings perform a second pass, reviewing what the machine recognized and/or interpreting those segments the machine might not have been able to recognize in the first place.

Have you tried it yet?

As a follow-up to my previous post, it was interesting to read David Pogue’s review of Google’s First Phone,  particularly in regards to some of the UI Compromises designers had to make on this first iteration of the Android-based phone:

1. The Menu Button – This feature provides context-relevant options based on the current task.  David compares it to the functionality of a mouse right-button that offers commands like Hold, Mute and Speaker when you’re on a call.  It also offers next-step related commands such as Archive and Delete once you’ve read an email.  This is a great strategy I always like to implement, particularly on Voice User Interfaces where callers can only be presented with a limited set of choices, and there’s a clear set of task-related options that callers would be looking for without having to ‘go back’ to a so-called Main Menu.  In my mind, this should be renamed as the “Common-Sense Button”
2. Two different programs for e-mail – Ouch, this one really hurts.  Granted Gmail has a different mental model and framework than other e-mail programs, I think this one shows a lack of understanding of what users look for: simplicity and efficiency We know complexity exists everywhere, but that complexity should be hidden, whenever possible, from the UI and the user interaction.  And to add insult to injury, it seems that replying to an email in the non-Gmail program puts your cursor in the To box…  I’m just glad they have an open architecture that allows anyone to improve these interfaces
3. (Useless?) Tilt sensor – This has to be the weirdest one of them all.  If the phone contains a sensor similar to the one powering the iPhone, why did they not hook it to the screen?  The fact that someone is turning the phone 90 degrees should be enough indication of intent, so why put users through the extra step of making a menu selection or pressing a key?  This one feels like those menu prompts that first ask you to press 1 for “Arrivals or Departures Information” – which gives intent information, albeit not in an ideal way – followed up by an absurd follow-up menu asking you to “press 1 for Arrivals or 2 for Departures”.

Wow, just when you think you’ve heard it all, you get an odd call at home. Is this what evolution has in store for us? I mean, I’ve heard how visionaries imagine a world where machines will interact with each other automatically, without the need of a human intermediary, but I’ve got to admit the call I got yesterday freaked me out a little bit.

What happened was that my phone at home rang, I picked up the phone, said “Hello?”, and was greeted by an automated system (outbound call) that told me:

“We are sorry to disturb you. This message was intended to be received only by an answering machine. Thank you!” <hang-up>

Apparently, I’m not the only one, and from what I read there, it seems some lucky recipients are getting calls very early in the morning, or non-stop throughout the night.
Wasn’t that the whole reason people demanded a do-not-call national registry? But I guess that doesn’t make a difference anymore now that certain companies offering “Low Cost Voice Broadcasting” services mention as part of their offering the fact that they are “Clean against the National Do-Not-Call List”.

The worst part is I didn’t even know who was calling, otherwise I could’ve let my machine know it would need to call them back