The session is entitled “Lessons in Multimodal Interaction Design”, and particularly, the topic I’m going to cover is “The Coexistence of IVRs and Small Screens”. If you’re attending SpeechTEK, I would love to have you join us tomorrow, August 3rd, at session D203 from 1:45 pm – 2:30 pm.

See you there!

Aside from the sad news about average success rates being around 59%, it was interesting to me to see how most of the Mobile Problems outlined in the report can be actually seen as opportunities to seriously consider the use of Speech Recognition.

I know most companies suggest Speech Recognition as the killer app for mobile devices, but I would argue that it should be seen instead as the ideal complementary mode of interaction when navigating the internet and retrieving information on mobile devices, not as the silver bullet that would solve all mobility hurdles.

For example, thinking about speech in the context of those problems raised in the report:

• Small screens: Yes, small size is a natural result of being portable. Yet, having a limited number of options at any given time and relying on short-term memory are the bread and butter of most Speech Recognition Systems. Therefore, adding an audible element and allowing users to express themselves in more natural ways helps compensate those visual limitations. Furthermore, multislot interactions and natural language understanding help alleviate the challenge of multiple windows and advanced behaviors present in purely visual interactions.
• Awkward input (especially for typing): Once again, Speech Recognition shines here since it’s the facto way of interaction amongst humans. Words can easily trump visual counterparts such as menus, buttons, and links not only because of how natural interactions are but also because it avoids the inherent limitations of tiny keypads, trackballs and mini-keyboards.
• Download delays: Even though Speech cannot solve the problem of being able to download screens faster, it can help in those instances where information can be delivered in an audible form since users can continue to interact with the system and move along their intended goal since prompts and logic can be embedded in a device without requiring network connectivity or optimized and compressed for faster delivery.

Therefore, I’ve never been a big fan of these types of services, except for those situations where I know we’re adding value to the conversation. Those situations where we’re providing a benefit to consumers, particularly in win-win scenarios where both parties benefit from the interaction.

One product/service I recently found out about that does exactly that is GlowCaps Connect. GlowCaps are electronic pill caps that use some very clever means to ensure patients take their medicine at the times and frequency that they should.

So picture this. If you know someone that needs to manage a chronic disease like diabetes or depression, daily medications are essential for their well being. What this device does is that every day, at the prescribed time, the GlowCap uses a myriad of modalities to remind users and attract their attention. For example, it may flash a visual reminder which is followed by sound if the bottle is not opened within the first hour. If the patient still doesn’t open the bottle, then the cap triggers a phone call to remind them and can even send weekly updates to friends and family as well as send reports to the patient’s doctor with a monthly summary of the bottle’s activity.

So, to summarize, better prescription handling which can be rewarded with coupons and incentives, better healthcare management with the doctor, and an opportunity for pharmacies to handle automatic refills. Those are the types of calls I wouldn’t mind at dinner time.

So I was very excited to read about a new smart phone app which seems to be tackling this problem in a very clever way by integrating the best aspects of disparate technologies such as Post-It notes, email, calendars, and voice for free! (with a Pro option available too)

It is called reQall and the way it works is that you call a free number to add “items” via your voice or text such as notes, appointments and memos.  If using your voice, they use speech recognition software to transcribe your message into text so that based on your situation (time, location, etc.), the system can remind you of those items via email, SMS, IM or even a “phone shake”.

A very nice feature is that you can also share your account with other family members and friends, so they can enter reminders for you.  Mmm, I wonder why wives love this feature so much…

Then of course, comes the part of the search itself.  Some services for example, provide you with a list of links to Web pages (such as Google and Yahoo). Others, like ChaCha, uses humans to find the answers for you and then send you the response via text (and yes, you can become a “guide” for them). While some others attempt to integrate other features and capabilities of the devices such as the use of GPS and maps, or trigger subsequent reactions on other services such as changing your status in Facebook or Twitter (as is the case with Vlingo).

Now, if someone could simply find a way to use voice to find out where I left my phone, or my remote, or the car keys…

We’ve talked in the past about the use of speech recognition in the realm of note taking, where tools such as Jott allow you to obtain a text version of a voice message, making it easier to document and search for information.

Well, Microsoft just recently unveiled a new application of speech recognition, but this time with a twist. Microsoft Recite (available as a preview which can be downloaded) allows anyone using a Windows Mobile phone to record a voice message or “remembrance”, store it, and then retrieve it later using speech pattern recognition.

The obvious advantage of pattern recognition compared to other types of speech searches is that the message itself doesn’t have to be decoded, transcribed or converted.  It simply uses a “search” sample as a pattern to match one or more of the words against existing “remembrances”.

Even though initial test have received possitive feedback, I’m hoping they’ll expand the tool to include other devices and languages (it currently only works with US English).

Some of those suggestions included adding speech-to-text capabilities to the Kindle 2.0, making it indeed an e-book “reader”.  I think this is a magnificent idea not only because it not only addresses how devices should evolve to support people with disabilities but also gives control back to the users on how to best interact with the information.

One company worth mentioning is JOYity which was recently covered by TechCrunch. They are leveraging the GPS capabilities of the phone, allowing users to engage in location-based games such as YouCatch, Roads of San Francisco and City Race Munich).

The most engaging by far is YouCatch which is an enhanced version of Manhunt. The concept is pretty simple: you and a handful of friends sign up to play the game, and then each one is randomly assigned a target, making everyone both a hunter and a target.

I hope they add voice features soon, which could allow you to play the game in a less obvious way (running around watching a phone screen kinda give you away) and maybe even team up with others for the hunt.

Here’s a quick review of the game and the interface:

1. The Menu Button – This feature provides context-relevant options based on the current task.  David compares it to the functionality of a mouse right-button that offers commands like Hold, Mute and Speaker when you’re on a call.  It also offers next-step related commands such as Archive and Delete once you’ve read an email.  This is a great strategy I always like to implement, particularly on Voice User Interfaces where callers can only be presented with a limited set of choices, and there’s a clear set of task-related options that callers would be looking for without having to ‘go back’ to a so-called Main Menu.  In my mind, this should be renamed as the “Common-Sense Button”
2. Two different programs for e-mail – Ouch, this one really hurts.  Granted Gmail has a different mental model and framework than other e-mail programs, I think this one shows a lack of understanding of what users look for: simplicity and efficiency We know complexity exists everywhere, but that complexity should be hidden, whenever possible, from the UI and the user interaction.  And to add insult to injury, it seems that replying to an email in the non-Gmail program puts your cursor in the To box…  I’m just glad they have an open architecture that allows anyone to improve these interfaces
3. (Useless?) Tilt sensor – This has to be the weirdest one of them all.  If the phone contains a sensor similar to the one powering the iPhone, why did they not hook it to the screen?  The fact that someone is turning the phone 90 degrees should be enough indication of intent, so why put users through the extra step of making a menu selection or pressing a key?  This one feels like those menu prompts that first ask you to press 1 for “Arrivals or Departures Information” – which gives intent information, albeit not in an ideal way – followed up by an absurd follow-up menu asking you to “press 1 for Arrivals or 2 for Departures”.

As is typical with these types of breakthroughs”, the G1 attempts to offer those things the iPhone was lacking – full keyboard, Bluetooth, etc. – at the price of making the UI more complex, and the design definitively less slick than the original (more buttons, bulkier, etc.).

On the other hand, the huge advantage of a device and platform as open as this one is that you can choose whatever carrier you want, developers are free to create any sort of application for the device without much censorship, and users are free to personalize their phones in any way they want. And if there’s anything we’ve learned from other open initiatives (Linux, Apache, Firefox, etc.), users are the ones who’ll win the most. This is definitively just the tip of the iceberg…

As for what that means for UI designers, well, I think more and more users are going to expect (and demand) multimodality (even if they don’t refer to it in those terms). They’ll be able to choose the interaction mode that’s more convenient to them (speech, keypad, pen, gesture, etc.) and switch between them, they’ll expect preferences to remain active no matter the interaction mode they choose (for example, notification preferences set up on the website should carry over to all other contact points), and if they aren’t happy, they have all the tools they need to make you the next Consumerist or Saturday Night Live star.

So yeah, bring it on!