TowardsaNaturalUserInterface英.ppt

Towards a Natural User Interface,Talk Outline,The NUI revolution The vision. The ingredients and technologies. 5 challenges to getting NUI started 10 steps to overcome challenges Conclusion,User Interface Evolution,User Interface Evolution Graphical User Interface,GUI Interaction User is the expert and knows What functions exist Where to go What to click (and in what order) What to type,GUI Technology Direct manipulation Mouse / keyboard input Mouse input mapped to Windows, menus, etc. Keyboard input (search) mapped to text index,Enables,User Interface Evolution,NUI “Do What I Mean” UI,Users naturally articulate what they mean, on any device, to any application or web service, and have their intention interpreted and executed accurately. Why NUI? Expressive. Natural. Scalable.,Natural User Interface Motivated by human-human interaction,Interacting with a Librarian,User asks naturally “Find information about Chicago”,NUI Technology,Natural User Interface (NUI),User says / types: “Information about Chicago” “I want to travel to Chicago” “Book my flight to Chicago using Expedia”,Enables,NUI Interaction,NUI: Hear What You Say,User says / types: “Information about Chicago” “I want to travel to Chicago” “Book my flight to Chicago using Expedia”,Enables,NUI Interaction,Hear What You Say,Human language : “Invented” for interaction “Language is a biological adaptation to communicate information One of natures engineering marvels” Steven Pinker “Vision evolved from the need to survive; speech evolved from the need to communicate” Michael Dertouzos. Speech is central but also typing, handwriting, gestures Best form depends on: Habit / skills (typing to PCs) Form factor (speaking to phones) Situation (writing at meetings) Eventually, multimodal,NUI: Know What You Mean,User says / types: “Information about Chicago” “I want to travel to Chicago” “Book my flight to Chicago using Expedia”,Enables,NUI Interaction,Syntax (rules of the humans language) Nouns, verbs, etc. and how they combine “Book about a trip to Chicago” vs. “Book a trip to Chicago” Normalize linguistic variations . Semantics Meaning of the words Book means reserve a ticket; requires from-city, to-city, etc. Context (additional hints) Domain knowledge : No train from Hawaii to Chicago Statistics : Book as a noun Book as a verb “Book Chicago” Personal Preferences : Where you live, your calendar, how you pay Model of time, urgency, presence Dialog (resolving ambiguity & determine intent) “Buy a book or book travel?” “What date would you like to travel?”,Know What You Mean by combining:,NUI: Do What You Want,User says / types: “Information about Chicago” “I want to travel to Chicago” “Book my flight to Chicago using Expedia”,Enables,NUI Interaction,Do What You Want (by brokering & combining),No system can know everything. Broker intent to experts: Take user intent and determine “Who are the possible experts for this intent?” Each expert (web service) registers what it knows Encarta knows about history, geography, Expedia knows about travel (to various places) Amazon knows about books, book reviews Combine all experts answers, Content (like a search) Find history of Chicago. Action (like a service) Buy a ticket to Chicago.,NUI Will Enable,Find the Bill Gates book on future When is the Britney Spears concert? How do I replace my printer cartridge? Buy the Gladiator DVD for less than $15 Send flowers to mom on her birthday Continue working on my annual report,Challenges to NUI,Technology limitations User reluctance Unproven business value High development effort More than UI infrastructure needed,So, how do we make NUI work? 10 evolutionary steps.,Challenges to NUI,Technology limitations User reluctance Unproven business value High development effort More than UI infrastructure needed,1. Change the world, one domain at a time,General understanding is hard. Domain knowledge can: Reduce linguistic ambiguity: proceeds, IRA Reduce semantic ambiguity: I want to check in. Can get benefit even for “very big domains” Use of contextual cues helps further.,“Reasoning Engine”,Domain Constraints Explicit Input Context = implicit input,2. Dont try to solve all the problems at once.,Probably Solvable Now Long-Term Research Problems,Authoring tool that learns from user behavior,Modeling of context / domain knowledge,Basic NL / disambiguation dialog,One model for speech and NL,Handle complex sentences,Natural dialogs,General Q&A over documents,General Context Model,Multi-domain reasoning,Planning, web services composition,Re-usable domain and general language libraries,Integrated search (documents, schema, database),Help and commands (slot filling),Q&A for databases (e.g., stored procedures),3. Be religious about solution, but be pragmatic about technology,Dont fall in love with one technology! Take whatever technology works. If there are several approaches, Take the simplest! Rule of 80-20 could be 99-1 for hi-tech! Combine them if that helps.,4. Use UI to hide technology imperfections.,Push the intelligence-required problems back to the human. Request-and-choose UI Natural (in search). Top N precision, not top 1 precision. Other UI tricks.,Challenges to NUI,Technology limitations User reluctance Unproven business value High development effort More than UI infrastructure needed,5. NUI complements and extends GUI,Its all about discoverability. If it fits in one screen, GUI works great. Always want to combine GUI and NUI. Search & browse is a good example,NUI extends GUI even for small devices,When the screen is there, use it! Example: SALT for multimodal and telephony-only One authoring tool for all devices. Extend presentation layer developers already know (e.g., html, xhtml).,6. NUI should start with users comfort zone,Dont try to add revolutionary UI elements, and make users learn. Instead, find out where users type/speak, and start there. Speaking : telephone (routing). Typing : search engine (finding). Handwriting : annotation. Accommodate the user first; then expand naturally from there. Telephone PDA PC. Search navigation help command.,Challenges to NUI,Technology limitations User reluctance Unproven business value High development effort More than UI infrastructure needed,7. NUI needs early adopters with real economic motivation,Cool demos or checkbox items will not lead to adoption. Find an early adopter with the economic inventive or end-user pain. Speech recognition for call centers, mobile devices, automobiles. NL translation for manuals. Smart search for CRM, self-help, e-commerce. Use early adopters to improve technology and tools. Then look for adjacent markets.,Challenges to NUI,Technology limitations User reluctance Unproven business value High development effort More than UI infrastructure needed,8. NUI Need Unified Authoring Tools,The most important enabler of NUI is great authoring tools. Same concept needed for speech, search, commands, help, or Q&A. Bridge the gap between “problem statement” and “solution statement”: “I want something that pays interest and maybe appreciate a little, but not too risky” problem statement (the way user thinks) Investment Articles Bonds Zero Coupons Tutorial solution statement (the way author thinks) Organize “problems” with user log mining/analysis. Organize “solutions” with re-usable data: Common types (date, time, synonyms) Linguistic objects (synonyms, morphology, linguistic normalization modules) Domain objects (e.g., financial terms) Organized ontology? Need to avoid Cyc problems.,Challenges to NUI,Technology limitations User reluctance Unproven business value High development effort More than UI infrastructure needed,9. NUI Needs Structure.,Unstructured document search Text Text.,Fortunately, structure is happening!,Structure allows for: Slot filling, understanding. Structure is definitely coming! Databases, XML, SOAP, WSDL.