理解用户意图：超越AB测试--Chad Davis.pdf

chad davis, qcon 2014 shanghai1 understanding user intent beyond a / b testing chad davis, qcon 2014 shanghai2 technology to product sereal: faster object serialization css:compressor : faster css compression brick: scalable on-site search hadoop: loyalty metrics and deals redis: server-side session storage perl: b ! chad davis, qcon 2014 shanghai3 user research studies chad davis, qcon 2014 shanghai4 iterative development via a / b tests chad davis, qcon 2014 shanghai5 iterative development via a / b tests product to technology chad davis, qcon 2014 shanghai6 brick: the search engine demand for flexible / forgiving search customizing elasticsearch, closer to lucene prototype a small server wrapping lucene chad davis, qcon 2014 shanghai7 a / a experiments: same experience customers in base are performing n searches customers in variant are searching 20% more what is happening? chad davis, qcon 2014 shanghai8 simulations and false positive rate auto-complete search a / a / a experiment ( three identical cases ) variantcancellations (conf.) v0 ( control group )= 0 % ( by definition ) v1 ( no change vs v0 )- 6 % ( not significant ) v2 ( no change vs v0 )+ 97 % ( significant ) chad davis, qcon 2014 shanghai9 simulations and false positive rate simulating 1000 a / a experiments : at some point during the experimentation : 771 experiments out of 1.000 reached 90% significance only at the end of the experimentation : 100 experiments out of 1.000 were significant at 90% /is-your-ab-testing-effort-just-chasing- statistical-ghosts.html chad davis, qcon 2014 shanghai10 interacting experiments feature x user feedback y without feature xwith feature x without user feedback y without feature x without user feedback y with feature x without user feedback y with user feedback y without feature x with user feedback y with feature x with user feedback y chad davis, qcon 2014 shanghai11 beyond conversion: search metrics click-through-rate (ctr) disambiguations searches per session (cf. web search) chad davis, qcon 2014 shanghai12 web marketing: search is intent chad davis, qcon 2014 shanghai13 context and ambiguity chad davis, qcon 2014 shanghai14 context and ambiguity chad davis, qcon 2014 shanghai15 80 / 20 rule high demand for : london, new york, paris over-represented results unpopular places : drown in the noise chad davis, qcon 2014 shanghai16 location relevance? chad davis, qcon 2014 shanghai17 dd y mn t nrmlize tht? mu(e)nchen upranks mnchen, germany caen fails to uprank caen, france chad davis, qcon 2014 shanghai18 how much feedback do you need? chad davis, qcon 2014 shanghai19 dialogue on spell check results chad davis, qcon 2014 shanghai20 “one king-size bed for two persons in honeymoon, no smoking, quiet for sleeping, free-wifi, need stove and cooking utensils. guranteed seaview in high floor room, thanks a lot.” we asked for : someone replied : chad davis, qcon 2014 shanghai21 what was this person looking for? b ba bar bare barel barelc barelce barelcel barelcelo barelcelon barelcelona spell check click on “barcelona“ click for district: barcelona city centre, spain s se sev sevi sevia disambiguation : single hotel result (redirect) sevi (backspace) sevil sevila disambiguation : single hotel result (redirect) sevil (backspace) click for city: seville, andaluca, spain booked a hotel in carmona, spain (30 km from seville) chad davis, qcon 2014 shanghai22 knowing what you dont know look at the search logs. how many people are searching for cities that are not available on our site? chad davis, qcon 2014 shanghai23 knowing what you dont know look at the search logs. how many people are searching for cities that are not available on our site? is “ 丹 asdghy3yh23” a new city ? a new 5-star villa ? a cat on your keyboard ? chad davis, qcon 2014 shanghai24 developer to product chad davis, qcon 2014 shanghai25 do your developers surprise the po hey, did you know we can . do spell check for search in the native apps? really? lets do it ! expect developers to develop ideas technical constraints