2017人工智能指数报告 AI report

,!,!,AI INDEX, NOVEMBER 2017,! 1,!,!,AI INDEX, NOVEMBER 2017,STEERING COMMITTEEYoav Shoham (chair)Stanford UniversityRaymond PerraultSRI InternationalErik BrynjolfssonMITJack ClarkOpenAIPROJECT MANAGERCalvin LeGassick,! 2,!,!,AI INDEX, NOVEMBER 2017,TABLE OF CONTENTSIntroduction to the AI Index 2017 Annual ReportOverviewVolume of ActivityAcademiaPublished PapersCourse EnrollmentConference AttendanceIndustryAI-Related StartupsAI-Related Startup FundingJob OpeningsRobot ImportsOpen Source SoftwareGitHub Project StatisticsPublic InterestSentiment of Media CoverageTechnical PerformanceVisionObject DetectionVisual Question AnsweringNatural Language UnderstandingParsingMachine TranslationQuestion AnsweringSpeech Recognition,579991114161617182123232525262626272828293031,! 3,!,!,AI INDEX, NOVEMBER 2017,Theorem ProvingSAT SolvingDerivative MeasuresTowards Human-Level Performance?Whats Missing?Expert ForumGet Involved!AcknowledgementsAppendix A: Data Description & Collection Methodology,323334374144687072,! 4,!,!,AI INDEX, NOVEMBER 2017,INTRODUCTION TO THE AI INDEX 2017ANNUAL REPORTArtificial Intelligence has leapt to the forefront of global discourse, garnering increasedattention from practitioners, industry leaders, policymakers, and the general public.The diversity of opinions and debates gathered from news articles this year illustratesjust how broadly AI is being investigated, studied, and applied.However, the field of AI is still evolving rapidly and even experts have a hard timeunderstanding and tracking progress across the field.,! 5,!,!,AI INDEX, NOVEMBER 2017,Without the relevant data for reasoning about the state of AI technology, we areessentially “flying blind” in our conversations and decision-making related to AI.We are essentially “flying blind” in our conversations anddecision-making related to Artificial Intelligence.Created and launched as a project of the One Hundred Year Study on AI at StanfordUniversity (AI100), the AI Index is an open, not-for-profit project to track activity andprogress in AI. It aims to facilitate an informed conversation about AI that is groundedin data. This is the inaugural annual report of the AI Index, and in this report we look atactivity and progress in Artificial Intelligence through a range of perspectives. Weaggregate data that exists freely on the web, contribute original data, and extract newmetrics from combinations of data series.All of the data used to generate this report will be openly available on the AI Indexwebsite at . Providing data, however, is just the beginning. To become trulyuseful, the AI Index needs support from a larger community. Ultimately, this report is acall for participation. You have the ability to provide data, analyze collected data, andmake a wish list of what data you think needs to be tracked. Whether you haveanswers or questions to provide, we hope this report inspires you to reach out to theAI Index and become part of the e f ort to ground the conversation about AI.,! 6,!,!,AI INDEX, NOVEMBER 2017,OVERVIEWThe first half of the report showcases data aggregated by the AI Index team. This isfollowed by a discussion of key areas the report does not address, expert commentaryon the trends displayed in the report, and a call to action to support our data collectione f orts and join the conversation about measuring and communicating progress in AItechnology.Data SectionsThe data in the report is broken into four primary parts: Volume of Activity Technical Performance Derivative Measures Towards Human-Level Performance?The Volume of Activity metrics capture the “how much” aspects of the field, likeattendance at AI conferences and VC investments into startups developing AIsystems. The Technical Performance metrics capture the “how good” aspects; forexample, how well computers can understand images and prove mathematicaltheorems. The methodology used to collect each data set is detailed in the appendix.These first two sets of data confirm what is already well recognized: all graphs are “upand to the right,” reflecting the increased activity in AI e f orts as well as the progressof the technology. In the Derivative Measures section we investigate the relationshipbetween trends. We also introduce an exploratory measure, the AI Vibrancy Index, thatcombines trends across academia and industry to quantify the liveliness of AI as afield. When measuring the performance of AI systems, it is natural to look for comparisonsto human performance. In the Towards Human-Level Performance section we outline ashort list of notable areas where AI systems have made significant progress towards! 7,!,!,AI INDEX, NOVEMBER 2017,matching or exceeding human performance. We also discuss the di f iculties of suchcomparisons and introduce the appropriate caveats.Discussion SectionsFollowing the display of the collected data, we include some discussion of the trendsthis report highlights and important areas this report entirely omits.Part of this discussion centers on the limitations of the report. This report is biasedtowards US-centric data sources and may overestimate progress in technical areas byonly tracking well-defined benchmarks. It also lacks demographic breakdowns of dataand contains no information about AI Research & Development investments bygovernments and corporations. These areas are deeply important and we intend totackle them in future reports. We further discuss these limitations and others in theWhats Missing section of the report.As the reports limitations illustrate, the AI Index will always paint a partial picture. Forthis reason, we include subjective commentary from a cross-section of AI experts. ThisExpert Forum helps animate the story behind the data in the report and addsinterpretation the report lacks.Finally, where the experts dialogue ends, your opportunity to Get Involved begins. Wewill need the feedback and participation of a larger community to address the issuesidentified in this report, uncover issues we have omitted, and build a productiveprocess for tracking activity and progress in Artificial Intelligence.,! 8,!,!,AI INDEX, NOVEMBER 2017,VOLUME OF ACTIVITYAcademiaPublished Papersview more information in appendix A1The number of Computer Science papers published and tagged with the keyword“Artificial Intelligence” in the Scopus database of academic papers.,! 9,9x,The number of AI papers produced each year hasincreased by more than 9x since 1996.,!,!,AI INDEX, NOVEMBER 2017,A comparison of the annual publishing rates of di f erent categories of academicpapers, relative to their publishing rates in 1996. The graph displays the growth ofpapers across all fields, papers within the Computer Science field, and AI papers withinthe Computer Science field.,The data illustrates that growth in AI publishing is driven by more than a growinginterest in the broader field of Computer Science. Concretely, while the number ofpapers within the general field of Computer Science has grown by 6x since 1996 thenumber of AI papers produced each year has increased by more than 9x in that sameperiod.,! 10,!,!,AI INDEX, NOVEMBER 2017,Course Enrollmentview more information in appendix A2The number of students enrolled in introductory Artificial Intelligence & MachineLearning courses at Stanford University.ML is a subfield of AI. We highlight ML courses because of their rapid enrollmentgrowth and because ML techniques are critical to many recent AI achievements.,! 11,Introductory AI class enrollment at Stanford hasincreased 11x si nce 1996.Note : The dip in Stanford ML enrollment for the 2016 academic yearreflects an administrative quirk that year, not student interest. Details inappendix.,11x,!,!,AI INDEX, NOVEMBER 2017,We highlight Stanford because our data on other universities is limited. However, wecan project that past enrollment trends at other universities are similar to Stanfords.,! 12,Note : Many universities have o f ered AI courses since before the 90s. The graphs above represent the yearsfor which we found available data.,!,!,AI INDEX, NOVEMBER 2017,! 13,Note : Many universities have o f ered ML courses since before the 90s. The graphs above represent theyears for which we found available data.It is worth noting that these graphs represent a specific sliver of the higher educationlandscape, and the data is not necessarily representative of trends in the broaderlandscape of academic institutions.,!,!,AI INDEX, NOVEMBER 2017,Conference Attendanceview more information in appendix A3The number of attendees at a representative sample of AI conferences. The data issplit into large conferences (over 1000 attendees in 2016) and small conferences(under 1000 attendees in 2016).,ShiftingFocus,These attendance numbers show that researchfocus has shifted from symbolic reasoningto machine learning and deep learning .Note : Most of the conferences have existed since the 1980s. The dataabove represents the years attendance data was recorded.! 14,!,!,AI INDEX, NOVEMBER 2017,! 15,Despite shifting focus, there is still asmaller research community makingsteady progress on symbolic reasoningmethods in AI.,SteadyProgress,!,!,AI INDEX, NOVEMBER 2017,IndustryAI-Related Startupsview more information in appendix A4The number of active venture-backed US private companies developing AI systems.,! 16,The number of active US startups developing AIsystems has increased 14x since 2000.,14x,!,!,AI INDEX, NOVEMBER 2017,AI-Related Startup Fundingview more information in appendix A5The amount of annual funding by VCs into US AI startups across all funding stages.,! 17,Annual VC investment into US startups developing AIsystems has increased 6x since 2000.,6x,!,!,AI INDEX, NOVEMBER 2017,Job Openingsview more information in appendix A6We obtained AI-related job growth data from two online job listing platforms, Indeedand Monster. AI-related jobs were identified with titles and keywords in descriptions.The growth of the share of US jobs requiring AI skills on the I platform.Growth is a multiple of the share of jobs on the Indeed platform that required AI skillsin the US in January 2013.,! 18,The share of jobs requiring AI skills in the US hasgrown 4.5x since 2013.,4.5x,!,!,AI INDEX, NOVEMBER 2017,The growth of the share of jobs requiring AI skills on the I platform, bycountry.,! 19,Note : Despite the rapid growth of the Canada and UK AI job markets, I reports they arerespectively still 5% and 27% of the absolute size of the US AI job market.,!,!,AI INDEX, NOVEMBER 2017,The total number of AI job openings posted on the Monster platform in a given year,broken down by specific required skills.,! 20,Note: A single AI-related job may be double counted (belong to multiple categories). Forexample, a job may specically require natural language processing and computer vision skills.,!,!,AI INDEX, NOVEMBER 2017,Robot Importsview more information in appendix A7The number of imports of industrial robot units into North America and globally.,!,!,AI INDEX, NOVEMBER 2017,The growth of imports of industrial robot units into North America and globally.,! 22,!,!,AI INDEX, NOVEMBER 2017,Open Source SoftwareGitHub Project Statisticsview more information in appendix A8The number of times the TensorFlow and Scikit-Learn software packages have beenStarred on GitHub. TensorFlow and Scikit-Learn are popular software packages fordeep learning and machine learning.,Software developers “Star ” software projects on GitHub to indicate projects they areinterested in, express appreciation for projects, and navigate to projects quickly. Starscan provide a signal for developer interest in and usage of software. ! 23,!,!,AI INDEX, NOVEMBER 2017,The number of times various AI & ML software packages have been Starred onGitHub.,! 24,Note: Forks of GitHub repositories follow almost identical trends (though, the absolute numberof forks and stars for each repo di er). See the appendix for info on gathering Forks data.,!,!,AI INDEX, NOVEMBER 2017,Public InterestSentiment of Media Coverageview more information in appendix A9The percentage of popular media articles that contain the term “Artificial Intelligence”and that are classified as either Positive or Negative articles.,! 25,!,!,AI INDEX, NOVEMBER 2017,TECHNICAL PERFORMANCEVisionObject Detectionview more information in appendix A10The performance of AI systems on the object detection task in the Large Scale VisualRecognition Challenge (LSVRC) Competition.,! 26,Error rates for image labeling have fallenfrom 28.5% to below 2.5% since 2010.,2.5%,!,!,AI INDEX, NOVEMBER 2017,Visual Question Answeringview more information in appendix A11The performance of AI systems on a task to give open-ended answers to questionsabout images.,! 27,Note: The VQA 1.0 data set has already been surpassed by the VQA 2.0 data set and it isunclear how much further attention the VQA 1.0 data set will receive.,!,!,AI INDEX, NOVEMBER 2017,Natural Language UnderstandingParsingview more information in appendix A12The performance of AI systems on a task to determine the syntactic structure ofsentences.,! 28,!,!,AI INDEX, NOVEMBER 2017,Machine Translationview more information in appendix A13The performance of AI systems on a task to translate news between English andGerman.,! 29,!,!,AI INDEX, NOVEMBER 2017,Question Answeringview more information in appendix A14The performance of AI systems on a task to find the answer to a question within adocument.,! 30,!,!,AI INDEX, NOVEMBER 2017,Speech Recognitionview more information in appendix A15The performance of AI systems on a task to recognize speech from phone call audio.,! 31,!,!,AI INDEX, NOVEMBER 2017,Theorem Provingview more information in appendix A16The average tractability of a large set of theorem proving problems for AutomaticTheorem Provers. “Tractability” measures the fraction of state-of-the-art AutomaticTheorem Provers that can solve a problem. See appendix for details about the“tractability” metric.,! 32,Note: Average tractability can go down if state-of-the-art solvers are introduced that performwell on novel problems but poorly on problems other solvers are good at.,!,!,AI INDEX, NOVEMBER 2017,SAT Solvingview more information in appendix A17The percentage of problems solved by competitive SAT solvers on industry-applicableproblems.,! 33,!,!,AI INDEX, NOVEMBER 2017,DERIVATIVE MEASURESWe can glean additional insights from the measurements in the previous sections byexamining the relationships between trends. This section demonstrates how the datagathered by the AI Index can be used for further analysis and to spur the developmentof refined and wholly original metrics.As a case-study for this demonstration, we look at trends across academia andindustry to explore their dynamics. Further, we aggregate these metrics into acombined AI Vibrancy Index.Academia-Industry DynamicsTo explore the relationship between AI-related activity in academia and industry, wefirst select a few representative measurements from the previous sections. Inparticular, we look at AI paper publishing, combined enrollment in introductory AI andML courses at Stanford, and VC investments into AI-related startups.These metrics represent quantities that cannot be compared directly: paperspublished, students enrolled, and amount invested. In order to analyze the relationshipbetween these trends, we first normalize each measurement starting at the year 2000.This allows us to compare how the metrics have grown instead of the absolute valuesof the metrics over time.,! 34,!,!,AI INDEX, NOVEMBER 2017,! 35,Note: The dip in enrollment for the 2016 academic year reects an administrative quirk thatyear, not student interest. Details in appendix A2.The data shows that, initially, academic activity (publishing and enrollment) drovesteady progress. Around 2010 investors started to take note and by 2013 became thedrivers of the steep increase in total activity. Since then, academia has caught up withthe exuberance of industry.,!,!,AI INDEX, NOVEMBER 2017,The AI Vibrancy IndexThe AI Vibrancy Index aggregates the measurements from academia and industry(publishing, enrollment and VC investment) to quantify the liveliness of AI as a field.To compute the AI Vibrancy Index, we average normalized publishing, enrollment andinvestment metrics over time.,We hope this brief investigation sparks interest in how metrics from the AI Index canbe further analyzed and creates discussion about what derived measures may beuseful to track over time.,! 36,!,!,AI INDEX, NOVEMBER 2017,TOWARDS HUMAN-LEVELPERFORMANCE?It is natural to look for comparisons between the performance of AI systems andhumans on the same task. Obviously, computers are vastly superior to humans incertain tasks; 1970-era hand calculators can perform arithmetic better than humans.However, the competence of AI systems becomes more di f icult to assess whendealing with more general tasks like answering questions, playing games, and makingmedical diagnoses.Tasks for AI systems are often framed in narrow contexts for the sake of makingprogress on a specific problem or application. While machines may exhibit stellarperformance on a certain task, performance may degrade dramatically if the task ismodified even slightly. For example, a human who can read Chinese characters wouldlikely understand Chinese speech, know something about Chinese culture and evenmake good recommendations at Chinese restaurants. In contrast, very di f erent AIsystems would be needed for each of these tasks.Machine performance may degrade dramatically if theoriginal task is modified even slightly.Despite the di f iculty of comparing human and AI systems, it is interesting tocatalogue credible claims that computers have reached or exceeded human-levelperformance. Still, it is important to remember that these achievements say nothingabout the ability of these systems to generalize. We also note the list below containsmany game playing achievements. Games provide a relatively simple, controlled,experimental environment and so are often used for AI research.,! 37,!,!,AI INDEX, NOVEMBER 2017,MilestonesBelow is a brief description of the achievements and their circumstances. Somemilestones repres