人工智能指数报告 AI report

!!AIINDEX,NOVEMBER2021!

1!!AIINDEX,NOVEMBER2021STEERING

COMMITTEE

Yoav

Shoham

(chair)

Stanford

University

Raymond

Perrault

SRI

International

Erik

Brynjolfsson

MIT

Jack

Clark

OpenAI

PROJECT

MANAGER

Calvin

LeGassick!

2!!AIINDEX,NOVEMBER2021TABLEOFCONTENTSIntroductiontotheAIIndex2021AnnualReportOverviewVolumeofActivity Academia PublishedPapers CourseEnrollment ConferenceAttendance Industry AI-RelatedStartups AI-RelatedStartupFunding JobOpenings RobotImports OpenSourceSoftware GitHubProjectStatistics PublicInterest SentimentofMediaCoverageTechnicalPerformance Vision ObjectDetection VisualQuestionAnswering NaturalLanguageUnderstanding Parsing MachineTranslation QuestionAnswering SpeechRecognition

5

7

9

9

9

11

14

16

16

17

18

2123232525262626

2728282930

31!

3!!AIINDEX,NOVEMBER2021

Theorem

Proving

SAT

SolvingDerivative

MeasuresTowards

Human-Level

Performance?What’s

Missing?Expert

ForumGet

Involved!AcknowledgementsAppendix

A:

Data

Description

&

Collection

Methodology323334

37

41446870

72!

4!!AIINDEX,NOVEMBER2021INTRODUCTIONTOTHEAIINDEX2021ANNUALREPORTArtificialIntelligencehasleapttotheforefrontofglobaldiscourse,garneringincreasedattentionfrompractitioners,industryleaders,policymakers,andthegeneralpublic.ThediversityofopinionsanddebatesgatheredfromnewsarticlesthisyearillustratesjusthowbroadlyAIisbeinginvestigated,studied,andapplied.However,thefieldofAIisstillevolvingrapidlyandevenexpertshaveahardtimeunderstandingandtrackingprogressacrossthefield.!

5!!AIINDEX,NOVEMBER2021WithouttherelevantdataforreasoningaboutthestateofAItechnology,weareessentially“flyingblind〞inourconversationsanddecision-makingrelatedtoAI.Weareessentially“flyingblind〞inourconversationsanddecision-makingrelatedtoArtificialIntelligence.CreatedandlaunchedasaprojectoftheOneHundredYearStudyonAIatStanfordUniversity(AI100),theAIIndexisanopen,not-for-profitprojecttotrackactivityandprogressinAI.ItaimstofacilitateaninformedconversationaboutAIthatisgroundedindata.ThisistheinauguralannualreportoftheAIIndex,andinthisreportwelookatactivityandprogressinArtificialIntelligencethrougharangeofperspectives.Weaggregatedatathatexistsfreelyontheweb,contributeoriginaldata,andextractnewmetricsfromcombinationsofdataseries.AllofthedatausedtogeneratethisreportwillbeopenlyavailableontheAIIndexwebsiteat.Providingdata,however,isjustthebeginning.Tobecometrulyuseful,theAIIndexneedssupportfromalargercommunity.Ultimately,thisreportisacallforparticipation.Youhavetheabilitytoprovidedata,analyzecollecteddata,andmakeawishlistofwhatdatayouthinkneedstobetracked.Whetheryouhaveanswersorquestionstoprovide,wehopethisreportinspiresyoutoreachouttotheAIIndexandbecomepartoftheeforttogroundtheconversationaboutAI.!

6!!AIINDEX,NOVEMBER2021OVERVIEWThefirsthalfofthereportshowcasesdataaggregatedbytheAIIndexteam.Thisisfollowedbyadiscussionofkeyareasthereportdoesnotaddress,expertcommentaryonthetrendsdisplayedinthereport,andacalltoactiontosupportourdatacollectionefortsandjointheconversationaboutmeasuringandcommunicatingprogressinAItechnology.DataSectionsThedatainthereportisbrokenintofourprimaryparts: •VolumeofActivity •TechnicalPerformance •DerivativeMeasures •TowardsHuman-LevelPerformance?TheVolumeofActivitymetricscapturethe“howmuch〞aspectsofthefield,likeattendanceatAIconferencesandVCinvestmentsintostartupsdevelopingAIsystems.TheTechnicalPerformancemetricscapturethe“howgood〞aspects;forexample,howwellcomputerscanunderstandimagesandprovemathematicaltheorems.Themethodologyusedtocollecteachdatasetisdetailedintheappendix.Thesefirsttwosetsofdataconfirmwhatisalreadywellrecognized:allgraphsare“upandtotheright,〞reflectingtheincreasedactivityinAIefortsaswellastheprogressofthetechnology.IntheDerivativeMeasuressectionweinvestigatetherelationshipbetweentrends.Wealsointroduceanexploratorymeasure,theAIVibrancyIndex,thatcombinestrendsacrossacademiaandindustrytoquantifythelivelinessofAIasafield.
WhenmeasuringtheperformanceofAIsystems,itisnaturaltolookforcomparisonstohumanperformance.IntheTowardsHuman-LevelPerformancesectionweoutlineashortlistofnotableareaswhereAIsystemshavemadesignificantprogresstowards !7!!AIINDEX,NOVEMBER2021matching

or

exceeding

human

performance.

We

also

discuss

the

di

f

iculties

of

suchcomparisons

and

introduce

the

appropriate

caveats.Discussion

SectionsFollowing

the

display

of

the

collected

data,

we

include

some

discussion

of

the

trendsthis

report

highlights

and

important

areas

this

report

entirely

omits.Part

of

this

discussion

centers

on

the

limitations

of

the

report.

This

report

is

biasedtowards

US-centric

data

sources

and

may

overestimate

progress

in

technical

areas

byonly

tracking

well-defined

benchmarks.

It

also

lacks

demographic

breakdowns

of

dataand

contains

no

information

about

AI

Research

&

Development

investments

bygovernments

and

corporations.

These

areas

are

deeply

important

and

we

intend

totackle

them

in

future

reports.

We

further

discuss

these

limitations

and

others

in

theWhat’s

Missing

section

of

the

report.As

the

report’s

limitations

illustrate,

the

AI

Index

will

always

paint

a

partial

picture.

Forthis

reason,

we

include

subjective

commentary

from

a

cross-section

of

AI

experts.

ThisExpert

Forum

helps

animate

the

story

behind

the

data

in

the

report

and

addsinterpretation

the

report

lacks.Finally,

where

the

experts’

dialogue

ends,

your

opportunity

to

Get

Involved

begins.

Wewill

need

the

feedback

and

participation

of

a

larger

community

to

address

the

issuesidentified

in

this

report,

uncover

issues

we

have

omitted,

and

build

a

productiveprocess

for

tracking

activity

and

progress

in

Artificial

Intelligence.

!

8!!AIINDEX,NOVEMBER2021VOLUMEOFACTIVITYAcademiaPublishedPapersviewmoreinformationinappendixA1ThenumberofComputerSciencepaperspublishedandtaggedwiththekeyword“ArtificialIntelligence〞intheScopusdatabaseofacademicpapers.!

99xThe

number

of

AIpapers

produced

each

year

hasincreased

by

more

than

9x

since

1996.!!AIINDEX,NOVEMBER2021A

comparison

of

the

annual

publishing

rates

of

di

f

erent

categories

of

academicpapers,

relative

to

their

publishing

rates

in

1996.

The

graph

displays

the

growth

ofpapers

across

all

fields,

papers

within

the

Computer

Science

field,

and

AI

papers

withinthe

Computer

Science

field.The

data

illustrates

that

growth

in

AI

publishing

is

driven

by

more

than

a

growinginterest

in

the

broader

field

of

Computer

Science.

Concretely,

while

the

number

ofpapers

within

the

general

field

of

Computer

Science

has

grown

by

6x

since

1996

thenumber

of

AI

papers

produced

each

year

has

increased

by

more

than

9x

in

that

sameperiod.

!

10!!AIINDEX,NOVEMBER2021Course

EnrollmentviewmoreinformationinappendixA2The

number

of

students

enrolled

in

introductory

Artificial

Intelligence

&

MachineLearning

courses

at

Stanford

University.ML

is

a

subfield

of

AI.

We

highlight

ML

courses

because

of

their

rapid

enrollmentgrowth

and

because

ML

techniques

are

critical

to

many

recent

AI

achievements.!

11IntroductoryAIclassenrollmentatStanfordhasincreased11xsince1996.Note:ThedipinStanfordMLenrollmentforthe2021academicyearreflectsanadministrativequirkthatyear,notstudentinterest.Detailsinappendix.11x!!AIINDEX,NOVEMBER2021We

highlight

Stanford

because

our

data

on

other

universities

is

limited.

However,

wecan

project

that

past

enrollment

trends

at

other

universities

are

similar

to

Stanford’s.!

12Note

:

Many

universities

have

o

f

ered

AI

courses

since

before

the

90’s.

The

graphs

above

represent

the

yearsfor

which

we

found

available

data.!!AIINDEX,NOVEMBER2021!

13

Note

:

Many

universities

have

o

f

ered

ML

courses

since

before

the

90’s.

The

graphs

above

represent

the

years

for

which

we

found

available

data.It

is

worth

noting

that

these

graphs

represent

a

specific

sliver

of

the

higher

educationlandscape,

and

the

data

is

not

necessarily

representative

of

trends

in

the

broaderlandscape

of

academic

institutions.!!AIINDEX,NOVEMBER2021ConferenceAttendanceviewmoreinformationinappendixA3ThenumberofattendeesatarepresentativesampleofAIconferences.Thedataissplitintolargeconferences(over1000attendeesin2021)andsmallconferences(under1000attendeesin2021).ShiftingFocusThese

attendance

numbers

show

that

researchfocus

has

shifted

fromsymbolicreasoningtomachinelearninganddeeplearning

.Note

:

Most

of

the

conferences

have

existed

since

the

1980s.

The

dataabove

represents

the

years

attendance

data

was

recorded.

!

14!!AIINDEX,NOVEMBER2021!

15Despite

shifting

focus,

there

is

still

asmallerresearchcommunity

makingsteady

progress

onsymbolicreasoningmethods

in

AI.SteadyProgress!!AIINDEX,NOVEMBER2021IndustryAI-Related

StartupsviewmoreinformationinappendixA4The

number

of

active

venture-backed

US

private

companies

developing

AI

systems.!

16The

number

of

activeUSstartups

developing

AIsystems

has

increased

14x

since

2000.14x!!AIINDEX,NOVEMBER2021AI-Related

Startup

FundingviewmoreinformationinappendixA5The

amount

of

annual

funding

by

VC’s

into

US

AI

startups

across

all

funding

stages.

!

17AnnualVCinvestment

into

US

startups

developing

AIsystems

has

increased

6x

since

2000.6x!!AIINDEX,NOVEMBER2021JobOpeningsviewmoreinformationinappendixA6WeobtainedAI-relatedjobgrowthdatafromtwoonlinejoblistingplatforms,IndeedandMonster.AI-relatedjobswereidentifiedwithtitlesandkeywordsindescriptions.ThegrowthoftheshareofUSjobsrequiringAIskillsontheIndeedplatform.GrowthisamultipleoftheshareofjobsontheIndeedplatformthatrequiredAIskillsintheUSinJanuary2021.

!

18TheshareofjobsrequiringAIskillsintheUShasgrown4.5xsince2021.4.5x!!AIINDEX,NOVEMBER2021ThegrowthoftheshareofjobsrequiringAIskillsontheIndeedplatform,bycountry.!

19Note:DespitetherapidgrowthoftheCanadaandUKAIjobmarkets,Indeedreportstheyarerespectivelystill5%and27%oftheabsolutesizeoftheUSAIjobmarket.!!AIINDEX,NOVEMBER2021The

total

number

of

AI

job

openings

posted

on

the

Monster

platform

in

a

given

year,broken

down

by

specific

required

skills.!

20Note:

A

single

AI-related

job

may

be

double

counted

(belong

to

multiple

categories).

Forexample,

a

job

may

specifically

require

natural

language

processing

and

computer

vision

skills.!!AIINDEX,NOVEMBER2021Robot

ImportsviewmoreinformationinappendixA7The

number

of

imports

of

industrial

robot

units

into

North

America

and

globally.!

21!!AIINDEX,NOVEMBER2021The

growth

of

imports

of

industrial

robot

units

into

North

America

and

globally.!

22!!AIINDEX,NOVEMBER2021Open

Source

SoftwareGitHub

Project

StatisticsviewmoreinformationinappendixA8The

number

of

times

the

TensorFlow

and

Scikit-Learn

software

packages

have

beenStarred

on

GitHub.

TensorFlow

and

Scikit-Learn

are

popular

software

packages

fordeep

learning

and

machine

learning.Softwaredevelopers“Star〞softwareprojectsonGitHubtoindicateprojectstheyareinterestedin,expressappreciationforprojects,andnavigatetoprojectsquickly.Starscanprovideasignalfordeveloperinterestinandusageofsoftware.
 !23!!AIINDEX,NOVEMBER2021The

number

of

times

various

AI

&

ML

software

packages

have

been

Starred

onGitHub.!

24Note:

Forks

of

GitHub

repositories

follow

almost

identical

trends

(though,

the

absolute

numberof

forks

and

stars

for

each

repo

di

ff

er).

See

the

appendix

for

info

on

gathering

Forks

data.!!AIINDEX,NOVEMBER2021PublicInterestSentimentofMediaCoverageviewmoreinformationinappendixA9Thepercentageofpopularmediaarticlesthatcontaintheterm“ArtificialIntelligence〞andthatareclassifiedaseitherPositiveorNegativearticles.!

25!!AIINDEX,NOVEMBER2021TECHNICAL

PERFORMANCEVisionObject

DetectionviewmoreinformationinappendixA10The

performance

of

AI

systems

on

the

object

detection

task

in

the

Large

Scale

VisualRecognition

Challenge

(LSVRC)

Competition.!

26Errorratesforimagelabelinghavefallenfrom28.5%tobelow2.5%since2021.2.5%!!AIINDEX,NOVEMBER2021Visual

Question

AnsweringviewmoreinformationinappendixA11The

performance

of

AI

systems

on

a

task

to

give

open-ended

answers

to

questionsabout

images.!

27Note:

The

VQA

1.0

data

set

has

already

been

surpassed

by

the

VQA

2.0

data

set

and

it

isunclear

how

much

further

attention

the

VQA

1.0

data

set

will

receive.!!AIINDEX,NOVEMBER2021Natural

Language

UnderstandingParsingviewmoreinformationinappendixA12The

performance

of

AI

systems

on

a

task

to

determine

the

syntactic

structure

ofsentences.!

28!!AIINDEX,NOVEMBER2021Machine

TranslationviewmoreinformationinappendixA13The

performance

of

AI

systems

on

a

task

to

translate

news

between

English

andGerman.!

29!!AIINDEX,NOVEMBER2021Question

AnsweringviewmoreinformationinappendixA14The

performance

of

AI

systems

on

a

task

to

find

the

answer

to

a

question

within

adocument.

!

30!!AIINDEX,NOVEMBER2021Speech

RecognitionviewmoreinformationinappendixA15The

performance

of

AI

systems

on

a

task

to

recognize

speech

from

phone

call

audio.

!

31!!AIINDEX,NOVEMBER2021TheoremProvingviewmoreinformationinappendixA16TheaveragetractabilityofalargesetoftheoremprovingproblemsforAutomaticTheoremProvers.“Tractability〞measuresthefractionofstate-of-the-artAutomaticTheoremProversthatcansolveaproblem.Seeappendixfordetailsaboutthe“tractability〞metric.!

32Note:

Average

tractability

can

go

down

if

state-of-the-art

solvers

are

introduced

that

performwell

on

novel

problems

but

poorly

on

problems

other

solvers

are

good

at.!!AIINDEX,NOVEMBER2021SAT

SolvingviewmoreinformationinappendixA17The

percentage

of

problems

solved

by

competitive

SAT

solvers

on

industry-applicableproblems.!

33!!AIINDEX,NOVEMBER2021DERIVATIVE

MEASURESWe

can

glean

additional

insights

from

the

measurements

in

the

previous

sections

byexamining

the

relationships

between

trends.

This

section

demonstrates

how

the

datagathered

by

the

AI

Index

can

be

used

for

further

analysis

and

to

spur

the

developmentof

refined

and

wholly

original

metrics.As

a

case-study

for

this

demonstration,

we

look

at

trends

across

academia

andindustry

to

explore

their

dynamics.

Further,

we

aggregate

these

metrics

into

acombined

AI

Vibrancy

Index.Academia-Industry

DynamicsTo

explore

the

relationship

between

AI-related

activity

in

academia

and

industry,

wefirst

select

a

few

representative

measurements

from

the

previous

sections.

Inparticular,

we

look

at

AI

paper

publishing,

combined

enrollment

in

introductory

AI

andML

courses

at

Stanford,

and

VC

investments

into

AI-related

startups.These

metrics

represent

quantities

that

cannot

be

compared

directly:

paperspublished,

students

enrolled,

and

amount

invested.

In

order

to

analyze

the

relationshipbetween

these

trends,

we

first

normalize

each

measurement

starting

at

the

year

2000.This

allows

us

to

compare

how

the

metrics

have

grown

instead

of

the

absolute

valuesof

the

metrics

over

time.!

34!!AIINDEX,NOVEMBER2021!

35 Note:Thedipinenrollmentforthe2021academicyearreflectsanadministrativequirkthat year,notstudentinterest.DetailsinappendixA2.Thedatashowsthat,initially,academicactivity(publishingandenrollment)drovesteadyprogress.Around2021investorsstartedtotakenoteandby2021becamethedriversofthesteepincreaseintotalactivity.Sincethen,academiahascaughtupwiththeexuberanceofindustry.!!AIINDEX,NOVEMBER2021The

AI

Vibrancy

IndexThe

AI

Vibrancy

Index

aggregates

the

measurements

from

academia

and

industry(publishing,

enrollment

and

VC

investment)

to

quantify

the

liveliness

of

AI

as

a

field.To

compute

the

AI

Vibrancy

Index,

we

average

normalized

publishing,

enrollment

andinvestment

metrics

over

time.We

hope

this

brief

investigation

sparks

interest

in

how

metrics

from

the

AI

Index

canbe

further

analyzed

and

creates

discussion

about

what

derived

measures

may

beuseful

to

track

over

time.!

36!!AIINDEX,NOVEMBER2021TOWARDS

HUMAN-LEVELPERFORMANCE?It

is

natural

to

look

for

comparisons

between

the

performance

of

AI

systems

andhumans

on

the

same

task.

Obviously,

computers

are

vastly

superior

to

humans

incertain

tasks;

1970-era

hand

calculators

can

perform

arithmetic

better

than

humans.However,

the

competence

of

AI

systems

becomes

more

di

f

icult

to

assess

whendealing

with

more

general

tasks

like

answering

questions,

playing

games,

and

makingmedical

diagnoses.Tasks

for

AI

systems

are

often

framed

in

narrow

contexts

for

the

sake

of

makingprogress

on

a

specific

problem

or

application.

While

machines

may

exhibit

stellarperformance

on

a

certain

task,

performance

may

degrade

dramatically

if

the

task

ismodified

even

slightly.

For

example,

a

human

who

can

read

Chinese

characters

wouldlikely

understand

Chinese

speech,

know

something

about

Chinese

culture

and

evenmake

good

recommendations

at

Chinese

restaurants.

In

contrast,

very

di

f

erent

AIsystems

would

be

needed

for

each

of

these

tasks.Machine

performance

may

degrade

dramatically

if

theoriginal

task

is

modified

even

slightly.Despite

the

di

f

iculty

of

comparing

human

and

AI

systems,

it

is

interesting

tocatalogue

credible

claims

that

computers

have

reached

or

exceeded

human-levelperformance.

Still,

it

is

important

to

remember

that

these

achievements

say

nothingabout

the

ability

of

these

systems

to

generalize.

We

also

note

the

list

below

containsmany

game

playing

achievements.

Games

provide

a

relatively

simple,

controlled,experimental

environment

and

so

are

often

used

for

AI

research.

!

37!!AIINDEX,NOVEMBER2021MilestonesBelow

is

a

brief

description

of

the

achievements

and

their

circumstances.

Somemilestones

represent

significant

progress

towards

human

performance

and

othersrepresent

super-human

performance

achievements.OthelloIn

the

1980s

Kai-Fu

Lee

and

Sanjoy

Mahajan

developed

BILL,

a

Bayesianlearning-based

system

for

playing

the

board

game

Othello.

In

1989

theprogram

won

the

US

national

tournament

of

computer

players,

and

beatthe

highest

ranked

US

player,

Brian

Rose,

56-8.

In

1997

a

program

namedLogistello

won

every

game

in

a

six

game

match

against

the

reigningOthello

world

champion.CheckersIn

1952,

Arthur

Samuels

built

a

series

of

programs

that

played

the

game

ofcheckers

and

improved

via

self-play.

However,

it

was

not

until

1995

that

acheckers-playing

program,

Chinook,

beat

the

world

champion.ChessSome

computer

scientists

in

the

1950s

predicted

that

a

computer

woulddefeat

the

human

chess

champion

by

1967,

but

it

was

not

until

1997

thatIBM’s

DeepBlue

system

beat

chess

champion

Gary

Kasparov.

Today,

chessprograms

running

on

smartphones

can

play

at

the

grandmaster

level.

!

38198019951997!AIINDEX,NOVEMBER2021 Jeopardy! In2021,theIBMWatsoncomputersystemcompetedonthepopularquiz- showJeopardy!againstformerwinnersBradRutterandKenJennings. Watsonwonthefirstplaceprizeof$1million. AtariGames In2021,ateamatGoogleDeepMindusedareinforcementlearningsystem tolearnhowtoplay49Atarigames.Thesystemwasabletoachieve human-levelperformanceinamajorityofthegames(e.g.,Breakout), thoughsomearestillsignificantlyoutofreach(e.g.,Montezuma’s Revenge). ObjectDetectioninImageNet In2021,theerrorrateofautomaticlabelingofImageNetdeclinedfrom 28%in2021tolessthan3%.Humanperformanceisabout5%. Go InMarchof2021,theAlphaGosystemdevelopedbytheGoogleDeepMind teambeatLeeSedol,oneoftheworld’sgreatestGoplayers,4-1. DeepMindthenreleasedAlphaGoMaster,whichdefeatedthetopranked player,KeJie,inMarchof2021.InOctober2021aNaturepaperdetailed yetanothernewversion,AlphaGoZero,whichbeattheoriginalAlphaGo system100-0.!

39 ! 2021202120212021!AIINDEX,NOVEMBER2021 SkinCancerClassification Ina2021Naturearticle,Estevaetal.describeanAIsystemtrainedona datasetof129,450clinicalimagesof2,032diferentdiseasesand compareitsdiagnosticperformanceagainst21board-certified dermatologists.TheyfindtheAIsystemcapableofclassifyingskincancer atalevelofcompetencecomparabletothedermatologists. SpeechRecognitiononSwitchboard In2021,

MicrosoftandIBMbothachievedperformancewithincloserange of“human-parity〞speechrecognitioninthelimitedSwitchboarddomain. Poker InJanuary2021,aprogramfromCMUcalledLibratusdefeatedfourtop humanplayersinatournamentof120,000gamesoftwo-player,headsup, no-limitTexasHold’em.InFebruary2021,aprogramfromtheUniversityof AlbertacalledDeepStackplayedagroupof11professionalplayersmore than3,000gameseach.DeepStackwonenoughpokergamestoprovethe statisticalsignificanceofitsskillovertheprofessionals. Ms.Pac-Man Maluuba,adeeplearningteamacquiredbyMicrosoft,createdanAI systemthatlearnedhowtoreachthegame’smaximumpointvalueof 999,900onAtari2600.!

40 !2021202120212021!!AIINDEX,NOVEMBER2021WHAT

’S

MISSING?This

inaugural

annual

report

covers

a

lot

of

ground,

but

certainly

not

all

of

it.

Manyimportant

areas

were

omitted

for

lack

of

available

data,

time,

or

both.

We

hope

toaddress

the

limitations

below

in

future

editions

of

the

report.We

also

believe

it

will

take

the

support

of

the

broader

community

to

e

f

ectively

engagein

this

broad

range

of

challenges,

and

we

invite

you

to

reach

out

to

the

AI

Index

if

youhave

ideas

or

relevant

data

for

tackling

these

challenges.Technical

PerformanceWe

did

not

cover

progress

in

many

important

technical

areas.

For

some

areas

there

arenot

clear

standardized

benchmarks

(e.g.

dialogue

systems,

planning,

continuouscontrol

in

robotics).

In

other

areas

it

is

hard

to

measure

performance

when

there

hasnot

been

significant

progress,

like

in

commonsense

reasoning.

And

still,

other

areasare

waiting

to

be

tracked

but

we

simply

have

not

had

the

opportunity

to

collect

thedata

(e.g.

recommender

systems,

standardized

testing).Tracking

areas

that

have

traditionally

lacked

concrete

measurements

may

alsofacilitate

a

more

sober

assessment

of

AI

progress.

Progress

is

typically

trackedconsistently

when

good

progress

has

been

made.

As

a

result,

this

report

may

presentan

overly

optimistic

picture.Indeed,

chatbot

dialog

falls

far

short

of

human

dialog

and

we

lack

widely

acceptedbenchmarks

for

progress

in

this

area.

Similarly,

while

today’s

AI

systems

have

far

lesscommon

sense

reasoning

than

that

of

a