大语言模型与知识图谱：机遇与挑战 Large Language Models and Knowledge Graphs - Opportunities and Challenges

arXivv[cs.AI]11Aug2023arXivv[cs.AI]11Aug2023JeffZ.Pan回仆OTheUniversityofEdinburgh,UnitedKingdomSimonRazniewski回BoschCenterforAI,GermanyJan-ChristophKalo回VrijeUniversiteitAmsterdam,TheNetherlandsSnehaSinghania回MaxPlanckInstituteforInformatics,GermanyJiaoyanChen回TheUniversityofManchester&UniversityofOxford,UnitedKingdomStefanDietze回GESIS-LeibnizInstitutefortheSocialSciences&Heinrich-Heine-UniversitätDüsseldorf,GermanyHajiraJabeen回GESIS-LeibnizInstitutefortheSocialSciences,GermanyJannaOmeliyanenko回UniversityofWürzburg,GermanyWenZhang回ZhejiangUniversity,ChinaMatteoLissandrini回AalborgUniversity,DenmarkRussaBiswas回Hasso-PlattnerInstitute,GermanyGerarddeMelo回Hasso-PlattnerInstitute,GermanyAngelaBonifati回Lyon1University,FranceEdliraVakaj回BirminghamCityUniversity,UnitedKingdomMauroDragoni回FondazioneBrunoKessler,ItalyDamienGraux回EdinburghResearchCentre,CSI,HuaweiTechnologiesUK,UnitedKingdom tractLargeLanguageModels(LLMs)havetakenKnowledgeRepresentation—andtheworld—bystorm.Thisinflectionpointmarksashiftfromex-plicitknowledgerepresentationtoarenewedfocusonthehybridrepresentationofbothexplicitknow-ledgeandparametricknowledge.Inthispositionpaper,wewilldiscusssomeofthecommondebatepointswithinthecommunityonLLMs(parametricknowledge)andKnowledgeGraphs(explicitknow-ledge)andspeculateonopportunitiesandvisionsthattherenewedfocusbrings,aswellasrelatedresearchtopicsandchallenges.©JeffZ.Pan,SimonRazniewski,Jan-ChristophKalo,SnehaSinghania,JiaoyanChen,StefanDietze,HajiraJabeen,WenZhang,MatteoLissandrini,RussaBiswas,GerarddeMelo,AngelaBonifati,EdliraVakaj,MauroDragoniandDamienGraux;licensedunderCreativeCommonsAttribution4.0International(CCBY4.0)...,Vol.000,Issue111,ArticleNo.42,pp.42:1–42:30......LLMsandKGs:OpportunitiesandChallenges22012ACMSubjectClassificationGeneralandreference→Generalliterature,KeywordsandPhrasesLargeLanguageModel,Pre-trainedLanguageModel,tology,RetrievalAugmentedLanguageModelsDigitalObjectIdentifier10.1234/0000000.00000000ReceivedDateofsubmissionAcceptedDateofacceptancePublishedDateofEditorEditorNameGeneralandreferenceKnowledgeGraph,On-publishing 1IntroductionplearedelsThearrivalofLLMsannouncestheeraofKnowledgeComputingdonredtoanother,untilaroundthe1960s,whenresearchersstartedtostudyknowledgerepresentationscaleusingwhichthelargescaleknowledgebasesarethenArelatedsurveypaper[204]presentsacomprehensivereviewofusingLLMsforKGcon-hontologiesasschemasaswellasotherOtherworksontheintersectionofLLMsandKGshaveaminoroverlapwiththetopicscoveredinourpaper;e.g.,onstudiesusingLLMsasKGs[5],onusingKGstoaugmentLLMs[185],oringOverall,noneofthesepaperslookintotheimplicationsoftheinflectionpointwithconcreteJeffJeffZ.Panetal.42:3ndebatepointswithinthecomLMs 2CommonDebatePointswithintheCommunityKnowledgeRepresentationandReasoning:KGsofferastructuredrepresentationofparametricknowledgeinLLMsreliesonstatisticalpatternsratherthantrueunderstandingandreasoning[13].ProponentsofLLMslikeChatGPT,highlighttheirabilitytogeneralizefromrestageThusparametricknowledgeisnottheonlyssHighPrecisionMethods:ThesuccessofKGscanlargelybeattributedtotheirabilitytoprovidefactualinformationaboutentitieswithhighaccuracy.Forinstance,YAGO[154]utilizedinproduction,partlyduetoitsinabilitytoreachthedesired99%accuracyintheirandGPT-3,equippedwithhundredsofbillionsofparameters[4].ThiscallsfornovelhighprecisionmethodsforKGconstructionbasedonLLMs.NumericalValues:ItiswidelyrecognizedthatLLMsgrapplewithhandlingnumericalval-tlyduringpretraininginawaythatenablesthemforsubsequentuseinKGsmodifyingLLMstohandlenumericalvaluesremainsunresolved,makingtheirutilizationforLLMsandKGs:OpportunitiesandChallengesBias,FairnessandBeyond:CriticsarguethatLLMscanperpetuateandamplifybiasessayondbiasandfairnessincludingbutnotlimitedExplainabilityandInterpretability:KGsareoftenpreferredinscenarioswhereexplain-abilityandinterpretabilityarecrucial[28],astheyexplicitlyrepresentrelationshipsbetweenmodelslacktransparencyandinterpretability,makingitdifficulttounderstandhowtheyar- 3OpportunitiesandVisions1.Instantaccesstohugetextcorpora:AsmentionedintheIntroduction,foralongtime,isallowsAIdeveloperstoavoidgettingRicherknowledgeformanysubtasks:AlthoughthemostprominentcapabilitiesofLLMs,JeffJeffZ.Panetal.42:5prompting,LLMshaveadvancedmanytaskssuchasdependencyandstructuredparsing,en-Byinjectingexplicit,andinparticularstructured,knowledgeintoLLMs,suchasthrough3.Evenmoreadvancedlanguageunderstanding:LLMsalonealreadysignificantlyad-wwithonstepintraditionalknowledgeengineerInExplicit-Knowledge-Firstusecases,ourvisionisthatLLMswillenable,advance,andsimplifycrucialstepsintheknowledgeengineeringpipelinesomuchastoenableKGsatunprecedentedscale,quality,andutility.InParametric-Knowledge-Firstusecases,ourvisionisthatKGswillimprove,ground,andverifyLLMgenerationssoastosignificantlyincreasereliabilityandtrustinLLMusage.approachessomeofwhichfavorParametricKnowledgeFirstblackboxap opicsandRelatedChallengesWiththeopportunitiesandvisionsrelatedtotheavailabilityofbothparametricandexplicitGsKnowledgeExtractionandCanonicalisationomaTradiLLMsandKGs:OpportunitiesandChallengesitedfor4.1.1EntityResolutionandMatchinggiesiesrepresentedbyflatstructureddataHoweverentityGeneralapproaches,suchasCG-MuAlign[203]whichemploysGraphNeuralNetworksLLMsareusedinentityresolutionandlinkingforKGsinmultipleways[7].First,LLMsstep,hinderingtheperformanceofentityalignmentforKGs.Similartohow[146]employsGenerativeAdversarialNetworks(GANs)toreducetheeffortoflabelingdata,wearguethatLLMscanprovidelabeledsamplesofKGsandcontroltheperformancesoftheaforementionedegraphlarlyedbasedones,aswellastheleveragingofLLM-drivenruleandlabeleddataconstruction,canleadtobetterintegrationofLLMswithknowledgereasoning[66].Thelatterintegrationof4.1.2KnowledgeExtractionfromTabularDatawayforKGconstruction.Fortableswhosesemantics(metainformation)arealreadyknown,heuristicrulescanbedefinedandusedtotransformtheirdataintoKGfacts.However,real-JeffJeffZ.Panetal.42:7ofotherpredictionusesBERTextractionSimilarlyRPT[162]usesBERTandGPTtopre-trainatablerepresentationmodel.sBERTwithaTherehavebeenseveralattemptsthatuseLLMsforthesetasksDoduoeRoBERTamodelAlthoughsomeattentionhasbeengiventoutilizingLLMsforTransformingtablecontentsintosequences:Atableoratableelementwithitsstruc-turedcontextneedstobetransformedintoasequencebeforeitcanbefedintoLLMs.Dif-ferenttransformationmethodsarerequiredfordifferentLLMutilizationscenarios,suchasfine-tuningLLMs,LLMinferencewithprompts,andinstructiontuningofLLMs.Representingandutilizingnonetextualtabulardata:AtableoftencontainsnotonlytillfewExtractingtabularknowledge:LLMsaremostlyappliedtoprocessandunderstandtablesntoGPTwhichextractsinstancesfromtextstopopulateanontologyusingChatGPTisknown,butthereareno4.1.3KnowledgeExtractionfromTextForexample,(1)NamedEntityRecognition(NER)andEntityLinking,asmentionedinstextandlinkingmoreinSectionthemtoKGsRelationextractionotenceofexplicittrainingonspecificdomains,therebyimprovingtheirversatilityandadaptability.LLMsandKGs:OpportunitiesandChallengesxtstiesinthefieldofNLP.LLMscangeneratehigh-qualitysyntheticdata,whichcanthenbeusedtofine-tuneakessaryHoweveritisionalresourcerequirementsfortrainingandEfficientextractionfromverylongdocuments.CurrentLLMsareincapableofpro-High-coverageinformationextraction.AlmostallextractionpipelinesfocusonhighgeexMsforKGsKnowledgeGraphConstructionanLLMAsaconclusionofthissection,wediscussthechallengesofLLM-basedmethodsforKG4.2.1LinkPredictionKGlinkpredictionmethodshavemostlybeenstudiedforstaticsnapshotsofaKG.Manyapproaches,infact,assumeatrainingphaseinwhichthecurrentstateoftheKGisusedtoproducts.Inductivelinkprediction(ILP),incontrast,focusesontechniquesthatcanpredictlinkstonewentitiesnotoriginallycontainedinaKG.Furthermore,existingKGembedding-JeffJeffZ.Panetal.42:9initialentityembedepresentationsatwordsentenceanddocumentlevelsmergingthemwiththeILPsetting.TostillpredictentitieswithinILP,existingworksthatcombinetextembeddingsforKGCompletion.Forexample,KG-BERT[187]representsatripleasanorderedsequenceofheadtext,includingsurfaceform,descriptions,andattributes,relationtext,tailtextsep-aratedwithspecialtokens.KG-BERT[187]optimizestheBERTmodelonKGs,followedbyKGGPTthatfine-tunesGPT-2model.MTL-KGC[84]enhancestheeffectivenessofKG-BERTbycombiningpredictionandrelevancerankingtasks.PKGCevaluatestriplevaliditybyEMKGCMEMKGCwithmodulesEntityDescriptionPrediction(EDP)andIncompleteTriplePredictiontextualencoder.LP-BERT[94]isahybridKGcompletionmethodthatcombinesMLMen-ncodingforfinetuningusingacontrastivethemodeledinmptingsettingtofindsuitablelinksbetweenentitiesthefollowingSectionAswiththepreviousChallengesandOpportunities:LLMsmakeitsignificantlyeasiertojointlyutilizestruc-tureresultsfromLLMs.LLMsandKGs:OpportunitiesandChallengesiderwwellamethodcouldlearntoinfernewtriplesbasedonexistingones.LLMsaretrainedbasedonamassivecorpusthatmightLLMsprovideseveralbenefitsforILPasdemonstratedbythemanyexistingapproaches,inceILPentitiesmayoccurthatarenotheunderlyingKGthetaskisinherentlyreliantonauxiliaryinformationapeiscostlyespeciallyforthoserecentcommercialLLMslikeGPT-4.n4.2.2LLMsforKGs:TripleExtractionfromLLMswtionsralizefromLLMparametersasknowledgeretrieval.BenchmarksandBaselines:LAMAisthefirstbenchmarkdatasettoevaluateknowledgeretrievalinLLMs[132].Relatedworksshowthatknowledgeretrievalthroughpromptsisin-eJeffJeffZ.Panetal.42:11data,notactuallyinferringknowledge.SimilartoLAMA,theexperimentsonamorerecentieselsBiasesinTripleExtractionEvaluation:LLMsmayexhibitvarioustypesofbiases;rep-nnlyedaanassPromptEngineeringforTripleExtraction:Caoetal.[24]proposethreeparadigmsforsaimstocreatepromptsthatefficientlyelicitdesiredresponsesfromLLMsforaspecifictask.suchasGPT-3,togeneratehigh-qualitypromptsforavarietyoftasks.InitialexperimentsonTosummarise,priorworkshaveshownthatrelationalknowledgeiscapturedbyLLMstoLLMsandKGs:OpportunitiesandChallengesChallengesandOpportunities:EntityDisambiguation:EntitydisambiguationisessentialforKGconstructiontoensuredisambiguationpresentsseveralchallengessinceLLMssentitiesExistingLLMsstillmanifestalowlevelofprecisiononlongtailentitiesProvenance:ExtractingfactualknowledgedirectlyfromLLMsdoesnotprovideproven-esitdifficulttoaccountforpotentialbiasesinthedatausedfortrainingProvenancealsoprovidescriticalcontext,withoutwhich,informationcanbemisunderstoodLMsforKGsOntologicalSchemaConstructionJeffJeffZ.Panetal.42:13canalsoberegardedasaKGInthispartweintroducetopicsthatLLMsareappliedtolearn4.3.1ConstraintandRuleMiningfromKGsrenoamissingwnqualityassessmentandvalidation85,136,2].SinceKGscontainhugeamountsofdata,itishattheKGneedstosatisfyatalltimesandthatshouldnotbeviolatedbytheintroductionofnewfactsortheirdeletion.InKGs,rulesandconstraintscanteanyofthemoremploythemtodeleteirr1:hasParent(x,y)⇒hasChild(y,x);r2:hasParent(x,y)⇒3v1,v2∈N|hasBirthYear(x,v1)∧hasBirthYear(y,v2);r3:hasChild(x,y)∧hasBirthYear(x,v1)∧hasBirthYear(y,v2)⇒v1<v2;Wherer1statesthathasChildistheinverseequivalentrelationofhasParent,r2statesthatforeyisprohibitivelydifficultandexpensiveOntheonehandthedomainexperts,whoknowthesemanticsforthedatasetathand,maynothavetheskillsetorthechintheKGisexploitedasasilverstandard[128].Thatis,whilethedataarenotperfect,weformationfrominputtextLLMsandKGs:OpportunitiesandChallengesesbeinterrogatedtoidentifywhetherhasParentbetweentwoentitiesrefersmorelikelytothenbereasoning,rulescanbetranslatedintheformofquestionstotheLLM,e.g.,“ifXischildofYisittruethatthebirthyearofYcomesalwaysbeforethebirthyearofX?”.Thus,anLLMcanonetthechallengearisesinthengeNLPalone,andusingthemdirectlyonlargesetsoffactswithinaKG.WecanthinkofthisgsforARuleMiningCasewithChatGPTLet’shavealookatthefollowingrealexampleaskingChatGPTtogeneraterules.Rule1:Ax,y(child(x,y)∧male(y)→father(y,x))ThisrulestatesthatforallsthefatherofxRule2:Ax,y(biologicalChild(x,y)→father(y,x))ThisrulestatesthatforallrofxRule3:Ax,y(fatherOf(y,x)→father(y,x))Thisrulestatesthatforallindividualsxandy,ifthereexistsa’fatherOf’relationshipbetweenyandx,thenyisthefatherofx.Theserules,expressedinfirst-orderlogic,demonstratehowtherelationshipsbetweenRuleisarulewithhighprobability;andRule3isalsocorrectsincefatherandfatherOfareinverseend,however,weneedtoaligntherulesfromtheLLMtotheexistingknowledgeintheKG,andfeedtheLLMabouttheexistingknowledgeandthecontextoftheKG.BothesJeffJeffZ.Panetal.42:154.3.2OntologyRefinementontologySNOMEDCT1andfoodontologyFoodOn2,alsoincludealotofmetainformationenattachedclassifierOnetheandpredictsthepositiontoplaceinSNOMEDCTomyExploitingLLMsisapromisingdirectionforontologyrefinement,butitstillneedsmuchandtosupportgenerativeLLMslikeLLaMAandGPT-4.OneobviouschallengeisthatthoseExploitingthegraphstructureandlogicsofanontologytogetherwithitstext.tuningwithsamplesconstructedbysometemplates,moreLLMtechniquessuchaspromptlearningandinstructiontuningcouldbeconsidered.CombingsymbolicreasoningwithLLMinference.Symbolicreasoning,suchasconsist-redbyLLMs.OneaspecttoincorporatesymbolicreasoningisconstructingsamplesforLLMzed12//LLMsandKGs:OpportunitiesandChallenges4.3.3OntologyAlignmenthipaconceptclass),anindividual(instance)oraproperty.Traditionalsystems(e.g.,LogMapetyearsThestudyofLLMapplicationinontologyalignmentissimilartoontologyrefinement.Pre-LLMslikeGPT-4havenotbeenappliedinontologyalignment,asfarasweknow,andthegeEvaluatingLLM-basedontologyalignmentsystems.Novelevaluationprotocolswith4.4KGsforLLMs:TrainingandAccessingLLMsInSections4.1to4.3,wediscussedonthreedifferentaspectsonusingLLMsforKGs.InthisctionieusingKGsforLLMsThereareafewdimensionshere.Firstly,KGscanbeusedastrainingdataforLLMs.Secondly,triplesinKGscanbeused4.4.1KGsfor(Pre-)TrainingLanguageModelsedageedgeJeffJeffZ.Panetal.42:17ss4.4.2KGsforPromptConstructionTheattentionreceivedbytheintegrationofKGsandLLMshasgrownrecently.Ontheoneinordertomakeitavailabletotheend-users.Ontheotherhand,thereisthelessexploredhnemalxtsegattributesKhopneighbors)oftheentitiesinordertohelptheLLMstogeneratebetterpredictions[22].aventitiesandrmsinChallengesandOpportunities:ThecurrentresearchinthefieldofKGutilizationforpromptficacycomparedtoLLMsLLMspresentsignificantpotentialforadvancingpromptcreationC1:KGscanbeemployedtoautomaticallyextractandrepresentrelevantknowledgetodbyntsC4:KGscancreatepromptsthataskquestions(e.g.,inferringmissingrelationsinanincom-nktoLLMsandKGs:OpportunitiesandChallengesys−→C1,C2,C3.RQ2:HowcanKGsbeexploitedtodrivethegenerativecapabilitiesofLLMsinorderto1.KGsforHallucinationDetectioninLLMs:ThereliabilityofLLMsisgreatlyaffectedbybleserveasafoundationBycombiningLLMsandKGsresearcherscan2.KGsforEditingKnowledgeinLLMs:LLMspossesstheabilitytostoreextensivereal-3.KGsforBlack-boxLLMsKnowledgeInjection:Whilepre-trainingandknowledgeedit-4.4.3RetrievalAugmentedMethodscreasestrainingLLMsisexpensiveegGPTbeputintoLLMs.JeffJeffZ.Panetal.42:19Oneideatodealwiththeabovelackof(updated)knowledgeistoedittheknowledgeinLLMs.Aobviousstrategyistoretrainandfine-tunethemodelbasedonthemodifieddata.ypernetworktolearnaparametershiftforthebasetytothedecoder.ItisexpectedthatstructuredknowledgewillbethemainsourceofexternalsUnifyingKnowledgeEditingandRetrievalAugmentation:KGscanbeusedforedit-toassistLLMsinretrievalaugmentedmethods.Infact,knowledgeeditingandretrievalAugmentedCounterfactualModel(SERAC),whichstoreseditsinanexplicitmemoryforSemi-parametricLLMs:Thisdirectionishighlyrelatedtothetopicofthispositionpaper.TheideaistomakeuseofexplicitknowledgetoaugmentLLMs.OneofthekeyissueisSupportofComplexReasoning:Canwegobeyondsimplyretrievingexplicitknow-alTheintegrationofKGsandLLMsinaunifiedapproachholdssignificantpotential,astheirceKGsprovideveryaccurateandexplicitknowledge,whichiscrucialforsomeapplicationsi.e.healthcare,whereasLLMshavebeencriticizedfortheirlackoffactualknowledgeleadingtontheirsymbolicreasoningability,areabletogenerateinterpretableresults.Ontheotherhand,KGsLLMsandKGs:OpportunitiesandChallengesThemajorityofKGscapturefactsofthesortonemightencounterinanencyclopediaorinaforAIsystemsForinstancewemaywishforaKGtonotonlycapturethattheCongoCommonsenseKnowledgefromLLMs:Thefirststudytoinvestigateextractingknow-lyw4.5.2DigitalBuildEnvironmentInthedomainofBuildEnvironment,whereitisvitaltodesignandconstructinasafeandeCodeCheckinghasbloomedtosupportthis,butstill,inthepast,itwasquitechallengingforextensiveexpertknowledgeforinterpretationManyapplications3/blog/doctor-ai-a-voice-chatbot-for-healthcare-powered-by-neo4j-and-aws/JeffJeffZ.Panetal.42:21nowinthisareahavecombinedLLMs+KGsinaddressingthesechallengesbyfollowinganLLMsenhancedKGapproach[204].LLMsinterpretthetextintheregulationsandenhanceaKGofrules,whicharefurtherserializedusingdomeDomainSpecificLanguage[195,176].Interrogating3DmodelsmodeledasgraphsusingtheLinkedBuildingDataapproachisanotherchallengeforthedomainasitrequiresskillsetsinquerylanguageslikeSPARQL.LLMsarehisisAISpeckle4.5.3DigitalHealthcareociatedwithemployingLLMsorthetwoAccuracy.SomedemonstrationsofLLMshaveshowcasedimpressivecapabilities.Neverthe-less,therehavealsobeendocumentedinstanceswhereLLMshavemademistakesorexhibitedusage.Whenutilizedtodiagnosehypotheticalpatientcases,LLMshaveexhibitedaccuracyatotowhichLLMs+KGmaycontributeinasignificantmanner.ioneLLMsmustrecognizethattheirdataarepotentiallymoresusceptibletobreachesormisuse.TheritizingprivacyinvolvesrunninganopensourceLLMwithin4https://speckle.systems/blog/ai-powered-conversations-with-speckle/LLMsandKGs:OpportunitiesandChallengesontheparametersofLLMsgiventhepossibilityofhavingcontroloftheinjectedknowledgeeThetworisksdescribedaboveleadtolikewisechallengesthatmustbetackledinorderto4.5.4DomainSpecificContentSearchGPTinamultitudeofNLPapplicationsthatinvolvemultiplemodalitiesanddomainspecificadaptations.WhileLLMsarenscanectorembeddingsforimagesandtextsuchthattheembeddingoftheactionsorpositioningrelationships)asedges.Therefore,MultimodalLLMcanbetrainedtographareconnectedtoabackgroundtaxonomyorKG,thenquestionsthatrequireformsofabstractionandcomputation,e.g.,Whatisthegenusofthelargestanimalinthepicture?Whatistheaveragepriceofthecarinthepicture?Retrieveimagesdepictingkidswearingveganfriendlyitemsofclothinginthecatalog.rmspecificapplicationsofNLPandLLMmethods2,186].Yet,theydonothaveaccesstoall5E.g.,https://haystack.deepset.ai/JeffZJeffZ.Panetal.1OshinAgarwal,HemingGe,SiamakShakeri,etal.42:23ficmanyweeksareleftbeforetheexpirationofthiscontract?Whataretheconflictsbetweenthisliabilityclauseandpreviouscontractsmycompanysigned?Whatistheriskofside-effectsofthistreatmentforapatientwiththisgivenhealthrecord?GettheaverageyearlyyieldandspreadforcompetitorcompaniesofACorpandtheirrecentacquisitionsintheEUmarket. 5OutlooknitiesforfutureresearchThecommunityisalreadyrapidlyadaptingtheirresearchfocus,withnovelforumsliketheKBC-LM1.Don’tthrowouttheKGwiththeparadigmshift:Forarangeofreliabilityorsafety-criticalwhichKGsandLLMscanfertilizeeachother.KGsareheretostay,donotjustditchthem2.Murderyour(pipeline)darlings:LLMshavesubstantiallyadvancedmanytasksintheKGnlyeoftheart3.Staycurious,staycritical:LLMsarearguablythemostimpressiveartifactofAIresearchofinthepublicaswellasintheresearchliterature,andoneshouldretainahealthydoseofourneyTheadvancestriggeredbyLLMshaveuprootedetoorKnowledgegraphbasedsyntheticcorpusgenera-tionforknowledge-enhancedlanguagemodelpre-training.InNAACL,pages3554–3565,June2021.2NaserAhmadi,Viet-PhiHuynh,VamsiMeduri,StefanoOrtona,andPaoloPapotti.Miningex-pressiverulesinknowledgegraphs.J.DataandInformationQuality,12(2),may2020.3MirzaMohtashimAlam,MdRashadAlHasanRony,MojtabaNayyeri,KarishmaMohiuddin,MSTMahfujaAkter,SaharVahdati,andJensLehmann.Languagemodelguidedknowledgegraphembeddings.IEEEAccess,10:76008–76020,4DimitriosAlivanistos,SeleneBáezSantamaría,MichaelCochez,Jan-ChristophKalo,EmilevanKrieken,andThiviyanThanapalasingam.Prompt-Krieken,andThiviyanThanapalasingam.Prompt-ingasprobing:Usinglanguagemodelsforknow-ledgebaseconstruction.LM-KBC,2022.5BadrAlKhamissi,MillicentLi,AsliCelikyilmaz,MonaDiab,andMarjanGhazvininejad.Areviewonlanguagemodelsasknowledgebases.arXiv,6MonaAlshahrani,MohammadAsifKhan,OmarMaddouri,AkiraRKinjo,NúriaQueralt-Rosinach,andRobertHoehndorf.Neuro-symbolicrepresentationlearningonbiologicalknowledgegraphs.Bioinformatics,33(17):2723–2730,047SihemAmer-Yahia,AngelaBonifati,LeiChen,GuoliangLi,KyuseokShim,JianliangXu,andXiaochunYang.Fromlargelanguagemodelstodatabasesandback:Adiscussiononresearchandeducation.DASFAA,abs/2306.01388,2023.8AlejandroBarredoArrieta,NataliaDíazRodríguez,JavierDelSer,AdrienBennetot,SihamTabik,A.Barbado,SalvadorGarcía,SergioGil-Lopez,DanielMolina,RichardBenjamins,RajaChatila,andFranciscoHerrera.Explainableartificialintelligence(XAI):Concepts,taxonomies,opportunitiesandchallengestowardresponsibleai.InformationFusion,2020.9SörenAuer,ChristianBizer,GeorgiKobilarov,JensLehmann,RichardCyganiak,andZacharyIves.Dbpedia:Anucleusforawebofopendata.InThesemanticweb,pages722–735,2007.10StephenHBach,VictorSanh,Zheng-XinYong,AlbertWebson,ColinRaffel,NihalVNayak,Ab-heeshtSharma,TaewoonKim,MSaifulBari,ThibaultFevry,etal.Promptsource:Aninteg-rateddevelopmentenvironmentandrepositoryfornaturallanguageprompts.ACL,2022.11ParishadBehnamGhader,SantiagoMiret,andSivaReddy.Canretriever-augmentedlanguagemodelsreason?theblamegamebetweenthere-trieverandthelanguagemodel.InarXiv,2022.12EmilyMBender,TimnitGebru,AngelinaMcMillan-Major,etal.Onthedangersofstochasticparrots:Canlanguagemodelsbetoobig?InFAT,pages610–623,2021.13EmilyM.BenderandAlexanderKoller.ClimbingtowardsNLU:Onmeaning,form,andunderstand-ingintheageofdata.InACL,2020.14RussaBiswas,HaraldSack,andMehwishAlam.Madlink:Attentivemultihopandentitydescrip-tionsforlinkpredictioninknowledgegraphs.SWJ,pages1–24,2022.15RussaBiswas,RadinaSofronova,MehwishAlam,andHaraldSack.Contextuallanguagemodelsforknowledgegraphcompletion.InMLSMKG,2021.16SuLinBlodgett,SolonBarocas,HalDaum’e,andHannaM.Wallach.Language(technology)ispower:Acriticalsurveyof“bias”innlp.ACL,17BerndBohnet,VinhQTran,PatVerga,RoeeAharoni,DanielAndor,LivioBaldiniSoares,JacobEisenstein,KuzmanGanchev,JonathanHerzig,KaiHui,etal.Attributedquestionanswering:Evaluationandmodelingforattrib-utedlargelanguage