LLM辅助知识图谱工程 ChatGPT实验 -LLM-assisted Knowledge Graph Engineering -Experiments with ChatGPT

arXiv:2307.06917v1[cs.AI]13Jul2023

LLM-assistedKnowledgeGraphEngineering:

ExperimentswithChatGPT

Lars-PeterMeyer1,2,3[0000—0001—5260—5181],

ClausStadler1,2,3[0000—0001—9948—6458],JohannesFrey1,2,3[0000—0003—3127—0815],

NormanRadtke1,2[0000—0001—9155—8920],KurtJunghanns1,2[0000—0003—1337—2770],

RoyMeissner2,3[0000—0003—4193—8209],GordianDziwis1[0000—0002—9592—418X],

KirillBulert1,2[0000—0002—1459—3754],andMichaelMartin1,2[0000—0003—0762—8688]

1InfAIe.V.Leipzig,Germany,lpmeyer@,

2AKSWresearchgroup,

3LeipzigUniversity,Germany,

https://www.uni-leipzig.de

Abstract.KnowledgeGraphs(KG)provideuswithastructured,ex-ible,transparent,cross-system,andcollaborativewayoforganizingourknowledgeanddataacrossvariousdomainsinsocietyandindustrialaswellasscientiicdisciplines.KGssurpassanyotherformofrepresenta-tionintermsofefectiveness.However,KnowledgeGraphEngineering(KGE)requiresin-depthexperiencesofgraphstructures,webtechnolo-gies,existingmodelsandvocabularies,rulesets,logic,aswellasbestpractices.Italsodemandsasigniicantamountofwork.

Consideringtheadvancementsinlargelanguagemodels(LLMs)andtheirinterfacesandapplicationsinrecentyears,wehaveconductedcom-prehensiveexperimentswithChatGPTtoexploreitspotentialinsup-portingKGE.Inthispaper,wepresentaselectionoftheseexperimentsandtheirresultstodemonstratehowChatGPTcanassistusinthede-velopmentandmanagementofKGs.

Keywords:ChatGPT·knowledgegraphengineering·RDF·largelan-guagemodelusecases·AIapplication.

1Introduction

Inthelastyears,ArtiicialIntelligence(AI)hasshowngreatpromiseinimprov-ingorrevolutionizingvariousieldsofresearchandpractice,includingknowledgeengineering.TherecentbigleapinAI-basedassistantchatbots,likeChatGPT(GenerativePre-trainedTransformer)model,hascreatednewopportunitiestoautomateknowledgeengineeringtasksandreducetheworkloadonhumanex-perts.Withthegrowingvolumeofinformationindiferentields,theneedforscalableandecientmethodstomanageandextractknowledgefromdatathatalsoadapttonewsourcesiscritical.Despitetheadvancesinresearchw.r.t.(semi)automation,knowledgeengineeringtasksstillrelyvastlyonhumanex-perts.Ononehand,thisprocesscanbetime-consuming,resource-intensive,andsusceptibletoerrors.Ontheotherhand,therelianceonhumanexpertisein

2L.-P.Meyeretal.

knowledgeengineeringexposesittoworkforceshortages(asknowledgeengineersarescarceandthedemandisgrowing)andtheriskofexpertiseloss.Thesefac-torscanimpacttheresilienceandsustainabilityofsystemsandoperationsthatrelyonknowledgeengineering.AI-basedassistantbotapproaches,suchasChat-GPT,couldbridgethisgapbyprovidingauniiedtoolfortasksinknowledgeengineering,toreducetheworkloadofknowledgeengineersthemselves,butalsomakeknowledgeengineeringmoreaccessibletoabroaderaudience.ChatGPT,inparticular,hasshownpromiseingeneratingresponsesinavarietyofsyntac-ticalrepresentations(includingcodeandmarkuplanguages)touserqueriesortaskdescriptionswritteninnaturallanguage.

Inthispaper,wediscussandinvestigatethepotentialofChatGPTtosup-portorautomatevariousknowledgeengineeringtasks(e.g.ontologygeneration,SPARQLquerygeneration).Wewillexplorethebeneits,pitfallsandchallengesofusingitandidentifypotentialavenuesforfutureresearch.

2RelatedWork

ChatGPT,aLargeLanguageModel(LLM)publishedbyOpenAI

4

,raisedtheinterestinthebroadieldofMachineLearning(ML)

5

andespeciallyLLMs

[4]

onabroadscale.WhiletherearecurrentdiscussionsandanalysisonthecapabilitiesofLLMslikeChatGPTingeneral(e.g.

[1]),

thereislittleintheareaofknowledge

graphengineering.Ekaputraetal.

[3]

givesageneraloverviewofcurrentresearchonthecombinationofthebroadieldofMLandsemanticweb.

SearchingGoogleScholarandSemanticScholarwith”knowledgegraphChat-GPT”,”ontologyChatGPT”and”rdfChatGPT”inthebeginningofApril2023resultsinonlytworelevantpapers.Theirstone,

[7],

reviewsthediferencesbetweenconversationalAImodels,prominentChatGPT,andstate-of-the-artquestion-answeringsystemsforknowledgegraphs.Intheirsurveyandexperi-ments,theydetectcapabilitiesoftheirusedframeworksbuthighlightChatG-PTsexplainabilityandrobustness.Thesecondone,

[6],

discussestheusageofChatGPTfordatabasemanagementtaskswhentabularschemaisexpressedinanaturallanguage.TheyconcludeamongothersthatChatGPTisabletoassistincomplexsemanticintegrationandtablejoinstosimplifydatabasemanage-mentandenhanceproductivity.TheappliedapproachesandresultsofthesetwopapersindicatethattheideaofusingLLMslikeChatGPTintheieldofKGengineeringisencouragingandthattheLLMsmightassistKGengineersintheirworklows.Still,theresearchontheusageofLLMsforknowledgegraphengineersisscarceandseemstobeanewresearcharea.

Thereexistsomenon-andsemi-scientiicresourceswhichrenderthetopicfromapracticalandexperienceperspective.WewanttohighlighthereahelpfulblogpostbyKurtCagle

[2]

onChatGPTfor”knowledgegraphworkers”andablogpostbyKonradKalicinski

[5]

onknowledgegraphgenerationinNeo4JassistedbyChatGPT.

4

/blog/chatgpt

5

/wp-content/uploads/2023/04/HAI

AI-Index-Report2023.pdf

LLM-assistedKnowledgeGraphEngineering:ExperimentswithChatGPT3

3LLM-AssistedKnowledgeGraphEngineering-PotentialApplicationAreas

IndiscussionroundswithknowledgegraphengineeringexpertsweidentiiedthefollowingpreliminarylistofpotentialusecasesinthedomainofknowledgegraphengineeringapplicabletoLLMsassistance:

–Assistanceinknowledgegraphusage:

GenerateSPARQLqueriesfromnaturallanguagequestions(relatedex-

perimentinSection

4.1

andSection

4.3

)

Explorationandsummarizationofexistingknowledgegraphs(related

experimentinSection

4.5

)

ConversionofcompetencyquestionstoSPARQLqueries

Codegenerationorconigurationoftool(chain)sfordatapipelines–Assistanceinknowledgegraphconstruction

Populatingknowledgegraphs(relatedexperimentinSection

4.4

)and

viceversa

Creationorenrichmentofknowledgegraphschemas/ontologiesGethintsforproblematicgraphdesignbyanalysingChatGPTusages

problemswithaknowledgegraph

Semanticsearchforconceptsorpropertiesdeinedinotheralreadyex-

istingknowledgegraphs

Creationandadjustmentofknowledgegraphsbasedoncompetency

questions

Giventhelimitedspaceofthispaper,weevaluateasubsetoftheapplicationareaswithexperimentsinthefollowingsection.

4Experiments

ToevaluatethecapabilitiesofLLMsattheexampleofChatGPTforassistingwithknowledgegraphengineering,wepresentseveralexperimentsandtheirre-sults.Furtherdetailsaboutthemisgiveninthe

SupplementalOnlineResources

.MostexperimentswereconductedwithChatGPTwiththeLLMGPT-3.5-turbo

6

(namedChatGPT-3fromhereon),someadditionallywithChatGPTwiththeLLMGPT-4

7

(namedChatGPT-4fromhereon).

4.1SPARQLQueryGenerationforaCustomSmallKnowledgeGraph

Forairstevaluation,wedesignedasmallknowledgegraphasshowninListing

1

.Speciically,wewantedtoknowwhether(1)GPTcanexplainconnectionsbetweenindirectlyrelatedentities,(2)createSPARQLqueriesoverthegivenmodeland(3)reconstructthemodelifallpropertiesandclasseswererelabelled.

6

/docs/models/gpt-3-5

7

/docs/models/gpt-4

4L.-P.Meyeretal.

1:anneafoaf:Person;foaf:firstName"Anne";foaf:surname"Miller";

2vcard:hasAddress[avcard:Home;vcard:country-name"UK"].

3:bobafoaf:Person;foaf:firstName"Bob";foaf:surname"Tanner";

4vcard:hasAddress[avcard:Home;vcard:country-name"US"].5:wonderOrgaorg:Organization.

6:researchDepaorg:OrganizationalUnit;org:unitOf:wonderOrg;

7rdfs:label"ResearchDepartment".

8:marketingDepaorg:OrganizationalUnit;org:unitOf:wonderOrg;

9rdfs:label"MarketingDepartment".

10:chiefResearchOfficeraorg:Role.:marketingManageraorg:Role.

11[aorg:Membership;org:member:anne;org:organization:researchDep;

12org:role:chiefResearchOfficer].

13[aorg:Membership;org:member:bob;org:organization:marketingDep;

14org:role:marketingManager].

Listing1:AnorganizationalKGwithtwopeopleworkingindiferentdepart-mentsofthesameorganization.

Weissuedthefollowingprompt,whichincludestheknowledgegraphfromListing

1

,onChatGPT-3andChatGPT-4:

Prompt1:GiventheRDF/Turtlemodelbelow,arethereanyconnectionsbetweenUSandUK?<rdf-model>

IntheknowledgegraphofListing

1

,thereisaconnectionbetweenthetwocountriesviathetwopeoplelivinginthese,whichgotajobindiferentdepart-mentsofthesamecompany.WhileChatGPT-3failstoidentifythisrelation,ChatGPT-4successfullyidentiiesitinallcases.

WefurtheraskedbothChatGPTmodelswithprompt

2

andreceivediveSPARQLquerieseach,whichweanalysedfortheirsyntacticcorrectness,plau-siblequerystructure,andresultquality.Theresultsforprompt

2

arelistedintable

1

andshowthatbothmodelsproducesyntacticallycorrectqueries,whichinmostcasesareplausibleandproducecorrectsresultsin3/5(ChatGPT3)and2/5(ChatGPT4)cases.

Prompt2:GiventheRDF/Turtlemodelbelow,createaSPARQLquerythatlistsforeverypersonthecountry,companyanddepartmentandrole.Pleaseadherestrictlytothegivenmodel.<rdf-model>

Inessence,AI-basedquerygenerationispossibleanditcanproducevalidqueries.However,theprocessneedsresultvalidationintwodimensions:1)val-idatingthequeryitselfbymatchingtostaticinformation,likeavailableclassesandpropertiesinthegraph,aswellas2)validatingtheexecutedqueryresultstoletChatGPTgeneratenewqueriesincaseofemptyresultsetsinordertoindworkingqueriesinatry&errorapproach.

LLM-assistedKnowledgeGraphEngineering:ExperimentswithChatGPT5

Table1.FindingsingeneratedSPARQLqueriesforprompt

2

.

ChatGPT-3ChatGPT-4

syntacticallycorrect

5/5

5/5

plausiblequerystructure

4/5

3/5

producingcorrectresult

3/5

2/5

usingonlydeinedclassesandproperties

correctusageofclassesandproperties

correctpreixforthegraph

3/5

5/5

5/5

4/5

5/5

4/5

AsalastpromptontheknowledgegraphfromListing

1

,wecreatedaderivedRDFgraphbyrelabellingallclassesandpropertieswithsequentiallynumberedIRIsintheexamplenamespace,likeeg:prop1andeg:class2.Giventherelabelledmodel,wetaskedChatGPT:

Prompt3:GiventheRDF/Turtlemodelbelow,pleasereplaceallprop-ertiesandclasseswiththemostlikelystandardones.<rdf-model>

WithChatGPT-3only2/5iterationssucceededincarryingoutallsubstitu-tions.Inthosesucceedingcases,thequalitywasstillnotasexpectedbecauseoflimitedontologyreuse:OnlyIRIsintheexamplenamespacewereintroduced,ratherthanreusingthefoaf,vcard,andorgvocabularies.Yet,thead-hocproper-tiesandclasseswerereasonablynamed,suchaseg:hrstName,eg:countryNameoreg:departmentName.Incontrast,ChatGPT-4deliveredbetterresults:Allclassesandpropertiesweresubstitutedwiththosefromstandardvocabularies-foaf,vcard,andorgwerecorrectlyidentiied.Forsomeiterations,ChatGPT-4usedthevocabularyinsteadoftheorgvocabularyasanalternativeap-proach.

4.2TokenCountsforKnowledgeGraphsSchemas

AftertheresultswiththesmallcustomknowledgegraphwewantedtocheckthesizeofsomewellknownknowledgegraphswithrespecttoLLMs.

TheLLMsbehindChatGPTcanhandleatthemomentonly4096tokens(GPT-3.5

6

)or8192respective32,768tokensforGPT-4

7

.

Wecountedtokensforvariouspublicknowledgegraphsindiferentserializa-tionformatswiththelibrarytiktoken

8

asrecommendedforChatGPT.Table

2

liststhetokencountsforacoupleofcombinationsorderedbytokencount.Moredataandinformationisavailableinthe

SupplementalOnlineResources

.Theturtleserializationseemtoresultinminimaltokencount,butisstillbiggerthanthesimilarSQLschemaaddedforcomparison.Allknowledgegraphsexceedthe

tokenlimitforGPT-3.5and3of4knowledgegraphslistedhereexceedthelimitforGPT-4.

8

/openai/tiktoken

6L.-P.Meyeretal.

Table2.Tokencountsforselectedknowledgegraphsandserialisations

GraphSerialisationTypeTokenCount

MondialOracleDBschemaMondialRDFschema

SQLschema

turtle

2,608token

5,339token

MondialRDFschemaMondialRDFschemaMondialRDFschemaMondialRDFschemaWineOntologyWineOntologyPizzaOntologyPizzaOntology

DBpediaRDFschemaDBpediaRDFschema

functionalsyntax9,696token

manchestersyntax11,336token

xml/rdf17,179token

json-ld47,229token

turtle13,591token

xml/rdf24,217token

turtle5.431token

xml/rdf35,331token

turtle471,251token

xml/rdf2,338,484token

Table3.Findingsingeneratedsparqlqueriesforprompt

4

.

ChatGPT-3ChatGPT-4

syntacticallycorrectplausiblequerystructureproducingcorrectresult

5/52/50/5

5/5

4/5

0/5

usingonlydeinedclassesandproperties

correctusageofclassesandproperties

correctpreixformondialgraph

1/5

0/5

0/5

3/5

3/5

1/5

4.3SPARQLQueryGenerationfortheMondialKnowledgeGraph

Inadditiontotheexperimentswiththesmallcustomknowledgegraph(seeSection

4.1

)wetestedChatGPTwiththebiggermondialknowledgegraph

9

whichispublishedsincedecadeswiththelatest”mainrevision”2015.

WeaskedChatGPTtogenerateaSPARQLqueryforanaturallanguagequestionfromasparqluniversitylecture

10

.Weusedthefollowingpromptive

timeswithChatGPT-3andChatGPT-4each:

Prompt4:Pleasecreateasparqlquerybasedonthemondialknowledge

graphforthefollowingquestion:whichriverhasthemostriparianstates?

Theresultsaredocumentedinthe

SupplementalOnlineResources

togetherwithdetailedcommentsonthegivenqueries.Table

3

givessomestatistics.Insummary,allSPARQLqueriesgivenbyChatGPTweresyntacticallycorrect,butnoneofthemworkedwhenexecuted.Actuallyallquerieshadatleastoneerrorpreventingthecorrectexecutionlikereferencingawrongnamespace,wrongusageofpropertiesorreferencingundeinedclasses.

9

rmatik.uni-goettingen.de/Mondial

10

rmatik.uni-goettingen.de/Teaching/SWPr-SS20/swpr-1.pdf

LLM-assistedKnowledgeGraphEngineering:ExperimentswithChatGPT74.4KnowledgeExtractionfromFactSheets

Asanexperimenttoevaluateknowledgeextractioncapabilities,weusedPDFfactsheetsof3Dprinterspeciicationsfromdiferentadditivemanufacturing(AM)vendorwebsites.ThegoalistobuildaKGaboutexisting3Dprintersandtheirtypeaswellascapabilities.Wefedplaintextexcerpts(extractedviapdfplumber)fromthesePDFsintoChatGPT-3andprompteditto:

Prompt5:Convertthefollowing$$vendor$$3dprinterspeciicationintoaJSONLDformattedKnowledgeGraph.ThenodeforthisKGshouldbePrinterasamainnode,Typeof3dprintersuchasFDM,SLA,andSLS,Manufacturer,Material,Applications,andTechnique.

Sincethefactsheetsareusuallyformattedusingatablescheme,thenatureoftheseplaintextsisthatmostlytheprinterentityismentionedinthebeginningofthetextwhichthenisfurthercharacterizedinakey-valuestyle.Asaresult,thetexttypicallydoesnotusefullsentencesandcontainsonlyoneentitythatisdescribedindetail,butseveraldependantentities(likeprintingmaterials).However,theformatofthekey-valuepairscanbenoisy.Keynamescanbeseparatedwithcolons,newlinefeeds,orincontrastmultiplekey-valuepairscanbeinthesameline,whichcouldimposeachallenge.Nevertheless,ChatGPTwasabletoidentifythekey-valuepairsoftheevaluationdocumentinareliablyway.Unfortunately,itdeliveredoutof5testrunsforthisdocument4partialand1completeJSONdocument.Inspiteofthat,wesummarizeirstinsightsgainedfromaknowledgeengineeringperspective(butforthesakeofbrevity,werefertotheoutputdocumentsintheexperimentsupplements)

–TheJSON-LDoutputformatprioritizesusageofvocabularyinthe5evaluationruns.Thisworksgoodforwell-knownentitiesandproperties(e.g.Organization@typeforthemanufacturer,orthenameproperty),how-ever,fortheAM-speciicfeaturekeynamesortermslikeprinterChatGPT-

3inventsreasonablebutnon-existentpropertynames(inthenamespace)insteadofaccuratelycreatinganewnamespaceorusingaded-icatedAMontologyforthatpurpose.

–Requestingturtleasoutputformatinstead,leadstodiferentresults.E.g.thepropertynamespacepreixisbasedontheprinterIDandthereforeprinterdescriptionsarenotinteroperableandcannotbequeriedinuni-iedwayinajointKG.

–Successfullysplittingx,yandzvaluesofthemaximumprintdimension(in-steadofextractingalldimensionsintoonestringliteral)worksin3runs.AlthoughChatGPT-3accuratelyappendstheunitofmeasurementtoallx,y,zvalues(whichisonlymentionedafterthezvalueintheinput)inthosecases,thisisamodellinglaw,asqueryingtheKGwillbemorecomplex.Inonerunitaddressedthisissuebyseparatingunitsintoaseparateunitcodeield.

8L.-P.Meyeretal.

–Asimilarefectwasobservedwhenitcomestomodellingthedependenten-tities.E.g.,in4runs,themanufacturerwasmodelledcorrectlyasaseparatetypedentity,in1asstringliteralinstead.

Asageneralconclusionoftheexperiment,ChatGPT-3hasoverallsolidskillstoextractthekeyvaluepairsfromthesheets,butthecorrectmodellingorrep-resentationintermsofaKGsigniicantlyvariesfromruntorun.Subsequently,noneofthegeneratedJSONdocumentscontainedsucientinformationontheirown,butonlyasubsetthatwasmodelledaccurately.Aquestionforfuturere-searchiswhethercherrypickingofindividualJSONelementsfromoutputsofseveralrunsandcombiningthemintooneinaldocumentoriterativelyreiningtheoutputbygivingChatGPTgenericmodellingfeedback(likeuseanontology,orseparateunitinformation,etc.)canbeautomatedinagoodandscalableway.

4.5KnowledgeGraphExploration

ExpertsintheieldofknowledgegraphsarefamiliarwithconceptsfromRDFSchema(RDFS)(domain/range,subPropertyOf,subClassOf)andWebOntol-ogyLanguage(OWL)(ObjectProperty,DatatypeProperty,FunctionalProperty,...).Often,eachoftheseexpertshastheirpreferredtoolsandmethodsforgaininganoverviewofanontologytheyarenotyetfamiliarwith.WeaskedChatGPT-3twodiferentquestionsrequestingthemermaid

11

visualizationofthemostimportantconceptsandtheirconnections:

Prompt6:Canyoucreatemeavisualizationshowingthemostimportantclassesandconceptsandhowtheyarelinkedfordbpediaontology,serializedformermaid?

Prompt7:CanyoucreatemeavisualizationofthemostcommonconceptsoftheDBpediaontologyandtheirconnectionsfocusingondomainandrangedeinedinproperties.

Weexpectedagraphwithatleasteightnodesandtheircorrespondingedges.TheidentiiersforthenodesandedgesareexpectedtofollowtheTurtleorSPARQLprefix:conceptnotation.Iftheirstquestiondidnotachievethegoal,weaskedadditionalquestionsordemandstoChatGPT-3.Theresultsarepresentedintable

4

andweevaluatedthedisplayedgraphsbasedonthefollowingcriteria:

Prompt

6

ledtoananswerwithahierarchicalgraphrepresentationoftheimportantclassesdeinedintheDBpediaontology.Thediagramalreadymetourrequirementsregardingtheminimumnumberandlabellingaftertheirstanswerandcanbeseeninthe

SupplementalOnlineResources

.

Theclasshierarchywasrepresentedbytherdfs:subPropertyOfrelation,andthenodeswerelabelledinpreixnotation,asweretheedges.Byarranging

11“...aJavaScript-baseddiagrammingandchartingtool...”

/

9

LLM-assistedKnowledgeGraphEngineering:ExperimentswithChatGPT

Table4.Diagramcontentoverview.

Prompt

6

Prompt

7

MermaidType

LabelsofNodes

LabelsofEdges

NumberofNodes(total/existing/dbo)NumberofEdges(total/unique)

graphgraph*preixandconceptpreixandconcept**preixandconceptpreixandconcept**

10/10/8

12/2

13/12/12

17/17

*Onemorepromptwasneededtoserializeagraph

**Onemorepromptwasneededtoaddpreixedlabels

itasatreeusingthesubClassOf-pattern,onlytwodiferentpropertieswereusedfortherelations(edges).Therootnodewasoftypeowl:Thingothernodesareconnectedas(sub)classesfromtheDBpediaontology.Thesewere:Place,Organi-zation,Event,Work,Species,andPerson.TheclassWorkhadonemoresubClas-sOfrelationtotheclassMusicalWork.TheclassPersonhadthemostcomplexrepresentation,withtwomoresubClassOfrelationsleadingtofoaf:Personandfoaf:Agent,thelatterofwhichisasubclassoftherootnode(owl:Thing).

Inthesecondprompt(Prompt

7

ChatGPT-3referredtoagraphicilewithintheanswertextthatnolongerexisted.Uponfurtherinquiry,amermaiddia-gramwasgenerated.Itwasoftype”Graph”andcontainedthirteencommonconceptsandseventeenedges,whichwereallunique.Thelabelsofboth,nodesandedgescontainnopreixes,butwereaddablewithfurtherinquiry.Onlythegeneratedconceptdbo:Occupationisnon-existent.Allremainingnodesandedgescomplywiththerulesoftheontology,eveniftheconceptsusedarede-rivedthroughfurthersubclassrelationships.Theresultingdiagramisshowninthe

SupplementalOnlineResources

.

Whileprompt

6

leadstoaresultthatcanbemorecomprehensivelyachievedwithconventionaltoolsforvisualizingRDF,theresultfromprompt

7

providesanoverviewofconcepts(classes)andpropertiesthatcanbeusedtorelateinstancesoftheseclassestoeachother.

5ConclusionandFutureWork

Fromtheperspectiveofaknowledgegraphengineer,ChatGPThasdemon-stratedimpressivecapabilities.Itsuccessfullygeneratedknowledgegraphsfromsemi-structuredtextualdata,translatednaturallanguagequestionsintosyn-tacticallycorrectandwell-structuredSPARQLqueriesforthegivenknowl-edgegraphs,andevengeneratedoverviewdiagramsforlargeknowledgegraphschemas,asoutlinedinsection

4

.Andetailedanalysisrevealedthatthegener-atedresultscontainmistakes,ofwhichsomearesubtle.Forsomeusecases,thismightbeharmlessandcanbetackledwithadditionalvalidationstepsingeneral,

likewiththemetricsweusedforSPARQLqueries.Ingeneral,ourconclusionis,

thatoneneedstokeepinmindChatGPT’stendencytohallucinate

12

,especially

12Generationofcontentwithoutanyfoundation

10L.-P.Meyeretal.

whenappliedtotheieldofknowledgegraphengineeringwheremanyengineersareusedtomathematicalprecisionandlogic.

Theclosed-sourcenatureofChatGPTchallengesscientiicresearchonitintwoways:1.Detailedcapabilityratingsofclosed-sourceprobabilisticmodelsrequiremuchefort2.Resultreproducibilityisboundtoserviceavailabilityandresultsmightbeirreproducibleatalaterdate(duetoservicechanges)Thus,opentrainingcorporaandLLMSaremandatoryforproperscientiicresearch.

Inthefuture,metricsaretobefoundtorategeneratedChatGPTanswersautomatically,likewebroachedwithSPARQLqueries.Thisagainenablestoex-tendthenumberoftestcasesforaspeciicexperimentandtogenerateprofoundstatisticalresults.AnotherresearchfocusshouldbegiventomethodsthatlettheLLMaccessabroader/necessarycontexttoincreasethechanceforcorrectanswers.

AcknowledgementsThisworkwaspartiallysupportedbygrantsfromtheGermanFederalMinistryforEconomicAfairsandClimateAction(BMWK)totheCoyPuproject(01MK21007A)andKISSproject(01MK22001A)aswellasfromtheGermanFederalMinistryofEducationandResearch(BMBF)totheprojectSt