2026年大模型与生成式AI面试与工程实践手册（英文版）

2026

LanguageModelsInterviewHandbook

151InterviewQuestions,FoundationRoadmaps,PythonExamples,

ArchitectureDiagrams,andProductionPlaybooksforModernLLMand

GenAIRoles

LamhotSiagian

AIEngineeringInsider

Copyright

Copyright©2026LamhotSiagian.

Imprint:AIEngineeringInsider.

Allrightsreserved.

Thishandbookisintendedforeducationalandprofessionalinterview-preparationuse.Itiswrittenasacompacttechnicalreferenceforengineers,researchers,students,andpractitionersworkingwithlargelanguagemodelsandretrieval-centeredAIsystems.

LanguageModelsInterviewHandbookLamhotSiagian»

—iii—AIEngineeringInsider

Preface

Largelanguagemodelsareoftenintroducedeitherasintimidatingresearchartifactsorasmagicproductivitytools.Neitherframinghelpsmuchinarealinterview.Hiringpanelswantcandidateswhocanexplainhowtokenization,attention,retrieval,prompting,fine-tuning,anddeploymentactuallyworktogetherunderproductionconstraints.Thishandbookwasrevisedtomeetthatneeddirectly.

Thebookisnoworganizedacrosssixteenchaptersandonehundredfifty-oneinterviewquestions,withastrongeremphasisonfoundations,careerroadmapframing,architecturediagrams,premiumchaptersummaries,codewalkthroughs,andinterviewpositioning.ThenewopeningchapterestablisheswhatanLLMis,howthefieldisevolving,howtosequenceyourlearning,andhowtopositionyourselfforGenAIroles.Thenextchaptersbuildthetechnicalfoundations:tokens,embeddings,attention,pretraining,andmodelfamilies.Middlechaptersmoveintoclassification,themediscovery,retrieval,RAG,andprompting.Laterchapterscovermultimodalsystems,embeddingoptimization,PEFT,trainingmath,decoding,serving,andproductiondeployment.

Eachchapternowincludestwodeliberateinterviewaids.InterviewAnchorsectionsexplainwhatastrongcandidateshouldemphasizewhenansweringaloud.INTERVIEWCHEAT-SHEETpanelsconvertthatintocompacttalkingpoints,trade-offs,andredflagsthatareeasytoreviewbeforeascreen,onsite,ortake-homediscussion.

Thegoalofthishandbookisnotmemorizationforitsownsake.Thestrongergoalistohelpyousoundlikeanengineerwhocanreasonfromfirstprinciples,choosetherighttoolfortheworkload,articulatefailuremodes,andjustifytrade-offswithclarity.Thatisthedifferencebetweenrecitingterminologyanddemonstratingrealtechnicaljudgment.

LanguageModelsInterviewHandbookLamhotSiagian»

—iv—AIEngineeringInsider

Contents

Preface

iii

1Introduction,Foundations,andCareerRoadmapforLLMs

1

2Tokens,Tokenization,andContextWindows

8

2.1WhatisatokenandwhyisittherealunitofcomputationinanLLM?

10

2.2Whydotokensnotmapcleanlytowords?

11

2.3Howdoesbyte-pairencodinghelpmodernlanguagemodels?

11

2.4WhatisSentencePieceandwhenisitpreferabletoclassicwhitespace-based

.........12

2.5Whatisacontextwindow?

12

2.6Whydoestokenizationdirectlyaffectcostandlatency?

13

2.7Whathappenswhenaninputislongerthanthemodelcanaccept?

13

2.8Whatisthedifferencebetweentruncation,slidingwindows,andsummarization?

14

2.9Whyarespecialtokensimportantinmodelbehavior?

14

2.10HowshouldengineersbudgettokensinaproductionLLMsystem?

15

3EmbeddingsandSemanticRepresentations

16

3.1Whatisanembedding?

18

3.2Whydoembeddingsmakesemanticsearchpossible?

18

3.3Whatisthedifferencebetweentokenembeddings,sentenceembeddings,and

........19

3.4WhydoengineersoftenL2-normalizeembeddings?

19

3.5Whenshouldyouusecosinesimilarityinsteadofdotproduct?

20

3.6Whatarehubnessandanisotropyinembeddingspaces?

20

3.7Whatisthedifferencebetweendenseandsparserepresentations?

21

3.8Whatisthedifferencebetweenabi-encoderandacross-encoder?

21

3.9Howdoesembeddingdimensionaffectsystemdesign?

22

3.10Howdoyouevaluateanembeddingmodelbeforeusingitinproduction?

22

4TransformerArchitecture,Attention,andPositionalReasoning

23

4.1Whywasthetransformersuchamajorbreakthrough?

25

4.2Whatisself-attentioninsimpleterms?

25

4.3Whatrolesdoquery,key,andvaluevectorsplayinattention?

26

4.4Whydotransformersusemultipleattentionheads?

26

4.5Whydotransformersneedpositionalencodingsorpositionalembeddings?

27

4.6Whatisthedifferencebetweenencoder-only,decoder-only,and

...................27

4.7Whatdothefeed-forwardblock,residualpath,andlayernormalization

............28

LanguageModelsInterviewHandbookLamhotSiagian»

—v—AIEngineeringInsider

4.8Whydotransformersscalewellbutbecomeexpensiveonlongsequences?

28

4.9Whatisthedifferencebetweencausalmaskingandbidirectionalattention?

29

4.10Whatarecommontransformerfailuremodesengineersshouldunderstand?

29

5PretrainingObjectives,ModelFamilies,andClassicalComparisons

30

5.1Whatdefinesalanguagemodelandwhyisitcalled“large”?

32

5.2Howdoautoregressiveandmaskedmodelsdiffer?

32

5.3Whatismaskedlanguagemodelingandwhatdoesitteachthemodel?

33

5.4Whatisnextsentencepredictionandwhydoesitmatterhistorically?

33

5.5Howdolanguagemodelshandleout-of-vocabularywords?

34

5.6Whatisasequence-to-sequencemodelandwhereisitmostuseful?

34

5.7WhydidtransformersreplacemanyRNN-basedSeq2Seqsystems?

35

5.8Howdofoundationmodelsdifferfromtask-specificmodels?

35

5.9Whatisthedifferencebetweengenerativeanddiscriminativemodels?

36

5.10HowdoLLMsdifferfromtraditionalstatisticallanguagemodels?

36

6ClassificationwithLargeLanguageModels

37

6.1HowcanagenerativeLLMperformclassification?

39

6.2Whenshouldyouusepromptinginsteadoffine-tuningforclassification?

39

6.3Whatisthedifferencebetweenzero-shotandfew-shotclassification?

40

6.4HowshouldyoudesignalabeltaxonomyforanLLMclassifier?

40

6.5HowdoyouhandleclassimbalanceinLLM-basedclassification?

41

6.6Howismulti-labelclassificationdifferentfromsingle-labelclassification?

41

6.7WhichmetricsmattermostforclassificationsystemsbuiltwithLLMs?

42

6.8HowdoyouestimateconfidenceforanLLMclassifier?

42

6.9Whenshouldaclassificationpipelineincludeahumanintheloop?

43

6.10WhatarecommonproductionfailuremodesinLLMclassificationsystems?

43

7TopicModeling,Clustering,andThemeDiscoveryatScale

44

7.1Howistopicmodelingdifferentfromclassification?

45

7.2Whyhaveembedding-basedclusteringmethodsbecomepopularfortopicdiscovery?

46

7.3Whatisapracticalpipelinefortopicdiscoveryatscale?

46

7.4Whydoengineersoftenreducedimensionalitybeforeclustering?

47

7.5Howdoyouchooseaclusteringalgorithmfortopicdiscovery?

47

7.6Howdoyounameclusterssobusinessteamscanactuallyusethem?

48

7.7Howdoyouhandleevolvingtopicsovertime?

48

7.8Howdoyouevaluatewhetherdiscoveredtopicsaregood?

49

7.9HowcanLLMsimprovetopicmodelingworkflows?

49

7.10Whatarecommonmistakeswhenteamsruntopicmodelingatscale?

50

8RetrievalFoundationsforLargeLanguageModelSystems

51

8.1Whatisretrieval-augmentedgeneration,orRAG?

54

8.2Whatisthedifferencebetweenlexicalretrievalanddenseretrieval?

54

8.3Whyishybridretrievaloftenbetterthanusingonlyonemethod?

55

LanguageModelsInterviewHandbookLamhotSiagian»

—vi—AIEngineeringInsider

8.4WhyischunkingsoimportantinRAG?

55

8.5Howdometadatafiltersimproveretrievalquality?

56

8.6Whatisavectordatabaseandwhatproblemdoesitsolve?

56

8.7Whydoproductionsystemsrelyonapproximatenearest-neighborsearch?

57

8.8Whatisrerankingandwhyisituseful?

57

8.9Howdoesqueryrewritinghelpretrieval?

58

8.10Whichofflinemetricsmattermostforretrievalquality?

58

9ProductionRAGArchitecturesandGroundedAnswering

59

9.1WhatisthedifferencebetweennaiveRAGandproductionRAG?

61

9.2Whatisthedifferencebetweensingle-hopandmulti-hopretrieval?

61

9.3HowdoyoureducehallucinationsinaRAGsystem?

62

9.4Whyarecitationsandprovenancesoimportantingroundedsystems?

62

9.5HowshouldaRAGsystemhandlefreshnessandknowledgeupdates?

63

9.6WhatisagenticRAGandwhenisituseful?

63

9.7WhydocachinglayersmatterinproductionRAG?

64

9.8HowdopermissionsandaccesscontrolaffectRAGdesign?

64

9.9HowdoyouevaluateaproductionRAGsystemofflineandonline?

65

9.10WhenshouldyoudecidenottouseRAG?

65

10Prompting,In-ContextLearning,andLLMOrchestration

66

10.1Whatrolesdosystem,user,andtoolmessagesplayinchat-basedLLMsystems?

67

10.2Whatmakesapromptreliablygoodratherthanmerelyverbose?

68

10.3Whendoesfew-shotpromptingmateriallyhelp?

68

10.4Howshouldyouthinkaboutchain-of-thoughtpromptinginaproductsetting?

69

10.5Howdoyoupromptforstructuredoutputs?

69

10.6Whatistoolorfunctioncallingandwhydoesitmatter?

70

10.7Whydoprompttemplatesandversioningmatterinengineeringteams?

70

10.8Whatispromptinjectionandwhyisitdangerous?

71

10.9Howdoyouevaluatewhetherapromptchangeisactuallybetter?

71

10.10Whendopromptsstopbeingenoughandastrongerinterventionbecomenecessary?

72

11MultimodalLargeLanguageModels

73

11.1WhatisamultimodalLLM?

75

11.2Whatisthecommonarchitecturepatternbehindtext-imagesystems?

75

11.3WhyisCLIPimportantinthehistoryofmultimodalsystems?

76

11.4Whatdoesvisualgroundingmeaninamultimodalmodel?

76

11.5WhenshouldyourelyonOCRversusnativevision-languageunderstanding?

77

11.6Howdoesmultimodalpromptingdifferfromtext-onlyprompting?

77

11.7Howdoyouevaluateamultimodalsystem?

78

11.8WhatarecommonfailuremodesinmultimodalLLMs?

78

11.9Howdoaudioandvideochangethedesigncomparedwithstaticimages?

79

11.10Whichmultimodalusecasesusuallydeliverthebestbusinessvaluefirst?

79

LanguageModelsInterviewHandbookLamhotSiagian»

—vii—AIEngineeringInsider

12CustomEmbeddingsandRetrievalOptimization

80

12.1Whywouldateamchoosecustomembeddingsinsteadofageneralembeddingmodel?

82

12.2Whatarethemainapproachestodomainadaptationforembeddings?

82

12.3Whyarehardnegativesimportantwhentrainingretrievalembeddings?

83

12.4Whichtraininglossesarecommonforembeddingfine-tuning?

83

12.5Howdoyourepresentlongdocumentswhenoneembeddingisnotenough?

84

12.6Whatspecialconsiderationsapplytomultilingualembeddingsystems?

84

12.7Howdoindexcompressionandquantizationaffectretrievalquality?

85

12.8Howshouldyouchoosesimilaritythresholdsinretrievalsystems?

85

12.9Howdoyoumonitorretrievaldriftafterdeployingcustomembeddings?

86

12.10Whatshouldateamplanforwhenmigratingfromoneembeddingmodeltoanother?

86

13Fine-Tuning,PEFT,andAdaptationStrategies

87

13.1Whatisthedifferencebetweenfullfine-tuningandparameter-efficient

.............90

13.2WhatareLoRAandQLoRA,andhowdotheydiffer?

90

13.3Whatisthedifferencebetweensupervisedfine-tuning,instructiontuning,

..........91

13.4Whatismodeldistillationandwhenisituseful?

91

13.5Whenisfine-tuningactuallyworththeeffort?

92

13.6Whatmakesafine-tuningdatasethighquality?

92

13.7Whatiscatastrophicforgettingandwhydoesitmatter?

93

13.8Howshouldyouevaluateafine-tunedmodelbeforerelease?

93

13.9Howdoesalignmentrelatetofine-tuning?

94

13.10Whatarethemaincosttrade-offsinfine-tuningprojects?

94

13.11Whenshouldateamavoidfine-tuningaltogether?

95

14OptimizationandMathFoundationsforLanguageModels

96

14.1Howisthesoftmaxfunctionusedinattention?

98

14.2Whydoesthedotproductappearinself-attention?

98

14.3Whyiscross-entropythestandardlossforlanguagemodeling?

99

14.4Howaregradientscomputedforembeddingsduringbackpropagation?

99

14.5WhatdoestheJacobianmatrixrepresentindeeplearning?

100

14.6Howdoeigenvaluesandeigenvectorsconnecttodimensionalityreduction?

100

14.7WhatisKLdivergenceandwhenisitusefulinLLMtraining?

101

14.8WhydoestheReLUderivativematter?

101

14.9Howdoesthechainrulemakebackpropagationpossible?

102

14.10Howdoresidualconnectionsandnormalizationhelpwithvanishinggradients?

102

15TextGeneration,Decoding,andServingatScale

104

15.1Howdotemperature,top-k,andtop-pchangemodeloutputs?

106

15.2Howdoesbeamsearchcomparewithgreedydecoding?

106

15.3Whyisstreaminggenerationimportantinuser-facingsystems?

107

15.4Howdobatchingandconcurrencyimproveservingefficiency?

107

15.5WhatistheKVcacheandwhydoesitmatterforautoregressivedecoding?

108

LanguageModelsInterviewHandbookLamhotSiagian»

—viii—AIEngineeringInsider

15.6Howdoesquantizationhelpdeploylargemodels?

108

15.7Howshouldengineersthinkaboutthroughputversuslatency?

109

15.8Whatmakeslong-contextservingdifficult?

109

15.9Howdosafetyandmoderationfitintogenerationpipelines?

110

15.10HowwouldyoudescribeascalableLLMgenerationserviceinsystem-designterms?

110

16Architectures,Extensions,andPracticalDeployment

111

16.1WhatisaMixtureofExpertsmodelandwhyisitattractive?

113

16.2WhatnewfailuremodesdoMoEsystemsintroduce?

113

16.3Howcanknowledgegraphscomplementlanguagemodels?

114

16.4Whendoesaknowledgegraphhelpmorethanplainvectorretrieval?

114

16.5Whatisadaptivesoftmaxandwhenisituseful?

115

16.6HowdoClaude-styleandGPT-styleecosystemscommonlydifferfordevelopers?

115

16.7Whydohyperparametersmatterbeyondthelearningrate?

116

16.8Howdoyouaddressbiasedorsystematicallyincorrectoutputs?

116

16.9WhyareinterpretabilityandprivacyhardinLLMdeployment?

117

16.10Whatdeploymentbottlenecksdoteamsunderestimatemostoften?

117

References

118

LanguageModelsInterviewHandbookLamhotSiagian»

—1—AIEngineeringInsider

Chapter1

Introduction,Foundations,andCareerRoadmapforLLMs

Chapteroverview.Thisopeningchaptergivesthereaderamapbeforethedeepdivebegins.Itdefineswhatalargelanguagemodelisatasystemslevel,showshowthemodernLLMstackfitstogether,outlinesapracticallearningroadmap,andsummarizesthetrendsshapinghiringinGenAIroles.Thegoalistomaketherestofthebookeasiertonavigatebecausereadersunderstandwhytokenization,embeddings,attention,retrieval,adaptation,evaluation,andservingappearinthisorder.

Astronginterviewcandidaterarelystartsbyrecitingarchitecturejargon.Theyfirstframetheworkload,identifywherethemodelcreatesvalue,andexplainthesurroundingsystemthatturnsrawmodelcapabilityintoaproductionproduct.Thatframingmindsetiswhatthisintroductorychapterisdesignedtobuild.

InterviewAnchor

Whattheinterviewerisreallytesting.WhetheryoucanexplainLLMsasengineeringsystemsratherthanasisolatedresearchbuzzwords.

Stronganswerpattern.DefinetheLLMasapretrainednext-tokenmodelembeddedinabroaderapplicationstack,thenconnectthestacktoretrieval,prompting,evaluation,serving,governance,andmeasurableproductoutcomes.

Commonmiss.Candidatesoftenjumpstraightintomodelnamesorhype.Strongcandidatesexplainthejobtobedone,thedatapath,thereliabilitycontrols,andthetrade-offsbetweenflexibility,cost,andrisk.

LanguageModelsInterviewHandbookLamhotSiagian»

—2—AIEngineeringInsider

INTERVIEWCHEATSHEET

Signaltohit

AnLLMisnotthewholeproduct.Itisthereasoningand

Bestexample

generationengineinsidealargerretrieval,tool-use,evaluation,anddeliveryworkflow.

Explainwhyacustomer-supportassistantneedspromptdesign,retrievalquality,monitoring,escalationrules,andoutputcontrolsinadditiontoacapablebasemodel.

Follow-upangle

MentiontheroadmapfromtokenizationandattentiontoRAG,PEFT,serving,evaluation,andgovernance.

Whatseniorsadd

Theyconnectcurrenttrendssuchasmultimodality,smaller

specializedmodels,andinferenceoptimizationtorealproductchoices.

Redflag

TreatingLLMengineeringlikepurepromptingwithout

discussingdata,qualitymeasurement,oroperationalconstraints.

Foundations:WhatanLLMreallyis

Alargelanguagemodelisaneuralnetworktrainedtopredictthenexttokeninasequenceatverylargescale.Thatsimpleobjectivebecomespowerfulbecausethemodelinternalizesstatisticalpatternsaboutsyntax,facts,structure,style,andtaskbehavioracrossenormouscorpora.Inpractice,however,aninterview-qualityexplanationshouldgoonelevelhigher:anLLMisvaluablenotmerelybecauseitgeneratestext,butbecauseitcanbeembeddedintoworkflowsthatclassify,retrieve,summarize,reasonovertools,anddraftstructuredoutputs.

Thisiswhytherestofthehandbookmovesfromtokenizationandembeddingsintoretrieval,adaptation,prompting,evaluation,andserving.Thoselayersarenotseparatetopicsgluedtogetherforstudyconvenience.TheyaretherealoperationallayersthatdeterminewhetheraGenAIsystemfeelsuseful,grounded,fast,safe,andeconomicallysustainable.

Theroadmapfigurebelowturnsthebookintoasequenceoflearninglayers.Ithelpsthereaderseewhytheearlymechanicschapterscomefirstandwhylaterchaptersfocusonretrieval,adaptation,anddeploymentratherthanstoppingatbase–modelconcepts.

Theroadmapmattersbecausemanycandidatesjumptofashionabletopicsbeforetheycanexplainthemechanicsunderneaththem.Thebettersequenceistobuildmechanism-levelunderstandingfirst,thenmoveupwardintoproductpatterns,evaluation,anddeployment.

LanguageModelsInterviewHandbookLamhotSiagian»

—3—AIEngineeringInsider

Layer1

Text,Tokens,Context

Layer2

EmbeddingsandAttention

Layer3

PretrainingandModelFamilies

Layer4

Retrieval,RAG,andPrompting

Layer6

Serving,Governance,andProducts

Layer5

Adaptation,PEFT,andEvaluation

Usethisasastudyplan:mechan-

icsfirst,systemdesignsecond,

productiontrade-offsthird.

Hiringloopsoftentesttheselaterlayersmostbecausethey revealengineeringjudgment.

Figure1.1:ApracticalroadmapforlearningandinterviewingacrossthemodernLLMstack.

RoadmapforLLMandGenAIroles

Formostengineers,themosteffectiveroadmapislayeredratherthanchronological.Startwithtextfundamentalsandmodelmechanics,thenlearnhowretrievalchangescontextquality,thenlearnhowadaptationandservingmakethesystemproduction-ready.Afterthat,specializeintodomainssuchasevaluation,agents,multimodalsystems,safety,ordomain-specificcopilots.

Thesamelayeredviewisalsousefulforresumeandinterviewstorytelling.Itletsyoupositionyourselfclearly.Youcansaythatyouarestrongestinretrievalandevaluation,orinservingandoptimization,orinproductizingagentworkflows.Thatsoundsmorecrediblethanclaimingbroadexpertisewithoutavisiblestack-shapednarrative.

Thetrendstablebelowisincludedtoanchorthehandbookincurrentindustrydirection.Itisnotalistofhypeterms.Eachrowpointstoaskillareathatchangeshowteamshire,scopeprojects,andevaluatetechnicaldepth.

Readthetrendstableasaprioritizationfilter.Thedifferentiatorisoftennotanothergenericmodeltutorial,butyourabilitytoreasonaboutretrievalquality,evaluation,inferenceconstraints,andwherehumansremainintheloop.”

LanguageModelsInterviewHandbookLamhotSiagian»

—4—AIEngineeringInsider

Table1.1:LLMtrendsthatmostaffectengineeringroadmapsandinterviewexpectations

Trend

Whyitmatters

Whatstrongcandidatesshouldbereadytodiscuss

Longercontextwindows

Moreinformationcanfitintoaprompt,but

irrelevantcontextstillhurtsanswersandcost.

Whyretrieval,ranking,andcontext

compressionstillmatterevenwhenthemodelsupportsverylargewindows.

Multimodalsystems

Text-onlyproductsarenolongerthedefaultformanyenterpriseand

consumerworkflows.

Howimage,audio,ordocumentinputschangeevaluation,latency,and

user-experiencedesign.

Smaller

specializedmodels

Manyteamsnowbalancefrontier-modelquality

againstcost,control,anddeploymentflexibility.

Whentouseasmallertask-shapedmodel,PEFT,orroutinginsteadofalwayscallingthelargestmodel.

Evaluationandgovernance

Asproductsscale,trustandmeasurement

becomeharderthandemoquality.

Offlineandonlineevaluation,hallucinationcontrol,guardrails,escalation,and

monitoring.

Inference

optimization

Costandlatency

increasinglydefine

whetheranLLMproductisviable.

Quantization,batching,caching,structuredoutputs,andoutput-budgetdiscipline.

Tooluseandagents

Realproducts

increasinglycombinelanguagemodelswithAPIs,databases,andworkflowengines.

Planningversusexecution,toolselection,statemanagement,andhuman-in-the-loopcontrols.

MindMapofThisHandbook

Thismindmapisintentionallyhighlevel.ItgivesthereaderavisualindexofhowthechaptersconnectsolatertopicssuchasPEFTorservingfeellikeextensionsofthesamesystemratherthanunrelatedinterviewtrivia.

Noticehowthemapmovesfromrepresentationtosystems.Stronganswersbecomepersua-sivewhentheyconnectmechanism-levelunderstandingtoproductbehaviorandproductionconstraints.

LanguageModelsInterviewHandbookLamhotSiagian»

—5—AIEngineeringInsider

TokensandContext

Fine-TuningandPEFTEmbeddingsandRetrieval

GovernanceandDeployment

MultimodalSystems

LLMInterviewMastery

ServingandDecodingAttentionandModelDesign

RAGandGrounding

Figure1.2:Amindmapoftheconceptsthishandbookbuildsfromfoundationtodeployment.

Bonus:ResumeStructureforLLMandGenAIRoles

BonusLayer

Astronglanguage-modelresumeshouldreadlikeanengineeringsystemsdocument,notabuzzwordcollage.Recruitersandinterviewerslookforevidenceofrealworkloadownership:evaluation,retrievalquality,agentorchestration,reliability,safety,andmeasurableimpact.

Thebonusmaterialbelongsherebecausecareerpositioningshouldtrackthesamestackthebookteaches:whatyoubuilt,howyoumeasuredit,andwhattrade-offsyouowned.

ThetablebelowconvertsthebroadideaofanLLMresumeintoconcretesections.Readitasachecklistforevidence