人工智能基础实践教程 习题及答案 第8章 习题

第8章习题8.1在深度神经网络训练中，梯度消失和梯度爆炸是常见问题。请解释其数学成因，并列举至少3种解决方案。答：成因是通过链式法则计算梯度时，深层网络的梯度是各层梯度的连乘积。若每层梯度绝对值小于1（如使用Sigmoid激活函数），则反向传播时梯度会指数级衰减，出现梯度消失；若大于1（如初始化权重过大），则梯度会指数级增长，出现梯度爆炸。解决方案包括：1）权重初始化：使用Xavier/Glorot初始化（均匀或正态分布，方差与输入维度相关）或He初始化（适配ReLU）。2）激活函数选择：用ReLU/LeakyReLU替代Sigmoid/Tanh，缓解梯度消失。3）梯度裁剪：设置阈值，当梯度超过阈值时按比例缩放。4）批归一化（BatchNormalization）：规范化每层输入，稳定梯度传播。5）残差连接（ResNet）：通过跳跃连接直接传递梯度，缓解深层网络梯度消失。8.2说明批归一化在深度学习中的核心作用，并写出其前向传播的数学公式。答： 批归一化的核心作用如下：1）加速训练收敛，通过规范化每层输入，减少内部协变量偏移（InternalCovariateShift）。2）缓解梯度消失/爆炸，稳定梯度传播。3）允许更高的学习率，因输入分布稳定，可增大学习率而不导致发散。4）轻微正则化效果，基于批次统计的随机性引入噪声。给定一个批次的数据X∈RB×D，B为批次大小，D1）计算批次均值：μ2）计算批次方差：σ3）规范化：X4）缩放与平移：Y=γX+β8.3解释注意力机制的数学本质，并写出缩放点积注意力的计算公式。答： 注意力机制的核心思想是：通过动态计算输入序列中各位置之间的相关性权重，分配不同注意力资源。其数学本质是加权求和，权重由输入通过线性变换后的相似度决定。 缩放点积注意力公式如下：给定查询矩阵Q∈RN×dkAttention其中，QKT计算查询与键的相似度（未归一化注意力权重）；dk为缩放因子，防止点积结果过大导致softmax梯度消失；softmax8.4生成对抗网络的生成器和判别器通过博弈优化，请写出其损失函数的原始形式，并解释模式崩溃的成因与缓解方法。答： 生成对抗网络的损失函数为min其中，判别器(D)用于最大化真实样本与生成样本的区分能力；生成器(G)用于最小化生成样本被判别为假的概率。模式崩溃成因是：生成器发现某一类样本能轻松欺骗判别器后，会过度生成该类样本，忽略其他模式，如只生成数字“1”的图片而忽略其他数字。 缓解方法有：1）WassersteinGAN（WGAN）：用Wasserstein距离替代JS散度，改善梯度稳定性。2）UnrolledGAN：在生成器更新时考虑判别器未来几步的反应，避免短视优化。3）多样性正则化：在生成器损失中添加惩罚项，鼓励生成多样样本（如特征匹配损失）。4）多生成器/判别器架构：如MultipleGeneratorsGAN，强制生成器覆盖不同模式。5）经验回放或记忆机制：存储生成过的样本，在训练中约束新样本不能过度偏离已生成过的模式，或显式鼓励覆盖未生成的模式。8.5预训练模型在下游任务微调时可能遭遇灾难性遗忘（CatastrophicForgetting）。请解释其含义，并列举至少2种微调策略以缓解该问题。答：灾难性遗忘是指微调时模型过度适应下游任务数据，丢失预训练阶段学到的通用知识，导致在原始任务（如语言理解）上性能下降。缓解策略包括：1）分层微调（Layer-wiseTuning）：冻结底层参数（如BERT的嵌入层和前几层），仅微调高层任务相关层。底层提取通用特征，高层适配特定任务，减少知识丢失。2）弹性权重巩固（ElasticWeightConsolidation,EWC）：在损失函数中添加正则化项，约束重要参数（对预训练任务敏感的参数）的变化幅度。数学形式为ℒnew=ℒ3）LoRA（Low-RankAdaptation）：在预训练模型参数旁插入低秩矩阵，仅训练新增参数，保持原参数不变。显著减少可训练参数量，降低遗忘风险。8.6假设在使用长短时记忆网络对IMDB电影评论进行情感分类时，模型出现梯度爆炸问题，请实现梯度裁剪和余弦退火学习率调度策略，并考察使用后训练稳定性提升效果。答：示例代码如下。importtorchimporttorch.nnasnnfromtorch.utils.dataimportDataLoader,Datasetfromtorchtext.data.utilsimportget_tokenizerfromtorchtext.vocabimportbuild_vocab_from_iteratorimporttorch.optimasoptimfromtorch.optim.lr_schedulerimportCosineAnnealingLRimportmatplotlib.pyplotaspltimportnumpyasnpfromtqdmimporttqdmimportos#设置随机种子torch.manual_seed(42)np.random.seed(42)tokenizer=get_tokenizer('basic_english')#==================1.从手动数据集构建词汇表==================defyield_tokens_from_dir(data_dir,split='train'):

"""遍历数据目录，逐条文本生成tokens"""

split_dir=os.path.join(data_dir,split)

forlabel_namein['pos','neg']:

folder=os.path.join(split_dir,label_name)

forfnameinos.listdir(folder):

iffname.endswith('.txt'):

withopen(os.path.join(folder,fname),'r',encoding='utf-8')asf:

text=f.read()

yieldtokenizer(text)print("Buildingvocabularyfrommanualdataset...")data_dir='./data/aclImdb'

#请确保路径正确vocab=build_vocab_from_iterator(

yield_tokens_from_dir(data_dir,split='train'),

specials=['<unk>'])vocab.set_default_index(vocab['<unk>'])print(f"Vocabularysize:{len(vocab)}")#==================2.自定义Dataset==================classManualIMDBDataset(Dataset):

def__init__(self,data_dir,split='train',max_samples=None):

self.samples=[]

label_map={'pos':1,'neg':0}

split_dir=os.path.join(data_dir,split)

forlabel_namein['pos','neg']:

folder=os.path.join(split_dir,label_name)

files=[fforfinos.listdir(folder)iff.endswith('.txt')]

ifmax_samples:

#正负各取一半

half=max_samples//2

files=files[:half]

forfnameintqdm(files,desc=f"Loading{split}/{label_name}",leave=False):

withopen(os.path.join(folder,fname),'r',encoding='utf-8')asf:

text=f.read()

self.samples.append((text,label_map[label_name]))

ifmax_samplesandlen(self.samples)>max_samples:

self.samples=self.samples[:max_samples]

#打乱顺序，避免正负样本连续

importrandom

random.shuffle(self.samples)

print(f"Loaded{len(self.samples)}samplesfrom{split}set")

def__len__(self):

returnlen(self.samples)

def__getitem__(self,idx):

text,label=self.samples[idx]

token_ids=vocab(tokenizer(text))

returntorch.tensor(token_ids,dtype=torch.long),label#==================3.填充函数==================defcollate_batch(batch):

texts,labels=zip(*batch)

lengths=[len(t)fortintexts]

max_len=max(lengths)

padded_texts=torch.zeros(len(texts),max_len,dtype=torch.long)

fori,textinenumerate(texts):

padded_texts[i,:len(text)]=text

labels=torch.tensor(labels,dtype=torch.float)

returnpadded_texts,labels#==================4.LSTM模型==================classLSTMClassifier(nn.Module):

def__init__(self,vocab_size,embed_dim=100,hidden_dim=128,num_layers=2):

super(LSTMClassifier,self).__init__()

self.embedding=nn.Embedding(vocab_size,embed_dim)

self.lstm=nn.LSTM(embed_dim,hidden_dim,num_layers,batch_first=True,dropout=0.3)

self.fc=nn.Linear(hidden_dim,1)

defforward(self,x):

x=self.embedding(x)

_,(hidden,_)=self.lstm(x)

out=self.fc(hidden[-1])

returnout.squeeze()#==================5.训练函数==================deftrain_model(model,dataloader,epochs=5,lr=0.001,use_clipping=False,use_cosine=False):

device=torch.device('cuda'iftorch.cuda.is_available()else'cpu')

print(f"Usingdevice:{device}")

model=model.to(device)

criterion=nn.BCEWithLogitsLoss()

optimizer=optim.Adam(model.parameters(),lr=lr)

scheduler=CosineAnnealingLR(optimizer,T_max=epochs)ifuse_cosineelseNone

epoch_losses=[]

forepochinrange(epochs):

model.train()

total_loss=0.0

progress_bar=tqdm(dataloader,desc=f"Epoch{epoch+1}/{epochs}",leave=False)

fortexts,labelsinprogress_bar:

texts,labels=texts.to(device),labels.to(device)

optimizer.zero_grad()

outputs=model(texts)

loss=criterion(outputs,labels)

loss.backward()

ifuse_clipping:

torch.nn.utils.clip_grad_norm_(model.parameters(),max_norm=1.0)

optimizer.step()

total_loss+=loss.item()

progress_bar.set_postfix(loss=loss.item())

avg_loss=total_loss/len(dataloader)

epoch_losses.append(avg_loss)

ifscheduler:

scheduler.step()

print(f"Epoch{epoch+1},AvgLoss:{avg_loss:.4f}")

returnepoch_losses#==================6.主程序==================defmain():

#数据集路径（请根据实际情况修改）

data_dir='./data/aclImdb'

#确保该路径下存在train/和test/文件夹

max_train_samples=5000

#训练集使用前5000条（可改为None使用全部）

print("Creatingtrainingdataset...")

train_dataset=ManualIMDBDataset(data_dir,split='train',max_samples=max_train_samples)

batch_size=64

dataloader=DataLoader(train_dataset,batch_size=batch_size,shuffle=True,collate_fn=collate_batch)

vocab_size=len(vocab)

epochs=8

#配置A

print("\n===配置A：无梯度裁剪+固定学习率===")

model_a=LSTMClassifier(vocab_size)

losses_a=train_model(model_a,dataloader,epochs=epochs,lr=0.003,

use_clipping=False,use_cosine=False)

#配置B

print("\n===配置B：梯度裁剪+余弦退火学习率===")

model_b=LSTMClassifier(vocab_size)

losses_b=train_model(model_b,dataloader,epochs=epochs,lr=0.003,

use_clipping=True,use_cosine=True)

#绘图对比

plt.figure(figsize=(10,6))

plt.plot(range(1,epochs+1),losses_a,marker='o',label='NoClipping+FixedLR')

plt.plot(range(1,epochs+1),losses_b,marker='s',label='GradientClipping+CosineAnnealing')

plt.xlabel('Epoch')

plt.ylabel('AverageLoss')

plt.title('TrainingStabilityComparison')

plt.legend()

plt.grid(True)

plt.show()

print("\n===最终损失对比===")

print(f"配置A:最终损失={losses_a[-1]:.4f},最小损失={min(losses_a):.4f}")

print(f"配置B:最终损失={losses_b[-1]:.4f},最小损失={min(losses_b):.4f}")if__name__=="__main__":

main()#推荐从/~amaas/data/sentiment/提前下载好代码所需的数据集aclImdb_v1.tar.gz输出：Buildingvocabularyfrommanualdataset...Vocabularysize:100683Creatingtrainingdataset...Loaded5000samplesfromtrainset===配置A：无梯度裁剪+固定学习率===Usingdevice:cpuEpoch1,AvgLoss:0.6946Epoch2,AvgLoss:0.6933Epoch3,AvgLoss:0.6936Epoch4,AvgLoss:0.6936Epoch5,AvgLoss:0.6935Epoch6,AvgLoss:0.6932Epoch7,AvgLoss:0.6932Epoch8,AvgLoss:0.6930===配置B：梯度裁剪+余弦退火学习率===Usingdevice:cpuEpoch1,AvgLoss:0.6943Epoch2,AvgLoss:0.6935Epoch3,AvgLoss:0.6934Epoch4,AvgLoss:0.6920Epoch5,AvgLoss:0.6908Epoch6,AvgLoss:0.6883Epoch7,AvgLoss:0.6878Epoch8,AvgLoss:0.6862===最终损失对比===配置A:最终损失=0.6930,最小损失=0.6930配置B:最终损失=0.6862,最小损失=0.68628.7在医学图像分类任务中，正样本（病变）仅占5%，存在严重类别不平衡。请实现FocalLoss，并在模拟的不平衡二分类数据集上训练ResNet-18，对比其相比标准交叉熵损失的改善效果。答：示例代码如下。importtorchimporttorch.nnasnnimporttorch.optimasoptimimporttorch.nn.functionalasFfromtorch.utils.dataimportDataLoader,TensorDatasetfromtorchvision.modelsimportresnet18importnumpyasnpfromsklearn.metricsimportaccuracy_score,precision_score,recall_score,f1_score,roc_auc_scoreimportmatplotlib.pyplotaspltfromcopyimportdeepcopy#-----------------------------1.FocalLoss）-----------------------------classFocalLoss(nn.Module):

def__init__(self,alpha=0.75,gamma=2,reduction='mean'):

super(FocalLoss,self).__init__()

self.alpha=alpha

self.gamma=gamma

self.reduction=reduction

defforward(self,inputs,targets):

BCE_loss=F.binary_cross_entropy_with_logits(inputs,targets,reduction='none')

pt=torch.exp(-BCE_loss)

F_loss=self.alpha*(1-pt)**self.gamma*BCE_loss

ifself.reduction=='mean':

returnF_loss.mean()

elifself.reduction=='sum':

returnF_loss.sum()

else:

returnF_loss#--------------------2.生成模拟医学图像数据（拆分为train/val）-------------------------np.random.seed(42)torch.manual_seed(42)N=2000

#总样本数pos_ratio=0.05

#正样本比例n_pos=int(N*pos_ratio)

#100n_neg=N-n_pos

#1900#随机生成3x224x224的图像数据（实际应为真实医学图像，这里仅作演示）X=torch.randn(N,3,224,224)y=torch.zeros(N,1,dtype=torch.float)y[n_neg:]=1.0

#最后n_pos个为正样本#打乱顺序（保证正负样本随机分布）indices=np.random.permutation(N)X,y=X[indices],y[indices]#划分训练集(80%)和验证集(20%)split=int(0.8*N)X_train,y_train=X[:split],y[:split]X_val,y_val=X[split:],y[split:]#创建DataLoaderbatch_size=64train_dataset=TensorDataset(X_train,y_train)val_dataset=TensorDataset(X_val,y_val)train_loader=DataLoader(train_dataset,batch_size=batch_size,shuffle=True)val_loader=DataLoader(val_dataset,batch_size=batch_size,shuffle=False)#--------------------------3.真正的ResNet-18（修改输出层为二分类）--------------------------defget_resnet18_model():

model=resnet18(pretrained=False)

#医学图像通常随机初始化即可

model.fc=nn.Linear(model.fc.in_features,1)

#二分类输出

returnmodel#--------------------4.训练与评估函数（支持FocalLoss/CrossEntropy）------------------------deftrain_and_evaluate(model,criterion,optimizer,train_loader,val_loader,epochs=20,device='cuda'):

model=model.to(device)

train_losses,val_losses=[],[]

val_metrics={'accuracy':[],'precision':[],'recall':[],'f1':[],'auc':[]}

forepochinrange(epochs):

#训练阶段

model.train()

total_loss=0.0

forX_batch,y_batchintrain_loader:

X_batch,y_batch=X_batch.to(device),y_batch.to(device)

optimizer.zero_grad()

outputs=model(X_batch)

loss=criterion(outputs,y_batch)

loss.backward()

optimizer.step()

total_loss+=loss.item()

avg_train_loss=total_loss/len(train_loader)

train_losses.append(avg_train_loss)

#验证阶段

model.eval()

all_preds=[]

all_labels=[]

all_probs=[]

val_loss=0.0

withtorch.no_grad():

forX_batch,y_batchinval_loader:

X_batch,y_batch=X_batch.to(device),y_batch.to(device)

outputs=model(X_batch)

loss=criterion(outputs,y_batch)

val_loss+=loss.item()

probs=torch.sigmoid(outputs)

preds=(probs>=0.5).float()

all_preds.extend(preds.cpu().numpy())

all_labels.extend(y_batch.cpu().numpy())

all_probs.extend(probs.cpu().numpy())

avg_val_loss=val_loss/len(val_loader)

val_losses.append(avg_val_loss)

#计算指标（处理可能无正样本的情况）

accuracy=accuracy_score(all_labels,all_preds)

precision=precision_score(all_labels,all_preds,zero_division=0)

recall=recall_score(all_labels,all_preds,zero_division=0)

f1=f1_score(all_labels,all_preds,zero_division=0)

auc=roc_auc_score(all_labels,all_probs)iflen(np.unique(all_labels))>1else0.5

val_metrics['accuracy'].append(accuracy)

val_metrics['precision'].append(precision)

val_metrics['recall'].append(recall)

val_metrics['f1'].append(f1)

val_metrics['auc'].append(auc)

print(f"Epoch{epoch+1:2d}|TrainLoss:{avg_train_loss:.4f}|ValLoss:{avg_val_loss:.4f}|"

f"Acc:{accuracy:.3f}|Recall:{recall:.3f}|F1:{f1:.3f}|AUC:{auc:.3f}")

returntrain_losses,val_losses,val_metrics#-----------------------------5.主程序：对比两种损失函数-----------------------------defmain():

device=torch.device('cuda'iftorch.cuda.is_available()else'cpu')

epochs=20

lr=0.001

#模型1：交叉熵损失

model_ce=get_resnet18_model()

criterion_ce=nn.BCEWithLogitsLoss()

optimizer_ce=optim.Adam(model_ce.parameters(),lr=lr)

print("==========TrainingwithCross-EntropyLoss==========")

train_loss_ce,val_loss_ce,metrics_ce=train_and_evaluate(

model_ce,criterion_ce,optimizer_ce,train_loader,val_loader,epochs,device

)

#模型2：FocalLoss（alpha可设为正样本比例，此处取0.95/0.05的倒数归一化，或简单用0.75）

model_fl=get_resnet18_model()

#alpha推荐设为pos_weight=n_neg/n_pos=1900/100=19，但FocalLoss中alpha是标量乘在损失上，通常取0.75~0.9

criterion_fl=FocalLoss(alpha=0.75,gamma=2)

optimizer_fl=optim.Adam(model_fl.parameters(),lr=lr)

print("\n==========TrainingwithFocalLoss==========")

train_loss_fl,val_loss_fl,metrics_fl=train_and_evaluate(

model_fl,criterion_fl,optimizer_fl,train_loader,val_loader,epochs,device

)

#-----------------------------6.结果可视化与对比-----------------------------

plt.figure(figsize=(14,5))

#损失曲线对比

plt.subplot(1,2,1)

plt.plot(range(1,epochs+1),train_loss_ce,label='CETrain',marker='o')

plt.plot(range(1,epochs+1),val_loss_ce,label='CEVal',marker='s')

plt.plot(range(1,epochs+1),train_loss_fl,label='FLTrain',marker='^')

plt.plot(range(1,epochs+1),val_loss_fl,label='FLVal',marker='d')

plt.xlabel('Epoch')

plt.ylabel('Loss')

plt.title('Training&ValidationLoss')

plt.legend()

plt.grid(True)

#F1分数对比（对不平衡任务最关键）

plt.subplot(1,2,2)

plt.plot(range(1,epochs+1),metrics_ce['f1'],label='CEF1',marker='o')

plt.plot(range(1,epochs+1),metrics_fl['f1'],label='FLF1',marker='s')

plt.xlabel('Epoch')

plt.ylabel('F1Score')

plt.title('F1ScoreonValidationSet')

plt.legend()

plt.grid(True)

plt.tight_layout()

plt.savefig('loss_f1_comparison.png')

plt.show()

#打印最终结果

print("\n==================FinalPerformance==================")

print(f"Cross-EntropyLoss:

ValLoss={val_loss_ce[-1]:.4f},F1={metrics_ce['f1'][-1]:.4f},AUC={metrics_ce['auc'][-1]:.4f}")

print(f"FocalLoss:

ValLoss={val_loss_fl[-1]:.4f},F1={metrics_fl['f1'][-1]:.4f},AUC={metrics_fl['auc'][-1]:.4f}")if__name__=="__main__":main()输出结果：==========TrainingwithFocalLoss==========Epoch1|TrainLoss:0.0579|ValLoss:0.0506|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.474Epoch2|TrainLoss:0.0137|ValLoss:1.7863|Acc:0.035|Recall:1.000|F1:0.068|AUC:0.429Epoch3|TrainLoss:0.0038|ValLoss:0.1148|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.566Epoch4|TrainLoss:0.0011|ValLoss:0.0592|Acc:0.963|Recall:0.000|F1:0.000|AUC:0.556Epoch5|TrainLoss:0.0043|ValLoss:0.2203|Acc:0.568|Recall:0.500|F1:0.075|AUC:0.544Epoch6|TrainLoss:0.0154|ValLoss:0.0588|Acc:0.963|Recall:0.000|F1:0.000|AUC:0.532Epoch7|TrainLoss:0.0063|ValLoss:0.0459|Acc:0.935|Recall:0.143|F1:0.133|AUC:0.594Epoch8|TrainLoss:0.0003|ValLoss:0.0655|Acc:0.963|Recall:0.000|F1:0.000|AUC:0.575Epoch9|TrainLoss:0.0001|ValLoss:0.0665|Acc:0.963|Recall:0.000|F1:0.000|AUC:0.579Epoch10|TrainLoss:0.0000|ValLoss:0.0825|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.584Epoch11|TrainLoss:0.0000|ValLoss:0.0848|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.588Epoch12|TrainLoss:0.0000|ValLoss:0.0839|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.590Epoch13|TrainLoss:0.0000|ValLoss:0.0843|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.590Epoch14|TrainLoss:0.0000|ValLoss:0.0882|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.592Epoch15|TrainLoss:0.0000|ValLoss:0.0860|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.591Epoch16|TrainLoss:0.0000|ValLoss:0.0866|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.594Epoch17|TrainLoss:0.0000|ValLoss:0.0883|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.596Epoch18|TrainLoss:0.0000|ValLoss:0.0889|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.597Epoch19|TrainLoss:0.0000|ValLoss:0.0878|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.598Epoch20|TrainLoss:0.0000|ValLoss:0.0922|Acc:0.965|Recall:0.000|F1:0.000|AUC:0.597==================FinalPerformance==================Cross-EntropyLoss:ValLoss=0.2563,F1=0.0000,AUC=0.6756FocalLoss:ValLoss=0.0922,F1=0.0000,AUC=0.59668.8不使用任何深度学习框架的注意力模块，从零实现一个单头自注意力机制的Transformer编码模块，并在序列分类任务（如情感分析）中测试其有效性。答：示例代码如下。importtorchimporttorch.nnasnnimporttorch.optimasoptimfromtorch.utils.dataimportDataLoader,Datasetfromtorchtext.data.utilsimportget_tokenizerfromtorchtext.vocabimportbuild_vocab_from_iteratorimportmathimportosfromglobimportglobfromtqdmimporttqdmimportwarnings#----------------------------1.单头自注意力（从零实现）----------------------------classSingleHeadSelfAttention(nn.Module):

"""单头缩放点积自注意力"""

def__init__(self,embed_size):

super().__init__()

self.embed_size=embed_size

self.query=nn.Linear(embed_size,embed_size)

self.key

=nn.Linear(embed_size,embed_size)

self.value=nn.Linear(embed_size,embed_size)

self.scale=math.sqrt(embed_size)

defforward(self,x,mask=None):

#x:(batch,seq_len,embed_size)

Q=self.query(x)

K=self.key(x)

V=self.value(x)

scores=torch.matmul(Q,K.transpose(-2,-1))/self.scale

#(B,L,L)

ifmaskisnotNone:

#mask原本形状(B,L)->扩展为(B,1,L)以广播到scores

mask=mask.unsqueeze(1)

#(B,1,L)

scores=scores.masked_fill(mask==0,float('-inf'))

attn=torch.softmax(scores,dim=-1)

out=torch.matmul(attn,V)

returnout#----------------------------2.TransformerEncoderBlock----------------------------classTransformerEncoderBlock(nn.Module):

def__init__(self,embed_size,ff_hidden_dim,dropout=0.1):

super().__init__()

self.attention=SingleHeadSelfAttention(embed_size)

self.norm1=nn.LayerNorm(embed_size)

self.norm2=nn.LayerNorm(embed_size)

self.ffn=nn.Sequential(

nn.Linear(embed_size,ff_hidden_dim),

nn.ReLU(),

nn.Linear(ff_hidden_dim,embed_size)

)

self.dropout=nn.Dropout(dropout)

defforward(self,x,mask=None):

attn_out=self.attention(x,mask)

x=self.norm1(x+self.dropout(attn_out))

ffn_out=self.ffn(x)

x=self.norm2(x+self.dropout(ffn_out))

returnx#----------------------------3.位置编码（正弦余弦）----------------------------classPositionalEncoding(nn.Module):

def__init__(self,embed_size,max_len=5000):

super().__init__()

pe=torch.zeros(max_len,embed_size)

position=torch.arange(0,max_len,dtype=torch.float).unsqueeze(1)

div_term=torch.exp(torch.arange(0,embed_size,2).float()*(-math.log(10000.0)/embed_size))

pe[:,0::2]=torch.sin(position*div_term)

pe[:,1::2]=torch.cos(position*div_term)

self.register_buffer('pe',pe.unsqueeze(0))

defforward(self,x):

returnx+self.pe[:,:x.size(1),:]#----------------------------4.完整的Transformer分类模型（健壮版本）----------------------------classTransformerForSequenceClassification(nn.Module):

def__init__(self,vocab_size,embed_size=128,num_layers=2,ff_hidden_dim=256,

dropout=0.1,max_len=500,num_classes=1):

super().__init__()

self.embedding=nn.Embedding(vocab_size,embed_size)

self.pos_encoder=PositionalEncoding(embed_size,max_len)

self.encoder_blocks=nn.ModuleList([

TransformerEncoderBlock(embed_size,ff_hidden_dim,dropout)

for_inrange(num_layers)

])

self.classifier=nn.Linear(embed_size,num_classes)

self.num_classes=num_classes

defforward(self,x,mask=None):

x=self.embedding(x)

x=self.pos_encoder(x)

forblockinself.encoder_blocks:

x=block(x,mask)

pooled=x.mean(dim=1)

#(batch,embed_size)

logits=self.classifier(pooled)

#(batch,num_classes)

#健壮处理：如果num_classes不是1，则取第一列作为二分类输出并给出警告

iflogits.size(1)!=1:

warnings.warn(f"Expectednum_classes=1,butgot{logits.size(1)}.Usingfirstcolumnaslogits.")

logits=logits[:,0]

#取第一列

else:

logits=logits.squeeze(1)

#(batch,)

returnlogits#----------------------------5.从本地aclImdb目录加载数据---------------------------classLocalIMDBDataset(Dataset):

def__init__(self,data_dir,split='train',vocab=None,max_length=200,tokenizer=None):

"""

data_dir:包含train/和test/的根目录，例如'./data/aclImdb'

split:'train'或'test'

"""

self.max_length=max_length

self.tokenizer=tokenizeriftokenizerelseget_tokenizer('basic_english')

self.split=split

self.raw_texts=[]

self.labels=[]

#构建路径

split_dir=os.path.join(data_dir,split)

pos_dir=os.path.join(split_dir,'pos')

neg_dir=os.path.join(split_dir,'neg')

#读取所有txt文件

pos_files=glob(os.path.join(pos_dir,'*.txt'))

neg_files=glob(os.path.join(neg_dir,'*.txt'))

forfileinpos_files:

withopen(file,'r',encoding='utf-8')asf:

text=f.read()

self.raw_texts.append(text)

self.labels.append(1)

#正面

forfileinneg_files:

withopen(file,'r',encoding='utf-8')asf:

text=f.read()

self.raw_texts.append(text)

self.labels.append(0)

#负面

#如果提供了vocab，立即将文本转为tokenids

ifvocabisnotNone:

self._convert_to_ids(vocab)

else:

self.token_ids=None

#稍后构建词汇表时使用raw_texts

def_convert_to_ids(self,vocab):

self.token_ids=[]

fortextinself.raw_texts:

tokens=self.tokenizer(text)[:self.max_length]

ids=[vocab[token]fortokenintokens]

self.token_ids.append(torch.tensor(ids,dtype=torch.long))

#释放原始文本以节省内存

self.raw_texts=None

def__len__(self):

returnlen(self.labels)

def__getitem__(self,idx):

ifself.token_idsisnotNone:

returnself.token_ids[idx],torch.tensor(self.labels[idx],dtype=torch.float)

else:

#如果尚未转换，则返回原始文本（用于构建词汇表）

returnself.raw_texts[idx],self.labels[idx]defbuild_vocab_from_local(data_dir,tokenizer,max_length=200):

"""从本地数据构建词汇表"""

train_dataset=LocalIMDBDataset(data_dir,split='train',vocab=None,max_length=max_length,tokenizer=tokenizer)

defyield_tokens():

fortext,_intrain_dataset:

yieldtokenizer(text)[:max_length]

vocab=build_vocab_from_iterator(yield_tokens(),specials=['<pad>','<unk>'])

vocab.set_default_index(vocab['<unk>'])

returnvocabdefload_imdb_data(data_dir,batch_size=32,max_length=200):

"""从本地aclImdb目录加载数据，返回DataLoader和vocab信息"""

tokenizer=get_tokenizer('basic_english')

#构建词汇表

print("BuildingvocabularyfromlocalIMDBdata...")

vocab=build_vocab_from_local(data_dir,tokenizer,max_length)

pad_idx=vocab['<pad>']

print(f"Vocabularysize:{len(vocab)}")

#创建带vocab的数据集

train_dataset=LocalIMDBDataset(data_dir,split='train',vocab=vocab,max_length=max_length,tokenizer=tokenizer)

test_dataset

=LocalIMDBDataset(data_dir,split='test',

vocab=vocab,max_length=max_length,tokenizer=tokenizer)

defcollate_fn(batch):

texts,labels=zip(*batch)

#填充至本批次最大长度

lengths=[len(t)fortintexts]

max_len=max(lengths)

padded=