推荐系统案例-第二天3mllib与

MLlib与MLlib与推荐系主要内1SparkMLlib1SparkMLlib2MLlib推荐算法3MLlib推荐算法4总主要内1SparkMLLib1SparkMLLib2MLlib推荐算法3MLlib推荐算法4总SparkSparkMLlibMLLib是基于Spark引擎实现的机器学良好的扩展性和容充分利用了Spark扩展性属于Spark生态系统重要组成实现了大部分常用的数据挖聚类分类推荐

MLlib分类算(summarystatisticsstratifiedsamplingrandomdatagenerationClassificationandregression分类一般针对离散型数linearmodels(SVMs,logisticregressionlinearregressionnaive decisiontreesensemblesoftrees(RandomForestsandGradient-BoostedTreesMLlib分类算 alternatingleastsquaresALS)交替最小二乘法(ALS singular position 奇异值分principal ysis(PCA)主成分分stochasticgradientdescent随机梯度下limited-memoryBFGS(L-BFGS)短时 的BFGS(拟 DeveloperApi在未来的发布种可能会被修改主要内1SparkMLlib1SparkMLlib2MLlib推荐算法3MLlib推荐算法4总MLlib协同过滤算MLlibMLlib基于ALS（alternatingleastsquares）的协同numBlocks:计算并行度(若为-1表示自动化配置Rank：模型中隐含影响因子，默认是Iterations：迭代次数，默认是Lambda：ALS中正则化参implicitPrefs：是否使用显式反馈变量或使用隐式反馈数据的变Alpha：ALS中的一个参数，作用于隐式反馈变量，控制基本的信心主要内1SparkMLlib1SparkMLlib2MLlib推荐算法3MLlib推荐算法4总MLlibMLlibALS加载数据将数据集解析成ALS要求的格将数据集分割成两部分：训练集和测试运行ALS，产生并评估模将最终模型用于推MLlibMLlibALS推荐步骤1valvalconf=newvalsc=newvalvalmovieLensHomeDir=valdataSet=sc.textFile(newFile(movieLensHomeDir,MLlibMLlibALS推荐步骤2valvalrating=dataSet.map{linevalfields=//format:(timestamp%10,Rating(userId,movieId,}MLlibMLlibALS推荐步骤3valnumPartitions=valtraining=ratings.filter(x=>x._1<valvalidation=ratings.filter(x=>x._1>=6&&x._1<valtest=ratings.filter(x=>x._1>=valnumTraining=training.count()valnumTest=test.count()MLlibALSMLlibALS推荐步骤4：运行defdeftrain(ratings:RDD[Rating],rank:Int,iterations:Int,lambda::MatrixFactorizationModel=你的代}MLlibALSMLlibALS推荐步骤4(2)：运行ranks=List(8,vallambdas=List(1.0,valnumIters=List(10,varbestModel:Option[MatrixFactorizationModel]=NonevarbestValidationRmse=Double.MaxValuevarbestRank=varbestLambda=-1.0varbestNumIter=-1//findbestmodelherevaltestRmse=computeRmse(bestModel.get,test,println("Thebestmodelwastrainedwithrank="+bestRank+"andlambda="++",andnumIter="+bestNumIter+",anditsRMSEonthetestsetis"+testRmse+MLlibALS推荐步骤4(2)：运行for(rank<-ranks;lambda<-lambdas;numIter<-numIters)valmodel=ALS.train(training,rank,numIter,valvalidationRmse=computeRmse(model,validation,numValidation)println("RMSE(validation)="+validationRmse+"forthemodeltrainedwithrank=+rank+",lambda="+lambda+",andnumIter="+numIter+".")if(validationRmse<bestValidationRmse){bestModel=Some(model)bestRank=rankbestLambda=bestNumIter=}}MLlibALS推荐步骤MLlibALS推荐步骤5：使用模 //...你的代}valcandida