一簡答題1.什么是數據降維？答：從矩陣的角度來看，所處理的數據集就是二維矩陣。如果這個矩陣中充斥了一些多余的、有缺陷的、彼此線性相關的行或列，那么既不利于存儲，也不利于建模和運算。此時需要進行降維處理，把冗余的行或列去掉，這就是數據降維的概念。2.數據降維可以分為行降維和列降維。基于數據篩選的算法應用于________。答：行降維3.列降維的算法有：PCA、SVD、LDA。其中哪些算法是有監督的？哪些算法是無監督的？答：LDA是有監督的，PCA和SVD是無監督的。4.列降維的算法有：PCA、SVD、LDA。其中哪一種算法常常用來做數據壓縮？答：SVD二計算題1根據數據篩選的方法，將表7-5的數據進行行降維操作。其中屬性A-D是預測字段，class是目標字段。取用戶自定義比例m%=40%。行降維以后的數據空缺，采用列均值進行填充。表7-5：XXX數據表屬性A屬性B屬性C屬性DClass1.013.222.541.3412.34Null3.224.2023.224.232.112.13Null1.342.33NullNull22.112.34Null4.232解：（1）按照行降維的原則進行降維：屬性A屬性B屬性C屬性DClass1.013.222.541.3412.34Null3.224.2022.112.34Null4.232（2）行降維以后的數據空缺，采用列均值進行填充：屬性A屬性B屬性C屬性DClass1.013.222.541.3412.342.783.224.2022.112.342.884.2322對于表7-2的牛奶與價格數據進行SVD降維處理，設定信息損失比為90%。利用SVD實現數據壓縮，并對比壓縮前后的存儲空間需求量。表7-2：牛奶容量與價格ID價格/元容量/升12022151.43171.74303.25131.16252.6解：（1）元素數據X=[[20.2.][15.1.4][17.1.7][30.3.2][13.1.1][25.2.6]]（2）對原始數據進行SVD分解：X=U*sigma*VTU=[[-0.39-0.11-0.32-0.66-0.16-0.52][-0.29-0.41-0.140.4-0.740.12][-0.33-0.090.92-0.13-0.07-0.11][-0.590.49-0.110.510.21-0.31][-0.25-0.73-0.090.150.610.02][-0.490.19-0.1-0.320.060.78]]sigma=diag([51.331942990.30270313])VT=[[-0.99-0.1][-0.10.99]]（3）保留1維，計算信息損失比：F1=51.33194299**2F=51.33194299**2+0.30270313**2F1/F=0.9999>90%因此降維以后的數據為：[-0.39-0.29-0.33-0.59-0.25-0.49]T（4）利用SVD進行數據壓縮：需要保存的數據：U1=[-0.39-0.29-0.33-0.59-0.25-0.49]T，diag(51.33)，V1T=[-0.99-0.1]壓縮以后恢復的數據：X1=U1*diag(51.33)*V1T=[[19.996596652.0334677][14.98738251.52407726][16.997107151.72844754][30.015024953.05224858][12.977657851.31970697][25.00581082.54285814]]存儲原始數據需要的空間：2*6=12存儲壓縮數據需要的空間：6+1+2=9三編程題1編寫Python程序，對表7-3的數據進行PCA降維處理。設定降低為2維數據集。代碼：fromsklearn.decompositionimportPCAimportnumpyasnpX=np.array([[1.0,1.5,2.0,2.5,3.0,3.5,4.0,4.5,5.0,5.5],[1.39,2.2,2.72,3.45,4.44,5.22,5.76,6.58,7.31,8.08],[12.88,12.67,10.98,15.29,12.96,9.83,12.34,14.15,12.38,8.77]]).Tprint("X=\n",X)#實例化model=PCA(n_components=2)#擬合模型model=model.fit(X)#獲取新矩陣X_dr=model.transform(X)print("X_dr=\n",X_dr)輸出：X_dr=[[-3.96384765-0.91368507][-3.00385908-0.74830481][-1.70685128-2.0423719][-2.519473832.28169145][-0.618410140.54058736][1.42384171-2.00586786][1.146349040.59349307][1.349371112.63154458][2.838311741.32835828][5.05456837-1.66544509]]2編寫Python程序，對鳶尾花數據集進行LDA降維處理。設定降低為2維數據集。降維之后，利用支持向量機SVC算法進行分類測試，輸出精確率、召回率、f1分數等性能參數。代碼：importmatplotlib.pyplotaspltfromsklearn.datasetsimportload_iris#引入鳶尾花數據集iris=load_iris()y=iris.targetx=iris.datafromsklearn.model_selectionimporttrain_test_splitX_train,X_test,y_train,y_test=train_test_split(x,y,random_state=0)fromsklearn.preprocessingimportStandardScalersc=StandardScaler()x_train_std=sc.fit_transform(X_train)x_test_std=sc.fit_transform(X_test)fromsklearn.discriminant_analysisimportLinearDiscriminantAnalysisasLDAlda=LDA(n_components=2)#做降維處理x_train_lda=lda.fit_transform(x_train_std,y_train)x_test_lda=lda.transform(x_test_std)print("x_train_lda=\n",x_train_lda)print("x_test_lda=\n",x_test_lda)#引入支持向量機模型，并用訓練集進行訓練fromsklearn.svmimportSVCcls=SVC()cls.fit(x_train_lda,y_train)##評估模型y_pred=cls.predict(x_test_lda)fromsklearn.metricsimportclassification_reportdefevaluation(y_test,y_predict):accuracy=classification_report(y_test,y_predict,output_dict=True)['accuracy']s=classification_report(y_test,y_predict,output_dict=True)['weightedavg']precision=s['precision']recall=s['recall']f1_score=s['f1-score']returnaccuracy,precision,recall,f1_scorelist_evalucation=evaluation(y_test,y_pred)print("accuracy:{:.3f}".format(list_evalucation[0]))print("weightedprecision:{:.3f}".format(list_evalucation[1]))print("weightedrecall:{:.3f}".format(list_evalucation[2]))print("F1score:{:.3f}".format(list_evalucation[3]))#####################################定義一個畫圖函數###########################importnumpyasnpfrommatplotlib.colorsimportListedColormapdefplot_decision_regions(X,y,classifier,resolution=0.02):#setupmarkergeneratorandcolormapmarkers=('s','x','o','^','v')colors=('red','blue','lightgreen','gray','cyan')cmap=ListedColormap(colors[:len(np.unique(y))])#plotthedecisionsurfacex1_min,x1_max=X[:,0].min()-1,X[:,0].max()+1x2_min,x2_max=X[:,1].min()-1,X[:,1].max()+1xx1,xx2=np.meshgrid(np.arange(x1_min,x1_max,resolution),np.arange(x2_min,x2_max,resolution))Z=classifier.predict(np.array([xx1.ravel(),xx2.ravel()]).T)Z=Z.reshape(xx1.shape)plt.contourf(xx1,xx2,Z,alpha=0.4,cmap=cmap)plt.xlim(xx1.min(),xx1.max())plt.ylim(xx2.min(),xx2.max())#plotclasssamplesforidx,clinenumerate(np.unique(y)):plt.scatter(x=X[y==cl,0],y=X[y==cl,1],alpha=0.6,edgecolor='black',marker=markers[idx],label=cl)plot_decision_regions(x_train_lda,y_train,classifier=cls)plt.legend(loc='lowerleft')plt.title("svcoftrain_LDA")plt.tight_layout()plt.show()plot_decision_regions(x_test_lda,y_test,classifier=cls)plt.legend(loc='lowerleft')plt.title("svcoftrain_LDA")plt.tight_layout()plt.show()程序輸出：accuracy:0.921weightedprecision:0.941weightedrecall:0.921F1score:0.9233編寫spark程序，對葡萄酒數據進行PCA降維處理。輸出每個主成分維度的解釋方差，并以此作為參考指標，調整保留的維度數量，使得每個主成分維度的解釋方差之和大于等于70%。代碼：importorg.apache.log4j.{Level,Logger}importorg.apache.spark.ml.feature.{PCA,StandardScaler,StringIndexer,VectorAssembler}importorg.apache.spark.sql.SparkSessionobjectxiti07_05{defmain(args:Array[String]):Unit={Logger.getLogger("akka").setLevel(Level.OFF)Logger.getLogger("org").setLevel(Level.OFF)valspark=SparkSession.builder().master("local[*]").appName("aaa").getOrCreate()valdf01=spark.read.option("inferSchema",true).csv("wine.data")df01.show(5)vallabelIndex=newStringIndexer().setInputCol("_c0").setOutputCol("label")valdf02=labelIndex.fit(df01).transform(df01)df02.show(5)valfeatureArray=Array("_c1","_c2","_c3","_c4","_c5","_c6","_c7","_c8","_c9","_c10","_c11","_c12","_c13")valassembler=newVectorAssembler().setInputCols(featureArray).setOutputCol("features")valdf03=assembler.transform(df02)df03.show(5,false)valscaler=newStandardScaler().setInputCol("features").setOutputCol("scalerFeatures").setWithStd(true).setWithMean(true)valdf04=scaler.fit(df03).transform(df03)df04.show(5,false)valpca=newPCA().setInputCol("scalerFeatures").setOutputCol("pcaFeatures").setK(4).fit(df04)valdf05=pca.transform(df04)df05.select("pcaFeatures").show(5,false)println(pca.explainedVariance)}}程序輸出：+----------------------------------------------------------------------------------+|pcaFeatures