1、Chapter 4 Multiple Feedforward Neural Network4.1 Error Backpropagation Algorithm4.2 Improving on BP Algorithm4.3 Practical aspects of neural computing4.4 Introduction to special neural network architectures4.5 Applications4.1 Error Backpropagation AlgorithmError Back-PropagationInputlayerOutputlayer

2、Hidden layer+-DesiredoutputNeuron jwijxiw1jw2jwijErrorx1x2xjYj,desiredYj = g(hj)Transfer functionujk j vjkulk+1 l vlk+1 uik-1 i vik-1wijwjlAssumption:Transfer function : SigmoidInput sample (xk, yk) , k=1,2,Nujk : Total input to jth neuron on the kth layervjk : Output from jth neuron on the kth laye

3、rwij : interconnecting weight between the ith node of layer k-1 and the jth node of layer kjmjjkjkjikiijkjvyEufvvwu21)(21)(Where, yj is the desired outputmin E(w, x)ijijwEwijijwEw is the learning rateujk j vjkulk+1 l vlk+1 uik-1 i vik-1wijwjl11)(kikjikiijijkjijkjkjijvuEvwwuEwuuEwE1kikjijvuEwAssuming

4、:)( kjkjkjkjkjkjkjufvEuvvEuE)( 11kjkikjkikjijufvvEvwLearning formulaujk j vjkwijSigmoid function:)1 ()(1)()( )exp(11)(kjkjkjkjkjkjkjvvufufufuuf)( 11kjkikjkikjijufvvEvwLearning formula1. jth node on the output layer mjmjjvyE2)(21)(mjjkjvyvE)(1 ()( mjjmjmjmjkjmjvyvvufvE2. jth node on the hidden layer

5、kkjklklkjvuuEvE11ljlkjklklklwvuuE111,lkljlkjkjkjwvv1)1 (ujk j vjkulk+1 l vlk+1 uik-1 i vik-1wijwjl)( 11kjkikjkikjijufvvEvw)( 11kjkikjkikjijufvvEvw)(1(mjjmjmjmjvyvvlkljlkjkjkjwvv1)1 (wij,new= wij,old+ wijBP algorithm:New weightfactor=Old weightfactor+LearningrateInputtermGradient-descentcorrection te

6、rmNN with thresholds11kbiasvkjkbiasjw,kjkbiasoldjkbiasnewjww,Calculating steps of BP algorithm1. Randomly assign values for weights wij2. Repeat the following procedures until the NN is converged(1) forward activation flow to calculate vjk on each layer backward error flow to calculate jk through th

7、e network(2) Adjust weight factorswij,new= wij,old+ wijAn illustrative example: fault diagnosisInput and output for fault-diagnosis networkInput vectorOutput vector I1: reactor inlet temperature, 0F c1 : low conversion I2: reactor inlet pressure, psia c2 : low catalyst selectivity I3: feed flow rate

8、, 1b/min c3 : catalyst sinteringThe desired output,dk from the neural network is Boolean: 0 indicates that no operational fault exist, 1 indicates that a fault does exist.The actual output from the neural network is a numeric value between 0 and 1, and can be viewed almost the “probability” that a g

9、iven input produce the operational fault. 0 = the fault definitely will not occur, 1 = the fault definitely will occur.Specific conditions:Input:I1=300/1000 0F=0.3Output:d1=1 (low conversion)I2=100/1000 psia=0.1d2=0 (no problem)I3=200/1000 lb/min=0.2d3=0 (no problem)In neural network training, we re

10、commend normalizing the input and output values to a finite range, such as 0,1 or -1,1.A three-layer perceptron neural networkAttempts to properly map given inputs with desired outputs by minimizing an error function.The total mean-square error function:kkkkkcdE)(2here, k is the output error vector

11、from the kth node on the output layer dk is the desired output value ck is the calculated value Step 1:Randomly specify numerical values for all weight factors (vijs and wiks) within the interval -1,+1. Likewise, assign internal threshold values (Tai, TBj, TCk) for every node, also between 1 and +1.

12、5 . 05 . 05 . 05 . 00 . 00 . 15 . 05 . 00 . 1ijv5 . 05 . 05 . 05 . 00 . 00 . 15 . 05 . 00 . 1jkwFor the internal threshold values (T1i, T2j, T3k), we set T1i=0 because usually there is no threshold for every node on the first layerTo use computer, let layer A=layer 1, layer B=layer 2, and layer C=la

13、yer 3.5 . 05 . 00 . 05 . 00 . 05 . 00 . 00 . 00 . 0ijTExample:Step 2:Introduce the input Ii into the neural network and calculate all outputs from the first layer, using the standard sigmoid function.ixiAiiiaexfTIxi11)(Example:2 . 002 . 01 . 001 . 03 . 003 . 0133312221111TIxTIxTIx54983. 011)(52498.

14、011)(57444. 011)(321332211xxxexfaexfaexfaWe introduce the input vector into the neural network and calculate outputs from layer 1:Note that had we not normalized our input values to the ranger 0,1, we would have the following results:200020010001003000300133312221111TIxTIxTIx0 . 111)(0 . 111)(0 . 11

15、1)(321332211xxxexfaexfaexfaIf I1=300 0F, then a1=1. If I1=150 0F, then a1=1 again.That is, the sigmoid transfer function is insenstitive to changes in the input when I12.Step 3:Given the output layer A, calculate the output from layer B, using the equation:LiBjiijjTavfb1)(where f ( ) is the same sig

16、moid function.Example:We calculate the output from each node in layer 2:43179. 0)27455. 0()5 . 054983. 0*5 . 052498. 0*0 . 157444. 0*0 . 1()(213312211111ffTavavavfb68990. 0)79965. 0()(36305. 0)56214. 0()(233332231133223322221122fTavavavfbfTavavavfbStep 4:Given the output from layer B, calculate the

17、output from layer C, using the equation:mjckjjkkTbwfc1)(Where f ( ) is the same sigmoid functionExample:We calculate the output from each node in layer 3:77598. 0)24237. 1 ()(25715. 0)06085. 1()(56862. 0)27621. 0()(333332231133323322221122313312211111fTbwbwbwfcfTbwbwbwfcfTbwbwbwfcObviously, the actu

18、al output values ck deviate from the desired output values (d1=1, d2=0, and d3=0).Step 1 to 4 represent the forward activation flow.The next few steps of the backpropagation algorithm represent the backward error flow in which we propagate the error between the desired dk (k=1 to 3) and the actual o

19、utput ck backward through the network, and try to find the best set of network parameters (vij, wij, and Tij).Step 5:Now backpropagate the error through the network, starting at the output and moving backward toward the input. Calculate the kth component of the output error, k, for each node in laye

20、r C, according to the equation: k=ck (1-ck) (dk-ck)Example:We backpropagate, first calculating the error for each node in layer 3:13498. 0)(1 (04912. 0)(1 (10581. 0)(1 (333332222211111cdcccdcccdcc)(1 ()( mjjmjmjmjkjmjvyvvufvEStep 6:Continue backpropagation, moving to layer B. Calculate the jth compo

21、nent of the error vector, ej, of layer B relative to each k, using the equation:nkkjkjjjwbbe1)()(1 (Example:We calculate the errors associated with layer 2, the hidden layer:002144. 0)(1 (008872. 0)(1 (03648. 0)(1 (333232131333323222121222313212111111wwwbbewwwbbewwwbbelkljlkjkjkjwvv1)1 (Step 7:Adjus

22、t weight factors, calculating the new wjk, i.e., wjk,new, as:New weightfactor=Old weightfactor+LearningrateInputtermGradient-descentCorrection term)(1 (,kkkkjcoldjknewjkkjcoldjknewjkcdccbwworbwwExample:We adjust weight factors for interconnections between layer 2 and 3.For simplicity, we assume that

23、 =0.7 for all values of .4593. 05149. 09680. 03113,132112,121111,11bwwbwwbwwnewnewnewWe continue these adjustments for the rest of the wjks.A comparison of the old and new weight factors shows:5 . 05 . 05 . 05 . 00 . 00 . 15 . 05 . 00 . 1,oldjkw4349. 05237. 05511. 04657. 00125. 00269. 14593. 05149.

24、09680. 0,newjkwStep 8:Adjust the thresholds Tck (k=1 to n) in layer C,according to the equation: Tck,new=Tck+ c kExample:We adjust the internal thresholds for layer three:5944. 04656. 00741. 0333,33232,32131,31TTTTTTnewnewnewThus, new and old threshold values are: old : T31=0.0 T32=0.5 T33=-0.5 new:

25、 T31=0.0741 T32=0.4656 T33=-0.5944Step 9:Adjust weight factors vij for interconnections between layer 1 and 2, according to the equation: vij,new= vij + BaiejExample:Again, =0.7.5009. 04964. 00147. 13113,132112,121111,11eavveavveavvnewnewnewWe continue these adjustments for the rest of the vijs, and

26、 the old and new weight factors are: 5 . 05 . 05 . 05 . 00 . 00 . 15 . 05 . 00 . 1ijv5008. 04966. 04860. 04992. 00033. 09866. 05009. 04964. 00147. 1,newijvStep 10:Adjust the thresholds TBj (j=1 to m) in layer B,according to the equation:jBBjnewBjeTT,Example: We adjust the internal thresholds for lay

27、er three:4985. 00062. 04745. 0323,23222,22121,21eTTeTTeTTnewnewnewThus, new and old internal threshold values are:Old : T21=0.5 T22=0.0 T23=-0.5New : T21=0.4745 T22=0.0062 T23=-0.4985Step 11:Repeat steps 2-10 until the squared error, E, or the output-error vector, , is zero or sufficiently small.Exa

28、mple:This problem requires 3860 time steps to get results less than 2% error on variable dk.Number of time steps: 3860Desired values: d1=1 d2=0 d3=0Actual values: c1=0.9900 c2=0.0156 c3=0.0098Percent error: e1=1.00% e2=1.56% e3=0.98%4.2 Improving on BP Algorithm4.2.1 Generalized Delta-Rule Algorithm

29、 and Its Application to Fault DiagnosisThe delta rule uses a technique known as momentum to speed up the training.Momentum is an extra weight added onto the weight factors when they are adjusted. By accelerating the change in the weight factors, we improve the training rate.A. Adding momentum term+

30、momentumNew weightfactor=Old weightfactor+LearningrateInputtermGradient-descentcorrection term+MomentumcoefficientPreviousweight changeHere, the momentum coefficient, , is restricted such that 0 E(n-1) Then e-E(n) e-E(n-1) , (n) 0.30.1390.0950.0620.0450.0350.0260.140390.0820.0540.0370.0350.

31、030.1380.0630.0380.03300.090.3A typical learning schedule for a 3-hidden-layer BP networkF. Selecting the proper transfer functionA comparison of the effectiveness of the hyperbolic tangent and sigmoid transfer function in training a prediction network

32、G. Generating a network learning curveA general procedure for generating a learning curve below:1. Divide all variable training examples into selected groups(t1,t2,t7).2. Use one group to train the network.3. Test the network for both recall of the training data set and generalization of the remaini

33、ng data groups not used in network training.4. Add one randomly selected generalization data group to the training data set.5. Retrain the network using the new training data set, reducing the learning rates and momentum coefficients. (this is done to maintain some of the integrity of the previously trained network.)6. Repeat steps 2-5 until there are no more groups remaining in the generalization data set.TrainingRecallGeneralizationGroup numbers in training setExperiments in training setExperiments in testing setAvg error reaction rateAvg error reaction rate11