1、CNN的早期歷史卷積神經網絡CNNK. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, vol. 36, pp. 193202, 1980Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “

2、Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541551, 1989Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 22782324, 19981DL時代的CNN擴展A Krizhe

3、vsky, I Sutskever, GE Hinton. ImageNet classification with deep convolutional neural networks. NIPS2012Y. Jia et al. Caffe: Convolutional Architecture for Fast Feature Embedding. ACM MM2014K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint a

4、rXiv:1409.1556, 2014C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014)2卷積示例3卷積形式化4卷積why?1. sparse interactions有限連接，Kernel比輸入小連接數少很多，學習難度小，計算復雜度低m個節點與n個節點相連O(mn)限定k(m)個節點與n個節點相連，則為O(kn)

5、5卷積why?1. sparse interactions有限連接，Kernel比輸入小連接數少很多，學習難度小，計算復雜度低m個節點與n個節點相連O(mn)限定k(m)個節點與n個節點相連，則為O(kn)6卷積why?1. sparse interactions有限(稀疏)連接Kernel比輸入小局部連接連接數少很多學習難度小計算復雜度低層級感受野（生物啟發）越高層的神經元，感受野越大7卷積why?2. Parameter Sharing（參數共享）Tied weights進一步極大的縮減參數數量3. Equivariant representations等變性配合Pooling可以獲得平移

6、不變性對scale和rotation不具有此屬性8CNN的基本結構三個步驟卷積突觸前激活，net非線性激活DetectorPoolingLayer的兩種定義復雜定義簡單定義有些層沒有參數9Pooling10定義（沒有需要學習的參數）replaces the output of the net at a certain location with a summary statistic of the nearby outputs種類max pooling (weighted) average poolingWhy Pooling?11獲取不變性小的平移不變性：有即可，不管在哪里很強的先驗假設Th

7、e function the layer learns must be invariant to small translationsWhy Pooling?12獲取不變性小的平移不變性：有即可，不管在哪里旋轉不變性？9個不同朝向的kernels（模板）0.20.610.10.50.30.020.050.1Why Pooling?13獲取不變性小的平移不變性：有即可，不管在哪里旋轉不變性？9個不同朝向的kernels（模板）0.50.30.0210.40.30.60.30.1Pooling與下采樣結合更好的獲取平移不變性更高的計算效率（減少了神經元數）14從全連接到有限連接部分鏈接權重被強制設

8、置為0通常：非鄰接神經元，僅保留相鄰的神經元全連接網絡的特例，大量連接權重為015Why Convolution & Pooling？a prior probability distribution over the parameters of a model that encodes our beliefs about what models are reasonable, before we have seen any data.模型參數的先驗概率分布(No free lunch)在見到任何數據之前，我們的信念（經驗）告訴我們，什么樣的模型參數是合理的Local connections；對

9、平移的不變性；tied weigts來自生物神經系統的啟發16源起：Neocognitron (1980)SimplecomplexLower orderhigh order17K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, vol. 36, pp. 193202, 1980Local Connection源起：Ne

10、ocognitron (1980)18源起：Neocognitron (1980)訓練方法分層自組織competitive learning 無監督輸出層獨立訓練有監督19LeCun-CNN1989用于字符識別簡化了Neocognitron的結構訓練方法監督訓練BP算法正切函數收斂更快，Sigmoid Loss，SGD用于郵編識別大量應用20LeCun-CNN1989用于字符識別輸入16x16圖像L1H112個5x5 kernel8x8個神經元L2-H212個5x5x8 kernel4x4個神經元L3H330個神經元L4輸出層10個神經元總連接數5*5*12*64+5*5*8*12*16+19

11、2*30，約66,000個21LeCun-CNN1989用于字符識別Tied weights對同一個feature map，kernel對不同位置是相同的！22LeCun-CNN1989用于字符識別231998年LeNet數字/字符識別LeNet-5Feature mapa set of units whose weighs are constrained to be identical. 241998年LeNet數字/字符識別 例如：C3層參數個數(3*6+4*9+6*1)*25 + 16 = 1516 25后續：CNN用于目標檢測與識別26AlexNet for ImageNet (201

12、2)大規模CNN網絡650K神經元60M參數使用了各種技巧DropoutData augmentReLULocal Response NormalizationContrast normalization.27Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems. 2012.AlexNet for Image

13、Net (2012)ReLU激活函數28AlexNet for ImageNet (2012)實現2塊GPU卡輸入層150,528其它層253,440186,624 64,896 64,896 43,264 4096 4096 1000.29Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems. 2012.A

14、lexNet for ImageNet (2012)ImageNet物體分類任務上1000類，1,431,167幅圖像30RankNameError rates(TOP5)Description1U. Toronto0.153Deep learning2U. Tokyo0.261Hand-crafted features and learning models.Bottleneck.3U. Oxford0.2704Xerox/INRIA0.271Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classifi

15、cation with deep convolutional neural networks.Advances in neural information processing systems. 2012.AlexNet for ImageNet深度的重要性31網絡深度87664參數數量60M44M10M59M10M性能損失0%1.1%5.7%3.0%33.5%Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural netwo

16、rks.Advances in neural information processing systems. 2012.VGG Net (2014)多個stage每個stage多個卷積層卷積采樣間隔1x1卷積核大小3x31個Pooling層(2x2)16-19層多尺度融合K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014VGG Net (2014)幾種配置Cov3-64:3x3感受野6

17、4個channel33K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014VGG Net (2014)34K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014GoogLeNet (

18、2014)超大規模22個卷積層的網絡計算復雜度是AlexNet的4倍左右C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014)GoogLeNet (2014)超大規模24層網絡Inception結構提取不同scale的特征然后串接起來1x1 convolutions3x3 convolutions5x5 convolut

19、ionsFilter concatenationPrevious layerC. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A.Rabinovich. Going deeper with convolutions. CVPR2015 (&arXiv:1409.4842, 2014)GoogLeNet (2014)超大規模24層網絡Inception結構提取不同scale的特征，然后串接起來增加1x1的卷積：把響應圖的數量縮小了1x1 convolutions3x3 convolutions5x5 convolutionsFilter concatenationPre