1、2015/8/29caffe源碼分析­­data_layer.cpp ­ linger(心懷夢想，活在當下) ­ 博客頻道 ­ CSDN.NET登錄 |linger(心懷夢想，活在當下) 學習，深度學習，數據挖掘，推薦系統，分布式算法目錄視圖 摘要視圖 訂閱個人資料caffe源碼分析-data_layer.cpp分類： 深度學習（deep learning）2014­05­31 21:24 5375人閱讀 評論(5) 收藏神經網絡 caffe 深度學習 deep learninglingerlanlandataLayer作為

2、整個網絡數據從leveldb中取。lev過圖片轉換過來的。網絡建立的時候，datalayer主要是負責設置一些參數，比如batchsize，channels，height，width等。這次會通過讀leveldb一個數據塊來獲取這些信息。然后啟動一個線程來預先從leveldb拉取一批數據，這些數據是圖像數據和圖像 。正向 的時候，： 156篇 ： 0篇譯文： 2篇 評論： 260條datalayer就把預先拉取好數據拷貝到指定的cpu或者gpu的內存。然后啟動新線程再預先拉取數據，這些數據留到下一次正向文章搜索文章分類深度學習（deep learning） (28)學習 (16)cuda(GP

3、U programming) (20)文本挖掘 (5)c/c+ (15)dota (5)hack programming (6)web2.0 (5)工具源碼 (7)語言學習 (22)設計模式 (2)讀書筆記 (1)翻譯 (1)足球大數據 (2)大雜燴 (18)Hadoop (12)Spark (2)sklearn (1)文章存檔2015年08月 (1)1/601.02.03.04.05.06.07.08.07.28.29.cpp/ Copyright 2013 Yangqing Jia#

4、include <stdint.h> #include <leveldb/db.h> #include <pthread.h>#include <string> #include <vector>#include "caffe/layer.hpp" #include "caffe/util/io.hpp" #include "caffe/vision_layers.hpp"using std:string;namespace caffe template <typena

5、me Dtype>void* DataLayerPrefetch(void* layer_pointer) CHECK(layer_pointer);DataLayer<Dtype>* layer = reinterpret_cast<DataLayer<Dtype>*>(layer_pointer); CHECK(layer);Datum datum; CHECK(layer>prefetch_data_);Dtype* top_data = layer>prefetch_data_>mutable_cpu_data();/數據Dt

6、ype* top_label = layer>prefetch_label_>mutable_cpu_data();/ const Dtype scale = layer>layer_param_.scale();const int batchsize = layer>layer_param_.batchsize(); const int cropsize = layer>layer_param_.cropsize();： 219485次： 3991等級：排名： 第3738名e2015/8/29caffe源碼分析­­data_layer.cpp

7、 ­ linger(心懷夢想，活在當下) ­ 博客頻道 ­ CSDN.NET2015年07月 (3)2015年06月 (3)2015年05月 (3)2015年04月 (8)2.const bool mirror = layer>layer_param_.mirror();if (mirror && cropsize = 0) /當前實現需要同時設置mirror和cropsize LOG(FATAL) << "Current implementati

8、on requires mirror and cropsize to be "<< "set at the same time."/ datum scalesconst int channels = layer>datum_channels_; const int height = layer>datum_height_; const int width = layer>datum_width_;const int size = layer>datum_size_;const Dtype* mean = layer>dat

9、a_mean_.cpu_data();for (int itemid = 0; itemid < batchsize; +itemid) /每一批數據的數量是batchsize，一個循環拉取一張？/ get a blob CHECK(layer>iter_); CHECK(layer>iter_>Valid();datum.ParseFromString(layer>iter_>value().ToString();/利用迭代器拉取下一批數據const string& data = datum.data(); if (cropsize) /如果需要裁

10、剪CHECK(data.size() << "Image cropping only support uint8 data" int h_off, w_off; 展開最新評論總結一下用caffe跑圖片數據的研liangzhituzi: zzq1989_:可能是那兩個文件路徑的問題，可以看看train_prototxt里面的路徑deep learning實踐經驗總結查志強: 問下，怎樣 “錯誤”的？神經網絡：caffe特征可視化的代fqss0436: 博主，您好， 您代碼。在調試您的代碼時，程序中斷于175行caffe_test_net.For.我所寫的 框架 V

11、S caffe gzp95: 樓主碼的速度和caf差距。因為最近在的cud.總結一下用caffe跑圖片數依然_范佩西11: 訓練完的模型如是調用呢，能說下能么測試單張圖像或者批量圖像的流程么Dota全圖那些事兒女圭丶女圭: 。單機理論效果，實際不好用啊。支持一下不錯的caffe源碼修改：抽取任意一張圖wwdzhtxnjwcnmd: 想請教一下博主，caffe網絡中batch_size和crop_size這兩個參數的含義是什么？哪一.caffe源碼分析­­data_layer.cpp 滄海1夢: 請問caffe中如何修改輸入和裁剪小是48的，想通過修改alexnet來訓練，還

12、.caffe卷積神經網絡框架安裝yang123jx: 我也遇到relu_layer.cu:29 check failed error = cudaSuc.caffe卷積神經網絡框架安裝yang123jx: 我也遇到relu_layer.cu:29 check failed error = cudaSuc.9.50.596./if (mi/ Copy for (int c0; h < cropsize; +h) 3.8for

13、 (int w =) top_data(ite+ cropsiz (static_cast<Dtyp >(uint8_t)da+ c) * cropsize + h) * cropsize w =* height + h + h_off) * width w_off)h_off) * width + w + w_off) mean* sca el閱讀排行總結一下用caffe跑圖片數 92)word2vector學習筆記（ 42)caffe神經網絡框架的輔助 7)caffe源碼修改：抽取任意 ) caffe卷積神經網絡框架安caffe源碼分析­­data_lay神

14、經網絡：caffe特征 視word2vec源碼caffe源碼分析­­Blob類代 (4386)deep learning實踐經驗 (4225)0007. else /如果不需要裁剪/ we will prefer to use data() first, and then try float_data()/我們優先考慮data()，然后float_data() if (data.size() for (int j = 0; j < size; +j) top_data

15、itemid * size + j =(static_cast<Dtype>(uint8_t)dataj) meanj) * scale; else for (int j = 0; j < size; +j) top_dataitemid * size + j =(datum.float_data(j) meanj) * scale;推薦文章2/62015/8/29caffe源碼分析­­data_layer.cpp ­ linger(心懷夢想，活在當下) ­ 博客頻道 ­ CSDN.NET11.112

16、.19.120.top_labelitemid = datum.label();/ go to the next iter layer>iter_>Next();if (!layer>iter_>Valid() / We have reached the end. Restart from the first. DLOG(INFO) << "Restarting data prefetching from start." layer>iter_>SeekToFirst();re

17、turn reinterpret_cast<void*>(NULL);121. 122.123. template <typename Dtype>124. DataLayer<Dtype>:DataLayer<Dtype>() 125.126./ Finally, join the threadCHECK(!pthread_join(thread_, NULL) << "Pthread joining failed."127. 128.129. template <typename Dtype>130

18、. void DataLayer<Dtype>:SetUp(const vector<Blob<Dtype>*>& bottom,82.183.1

19、84.185.186.vector<Blob<Dtype>*>* top) CHECK_EQ(bottom.size(), 0) << "Data Layer takes no input blobs." CHECK_EQ(top>size(), 2) << "Data Layer takes two blobs as output."/ Initialize the leveldb leveldb:DB* db_temp; leveldb:Options options; options.creat

20、e_if_missing = false; options.max_open_files = 100;LOG(INFO) << "Opening leveldb " << this>layer_param_.source(); leveldb:Status status = leveldb:DB:Open(options, this>layer_param_.source(), &db_temp); CHECK(status.ok() << "Failed to open leveldb "<

21、< this>layer_param_.source() << std:endl << status.ToString(); db_.reset(db_temp);iter_.reset(db_>NewIterator(leveldb:ReadOptions();/通過迭代器來 leveldbiter_>SeekToFirst();/ Check if we would need to randomly skip a few data points/是否要隨機跳過一些數據if (this>layer_param_.rand_skip() /

22、 NOLINT_NEXT_LINE(runtime/threadsafe_fn)unsigned int skip = rand() % this>layer_param_.rand_skip(); LOG(INFO) << "Skipping first " << skip << " data points." while (skip > 0) /循環次數iter_>Next();if (!iter_>Valid() iter_>SeekToFirst();/ Read a data p

23、oint, and use it to initialize/可以觀察到下面iter_調用調用next。所以這次Datum datum;datum.ParseFromString(iter_>value().ToString();/利用迭代器/ image圖像數據int cropsize = this>layer_param_.cropsize();/裁剪大小if (cropsize > 0) /需要裁剪(*top)0>Reshape(this>layer_param_.batchsize(), datum.channels(), cropsize, cropsi

24、ze); prefetch_data_.reset(new Blob<Dtype>(this>layer_param_.batchsize(), datum.channels(), cropsize, cropsize); else /不需要裁剪(*top)0>Reshape(this>layer_param_.batchsize(), datum.channels(), datum.height(), datum.width();prefetch_data_.reset(new Blob<Dtype>(this>layer_param_.bat

25、chsize(), datum.channels(), datum.height(), datum.width();LOG(INFO) << "output data size: " << (*top)0>num() << ","<< (*top)0>channels() << "," << (*top)0>height() << ","<< (*top)0>width();/ label 數據(

26、*top)1>Reshape(this>layer_param_.batchsize(), 1, 1, 1); prefetch_label_.reset(new Blob<Dtype>(this>layer_param_.batchsize(), 1, 1, 1);3/62015/8/29caffe源碼分析­­data_layer.cpp ­ linger(心懷夢想，活在當下) ­ 博客頻道 ­ CSDN.NET99

27、.00045./ datum sizedatum_channels_ = datum.channels(); datum_height_ = datum.height(); datum_width_ = datum.width();datum_

28、size_ = datum.channels() * datum.height() * datum.width(); CHECK_GT(datum_height_, cropsize);CHECK_GT(datum_width_, cropsize);/ check if we want to have mean是否要減去均值if (this>layer_param_.has_meanfile() BlobProto blob_proto;LOG(INFO) << "Loading mean file from" << this>laye

29、r_param_.meanfile(); ReadProtoFromBinaryFile(this>layer_param_.meanfile().c_str(), &blob_proto); data_mean_.FromProto(blob_proto);CHECK_EQ(data_mean_.num(), 1); CHECK_EQ(data_mean_.channels(), datum_channels_); CHECK_EQ(data_mean_.height(), datum_height_); CHECK_EQ(data_mean_.width(), datum_w

30、idth_); else / Simply initialize an allempty mean.data_mean_.Reshape(1, datum_channels_, datum_height_, datum_width_);/ Now, start the prefetch thread. Before calling prefetch, we make two/ cpu_data calls so that the prefetch thread does not accidentally make/ simultaneous cudaMalloc calls when the

31、main thread is running. In some/ GPUs this seems to cause failures if we do not so. prefetch_data_>mutable_cpu_data(); prefetch_label_>mutable_cpu_data(); data_mean_.cpu_data();DLOG(INFO) << "Initializing prefetch" CHECK(!pthread_create(&thread_, NULL, DataLayerPrefetch<

32、Dtype>,reinterpret_cast<void*>(this) << "Pthread execution failed." DLOG(INFO) << "Prefetch initialized."template <typename Dtype>void DataLayer<Dtype>:Forward_cpu(const vector<Blob<Dtype>*>& bottom, vector<Blob<Dtype>*>

33、;* top) / First, join the thread 等待線程結束CHECK(!pthread_join(thread_, NULL) << "Pthread joining failed."/ Copy the data拷貝數據到top，即該層的輸出memcpy(*top)0>mutable_cpu_data(), prefetch_data_>cpu_data(),sizeof(Dtype) * prefetch_data_>count(); memcpy(*top)1>mutable_cpu_data(), prefet

34、ch_label_>cpu_data(),sizeof(Dtype) * prefetch_label_>count();/ Start a new prefetch thread啟動新線程拉取下一批數據CHECK(!pthread_create(&thread_, NULL, DataLayerPrefetch<Dtype>,reinterpret_cast<void*>(this) << "Pthread execution failed."/ The backward operations are dummy t

35、hey do not carry any computation. template <typename Dtype>Dtype DataLayer<Dtype>:Backward_cpu(const vector<Blob<Dtype>*>& top, const bool propagate_down, vector<Blob<Dtype>*>* bottom) return Dtype(0.);INSTANTIATE_CLASS(DataLayer); / namespace caffe本文作者：linger本文 ：本文為博主 文章， 博主 不得 。4/62015/8/29caffe源碼分析­­data_layer.cpp ­ linger(心懷夢想，活在當下) ­ 博客頻道 ­ CSDN.NET上一篇 全局內存性能測試下一篇 使用squid架設自己的主題推薦源碼 網絡 線程 圖片內存猜你在找韋東山HTML 5移動開發從入門到精通C語言及程序設計初步Android入門實零基礎學HTML 5實戰開發(第一季)任意一張圖片的特征