caffe: 数据层

在caffe中layer分为以下几类：

1. Data Layers
2. Vision Layers
3. Recurrent Layers
4. Common Layers
5. Normalization Layers
6. Activation / Neuron Layers
7. Utility Layers

8. Loss Layers

   首先介绍数据层，数据层是每个模型的最底层，是模型的入口，不仅提供数据的输入，也提供数据从Blobs转换成别的格式进行保存输出。通常数据的预处理（如减去均值, 放大缩小, 裁剪和镜像等），也在这一层设置参数实现。数据来源可以来自高效的数据库（如LevelDB和LMDB），也可以直接来自于内存。如果不是很注重效率的话，数据也可来自磁盘的hdf5文件和图片格式文件。所有的数据层的都具有的公用参数：先看示例：

 layer {name: "cifar"type: "Data"top: "data"top: "label"include {phase: TRAIN}transform_param {mean_file: "examples/cifar10/mean.binaryproto"}data_param {source: "examples/cifar10/cifar10_train_lmdb"batch_size: 100backend: LMDB}
}

下面分别介绍这些参数的含义：
1）name：表示该层的名字，一般选取有意义的名称;
2) type: 表示层类型，如果是Data，表示数据来源于LevelDB或LMDB。根据数据的来源不同，数据层的类型也不同（后面会详细阐述）。通常都是采 用的LevelDB或LMDB数据，因此层类型设置为Data。
3）top或bottom: 每一层用bottom来输入数据，用top来输出数据。如果只有top没有bottom，则此层只有输出，没有输入。反之亦然。如果有多个 top或多个bottom，表示有多个blobs数据的输入和输出。
4）data 与 label: 在数据层中，至少有一个命名为data的top。如果有第二个top，若是分类问题则一般是label。根据任务的不同top的有所不同，例如：siftflow-fcn16s 的datalayer如下：

layer {name: "data"type: "Python"top: "data"top: "sem"top: "geo"python_param {module: "siftflow_layers"layer: "SIFTFlowSegDataLayer"param_str: "{\'siftflow_dir\': \'../data/sift-flow\', \'seed\': 1337, \'split\': \'trainval\'}"}
}

5）include: 一般训练的时候和测试的时候，模型的层是不一样的。该层（layer）是属于训练阶段的层，还是属于测试阶段的层，需要用include来指定。如果没有include参数，则表示该层既在训练模型中，又在测试模型中

6）Transform_param: 数据的预处理，可以将数据变换到定义的范围内。如设置scale、mirror、crop_size、mean_file等。例如：

transform_param {scale: 0.00390625mean_file_size: "examples/cifar10/mean.binaryproto"# 用一个配置文件来进行均值操作mirror: 1  # 1表示开启镜像，0表示关闭，也可用ture和false来表示# 剪裁一个 227*227的图块，在训练阶段随机剪裁，在测试阶段从中间裁剪crop_size: 227}

参考caffe.proto 文件，有关transform_param 的code如下：

// Message that stores parameters used to apply transformation
// to the data layer's data
message TransformationParameter {// For data pre-processing, we can do simple scaling and subtracting the// data mean, if provided. Note that the mean subtraction is always carried// out before scaling.optional float scale = 1 [default = 1];// Specify if we want to randomly mirror data.optional bool mirror = 2 [default = false];// Specify if we would like to randomly crop an image.optional uint32 crop_size = 3 [default = 0];// mean_file and mean_value cannot be specified at the same timeoptional string mean_file = 4;// if specified can be repeated once (would subtract it from all the channels)// or can be repeated the same number of times as channels// (would subtract them from the corresponding channel)repeated float mean_value = 5;// Force the decoded image to have 3 color channels.optional bool force_color = 6 [default = false];// Force the decoded image to have 1 color channels.optional bool force_gray = 7 [default = false];
}

type:参数输入类型很多，所以单独拿出来做解释，下面详细介绍：
1) tpye：Data，这种最常用 ，例如：

layer {name: "mnist"type: "Data"top: "data"top: "label"include {phase: TRAIN}transform_param {scale: 0.00390625}data_param {source: "examples/mnist/mnist_train_lmdb"batch_size: 64backend: LMDB}
}

必须设置的参数：
source: 包含数据库的目录名称，如examples/mnist/mnist_train_lmdb
batch_size: 每次处理的数据个数，如64
可选的参数：
rand_skip: 在开始的时候，路过某个数据的输入。通常对异步的SGD很有用。
backend: 选择是采用LevelDB还是LMDB, 默认是LevelDB.
参考源码：

 **message DataParameter {enum DB { LEVELDB = 0; LMDB = 1;}// Specify the data source.optional string source = 1;// Specify the batch size.optional uint32 batch_size = 4;// The rand_skip variable is for the data layer to skip a few data points// to avoid all asynchronous sgd clients to start at the same point. The skip// point would be set as rand_skip * rand(0,1). Note that rand_skip should not// be larger than the number of keys in the database.// DEPRECATED. Each solver accesses a different subset of the database.optional uint32 rand_skip = 7 [default = 0];optional DB backend = 8 [default = LEVELDB];// DEPRECATED. See TransformationParameter. For data pre-processing, we can do// simple scaling and subtracting the data mean, if provided. Note that the// mean subtraction is always carried out before scaling.optional float scale = 2 [default = 1];optional string mean_file = 3;// DEPRECATED. See TransformationParameter. Specify if we would like to randomly// crop an image.optional uint32 crop_size = 5 [default = 0];// DEPRECATED. See TransformationParameter. Specify if we want to randomly mirror// data.optional bool mirror = 6 [default = false];// Force the encoded image to have 3 color channelsoptional bool force_encoded_color = 9 [default = false];// Prefetch queue (Increase if data feeding bandwidth varies, within the// limit of device memory for GPU training)optional uint32 prefetch = 10 [default = 4];
}**

2）type：memory_data

layer {top: "data"top: "label"name: "memory_data"type: "MemoryData"memory_data_param{batch_size: 2height: 100width: 100channels: 1}transform_param {scale: 0.0078125mean_file: "mean.proto"mirror: false}
}

必须设置的参数： batch_size：每一次处理的数据个数，比如2; channels：通道数; height：高度; width: 宽度;
参考源码：

message MemoryDataParameter {optional uint32 batch_size = 1;optional uint32 channels = 2;optional uint32 height = 3;optional uint32 width = 4;
}

3) type:HDF5data

layer {name: "data"type: "HDF5Data"top: "data"top: "label"hdf5_data_param {source: "examples/hdf5_classification/data/train.txt"batch_size: 10}
}

必须设置的参数：source: 读取的文件名称;batch_size: 每一次处理的数据个数

// Message that stores parameters used by HDF5DataLayer
message HDF5DataParameter {// Specify the data source.optional string source = 1;// Specify the batch size.optional uint32 batch_size = 2;// Specify whether to shuffle the data.// If shuffle == true, the ordering of the HDF5 files is shuffled,// and the ordering of data within any given HDF5 file is shuffled,// but data between different files are not interleaved; all of a file's// data are output (in a random order) before moving onto another file.optional bool shuffle = 3 [default = false];
}

4） type:ImageData

layer {name: "data"type: "ImageData"top: "data"top: "label"transform_param {mirror: falsecrop_size: 227mean_file: "data/ilsvrc12/imagenet_mean.binaryproto"}image_data_param {source: "examples/_temp/file_list.txt"batch_size: 50new_height: 256new_width: 256}
}

必须设置的参数：
source: 一个文本文件的名字，每一行给定一个图片文件的名称和标签（label)
batch_size: 每一次处理的数据个数，即图片数
可选参数：
rand_skip: 在开始的时候，路过某个数据的输入。通常对异步的SGD很有用。
shuffle: 随机打乱顺序，默认值为false
new_height,new_width: 如果设置，则将图片进行resize
参考源码：

message ImageDataParameter {// Specify the data source.optional string source = 1;// Specify the batch size.optional uint32 batch_size = 4 [default = 1];// The rand_skip variable is for the data layer to skip a few data points// to avoid all asynchronous sgd clients to start at the same point. The skip// point would be set as rand_skip * rand(0,1). Note that rand_skip should not// be larger than the number of keys in the database.optional uint32 rand_skip = 7 [default = 0];// Whether or not ImageLayer should shuffle the list of files at every epoch.optional bool shuffle = 8 [default = false];// It will also resize images if new_height or new_width are not zero.optional uint32 new_height = 9 [default = 0];optional uint32 new_width = 10 [default = 0];// Specify if the images are color or grayoptional bool is_color = 11 [default = true];// DEPRECATED. See TransformationParameter. For data pre-processing, we can do// simple scaling and subtracting the data mean, if provided. Note that the// mean subtraction is always carried out before scaling.optional float scale = 2 [default = 1];optional string mean_file = 3;// DEPRECATED. See TransformationParameter. Specify if we would like to randomly// crop an image.optional uint32 crop_size = 5 [default = 0];// DEPRECATED. See TransformationParameter. Specify if we want to randomly mirror// data.optional bool mirror = 6 [default = false];optional string root_folder = 12 [default = ""];
}

5）type：Python 这个有点牛，可以自己定义
例如：鼎鼎大名的fcn 就采用了这个：

layer {name: "data"type: "Python"top: "data"top: "sem"top: "geo"python_param {module: "siftflow_layers"layer: "SIFTFlowSegDataLayer"param_str: "{\'siftflow_dir\': \'../data/sift-flow\', \'seed\': 1337, \'split\': \'trainval\'}"}
}

具体可去caffe官网查看内容，先写这么多吧！！！