【mediacodec】MediaRecorder--MediaCodec_综合

文章目录

前言
查看代码重点关注log
1 应用层
- 1.1 创建
- 1.2 获取
- 1.3 设置VideoSource
- 1.4 设置输出模式
- 1.5 设置文件名
- 1.6 设置编码方式
2 获取video codec表——mediaprofile
3 MediaRecorder: prepare
3.1 创建MediaCodecSource（重点看MediaCodec的log）
- 3.2 Audio
4 MediaRecorder: start
5 小结

前言

从应用层到native层查看mediacodec的创建
后续再看Acodec到omx

查看代码重点关注log

Codec相关：MediaCodec\MediaCodecSource\Acodec\MPEG4Writer\omx
音频数据：AudioSource
视频数据：GraphicBufferSource

1 应用层

以camera代码为例，因为camera使用的是mediarecorder，而audiorecord只能录制pcm数据。

android\packages\apps\SnapdragonCamera\src\com\android\camera\captureModule.java

1.1 创建

mMediaRecorder = new MediaRecorder()

1.2 获取

mMediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);

int audioEncoder = SettingTranslation.getAudioEncoder(mSettingsManager.getValue(SettingsManager.KEY_AUDIO_ENCODER));
在这里插入图片描述

1.3 设置VideoSource

mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.SURFACE);
MediaRecorder: setVideoSource(2)
视频数据来源为surface(屏幕显示？)，非camera，存疑
在这里插入图片描述

1.4 设置输出模式

mMediaRecorder.setOutputFormat(mProfile.fileFormat);
MediaRecorder: setOutputFormat(2)
选择输出模式为mpeg4，目前看起来输出模式只有两种
默认为OUTPUT_FORMAT_THREE_GPP
存疑：为什么输出模式有两种，如何对应encoder，而编码方式中也有mpeg4
解答：这里的mpeg_4和three_gpp都只是文件的存储类型，当前视频的存储格式支持这两种。
而在videoencoder看到的mpeg4是编码类型，同类型的还有h263/h264/h265(hevc)

参考文章：媒体篇　第一章音频 | 不只是看客
在这里插入图片描述

在这里插入图片描述
MediaRecorder官方文档导读

在这里插入图片描述

1.5 设置文件名

mMediaRecorder.setOutputFile(fileName);

1.6 设置编码方式

mMediaRecorder.setVideoEncoder(mVideoEncoder);
MediaRecorder: setVideoEncoder(2)
选择编码方式为h264
在这里插入图片描述
编码格式不同于输出格式，以音频为类比。
视频编码可以分为h264/mpeg4，音频编码为lpcm/aac
视频输出格式常见的是mp4/m4a(outputformat为mpeg4)，音频输出格式mp3/wav/amr

在这里插入图片描述
视频编码为h264（camera app默认设置）
07-26 22:23:32.636 3772 3772 V MediaRecorder: setVideoEncoder(2)
07-26 23:01:36.513 1129 3967 I OMXMaster: makeComponentInstance(OMX.qcom.video.encoder.avc) in android.hardwar process

视频编码为mpeg4
07-26 23:02:09.983 3772 3772 V MediaRecorder: setVideoEncoder(3)
07-20 23:30:40.325 972 1051 I OMXMaster: makeComponentInstance(OMX.qcom.video.encoder.mpeg4) in android.hardwar process

在这里插入图片描述

视频编码为h265
07-26 23:02:28.156 3772 3772 V MediaRecorder: setVideoEncoder(5)
07-26 23:02:28.158 1121 1216 V MPEG4Writer: initInternal
07-26 23:02:28.158 1121 1216 V MediaCodecList: matching ‘OMX.qcom.video.encoder.hevc’
07-26 23:02:28.158 1121 1216 V MediaCodecList: matching ‘c2.android.hevc.encoder’
07-26 23:02:28.158 1121 1216 V ACodec : Now uninitialized
07-26 23:02:28.159 1121 4879 V ACodec : onAllocateComponent
07-26 23:02:28.161 1121 4879 I OMXClient: IOmx service obtained
07-26 23:02:28.162 1129 4672 I OMXMaster: makeComponentInstance(OMX.qcom.video.encoder.hevc) in android.hardwar process
在这里插入图片描述

c2.android.hevc.encoder的size不符合

设置h265编码时可选两种编码器，但是最终选择了qcom的硬件编码

1 判断是否是软解码，依据编码器名称是否包含OMX.google.和c2.android.
ps：startsWithIgnoreCase表示不区分大小写

2 判断软解码
在这里插入图片描述
以soundrecorder为例，可以看到输出格式为audio/3gpp

以camera mpeg4为例，可以看到输出格式为video/mp4v-es，输入格式为video/raw

以camera h265为例，可以看到输出格式为video/hevc，输入格式为video/raw
在这里插入图片描述

2 获取video codec表——mediaprofile

java代码
在这里插入图片描述

上层传入encoder类型

根据video_encoder类型，获取编码信息，码率/帧率/宽/高的最大最小值

存疑：sProfiles哪里来的
来源于/vendor/etc/media_profiles_vendor.xml

在这里插入图片描述

在这里插入图片描述
目前看起来不支持vp8，参见/vendor/etc/media_profiles_vendor.xml

3 MediaRecorder: prepare

在这里插入图片描述

3.1 创建MediaCodecSource（重点看MediaCodec的log）

在这里插入图片描述

status_t MediaCodecSource::initEncoder() {
    //1. 根据传入的mimi等信息，查找mediacodeclist中合适的codec名字存入Vector<AString> matchingCodecs中MediaCodecList::findMatchingCodecs(outputMIME.c_str(), true /* encoder */,mFlags,&matchingCodecs);//2. 遍历matchingCodecs，根据名字创建mediacodec//ps:还有一种方式MediaCodec::CreateByType创建mediacodec，或者函数有的时候会用到mEncoder = MediaCodec::CreateByComponentName(mCodecLooper, matchingCodecs[ix]);//3. 打印输出格式ALOGV("output format is '%s'", mOutputFormat->debugString(0).c_str());//4. 配置mediacodecerr = mEncoder->configure();//5. 启动mediacodecerr = mEncoder->start();
}

1 查找匹配的codec名字
输入mime，在MediaCodecList中查找合适的codec名字，存入matches中
mime --> MediaCodecList.matches.name

07-28 03:44:05.130   945  3290 D MediaCodecSource: youkai in MediaCodecSource::initEncoder
07-28 03:44:05.130   945  3290 V MediaCodecList: youkai in MediaCodecList::findMatchingCodecs
07-28 03:44:05.130   945  3290 V MediaCodecList: matching 'OMX.qcom.video.encoder.mpeg4'
07-28 03:44:05.130   945  3290 V MediaCodecList: matching 'c2.android.mpeg4.encoder'
07-28 03:44:05.130   945  3290 V MediaCodecList: matching 'OMX.google.mpeg4.encoder'

至此匹配到三个encoder

07-28 03:44:05.130 945 3290 V MediaCodecList: matching ‘OMX.qcom.video.encoder.mpeg4’
Media_codecs.xml
07-28 03:44:05.130 945 3290 V MediaCodecList: matching ‘c2.android.mpeg4.encoder’
07-28 03:44:05.130 945 3290 V MediaCodecList: matching ‘OMX.google.mpeg4.encoder’
Media_codecs_sw.xml (frameworks\av\media\libstagefright\data)
Size不符合淘汰

2 根据名字创建mediacodec
在这里插入图片描述

3 打出输出格式，这个参数是从app传来的

07-28 03:44:05.161   945 15206 V ACodec  : [OMX.qcom.video.encoder.mpeg4] Now Loaded
07-28 03:44:05.163   945  3290 V MediaCodecSource: output format is 'AMessage(what = 0x00000000) = {
    
07-28 03:44:05.163   945  3290 V MediaCodecSource:   string mime = "video/mp4v-es"
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t width = 1920
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t height = 1080
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t stride = 1920
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t slice-height = 1080
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t color-format = 2130708361
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t bitrate = 20000000
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t frame-rate = 30
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t i-frame-interval = 1
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t priority = 0
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t feature-nal-length-bitstream = 1
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t nal-length-in-bytes = 4
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t isNativeRecorder = 1
07-28 03:44:05.163   945  3290 V MediaCodecSource:   int32_t create-input-buffers-suspended = 1
07-28 03:44:05.163   945  3290 V MediaCodecSource: }'
07-28 03:44:05.163   945 15205 I MediaCodec: MediaCodec will operate in async mode
07-28 03:44:05.163   945 15205 V MediaCodec: kWhatConfigure: Old mCrypto: 0x0 (0)
07-28 03:44:05.163   945 15205 V MediaCodec: kWhatConfigure: New mCrypto: 0x0 (0)
07-28 03:44:05.163   945 15205 V MediaCodec: Found 0 pieces of codec specific data.
07-28 03:44:05.163   945 15206 V ACodec  : onConfigureComponent

4 mediacodec的消息机制接受到kWhatComponentConfigured，打印出输入和输出格式
在这里插入图片描述

07-28 03:44:05.226   945 15205 V MediaCodec: [OMX.qcom.video.encoder.mpeg4] configured as input format: AMessage(what = 0x00000000) = {
    
07-28 03:44:05.226   945 15205 V MediaCodec:       string mime = "video/raw"
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t stride = 1920
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t slice-height = 1080
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t color-format = 2130708361
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t color-range = 0
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t color-standard = 0
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t color-transfer = 0
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t width = 1920
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t height = 1080
07-28 03:44:05.226   945 15205 V MediaCodec:     }, output format: AMessage(what = 0x00000000) = {
    
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t bitrate = 20000000
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t max-bitrate = 20000000
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t frame-rate = 30
07-28 03:44:05.226   945 15205 V MediaCodec:       string mime = "video/mp4v-es"
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t width = 1920
07-28 03:44:05.226   945 15205 V MediaCodec:       int32_t height = 1080
07-28 03:44:05.226   945 15205 V MediaCodec:     }
07-28 03:44:05.227   945 15206 V ACodec  : onCreateInputSurface
07-28 03:44:05.227   953  2087 V GraphicBufferSource: GraphicBufferSource

5 接受到GraphicBufferSource传来的创建显示消息kWhatInputSurfaceCreated

在这里插入图片描述

07-28 03:44:05.232   945 15205 V MediaCodec: [OMX.qcom.video.encoder.mpeg4] input surface created as input format: AMessage(what = 0x00000000) = {
    
07-28 03:44:05.232   945 15205 V MediaCodec:       string mime = "video/raw"
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t stride = 1920
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t slice-height = 1080
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t color-format = 2130708361
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t color-range = 2
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t color-standard = 1
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t color-transfer = 3
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t width = 1920
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t height = 1080
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t android._dataspace = 260
07-28 03:44:05.232   945 15205 V MediaCodec:       Buffer android._color-aspects = {
    
07-28 03:44:05.232   945 15205 V MediaCodec:         00000000:  02 00 00 00 01 00 00 00  03 00 00 00 01 00 00 00  ................
07-28 03:44:05.232   945 15205 V MediaCodec:       }
07-28 03:44:05.232   945 15205 V MediaCodec:     }, output format: AMessage(what = 0x00000000) = {
    
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t bitrate = 20000000
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t max-bitrate = 20000000
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t frame-rate = 30
07-28 03:44:05.232   945 15205 V MediaCodec:       string mime = "video/mp4v-es"
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t width = 1920
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t height = 1080
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t color-range = 2
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t color-standard = 1
07-28 03:44:05.232   945 15205 V MediaCodec:       int32_t color-transfer = 3
07-28 03:44:05.232   945 15205 V MediaCodec:     }
07-28 03:44:05.232   945  3290 V MediaCodecSource: setting dataspace 0x104, format 0x22
07-28 03:44:05.232   945 15206 V ACodec  : onStart
07-28 03:44:05.232   953  1026 V GraphicBufferSource: stop

3.2 Audio

07-28 03:44:05.303   945  3290 D MediaCodecSource: youkai in MediaCodecSource::initEncoder
07-28 03:44:05.303   945  3290 V ACodec  : Now uninitialized
07-28 03:44:05.304   945 15212 V ACodec  : onAllocateComponent
07-28 03:44:05.306   945 15212 I OMXClient: IOmx service obtained
07-28 03:44:05.306   714 15211 V AudioFlinger: acquireWakeLock_l() AudioIn_7E status 0
07-28 03:44:05.306   714 15211 D AudioFlinger: updateWakeLockUids_l AudioIn_7E uids:
07-28 03:44:05.306   953  1026 I OMXMaster: makeComponentInstance(OMX.qcom.audio.encoder.aac) in android.hardwar process
07-28 03:44:05.306   714 15211 V AudioFlinger: updateWakeLockUids_l() AudioIn_7E status 0
07-28 03:44:05.308   714 15211 V AudioFlinger: releaseWakeLock_l() AudioIn_7E
07-28 03:44:05.309   714 15211 V AudioFlinger: RecordThread: loop stopping
07-28 03:44:05.314   953  1026 E         :  component init: role = OMX.qcom.audio.encoder.aac
07-28 03:44:05.324     0     0 I aac_in_open: session id 2: NT mode encoder success
07-28 03:44:05.324     0     0 I aac_in_open: session id 2: success
07-28 03:44:05.327   953  1026 W media.codec: Failed to obtain quirks for omx component 'OMX.qcom.audio.encoder.aac' from XML files
07-28 03:44:05.328   945 15212 V ACodec  : [OMX.qcom.audio.encoder.aac] Now Loaded
07-28 03:44:05.329   945  3290 V MediaCodecSource: output format is 'AMessage(what = 0x00000000) = {
    
07-28 03:44:05.329   945  3290 V MediaCodecSource:   string mime = "audio/mp4a-latm"
07-28 03:44:05.329   945  3290 V MediaCodecSource:   int32_t aac-profile = 2
07-28 03:44:05.329   945  3290 V MediaCodecSource:   int32_t max-input-size = 2048
07-28 03:44:05.329   945  3290 V MediaCodecSource:   int32_t channel-count = 2
07-28 03:44:05.329   945  3290 V MediaCodecSource:   int32_t sample-rate = 48000
07-28 03:44:05.329   945  3290 V MediaCodecSource:   int32_t bitrate = 96000
07-28 03:44:05.329   945  3290 V MediaCodecSource:   int32_t priority = 0
07-28 03:44:05.329   945  3290 V MediaCodecSource: }'
07-28 03:44:05.330   945 15212 I MediaCodec: MediaCodec will operate in async mode
07-28 03:44:05.331   945 15212 V MediaCodec: kWhatConfigure: Old mCrypto: 0x0 (0)
07-28 03:44:05.332   945 15212 V MediaCodec: kWhatConfigure: New mCrypto: 0x0 (0)
07-28 03:44:05.332   945 15212 V MediaCodec: Found 0 pieces of codec specific data.
07-28 03:44:05.332   945 15212 V ACodec  : onConfigureComponent07-28 03:44:05.334   945 15212 V MediaCodec: [OMX.qcom.audio.encoder.aac] configured as input format: AMessage(what = 0x00000000) = {
    
07-28 03:44:05.334   945 15212 V MediaCodec:       string mime = "audio/raw"
07-28 03:44:05.334   945 15212 V MediaCodec:       int32_t channel-count = 2
07-28 03:44:05.334   945 15212 V MediaCodec:       int32_t sample-rate = 48000
07-28 03:44:05.334   945 15212 V MediaCodec:       int32_t pcm-encoding = 2
07-28 03:44:05.334   945 15212 V MediaCodec:     }, output format: AMessage(what = 0x00000000) = {
    
07-28 03:44:05.334   945 15212 V MediaCodec:       int32_t bitrate = 96000
07-28 03:44:05.334   945 15212 V MediaCodec:       int32_t max-bitrate = 96000
07-28 03:44:05.334   945 15212 V MediaCodec:       string mime = "audio/mp4a-latm"
07-28 03:44:05.334   945 15212 V MediaCodec:       int32_t channel-count = 2
07-28 03:44:05.334   945 15212 V MediaCodec:       int32_t sample-rate = 48000
07-28 03:44:05.334   945 15212 V MediaCodec:     }
07-28 03:44:05.335   945  3290 V MediaCodecSource: setting dataspace 0x10c10000, format 0x22
07-28 03:44:05.335   945 15212 V ACodec  : onStart07-28 03:44:05.367   945 15212 V MediaCodec: [OMX.qcom.audio.encoder.aac] output format changed to: AMessage(what = 0x00000000) = {
    
07-28 03:44:05.367   945 15212 V MediaCodec:       int32_t bitrate = 96000
07-28 03:44:05.367   945 15212 V MediaCodec:       int32_t max-bitrate = 96000
07-28 03:44:05.367   945 15212 V MediaCodec:       string mime = "audio/mp4a-latm"
07-28 03:44:05.367   945 15212 V MediaCodec:       int32_t channel-count = 2
07-28 03:44:05.367   945 15212 V MediaCodec:       int32_t sample-rate = 48000
07-28 03:44:05.367   945 15212 V MediaCodec:     }
07-28 03:44:05.367   945 15212 V MediaCodec: -- returned buffer timestamp 0 <= 0, ignore it
07-28 03:44:05.367   945 15204 E Utils   : csd0 too small
07-28 03:44:05.367   945 15204 E ExtendedUtils: csd0 too small
07-28 03:44:05.368   945 15212 V ACodec  : [OMX.qcom.audio.encoder.aac] calling fillBuffer 3

4 MediaRecorder: start

07-28 03:44:05.756 14402 14402 V MediaRecorder: start
07-28 03:44:05.756   945  1028 V StagefrightRecorder: start
07-28 03:44:05.756   945  1028 V MPEG4Writer: movie time scale: 1000
07-28 03:44:05.756   945  1028 V MPEG4Writer: muxer starting: mHasMoovBox 1, mHasFileLevelMeta 0
07-28 03:44:05.757   945  1028 I MPEG4Writer: limits: 11065696640/600000000 bytes/us, bit rate: 20096000 bps and the estimated moov size 405120 bytes
07-28 03:44:05.757   945  1028 V MPEG4Writer: startWriterThread
07-28 03:44:05.757   945  1028 V MPEG4Writer: initTrackingProgressStatus
07-28 03:44:05.757   945  1028 I MPEG4Writer: Start time offset: 100000 us
07-28 03:44:05.757   945 15239 V MPEG4Writer: ThreadWrapper: 0xeb153180
07-28 03:44:05.757   945 15239 V MPEG4Writer: threadFunc
07-28 03:44:05.757   945 15239 V MPEG4Writer: findChunkToWrite
07-28 03:44:05.757   945 15239 V MPEG4Writer: Nothing to be written after all
07-28 03:44:05.757   945 15204 I MediaCodecSource: MediaCodecSource (video) starting
07-28 03:44:05.758   945 15204 I MediaCodecSource: MediaCodecSource (video) started
07-28 03:44:05.758   945  1028 V MPEG4Writer: initTrackingProgressStatus
07-28 03:44:05.759   945  1028 I MPEG4Writer: Start time offset: 100000 us
07-28 03:44:05.759   945 15204 I MediaCodecSource: MediaCodecSource (audio) starting
07-28 03:44:05.759   945 15204 V MediaCodecSource: puller (audio) start
07-28 03:44:05.759   953 14820 D GraphicBufferSource: setStartTimeUs: skipFramesBeforeUs=81188109321
07-28 03:44:05.760   953  1026 V GraphicBufferSource: setSuspend=0 at time -1 us
07-28 03:44:05.763   945 15241 V AudioRecord: start(48): sync event 0 trigger session 0

5 小结

本章主要介绍从java的mediarecorder到mediacodec创建流程。

从app传下来结果关键参数(编码类型)
经过mediaprofile的过滤，选择出支持的格式
将mime传给mediacodeclist选择合适的encoder名字
根据encoder名字创建mediacodec(这里会创建Acodec，然后到omx和MPEG4writer，后续文章介绍)
mediarecorder的prepare阶段重点创建音视频的encoder，start阶段简单介绍