项目暂时告一段落,也是一知半解,不过我的分享可以帮助我这样菜鸟了。
先来下知识结构:

1、h264

视频编码处理的最后一步就是熵编码,在H.264中采用了两种不同的熵编码方法:通用可变长编码(UVLC)和基于文本的自适应二进制算术编码(CABAC)。

2、aac

Advanced Audio Coding。一种专为声音数据设计的文件压缩格式,与MP3不同,它采用了全新的算法进行编码,更加高效,具有更高的“性价比”。利用AAC格式,可使人感觉声音质量没有明显降低

3、pcm

音频采集的原始数据,硬编码数据

4、yuv

视频采集的原始数据,硬编码数据

5、时间戳

直播音视频同步的关键参数

6、rtmp推流

直播的推流手段

一、我们首先要做的就是采集,我们需要采集硬编码数据yuv将其转化成h264,然后采集pcm数据,并将其转化成aac数据,并发送

1、视频采集,我们要采集最后要转化为h264编码的格式,需要用到VideoToolbox.framework及AVFoundation.framework
VideoToolbox.framework 的主要工作是编码,将yuv数据编码为h264。AVFoundation.framework的任务是采集yuv原始数据。

// 获取硬编码数据函数,一些初始化工作就不在这里熬述了,网上有很多
-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection
{
}

(1)初始化VTCompressionSession。
VTCompressionSession初始化的时候需要给出width宽,height长,编码器类型kCMVideoCodecType_H264等。然后通过调用VTSessionSetProperty接口设置帧率等属性,最后需要设定一个回调函数,这个回调是视频图像编码成功后调用。全部准备好后,使用VTCompressionSessionCreate创建session。

// 这个函数是初始化
- (void) initEncode:(int)width  height:(int)height bite:(int)iBite
{
    dispatch_sync(aQueue, ^{
        
        // For testing out the logic, lets read from a file and then send it to encoder to create h264 stream
        
        // Create the compression session   注意h264函数
        OSStatus status = VTCompressionSessionCreate(NULL, width, height, kCMVideoCodecType_H264, NULL, NULL, NULL, didCompressH264, (__bridge void *)(self),  &EncodingSession);
        NSLog(@"H264: VTCompressionSessionCreate %d", (int)status);
        
        if (status != 0)
        {
            NSLog(@"H264: Unable to create a H264 session");
            error = @"H264: Unable to create a H264 session";
            
            return ;
            
        }
        
        // 码率是清晰度
        // Set the properties
        VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_RealTime, kCFBooleanTrue);
        VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_AllowFrameReordering, kCFBooleanFalse);
        VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, (__bridge CFTypeRef _Nonnull)(@(GOP_SIZE)));
        VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Main_AutoLevel);
        VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_AverageBitRate, (__bridge CFTypeRef _Nonnull)@(iBite));
        VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_ExpectedFrameRate, (__bridge CFTypeRef _Nonnull)@(FRAME_RATE));
        VTSessionSetProperty(EncodingSession, kVTCompressionPropertyKey_DataRateLimits, (__bridge CFTypeRef _Nonnull)@[@(iBite/8),@(1)]);
    
        // Tell the encoder to start encoding
        VTCompressionSessionPrepareToEncodeFrames(EncodingSession);
    });
}

(2)提取摄像头采集的原始图像数据给VTCompressionSession来硬编码。

摄像头采集后的图像是未编码的CMSampleBuffer形式,利用给定的接口函数CMSampleBufferGetImageBuffer从中提取出CVPixelBufferRef,使用硬编码接口VTCompressionSessionEncodeFrame来对该帧进行硬编码,编码成功后,会自动调用session初始化时设置的回调函数。

    dispatch_sync(aQueue, ^{
        
        frameCount++;
        // Get the CV Image buffer  提取摄像头采集的原始图像数据给VTCompressionSession来硬编码 也就是给VTCompressionSessionCreate来编码
        CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
        
        // Create properties
        CMTime presentationTimeStamp = CMTimeMake(frameCount, 1000);
        //CMTime duration = CMTimeMake(1, DURATION);
        VTEncodeInfoFlags flags;
        
        // Pass it to the encoder
        OSStatus statusCode = VTCompressionSessionEncodeFrame(EncodingSession,
                                                              imageBuffer,
                                                              presentationTimeStamp,
                                                              kCMTimeInvalid,
                                                              NULL, NULL, &flags);
        // Check for error
        if (statusCode != noErr) {
            NSLog(@"H264: VTCompressionSessionEncodeFrame failed with %d", (int)statusCode);
            error = @"H264: VTCompressionSessionEncodeFrame failed ";
            
            // End the session
            VTCompressionSessionInvalidate(EncodingSession);
            CFRelease(EncodingSession);
            EncodingSession = NULL;
            error = NULL;
            return;
        }
        //            NSLog(@"H264: VTCompressionSessionEncodeFrame Success");
    });

(3)利用回调函数,将因编码成功的CMSampleBuffer转换成H264码流,通过网络传播。
基本上是硬解码的一个逆过程。

void didCompressH264(void *outputCallbackRefCon, void *sourceFrameRefCon, OSStatus status, VTEncodeInfoFlags infoFlags,
                     CMSampleBufferRef sampleBuffer )
{
//        NSLog(@"didCompressH264 called with status %d infoFlags %d", (int)status, (int)infoFlags);
    NSLog(@"H264");
    if (status != 0) return;
    
    if (!CMSampleBufferDataIsReady(sampleBuffer))
    {
        NSLog(@"didCompressH264 data is not ready ");
        return;
    }
    H264Encoder* encoder = (__bridge H264Encoder*)outputCallbackRefCon;
    
    // Check if we have got a key frame first
    bool keyframe = !CFDictionaryContainsKey( (CFArrayGetValueAtIndex(CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, true), 0)), kCMSampleAttachmentKey_NotSync);
    encoder->countFrame=encoder->countFrame+1;
    
//    NSLog(@"dzf  frameCount%d",encoder->countFrame);
    if (keyframe)
    {
//        NSLog(@"dzf  keyframe is true ");
        CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
        // CFDictionaryRef extensionDict = CMFormatDescriptionGetExtensions(format);
        // Get the extensions
        // From the extensions get the dictionary with key "SampleDescriptionExtensionAtoms"
        // From the dict, get the value for the key "avcC"
        
        size_t sparameterSetSize, sparameterSetCount;
        const uint8_t *sparameterSet;
        OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sparameterSet, &sparameterSetSize, &sparameterSetCount, 0 );
        if (statusCode == noErr)
        {
            // Found sps and now check for pps
            size_t pparameterSetSize, pparameterSetCount;
            const uint8_t *pparameterSet;
            OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pparameterSet, &pparameterSetSize, &pparameterSetCount, 0 );
            if (statusCode == noErr)
            {
                // Found pps
                encoder->sps = [NSData dataWithBytes:sparameterSet length:sparameterSetSize];
                encoder->pps = [NSData dataWithBytes:pparameterSet length:pparameterSetSize];
                if (encoder->_delegate)
                {
                    [encoder->_delegate gotSpsPps:encoder->sps pps:encoder->pps];
                }
            }
        }
    }
    CMBlockBufferRef dataBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    size_t length, totalLength;
    char *dataPointer;
    OSStatus statusCodeRet = CMBlockBufferGetDataPointer(dataBuffer, 0, &length, &totalLength, &dataPointer);
    if (statusCodeRet == noErr) {

        // 发送数据
        size_t bufferOffset = 0;
        static const int AVCCHeaderLength = 4;
        while (bufferOffset _delegate gotEncodedData:data isKeyFrame:keyframe];
            
            // Move to the next NAL unit in the block buffer
            bufferOffset += AVCCHeaderLength + NALUnitLength;
        }
        // 你小子存的数据
        [encoder->_delegate oneFrameEncodeEnd:keyframe];
    }
}

值得注意的是一段视频的头部是sps pps 组成的,我们在这个函数中要检查头部信息,筛选普通信息进行封装发送推流。先发送头部数据再发送普通视频数据。
解析出参数集SPS和PPS,加上开始码后组装成NALU。提取出视频数据,将长度码转换成开始码,组长成NALU。将NALU发送出去。

发送视频头部信息代码

- (void)gotSpsPps:(NSData*)sps pps:(NSData*)pps
{
//    NSLog(@"gotSpsPps");
    frameCount2 = [_h264Encoder getFreameCound];
    
    const char bytes[] = "x00x00x00x01";
    size_t length = (sizeof bytes) - 1; //string literals have implicit trailing ''
    NSData *ByteHeader = [NSData dataWithBytes:bytes length:length];
    
    mysps = sps;
    mypps = pps;
    [mutableData appendData:ByteHeader];
    [mutableData appendData:mysps];
    [mutableData appendData:ByteHeader];
    [mutableData appendData:mypps];
    pos = pos + sps.length + pps.length + ByteHeader.length*2;
    
    NSMutableData *mutableDataTem1 = [[NSMutableData alloc] init];;
    [mutableDataTem1 appendData:ByteHeader];
    [mutableDataTem1 appendData:mysps];
    long tem1 = sps.length + ByteHeader.length;
    [self sendData:sizeof(Byte)*tem1 data:(char*)[mutableDataTem1 bytes]];
    
    NSMutableData *mutableDataTem = [[NSMutableData alloc] init];;
    [mutableDataTem appendData:ByteHeader];
    [mutableDataTem appendData:mypps];
    long tem = pps.length + ByteHeader.length;
    [self sendData:sizeof(Byte)*tem data:(char*)[mutableDataTem bytes]];
}

发送实体部分代码

- (void)oneFrameEncodeEnd:(BOOL)isKeyFrame
{
    FrameData *frameData = [[FrameData alloc] init];
    
    frameData.Iframe = isKeyFrame;
    frameData.frame_len = (int) pos;
    frameData.frame_seq = total_vseq;
    frameData.stream_index = 0;
    
    
    frameData.frame_data = (Byte *)malloc(sizeof(Byte)*pos);//new Byte[pos];
    memcpy(frameData.frame_data,[mutableData bytes], pos*sizeof(Byte));
    
    [_videoArray addObject:frameData];
    total_vseq++;
    
    //if(isKeyFrame)
    //NSLog(@"add one h264 h264  h264  frame to videoArray---seq:%ld",total_vseq);
    
    mysps = nil;
    mypps = nil;
    
    [mutableData resetBytesInRange:NSMakeRange(0, [mutableData length])];
    [mutableData setLength:0];
    
    pos = 0;
}

音频的采集发送

将采集pcm数据进行aac编码,网上应该有相关的代码可以学习

-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection
{
    static BOOL             firstStartTimer = false;
    static long             num = 0;

    if (connection == _audioConnection) {
        NSLog(@"captureOutput audio");
        
        char szBuf[4096];
        memset(szBuf, 0, sizeof(szBuf));
        uint32_t  nSize = sizeof(szBuf);
//        AudioStreamBasicDescription inputFormat = *(CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sampleBuffer))); // 输入音频格式
        
        AudioStreamBasicDescription outputFormat = *(CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sampleBuffer)));
        nSize = CMSampleBufferGetTotalSampleSize(sampleBuffer);
        CMBlockBufferRef databuf = CMSampleBufferGetDataBuffer(sampleBuffer);
        if (CMBlockBufferCopyDataBytes(databuf, 0, nSize, szBuf) == kCMBlockBufferNoErr)
        {
            int32_t nOffSet = 0;
            while (nOffSet = 640) {
                    nInSize = 640;
                } else {
                    nInSize = nSize - nOffSet;
                }
                
                outsize = [ecdoer AACEncoderEncode:lHand inData:szBuf + nOffSet inSize:nInSize outData:szOutBuf maxOutSize:4096];
                //            [ecdoer AACEncoderClose:outsize];
                if (outsize > 0)
                {
                    [self sendAacDataLen:outsize data:szOutBuf ptsTime:0];
                }

                nOffSet += 640;
            }

        }

    }
}

音频数据的发送

- (void)sendAacDataLen:(int) totalLength data: (char*) dataPointer ptsTime:(int64_t)pts{

    int ret = WM_RTMPLIVESDK_InputData(WMRtmpLiveDataType_AAC, (const char* )dataPointer, totalLength, [self getNowTime]);
    NSLog(@"~~~~~~~~~iAAc[%lld]",[self getNowTime]);
    if (ret == 1) {
        NSLog(@"~~~~~aac~~~~~sendData ret[%d] totalLength[%d]",ret,(int)totalLength);
    }
    // fail 1 success 0
}

rtmp推流网上也有很多代码,调用rtmplib 可以自己用c++封装一个库用来调用。

二、最后的陈述

这里就先不解释了大体采集发送的过程就是这样,还有一点视频采集发送音频采集发送的时间获取的是当前时间,测试的时候也可以写间隔20ms来测试延迟的问题,它的逻辑是发送一堆音频数据再发送一个视频数据,因为音频数据比较多,音频数据如果丢帧会感觉出来明显的卡顿,视频则不是,视频丢一帧人眼是很难发现的
有些详细的理论推荐大家看这篇博客。
http://www.jianshu.com/p/a6530fa46a88

文章来源于互联网,如有雷同请联系站长删除:iOS 音视频采集及rtmp推流

发表评论