Converting kCVPixelFormatType_420YpCbCr8BiPlanarFullRange Frames to UIImage

===========================================================

In this article, we’ll explore the process of converting frames captured in the kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format to a UIImage. This format is commonly used for video recording on iOS devices and represents a bi-planar component Y’CbCr 8-bit 4:2:0, full-range image.

Understanding the kCVPixelFormatType_420YpCbCr8BiPlanarFullRange Format

The kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format is a component video format that consists of three planes:

• Y’ (luminance, 8-bit, full-range)
• Cb (blue, 8-bit, full-range)
• Cr (chrominance, 8-bit, full-range)

The format uses bi-planar storage, where each plane is stored in a separate buffer. The baseAddr points to the big-endian CVPlanarPixelBufferInfo_YCbCrBiPlanar struct.

Creating a UIImage from kCVPixelFormatType_420YpCbCr8BiPlanarFullRange Data

To create a UIImage from data in this format, we need to:

1. Lock the base address of the pixel buffer using CVPixelBufferLockBaseAddress.
2. Get the number of bytes per row for the pixel buffer.
3. Create a device-dependent RGB color space using CGColorSpaceCreateDeviceRGB.
4. Create a bitmap graphics context with the sample buffer data using CGBitmapContextCreate.
5. Unlock the pixel buffer.

However, we can’t use kCGImageAlphaPremultipliedFirst or kCGImageAlphaNoneSkipLast flags as it requires an alpha channel that is not present in our format. Instead, we need to create a context with no alpha channel and manually adjust the brightness values to match Apple’s implementation.

Modifying the Code

To modify the code provided by the OP, we’ll follow these steps:

1. Allocate memory for the RGB buffer based on the frame dimensions.
2. Extract the Y’ and Cb/Cr data from their respective buffers using CVPixelBufferGetBaseAddressOfPlane and CVPixelBufferGetBytesPerRowOfPlane.
3. Iterate through each pixel in the frame, applying adjustments to create a pseudo-alpha channel.

Here’s an example implementation that modifies the code provided by the OP:

#define clamp(a) (a>255?255:(a<0?0:a))

- (UIImage *)imageFromSampleBuffer:(CMSampleBufferRef)sampleBuffer {
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, 0);

    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);
    uint8_t *yBuffer = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
    size_t yPitch = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0);
    uint8_t *cbCrBuffer = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1);
    size_t cbCrPitch = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 1);

    int bytesPerPixel = 4;
    uint8_t *rgbBuffer = malloc(width * height * bytesPerPixel);

    for(int y = 0; y < height; y++) {
        uint8_t *rgbBufferLine = &rgbBuffer[y * width * bytesPerPixel];
        uint8_t *yBufferLine = &yBuffer[y * yPitch];
        uint8_t *cbCrBufferLine = &cbCrBuffer[(y >> 1) * cbCrPitch];

        for(int x = 0; x < width; x++) {
            int16_t y = yBufferLine[x];
            int16_t cb = cbCrBufferLine[x & ~1] - 128;
            int16_t cr = cbCrBufferLine[x | 1] - 128;

            uint8_t *rgbOutput = &rgbBufferLine[x*bytesPerPixel];

            // Add a pseudo-alpha channel
            rgbOutput[0] = (uint8_t)(clamp((y + cr) * 0.7));
            rgbOutput[1] = (uint8_t)(clamp(y + cb * -0.343 + cr * -0.711));
            rgbOutput[2] = (uint8_t)(clamp(y + cb * 1.765));
            rgbOutput[3] = (uint8_t)255; // Set alpha channel to max
        }
    }

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef context = CGBitmapContextCreate(rgbBuffer, width, height, 8, width * bytesPerPixel, colorSpace, kCGBitmapByteOrder32Little);
    CGImageRef quartzImage = CGBitmapContextCreateImage(context);

    // Release resources in the correct order
    CGContextRelease(context);
    CGColorSpaceRelease(colorSpace);
    CGImageRelease(quartzImage);
    free(rgbBuffer);

    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);

    return [UIImage imageWithCGImage:quartzImage];
}

Conclusion

Converting frames in the kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format to a UIImage requires careful manipulation of the pixel data and creation of a pseudo-alpha channel.

By following this guide, you can create an implementation that produces high-quality images from your video recordings.

Last modified on 2024-04-30