Video Processing Procedures#

The following pseudo code shows the video processing procedure:

MFXVideoVPP_QueryIOSurf(session, &init_param, response);
allocate_pool_of_surfaces(in_pool, response[0].NumFrameSuggested);
allocate_pool_of_surfaces(out_pool, response[1].NumFrameSuggested);
MFXVideoVPP_Init(session, &init_param);
mfxFrameSurface1 *in=find_unlocked_surface_and_fill_content(in_pool);
mfxFrameSurface1 *out=find_unlocked_surface_from_the_pool(out_pool);
for (;;) {
   sts=MFXVideoVPP_RunFrameVPPAsync(session,in,out,NULL,&syncp);
   if (sts==MFX_ERR_MORE_SURFACE || sts==MFX_ERR_NONE) {
      MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
      process_output_frame(out);
      out=find_unlocked_surface_from_the_pool(out_pool);
   }
   if (sts==MFX_ERR_MORE_DATA && in==NULL)
      break;
   if (sts==MFX_ERR_NONE || sts==MFX_ERR_MORE_DATA) {
      in=find_unlocked_surface_from_the_pool(in_pool);
      fill_content_for_video_processing(in);
      if (end_of_stream())
         in=NULL;
   }
}
MFXVideoVPP_Close(session);
free_pool_of_surfaces(in_pool);
free_pool_of_surfaces(out_pool);

Note the following key points about the example:

The application uses the MFXVideoVPP_QueryIOSurf() function to obtain the number of frame surfaces needed for input and output. The application must allocate two frame surface pools: one for the input and one for the output.
The video processing function MFXVideoVPP_RunFrameVPPAsync() is asynchronous. The application must use the MFXVideoCORE_SyncOperation() function to synchronize in order to make the output result ready.
The body of the video processing procedure covers the following three scenarios:
- If the number of frames consumed at input is equal to the number of frames generated at output, VPP returns mfxStatus::MFX_ERR_NONE when an output is ready. The application must process the output frame after synchronization, as the MFXVideoVPP_RunFrameVPPAsync() function is asynchronous. The application must provide a NULL input at the end of the sequence to drain any remaining frames.
- If the number of frames consumed at input is more than the number of frames generated at output, VPP returns mfxStatus::MFX_ERR_MORE_DATA for additional input until an output is ready. When the output is ready, VPP returns mfxStatus::MFX_ERR_NONE. The application must process the output frame after synchronization and provide a NULL input at the end of the sequence to drain any remaining frames.
- If the number of frames consumed at input is less than the number of frames generated at output, VPP returns either mfxStatus::MFX_ERR_MORE_SURFACE (when more than one output is ready), or mfxStatus::MFX_ERR_NONE (when one output is ready and VPP expects new input). In both cases, the application must process the output frame after synchronization and provide a NULL input at the end of the sequence to drain any remaining frames.

Configuration#

Intel^® VPL configures the video processing pipeline operation based on the difference between the input and output formats, specified in the mfxVideoParam structure. The following list shows several examples:

When the input color format is YUY2 and the output color format is NV12, Intel^® VPL enables color conversion from YUY2 to NV12.
When the input is interleaved and the output is progressive, Intel^® VPL enables deinterlacing.
When the input is single field and the output is interlaced or progressive, Intel^® VPL enables field weaving, optionally with deinterlacing.
When the input is interlaced and the output is single field, Intel^® VPL enables field splitting.

In addition to specifying the input and output formats, the application can provide hints to fine-tune the video processing pipeline operation. The application can disable filters in the pipeline by using the mfxExtVPPDoNotUse structure, enable filters by using the mfxExtVPPDoUse structure, and configure filters by using dedicated configuration structures. See the Configurable VPP Filters table for a complete list of configurable video processing filters, their IDs, and configuration structures. See the ExtendedBufferID enumerator for more details.

Intel^® VPL ensures that all filters necessary to convert the input format to the output format are included in the pipeline. Intel^® VPL may skip some optional filters even if they are explicitly requested by the application, for example due to limitations of the underlying hardware. To notify the application about skipped optional filters, Intel^® VPL returns the mfxStatus::MFX_WRN_FILTER_SKIPPED warning. The application can retrieve the list of active filters by attaching the mfxExtVPPDoUse structure to the mfxVideoParam structure and calling the MFXVideoVPP_GetVideoParam() function. The application must allocate enough memory for the filter list.

See the Configurable VPP Filters table for a full list of configurable filters.

Configurable VPP Filters#
Filter ID	Configuration Structure
`MFX_EXTBUFF_VPP_DENOISE2`	`mfxExtVPPDenoise2`
`MFX_EXTBUFF_VPP_MCTF`	`mfxExtVppMctf`
`MFX_EXTBUFF_VPP_DETAIL`	`mfxExtVPPDetail`
`MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION`	`mfxExtVPPFrameRateConversion`
`MFX_EXTBUFF_VPP_IMAGE_STABILIZATION`	`mfxExtVPPImageStab`
`MFX_EXTBUFF_VPP_PROCAMP`	`mfxExtVPPProcAmp`
`MFX_EXTBUFF_VPP_FIELD_PROCESSING`	`mfxExtVPPFieldProcessing`
`MFX_EXTBUFF_VPP_3DLUT`	`mfxExtVPP3DLut`

The following example shows video processing configuration:

/* enable image stabilization filter with default settings */
mfxExtVPPDoUse du;
mfxU32 al=MFX_EXTBUFF_VPP_IMAGE_STABILIZATION;

du.Header.BufferId=MFX_EXTBUFF_VPP_DOUSE;
du.Header.BufferSz=sizeof(mfxExtVPPDoUse);
du.NumAlg=1;
du.AlgList=&al;

/* configure the mfxVideoParam structure */
mfxVideoParam conf;
mfxExtBuffer *eb=(mfxExtBuffer *)&du;

memset(&conf,0,sizeof(conf));
conf.IOPattern=MFX_IOPATTERN_IN_SYSTEM_MEMORY | MFX_IOPATTERN_OUT_SYSTEM_MEMORY;
conf.NumExtParam=1;
conf.ExtParam=&eb;

conf.vpp.In.FourCC=MFX_FOURCC_YV12;
conf.vpp.Out.FourCC=MFX_FOURCC_NV12;
conf.vpp.In.Width=conf.vpp.Out.Width=1920;
conf.vpp.In.Height=conf.vpp.Out.Height=1088;

/* video processing initialization */
MFXVideoVPP_Init(session, &conf);

Region of Interest#

During video processing operations, the application can specify a region of interest for each frame as shown in the following figure:

Specifying a region of interest guides the resizing function to achieve special effects, such as resizing from 16:9 to 4:3, while keeping the aspect ratio intact. Use the CropX, CropY, CropW, and CropH parameters in the mfxVideoParam structure to specify a region of interest for each frame when calling MFXVideoVPP_RunFrameVPPAsync(). Note: For per-frame dynamic change, the application should set the CropX, CropY, CropW, and CropH parameters when calling MFXVideoVPP_RunFrameVPPAsync() per frame.

The VPP Region of Interest Operations table shows examples of VPP operations applied to a region of interest.

VPP Region of Interest Operations#
Operation	VPP Input Width X Height	VPP Input CropX, CropY, CropW, CropH	VPP Output Width X Height	VPP Output CropX, CropY, CropW, CropH
Cropping	720 x 480	16, 16, 688, 448	720 x 480	16, 16, 688, 448
Resizing	720 x 480	0, 0, 720, 480	1440 x 960	0, 0, 1440, 960
Horizontal stretching	720 x 480	0, 0, 720, 480	640 x 480	0, 0, 640, 480
16:9 4:3 with letter boxing at the top and bottom	1920 x 1088	0, 0, 1920, 1088	720 x 480	0, 36, 720, 408
4:3 16:9 with pillar boxing at the left and right	720 x 480	0, 0, 720, 480	1920 x 1088	144, 0, 1632, 1088

Multi-view Video Processing#

Intel^® VPL video processing supports processing multiple views. For video processing initialization, the application needs to attach the mfxExtMVCSeqDesc structure to the mfxVideoParam structure and call the MFXVideoVPP_Init() function. The function saves the view identifiers. During video processing, Intel^® VPL processes each view individually. Intel^® VPL refers to the FrameID field of the mfxFrameInfo structure to configure each view according to its processing pipeline. If the video processing source frame is not the output from the Intel^® VPL MVC decoder, then the application needs to fill the the FrameID field before calling the MFXVideoVPP_RunFrameVPPAsync() function. This is shown in the following pseudo code:

mfxExtBuffer *eb;
mfxExtMVCSeqDesc  seq_desc;
mfxVideoParam init_param;

init_param.ExtParam = &eb;
init_param.NumExtParam=1;
eb=(mfxExtBuffer *)&seq_desc;

/* init VPP */
MFXVideoVPP_Init(session, &init_param);

/* perform processing */
for (;;) {
    MFXVideoVPP_RunFrameVPPAsync(session,in,out,NULL,&syncp);
    MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
}

/* close VPP */
MFXVideoVPP_Close(session);

Video Processing 3DLUT#

Intel^® VPL video processing supports 3DLUT with Intel HW specific memory layout. The following pseudo code shows how to create a MFX_3DLUT_MEMORY_LAYOUT_INTEL_65LUT 3DLUT surface.

VADisplay va_dpy = 0;
VASurfaceID surface_id = 0;

vaInitialize(va_dpy, NULL, NULL);

// MFX_3DLUT_MEMORY_LAYOUT_INTEL_65LUT indicate 65*65*128*8bytes.
mfxU32 seg_size = 65, mul_size = 128;
mfxMemId memId = 0;

// create 3DLUT surface (MFX_3DLUT_MEMORY_LAYOUT_INTEL_65LUT)
VASurfaceAttrib    surface_attrib = {};
surface_attrib.type =  VASurfaceAttribPixelFormat;
surface_attrib.flags = VA_SURFACE_ATTRIB_SETTABLE;
surface_attrib.value.type = VAGenericValueTypeInteger;
surface_attrib.value.value.i = VA_FOURCC_RGBA;

vaCreateSurfaces(va_dpy,
                 VA_RT_FORMAT_RGB32,   // 4 bytes
                 seg_size * mul_size,  // 65*128
                 seg_size * 2,         // 65*2
                 &surface_id,
                 1,
                 &surface_attrib,
                 1);

*((VASurfaceID*)memId) = surface_id;

// configure 3DLUT parameters
mfxExtVPP3DLut lut3DConfig;
memset(&lut3DConfig, 0, sizeof(lut3DConfig));
lut3DConfig.Header.BufferId         = MFX_EXTBUFF_VPP_3DLUT;
lut3DConfig.Header.BufferSz         = sizeof(mfxExtVPP3DLut);
lut3DConfig.ChannelMapping          = MFX_3DLUT_CHANNEL_MAPPING_RGB_RGB;    
lut3DConfig.BufferType              = MFX_RESOURCE_VA_SURFACE;
lut3DConfig.VideoBuffer.DataType    = MFX_DATA_TYPE_U16;
lut3DConfig.VideoBuffer.MemLayout   = MFX_3DLUT_MEMORY_LAYOUT_INTEL_65LUT;
lut3DConfig.VideoBuffer.MemId       = memId;

// release 3DLUT surface
vaDestroySurfaces(va_dpy, &surface_id, 1);

The following pseudo code shows how to create a system memory mfx3DLutSystemBuffer 3DLUT surface.

// 64 size 3DLUT(3 dimension look up table)
// The buffer size(in bytes) for every channel is 64*64*64*sizeof(DataType)
mfxU16 dataR[64*64*64], dataG[64*64*64], dataB[64*64*64];
mfxChannel channelR, channelG, channelB;
channelR.DataType = MFX_DATA_TYPE_U16;
channelR.Size = 64;
channelR.Data16 = dataR;
channelG.DataType = MFX_DATA_TYPE_U16;
channelG.Size = 64;
channelG.Data16 = dataG;
channelB.DataType = MFX_DATA_TYPE_U16;
channelB.Size = 64;
channelB.Data16 = dataB;

// configure 3DLUT parameters
mfxExtVPP3DLut lut3DConfig;
memset(&lut3DConfig, 0, sizeof(lut3DConfig));
lut3DConfig.Header.BufferId         = MFX_EXTBUFF_VPP_3DLUT;
lut3DConfig.Header.BufferSz         = sizeof(mfxExtVPP3DLut);
lut3DConfig.ChannelMapping          = MFX_3DLUT_CHANNEL_MAPPING_RGB_RGB;    
lut3DConfig.BufferType              = MFX_RESOURCE_SYSTEM_SURFACE;
lut3DConfig.SystemBuffer.Channel[0] = channelR;
lut3DConfig.SystemBuffer.Channel[1] = channelG;
lut3DConfig.SystemBuffer.Channel[2] = channelB;

HDR Tone Mapping#

Intel^® VPL video processing supports HDR Tone Mapping with Intel HW. The following pseudo code shows how to perform HDR Tone Mapping.

The following pseudo code shows HDR to SDR.

// HDR to SDR (e.g P010 HDR signal -> NV12 SDR signal) in transcoding pipeline
// Attach input external buffers as the below for HDR input. SDR is by default, hence no
// extra output external buffer.
// The input Video Signal Information
mfxExtVideoSignalInfo inSignalInfo   = {};
inSignalInfo.Header.BufferId         = MFX_EXTBUFF_VIDEO_SIGNAL_INFO_IN;
inSignalInfo.Header.BufferSz         = sizeof(mfxExtVideoSignalInfo);
inSignalInfo.VideoFullRange          = 0; // Limited range P010
inSignalInfo.ColourPrimaries         = 9; // BT.2020
inSignalInfo.TransferCharacteristics = 16; // ST2084

// The content Light Level Information
mfxExtContentLightLevelInfo inContentLight = {};
inContentLight.Header.BufferId         = MFX_EXTBUFF_CONTENT_LIGHT_LEVEL_INFO;
inContentLight.Header.BufferSz         = sizeof(mfxExtContentLightLevelInfo);
inContentLight.MaxContentLightLevel    = 4000; // nits
inContentLight.MaxPicAverageLightLevel = 1000; // nits

// The mastering display colour volume
mfxExtMasteringDisplayColourVolume inColourVolume = {};
inColourVolume.Header.BufferId = MFX_EXTBUFF_MASTERING_DISPLAY_COLOUR_VOLUME_IN;
inColourVolume.Header.BufferSz = sizeof(mfxExtMasteringDisplayColourVolume);
// Based on the needs, Please set DisplayPrimaryX/Y[3], WhitePointX/Y, and MaxDisplayMasteringLuminance,
// MinDisplayMasteringLuminance

mfxExtBuffer *ExtBufferIn[3];
ExtBufferIn[0] = (mfxExtBuffer *)&inSignalInfo;
ExtBufferIn[1] = (mfxExtBuffer *)&inContentLight;
ExtBufferIn[2] = (mfxExtBuffer *)&inColourVolume;

mfxSession session      = (mfxSession)0;
mfxVideoParam VPPParams = {};
VPPParams.NumExtParam   = 3;
VPPParams.ExtParam      = (mfxExtBuffer **)&ExtBufferIn[0];
MFXVideoVPP_Init(session, &VPPParams);

The following pseudo code shows SDR to HDR.

// SDR to HDR (e.g NV12 SDR signal -> P010 HDR signal) in transcoding pipeline
// Attach output external buffers as the below for HDR output. SDR is by default, hence no
// extra input external buffer.
// The output Video Signal Information
mfxExtVideoSignalInfo outSignalInfo   = {};
outSignalInfo.Header.BufferId        = MFX_EXTBUFF_VIDEO_SIGNAL_INFO_OUT;
outSignalInfo.Header.BufferSz         = sizeof(mfxExtVideoSignalInfo);
outSignalInfo.VideoFullRange          = 0; // Limited range P010
outSignalInfo.ColourPrimaries         = 9; // BT.2020
outSignalInfo.TransferCharacteristics = 16; // ST2084

// The mastering display colour volume
mfxExtMasteringDisplayColourVolume outColourVolume = {};
outColourVolume.Header.BufferId = MFX_EXTBUFF_MASTERING_DISPLAY_COLOUR_VOLUME_OUT;
outColourVolume.Header.BufferSz = sizeof(mfxExtMasteringDisplayColourVolume);
// Based on the needs, Please set DisplayPrimaryX/Y[3], WhitePointX/Y, and MaxDisplayMasteringLuminance,
// MinDisplayMasteringLuminance

mfxExtBuffer *ExtBufferOut[2];
ExtBufferOut[0] = (mfxExtBuffer *)&outSignalInfo;
ExtBufferOut[2] = (mfxExtBuffer *)&outColourVolume;

mfxSession session      = (mfxSession)0;
mfxVideoParam VPPParams = {};
VPPParams.NumExtParam   = 2;
VPPParams.ExtParam      = (mfxExtBuffer **)&ExtBufferOut[0];
MFXVideoVPP_Init(session, &VPPParams);

The following pseudo code shows HDR to HDR.

// HDR to HDR (e.g P010 HDR signal -> P010 HDR signal) in transcoding pipeline
// Attach in/output external buffers as the below for HDR input/output. 
// The input Video Signal Information
mfxExtVideoSignalInfo inSignalInfo   = {};
inSignalInfo.Header.BufferId         = MFX_EXTBUFF_VIDEO_SIGNAL_INFO_IN;
inSignalInfo.Header.BufferSz         = sizeof(mfxExtVideoSignalInfo);
inSignalInfo.VideoFullRange          = 0; // Limited range P010
inSignalInfo.ColourPrimaries         = 9; // BT.2020
inSignalInfo.TransferCharacteristics = 16; // ST2084

// The content Light Level Information
mfxExtContentLightLevelInfo inContentLight = {};
inContentLight.Header.BufferId         = MFX_EXTBUFF_CONTENT_LIGHT_LEVEL_INFO;
inContentLight.Header.BufferSz         = sizeof(mfxExtContentLightLevelInfo);
inContentLight.MaxContentLightLevel    = 4000; // nits
inContentLight.MaxPicAverageLightLevel = 1000; // nits

// The mastering display colour volume
mfxExtMasteringDisplayColourVolume inColourVolume = {};
inColourVolume.Header.BufferId = MFX_EXTBUFF_MASTERING_DISPLAY_COLOUR_VOLUME_IN;
inColourVolume.Header.BufferSz = sizeof(mfxExtMasteringDisplayColourVolume);
// Based on the needs, Please set DisplayPrimaryX/Y[3], WhitePointX/Y, and MaxDisplayMasteringLuminance,
// MinDisplayMasteringLuminance

mfxExtVideoSignalInfo outSignalInfo   = {};
outSignalInfo.Header.BufferId         = MFX_EXTBUFF_VIDEO_SIGNAL_INFO_OUT;
outSignalInfo.Header.BufferSz         = sizeof(mfxExtVideoSignalInfo);
outSignalInfo.VideoFullRange          = 0; // Limited range P010
outSignalInfo.ColourPrimaries         = 9; // BT.2020
outSignalInfo.TransferCharacteristics = 16; // ST2084

// The mastering display colour volume
mfxExtMasteringDisplayColourVolume outColourVolume = {};
outColourVolume.Header.BufferId = MFX_EXTBUFF_MASTERING_DISPLAY_COLOUR_VOLUME_OUT;
outColourVolume.Header.BufferSz = sizeof(mfxExtMasteringDisplayColourVolume);
// Based on the needs, Please set DisplayPrimaryX/Y[3], WhitePointX/Y, and MaxDisplayMasteringLuminance,
// MinDisplayMasteringLuminance

mfxExtBuffer *ExtBuffer[5];
ExtBuffer[0] = (mfxExtBuffer *)&inSignalInfo;
ExtBuffer[1] = (mfxExtBuffer *)&inContentLight;
ExtBuffer[2] = (mfxExtBuffer *)&inColourVolume;
ExtBuffer[3] = (mfxExtBuffer *)&outSignalInfo;
ExtBuffer[4] = (mfxExtBuffer *)&outColourVolume;

mfxSession session      = (mfxSession)0;
mfxVideoParam VPPParams = {};
VPPParams.NumExtParam   = 5;
VPPParams.ExtParam      = (mfxExtBuffer **)&ExtBuffer[0];
MFXVideoVPP_Init(session, &VPPParams);

Camera RAW acceleration#

Intel^® VPL supports camera raw format processing with Intel HW. The following pseudo code shows how to perform camera raw hardware acceleration. For pipeline processing initialization, the application needs to attach the camera structures to the mfxVideoParam structure and call the MFXVideoVPP_Init() function.

The following pseudo code shows camera raw processing.

#ifdef ONEVPL_EXPERIMENTAL
// Camera Raw Format
mfxExtCamPipeControl pipeControl = {};
pipeControl.Header.BufferId      = MFX_EXTBUF_CAM_PIPECONTROL;
pipeControl.Header.BufferSz      = sizeof(mfxExtCamPipeControl);
pipeControl.RawFormat            = (mfxU16)MFX_CAM_BAYER_BGGR;

// Black level correction
mfxExtCamBlackLevelCorrection blackLevelCorrection   = {};
blackLevelCorrection.Header.BufferId                 = MFX_EXTBUF_CAM_BLACK_LEVEL_CORRECTION;
blackLevelCorrection.Header.BufferSz                 = sizeof(mfxExtCamBlackLevelCorrection);
mfxU16 black_level_B = 16, black_level_G0 = 16, black_level_G1 = 16, black_level_R = 16;
// Initialize the value for black level B, G0, G1, R as needed
blackLevelCorrection.B  = black_level_B;
blackLevelCorrection.G0 = black_level_G0;
blackLevelCorrection.G1 = black_level_G1;
blackLevelCorrection.R  = black_level_R;

mfxExtBuffer *ExtBufferIn[2];
ExtBufferIn[0] = (mfxExtBuffer *)&pipeControl;
ExtBufferIn[1] = (mfxExtBuffer *)&blackLevelCorrection;

mfxSession session      = (mfxSession)0;
mfxVideoParam VPPParams = {};
VPPParams.NumExtParam   = 2;
VPPParams.ExtParam      = (mfxExtBuffer **)&ExtBufferIn[0];
MFXVideoVPP_Init(session, &VPPParams);
#endif

Task submission synchronization#

Intel^® VPL can return synchronization object - syncpoint to notify application about submission a task to the GPU. The following example demonstrates the approach.

#ifdef ONEVPL_EXPERIMENTAL
mfxExtSyncSubmission syncSubmit = {};
syncSubmit.Header.BufferId      = MFX_EXTBUFF_SYNCSUBMISSION;
syncSubmit.Header.BufferSz      = sizeof(mfxExtSyncSubmission);

mfxFrameSurface1 in;
mfxFrameSurface1 out;
out.Data.ExtParam = (mfxExtBuffer **)&syncSubmit;
out.Data.NumExtParam=1;
sts=MFXVideoVPP_RunFrameVPPAsync(session,&in,&out,NULL,&syncp);
if (MFX_ERR_NONE == sts) {
    MFXVideoCORE_SyncOperation(session, *syncSubmit.SubmissionSyncPoint, INFINITE);
    run_your_GPU_kernel(&out);
}
#endif

Video Processing Procedures

Contents