Graph fusion

Intel® Extension for TensorFlow* provides graph optimization to fuse specified operator patterns into a new single operator for better performance.

Basic fusion

The basic list of supported fusions is shown below. These fusions require input and output of the same data type.

Pattern	Operator number
(Equal, NotEqual, GreaterEqual, Greater, LessEqual, Less)+Cast	2
L2loss+AddN	2
BatchMatMul+Mul	2
Mul+AddN+TrainingOp	3
Conv+Bias	2
Conv+Bias+(Relu, Relu6, Elu, LeakyRelu, Gelu_erf, Gelu_tanh, Tanh, Sigmoid)	3
MatMul+Bias	2
MatMul+Bias+(Relu, Relu6, Elu, Gelu_erf, Gelu_tanh, Tanh, Sigmoid)	3
FusedBatchNorm+Relu	2
FusedBatchNormGrad+ReluGrad	2
Conv+Bias+Add	3
Conv+Bias+Add+(Relu, Relu6, Elu, LeakyRelu, Gelu_erf, Gelu_tanh, Tanh, Sigmoid)	4
MatMul+Bias+Add	3
MatMul+Bias+Add+(Relu, Relu6, Elu, Gelu_erf, Gelu_tanh, Tanh, Sigmoid)	4
MatMul+BiasAddGrad	2
ConvGradFilter+BiasAddGrad	2
Pad+Conv	2
BatchMatMul with variable post-op	2+
Swish	2
LayerNorm	3+

Mixed data type fusion

As stock TensorFlow only supports same-data-type input and output, inserting a cast node during BF16 inference and training may break the existing fusion pattern and impact performance.

Intel® Extension for TensorFlow* provides mixed data type fusion, which removes the additional data type conversions on the graph level.

Here is the list of supported mixed data type fusions, and we’ll take a closer look at MatMul as an example.

Pattern	Fused operator	Input data type	Output data type	oneDNN FP32 Math mode
MatMul + Cast	AccMatMul	BF16	FP32	N/A
FusedMatMul + Cast	FusedAccMatMul	BF16	FP32	N/A
AccMatMul + any MatMul Fusion	FusedAccMatMul	BF16	FP32	N/A
Cast + MatMul + Cast	AccMatMul	FP32	FP32	BF16
Cast + FusedMatMul + Cast	FusedAccMatMul	FP32	FP32	BF16

Implementation Details

The Cast + (Fused)MatMul + Cast pattern is covered by pattern matcher; the rest is covered by remapper fusion. The new kernels are implemented（AccMatMul and FusedAccMatMul(WithSum)）as an extension of original MatMul with the following new attributes:

• Tout: Output data type ∈ {float32}.

• Tpost: Post op data type ∈ {bfloat16, float32}.

• is_bf16_math_mode: A Boolean to indicate whether to use oneDNN BF16 math mode if FP32 input, FP32 output.

Generic layout optimizer

As the channels_first format is not supported by stock TensorFlow on CPU, it inserts transpose nodes before and after the Conv3D/MaxPool3D nodes. However, this problem does not exist in GPU device. To avoid unnecessary layout transformation when running on a GPU device, Intel® Extension for TensorFlow* adds a separate layout optimizer.

Pattern	Fused operator	Conv data format (before optimization)	Conv data format (after optimization)
Transpose + Conv3D + Transpose	Conv3D	NDHWC	NCDHW