Distributed training example with Intel® Optimization for Horovod*¶
Dependency¶
others show as below
pip install gin gin-config tensorflow-addons tensorflow-model-optimization tensorflow-datasets
Model examples preparation¶
Model Repo¶
git clone https://github.com/horovod/horovod.git # top commit: 0b19c
cd horovod/examples/tensorflow2
git apply tensorflow2_keras_mnist.patch
Notes:
Refer to tensorflow2_keras_mnist.py for other changes about how to enable horovod.
Execution¶
To run on a machine with 4 XPUs:
Notes:
Check log looking for “### is_mpi” and “### gpus” to get how many XPUs are in your platform. Our example is on a machine with 2 Intel® Data Center GPU Max Series. The 2 tiles in each GPU are taken as 2 independent XPU devices, for a total of 4 XPU devices.
horovodrun -np 4 -H localhost:4 python ./tensorflow2_keras_mnist.py
Output¶
...
[0] ### len(sys.argv) 1
[0] ### is_mpi 4
[0] ### gpus [PhysicalDevice(name='/physical_device:XPU:0', device_type='XPU'), PhysicalDevice(name='/physical_device:XPU:1', device_type='XPU'), PhysicalDevice(name='/physical_device:XPU:2', device_type='XPU'), PhysicalDevice(name='/physical_device:XPU:3', device_type='XPU')]
...
[3] 2022-12-05 10:45:55.830045: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:XPU:0 with 0 MB memory) -> physical PluggableDevice (device: 3, name: XPU, pci bus id: <undefined>)
[2] 2022-12-05 10:45:55.830610: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:XPU:0 with 0 MB memory) -> physical PluggableDevice (device: 2, name: XPU, pci bus id: <undefined>)
[1] 2022-12-05 10:45:55.830696: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:XPU:0 with 0 MB memory) -> physical PluggableDevice (device: 1, name: XPU, pci bus id: <undefined>)
[0] 2022-12-05 10:45:55.831773: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:XPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: XPU, pci bus id: <undefined>)
...
[3] 2022-12-05 10:45:58.971265: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type XPU is enabled.
[2] 2022-12-05 10:45:58.982479: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type XPU is enabled.
[1] 2022-12-05 10:45:59.003327: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type XPU is enabled.
[0] 2022-12-05 10:45:59.070484: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type XPU is enabled.
...
0] Epoch 1/24
1/125 [..............................] - ETA: 20:53 - loss: 2.3020 - accuracy: 0.0859
[0] Epoch 2/24
125/125 [==============================] - xxms/step - loss: 0.1776 - accuracy: 0.9456 - lr: 0.0030
...