With PyTorch 1.5, the RPC framework can be used to build training applications that make use of distributed architectures if they’re available.

Soumith Chintala, PyTorch project lead, seems to share Zaharia's ideas about distributed training being the next big thing in deep learning, as it has been introduced in the latest version of PyTorch.

The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of- core DNN training, and 4) Hybrid (Data and Model) parallelism. Case ...

In this video from the Swiss HPC Conference, DK Panda from Ohio State University presents: Scalable and Distributed DNN Training on Modern HPC Systems. The current wave of advances in Deep Learning ...