Publication
Gradient compression is a promising approach to alleviating the communication bottleneck in data parallel DNN training. This paper introduces Tensor Homomorphic Compression (THC), a novel bi-directional compression framework that enables direct aggregation of compressed values while optimizing compression ratios, achieving significant speedups in data parallel training.
Youhui Bai, Cheng Li, Quan Zhou, Jun Yi, Ping Gong, Feng Yan, Ruichuan Chen, Yinlong Xu
Tensor Homomorphic Compression (THC) enables direct aggregation of compressed gradients while maintaining compression efficiency, significantly reducing communication bottlenecks in data parallel DNN training.