Parallel Computing Cluster for Solving Computational Problems in Data: A Framework for High-Throughput Data Processing - Softcover

Mutyala Rao, Chavala

 
9786209340734: Parallel Computing Cluster for Solving Computational Problems in Data: A Framework for High-Throughput Data Processing
Softcover
ISBN 10: 6209340733 ISBN 13: 9786209340734
Verlag: LAP LAMBERT Academic Publishing, 2025

Inhaltsangabe

Thisbook tackles a critical bottleneck in large-scale AI: the slow and communication-heavy training of massive Deep Neural Networks (DNNs) on multi-GPU systems. It addresses the trade-off between two main parallelization methods. Data parallelism suffers from severe communication overhead for large models, while pipelined model parallelism (like PipeDream) offers up to 8.91x speedup for large Fully Connected/Recurrent Neural Networks but causes "weight staleness," degrading model accuracy. To resolve this, the paper introduces SpecTrain, a novel technique. SpecTrain uses the momentum from optimizers to predict future weight updates, allowing pipelined computation with accurate, non-stale weights. This enables the high GPU utilization and speed of pipelining while maintaining the training robustness and final accuracy of synchronous methods.

Die Inhaltsangabe kann sich auf eine andere Ausgabe dieses Titels beziehen.

Über die Autorin bzw. den Autor

Chavala Mutyala Rao is an Assistant Professor in Information Technology at MANUU Polytechnic, Hyderabad.

„Über diesen Titel“ kann sich auf eine andere Ausgabe dieses Titels beziehen.

Verlag
LAP LAMBERT Academic Publishing
Erscheinungsdatum
2025
Sprache
Englisch
ISBN 10 
6209340733
ISBN 13 
9786209340734
Einband
Taschenbuch
Anzahl der Seiten
56
Kontakt zum Hersteller
preigu GmbH & Co. KG
mail@preigu.de

Lengericher Landstr. 19
Osnabrück
Niedersachsen
49078
Deutschland