Programming Massively Parallel Processors

Wen-mei W. Hwuis a Senior Director of Research of NVIDIA and the Sanders-AMD Endowed Chair Professor Emeritus of Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign. His work focuses on parallel computing—covering architecture, implementation, compilers, and algorithms. Dr. Hwu has received numerous honors, including the ACM/IEEE Eckert-Mauchly Award, ACM Grace Murray Hopper Award, IEEE B.R. Rau Award. He is an IEEE and ACM Fellow. He earned his Ph.D. in Computer Science from UC Berkele David B. Kirkis known for major contributions to graphics, hardware, and algorithms. Before pursuing his Ph.D. at Caltech, he earned B.S. and M.S. degrees in mechanical engineering from MIT and worked at Raster Technologies and Hewlett-Packard’s Apollo Systems Division. After completing his doctorate, he served as chief scientist and head of technology at Crystal Dynamics. In 1997, he became Chief Scientist at NVIDIA. Dr. Kirk has received numerous honors including the IEEE Seymour Cray Computer Engineering Award and ACM SIGGRAPH Computer Graphics Achievement Award. He is a member of the U.S. National Academy of Engineering. Izzat El Hajjis an Assistant Professor of Computer Science at the American University of Beirut. His research focuses on leveraging accelerator architectures to tackle challenging computations, with a focus on GPU computing, processing-in-memory, and performance modeling. He earned his Ph.D. in Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign. He has received the Dan Vivoli Endowed Fellowship (UIUC) and the Distinguished Graduate Award from the American University of Beirut.

• 1. IntroductionPart I. Fundamental Concepts2. Heterogeneous data parallel computing3. Multidimensional grids and data4. Compute architecture and scheduling5. Memory architecture and data locality6. Performance considerationsPart II. Parallel Patterns7. Convolution8. Stencil9. Parallel histogram10. Reduction11. Prefix sum (scan)12. MergePart III. Advanced Patterns and Applications13. Sorting14. Filtering (new)15. Sparse matrix computation16. Wavefront Algorithms (new)17. Graph traversal18. Deep learning19. Multi-GPU API (new)20. Electrostatic potential map21. Parallel programming and computational thinkingPart IV. Advanced Practices22. Programming a heterogeneous computing cluster23. Advanced Optimizations for Matrix Multiplication (new)24. Advanced practices and future evolution25. Conclusion and outlook