High Performance Computing

Title High Performance Computing (51087)
Quarter Spring 2020
Instructor Ronald Rahaman (rahaman@cs.uchicago.edu)


Course Description


Parallel programming is ubiquitous in both the largest, power-intensive data centers and the smallest, low-power embedded devices.  Though this has been the status quo for many years, achieving optimal parallel performance can still be a challenging, multi-disciplinary effort. 


In this course, we will implement and optimize C applications on large-scale, multicore CPU and GPU compute clusters.  We will cover details about widely-used parallel programming APIs (OpenMP, CUDA, and MPI).  Computer architecture will be a pervasive theme, and we will discuss how the C language and parallel APIs map to the underlying hardware. 


We will be programming exclusively in C, and no other knowledge is assumed.  Previous MPCS courses in computer architecture, compilers, or algorithms can be very helpful but are absolutely not required.  




  • Overview of CPU and GPU Architectures
    • Instruction sets
    • Functional units
    • Memory hierarchies
  • Performance Metrics
    • Latency and bandwidth
    • Roofline modeling
  • Single-core optimization
    • Compiler-assisted vectorization (data-level parallelism)
    • Design patterns for cache-based optimization
  • Multi-threaded CPU programming
    • Worksharing, synchronization, and atomic operations
    • Memory access patterns, including non-uniform memory access
    • The OpenMP API
  • GPU programming
    • Thread-mapping for optimal vectorization and memory access
    • Task-scheduling for latency reduction
    • The CUDA and OpenMP offload APIs
  • Distributed parallelism
    • Synchronous and asynchronous communication patterns
    • Data decomposition
    • Hybrid models for distributed multi-threaded and GPU programming
    • The MPI API


Throughout the course, will draw on examples from linear algebra, Monte Carlo simulations, discretized partial differential equations, and machine learning. 




This class is primarily project-based.  We will have 4-5 bi-weekly homework assignments and 2 brief concept-based exams. 




We will draw on material from the following texts.  None are required, but they can be helpful resources throughout your career. 


Prerequisites (Courses)

MPCS 51040 - C Programming or MPCS 51100 Advanced Programming or instructor consent.

Prerequisites (Other)


High Performance Computing Specialization (https://masters.cs.uchicago.edu/page/high-performance-computing)


Friday 5:30-8:30PM


JCL 011