MPCS 53113 Natural Language Processing (Summer 2024)

Section 1
Instructor(s) Chaudhary, Amitabh (amitabh)
Location Online Only
Meeting Times Wednesday 5:30pm - 8:30pm
Fulfills Elective Specialization - Artificial Intelligence (AI-2)

Syllabus

Natural language processing (NLP) is the application of computational techniques, particularly from machine learning, to analyze and synthesize  human language. The recent explosion in the amount of available text data has made natural language processing invaluable for businesses, social sciences, and even natural sciences.
 
In this course we study the fundamentals of modern natural language processing, emphasizing models based on deep learning.   These include language models,  word embeddings,  recurrent neural networks (Simple RNNs, LSTMs), context-free grammars and syntactic parsing, convolutional neural networks (CNNs), and attention-based models such as the Transformer, BERT, and the GPT models.
 
We use Python and the deep learning framework PyTorch to build models from scratch and apply them to tasks such as machine translation.   We also use the library of pre-trained models HuggingFace to build even more sophisticated models that can, for example, automatically generate summaries of articles.
 
A significant component is the course project in which students build and train models to solve a real-world problem of their choice.
 
A tentative list of topics follows.
  • Language Models
  • Word Embeddings
  • Recurrent Neural Networks (RNNs), LSTMs
  • Convolutional Neural Networks (CNNs) for NLP
  • Conditioned Generation, Sequence to Sequence Models
  • Sytactic Parsing, Context Free Grammars
  • Attention-Based Models, The Transformer
  • Advanced models: BERT, GPT 1-4, Google PaLM. 

Coursework and Evaluation

  • Homework Assignments:  There are about three assignments, designed to reinforce material and test a deeper understanding of the concepts and algorithms through theoretical questions, program implementation, and analysis of empirical results.  Tentatively worth 40% of the grade.
  • Midterm Examination: Tentatively worth 20% of the grade.
  • Course Project: Chosen by the student, and entatively worth 40% of the grade.

Textbook

  • Neural Network Methods for Natural Language Processing by Yoav Goldberg (https://doi.org/10.2200/S00762ED1V01Y201703HLT037)
  • Research papers 

Course Prerequisites

A grade of B+ or better in the following courses:
• MPCS 50103 Math for Computer Science (or placement exam waiver)
• MPCS 51042 Python Programming

A grade of B or better in MPCS 55001 Algorithms

A grade of B or better in one of the following courses:
• MPCS 53110 Foundations of Computational Data Analysis
• MPCS 53120 Applied Data Analysis

MPCS 53111 Machine Learning (recommended; see below).
Equivalent courses or experience will be accepted with instructor permission. A prior course in machine learning would be useful but is not necessary; if you haven't taken any please contact the instructor with your prior courses and experience.

Other Prerequisites

Programming experience in Python.

For MPCS students, this is an elective/specialization course and can only be taken after completion of three core classes or concurrent with third core class.

This course requires competency in Unix and Linux. Please plan to attend the MPCS Unix Bootcamp (https://masters.cs.uchicago.edu/page/mpcs-unix-bootcamp) and/or review the UChicago CS Student Resource Guide here: https://uchicago-cs.github.io/student-resource-guide/.

Overlapping Classes

This class is scheduled at a time that conflicts with these other classes:

  • MPCS 52072-1 -- GPU Programming
  • MPCS 52553-1 -- Web Development
  • MPCS 50101-1 -- Concepts of Programming

Eligible Programs

MS in Molecular Engineering MA in Computational Social Science (Year 1) MA in Computational Social Science (Year 2) Bx/MS in Computer Science (Option 2: Professionally-oriented - CS Majors) Bx/MS in Computer Science (Option 3: Profesionally-oriented - Non-CS Majors) Masters Program in Computer Science