CS846: FMxAI

Federico Mora · University of Waterloo · Fall '26 · Tues 3:00-5:50pm · DC 2585

Grading

Our meetings will be centered around paper discussions and in-class activities. Attendance, reading of assigned papers, and participation are essential.

  • 20% participation
  • 30% discussion leads
  • 50% project

Project

Every project must contain new research—a new idea or insight—and an element of both formal methods and artificial intelligence. Students can work in groups of up to three.

  1. Proposal (due Oct 09)
  2. Mini presentation (Nov 03)
  3. Final presentation (Dec 08)
  4. Report (due Dec 18)

Discussion Leads

For research papers, discussion leads present a review with four components:

  • research context (~5 minutes)
  • paper summary (~15 minutes)
  • strengths and weaknesses (~5 minutes)
  • what has changed or will change (~5 minutes)

For textbooks, discussion leads give a 30-minute tutorial. For blog posts, discussion leads give a 30-minute deep-dive.

Schedule

All due dates are end of day on the date listed.

Date Description Read
Sep 15 Introduction and overview 1, 2
Sep 15 Discussion preferences due
Sep 22 Agents and tool-use ATU 1, ?
Sep 29 Constraints on LLM outputs CLO 1, ?
Oct 06 Generating system constraints GSC ?
Oct 09 Proposals due
Oct 13 Reading week
Oct 20 Satisfiability (modulo theories) SAT 1, ?
Oct 27 Program verification PVE 1, ?
Nov 03 Mini presentations day
Nov 10 Superoptimization SUP ?
Nov 17 Theorem proving TPR ?
Nov 24 Machine-learning verification MLV 1, ?
Dec 01 Bonus 1, ?
Dec 08 Final presentations day
Dec 18 Reports due

Agents and Tool-Use (Sep 22)

ATU

LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers

Olausson et al., EMNLP '23

ATU
Diagram of the LINC pipeline: an LLM semantic parser samples logic formulas from a natural-language problem, a theorem prover checks each sample and filters out syntax/semantic errors, and majority voting over the remaining labels produces the final output

Reinforcement Learning from Human Feedback, Chapter 13

Lambert, Textbook '26

ATU
Diagram of tool use interleaving model generation with external tool execution
Figure 1: Tool use interleaves model generation with external execution: the model generates tokens until it emits a tool call (orange), an external system executes the tool and injects the output (purple) into the sequence, and then the model continues generating. Models can emit multiple tool calls in a single generation. During training, tool call and output tokens are typically masked from the loss.

Towards Verifiably Safe Tool Use for LLM Agents

Doshi et al., ICSE-NIER '26

ATU
Example agent conversation where an LLM agent reschedules a meeting and, without being asked, emails the other attendee an explanation that leaks the sensitive reason for the reschedule

Constraints on LLM Outputs (Sep 29)

CLO

How To Generate Text: Using Different Decoding Methods For Language Generation With Transformers

von Platen, Blog '20

CLO
Top-K sampling illustrated with two bar charts of word probabilities: the top 6 words after 'The' cover 68% of probability mass, while the top 6 after 'The car' cover 99%, showing that a fixed K can be too small or too large depending on the distribution
Example sampling startegy: top-K sampling redistributes probability mass among the K most likely next words, but a fixed K can cut off a flat distribution too early or admit too many words from a peaked one.

Constrained Adaptive Rejection Sampling

Parys et al., ICML '26

CLO
Example 3.1: a grammar for arithmetic expressions over digits 0 and 1, where strings like 1+0+1 are accepted but 0++ or +1 are not
Plot comparing approximation error versus number of samples for CARS against other exact and inexact constrained sampling methods, showing CARS reaches low error with fewer samples than most baselines

Type-Constrained Code Generation with Language Models

Mündler et al., PLDI '25

CLO
A partial TypeScript program awaiting completion, with five candidate completions shown alongside whether vanilla, syntax-constrained, and type-constrained decoding accept or reject each one; only type-constrained decoding correctly rejects all four invalid completions and accepts the well-formed one

Synthetic Programming Elicitation for Text-to-Code in Very Low-Resource Programming and Formal Languages

Mora et al., NeurIPS '24

CLO
A desire path meme: a person walks the worn dirt shortcut labeled 'LLM Outputs' instead of the paved path labeled 'Formal Language', illustrating that LLM-generated code naturally drifts toward informal syntax rather than the target formal language

Generating System Constraints (Oct 06)

GSC

Mining Specifications

Ammons et al., POPL '02

GSC
Problem 2.1: given an unlabelled training set of interaction traces with an API or ADT, find an automaton that generates exactly the correct traces; this automaton is called a specification, and an algorithm that finds it is called a specification miner

Learning Context-Free Grammars for Grammar-Constrained Decoding via Declarative Agentic Programming with Guarantees

Cheang et al., arXiv '26

GSC
Autogrammar's two-phase pipeline: in the learning phase, execution logs and API documentation are used to produce a context-free grammar for the target DSL; in the deployment phase, a downstream LM uses the grammar through a validator to generate correct DSL programs in response to natural-language tasks

A Neurosymbolic Approach to Natural Language Formalization and Verification

An et al., CAV '26

GSC
End-to-end architecture of ARC: a Policy Model Creator autoformalizes a natural-language policy document into a policy model, refines it via test-case enumeration, conflict detection, and user feedback, and an Answer Verifier autoformalizes a question-and-answer conversation, aggregates logical equivalence, and checks it against the policy model with an SMT solver

Speed Matching

  • Find a pair (30 seconds)
  • Describe a research project that you have worked on or that you would like to work on (1 minute)
  • Listen to a research project that your pair has worked on or would like to work on (1 minute)
  • Brainstorm with your pair on a project that you could work on together (2 minutes)
  • Find a new pair (30 seconds) and repeat

Satisfiability (Modulo Theories) (Oct 20)

SAT

Introduction to Neural Network Verification, Chapters 4, 6 (7 optional)

Albarghouthi, Textbook '26

SAT
Chapter 4 title banner: Logics and Satisfiability
Chapter 6 title banner: DPLL Modulo Theories

Guiding High-Performance SAT Solvers with Unsat-Core Predictions

Selsam and Bjørner, SAT '19

SAT

The NeuroSAT neural network architecture was introduced in [37] for predicting properties of propositional formulae. When trained to predict the satisfiability of toy problems, it was shown to find solutions and unsatisfiable cores on its own. However, the authors saw "no obvious path" to using the architecture to improve the state-of-the-art. In this work, we train a simplified NeuroSAT architecture to directly predict the unsatisfiable cores of real problems. We modify several high-performance SAT solvers to periodically replace their variable activity scores with NeuroSAT’s prediction of how likely the variables are to appear in an unsatisfiable core.

In short,

... we use it to help inform variable branching decisions within high-performance SAT solvers on real problems.

Learning to Solve SMT Formulas

Balunović et al., NeurIPS '18

SAT
Search tree over tactic applications: starting from formula phi, each transformation (tactic) branches into new formulas, some of which time out and others of which are solved quickly, illustrating the need to choose tactics wisely
The action space of available tactics, including constant folding, normalizing bounds, bit blasting, and calling a decision procedure over reals, integers, or bitvectors

SATzilla: Portfolio-based Algorithm Selection for SAT

Xu et al., JAIR '08

SAT

It has been widely observed that there is no single "dominant" SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis.

Flowchart of SATzilla's online phase: select a SAT instance from test data, compute features, use hardness models to predict algorithm runtimes, run the predicted fastest algorithm, and return the result
Source: Costa, Natarajan, Rosa, Shao, and Zhai, "SATzilla", Blog '20.

MedleySolver: Online SMT Algorithm Selection

Pimpalkhare et al., SAT '21

SAT
Pipeline diagram: queries from SMT-enabled tools like UCLID5, KLEE, and the Rosette language are streamed into MedleySolver, which learns online and outputs SAT/UNSAT for each query

SMT users rarely aim to solve a single query in isolation and usually care about [total] resource consumption. For example, verification engines generate many verification queries for one verification problem and aim to solve these queries in the least amount of time.

Domain-Specific Hyperspecialization (For SAT)

Green, Blog '26

SAT
Satirical diagram mapping four NP-hard problems (graph coloring, bin packing, set cover, SAT) each to its own bespoke custom solver, instead of reducing them all to a single general SAT solver

Program Verification (Oct 27)

PVE

Getting Started with Dafny: A Guide

Leino, Textbook '23

PVE
Dafny code for binary search: a sorted predicate and a BinarySearch method with requires/ensures contracts and a while loop annotated with invariants, verified statically against the specification

AutoVerus: Automated Proof Generation for Rust Code

Yang et al., OOPSLA '25

PVE
A Rust count_digits function alongside its Verus spec functions, with the proof annotations (loop invariants and asserts) needed to verify the implementation against the specification highlighted separately from the specification itself

VERINA: Benchmarking Verifiable Code Generation

Ye et al., ICLR '26

PVE

Verifiable code generation—jointly generating code, specifications, and proofs of code-specification alignment—offers a promising path to address this limitation and further unleash LLMs’ benefits in coding. Yet, there exists a significant gap in evaluation: current benchmarks often focus on only individual components rather than providing a holistic evaluation framework of all tasks.

But proof success is quite low:

The best model, OpenAI o3, achieves a 72.6% code correctness rate, 52.3% for specification soundness and completeness, and a mere 4.9% proof success rate (based on one trial per task).

Large Language Model Powered Symbolic Execution

Li et al., OOPSLA '25

PVE
Example workflow: source code is parsed into an AST, a path slice is generated and rendered back into source, then an LLM is prompted with a pre-condition, the slice, and a post-condition to decide whether the post-condition holds along that path, passing or failing accordingly

Superoptimization (Nov 10)

SUP

Program Synthesis is Possible

Sampson, Blog '18

SUP

you have the "slow" expression x * 2, and you know that there's a "faster" version to be had that can be written x << ?? for some value of ??. Let's ask Z3 what to write there...

Stochastic Superoptimization

Schkufza et al., ASPLOS '13

SUP
Cycling Through 3 Values benchmark: gcc -O3 translates the esoteric bit-twiddling implementation almost literally into assembly, while STOKE rediscovers the intuitive algorithm using conditional move instructions in far fewer instructions

Faster Sorting Algorithms Discovered Using Deep Reinforcement Learning

Mankowitz et al., Nature '23

SUP
Diagram of the AssemblyGame: AlphaDev selects an assembly instruction to append to the algorithm being built, then all test input sequences are run through the algorithm and the outputs are compared against expected outputs to compute a correctness-based reward

Mathematical Discoveries from Program Search with Large Language Models

Romera-Paredes et al., Nature '23

SUP
Diagram of the FunSearch loop: a specification and prompt built from high-scoring programs sampled from a programs database are fed to a pretrained LLM, which generates new programs that are evaluated and, if correct, stored back in the programs database, with the highest-scoring program retrievable at any point
Overview of FunSearch. The input to FunSearch is a specification of the problem in the form of an 'evaluate' function, an initial implementation of the function to evolve, which can be trivial, and potentially a skeleton. At each iteration, FunSearch builds a prompt by combining several programs sampled from the programs database (favouring high-scoring ones). The prompt is then fed to the pretrained LLM and new programs are created. Newly created programs are then scored and stored in the programs database (if correct), thus closing the loop. The user can at any point retrieve the highest-scoring programs discovered so far.

Speed Matching

  • Find a pair (30 seconds)
  • Describe a research project that you have worked on or that you would like to work on (1 minute)
  • Listen to a research project that your pair has worked on or would like to work on (1 minute)
  • Brainstorm with your pair on a project that you could work on together (2 minutes)
  • Find a new pair (30 seconds) and repeat

Theorem Proving (Nov 17)

TPR

Generative Language Modeling for Automated Theorem Proving

Polu and Sutskever, arXiv '20

TPR

We present an automated prover and proof assistant, GPT-f, for the Metamath formalization language, and analyze its performance. GPT-f found new short proofs that were accepted into the main Metamath library, which is to our knowledge, the first time a deep learning based system has contributed proofs that were adopted by a formal mathematics community.

Hilbert: Recursively Building Formal Proofs with Informal Reasoning

Varambally et al., ICLR '26

TPR
Diagram of the Hilbert algorithm: given a target theorem, a prover attempts formal proof generation; on failure the theorem is decomposed into subgoal theorems that are each retried with the prover and a reasoner (shallow solve), recursing into further decomposition until all subgoals are resolved

Postmortem for Kernel Soundness Bug #14576

de Moura, Blog '26

TPR

On July 25, Ramana Kumar published a repository containing a sorry-free "disproof" of the Collatz conjecture, produced with AI assistance. It is not a valid proof because it exploits a bug in the kernel's handling of nested inductive types.

Machine-Learning Verification (Nov 24)

MLV

Introduction to Neural Network Verification, Chapters 1, 2, 3, 5

Albarghouthi, Textbook '26

MLV

This book is about verifying that a neural network behaves according to some set of desirable properties.

Importantly,

A number of people in the verification community, the author included, argue that specification [deciding on the set of desirable properties] is harder than verification—that is, the hard part is asking the right questions!

VNN-LIB 2.0: Rigorous Foundations for Neural Network Verification

Roy et al., CAV '26

MLV
(vnnlib-version <2.0>)

; Network declaration
(declare-network f
    (declare-input  X float32 [2,2])
    (declare-output Y float32 [1])
)

; Input constraints
(assert (and (>= X[0,0] 0.0) (<= X[0,0] 1.0)))
(assert (and (>= X[0,1] 0.0) (<= X[0,1] 1.0)))
(assert (and (>= X[1,0] 0.0) (<= X[1,0] 1.0)))
(assert (and (>= X[1,1] 0.0) (<= X[1,1] 1.0)))

; Output constraints
(assert (or (<= Y[0] -3.0) (>= Y[0] 0.0)))

Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Robustness Verification

Wang et al., NeurIPS '21

MLV
The αβ-CROWN logo: a crown badge reading 'winner of International Verification of Neural Networks Competitions (VNN-COMP 2021-2025)'

Bonus or Guest Talks (Dec 01)

Discussion Preferences (Due Sep 15)

Submit your discussion preferences today through the LEARN quiz.

You will be matched to papers based on your submitted preferences on Sep 16.

Speed Matching

  • Find a pair (30 seconds)
  • Describe a research project that you have worked on or that you would like to work on (1 minute)
  • Listen to a research project that your pair has worked on or would like to work on (1 minute)
  • Brainstorm with your pair on a project that you could work on together (2 minutes)
  • Find a new pair (30 seconds) and repeat

References

  • Seshia, S. A. (2024). EECS 219C: Formal methods: Specification, verification, and synthesis Course website. University of California, Berkeley.
  • Chasins, S. E. (2024). CS 294-184: Building user-centered programming tools Course website. University of California, Berkeley.
  • Willsey, M. (2024). CS 294-260: Declarative program analysis and optimization Course website. University of California, Berkeley.
  • Artificial Intelligence Tool(s): Claude (Sonnet 5), Anthropic; Writing: Claude was used to fix spelling and grammatical mistakes in the syllabus; Project Administration: Claude was used to build and maintain Markdown-to-HTML scripts and keep the Schedule and Reading Bank tables in sync.
1 / 1