Training Flow and Diffusion Models

These are some notes I took, while watching MIT 6.S184: Lecture 03.

The big picture of arriving at a trainable objective in flow matching [1]:

1. Our ideal goal is to learn . So we can try to formulate the flow matching loss with the marginal vector field:

   Yet the issue is that, being intractable¹¹ because it requires an integral over the entire data distribution.

2. To fix this intractability, we use the Conditional Flow Matching (CFM) loss, which regresses against the easy-to-calculate conditional vector field .
   It can be proved that the Marginal flow matching loss equals the conditional flow matching loss () up to a constant that does not depend on the neural network parameters ().
   Because their gradients are the same (), we can minimize the easy, conditional loss to implicitly solve the hard, marginal one.

3. Now that we know we can use the conditional version, we pick a specific

   • For instance, we can use a Gaussian conditional probability path

   • So what’s the Target? For this path, we can analytically calculate exactly what should be using its time derivatives.

     using the standard reparametrization trick ²² we express as a function of standard Gaussian noise : This allows us to substitute every instance of in the equation with

4. Finally, we choose the simplest “schedulers” for and .
   The Setup: We set and .This specific linear interpolation gives us the Conditional Optimal Transport (CondOT) path.
   The Loss: This leads to the simple training objective:

   where

Doubts

1. how different is the diffusion objective from the flow matching loss?
2. Is cosine schedule of diffusion equivalent to linear interpolation in FM?
3. Is Euler method used in practice? or do people prefer higher order methods? Heun’s method?