Just skimmed the paper (haven't seen this before). Might need to read more carefully, but at first glance I don't understand their intuition why the "Markovian property limits the model’s ability to fully utilize the generation trajectory".
When it comes to multi-resolution training (e.g. matryoshka training), there are precedents that don't require this AR formulation.
They cite MAR (https://arxiv.org/pdf/2406.11838), which I think is a much clearer articulation of "autoregressive diffusion". MAR uses an autoregressive base (like an LLM) and staples on a small MLP on-top that's trained as a diffusion head. That makes more sense to me, since you can leverage the "knowledge prior" from an LLM and have it generate images. That's probably how nano-bannana and GPT-Image broadly work.
Zooming out, it's not clear to me from any of the work WHY autoregressive diffusion on it's own is better than regular diffusion. Most autoregressive diffusion models use speculative decoding, because pure autoregressive diffusion is too slow to run at inference time.
The one ~magical~ thing about auto-regressive diffusion (IMO) has nothing to do with autoregressive vs. fully-bidirectional diffusion. But simply, the face that you can do it on-top of a LLM. That means the LLM can look at it's generation, assess it's quality, think on what's broken, and then call itself to edit the image and fix it. With methods like RLVR, that means you can essentially guarantee the correctness of your image (along the axes you've RLVR-ed).
By schopra909 23 minutes ago
In my opinion its a great idea because you get reuse hidden states which allows you to do something akin to thinking/[online learning]. If you only evolve use input space outputs you're dealing with more decoding pressure.
When it comes to inference it should also be cheaper (since we’ve cut attention sequence length by 4x), but I don’t have a hard number for you how much cheaper for inference
By schopra909 4 hours ago
By E-Reverance 43 minutes ago
By schopra909 23 minutes ago
By E-Reverance 7 minutes ago
By popalchemist an hour ago
By schopra909 an hour ago