Nunchux AI Introduces VC-Attention for Video DiTs

Generating artificial intelligence video clips using Video Diffusion Transformers, or Video DiTs, often encounters heavy computational bottlenecks. A 5-second 720p clip from Wan2.2-14B expands into approximately 70,000 spatiotemporal tokens, causing attention mechanisms to consume more than 64% of generation time on an RTX 5090 graphics card. Meanwhile, researchers report that attention takes about two-thirds of every denoising step on a single B200 chip.

Although low-bit Tensor Cores accelerate the two matrix products QK and PV, two major obstacles remain. First, prior methods such as SageAttention2 smooth queries and keys, but after this smoothing and rotation, the value term accounts for 82% of output error on Wan2.2 models. Second, the softmax calculation between products continues to run in FP32, making the exponential function and its FP8 conversion the longest pipeline stages on B200 and H200 hardware.

Stock photo for illustration only, not from the actual event

To address these challenges, the research team implemented a grouping technique restricted to the first 25% of denoising steps, reusing permutations across four adjacent steps. Averaged over the full schedule, grouping incurs a cost of only 3% to 4% of attention time. Additionally, ExpCast-FP8 is employed to write bytes directly from log-domain scores, reducing error rates and enhancing processing efficiency.