Huggingface

This listing describes a custom GPU kernel written in Triton that accelerates training of a sparse feed-forward network (FFN). With K=256, 256 of 4,096 FFN neurons are active per token (6.25%). The author reports 36% higher training throughput than the dense counterpart.

The kernel has been tested so far on a single NVIDIA RTX PRO 6000 GPU and requires GPU hardware with BF16 support. The page notes that whether a sparse-trained, later densified model matches the quality of a densely trained model remains an open question.