Reproducing DeepSeek’s mHC: When Hyper-Connections Cause Explosions
TL;DR A small-scale reproduction of DeepSeek’s manifold-constrained Hyper-Connections (mHC) shows unconstrained mixing matrices can amplify signals exponentially, producing unstable training…