This code base provides the numerical experiments discussed in the paper
It contains the training setup for various narrow/non-augmented ResNets including neural ODEs and MLPs.
In the one-dimensional setting, we train models to approximate x^2 and analyze the present embedding restrictions
In the neural ODE regime, the narrow ResNet cannot express critical points. This topological constraint creates a "tunnel" in the prediction level sets, leading to unavoidable misclassifications on the 2D circle dataset.
Similarly, in the standard MLP regime, the skip connection's influence vanishes and the architecture once again loses the ability to embed critical points. The resulting input-output map generates the exact same "tunnel effect", failing to capture the enclosed topology of the data.
Standard ResNets are able to embed critical points and as a result express the desired topology of the data accurately.