This research analyzes the 'separation power' of equivariant neural networks, demonstrating how architectural choices, activation functions (like ReLU and sigmoid), and representation decomposition influence a model's ability to distinguish between inputs, providing a framework.
The paper investigates the separation power of equivariant neural networks, which serves as a measure of a model's expressivity. The authors characterize inputs that are indistinguishable by models of a given architecture. Key findings indicate that the choice of activation function—specifically non-polynomial ones like ReLU and sigmoid—results in maximum separation power, suggesting they are equivalent in expressivity. Furthermore, the study details how hyperparameters (depth, hidden layer width) and architectural features (adding invariant features, block decomposition) affect separability, revealing a hierarchy in separation power through block decomposition.