کنفرانس مهندسی برق ایران

صفحه اصلی / سی و چهارمین کنفرانس بین المللی مهندسی برق

Layer-wise Representation Analysis in Vision Transformers Using Separation Index

نویسندگان :

Navid Dehban¹ Ahmad Kalhor²

1- University of Tehran 2- University of Tehran

کلمات کلیدی :

Vision Transformer،Representation Analysis،Separation Index،Layer-wise Analysis،Neural Network Interpretability

چکیده :

Vision Transformers (ViTs) have achieved competitive performance on a wide range of image recognition benchmarks, but the way their intermediate layers build class-discriminative representations is still not fully understood. In this paper, we present a systematic, layer-wise analysis of representation quality in ViTs using the Separation Index (SI), a simple but powerful metric that quantifies how well feature embeddings separate classes based on nearest-neighbor consistency in feature space. We evaluate three standard ViT architectures (Tiny, Base, and Large) on three datasets with different levels of complexity (CIFAR-10, CIFAR-100, and Tiny-ImageNet), and compute SI after each transformer block using CLS-token features. Our study reveals several consistent behaviors: SI grows monotonically with depth and exhibits clear saturation, the saturation depth and maximum SI depend jointly on model capacity and dataset difficulty, and deeper blocks primarily refine already linearly separable features rather than creating separability from scratch. We further show that SI can highlight overcapacity and under capacity regimes across architectures and datasets, providing a representation-driven perspective on depth allocation. These findings suggest that SI is a useful tool for understanding and guiding architecture design in vision transformers, and they naturally motivate SI-based structural tuning and pruning strategies explored as future work.