GEOMETRIC–SPECTRAL STABILITY ANALYSIS OF TRANSFORMER REPRESENTATIONS: EXTENDING THE REPRESENTATION STABILITY SCORE TO NATURAL LANGUAGE PROCESSING

Abstract

To gain insights into the geometric and spectral characteristics of transformer representations, beyond typical downstream performance metrics. We generalize the Representation Stability Score (RSS) to pretrained transformer language models, a unified metric of the intrinsic dimensionality and numerical conditioning of the representations. The layer-wise representations extracted from BERT, RoBERTa, DistilBERT, and ALBERT are analyzed on the SST-2 and IMDb sentiment classification benchmarks using spectral entropy, effective rank, condition number, and RSS. All experiments are replicated with five independent random seeds, and the results are reported as mean, standard deviation, and 95\% bootstrap confidence intervals to ensure statistical reliability. Moreover, a cross domain test is carried out between the AG news and the 20 Newsgroups cross domain test (the sci.med subset) to study the sensitivity of the representation geometry to the domain shift. Different stability profiles are observed for different transformer models, as shown by the results. BERT shows the increasing stability of the representations from the deeper layers, while DistilBERT is able to reach similar representations even in a shallower architecture, RoBERTa shows a non-monotonic layer-wise stability trajectory, and ALBERT keeps comparatively compact and stable representations. The cross-domain experiment shows a significant drop in RSS from 5.455 to 2.042 (62.6\%), indicating substantial geometric degradation due to domain shift. The results show that the quantitative characterization of transformer representation geometry using RSS is model agnostic and interpretable, and can be used to gain insights beyond the standard accuracy-based evaluation.