For some reason the larger bucket sizes are causing a large perf hit
in spectral_norm.  It's unclear exactly why this is happening, but
theories are legion.  More investigation is warranted, but this gets us
back from the perf regression.

Also hoist the atom_idx calculation out of a couple of loops that were
iterating over object indices.