The ability to train deep architectures has led to many developments in parametric, non-linear dimensionality reduction but with little attention given to algorithms based on convolutional feature extraction without backpropagation training. This paper aims to fill this gap in the context of supervised Mahalanobis metric learning. Modifying two existing approaches to model latent space similarities with a Students t-distribution, we obtain competitive classification perfor- mance on CIFAR-10 and STL-10 with k-NN in a 50-dimensional space compared with a linear SVM with significantly more features. Using simple modifications to existing feature extraction pipelines, we obtain an error of 0.40% on MNIST, the best reported result without appending distortions for training.