Estimating object poses is a fundamental problem in computer vision in general as well as for robotic manipulation in particular. Most approaches require a known 3D model of the object. One step towards a more general formulation is to estimate the object’s width, height and depth with the pose, e. g. consider a generic box, cylinder or plate instead of one with known dimensions. This paper investigates the last stage of such a pipeline, namely least-squares estimating pose and scales from point correspondences aggregated into a fixed size matrix. Therefore it encapsulates the scaled \(SO(3)\) manifold in a so-called \(\boxplus \) -operator and derives a Gauss-Newton based optimizer with initial guess on that. We find that the resulting estimator is strongly biased towards small scales. This is due to the structure of the least-squares loss: Noise in recognized object points is multiplied with the to be estimated transformation matrix, violating the additive noise assumption. It has no effect in the prevalent use of this loss for pose estimation but affects the scale. We propose a solution to this bias based on an approximation of total least-squares that preserves the advantage of a fixed size representation and show that it provides relatively consistent uncertainty estimates.