Accurate knowledge of chronic condition status is essential for optimising healthcare policy and population outcomes. However, administrative claims data often lacks comprehensive diagnostic information, leading to data and label insufficiency problems. Recent advancements in transfer learning with Transformers have demonstrated that model pretraining can capture general patterns in large datasets, and be effectively applied to scenarios with limited labelled data. Although this approach has been explored in Electronic Health Record data, its application to claims data remains understudied. In this work, we propose the Claimsformer, a pretrained Transformer model specifically designed for administrative claims data, using information from Australian medical services and prescriptions. We explore various pretraining strategies to identify the optimal approach for the Claimsformer, and validate its effectiveness in predicting several chronic conditions for two cohorts: cancer and mental health. This work demonstrates, for the first time, the advantages of pretraining from Australian claims data, and highlights the opportunity to further leverage claims relations for improving health-related predictions.