The paper addresses a problem of packing jammed spheres in a cylindrical layer motivated by the need of increasing the heat transfer efficiency systems based on the heat exchangers made of modified thermoplastic materials. Corresponding nonlinear programming model is formulated, using the phi-function technique. A solution approach is proposed that involves three main stages. Firstly, the cylindrical layer is partitioned into equal sectors. Then feasible arrangements are generated in the sectors based on the homothetic transformations of flexible sphere radii. Finally, an original multistep decomposition procedure is applied. The latest is replaced the large-scale packing problem by a sequence of considerably smaller nonlinear programming problems. A local optimal solution found at the last iteration of the procedure we refer to a jammed packing of spheres. Several examples illustrating the work of our algorithm are provided.