Accelerated edge computing devices allowing user data to be processed at network periphery have seen exponential growth globally in recent years. Thus, deploying deep learning models on accelerated devices has emerged as an inevitable trend. However, one of the main obstacles is addressing the problem of optimizing models’ utilization on resource-constrained accelerated hardware while guaranteeing inference accuracy and automating the entire process of a model’s optimization and deployment. To tackle the problem, we propose an automated deep learning framework for optimizing the deployment of deep learning models on accelerated devices. Our proposed framework is a self-contained design flow consisting of model training from scratch, model compression techniques such as structural pruning, integer-only quantization, and firmware generation for a hardware accelerator using a hardware-specific compiler. Evaluations show that the proposed framework enhances the models’ inference on an accelerated device by up to 40 times faster than the complete processor-based inference, while it ensures an accuracy tolerance of 3% compared to the corresponding full precision model.