Encrypted traffic classification at packet granularity requires both real-time inference and minimal resource overhead. Traditional deep learning methods typically rely on flow-level data and large models, leading to high preprocessing latency and prohibiting deployment on edge devices. DE-MFLA addresses these challenges with a dual-branch embedding design that separates header and payload byte sequences, fusing them through multi-head focused linear attention to capture global context, and multi-scale one-dimensional convolutions to enrich local byte relationships. A lightweight classifier built on this representation totals only 4.6 million parameters, enabling sub-millisecond packet-level predictions. Experimental results demonstrate that DE-MFLA outperforms existing lightweight models in both accuracy and operational efficiency on two benchmark datasets.