In the field of medicine, image segmentation is highly important in automatic disease diagnosis from medical image and subsequent treatment. However, in deployment environments with limited computing and storage resources, it is necessary to balance the accuracy and complexity of the model. This paper presents a lightweight medical image segmentation network, dubbed ELU-Net, which includes two effective modules: Attention Edge Aware Module (AEAM) and Depthwise-Pointwise Convolution Group (DPG). AEAM combines low-level and high-level features with an attention mechanism to enhance and extract edge features in the image. This mechanism automatically focuses on the regions with significant boundary information, and can correctly segment the regions with ambiguous boundaries with edge detection. DPG uses a combination of depthwise and pointwise convolutions with different kernel sizes in each layer, remarkably reducing the number of model parameters while expanding the receptive field. With only 39KB of parameter amount, ELU-Net was tested on a self-built OCT dataset and the public ISIC 2018 dataset. Compared with the baseline, the mIoU scores increase by 0.1205 and 0.0321, respectively.