Graph convolutional networks (GCNs) provide a promising yet effective way to process 3D point clouds. However, two critical issues have significantly influenced the learning of robust GCNs, and thus limited the applications of most existing approaches for point cloud analysis: 1) local shapes are always sensitive to noisy points due to the cluttered background and missing cloud point parts, and 2) local geometric structure cannot be well preserved during the typical down-sampling procedure. We present a novel adaptive edge dual-graph convolutional networks (AE-DualGCN) to deal with these issues, which includes two key components: 1) adaptive-gate edge convolution on dual graph, which dynamically adjusts the edge features of the feature graph and position graph to preserve the topology of local shape, and 2) local attention graph sampling, which performs down-sampling according to local geometric structure. Our proposed AE-DualGCN can pass on the most important points with intrinsic 3D structure to the next layers, and at the same time, make the down-sampling of GCN in an end-to-end fashion. We demonstrate the robust of AE-DualGCN for different scenarios in point cloud classification and segmentation on several famous point cloud datasets.