3D Gaussian Splatting (3DGS) has recently emerged as a promising technique for synthesizing novel photorealistic views in real time. Nevertheless, it heavily relies on numerous discrete and interdependent Gaussians, resulting in blurred regional edges and ambiguous floaters. To address these limitations, we introduce an innovative two-stage methodology called SGAGS that sequentially constructs the semantic and appearance fields by leveraging regional features embedded in the semantics. We augment each Gaussian with an identity feature and adaptive properties, allowing instance-based classification of the Gaussians while adjusting their basis function frequency to more accurately fit the corresponding regions. Instead of using expensive 3D labels, we first leverage the 2D masks predicted by the SAM-based method to supervise identity splatting along with frequency adaptation and consistency regularization. Building on established properties, we then partly inherit these characteristics to utilize hyper-ellipsoid boundaries and regularization to filter and guide floaters during appearance splatting. Experiments on several scene datasets validate that our approach outperforms existing methods, delivering finer details and higher rendering quality.