In this chapter, an online self-organizing fuzzy modified least square network (SOFMLS) is proposed. The network generates a new rule, if the smallest distance between the new data and all the existing rules (the winner rule) is more than a prespecified radius. The major contributions of this chapter are as follows: (1) A new network is proposed. In this network, unidimensional membership functions are used, and only two parameters for each rule are employed, thus reducing the number of parameters. The network avoids the singularity produced by the widths in the antecedent part for online learning. (2) A new pruning algorithm based on the density is proposed, where the density is the number of times that each rule is used in the algorithm. The rule that has the smallest density (the looser rule) in a selected number of iterations is pruned if the value of its density is smaller than a prespecified threshold. (3) The stability of the proposed algorithm is proven, and the bound for the average of the identification error is found. The condition that led the algorithm to avoid the local minimum is found, and it is proven that the parameters error is bounded by the initial parameters error. Three simulations give the effectiveness of the suggested algorithm.