Social Education in Japan is non-formal education that refers to Community and Adult Education. Influenced by the European Enlightenment, social education theory was first raised in the late nineteenth century, and was addressed as an educational policy in the twentieth century. Later, during the educational reforms after World War II, the Social Education Act was enacted in 1949, and it has served as an educational system to this day. The Social Education Act is based on the principle of “freedom of social education” in a democratic nation, and defines the role of the national and local governments as “fostering an environment in which people can enhance their own cultural education that is relevant to their actual lives.” The “freedom of social education” reflects the philosophy of Sakuo Teranaka, an educational bureaucrat who played a central role in establishing social education in postwar Japan. In order for citizens to better their lives and learn freely about the various issues they face in their everyday lives, the government is mandated to provide a learning environment that is positioned by the educational administration as public education, while limiting the intervention of governmental authorities. Today, however, social education administration is regressing and weakening. Although the external context is that of structural reform of administrative and financial systems, there have also been cases where academic freedom has been violated due to pressure from state power against learning that handles content contrary to the agendas of the administration and government at the time. This paper suggests that there are two intrinsic factors behind the regression of social education administration: “atrophy of social education administration” and “disempowerment of social education administration,” and calls for a renewed appreciation of the principles of “freedom of social education,” and argues that social education that encourages free learning and discussion is important precisely to build the future.