Since the 1980s, but particularly after the 9/11 attacks, Islamic organizations in Europe, often in response to anti Islam activities of radical right-wing groups, have begun to reinforce the communities’ Muslim identity, and also government institutions started to cater more and more to religious demands of these communities. However, many secular Muslims in countries such as the Netherlands, Belgium, Germany or France do not identify with religious labels. Most of them can be located around left-wing political parties, where they defend equal rights for women, LGBT people, and non-believers. As a result, they frequently face criticism from within their own cultural communities. But their secular principles have also brought them into conflict with sectors of the political left who accuse them of ‘giving arguments to right-wing extremists’. By silencing these secular voices, part of the left is effectively furthering religious conservatism within Islamic communities in Europe. This chapter analyses the contested position of secular Muslims who attempt to combat tendencies of growing religiosity within their communities. The research is based on a dozen of in-depth interviews with secular Muslims in the Netherlands. The theoretical framework for the analysis of their strategies is an updated version of Hirschman’s model of Exit, Voice and Loyalty (Hirschman 1970). I argue that integration policies should focus more on secular members of Muslim communities instead of on religious interlocutors.