The SARS-CoV-2 outbreak prompted significant organizational adjustments in workplaces, necessitating the restructuring of shift schedules and guidelines to mitigate the risk of COVID-19 transmission. Workplace modifications encompassed ensuring physical distancing, restricting contact beyond the workplace, embracing remote work, and altering work conditions (e.g., smaller employee teams, reduced shift groups, and extended working hours). These changes primarily aimed to minimize employee interactions, particularly affecting shift workers who already experienced limited socialization due to their fluctuating schedules. The challenge of building a social life for shift workers was exacerbated as societal activities typically occurred after the standard 9–5 workday. The availability and quality of social networks and social engagement, in addition to stress, are indicated as potential mediators of disease in shift workers. This study analyzes longitudinal survey data from 631 garment factory workers in Poland from 2019 to 2021. Findings reveal that shift workers encountered heightened difficulties in general health, mental well-being, and social relationships during the pandemic (2020–2021) compared to other employees. Recognizing the adverse social implications for shift workers in extraordinary circumstances like the COVID-19 pandemic emphasizes the need for proactive employer measures to address these concerns.