With the widespread use of third-party code in software projects, Software Composition Analysis (SCA) tools emerged in order to help developers and security specialists automate the process of vulnerability detection within dependencies. Among SCA tools, the most common dependency detection techniques are metadata-based. However, there has not been a comprehensive evaluation of metadata-reliant SCA tools in regard to their resilience against metadata manipulations. To bridge this gap, we conducted a thorough evaluation of 5 state-of-the-art metadata-reliant SCA tools across 11 attack scenarios, each crafted to demonstrate a particular manifest feature, bundling, or dependency modification. Our findings reveal a concerning lack of resilience against metadata manipulations among these tools, with subtle modifications easily influencing their detection capabilities. Our findings not only uncover the limitations of existing metadata-based approaches but also offer valuable insights for SCA tool researchers, developers, and users.