Most digital cameras use color filter array (CFA) coupled sensors to capture color images. An image signal processor (ISP) pipeline is used to convert CFA captures to visually pleasing red-green-blue (RGB) images. These images are commonly used for downstream computer vision (CV) and machine learning (ML) tasks. Despite their widespread use, the impact of different components of the ISP pipeline on ML and CV tasks remains largely unexplored. We study the impact of demosaicing (color reconstruction) and processing of images by comparing it with neural network-based reconstruction of RGB images from the CFA signal. Since noise reduction is part of the motivation for the ISP pipeline, we repeat our experiments with added noise. We find that while demosaicing has little impact, operators such as median filtering, white balance and white level adjustment significantly reduce reconstruction quality. The ML reconstruction approach that bypasses the ISP pipeline performs better, even in the presence of noise. This motivates using the raw CFA data instead of processed RGB for ML and CV tasks.