Large Language Models (LLMs) have advanced tremendously in the last few years. Large-scale models such as GPT4, GEMINI, LLaMa, and Claude have opened new frontiers for what is possible for generative models. With the widespread utilization of these models for a plethora of applications, challenges associated with trust, safety, and fairness need to be addressed. One of these challenges for LLMs relates to their capability for causal reasoning. LLMs’ capability to generate reasoning based on causal knowledge and not simple correlation has been discussed in the literature. This can be critical for successfully utilizing LLMs in sensitive applications such as biomedical, healthcare, technology, law, and government. To address this, we investigate LLMs for whether they can identify cause and effect relations using a combination of benchmarked causal datasets (Tuebingen dataset), image datasets (Animals with Attributes 2, IITH-CANDLE), and LLM benchmark dataset (CRASS). We investigate the causal reasoning capabilities of LLMs on causal discovery and causal inference. For causal discovery, we present LLMs’ ability to identify causal relations and generate causal graphs given an observational dataset. For causal inference, we investigate whether they can generate counterfactual reasoning on natural language questions. Using multi-modal data, experimental results demonstrate the capability of LLMs to complement and contribute to the growing field of causal reasoning for AI systems by aiding in causal discovery and treatment effect estimation methods based on traditional techniques. This will help in ensuring the trust, safety, fairness, and robustness of LLMs towards responsible and ethical AI. However, we also highlight the limitations of LLMs to generate causal reasoning as the data complexity is increased. While they are able to generate causal relations between simple two variable datasets, they are limited when using robust multi-variable datasets that mimic real world data.