Relational tables on the web hold a vast amount of knowledge, and it is critical for machine learning models to capture the semantics of these tables such that the models can achieve good performance on table interpretation tasks, such as entity linking, column type annotation and relation extraction. However, it is very challenging for ML models to process a large amount of tables and/or retrieve inter-table context information from the tables. Instead, existing works usually rely on heavily engineered features, user-defined rules or pre-training corpus. In this work, we propose a unified Retrieval-Augmented Framework for tabular interpretation with Large language model ( \(\textsf{RAFL}\) ), a novel 2-step framework for addressing the table interpretation task. \(\textsf{RAFL}\) first adopts a graph-enhanced model to obtain the inter-table context information by retrieving schema-similar and topic-relevant tables from a large range of corpus; \(\textsf{RAFL}\) then conducts tabular interpretation learning by combining a light-weighted pre-ranking model with a re-ranking-based large language model. We verify the effectiveness of \(\textsf{RAFL}\) through extensive evaluations on 3 tabular interpretation tasks (including entity linking, column type annotation and relation extraction), where \(\textsf{RAFL}\) substantially outperforms existing methods on all tasks.