Fine-tuning ClimateBert transformer with ClimaText for the disclosure analysis of climate-related issues in corporates’ financial and non-financial reports
摘要
There is a growing demand from financial agents (especially particular and institutional investors), for companies to report on climate-related financial risks. A vast amount of information, in text format, is expected to be disclosed in the short term by firms in order to identify these risks in their financial and non-financial reports, particularly in response to the growing regulation that is being passed on the matter. This paper applies state-of-the-art NLP (Natural Language Processing) techniques for the detection of climate change in text corpora. We use transfer learning to fine-tune two transformer models, BERT and ClimateBert -a recently published DistillRoBERTa-based model specifically tailored for climate text classification-. We fine-tune both models on the novel “ClimaText” database, consisting of data collected from Wikipedia, 10K Files Reports, and web-based claims. We find empirical evidence that our text classification model obtained from the ClimateBert fine-tuning process on ClimaText, outperforms the models created with BERT and the current state-of-the-art transformer in this particular problem, achieving better results in accuracy, precision, F1, and specificity parameters. Furthermore, to achieve an even better model, we apply Bayesian optimization to automatically select hyperparameters that maximize validation accuracy. Our study is the first one to implement the recently published ClimateBert algorithm on the ClimaText database. Based on our results, it can be said that ClimateBert fine-tuned on ClimaText is an outstanding tool within the NLP pre-trained transformer models that may and should be used by investors, institutional agents, and companies themselves to monitor the disclosure of climate risk in financial reports, as well as in other textual sources (policies or new legislation passed on this matter. Furthermore, our transfer learning methodology is cheap in computational terms, thus allowing any organization to perform it. Concretely, these stakeholders can use our proposed model by embedding it in a software application that can load all the financial texts of a plethora of companies and generate a report with the percentage of texts belonging to climate change of every company in every year of activity and data visualizations automatically, dramatically speeding up the process of visualizing the actual climate change reporting of a company.