Considering the significant growth rate of population in the urban area, public transport has become vital to urban living. It has become unavoidable to promote the culture of using Mobility as a Service (MaaS) among travellers to address climatic challenges, especially the global warming phenomenon. To encourage the use of public transport, It is important to introduce innovative IT solutions to the ecosystem of TSPs (Transport Service Providers) backed by an in-depth impact analysis to meet the expectations and the needs of the TSPs and the travellers. This work introduces an assessment methodology to calculate the “Effectiveness” of the innovative IT solutions for TSPs and travellers through a series of data analysis methods using the Bayesian Network, Regression analysis, and ANOVA tests. The assessment is based on two types of quantitative datasets: operational KPIs (Key Performance Indicators) and USI (User Satisfaction Index) surveys to evaluate how the expectations and needs of travellers with different socio-demographic profiles (by gender, age, income level, and impairments) are met by these IT solutions. This methodology is the foundation for the IP4MaaS Project supported by the Europe’s Rail Joint Undertaking. The paper presents the results of applying this methodology to data collected in six sites (Athens, Barcelona, Liberec, Osijek, Padua, and Warsaw). The presented methodology will be helpful to the IT developers and TSPs for assessing their own IT solutions. The findings will help researchers, policymakers, and others in the transport sector to assess public transport services. This assessment methodology is scalable to other demo sites and datasets.