<p>Decision trees are powerful predictive models widely used in machine learning. However, simple trees like CART (Classification and Regression Trees) struggle to capture linear relationships. Linear model trees address this by fitting linear models in the nodes, combining tree structure with linear regression for better interpretability. Recently, a new approach called PILOT (Piecewise Linear Organic Tree) was introduced, offering the speed of traditional decision trees with the flexibility of linear model trees. This paper proposes a weighted extension of the PILOT tree algorithm, allowing its full potential to be exploited in a broader range of applications. During training, each observation can be assigned a weight that reflects its importance in the loss function. Crucially, incorporating these weights does not change the computational complexity of the PILOT algorithm. Three applications demonstrate the benefits of weighted PILOT trees. First, they are used in a one-step boosting model that fits two complementary PILOT trees. This approach strikes a balance between the simplicity of a single PILOT tree and the performance of an entire (random) forest of PILOT trees. Second, weighted PILOT is applied to imbalanced regression tasks. By increasing the weights of underrepresented points, prediction accuracy improves in critical regions. Finally, in the third application, we apply weighted PILOT to the covariance shift problem. Applying weights effectively mitigates covariate shift, leading to improved performance compared to the unweighted version.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Fast weighted linear model trees

  • Flor Debois,
  • Jakob Raymaekers,
  • Thomas Servotte,
  • Tim Verdonck

摘要

Decision trees are powerful predictive models widely used in machine learning. However, simple trees like CART (Classification and Regression Trees) struggle to capture linear relationships. Linear model trees address this by fitting linear models in the nodes, combining tree structure with linear regression for better interpretability. Recently, a new approach called PILOT (Piecewise Linear Organic Tree) was introduced, offering the speed of traditional decision trees with the flexibility of linear model trees. This paper proposes a weighted extension of the PILOT tree algorithm, allowing its full potential to be exploited in a broader range of applications. During training, each observation can be assigned a weight that reflects its importance in the loss function. Crucially, incorporating these weights does not change the computational complexity of the PILOT algorithm. Three applications demonstrate the benefits of weighted PILOT trees. First, they are used in a one-step boosting model that fits two complementary PILOT trees. This approach strikes a balance between the simplicity of a single PILOT tree and the performance of an entire (random) forest of PILOT trees. Second, weighted PILOT is applied to imbalanced regression tasks. By increasing the weights of underrepresented points, prediction accuracy improves in critical regions. Finally, in the third application, we apply weighted PILOT to the covariance shift problem. Applying weights effectively mitigates covariate shift, leading to improved performance compared to the unweighted version.