Reducing the high dimensionality of the original feature space through the use of feature selection algorithms is crucial in gene-expression-based predictive tasks to potentially improve performance and provide a better understanding of each feature’s power and biological meaning. Feature selection approaches like LASSO and other embedded techniques select small subsets of relevant features based solely on their quantitative contribution and predictive power, often leading to the selection of features with limited biological relevance. This work aims to provide a wide exploratory analysis of LASSO feature selection to assess the effects of different hyper-parameters on the selection of the most relevant features and their corresponding biological significance. Then, it introduces a new approach that can guide LASSO in the selection of the features by considering their predictive power as well as their biological relevance. With this intention, this work proposes a novel Gene Information Score to estimate each gene’s biological relevance and shows its use in enhancing the feature selection.