Traditional methods of optimizing investment portfolios often rely on rigid models that are not able to fully capture the complexity and unpredictability of financial markets. Previous research has primarily centered around statistical analysis, which struggles to capture the complex interdependencies between data points that ML-based models may be able to. In this paper, we have constructed a hybrid model incorporating machine learning and statistical methods to propose a dynamic and data-driven approach to optimize the portfolio using a Long Short-Term Memory (LSTM) network and a covariance matrix based on the Gerber statistic. It estimates the return prices and consequently constructs a Gerber estimator covariance matrix. This yields two-fold advantages, added robustness and improved risk management. Our study utilizes S&P500 tech stocks to evaluate the model and constructs 3 different portfolios for asset allocation, i.e., Minimum CVaR, Mean Risk, and Hierarchical Risk Parity. The annualized Sharpe Ratio for HRP achieved is 1.75, and the metrics show effective risk management. The model is unique due to its hybrid nature which combines the data-centricity and the flexibility of LSTM to capture any non-linear patterns which is capable of adjusting to evolving market conditions. By leveraging the statistical rigor of the covariance matrix to yield promising results, the model can gauge the stochastic nature of financial markets to a reasonable extent, thereby helping to maximize returns and minimize risks.