A comprehensive, end-to-end platform for acquiring data, cleaning, backtesting, generating, and analyzing forecasts for Forex and commodity markets using Markov Chains and a Random Forest confidence validator.
This project and all the code contained within are provided for educational and research purposes only. The forecasts generated by this system are based on historical statistical patterns and do not constitute financial advice or a recommendation to buy or sell any asset.
Financial markets are subject to high levels of risk and volatility. Past performance is not indicative of future results. The author and contributors of this project assume no liability for any financial losses or damages incurred as a result of using, modifying, or relying on this software, its data, or its forecasts. Trade at your own risk.
This repository contains an integrated platform designed to explore the application of Markov Chains for predicting directional movements in financial markets. It's more than a single model; it's a complete workflow that covers the entire data science lifecycle, from raw data acquisition to advanced performance analytics.
The core hypothesis is to test whether historical price action sequences can provide a statistical edge in forecasting the next period's direction (UP or DOWN). The system employs a two-layer approach:
- Primary Forecast: A configurable Markov Chain model generates the initial directional forecast based on historical transition probabilities.
- Confidence Validation: A Random Forest classifier, trained on the historical performance of the primary model, acts as a "validator" to score the confidence of each new forecast, providing a deeper layer of analysis.
Note: This project is presented as a completed, "frozen" piece of work. The data is not updated, and the goal for users is to run the pipeline to understand the methodology and reproduce the results.
The platform is organized into a series of modular scripts, designed to be run in a logical sequence. Each script represents a key stage in the pipeline.
1_Data_Acquisition_Pipeline.py: Fetches the latest daily financial data from the Alpha Vantage API for a predefined list of assets and maintains local CSV files.2_Data_Cleaning_Pipeline.py: A robust tool that reads raw data files, performs validation checks (duplicates, data types, logical errors), and saves a standardized, analysis-ready CSV file.3_Markov_Order_Optimizer.py: A backtesting script to perform a systematic search for the optimal Markov Order for a given asset over a defined historical period.4_Feature_Enrichment_Engine.py: Enriches the clean data by adding historical Markov-based features in a "leak-proof" manner, preparing the dataset for the confidence validator.5_Confidence_Validator_Optimizer.py: A hyperparameter tuning script that finds the optimaln_estimatorsfor the Random Forest validator model to maximize its predictive accuracy.6_Historical_Forecast_Generator.py: A powerful tool to generate (or regenerate) forecasts for a specific historical period, complete with validator scores.7_Live_Forecasting_Engine.py: The main operational engine that generates the next day's forecast for the entire configured asset portfolio.8_Unified_Performance_Analyzer.py: The final reporting tool that analyzes forecast logs and generates comprehensive, multi-dimensional performance reports in HTML and JSON formats.
- Automated Data Pipelines: From API fetching to cleaning and enrichment, the data workflow is streamlined and automated.
- Multi-Asset Support: Easily configurable to handle Forex, Commodities, and Crypto assets.
- Two-Layer Forecasting System: Combines a primary Markov model with a secondary Machine Learning validator for enhanced signal analysis.
- Advanced Performance Analytics: Generates detailed reports including directional accuracy, day-of-week performance, and intraday streak analysis.
- Dynamic & Flexible: Most tools are configuration-driven, allowing for easy experimentation and analysis of different assets and timeframes.
This platform was developed using an AI-Assisted Development approach, with the Gemini Pro model acting as a collaborator in the generation of Python code and the structuring of the data science workflow. My role was that of a Lead Data Scientist and Systems Architect, responsible for designing the end-to-end pipeline and validating its components.
My contributions were focused on the following high-level tasks:
- Concept and Requirements Definition: Defining the requirements for an integrated data science platform to explore the application of Markov Chains in financial forecasting. This included designing the novel two-layer system that combines a primary Markov model with a secondary Random Forest confidence validator.
- Architectural Design: Structuring the project as a Sequential Pipeline composed of eight distinct Python scripts. Each script was designed to represent a key stage in the data science lifecycle, from data acquisition and cleaning to model optimization, forecast generation, and comprehensive performance analysis.
- Prompt Engineering: Crafting detailed prompts for the AI to build each script in the pipeline, with a specific focus on the correct implementation of statistical algorithms (Markov Chains), machine learning models (Random Forest), and the development of robust modules for model optimization and performance reporting.
- Review, Testing, and Validation: Implementing systematic backtesting through dedicated optimizer scripts (
Markov_Order_Optimizer,Confidence_Validator_Optimizer) to find the best-performing model parameters. The integrity of the entire pipeline was validated by running it to generate and analyze historical forecast logs. - Full Documentation: Writing clear documentation in the
README.mdthat explains the research hypothesis, the two-layer forecasting system, the function of each of the eight scripts in the pipeline, and provides complete instructions for setup and execution.
This project serves as a comprehensive example of how AI can be guided to build a multi-stage, analytical platform, bridging the gap from raw data to actionable, model-driven insights.
- Clone the repository:
git clone [https://github.com/your-username/your-repository-name.git](https://github.com/your-username/your-repository-name.git)
- Install dependencies:
pip install pandas numpy requests scikit-learn matplotlib
- Configure API Key: Add your personal Alpha Vantage API key in
src/1_Data_Acquisition_Pipeline.py. - Run the Pipeline: Execute the scripts in their numerical order from within the
src/directory to experience the full workflow.
- Language: Python 3.11+
- Core Libraries: Pandas, NumPy
- Machine Learning: Scikit-learn
- Data Fetching: Requests
- Visualization: Matplotlib
Computer Engineering Student (4th Year) | Machine Learning | Deep Learning | Artificial Intelligence | Algorithmic Trading | Financial Markets.
This project is open-source and available under the MIT License.