Feature/forecasting task

[ivan-salas]

Add Native Time Series Forecasting Support

Summary

Adds native time series forecasting as a first-class task type in DashAI. This includes a new ForecastingTask, four model families (Prophet, ARIMA, SARIMAX, SklearnMultiStep), a dedicated ForecastingJob with causal splitting, three explainers (decomposition, feature importance, uncertainty), forecasting-specific metrics (MAPE, sMAPE), automatic temporal column detection on CSV upload, and all required frontend modals and panels to support the full workflow.

---

Type of Change

• Backend change
• Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

---

Changes (by file)

Backend — Task & Models

• DashAI/back/tasks/forecasting_task.py: new ForecastingTask class with timestamp/target/exogenous column validation and temporal metadata exposure.
• DashAI/back/models/forecasting/base_forecasting_model.py: abstract base class defining fit, predict, exogenous helpers, and frequency utilities.
• DashAI/back/models/forecasting/prophet_model.py: Prophet-based model with seasonality, holidays, and component decomposition.
• DashAI/back/models/forecasting/statsmodels_arima_model.py: ARIMA model with configurable (p, d, q) and trend options.
• DashAI/back/models/forecasting/statsmodels_sarimax_model.py: SARIMAX model extending ARIMA with seasonal (P, D, Q, m) components.
• DashAI/back/models/forecasting/sklearn_multistep_forecaster.py: scikit-learn-based model with lag features and direct/recursive strategies.

Backend — Job & Factory

• DashAI/back/jobs/forecasting_job.py: new job class with temporal splitting, fit orchestration, and Huey integration.
• DashAI/back/job_queues/model_factory.py: updated to handle forecasting evaluation edge cases (small splits, NaN/Inf metric sanitization).

Backend — Explainers

• DashAI/back/explainability/forecasting/base_forecasting_explainer.py: shared base for all forecasting explainers.
• DashAI/back/explainability/forecasting/forecast_decomposition.py: STL trend/seasonal/residual decomposition.
• DashAI/back/explainability/forecasting/forecast_feature_importance.py: permutation-based exogenous variable importance.
• DashAI/back/explainability/forecasting/forecast_uncertainty.py: prediction interval and confidence estimation.
• DashAI/back/routers/explainers_router.py: added name auto-deduplication logic.

Backend — Predict

• DashAI/back/jobs/predict_job.py: added forecasting-specific validation (_validate_forecasting_dataset), auto-generate mode, temporal metadata propagation, and sanitize_for_json() utility.

Backend — Dataset / Converters

• DashAI/back/routers/datasets_router.py: new GET /datasets/{id}/temporal-info endpoint.
• DashAI/back/converters/time_series_window_converter.py: supervised lag-feature transformation.
• DashAI/back/converters/extend_time_series_converter.py: future-row appender for out-of-sample prediction.

Backend — Metrics

• DashAI/back/metrics/mape.py: MAPE metric with zero-value guard.
• DashAI/back/metrics/smape.py: sMAPE metric, bounded 0–200%.

Backend — Registration

• DashAI/back/initial_components.py: registered all new task, models, converters, metrics, explainers, and job.
• requirements.txt: added prophet and statsmodels.

Frontend

• DashAI/front/src/components/splits/SplitDatasetTemporal.tsx: temporal split panel with train/val/test sliders and gap parameter.
• DashAI/front/src/components/predictions/ForecastingOptions.tsx: prediction modal panel for auto-generate vs. dataset mode, with frequency validation.
• DashAI/front/src/components/explainers/NewGlobalExplainerModal.tsx: extended to fetch and display temporal metadata for forecasting explainers.
• DashAI/front/src/components/explainers/ForecastingExplainerInfo.tsx: new component showing model, horizon, and frequency context.
• DashAI/front/src/components/runs/RunnerDialog.tsx: passes task_name when enqueuing jobs.

Tests

• DashAI/tests/back/models/test_forecasting_models.py: 473-line suite covering all four model families.
• DashAI/tests/back/tasks/test_forecasting_task.py: task validation tests.
• DashAI/tests/back/jobs/test_forecasting_job.py: job splitting and training tests.

---

Testing (optional)

• Run pytest DashAI/tests/back/models/test_forecasting_models.py to verify all model families.
• Upload a daily CSV (e.g., date,value) to a ForecastingTask experiment and confirm automatic frequency detection.
• Train a ProphetModel, then run a prediction using auto-generate mode (e.g., 30 future periods) and verify the output in the UI.
• Add a ForecastDecomposition explainer to a trained run and confirm trend/seasonal/residual components are displayed.

---

Notes (optional)

• prophet and statsmodels are new dependencies; ensure they are installed before running forecasting experiments.
• All i18n strings are available in both English and Spanish.
• For a detailed description of each component (task contract, model internals, explainer algorithms, metric formulas, frontend UX), see sections 1–10 below.

💡 Reviewer Guide: Key Architectural Adaptations for Forecasting (Click to expand)

Forecasting introduces specific domain requirements that differ from classical ML (Regression/Classification). Here is a quick guide to the 6 key architectural adaptations implemented in this PR to integrate Forecasting seamlessly into the DashAI ecosystem:

1. Temporal Splitting — Strict Chronological Order

• Key files: dashai_dataset.py
• Design Decision: Unlike classification where random shuffling is standard, forecasting requires preserving causality. Shuffling is bypassed for this task in favor of a strict chronological split (Train -> Val -> Test) with an optional gap parameter to prevent data leakage.

2. Frequency Detection and Metadata Propagation

• Key files: forecasting_task.py, predict_job.py
• Design Decision: The timestamp column requires special handling. Its metadata (frequency, start/end dates) is automatically detected during task creation and propagates through the model's lifecycle to enable accurate future timestamp generation.

3. Prediction Logic — Auto-generate vs. Exogenous Dataset

• Key files: predict_job.py
• Design Decision: The prediction flow was expanded to support two distinct modes: auto-generating future timestamps for a given horizon (forecast_periods), or validating a user-provided CSV containing future values for exogenous variables.

4. Richer Interface — BaseForecastingModel

• Key file: base_forecasting_model.py
• Design Decision: Extends the standard BaseModel interface. Forecasting models utilize a richer interface to store temporal metadata (frequency, last observed date) so that the downstream infrastructure (Explainers, PredictJob) can consume it uniformly.

5. Supervised Transformation — Windowing Converters

• Key files: time_series_window_converter.py
• Design Decision: Implements the Adapter pattern. It transforms time series into supervised learning problems (using lags) to cleanly reuse classical sklearn algorithms for multi-step forecasting without modifying their core logic.

6. Metric Sanitization — JSON Safety

• Key files: mape.py, smape.py
• Design Decision: Time series with zero values can mathematically produce inf or NaN metric results, which invalidate JSON serialization for the frontend. Masks and semantic fallbacks were added to ensure robust API stability.

---

Overview

This PR introduces native time series forecasting capabilities to DashAI, enabling users to build, train, evaluate, and explain forecasting models directly within the platform. The implementation covers the full ML pipeline: data ingestion with automatic temporal column detection, task/job orchestration, four model families, specialized metrics, explainability tools, and a dedicated frontend experience.

---

1. Forecasting Task

A new ForecastingTask class defines the contract for time series problems. It expects one timestamp column (required), one target column (output), and optionally exogenous/regressor columns as additional inputs. The task validates column types at configuration time, provides informative error messages when columns are missing or mistyped, and exposes temporal metadata that flows downstream to training, prediction, and explainability components.

---

2. Forecasting Models

Four model families are available out of the box:

Model	Backend	Notes
ProphetModel	Facebook Prophet	Automatic seasonality, holiday effects, exogenous regressors, component decomposition
StatsmodelsARIMAModel	statsmodels	Configurable (p, d, q) orders, trend options (n/c/t/ct), exogenous support
StatsmodelsSARIMAXModel	statsmodels	Extends ARIMA with seasonal (P, D, Q, m) components
SklearnMultiStepForecaster	scikit-learn	Transforms the series into a supervised problem via lag features; supports linear, ridge, and random_forest base estimators with direct or recursive multi-step strategies

All models support both in-sample (validation) and out-of-sample (future) prediction modes, and all share a unified interface for handling exogenous variables.

---

3. Base Forecasting Model

BaseForecastingModel is the abstract base class that all forecasting models extend. It defines:

• fit(X, y, temporal_metadata) — trains the model while storing timestamp column, target column, and frequency info.
• predict(X) — dispatches to in-sample or out-of-sample logic depending on whether future timestamps are present.
• get_exogenous_columns() / has_exogenous_variables() — unified exogenous variable management across implementations.
• Seasonal period inference utilities (e.g., mapping frequency codes like D, W, M to integer periods).

This design ensures that any new forecasting model added to DashAI only needs to implement the core logic while inheriting consistent behaviour for metadata management, column tracking, and prediction dispatch.

---

4. Forecasting Job

ForecastingJob orchestrates the full training lifecycle for forecasting experiments:

• Retrieves the dataset and applies temporal (causal) splitting — no shuffling, strictly train → validation → test in chronological order, with an optional gap between splits to prevent leakage.
• Prepares the data (timestamp parsing, dtype coercion) and passes temporal_metadata to model.fit().
• Manages job status (queued → running → finished/failed) and persists results to the database.
• Integrated with the Huey task queue, consistent with other DashAI job types.

The model factory was also updated to handle forecasting-specific edge cases during evaluation: small splits that fail prediction are gracefully skipped, NaN/Inf metric values are sanitized to null before JSON serialization, and per-metric try/except blocks prevent a single failing metric from aborting the entire evaluation.

---

5. Explainers

Three new forecasting explainers provide interpretability at different levels:

ForecastDecomposition — breaks down any forecast into trend, seasonal, and residual components using STL (Seasonal-Trend decomposition via Loess). Works across all four model families. Configurable horizon (1–365 periods) and an include_historical flag.

ForecastFeatureImportance — permutation-based importance for exogenous variables. Measures the degradation in forecast quality (MAE, RMSE, or MAPE) when each external regressor is randomly shuffled. Useful for understanding which external signals actually help the model. Configurable n_repeats (1–50).

ForecastUncertainty — quantifies prediction intervals and confidence around each forecast point, helping users assess how reliable a forecast is over the horizon.

All three share a ForecastingGlobalExplainer base class that handles timestamp detection, frequency inference, and dataset preparation, reducing duplication across implementations.

Additionally, the explainers API endpoint now auto-deduplicates names (e.g., "Explainer" → "Explainer_1") to avoid conflicts.

---

6. Predict — Forecasting Particularities

Forecasting prediction is fundamentally different from classification/regression, so it required dedicated handling in the predict job:

• Two prediction modes:

  • Auto-generate: the user specifies a number of future periods (forecast_periods); DashAI automatically generates the future timestamp sequence using the detected frequency, so no prediction dataset is needed.
  • Dataset: the user uploads or selects an existing dataset containing future timestamps (and optionally exogenous values for those periods).

• Validation (_validate_forecasting_dataset): when a dataset is provided, the job validates it thoroughly — strict chronological ordering, no duplicate timestamps, consistent frequency, no backcasting (no predictions before the training window ends), and presence of all required exogenous columns without NaN values.

• Temporal metadata propagation: the training run's temporal config (timestamp column, frequency, training date range) is passed to the prediction job so it can validate compatibility without re-reading the training data.

• A new sanitize_for_json() utility converts any NaN/Infinity float values in prediction outputs to null before returning them via the API.

---

7. CSV Upload with Automatic Temporal Column Detection

When a user uploads a CSV for a ForecastingTask, DashAI automatically scans column types to detect the timestamp column. The detection logic:

1. Looks for a column named ds first (Prophet convention).
2. Falls back to any column with a datetime-parseable type.
3. Infers the temporal frequency from the column values (minute / hour / day / week / month / annual), detecting gaps or irregularities.

A new API endpoint GET /datasets/{dataset_id}/temporal-info?timestamp_column={col} returns the detected frequency_code, frequency_label, frequency_description, start_date, end_date, total_periods, and any detected_gaps. The frontend uses this to populate the split configuration and to validate that a prediction dataset matches the training frequency before allowing the user to proceed.

Two new converters support the data transformation pipeline:

• TimeSeriesWindowConverter: transforms the series into a supervised format with configurable lag features (lag_1 … lag_w) and a shifted target (y_target_h).
• ExtendTimeSeriesConverter: appends n_steps future rows with inferred timestamps and NaN for unobserved columns, used to create the prediction input for out-of-sample forecasting.

---

8. Forecasting Metrics

MAPE (Mean Absolute Percentage Error)

MAPE = (1/n) * Σ | (y_true - y_pred) / y_true | × 100

Scale-independent and returns a percentage (lower is better). Zero values in y_true are masked with a 1e-8 threshold to avoid division by zero. Compatible with both ForecastingTask and RegressionTask.

sMAPE (Symmetric Mean Absolute Percentage Error)

sMAPE = (2/n) * Σ |y_true - y_pred| / (|y_true| + |y_pred|) × 100

Bounded between 0–200%, more numerically stable than MAPE, and symmetric with respect to over/under-forecasting. Also handles zero values gracefully.

Both metrics are registered in initial_components.py and are selectable from the UI when configuring a forecasting experiment.

---

9. Frontend Changes

Temporal Dataset Splitting (SplitDatasetTemporal)
A new split configuration panel replaces the standard random split for ForecastingTask. It provides train/validation/test percentage sliders, a gap parameter (rows to skip between splits to prevent leakage), floating-point tolerance validation, and auto-scaled minimum sizes based on dataset length. Validation errors are shown inline with clear explanations.

Prediction Modal — Forecasting Options (ForecastingOptions)
When running a prediction for a forecasting model, the prediction modal opens a new ForecastingOptions panel that lets the user choose between the two prediction modes (auto-generate vs. dataset). It displays a summary of the training data's temporal context (frequency, date range, total periods) and — when a dataset is selected — fetches its temporal info and shows a frequency match/mismatch indicator with visual feedback before allowing the user to submit.

New Explainer Modal
The NewGlobalExplainerModal was extended to fetch and display temporal metadata for forecasting models. A new ForecastingExplainerInfo component shows the model name, configured horizon, and frequency context inline within the ConfigureExplainerStep, giving users clarity on what the explainer will operate over.

Runner Dialog
The RunnerDialog now passes task_name when enqueuing jobs, enabling the backend to route to the correct job class (including ForecastingJob).

---

10. Additional Changes

• Dependencies: prophet and statsmodels added to requirements.txt.
• i18n: All new UI strings translated to English and Spanish.
• initial_components.py: All new task, models, converters, metrics, explainers, and job are registered.
• Tests: 473-line test suite covering all four model families (with/without exogenous variables, in-sample/out-of-sample, small dataset auto-adjustment, metrics computation), plus additions to task and dataset split tests.
• Pre-commit / code style: formatting fixes applied across modified files.