> ## Documentation Index
> Fetch the complete documentation index at: https://systematica.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Sandbox
> Fine-tuning models at scale.
## Introduction
The `Sandbox` class serves as a flexible development environment for testing and optimizing investment strategies. Its main purposes are:
* **Experimentation & Prototyping**: It allows users to rapidly test new strategies, feature configurations, and signal models in isolation from production systems.
* **Hyperparameter Tuning**: By integrating with [optuna](https://optuna.org/), it supports automated hyperparameter optimization, enabling efficient exploration of parameter spaces for better model performance.
* **Signal Generation & Evaluation**: It computes trading signals and evaluates them using customizable metrics, supporting both single-run and cross-validation workflows.
The `Sandbox` explores the full workflow of the system:
* **Data Loading**: It manages the selection and filtering of feature/model configurations, which define how data is loaded and preprocessed.
* **Model Execution**: It runs models with various parameter sets, leveraging `optuna` for systematic search and optimization.
* **Output Configuration**: It supports flexible metric selection and output formatting, making it easy to compare results across experiments.
* **Logging & Tracking**: It integrates with trackers (e.g., Neptune) and uses structured logging to monitor experiments, making debugging and analysis straightforward.
Overall, `Sandbox` is designed to streamline the research and development cycle for systematic trading strategies, providing a controlled environment to explore, tune, and validate ideas before deployment.
## Model Processes
The `Sandbox` integrates a variety of model classes—such as `ArbitrageClipIndex`, `RollingOUProcess`, and others—by using a registry of feature configurations. Each configuration specifies the model class, its parameters, signal definitions, and target metrics. This allows users to easily select and experiment with different strategies by referencing their configuration in the registry.
The modular approach means that:
* **Each Model Encapsulates Its Own Logic**: Parameters, signal models, and metrics are defined per model, ensuring that each strategy is self-contained and reusable.
* **Consistent Pipeline Usage**: Regardless of the model chosen, the Sandbox can run optimization, evaluation, and reporting in a uniform way, thanks to the standardized configuration structure.
* **Easy Extensibility**: New models or strategies can be added by simply creating a new configuration entry, without modifying the core workflow logic.
This design enables systematic experimentation and benchmarking of diverse strategies within a unified development environment.
## Pipeline Execution
Data is pulled using vectorbtpro’s `Data` objects, using `load_clean_data` by default. This function fetches and updates historical market data for the specified symbols and timeframes, ensuring the latest information is available for modeling.
Once the data is loaded, it is passed into the `OptunaTuner` class. This class executes the strategy logic, applying the model’s parameters and signal definitions to generate trading signals and portfolio analytics. `OptunaTuner` is responsible for automating the exploration of hyperparameters during strategy optimization. It leverages the `optuna` framework to systematically search for the best parameter combinations that maximize (or minimize) specified performance metrics.
Signals—such as long/short entries and exits—are generated within the pipeline by applying the model’s signal logic to the input data. Each model configuration specifies a `signal_model` (e.g., "twin\_spread" or "crossover1d") along with threshold parameters for entries and exits.
During execution, the pipeline uses these parameters to transform raw data into actionable trading signals. The parameter search systematically explores different combinations of these thresholds and logic settings, optimizing for the best-performing signal generation rules.
Key aspects include:
* **Automated Search**: `OptunaTuner` runs multiple trials, each with different hyperparameter values, to efficiently explore the parameter space.
* **Sampler/Pruner**: By default, the `Sandbox` uses the `TPESampler` (Tree-structured Parzen Estimator) for intelligent sampling and a pruner such as `SuccessiveHalvingPruner` to stop unpromising trials early, saving computation time.
* **Configuration**: The `Sandbox` allows users to set the number of trials (`n_trials`), control parallelization (`n_jobs`), and enable memory checks (`gc_after_trial`) to manage resource usage during optimization, among other tools.
* **Integration**: All these options are passed to `OptunaTuner`, which manages the study lifecycle, trial execution, and result collection, making hyperparameter tuning seamless and reproducible.
* **Cross-Validation**: The system supports cross-validation (when `cross_validate=True`), allowing strategies to be evaluated on multiple data splits for robust performance assessment. The pipeline can apply various transformations, such as rolling windows or expanding windows, to compute features or signals dynamically over time.
This approach ensures that strategies are tested under realistic, diverse conditions, supporting reliable research and development.
* **Multi-Asset Processing**: Data objects can be tuned using multiple pair of symbols, enabling simultaneous processing and evaluation of strategies across several assets.
* **Performance Metrics & Metric Registry**: Single or multiple quantitative measures such as Sharpe ratio, Sortino ratio, and annualized return, which assess the effectiveness of a strategy, are structured around a "metric registry". First, metrics are specified as a list or dictionary (e.g., metrics=\["sharpe\_ratio", "sortino\_ratio"]) in the Sandbox class. During pipeline execution, each metric is computed using the relevant method or function, often leveraging vectorbtpro’s built-in analytics or custom metric functions. If cross-validation or rolling windows are used, metrics are aggregated across splits or windows using a reduce function (e.g., `np.nanmedian`). Then, The computed metrics are used as objectives for hyperparameter optimization (e.g., maximizing Sharpe ratio), guiding the search process in `optuna`.
* **Structured Formats**: The Sandbox class returns a `FeatureOutput` namedtuple, which contains both the `optuna` study object and a dictionary of computed metrics. This approach ensures that all relevant outputs—optimization results, performance statistics, and signal data—are easily accessible for further analysis or reporting.
## Orchestration
The Sandbox, trackers, analyzers, and composers work together to form a cohesive workflow for strategy optimization and analysis:
* **Sandbox**: The Sandbox sets up the study by managing feature configurations, running optimization trials, and computing metrics. It acts as the central hub for experimentation and hyperparameter tuning.
* **Trackers (e.g., Neptune)**: Trackers like `NeptuneTracker` record intermediate and final results during optimization. They log trial data, metrics, and (optionally) visualizations, enabling detailed monitoring and reproducibility of experiments.
* **Analyzers and Composers**: Analyzers like `NeptuneAnalyzer` fetch and process logged data to generate advanced views, such as Pareto fronts, parameter importance plots, and optimization histories. Composers like `PortfolioComposer`, `SignalComposer` or `ColumnStackComposer` use the best trial(s) parameters to simulate and analyze a portfolio or multuple aggregated portfolios performance.
## Configuration Management
`FeatureConfig` objects serve as structured containers for storing model and parameter sets.
Each FeatureConfig defines:
* **Model Class**: Specifies the trading strategy or feature model (e.g., `ArbitrageClipIndex`, `RollingOUProcess`).
* **Parameters**: Includes signal thresholds (e.g., long\_entries, short\_exits) and model-specific configurations (e.g., training\_window, splitter).
* **Metadata**: Attributes like `study_name`, `timeframe`, and `id` help identify and organize configurations.
These objects are registered in the `FEATURE_MODEL_REGISTRY`, enabling modular experimentation and consistent usage across pipelines.
```python theme={null}
import systematica as sma
FEATURE_MODEL_REGISTRY = [
sma.FeatureConfig(
id="test",
study_name="ArbitrageClipIndex-1h",
timeframe="1h",
model=sma.ArbitrageClipIndex,
...
),
...
]
```
## Trial Monitoring
The `NeptuneTracker` is designed to log trial parameters and metrics during `optuna` optimization. It integrates seamlessly with Neptune to provide real-time tracking and post-experiment analysis.
Key Features:
* **Live Streaming**: When `live_stream=True`, `NeptuneTracker` uses the `NeptuneCallback` to log trial parameters, metrics, and visualizations during optimization. Intermediate results, such as trial distributions and optimization history, are logged at specified intervals.
* **Post-Experiment Logging**: If `live_stream=False`, the tracker logs all trial data and visualizations after the study is completed using `log_study_metadata`.
* **Trial Parameters**: Each trial's hyperparameters are logged under the `study/distributions` namespace.
* **Metrics**: Performance metrics like Sharpe ratio or Sortino ratio are logged under `best/values` and `trials/trials`.
* **Visualizations**: `optuna` visualizations (e.g., Pareto fronts, parameter importance) are (optionally) logged using the `NeptuneCallback`.
## Advanced Experiment Analysis
The `NeptuneAnalyzer` provides tools for analyzing and visualizing experiment data logged by `NeptuneTracker`. It enables detailed insights into optimization results and trial performance.
* **Best-Trial Selection**: Fetches the best trials based on metrics using `get_best_trials`. Allows users to identify optimal parameter sets for further analysis.
* **Hypervolume Analysis**: Computes and visualizes the hypervolume history for multi-objective studies using `plot_hypervolume_history`. Helps assess the trade-offs between competing objectives.
* **Score Distributions**: Generates histograms and correlation matrices for metrics and parameters using `plot_metrics_correlation` and `plot_params_correlation`. Provides insights into parameter sensitivity and metric relationships.
* **And More...**
The `NeptuneAnalyzer` complements the NeptuneTracker by enabling advanced post-experiment analysis, making it easier to refine strategies and optimize future experiments.