GBDTs as Hyper-Models for Classical Forecasting Models
+
+
+
Open Source
+
+
+
+
+
+
CI/CD
+
+
+
+
+
+
+
Package
+
+
+
+
+
+
+
Downloads
+
+
+
+
+
+
Paper
+
+
+
+
+
+
Release
+
+
+
+
+
+
+
+
---
@@ -34,6 +71,7 @@ Hyper-Trees offer several advantages:
---
# News
+[2026-06-03] v0.2.0 adds support for forecast intervals via conformal prediction.
[2026-06-01] v0.1.0 released on [PyPI](https://pypi.org/project/hypertrees-forecasting/).
[2024-05-01] Create repository and initial commits.
@@ -45,11 +83,16 @@ Hyper-Trees offer several advantages:
| :--- | :--- | :---: |
| **`Hyper-Tree-AR`** | Autoregressive model with tree-learned, time-varying AR(p) parameters. |   |
| **`Hyper-TreeNet-AR`** | Hybrid model combining tree embeddings with a neural network to learn AR(p) parameters. |   |
+| **`Hyper-Tree-ARMA`** | Autoregressive moving-average model with tree-learned, time-varying AR(p) and MA(q) parameters, fitted recursion-free via the two-stage Hannan-Rissanen procedure. |   |
+| **`Hyper-TreeNet-ARMA`** | Hybrid model combining tree embeddings with a neural network to learn the ARMA(p, q) parameters. |   |
| **`Hyper-Tree-ETS`** | Exponential smoothing model where ETS parameters are estimated by trees. |   |
| **`Hyper-Tree-STL`** | STL decomposition with tree-learned parameters for trend and seasonality. |  |
+| **`Hyper-Tree-VAR`** | Vector autoregression with tree-learned, time-varying VAR(p) coefficient matrices, capturing cross-series lead/lag dependence. Intended for small aligned panels. |  |
+| **`Hyper-TreeNet-VAR`** | Hybrid model combining tree embeddings with a neural network to learn the VAR(p) coefficient matrices; recommended VAR variant, since its runtime is independent of the number of coefficients. |  |
+| **`Hyper-Tree-TSB`** | Intermittent demand model (Teunter-Syntetos-Babai) with tree-learned, time-varying smoothing rates for demand probability and demand size. |   |
`Global` means a single model is trained across multiple time series; `Local` means a separate model is trained for each individual series.
-All models currently provide point forecasts only. Probabilistic forecasting is planned for future releases. Note on `Hyper-Tree-STL`: it is designed to decompose time series into trend and seasonal components and is not intended for forecasting. However, the STL-parameters can still be used to generate forecasts.
+All models produce point forecasts and support conformal prediction intervals via `ForecastIntervals` (see [Getting Started](#getting-started)). Full distributional (probabilistic) forecasting is planned for future releases. Note on `Hyper-Tree-STL`: it is designed to decompose time series into trend and seasonal components and is not intended for forecasting. However, the STL-parameters can still be used to generate forecasts.
---
@@ -59,6 +102,7 @@ The example below trains a `Hyper-Tree-AR` model on the classic AirPassengers se
```python
from hypertrees.models import HyperTreeAR
+from hypertrees import ForecastIntervals
from examples.utils import (load_air_passengers, plot_example_forecast)
# Load data and add 'month' as a feature
@@ -67,13 +111,18 @@ dta["month"] = dta["date"].dt.month
# Split the data into training and testing sets, reserving the last 12 months for testing
fcst_h = 12
-test = dta.tail(fcst_h)
+test = dta.tail(fcst_h).drop(columns="value")
train = dta.drop(test.index)
-# Initialize an AR-12 model for monthly data, train, and forecast
+# Initialize an AR-12 model for monthly data, calibrate conformal intervals, and forecast
+ci_levels = [80, 90]
ht_model = HyperTreeAR(p=12, freq="M", fcst_h=fcst_h)
-ht_model.train(lgb_params={"learning_rate": 0.1}, num_iterations=100, train_data=train)
-forecasts = ht_model.forecast(test_data=test)
+ht_model.train(
+ lgb_params={"learning_rate": 0.1},
+ train_data=train,
+ forecast_intervals=ForecastIntervals(n_windows=5), # calibrate intervals
+)
+forecasts = ht_model.forecast(test_data=test, level=ci_levels)
# Plot actuals vs. forecast
plot_example_forecast(dta, forecasts)
@@ -157,7 +206,7 @@ This work draws on and integrates methods and implementations from the following
- [**LightGBM**](https://github.com/microsoft/LightGBM) – Gradient boosting framework for efficient tree-based learning.
- [**PyTorch**](https://github.com/pytorch/pytorch) – Deep learning framework for tensor computation and neural network modeling.
-- [**Nixtla**](https://github.com/Nixtla) – Open Source Time Series Ecosystem.
+- [**Nixtla**](https://github.com/Nixtla) – Open Source Time Series Ecosystem. The conformal prediction intervals in `hypertrees/conformal.py` are adapted from Nixtla's [statsforecast](https://github.com/Nixtla/statsforecast), [mlforecast](https://github.com/Nixtla/mlforecast), and [neuralforecast](https://github.com/Nixtla/neuralforecast) (Apache-2.0); see [`THIRD_PARTY_NOTICES`](THIRD_PARTY_NOTICES).
- [**sktime**](https://github.com/sktime/sktime) – A unified framework for machine learning with time series.
- [**GluonTS**](https://github.com/awslabs/gluonts) – Probabilistic time series modeling and forecasting with deep learning.
diff --git a/THIRD_PARTY_NOTICES b/THIRD_PARTY_NOTICES
new file mode 100644
index 0000000..307b20f
--- /dev/null
+++ b/THIRD_PARTY_NOTICES
@@ -0,0 +1,43 @@
+Hyper-Trees
+Copyright 2026 Alexander März
+
+Licensed under the Apache License, Version 2.0, with the Commons Clause License
+Condition v1.0. See the LICENSE file for the full terms.
+
+------------------------------------------------------------------------------
+Third-party attributions
+------------------------------------------------------------------------------
+
+This product includes software adapted from the Nixtla open-source forecasting
+libraries, each licensed under the Apache License, Version 2.0:
+
+ - statsforecast https://github.com/Nixtla/statsforecast
+ - mlforecast https://github.com/Nixtla/mlforecast
+ - neuralforecast https://github.com/Nixtla/neuralforecast
+
+The following components are adapted from these libraries and have been
+modified from the originals:
+
+ - Conformal prediction intervals (hypertrees/conformal.py): the
+ rolling-origin calibration, the two interval-construction methods
+ ("conformal_distribution" and "conformal_error"), the per-horizon-step
+ quantile logic, and the "-lo-" / "-hi-"
+ output column naming convention follow Nixtla's design.
+
+ - Exponential smoothing state initialization
+ (hypertrees/models/HyperTreeETS.py): the classical and additive
+ seasonal/level/trend initialization (centered 2 x m moving-average
+ detrending, per-slot seasonal indices, the OLS level/trend seed, and the
+ initial-seasonal clipping) follows the statsforecast / R forecast::ets
+ heuristic.
+
+ - TSB intermittent-demand method (hypertrees/models/HyperTreeTSB.py): the
+ probability/size smoothing recursion and the state initialization follow
+ the statsforecast TSB implementation.
+
+ - Time series preprocessing (hypertrees/utils.py): the frequency-alias
+ conversion and the lag-column ordering follow the mlforecast conventions.
+
+A copy of the Apache License, Version 2.0, is available at:
+
+ http://www.apache.org/licenses/LICENSE-2.0
\ No newline at end of file
diff --git a/examples/Basic Walkthrough.ipynb b/examples/Basic Walkthrough.ipynb
index 5d90be3..44893df 100644
--- a/examples/Basic Walkthrough.ipynb
+++ b/examples/Basic Walkthrough.ipynb
@@ -12,47 +12,66 @@
{
"cell_type": "code",
"metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T10:02:35.357511Z",
+ "iopub.status.busy": "2026-06-11T10:02:35.357511Z",
+ "iopub.status.idle": "2026-06-11T10:02:48.211980Z",
+ "shell.execute_reply": "2026-06-11T10:02:48.211980Z"
+ },
"ExecuteTime": {
- "end_time": "2026-06-01T06:21:25.499778900Z",
- "start_time": "2026-06-01T06:21:25.490665700Z"
+ "end_time": "2026-06-12T12:00:11.427786800Z",
+ "start_time": "2026-06-12T12:00:01.259843200Z"
}
},
"source": [
+ "import numpy as np\n",
"import pandas as pd\n",
"import torch\n",
"from hypertrees.models import (\n",
" HyperTreeAR,\n",
+ " HyperTreeARMA,\n",
+ " HyperTreeNetARMA,\n",
" HyperTreeETS,\n",
" HyperTreeNetAR,\n",
" HyperTreeSTL\n",
")\n",
- "from utils import calculate_metrics\n",
+ "from utils import calculate_metrics, plot_forecasts, plot_stl\n",
"import matplotlib.pyplot as plt\n",
"import shap"
],
"outputs": [],
- "execution_count": 13
+ "execution_count": 1
},
{
"cell_type": "markdown",
"metadata": {},
- "source": "## General Parameters"
+ "source": [
+ "## General Parameters"
+ ]
},
{
"cell_type": "code",
"metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T10:02:48.214319Z",
+ "iopub.status.busy": "2026-06-11T10:02:48.214319Z",
+ "iopub.status.idle": "2026-06-11T10:02:48.256445Z",
+ "shell.execute_reply": "2026-06-11T10:02:48.256445Z"
+ },
"ExecuteTime": {
- "end_time": "2026-06-01T06:21:25.575064800Z",
- "start_time": "2026-06-01T06:21:25.522020600Z"
+ "end_time": "2026-06-12T12:00:11.939266800Z",
+ "start_time": "2026-06-12T12:00:11.429758700Z"
}
},
"source": [
"# General Parameters\n",
"lag_p=12 # Lag order for AR(p) models\n",
+ "q=1 # MA order for ARMA models\n",
"freq='MS'\n",
"fcst_h=12\n",
"num_iterations=100\n",
"num_seasonal_components=1\n",
+ "seed=123\n",
"device=torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"\n",
"# Hyper-Tree Parameters\n",
@@ -67,40 +86,111 @@
" 'hidden_dim': 128, # hidden dimension for the MLP network\n",
" 'dropout': 0.1, # dropout rate for the MLP network\n",
" 'use_random_projection': True, # whether to use random projections for the embeddings\n",
- "\t'rp_embed_dim': lag_p, # dimension of the random projections (if used)\n",
+ " 'rp_embed_dim': lag_p, # dimension of the random projections (if used)\n",
"}"
],
"outputs": [],
- "execution_count": 14
+ "execution_count": 2
},
{
- "metadata": {},
"cell_type": "markdown",
+ "metadata": {},
"source": [
"## Load and Prepare Data\n",
"We'll use the Air Passengers dataset which contains monthly airline passenger numbers from 1949 to 1960."
]
},
{
+ "cell_type": "code",
"metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T10:02:48.258449Z",
+ "iopub.status.busy": "2026-06-11T10:02:48.258449Z",
+ "iopub.status.idle": "2026-06-11T10:02:48.701896Z",
+ "shell.execute_reply": "2026-06-11T10:02:48.701896Z"
+ },
"ExecuteTime": {
- "end_time": "2026-06-01T06:21:26.075224100Z",
- "start_time": "2026-06-01T06:21:25.605213Z"
+ "end_time": "2026-06-12T12:00:12.524240500Z",
+ "start_time": "2026-06-12T12:00:12.079900Z"
}
},
- "cell_type": "code",
"source": [
"# The data needs to have the following columns: 'date', 'series_id', 'value'. All other columns are automatically treated as features.\n",
"# For the AR-models, you don't have to add lag-values yourself, this happens automatically during training.\n",
"df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/air-passengers.csv', parse_dates=['ds'])\n",
"df.rename(columns={'unique_id': 'series_id', 'ds': 'date', 'y': 'value'}, inplace=True)\n",
+ "\n",
+ "# Add month and quarter as features\n",
"df['month'] = df['date'].dt.month\n",
"df[\"quarter\"] = df['date'].dt.quarter\n",
+ "\n",
+ "# Split into train and test\n",
"test = df.tail(fcst_h)\n",
"train = df.drop(test.index)"
],
"outputs": [],
- "execution_count": 15
+ "execution_count": 3
+ },
+ {
+ "metadata": {},
+ "cell_type": "markdown",
+ "source": "# STL Decomposition via Hyper-Tree-STL"
+ },
+ {
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2026-06-12T12:00:14.287387600Z",
+ "start_time": "2026-06-12T12:00:12.534241400Z"
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "np.random.seed(seed)\n",
+ "\n",
+ "# Initialize\n",
+ "ht_stl = HyperTreeSTL(\n",
+ " period=12,\n",
+ " num_seasonal_components=num_seasonal_components,\n",
+ " freq=freq,\n",
+ " fcst_h=fcst_h,\n",
+ ")\n",
+ "\n",
+ "# Add time feature\n",
+ "df_stl = df.copy()\n",
+ "df_stl['time'] = df_stl.groupby(\"series_id\").cumcount() + 1\n",
+ "test_stl = df_stl.tail(fcst_h)\n",
+ "train_stl = df_stl.drop(test_stl.index)\n",
+ "\n",
+ "# Train\n",
+ "ht_stl.train(\n",
+ " lgb_params=ht_params,\n",
+ " num_iterations=num_iterations,\n",
+ " train_data=train_stl,\n",
+ " seed=seed,\n",
+ " verbose=-1\n",
+ ")\n",
+ "\n",
+ "# Extract STL parameters\n",
+ "stl_df = ht_stl.forecast(\n",
+ " train_stl,\n",
+ " type=\"components\",\n",
+ ")\n",
+ "\n",
+ "plot_stl(stl_df)"
+ ],
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "
"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "execution_count": 10
+ },
+ {
+ "cell_type": "markdown",
+ "id": "65a6dc11",
+ "source": "## Accuracy Comparison\n\nPoint-forecast metrics need care on intermittent data. The flat TSB forecast estimates the *expected* demand per period, and squared error (RMSE) is minimized by exactly that conditional mean, so RMSE is the fairest yardstick here. The absolute metrics (MAE, WAPE) are minimized by forecasting close to zero on mostly-zero series, so they reward degenerate forecasts and can exceed 100% WAPE even for a well-calibrated demand rate. MAPE is undefined on zero-demand weeks and is computed only on the nonzero subset. The TSB model estimates each SKU's expected demand rate, while the AR baseline, lacking an intercept, decays toward zero on the mostly-zero history and underforecasts.",
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "id": "854cb679",
+ "source": "fcsts_df = pd.concat(\n [\n tsb_fcst,\n ar_fcst,\n ], axis=0).merge(\n test[[\"series_id\", \"date\", \"value\"]],\n on=[\"series_id\", \"date\"],\n how=\"inner\"\n)\n\nfcsts_df.groupby(\"model\")[[\"value\", \"fcst\"]].apply(calculate_metrics).round(3)",
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:29:06.724460100Z",
+ "start_time": "2026-06-11T13:29:06.680527700Z"
+ }
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ " MAE MAPE sMAPE WAPE RMSE\n",
+ "model \n",
+ "Hyper-Tree-AR(8) 2.573 76.377 156.345 129.468 4.012\n",
+ "Hyper-Tree-TSB 2.331 71.810 170.232 117.276 3.219"
+ ],
+ "text/html": [
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
MAE
\n",
+ "
MAPE
\n",
+ "
sMAPE
\n",
+ "
WAPE
\n",
+ "
RMSE
\n",
+ "
\n",
+ "
\n",
+ "
model
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
Hyper-Tree-AR(8)
\n",
+ "
2.573
\n",
+ "
76.377
\n",
+ "
156.345
\n",
+ "
129.468
\n",
+ "
4.012
\n",
+ "
\n",
+ "
\n",
+ "
Hyper-Tree-TSB
\n",
+ "
2.331
\n",
+ "
71.810
\n",
+ "
170.232
\n",
+ "
117.276
\n",
+ "
3.219
\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "execution_count": 7
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f9d73268",
+ "source": "## Practical Notes\n\n- **Equal-length panel**: TSB reshapes the panel to (n_series, T), so all series must have the same length. Pad shorter series and supply a `mask` column (1 = valid, 0 = padding); padded rows are excluded from the loss and from the conformity scores.\n- **Flat point forecast**: the TSB forecast is constant over the horizon at `p_T * z_T`. Horizon features therefore do not move the point forecast, though they do change `type=\"parameters\"`. This is the classical TSB behaviour: future occurrence is unobserved, so the expected one-step-ahead state propagates unchanged.\n- **Feature-driven smoothing**: the gain over classical TSB comes from letting `alpha_p` and `alpha_d` vary with features, so the probability and size estimates adapt their responsiveness by context (promotions, season, SKU).\n- **When to use it**: reach for TSB when the series is genuinely intermittent. On smooth, non-zero series an AR or ETS target model is usually the better fit.",
+ "metadata": {}
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.11"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/VAR Models.ipynb b/examples/VAR Models.ipynb
new file mode 100644
index 0000000..8e8850d
--- /dev/null
+++ b/examples/VAR Models.ipynb
@@ -0,0 +1,598 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "e99b797525414fdb",
+ "metadata": {},
+ "source": "# Vector Autoregression with Hyper-Trees\n\nThis walkthrough introduces the multivariate Hyper-Tree models. They extend the autoregressive Hyper-Trees to a VAR(p) target model over an aligned panel of k series:\n\n$$y_{i,t} = \\sum_{j=1}^{p} A_j[i, :](x_{i,t}) \\cdot y_{t-j}$$\n\nEvery series' forecast is a feature-dependent linear combination of the lagged values of *all* series, so cross-series lead/lag dependence is captured by the off-diagonal coefficients of the time-varying lag matrices.\n\n- **`HyperTreeNetVAR`** (GBDT encoder + MLP decoder): the boosting cost is independent of the number of coefficients. This is the **strongly recommended** variant.\n- **`HyperTreeVAR`** (one boosted tree per coefficient): fully interpretable and intended for small panels, since each boosting round grows `k * p` trees.\n- **`type=\"factor\"`** (restricted \"Global VAR\", available on both models): each equation keeps only its own lags plus a shared cross-sectional factor, so the coefficient count per equation is `2 * p` regardless of k. This makes it a scalable middle ground when the unrestricted VAR is over-parameterized.\n\nAll models require an *aligned panel*: all series must have the same length and identical dates."
+ },
+ {
+ "cell_type": "code",
+ "id": "4e68ded019b2d1f4",
+ "metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T09:50:54.355207Z",
+ "iopub.status.busy": "2026-06-11T09:50:54.355207Z",
+ "iopub.status.idle": "2026-06-11T09:51:05.430153Z",
+ "shell.execute_reply": "2026-06-11T09:51:05.430153Z"
+ },
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:43:08.481273300Z",
+ "start_time": "2026-06-11T13:43:04.798619300Z"
+ }
+ },
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import torch\n",
+ "\n",
+ "from hypertrees import ForecastIntervals\n",
+ "from hypertrees.models import HyperTreeAR\n",
+ "from hypertrees.models.HyperTreeVAR import HyperTreeVAR\n",
+ "from hypertrees.models.HyperTreeNetVAR import HyperTreeNetVAR\n",
+ "from utils import calculate_metrics, plot_panel_forecasts, simulate_var_panel\n",
+ "\n",
+ "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+ "\n",
+ "# Shared setup, used by all models\n",
+ "seed = 123\n",
+ "num_iterations = 300\n",
+ "level = [80]\n",
+ "forecast_intervals = ForecastIntervals(\n",
+ " n_windows=10, # number of rolling calibration windows\n",
+ " method=\"conformal_distribution\", # or \"conformal_error\"\n",
+ " step_size=1, # time steps between windows\n",
+ " refit=False, # train once on the oldest window for speed\n",
+ ")"
+ ],
+ "outputs": [],
+ "execution_count": 1
+ },
+ {
+ "cell_type": "markdown",
+ "id": "var-ex-sim-md",
+ "metadata": {},
+ "source": [
+ "## Simulate an Aligned Panel\n",
+ "\n",
+ "We simulate 10 monthly series over 20 years from a stable VAR(1) with a lead/lag *chain*: every series follows its own past and its neighbor's previous month (`A[i, i-1] = 0.3`). On top we add a common seasonal profile and strongly heterogeneous scales, which is exactly the situation the models' built-in per-series scaling (`scaling=\"mean\"`, the default) is for.\n",
+ "\n",
+ "Feature columns: `month`, `quarter`, and a series identifier `series_num`. The GBDT learns one global mapping from features to coefficients, so a feature must identify the series for the equations to differ. We cast it to pandas `category` dtype so LightGBM applies true categorical splits."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "id": "var-ex-sim",
+ "metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T09:51:05.432183Z",
+ "iopub.status.busy": "2026-06-11T09:51:05.432183Z",
+ "iopub.status.idle": "2026-06-11T09:51:05.447712Z",
+ "shell.execute_reply": "2026-06-11T09:51:05.447181Z"
+ },
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:43:08.516782100Z",
+ "start_time": "2026-06-11T13:43:08.483290200Z"
+ }
+ },
+ "source": [
+ "# Simulation parameters: k series, n_train months of training data, forecast horizon fcst_h, and VAR lag order var_p (number of coefficient matrices A1, A2, ...)\n",
+ "k, n_train, fcst_h, var_p = 10, 240, 12, 4\n",
+ "\n",
+ "df, train, test = simulate_var_panel(k=k, n_train=n_train, fcst_h=fcst_h, seed=seed)\n",
+ "print(f\"{k} series x {n_train} months, forecast horizon {fcst_h}\")"
+ ],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "10 series x 240 months, forecast horizon 12\n"
+ ]
+ }
+ ],
+ "execution_count": 2
+ },
+ {
+ "cell_type": "markdown",
+ "id": "eac95674e4754575",
+ "metadata": {},
+ "source": [
+ "## Hyper-TreeNet-VAR (recommended for runtime efficiency)\n",
+ "\n",
+ "The GBDT produces a low-dimensional embedding that an MLP decodes into all `k * p = 40` coefficients per row. Per-series scaling is on by default and forecasts come back on the original scale. We also calibrate conformal prediction intervals via rolling-origin cross-validation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "id": "d91f4020e74aaa26",
+ "metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T09:51:05.448747Z",
+ "iopub.status.busy": "2026-06-11T09:51:05.448747Z",
+ "iopub.status.idle": "2026-06-11T09:51:10.172680Z",
+ "shell.execute_reply": "2026-06-11T09:51:10.171671Z"
+ },
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:43:14.994445600Z",
+ "start_time": "2026-06-11T13:43:08.520793Z"
+ }
+ },
+ "source": [
+ "torch.manual_seed(seed)\n",
+ "np.random.seed(seed)\n",
+ "\n",
+ "htnet_var = HyperTreeNetVAR(\n",
+ " p=var_p,\n",
+ " freq=\"M\",\n",
+ " fcst_h=fcst_h,\n",
+ " device=device\n",
+ ")\n",
+ "\n",
+ "htnet_var.train(\n",
+ " lgb_params={\"learning_rate\": 0.1},\n",
+ " network_params={\n",
+ " \"learning_rate\": 1e-3,\n",
+ " \"embedding_dimension\": 1,\n",
+ " \"hidden_dim\": 64,\n",
+ " \"dropout\": 0.1,\n",
+ " \"use_random_projection\": True,\n",
+ " \"rp_embed_dim\": k * var_p,\n",
+ " },\n",
+ " num_iterations=num_iterations,\n",
+ " train_data=train,\n",
+ " seed=seed,\n",
+ " forecast_intervals=forecast_intervals,\n",
+ ")\n",
+ "\n",
+ "# Request interval columns via level: adds -lo-80 / -hi-80\n",
+ "htnet_var_fcst = htnet_var.forecast(test_data=test, level=level)"
+ ],
+ "outputs": [],
+ "execution_count": 3
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1880c6ce45f404",
+ "metadata": {},
+ "source": [
+ "## Hyper-Tree-VAR\n",
+ "\n",
+ "One boosted tree per coefficient, so every coefficient is a separate, SHAP-able tree output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "id": "ce70e089df123f2e",
+ "metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T09:51:10.175680Z",
+ "iopub.status.busy": "2026-06-11T09:51:10.174680Z",
+ "iopub.status.idle": "2026-06-11T09:51:26.093567Z",
+ "shell.execute_reply": "2026-06-11T09:51:26.092563Z"
+ },
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:43:46.075989900Z",
+ "start_time": "2026-06-11T13:43:15.234756800Z"
+ }
+ },
+ "source": [
+ "torch.manual_seed(seed)\n",
+ "np.random.seed(seed)\n",
+ "\n",
+ "ht_var = HyperTreeVAR(\n",
+ " p=var_p,\n",
+ " freq=\"M\",\n",
+ " fcst_h=fcst_h\n",
+ ")\n",
+ "ht_var.train(\n",
+ " lgb_params={\"learning_rate\": 0.01},\n",
+ " num_iterations=num_iterations,\n",
+ " train_data=train,\n",
+ " seed=seed,\n",
+ " forecast_intervals=forecast_intervals,\n",
+ ")\n",
+ "ht_var_fcst = ht_var.forecast(test_data=test, level=level)"
+ ],
+ "outputs": [],
+ "execution_count": 4
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3483d6fc",
+ "source": "## Hyper-Tree-FactorVAR (restricted, scales to large panels)\n\nA middle ground between the univariate AR and the unrestricted VAR. Each equation uses only its **own** lags plus the lags of a single cross-sectional **factor**. That factor is the equal-weighted mean of the (scaled) panel, the \"star variable\" of the Global VAR literature. This keeps a cross-series channel while cutting the coefficient count per equation from `k * p = 40` to just `2 * p = 8`, *independent of the number of series*. It is the principled choice when the panel is too large for the unrestricted `HyperTreeVAR`. Coefficients split into own-lag (`A{j}(own)`) and factor-lag (`A{j}(factor)`) blocks.",
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "id": "dff7ae00",
+ "source": [
+ "torch.manual_seed(seed)\n",
+ "np.random.seed(seed)\n",
+ "\n",
+ "ht_factor_var = HyperTreeVAR(\n",
+ " type=\"factor\",\n",
+ " p=var_p,\n",
+ " freq=\"M\",\n",
+ " fcst_h=fcst_h,\n",
+ ")\n",
+ "ht_factor_var.train(\n",
+ " lgb_params={\"learning_rate\": 0.05},\n",
+ " num_iterations=num_iterations,\n",
+ " train_data=train,\n",
+ " seed=seed,\n",
+ " forecast_intervals=forecast_intervals,\n",
+ ")\n",
+ "ht_factor_var_fcst = ht_factor_var.forecast(test_data=test, level=level)"
+ ],
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:43:49.384413200Z",
+ "start_time": "2026-06-11T13:43:46.103857700Z"
+ }
+ },
+ "outputs": [],
+ "execution_count": 5
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7f98e44d",
+ "source": "### How it works and how it differs from the other models\n\n**The mechanism.** Each equation forecasts a series from two blocks of lags: its **own** past, and the past of a single shared **factor** $f_t = \\frac{1}{k}\\sum_{i} y_{i,t}$. That factor is the equal-weighted mean of the *scaled* panel (the \"star variable\" of the Global VAR literature), computed internally:\n\n$$y_{i,t} = \\underbrace{\\sum_{j=1}^{p} a_{j}(x_{i,t})\\, y_{i,t-j}}_{\\text{own lags}} \\; + \\; \\underbrace{\\sum_{j=1}^{p} c_{j}(x_{i,t})\\, f_{t-j}}_{\\text{factor lags}}$$\n\nBoth blocks share the same lag order `p`: for every lag `j` there is one own-lag coefficient $a_j$ and one factor-lag coefficient $c_j$. As in every Hyper-Tree, the coefficients $a_j, c_j$ are **time-varying functions of features** generated by LightGBM. Here that means one boosted tree per coefficient, so `num_class = 2 * p`. When forecasting, the recursion advances all series jointly: at each step the fresh forecasts become the next own-lag, and **the factor is recomputed as the cross-sectional mean of those forecasts**, so the common signal keeps propagating over the horizon. `forecast(type=\"parameters\")` exposes the two blocks as the `A{j}(own)` and `A{j}(factor)` columns.\n\n**How it differs.** It sits between the univariate AR and the full VAR on the bias/variance spectrum:\n\n| Model | Coefficients per equation | Cross-series channel | Estimator |\n|---|---|---|---|\n| `HyperTreeAR` | `p` | none (own lags only) | one tree per coefficient |\n| **`HyperTreeVAR(type=\"factor\")`** | **`2p`** | **one shared common factor** | one tree per coefficient |\n| `HyperTreeVAR` | `k·p` | full pairwise lag matrices | one tree per coefficient |\n| `HyperTreeNetVAR` | `k·p` | full pairwise lag matrices | GBDT encoder → MLP decoder |\n\n- **vs `HyperTreeAR`**: AR sees only each series' own history. The factor design adds the factor block, so it can borrow the panel-wide signal at a coefficient count (`2p`) that barely grows beyond AR's `p`.\n- **vs the unrestricted `HyperTreeVAR`**: the full VAR regresses every series on *all* other series' lags (`k·p` coefficients, dense pairwise lead/lag links). The factor design replaces that dense block with a single factor, so its cost is **independent of k** and it resists the overparameterization that sinks unrestricted VARs on large panels. The tradeoff is that it can only capture cross-series dependence that flows through the common factor, not arbitrary pairwise links.\n- **vs `HyperTreeNetVAR`**: NetVAR still targets the *full* `k·p` matrices but keeps boosting cheap with a GBDT→MLP decoder. The factor design instead shrinks the *number of coefficients structurally*; on the direct model used here it stays fully interpretable (an explicit own/factor split, one SHAP-able tree per coefficient) with no neural network. `HyperTreeNetVAR` also accepts `type=\"factor\"` if you want the factor design with the MLP decoder.\n\nIn short: reach for it when the panel is large and cross-series dynamics plausibly run through a **common driver** (a market, category, or panel-wide factor) rather than dense pairwise relationships.",
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "id": "var-ex-ar-md",
+ "metadata": {},
+ "source": [
+ "## Univariate Baseline: Hyper-Tree-AR\n",
+ "\n",
+ "The univariate model sees only each series' own lags, so it cannot exploit the lead/lag chain."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "id": "dc0081e6b52f8d7c",
+ "metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T09:51:26.095568Z",
+ "iopub.status.busy": "2026-06-11T09:51:26.095568Z",
+ "iopub.status.idle": "2026-06-11T09:51:28.657279Z",
+ "shell.execute_reply": "2026-06-11T09:51:28.656777Z"
+ },
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:43:53.208257100Z",
+ "start_time": "2026-06-11T13:43:49.416767800Z"
+ }
+ },
+ "source": [
+ "torch.manual_seed(seed)\n",
+ "np.random.seed(seed)\n",
+ "\n",
+ "ht_ar = HyperTreeAR(\n",
+ " p=var_p,\n",
+ " freq=\"M\",\n",
+ " fcst_h=fcst_h\n",
+ ")\n",
+ "ht_ar.train(\n",
+ " lgb_params={\"learning_rate\": 0.1},\n",
+ " num_iterations=num_iterations,\n",
+ " train_data=train,\n",
+ " seed=seed,\n",
+ " forecast_intervals=forecast_intervals,\n",
+ ")\n",
+ "ht_ar_fcst = ht_ar.forecast(test_data=test, level=level)"
+ ],
+ "outputs": [],
+ "execution_count": 6
+ },
+ {
+ "cell_type": "markdown",
+ "id": "var-ex-plot-md",
+ "metadata": {},
+ "source": "## Forecasts\n\nAll four models on three of the ten series, with the 80% conformal interval of the recommended `Hyper-TreeNet-VAR`."
+ },
+ {
+ "cell_type": "code",
+ "id": "var-ex-plot",
+ "metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T09:51:28.659285Z",
+ "iopub.status.busy": "2026-06-11T09:51:28.659285Z",
+ "iopub.status.idle": "2026-06-11T09:51:29.945714Z",
+ "shell.execute_reply": "2026-06-11T09:51:29.945714Z"
+ },
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:43:53.936535300Z",
+ "start_time": "2026-06-11T13:43:53.241140Z"
+ }
+ },
+ "source": "datasets = [\n (df, \"date\", \"value\", \"Actual\", \"#2E86AB\", \"-\"),\n (htnet_var_fcst, \"date\", \"fcst\", \"Hyper-TreeNet-VAR Forecast\", \"green\", \"--\"),\n (ht_var_fcst, \"date\", \"fcst\", \"Hyper-Tree-VAR Forecast\", \"red\", \"--\"),\n (ht_factor_var_fcst, \"date\", \"fcst\", \"Hyper-Tree-FactorVAR Forecast\", \"purple\", \"--\"),\n (ht_ar_fcst, \"date\", \"fcst\", \"Hyper-Tree-AR Forecast\", \"orange\", \"--\"),\n]\n\nplot_panel_forecasts(\n datasets,\n series_ids=[\"Series 2\", \"Series 6\", \"Series 10\"],\n split_date=test[\"date\"].min(),\n interval_fcst=htnet_var_fcst,\n level=level[0],\n)",
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "
"
+ ],
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABc4AAAH2CAYAAABJFh/yAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQABAABJREFUeJzsnQeYG9XZ/V/17b25risuuIExvUNM74QAoYQY+EiABEho+QihfAmBQEhCKP8AoZMACTYdm96rwcY27t3e3lcrrer8n/dqr/ZKK+1KWzSj3fN7HtlaaVY7Go3m3nvuuec1aZqmEQAAAAAAAAAAAAAAAAAABObQfwAAAAAAAAAAAAAAAAAAgHAOAAAAAAAAAAAAAAAAAEQBxzkAAAAAAAAAAAAAAAAAoADhHAAAAAAAAAAAAAAAAABQgHAOAAAAAAAAAAAAAAAAAChAOAcAAAAAAAAAAAAAAAAAFCCcAwAAAAAAAAAAAAAAAAAKEM4BAAAAAAAAAAAAAAAAAAUI5wAAAAAAAAAAAAAAAACAAoRzAAAAYAQyYcIEMplMvd7+8pe/6L2bw46f/OQn4tg+/vjjZGRinQ+ZmZnivDn77LPp448/pnTj/fffF+/j8MMPp3RnyZIldPLJJ9Po0aPJbrdTfn4+TZkyhY499li6/fbbac2aNRHbb9u2Tbx3/vyMTLrsZzLXWH5PffHPf/5TbDtmzBgKBAJ9bv/tt9+K7a1WK1VVVfV4ns8N+b1dvXp1r691yy239PiuWywWKiwspP3335/+8Ic/kNPppMG8jkyePJkuuugi+u677yjdGE7XEQAAAAD0jRUHCQAAABi5HHTQQUJwi8XMmTNTvj/pDIvhLAZdeOGFhhfGE+WYY46hiooKcb+hoYG+/vpreu655+j555+ne++9l375y1/ScICFMEbTNDIyLKqef/759K9//Uv8vOeee9K+++4rxMgdO3bQhx9+SEuXLqXW1la6++67aaTDIucRRxxBhx12mLhvVM466yzxXWIRnD+/448/vk+hnTnuuOPE5IlKdXU1vf766+GfH330UfFd7Yvy8nIx8cL4fD7asmULffHFF+L25JNP0kcffUSlpaUDvo7U1dXRV199Ja6RzzzzDD399NPi/QMAAAAAGBEI5wAAAMAI5uKLLxYuaJAa7rjjDrrhhhto1KhRaXHIeV9VZ6XL5RLC7YsvvkjXXXcdnXnmmcIlC1LDQw89JETz3Nxceumll4QorMKfz6uvviqETxX+jNauXUs2mw0flQHJycmhH/3oR0LkZlG8N+Hc4/HQs88+K+4vWrSox/NPPPGEmGDhz3z37t1CmL7zzjvFyoTemD59eo8JP56I+cEPfkDr168XzvT7779/UK4jPLHzwx/+kN566y265JJLxN9ghzsAAAAAgNFAVAsAAAAAQIpgwZwFKo7WSEeysrLovvvuE/e9Xq9wx4LU8e9//1v8f8UVV/QQzeXnw+7dH//4xxGPs2DO5x1HZABjIkXwV155hRobG+NuxxMmTU1NVFZWRieeeGJcN/o999xDkyZNEitF+Hf6w6GHHipW0Mj9Giz4+vePf/xD3G9ra8N1BAAAAACGBcI5AAAAABJi165ddOWVV9LUqVMpIyNDiB8c9fL//t//i5nLy+5FjsBgRzsLPVdddZUQ7hwOR4982HfeeYdOP/10ISyzM5JFodNOO40+++yzuPvD7lrOYT/44IOFW5Fft7Kykk466aSwI1Oyfft24bo88sgjafz48WLbgoIC8bu8/8FgMObfWL58uXCCjh07VuxXXl6eEKPOOOOMCDGK84w5pkU6PtVMX/W9xss4lznD/H99fT1dfvnlNG7cOPE3+X8+7i0tLTH3keNFWCzbZ599hHBaXFwsIhw+/fTTIcnj5WgI/htMbW1tzG3+85//iNgHjnbg98Du1/POO4++//77AR3nRHLi1fOuL+Rxl0TnMav51C+88AIdffTR4r2zEM3/c5wRO2ZTldUsjzd/PwYrO1y+V4bdyRz9wg5o/uzOOeccEQEjz7O///3vNG/ePMrOzqaSkhJxjDl6Ixr1fI5Ff87LL7/8Uqxy4P3j2A8+TzhehL/vb7/9do/t+bXl5MIHH3wQ8bnGOg79uQbx+czOaT4WHJcza9YsEZGTSE55NAcccIA4n3hCij+HeEhh/IILLhAZ5yr8Pjdu3CjOTd53eU1iJ3t/mTNnTq/f9f7Cn0FRUZG4L79n/blOq+c2n6MsyM+fP1+co9xGLVy4sNfP0O12i0kGznPnv8Vt27Rp08S51tsEBgAAAABGBohqAQAAAECfcCYtC6EsgLOgceqpp4rl9iyAsUC7ePFievnll2PGAbDjkUVdFn4POeQQIWqo2/36178WwoXZbBbb8TYs1rFgyi7Hhx9+OCwASXbu3Cn2h4UrFotZwGexiKMJOIt31apVdO6554a3f+qpp+i3v/0tTZw4kfbYYw+xPWcBs6DyySef0LJly4TYq4qoLKSxAM2xF3PnzhXCFgti/Ddee+01cf+UU04R23Jkyeeffy5eiycHWOiRsNM3Ufh97b333uJv8j52dnaK12TBkrOG+X503AaL7A8++KA4fnzsWPjj989uUZ6sGGxYvJLFAlm4VPH7/cLtzBnoLHrxZ82i+YYNG0SeMUe88E1mKSd7nAcbFoHZUcuTHYx010pYQGZuu+02+t3vfieEygMPPFC8Jz7/+TxlUZKzxqXAOJTwd4+FUZmnP5grF2688UYh+vJ5w58HC9XscOdzbuXKlXTZZZeJ7zgL0jypwY/zceNClXx96CsKZKD85je/offee08caymMbt68WUTT8I0n0dTMfT7HWATlVRFqfjfDQrdKf65BXCCXX7Ojo0McD44b4Wsd7ydfC/rrOv/Vr35Fjz32WMz6ATx5yfEmcttopEDO30H+PHhig89b/h2+tvAkXLKwIzzWd30wriN87Bi+VvT3Oq3CnxFPmvLnx278FStWiPfOkTM8qbDffvtFbM+Z8vwZ8vWSRfwFCxaIGKRvvvmG/vSnP4nJMm7jeEIWAAAAACMUDQAAAAAjjsrKSq6CqD322GN9btvZ2Rne/rLLLtO8Xm/4uc2bN2sTJkwQz/3mN7+J+D1+bX6cb0cddZTW2tra47X/8Y9/iOenTJmirVy5MuK5Dz74QMvNzdXsdru2YcOG8OOBQEDbZ599xO8tXLhQq6uri/g9t9utvfbaaxGPffnll9qqVat6/P3du3drc+fOFa/1/PPPRzx3xBFHiMeffvrpHr/X0tKiffbZZzHf74UXXqjFg5+Lddx/97vfhY/VT37yE3HMJTt27NDGjBkjnnv22Wcjfu+ll14Sj+fk5GiffPJJxHP33HNP+DUPO+wwLRnk77333ns9nlu2bJl4jj8XPn4qfA7wc/vtt5+2ZcuWiOdeeOEFzWKxaIWFhVpzc3O/j3O8Y9jX58DvJd6xkO83FvxZZGZmimO8bt26Hs9v27ZNW7t2rZYKFi9eHN7X/Px87bzzztMeeOAB7fPPP9c8Hk/c39u6dav4Hf4eRyNfr7i4WFuxYkX4cZfLpR188MHiudmzZ2uTJ08W71VSX18vvrexPjt5PvP/sYj3WfS2n6+//rpWVVXV4/FPP/1Uy8vL02w2m7Zr166E/s5Ar0F8jRk3bpz4vauuukrz+/3h5/g1SkpKwseV31Oi8LWM3wf/3vLly3s8/3//93/iuQMPPDDmd4XPU35e/RyPOeYY8dhtt90W82/KzyreMeK/xc9ffvnlWrL0dh159dVXw8+/++67/b5Oy3NGnjfr168PP8efy09/+tNwW6ESDAa1gw46SDy3aNEira2tLfycz+fTfvWrX4nn+PqU7DkFAAAAgOEDhHMAAABgBCKF8Hg3VRR46qmnxGOjR4+OEHQl//nPf8TzLDCxoBQtYLIQxAJ7NCyA82vyNl9//XXM/bzrrrvE8yxiSJYsWSIeGzVqlNbe3j7gY7F06VLxej/84Q8jHp85c6Z4vKmpKaHXGQzhfOzYsVpHR0eP3/vjH/8onmcRSOXII48Uj994440x/96CBQsGTThnoZTFb/7MzGazEBxVGhsbhXCXkZHRQ8CU/PznPxeve9999/X7OKdaOGcxk5+bM2eOZgQeffRRIXJHf2f5uJ9++ulCfOyPcH7//ff3eO7FF18MPx89GaVOzlx00UVDLpz3Bp//sd5DXyJnf69BPFHAj7F4rk4kSu69995+CefMGWecEVeolhMVjzzySI/nHnzwQfHc/PnzIx5noZkfnzhxohCLExHO+T3xZBBP4vFz8+bN0xoaGrRkiXcd4QnAsrKy8Gvz59Df67QqnL/88ss9fq+6ulo853A4Ij6rN954I/z3WSiPhvdp1qxZYhtVzIdwDgAAAIwsENUCAAAAjGB4KfyUKVN6PK7Gi/BSdebss88OL6lX4Vxgzhhvbm4WWdX8mip77bWXiDKIhiMeeKk8R5tw9EIsZAYyx8FI3nzzTfE/R7HIKI1E8Hg8Yqk/x0pwLjP/zNpOe3u7eH79+vUR23OWMkfBcOwBxy9wBm50pvBgc9RRR4nomWhmzJgh/uf4EjUWRR6X6GKQEj5G/H77S6wClJzlzMeR91WFYzQ4L5gf5yiTeJ/nAw88IPabC1zqdZyTgbO+OT+Zc8w5RoMjMjiLWi9++tOfiu8ix5PwMf/666/FvnGsD8fgcLzIQw89RBdffHFSr3v88cf3eIzrGTD8eXBWdLzn+XucCjhzmuN7Vq9eLa43HO/DcHxNrO9wX/T3GiSviVyINTo6SUb+XH311dQf+HP773//KyJHOD5GXnM5amTTpk3imsf1AKJ55JFHwueHCscccYzV1q1b6d133+3xvZXIHPhoOEOe41EGEsUT6zrCcCwVn7MckdPf67SEz1E1jkfCefiyfeLzh39m+DxiuI5CrOsN7xPHFvG5xp8959cDAAAAYORhnFEJAAAAAFIOizR9FVGUYi3nzsaCxRZ+joUJVdiVxCrEx2zZskX8zznF8TJrJVwwU8IF5JLNDufMYRabZKHD3rJ8JXfccYcQJN944w1xY8GYhR4W0ljklWL2YGdYx4KLZTIsjko4T1n+HO8Yx3s8UY455hghNHEecU1NjcgKZnGcC31y5rA6ISI/T84sT+bz1OM4J8uTTz4pcuz//Oc/ixvnIXNeMudan3/++T0ys+OxZMkScYv1PVRz8fuCJ1dYtOUbw1nRfOx44oFFZM69ZxGRi60O5NyTE1Ocmx9LXOQ86OjzcqjgnHEWo2UudiLf4b7o7zWIs8Z7uyayUMv585yDnyw8QcFZ5JxJzrUjeJJELQrKn3n0hCFn0POkJWe6q7UdGBa8+Xv0t7/9TbxGPOFczYHnwsv8mlybgDPeOXeci3YO9DrC8EQAFxjmHHIW1NXj3p/rtITP0ViTGPL6ye2Tep7Kz57fG98S/ewBAAAAMLKAcA4AAACAIYWF0FiwGMuwoMLCSm8kKkzGgkUgLmZaW1srisf97Gc/Ey57FlMsFosQh6ZNmyZcjSq8X+zmZSfm22+/LYRiWaDzD3/4gxB8r7/+ehpMVOflYNCXGNgXN9xwQ9hxy7A7lz8rdmGyQMdF++TfkJ8nH9voVQfRqJMeg32c5X4MJizybdu2TbhUeT/ZgcpFJ1ms5uKLLHDGEyRVuFihLESqwsc4GeE8Gi6UycI+F1bloop8zvO+XXLJJYNy7g32eZnsZ8Si8P/8z/+I7ysLuOyCZqGfJxD4/PvHP/4hno/+Die6H0N9DUoGPtY8mXn77beLIqEsnLPbml3ffRUF5ckNLooZDTutGXZ3c5HmgoKCmN9JLjqrct9999EvfvELuuuuu+iwww6LuSqhP9eRwbxO9/cclZ89f+94xUFvcEFaAAAAAIxMIJwDAAAAoFdk7IZ06MWCYwDUbROBXZUMxwhECzaJOGPXrVuX0PbskmYxhl3M0rWpImMeYsGiHAs+UvRhxyLvKzt62d3LYmVfostQwceN3ZscZcAu/FjxISz2DibsFH3hhRdozpw5Qtx+5plnhPtc/TxZ3Erm80z2OMvICBndEI1ckTAUE0C8H3yTLtSbbrpJiLYcj5HI373lllvEbajg7x+fBzwRwSsS9GKwPyM+51gwvfLKK+m6665L6js8FNcgeZ2L9/1icbo/bnMJn0//93//JyaS2HnOkzQsLLO4feCBB0Zsy99//h4yTqdTTDjFg79XvC1/rxKBj/eXX35JTz/9NF1zzTXCDT9UMUoDuU4P5LPnKJtf//rXg/raAAAAABg+DK59BAAAAADDDilmPvfcczEjGdhty8vgObYhXk5wLBYsWCBcnJxvvWbNmoR/T8YJ/Otf/+o1tkHS1NTUawwKi0KJwlEIl112mRCO2bHIESPRYiFnj6cCjiVghzHDecix4GM02LB4x25QhkVg+X7Zcc3HgPOfOZt4IPR2nKVouXbt2h6/x+IqO62TRUY8JPPZcfY5O3EZjpbg78BQ05ejOhAIhOOSkolpGWx6+4zUfOlEkd/hysrKHs/xNYkzwWPR13eyv9cgdl8zzz//fDhnPTraZyBwxBJ/n/jcZ0FfCsnR+eXSRc7Hhye1+H3yORLrxrUFVHd6orDDnyeNOFv8qaeeoqFiMK/TiXDcccdFTMoAAAAAAMQCwjkAAAAAeuWHP/yhEDM4poNdh6oIxU5zLpgo3YkseCYjVnLMBYsWp512Gn388ccxhUAuaMfZt5KTTz5ZFBzl/eF9kzEEqpCmiqcyI5uzt1kgU2G3ME8IxOLuu++OmbXLTnfpflSFPClURv+NoYRjFBjOL1aPEfPXv/5VuMKHAnZac84yZ0PL6BHOSOZzgCczOEpj1apVPX6P3bEvv/xyxGqBZI/z0UcfLf5nEU891ixgcqRLf4qhys8ulnjK7mguvBgrW5nzn2WmtcyhH0o4hoOFzFjFONnlzBMa1dXVYl+kMKgHRx55pIjOYKc0R9tI+LvO52o8oTse8jvM55rqYufv+s9//vPwipd4nyufR7EE7v5eg3jVAU8O8Hl74403RkTPcIwRu8UHioxk4ePFkUjs9L7gggt6bCeFcF75wZEm8eDIF55I4IKoHBmUKCzI8/ea4fc1VBOD/b1O9xd2mvPECTvqORomVo45T4Zxod1UTYYCAAAAwIBoAAAAABhxVFZWssVOe+yxxxLa/ssvv9SKiorE7/Dv/uhHP9KOP/54LSMjQzx2zDHHaB6PJ+J3+LX5uQsvvLDX17722mvFdnzbc889tVNOOUU7++yztcMPP1wrKCgQjz/44IMRv7Nt2zZt2rRp4rmsrCxt4cKF2jnnnKMdeuihWn5+vthHFX5N3tZut4tt+fWnT5+umUwm7X//93/D70uFX4cf5+1OO+007dxzzxX7ZLVaxeMXXHBBxPb8/kePHi2e22uvvcTzixYt0u66667wNnwsYh333/3ud+Jx/j8W7733nnj+sMMO6/HcpZdeKp6zWCxi//g4zJo1S/x89dVXi+d+8IMfaMkgPw/+u/G4+eabxTYTJkzQvF6veMzn84njxI+bzWZxHM444wxxvhx00EFadna2eO6NN97o93FWP8/MzEzx3k4++WRt7NixWl5envbLX/4y5nnX2zH89a9/LZ4rKSnRzjrrLPG58a2hoUH79ttvxXM2m01bsGCBeJ5v/N74cT6HHnnkES0VzJ07N/w3Z8yYoZ166qnh74o8tnxMlixZEvF7W7dujXmOq591LHr7vb6Oqfwc5Hl5+umna5MnTxbH8YYbboj5e/H+XnNzc/iaVVxcLN43n1dlZWVabm5u3M+c2WeffcRzfL348Y9/LD7X66+/fsDXoPfff19ce/g5fl+8PZ+L/P74vcr95ffUHzo7O8PXXL7xe45my5Yt4lzg59esWdPna/J+8bZXXHFFj2tPrM9Q0tTUFD4O//jHPwb1OjLQ63Rf5ygT77PYvXu3Nm/ePPEcf38OPPBA8Tf5OPHjfO7yc263O6FzHgAAAADDDwjnAAAAwAgkWeGc2bFjh3b55ZdrkyZNEsIGC1YHHHCAEJRYMI0mUeGc+eSTT4SoxfvlcDjEa++xxx5CLGJRkoWbaNrb27U777xTiJm8Pf8e/z6LqP/+978jtmVh909/+pM2e/ZsIXaxIMXCzLJly+IKL08//bR20UUXCRGat5evf9xxx2mLFy/WgsFgj31atWqV+PulpaVCOI4WWIZCOOf9ePjhh7W9995bTGSwwMXv7cMPP9SefPJJ8XsspidDIoJXW1ubeJ+83UMPPRTx3Ouvvy7EpzFjxgghkfeJhV4WpZ599lmto6NjQMeZRcWbbrpJnIv8+iyg8nvctGlT3POut2PIwth1112nTZkyRZzb8v3zucHv8y9/+YsQ9adOnarl5OQIkY3PTxb1v/76ay1V8Pvj79sPf/hDIfCyiMziHk8+zJ8/X7wHnlSKRg/hnD+3e+65R3zufEz5sz3ppJO05cuXx/293v5efX299vOf/1yI1HyO8CTVeeedp23cuLHXa8327dvFZMyoUaPCkzGxXr8/1yD+vvN5Ls9bfq933HGHuB4OVDhnrrzyyvDn8/LLL/d4/re//a14jicHEoEnVHj7wsLCsBiciHDO8PuSxy56knSwhPP+XKcHIpzLawlfv4444gjxfeJzhK8nLJxze7d06dKI7SGcAwAAACMLE/+jt+sdAAAAAAAMPpyJ/Nhjj9E999wjYnYAAAAAAAAAACQGMs4BAAAAANIYzuWOLpLKmcsPP/ywKCzIufPnnHOObvsHAAAAAAAAAOmIVe8dAAAAAAAA/edPf/oTPf/886JgKhcsZBGdi+tt27ZNFAt84IEHaNSoUTjEAAAAAAAAAJAEEM4BAAAAANKYH/3oR9TW1kbLly+nFStWkN/vp7KyMvH4VVddRfvvv7/euwgAAAAAAAAAaQcyzgEAAAAAAAAAAAAAAAAABWScAwAAAAAAAAAAAAAAAAAKEM4BAAAAAAAAAAAAAAAAAAUI5wAAAAAAAAAAAAAAAACAAoRzAAAAAAAAAAAAAAAAAEABwjkAAAAAAAAAAAAAAAAAoADhHAAAAAAAAAAAAAAAAABQgHAOAAAAAAAAAAAAAAAAAChAOAcAAAAAAAAAAAAAAAAAFCCcAwAAAAAAAAAAAAAAAAAKEM4BAAAAAAAAAAAAAAAAAAUI5wAAAAAAAAAAAAAAAACAAoRzAAAAAAAAAAAAAAAAAEABwjkAAAAAAAAAAAAAAAAAoADhHAAAAAAAAAAAAAAAAABQgHAOAAAAAAAAAAAAAAAAAChAOAcAAAAAAAAAAAAAAAAAFCCcAwAAAAAAAAAAAAAAAAAKEM4BAAAAAAAAAAAAAAAAAAUI5wAAAAAAAAAAAAAAAACAAoRzAAAAAAAAAAAAAAAAAEABwjkAAAAAAAAAAAAAAAAAoADhHAAAAAAAAAAAAAAAAABQgHAOAAAAAAAAAAAAAAAAAChAOAcAAAAAAAAAAAAAAAAAFCCcAwAAAAAAAAAAAAAAAAAKEM4BAAAAAAAAAAAAAAAAAAUI5wAAAAAAAAAAAAAAAACAAoRzAAAAAAAAAAAAAAAAAEABwjkAAAAAAAAAAAAAAAAAoADhHAAAAAAAAAAAAAAAAABQgHAOAAAAAAAAAAAAAAAAAChAOAcAAAAAAAAAAAAAAAAAFCCcAwAAAAAAAAAAAAAAAAAKEM4BAAAAAAAAAAAAAAAAAAUI5wAAAAAAAAAAAAAAAACAAoRzAAAAAAAAAAAAAAAAAEABwjkAAAAAAAAAAAAAAAAAoADhHAAAAAAAAAAAAAAAAABQgHAOAAAAAAAAAAAAAAAAAChAOAcA9Mrhhx8ubgAAAABID9B2AwAAAMYEbTQA6QWEcwDSkFWrVtGZZ55JlZWVlJGRQWPGjKEf/OAHdN9991G643K56P7776eFCxfSqFGjKDc3l/baay968MEHKRAI6L17AAAAQL8Yzm23xOv10h/+8AeaPn26eI/l5eV0wgkn0K5du/TeNQAAAGDEttHLli2jRYsW0axZs8hisdCECRPibhsMBumuu+6iiRMnimMxZ84c+te//pXS/QXASJg0TdP03gkAQOJ8+umndMQRR9D48ePpwgsvpIqKCtq5cyd9/vnntHnzZtq0adOgD4IZu92eko9p9erVonE+6qijhHiel5dHS5cupcWLF9MFF1xATzzxREr2AwAAABgshnvbzfh8Pjr++OPFe73kkktEW97c3ExffPEF/e53v6M999wzZfsCAAAAJMpIaKN/8pOf0HPPPUd777037dixQ4jn27Zti7ntjTfeSH/84x9FW75gwQJ66aWX6LXXXhPi+dlnn52yfQbAKEA4ByDNYOfWV199RRs2bKCCgoKI5+rq6qisrGzQnN9ZWVmUahoaGqi2trbHAPunP/0pPfbYY7Rx40aaMmVKyvcLAAAA6C/Dve1m2J1200030ccff0z77ruvLvsAAAAAJMtIaKOrqqqotLSUbDYbnXjiicKsFks43717t3CaX3rppfT3v/9dPMZe28MOO4y2bt0qfodFdwBGEohqASDN4FlvFpWjG3UmVqP+9NNP0/z58ykzM5OKiorELDHPoEfnrPGyreXLl9Ohhx4qGvTf/OY3cTPOPR6PcI+xgO1wOGjcuHF03XXXicdV3nrrLTr44IPFvubk5NC0adPCrxuPkpKSmK600047Tfy/du3aXn8fAAAAMBrDve3mZd1//etfRVvNornf7xcCAQAAAGB0hnsbzYwePVqI5n3B7nJeQfbzn/88/JjJZKKf/exnInbts88+6/M1ABhuWPXeAQBAcnDuGjdYPEvMjXFv/P73v6ff/va3dNZZZ9HFF19M9fX1IqeNG+9vv/02onPQ2NhIxx13nGj4zzvvPJFLGm9wfPLJJwtHGc9Ez5gxQ2TC3XvvvWKWfsmSJWK7NWvWiNlsXqp92223iQ4AL3P75JNP+vWR19TUhIV1AAAAIJ0Y7m33999/L9xs/Hv8+hyrxkvRZ8+eLQR1XgIPAAAAGJHh3kYnA7+H7OxssQ8qciUZP8/CPQAjCs44BwCkD8uWLdMsFou4HXDAAdp1112nLV26VPN6vRHbbdu2TWzz+9//PuLxVatWaVarNeLxww47jGsdaA899FCPv8fP8U3y1FNPaWazWfvoo48ituPf5df45JNPxM/33nuv+Lm+vn7A79nj8WgzZ87UJk6cqPl8vgG/HgAAAJBKhnvb/eKLL4rfKy4u1qZOnao99thj4sb37Xa7tnLlyqReDwAAAEgVw72NjuaEE07QKisr4z43adKkHo93dHSIv33DDTcM6G8DkI4gqgWANIOre/OMOM9Kr1y5UmSKHnPMMaLy98svvxze7sUXXxSz1zwbzrnh8sbFTqZOnUrvvfdexOvyjPVFF13U599/4YUXxAz09OnTI173yCOPFM/L15Wz7bzci/djIFxxxRXCzcY5a1YrFsoAAABIL4Z72+10OsX/7e3t9M4774giZHx7++23RTYqv18AAADAiAz3NjoZ3G632O9oMjIyws8DMNKAcA5AGsLVrbnhbm5upi+//FJUvubB6plnnikEZoaLaPJglRtxLgSi3jgnnAudqHDHIJHK3vy6vEws+jX32GMP8bx83R/96Ed00EEHiSVsvCyNl6g9//zzSTfyf/rTn+jhhx+m22+/nY4//vikfhcAAAAwCsO57eacV4Z/l3NZJePHjxdLuj/99NN+HDEAAAAgNQznNjoZuD2PzlVnOjs7w88DMNKAdROANIYbYm7k+cYNK89o84w1FxbhBpQLebzxxhsxK19zMRGVRBtBfl3OLP3zn/8c83k5YObX+/DDD8UM+WuvvUZvvvkmPffcc2LmfNmyZQlV43788cfp+uuvp8suu4xuuummhPYPAAAAMDLDse3momNMrPxWLqzGmagAAACA0RmObXQyjBo1Srw+TxDwe5VUV1dHtPcAjCQgnAMwTNhnn30iGrXJkyeLBm/ixInh2erBgF+Xl7AdddRREY1pLMxms9iOb9wR+MMf/kD/+7//Kxrjo48+utff5SVoPJt++umn0/333z9o+w8AAAAYheHSdvOA32az0e7du3s8x0VD2TkHAAAApBPDpY1Ohnnz5tEjjzwiHPQzZ84MP/7FF1+EnwdgpIGoFgDSDDkDHM3rr78u/p82bZr4nwVnnnW+9dZbe2zPP3OV7/7AmW48MOb4lGg486yjo0Pcb2pq6vG8bGhjLf9S4Zl0XnrG1cmfeeYZ0UEAAAAA0pXh3nbn5uaKODWOZFm3bl34cR5482OcHwsAAAAYkeHeRifDKaecIibCH3jggYj39tBDD4nomQMPPHBQ/g4A6YSJK4TqvRMAgMSZNWsWuVwuOu2000QBEa/XKwalvEyLl3HxcmhZOOSPf/yjyGfjBu7UU08VA9utW7fS4sWL6dJLL6Vf//rXYrvDDz9cFCBZvXp1j7/HzzHvv/9+eCnZSSedJJaoyZy1QCAgBsqcsbZ06VIxO3/VVVcJAfyEE06gyspKkc3GDTDPovPfyc/Pj/n+tm/fTnPnzhXv6+6776a8vLyI5+fMmSNuAAAAQLow3NtuhjNg99tvP7G/v/jFL8Rjf/vb38jv94v3xwNuAAAAwGiMhDb6u+++Cxc6ffrpp6m2tpZ+9atfiZ957M1/X3LdddeJOmP8fjiyZsmSJSIahg1t55577qAeewDSAhbOAQDpwxtvvKH99Kc/1aZPn67l5ORodrtdmzJlinbllVdqtbW1Pbb/73//qx188MFadna2uPHvXX755dr69evD2xx22GHannvuGfPv8XN8U/F6vdqdd94pfsfhcGiFhYXa/PnztVtvvVVrbW0V27zzzjvaKaecoo0ePVrsI/9/zjnnaBs2bOj1/b333ns8mRf39rvf/a6fRw4AAADQh+HedkuWL1+uHX300WKfc3NzxWsl+rsAAACAHoyENvqxxx6LO76+8MILI7YNBALaH/7wB62yslL8Hd6np59+OuHjCcBwA45zAAAAAAAAAAAAAAAAAEABwcEAAAAAAAAAAAAAAAAAgAKEcwAAAAAAAAAAAAAAAABAAcI5AAAAAAAAAAAAAAAAAKAA4RwAAAAAAAAAADAQH374IZ100kk0evRoMplMtGTJkojnb7nlFpo+fTplZ2dTYWEhHX300fTFF19EbNPU1EQ//vGPKS8vjwoKCmjRokXkdDojtvnuu+/okEMOoYyMDBo3bhzdddddKXl/AAAAQDoA4RwAAAAAAAAAADAQHR0dNHfuXLr//vtjPr/HHnvQ3//+d1q1ahV9/PHHNGHCBFq4cCHV19eHt2HRfM2aNfTWW2/Rq6++KsT4Sy+9NPx8W1ub+J3Kykpavnw5/elPfxKC/D/+8Y+UvEcAAADA6Jg0TdP03gkAAAAAAAAAAAD0hB3nixcvplNPPTXu4WERPD8/n95++2066qijaO3atTRz5kz66quvaJ999hHbvPnmm3T88cfTrl27hJP9wQcfpP/93/+lmpoastvtYpsbbrhBuNvXrVuHjwIAAMCIxzpcj0AwGKSqqirKzc0VHQ0AAADAqPAcdnt7uxjEms0jdzEY2m4AAADpgpHabq/XK1ziLJyzS5357LPPRDyLFM0ZjnPhfeVIl9NOO01sc+ihh4ZFc+aYY46hO++8k5qbm0UETDQej0fc1LabI2GKi4sx7gYAADDs2u5hK5yzaM4ZbQAAAEC6sHPnTho7diyNVNB2AwAASDf0bLs5fuXss88ml8tFo0aNEpEsJSUl4jl2kZeVlUVsb7VaqaioSDwnt5k4cWLENuXl5eHnYgnnd9xxB916661D+K4AAAAA47Tdw1Y4Z6e5PBhcDEVv4s3YpwPpvO/pvv/Ydxx3nDcj4/vKy6t5sle2XSMVtN2DRzq3H+m+/9h3HHecN+lDurfdRxxxBK1YsYIaGhro4YcfprPOOku4yaMF88HkxhtvpGuuuSb8c2trK40fPx7j7hHefqT7/mPfcdxx3qQPzSluu4etcC7jWVg0N4JwHggEDLEfI23f033/se847jhvRtb3daRHi6HtHjzSuf1I9/3HvuO447xJH9K97c7OzqYpU6aI2/77709Tp06lRx99VIjbFRUVVFdXF7G93+8XsSr8HMP/19bWRmwjf5bbRONwOMQtGoy7R3b7ke77j33Hccd5kz4EUtx2j9wgVQAAAIbKGtuxY4coVoWa1QAAAIDxQdttPDhvXOaPH3DAAdTS0kLLly8PP//uu++Kbfbbb7/wNh9++CH5fL7wNhz3Mm3atLR1DgMAAIgP2u7kGbaOcwAAAOkDD/Iuv/xyMXBbsmQJZWRk6L1LAAAAAOgFtN1Di9PppE2bNoV/3rp1q4hl4YxyLsT5+9//nk4++WSRbc5RLffffz/t3r2bfvjDH4rtZ8yYQcceeyxdcskl9NBDD4k+1hVXXCEy0bkoGnPuueeKvPJFixbR9ddfT6tXr6a//vWvdO+99+LcBwCAYQja7uSBcA4AAMAQ8HIrr9er924AAAAAIEHQdg8dX3/9tcgwl8hc8QsvvFAI4evWraMnnnhCiOYspC9YsIA++ugj2nPPPcO/88wzzwix/KijjiKz2UxnnHEG/e1vfws/n5+fT8uWLRPmhfnz54vCojfffDNdeumlQ/jOAAAA6Ana7uSAcA4AAEB32GHOgzsu9AG3OQAAAGB80HYPLYcffniv8XUvvvhin6/B7vRnn322123mzJkjBHcAAADDH7TdyYOMcwAAAAAAAAAAAAAAAABAAcI5AAAAAAAAAAAAAAAAAKAA4RwAAIDucLb53XffTffddx9yzgEAAIA0AG03AAAAkF6g7U4eZJwDAADQnWAwSB988AH5fD5xHwAAAADGBm03AAAAkF6g7U4eCOcApCm+QJBsFiwaAcMDq9VKF198MTmdTnEfAAAAAMYGbTcAAADQEy7s7A9qhtRr0HYnD9QJANKQ9zZU0T8/W0eHTBlFlx40Q+/dAWBQGvBTTjmFmpubIZwDAAAY1MHrB5uqyeMP0A+mjyWzyYSjO0ig7QYAAAAiCWoa3bH0W9pQ30rXHDGH5o4tNtQhQtudPMab/gAA9MnStTvFDCYL6G6fH0cMAAAAACAG62pb6P99vJYe/3wDfbOzAccoxbR0tlB9Rz2OOwAAgBHBruYOWl3dTF5/kJat26n37oBBAMI5AGnonKppc4V/rm1z67o/AAzWeV1XV0f19fXiPgAAADAYbGloD9+vbu3uP4HUtN3tnnaq66jD4QYAADAiaOjo1me2NTrJaGDcnTyIagEgzWhxe8nj7y6eWNvupgnFubruEwADxePx0KJFi0Rx0CVLllBGRgYOKgAAgAHT6OoM3/cGAjiiKW67nV4nuXwu8ga8ZLfYcfwBAAAMaxqcnvD9JpeHWt1eys80TvuHcXfywHEOQJqhus2ZunY4zsHwwOFwkN1unE7FcOGOO+6gBQsWUG5uLpWVldGpp55K69evj9ims7OTLr/8ciouLqacnBw644wzqLa2NmKbHTt20AknnEBZWVnida699lry+yOjot5//33ae++9xWc5ZcoUevzxx1PyHgEAIB6NHYpwrhgPwNC33UEtSG6/mzwBD3X6uz8HAAAAYCT0O5htjd0r34wCxt3JAeEcgDSDHeaRP2PZMUh/2KX2n//8h5544gm4zQeZDz74QIjin3/+Ob311lvCGbhw4ULq6OgIb3P11VfTK6+8Qi+88ILYvqqqik4//fTw84FAQIjmXq+XPv30U/E5sSh+8803h7fZunWr2OaII46gFStW0FVXXUUXX3wxLV26dLDfEgAAJExTR7fzC47z1Lbd7DL3BXzif7cPRg8AAAAjTzjf2mQs4Rzj7uRBVAsAaUZ0PmddOxw8AID4vPnmmxE/s+DNjvHly5fToYceSq2trfToo4/Ss88+S0ceeaTY5rHHHqMZM2YIsX3//fenZcuW0ffff09vv/02lZeX07x58+j222+n66+/nm655RbhNnzooYdo4sSJdM8994jX4N//+OOP6d5776VjjjkGHxEAQBfgONcPIZwHfSKihSNbyqlcx70BAAAAhp6GNHCcg+SA4xyANKMmymEOxzkAIBlYKGeKiorE/yygswv96KOPDm8zffp0Gj9+PH322WfiZ/5/9uzZQjSXsBje1tZGa9asCW+jvobcRr5GvIw9fg31BgAAg4U/GKQWl7f7muNHxnmqhXMuQpZpzaQ2TxuKfwMAABj2NDghnA834DgHIM2obYtc6lrv7BQDQ6sZ82AgfWHhlh3LLpeLrrnmGrLZbHrv0rAkGAyKCJWDDjqIZs2aJR6rqakRjvGCgoKIbVkk5+fkNqpoLp+Xz/W2DYvhbrebMjMzY+av33rrrT0eb25uFvEwepPOQn4673u67z/23TjHvbHDQ75Ady2G9g6XuL4YkXQ8b7jt5hVKXCfjZz/7WY+2u8nVRP4OP2kOjZx+J9Vaa8lhddBwOe7p+JkBAAAYOoKaJgqCRkfturx+yrIbQ37FuDt5jPHJAQASgl070cVBNY2o0dlJ5XlZOIogbWGRlONAuCH/5S9/CeF8iOCs89WrV4sIFSNw4403iokSVYQYN24cFRYWUl5eHhkB3pd0JZ33Pd33H/tujONe620hq6V7uGOy2g392Rh532LBgjm3J9x2c3RXdM55Q7CBMoIZlJuZS+4ONzlyHFSYWThsjrvFYhn0fQEAAJC+NLs8Qp+JZntTO82oMEb7h3F38kA4ByCNaHF7yeMP9nicZzEhnIN0xmq10vnnn09Op1PcB4PPFVdcQa+++ip9+OGHNHbs2PDjFRUVouhnS0tLhOu8trZWPCe3+fLLLyNej5+Xz8n/5WPqNiyAx3Kby4rufAMAgFQU6EJx0NS13Wz2cPlcZLPYyGQyicc6/ajLA0B/4O/TC99uIafHR+fsM4UybegrA2D0fkemzUJuX2gF7dZG4wjnGHcnD7IdAEgjVLd5hq3b5VLXHhnfAkC6wQ34WWedRaeddhqE8yEYbLFovnjxYnr33XdFAU+V+fPnC4f/O++8E35s/fr1tGPHDjrggAPEz/z/qlWrqK6uLrzNW2+9JUTxmTNnhrdRX0NuI18DAABSTVNH5HJpZJynru3moqCccW4zh+JbWEBv6WwZ5D0AYGSwurqZFq/cRm+t202fbok0KQAAjJlvPm9scfg+C+dGAePu5IFwDkAawc5yyaxRhTEfB4NLIKjRzmYnClqBtI5nefrpp+nZZ5+l3NxckUXON84dZ/Lz82nRokUiMuW9994TxUIvuugiIXjvv//+YpuFCxcKgZydhStXrqSlS5fSTTfdJF5bOsYvu+wy2rJlC1133XW0bt06euCBB+j555+nq6++Wtf3DwAYufRwnMdYtQeGBhbNfQGfEMwZh8UhHOj+YHfmPAAgMXY0OeNe1wAAxoFrq0jmjikmc2jBFW1rMo5wDoZYOOciXgsWLBAD77KyMjr11FOFKy2azz77jI488kjKzs4WbrRDDz00PEBnmpqa6Mc//rF4jpeF84Cdl/ipfPfdd3TIIYeIrDzOO73rrrv68fYAGL6Oc74QSyCcDx13vb2CrlvyBT3z1aYh/CuAXdGtra0i45rvg8HjwQcfFMf28MMPp1GjRoVvzz33XHibe++9l0488UQ644wzRJvNsSsvvvhiRI4rx7zw/yyon3feeXTBBRfQbbfdFt6GneyvvfaacJnPnTuX7rnnHnrkkUfomGOOwccJANAFRLXo13azcB7QAmQ1h5zoXBTU4/cgrgWAflDb3j0G7PTrXzwdABCbBmVia1R+Fo0rzBH3d7d0GGbVG8bdyZNUONYHH3wg3GUsnvv9fvrNb34jXGjff/+9EMmlaH7ssceKgl/33XefWAbA7jSzuVujZ9G8urpaDK65mAw72y699FLhhmO488Wve/TRR9NDDz0klof/9Kc/FSI7bwfASKVaEc5njioUM5hBjai2DY7zocAXCNJ3u5vE/W92NdB5+04dkr8DiDwejxBjuU1YsmRJjwJjoP8kMhHBx/v+++8Xt3hUVlbS66+/3uvrsDj/7bff9ms/AQBgKJ1fTKw6MWBo2m4WzlVYQGchnXPOc+whIQEAkBh17d1inNuLVRsApENUS3F2Bk0ozqXtTbx6nWhHs5OmluaT3mDcPcTC+Ztvvhnx8+OPPy6c57ysmx1qDC/J/sUvfkE33HBDeLtp06aF769du1a8zldffUX77LOPeIwF9uOPP57uvvtuGj16ND3zzDOiUNk///lPstvttOeee9KKFSvoz3/+M4RzMKKRAjkL5uW5WVSamykeYxcCi2Oy+BIYHFrd3YO+Vnfk4BsAAAAA6RbVYgy310jA7XOT2dRzcTPHtQAAkgOOcwDSg6aufgfLMoVZDppQlEsfULV4bHtjuyGE80TgqDWe7M6wwsw24IxzXprHFBUVif+5aNgXX3whxPQDDzyQysvL6bDDDqOPP/44/DvsSGfnuBTNGXaWsyOdf1duw0I8i+YSXurNsTDNzc1xZ03Yqa7eABhOsDAuo1rKcjPJYjZReW5m2EHV1unTeQ+HH82KWO7yBoQDHQwN7FJ75ZVX6F//+hfc5gCAXkGcE0gEbrOj+0ZetOMpa7s7vB3hfHMJx7W0dobGjwCAxAhqGtUrLla3FxOAABg9qqUoy0Fmk4kmFucarkBoIuPuelc9bWneQkEN+kfSjnOVYDBIV111FR100EE0a9Ys8RgXBWNuueUW4R6fN28ePfnkk3TUUUfR6tWraerUqaIgGQvrKhznwuI7P8fw/5yVqsIivHyusLC7KKKav37rrbf2eJyF9kBA/8YlnYX8dN73dN9/dd9bO33U4Qk5oAvsZnFu51mJ/IHQcr2Nu2tosnJh1pvhcNx31TWHjy+zvbqOirNDhRCNzHA49iNt39P5fQOQSv72/ipaubuJrjxsT5o3tgQHH8SlyeWJWfCbb2w+AEMHFwD1BDxkM0cK5+xcc/vdIuucRXQAQGIrZ/i6Jen0I6oFACPCGeZOT+j7WZITEqTHF+UQ9zi0NCsQypPcje5GaulsoaLMkFF6JNNv4ZyzzlkMV93kLKYz//M//yNyy5m99tqL3nnnHRG7wuL2UMGZ6tdcc02ECMFFRVlk5yKkRiCW4J8upPO+p/v+y32vqWkmqyX0la0sLRSPV5a1kXV7KIPbTTbDvU+j7U+y+x6o7Qgfc0ZzZFJhYXosr0r3Yz/S9p2LbgIA+hZCP9taJ+6/9N12COeg9/MlKqZF4g0EKLOrYCUYGjjfnJd55zgis8ztFju1elpFzjmEcwASI7qWVadPf1MgAKD3eLiS7JBwnmmzUkV+FlW3umhHk5P8wSBZlfqPRp385lg1bstrnDVUkFEQM3ptJNGvd3/FFVfQq6++Su+99x6NHTs2/PioUaPE/zNnzozYfsaMGbRjxw5xv6KiQkS6qHCh0aamJvGc3Ka2tjZiG/mz3CYah8MhBHL1BsBwora9u9NUkZcl/pdRLdHPg8GhWck4j848Hw6s2NVAL3y7hdoNEPPDhcUefvhheuKJJ8R9AACIxuXpvjZsqGshtw+uO5BYgS4VLwqEDnnbLYTzoK+H41wMvDV2zMb+bAAAPalzRo7x0PYBYPx+R1GXcM7IuBZ/UKPdLR1k9HE3t9G8Mqwsq4ya3E3CdT7SMSebKcmi+eLFi+ndd9/tEacyYcIEUdyTs8hVNmzYQJWVleL+AQccQC0tLaKgqIRfi93q++23X3ibDz/8MOJDfOutt0SR0XR2IgIwEGS+OVORFxLMuUCopA7C+aATXRC0ZRgJ506Pj/787nf04oqt9PKqbXrvjojUevnll0XxaCPEawEw3PNS07Fmg0fZZ161vr4WHXkQn0YlqsWqRLPwUmowtG03C+dMrKL1FrMFBUIBSAI4zgFIDxo7PD0c5wwXCDVSznlf424WzrkwKK8M4wnvGmfNiM86tyYbz/Lss8/SSy+9RLm5ueFM8vz8fMrMzBSdo2uvvZZ+97vf0dy5c0XGOc9irFu3jv7zn/+E3efHHnssXXLJJfTQQw8JcZzF+LPPPluI7sy5554r8soXLVpE119/vYiE+etf/0r33nvv4JwpAKQh1RHCeVa4SKgEjvPBp9kVKZS3xMhLTVd4osUXCOUlGmHmm2tdnHXWWdTR0SHuAwCGBpfXT795+UvhWLvl+H1oVH73BKzR6YxymK+qakJcC0hoyTT3m3Z1tXVeCOdD3nazUy0ePAiH4xyAxIHjHID0KgzKlOR01/GYoNSh226AnPO+xt1un5tMIpmdRExLU5frfCRnnSelTjz44IPi/8MPPzzi8ccee4x+8pOfiPtcMLSzs5OuvvpqEb/CAjq7xSdPnhze/plnnhFiORcNNZvNdMYZZ9Df/va38PMsxC9btkwI9fPnz6eSkhK6+eab6dJLLx3o+wUg7d0GbN6RxSYybBbKz7SLCBEI54NPS5TjfDhFtbR1dr+XdiX+QC+40T7//PNF0VsI5wAMHd/ubAi3F19ur6NT5kxIm8Md7RReXdWs276A9Mo455V6UjhXVy6AoWm7nV4n2Sy22L9jtooYFwBA/xznbHxBkWMAjD1hX6w4zmVagFFWsPc17ubCoHarPbSt2Rp2nY/krHNrslEtiXDDDTeIWzyKioqEc7035syZQx999FEyuwfAsIW/ezKqhV3makEJ/pkFXb5xsRgW08Hg0BLtOI8S0tMZNdfcCBnnAIDUO9fYfZ5ORBdE29HsFG0fTyADEG/JNKe0lCor9OA4H1qCWpDcfrcoBBoLHnRz4THebqQOwAFIhljmKF41luOIPTkFANCHRmds4dxhtaRNH4Sj1rgNd1i6HfP5jnzhOmdBvTBzZEZno7cCQBrAwoCnq5hVhZJrHl0gFDnng5sBHC2UG2GGeLBQXeaq+1zPySFercS3RCdpAQADK1zUafDOezSyHVRZU92ky76A9HF+cYGujIhBKxznQ9l2i8KggZ6FQSUWk4UCwYC4gd7hml8nnXSSiDPlSNQlS5aEn+O4U440nT17NmVnZ4ttLrjgAqqqqop4DV4B/uMf/5jy8vKooKBARKE6nc6Ibb777js65JBDKCMjg8aNG0d33XUXPhoD1SSKNckdPZEMADBOVAsbGbPt1pjCeay+rJHG3aIwaMAj8s0lNotNRLeweD5SgXAOQNrlm3cL5T2E86iq62BgHVUuPjcSolrcvoDuhQI9Hg/98Ic/pIsuukjcBwAMDfWKcO5ON8d5DKGfc84BiBXr4/SEzu/ibAfZVeEcBaiHtO0WwnnQFzeqhYuDstucC4+B3uH8WY49vf/++3s853K56JtvvqHf/va34v8XX3yR1q9fTyeffHLEdiyar1mzRkSnvvrqq0KMV+NP29raaOHChVRZWUnLly+nP/3pT3TLLbfQP/7xD3w8BiBeFGd0zQ8AgL6wAC0n7LkwqFoc22YxG6pAeW/jbhbO+b1ErwizWWwi+3ykggpsAKRZp6m8qzBo+Ge1QGhUBh4YvJgW8ZjbIxoStSFMV9qi4ll4oqAwq3tmGQAwPKlXJljTzXEeSyhYXd08bK7LYPBo6oppkculHcqgFY7zoYULg8YadEv4cTjOE+O4444Tt1hwTTAWw1X+/ve/07777ks7duyg8ePH09q1a+nNN9+kr776ivbZZx+xzX333UfHH3883X333cKlzrXHvF4v/fOf/yS73U577rknrVixgv785z+jvpgBUFcTcyunKaYXAIBx4OhTrj8QHdPCmE0msllM4nmj90E6vB0x22+r2UreoFe03zwBPtKAcA5AmnWaejjOlZ9r27ud6WBgxMoz58aOO6pZytKrdKU9Kp6FHeh6CucOh4NeeOEFUaSE7wMAhtYNk46Oc3WwYbeYyRsIiugZbiOjJ5XByEYul2aKsqId58YetKYTsdpudqv1hohqoYDIOQeDS2trq5hE5EgW5rPPPhP3pWjOHH300WQ2m+mLL76g0047TWxz6KGHCtFccswxx9Cdd94pPtfCwp55tuxQVF2K7FoHQ2+eqsjPoupWVzjjHABgzH4Hr3SLhvshvoCfPAZY9RZv3M3jhDZPW0RMi8RmtpHL5xJtN4RzAIAh6VDEjbyMyGJL5UrmebzlfCB54uWZc1zLcBDOox3nehcI5YEeZ2vyLZ5ztN3TLpaAF2cVp3z/ABgO8PVLumHS0nHu724LZ40uom92Noj7q6qbIZyDCNQJInZ+8USLkZZJDxditd3t3va4hUHl77BtFlEtgwtn1XLm+TnnnCPyzJmamhoqKyuL2M5qtVJRUZF4Tm4zceLEiG3Ky8vDz8USzu+44w669dZbezzOIkzAAKJQOgv50fu+va6J/IFQ21eeZaWdTaH79c0t1JxlvNTd4XTs0wnsu/7HfltN93c10xQU10MVczAonu9we3o8pxe82qilpSX8M2ebt7W2UYY1g9zeSF0pEAyQ2+OmBnsDZdmy0vqc78/vpr/6A8AIQC0Kk2WL/NrmZdjIYTWLQhMQzgePFle3k6YkJyNcUI+d6KPy9W8sBkp0QVC1WKhR4YIk9a56yrRlGqLBBiDdqFfExHR0rKnF0BZUlnYL51WNdPS0MTruGTAaTUobzs4vtWaJF8L5kMFONM5A7U04l6A46ODBhULPOuss4RZ88MEHaai58cYb6ZprrokQIbioKIvsUrTXm1iCf7qg7nubfytZLaGx3x6jSmhFdUjwsToyDfsejbpfiYB9x3Hv73njqXaGv6vjy4p6nEvZmQ5q9wUpaDIb6jxT96Wls4WogygvOy+mka3D2UFZuVlUmGmM/e/vcbRYko+aMd40JQCgBy5vt6gZ7Xbmi5p0nde3uykYVRkZ9I9mxXFeWZTTpxM93YgWyqOF9FTj9/vpqaeeoueee07cj4YHg9yY82132+4eFcABAH3DbYSK26u/MzAZVKfw9PKCcHu4proZbd8QEghq9Na6XXTNi5/Rn9/9Li2OdVOU45wNBhJEtQxd28355rwyLCHhHMVBB1U03759u8g8V4XriooKqqur6/GZNTU1iefkNrW1tRHbyJ/lNtHw0n7+O+oNDA21baFolmyHVRQcTNcVYwAMdxq7THaM+l2VOLoi44wweR9v3C2j1nqrG+QLGt9sNxRAOAcgDXAp4kasmJCirhwtdlQ5dY7cGC60KhnnE4pyh5VwziJIh8dvqKgWbrSff/55WrJkibi/q6WDrl/yBf3t/VXkDwZFQ863oswiqnHWUKO7Udf9BSAdaVQKJkZHn6SbcJ5ps9Keo0JOE76ebW9q13HPhi9rqpvoxpe/oH9+tl5k6361vZ42Nxh/OXuDM6o4qJJxjqiWoWu7eZk3u85tFlufv+sLoL86WKL5xo0b6e2336bi4sgouwMOOEAsw1++fHn4sXfffZeCwSDtt99+4W0+/PBD8VoSFuCnTZtmKFfkSMQXCIYLHZfnZlKGrfs6huKgAKRfxjnjD2q6GxCi2241FpWLgPaGb4S23YhqASCNolq4GrNNyemU5Cu55+wczsvs2+kDekcVyCcU58YU1NMVZ4xYFr2jWnjJ1Mknn0wdHR3i/jvrd9COZqe47TW2hGaNyRAutoKMAtFg72zdSbn23JjFSwAAsalzRjnOfQGxeqM3Z4mRUIUCFhBmjSoSQi6zuqqZJhbD9ThY8Hnx/z5eSx9squ7xnBBySsnQNLpCA1ir2US5GTayd6A4aCrabk9nYn0kHphzmw56x+l00qZNm8I/b926lVasWCEyykeNGkVnnnkmffPNN/Tqq6+KbHGZW87Pc7HPGTNm0LHHHkuXXHIJPfTQQ0Icv+KKK+jss8+m0aNHi23PPfdckVe+aNEikZG+evVq+utf/0r33nsvPh6d4ZhIKa+FhPNu6aYzzaLWABju1Hf1sblHXRTLcR5Va4UNIEZpu5mgFhQ1SnobW1vNVhHHNhKBcA5AGtDRFdUS7wLLg8J4RR9B/2juykdlcYY7q8PJcd4eI5Yl1mOpxGaziYEdF0vh+3VKpMQb3++kiaWhAR4LfCye13bUUrWzmiYUTNBxrwFIL2StBgkbXji2QnXjGhnpFOZBCRd7nDW62w3JcS0nza7Uce+GFzubOyJEc44JkCuVuNaH0ZEuTV6RZzaZUBw0RW23s83Zp1uNMZvMEM4T4Ouvv6Yjjjgi/LPMFb/wwgvplltuoZdffln8PG/evIjfe++99+jwww8X95955hkhlh911FFkNpvpjDPOoL/97W/hbfPz82nZsmV0+eWX0/z586mkpIRuvvlmuvTSSxM9DcAQUdseimlhynIzKROOczDCVlw8+NEaUcftpwdME6vHjAo7yHc1d4j7ZXmZMY2O0nHOeP1Byux7YVbK2m6GV3Z7/V7KcXRH1MYUzv0QzgEABkUWcIsV08LkKY7zVp0F0OGCFMgLMu2Urzj4W1zpf3xjTa7oHdXSWz7t1sZ2WlVdR0U59gjxvKqtSvzPNwBA8sK5LLiZbsK5w2YR14FReVli4pivX5saWtPKPW90ahTB5qhpo2mf8aV051sr06Id5MG2XKlXmOWIMWDVP190OMLfP6fXmVC+ucVsgXCeACx+91bTJZF6L+w+f/bZZ3vdZs6cOfTRRx8lsksghdQqJhKuZwXHORhJfLW9jj7bGqrR0NjRSb87fr6uLu3eqG1zh+unVBbGFp7VWitGjIzjVWCcX95bG241W8U2XNyb2/GRBDLOATA4PIMpC7jFE85VYVfvIo/DAW7MWEySg+4ch43MpuET1RLrHNE7qiWaxi7HP6NpAcpYfRdV1C8jW8dW8ViGNYOCmpfa6r7ScS8BSB9YYJHLSGNNzKYD8rrMbnOGRfIpJaF4FnZD17SNTBfMUE+yTCsrCAvQ6eA4V9s4aSyQ5wyD4qBDA+ebJ1oYlB3nXByUl4YDAPoWzstyMyIc57I9BGC4sru1ewJ/e5OT/v7BGt2zwePB0aKScXGEc6NP4HN9EpNY0xkfm8UmIlNHYoFQCOcGLNr32uod9OmWUEYdANwxkk1Etj32mp48RLUMKq2K+5od57zMuyDTMXyiWjzGc5x3dnbSSSedROeccw61OTsi9qfYVENj3V9T8c5/U8GOp0MPahoVOddT4drb9NtpANLse8/LXaNJp8F3Z9dAQy2QNqUsP3x/U30rpVsk2OKVW2lro/GKbapxWaW5meE2kGk1eDsYKZyH+k2OqCXSYPDb7lZnK3mDiQnnFpNFONZ4oA4A6Ps6zI5z1W2L4qBgJJ3/zDc7G+jprzZSugrnEUXKu9zpRmi7+T7D7bEWVp3it93+oH9EFgiFcG4wXl+zQ1wQ7vtgDe1uCeUkgZGNXG7MZNotfUa1tBl8QJt2wnmXy066+jkKx6iz3QNxnPNjiSz5TQWNXdm0DKcuTDKtE/c7PAHqzJ8t7mfXv0NFzZ+T2dtIQR+ulQD0J6Yl3Rzn0qGTYe0WD6TjnNnUYDwBOh7cVv/uta/p+W+20D3vfGe4dqVeOV9KczJEJI5MwWk2eD+jzd3dhuc6Qm23mjfqDaTPZFG6Oc55zJ1IXBIv8Wa3OYvnAIDeHecWs0nUa1AnjdOp7QZgoCsu5MrvN9bspLfW7TLcAd2pCOfj4znO1X6IAR3nvGKsr/bb0tV2j8RJb2OGBI1gPtzcXYipps1FYwqydd0fYCzhPCtOrheiWoZOOC/sctnJY8zahrPTR3lKPE46Z5xzJ5wdp/6gJtycemXHORwOevrpp6mlpYXa/N0C0kGTKmjcVimcBymQExLOM1tWUkbzV0Ls97lryWGbpMt+A5AuqEKo3WoOu27TZfDNK/J8AS2ccS6ZUpovFpZqaeQ45wHTn95ZGf5MeLKQr8Px4tj0QMb6sGDDMS2hlVd2anZ5DR9ZFstxbjWbhPDPbXislRdg4G130BxMuMaAdK1xXAsAIHabV9vmCk9e8jVYbb/TabUYAP1Bnv/c9zhzr0n0yKeh8eBjn68XEXLji+IXsUw1O5qc4e9neV5m345znYVzte3m+4zb506ouDeDqBagK7uaneFqvOk0mAVDi0s5DxIpDhqr8CNIDvUYSsE8okCowd12fdGuiAqj87OUx/U7d3iwnZ+fT3l5eRH55hOLMml2xmYhjDm1TPqisUg8HrDlhcWygKdet/0GIF1Q883HKpPy6bLcWx1kZCiDD24XRxeErmPbGtsN6eJRYWf5Ax99T5vqI93xRurzhfLwQ6J+iSLY5CuRZUZzyMeLI5OT3NzGyEGrD47zIWm7ey0MylnmSp45HOcA9O1glZPFE4tze7R/RmozABhs+PyW43EWoo+aNoaO23Oc+Jm7H9/uajDMQedJLOmOH1eQE+4z9S6cBw3TdssJb141xpPaieD1p7cW0h8Q1WIgPt8WqhoswUwyYFze7gFgvIxzXoIsl+9xlAgYGOox5Flu1XmeDoXR+kIVyMfkZxuusGyTEtVSpm2iAjv/bKKNgT3o0y3t4vGAraDLPahRwGOczhMA6RDVouYvpotwLvPNGYc1svs6pSSUcx7UiLY1ha4RRuXfyzfTF1H9PaP1+Tq8/vD+lGZn9GgPedCqd12MZIuDqsuk9R6wDkf8AT91Bjp7COfZ9e9TxXfX0rgvzyVH+/oevwfHOQCx2drY3ZZNLO6OJMvsMlGpbWI60NjRSct31JNP52xnkI75/iEH9/4Tygw5Fq9S9nVcYfy0CHajS4xm8pA1RxJxnNssNnL7I/PnRwIQzg1E9EAqXQazIJUZ5/EvZvldg0OjiJ/DJqolKuNcPJ/mjnM5g89L18u6OiPxioamCr/fT88//zwtXryY6lq7c+JKPCuJ+xk2i4k2BvegmjaPcDoGbd0FAbXORp32GoD0jGpR8xc708S15lH6RBlRkVKRBUKNm3P+8eZqemXVdnGfJ/4qlWXGRhJB1NUJXBg0Wjg32qC1t1VjuQ5bD7eX0Qas6Yxsu/+7+L/k6nT1EM7duxrJv2k1BdpdZHPtiHjORKYRmZMKQCJsUWp2TClVhHPpOPf6DVObKJEYi18v/pzufuc7emV1qA0EINF8czlWlWNypklZnaw3u1tDkTLM+KLu1SFGLg6qjrv5PrfF0cJ5vOuLxWSBcA50jmmJKgaKJViAcXm7B3hZSq5rNFy4i+nw8IUPs/mDHdUSKRikt3De7vGGzxl53ojHdXQQcqP91FNPiUa8vr27A5Ln/IZH1yJnd6NnKmmcxe4LUkARzoNwnAOQsOOcv0sVeVmGdDonGtXS03HeLSpsNHDO+Rvf7wzf/8l+02jumOLwzyyCGIW69sjCoJICZeWVkSeQ1X1T65FIt5cXjsdBb7tfeP4F8vq8EYPunNpllLXqEXK3usm5y9lDONdIQ3FQAOKwpTEknHOIQqUixkkTFa+w4vpERofHFne/szLc11hf26L3LoE0dZyrfRCut2IUdrd27+v43hznBioOqo67w8J5wC9i1Jh3NmylC/79ED399coev2s1W8kX8I249ts4VYhGONExLQyEc8B0qFEtinMqGnU5MndS1FlZ0D/HOQtM0q0WmXFunFnu3uBG+dtdjbRHWX74fODZYzkxwOeM6sbTc7WCxWKhhQsXksvlolp3aP8yLT5yONeIYYOzOZdK7qwn+ywvuRZWUkGXcM6uzaCnSbf9BiBdkC7ikuwMylZWL6VLX0PdzwxrZPeVo2dkwTSjOs650Btn1jIVeZm0cMZYWrxyqyEd52qsT4RwntXdDjYbyO0Vjbp6Sm3j7BZLeBKG28JEC1mCvtvu2uZacV/F1riKPP6O0CoxCpKlfhORUsebj/9ILDAGQF9wnMn2rmKDo/KzImpcqTU+uF20xasrYJB272/vr4pY8YZaXCBZx3l5blY4mjbHYSWnx08tBuqD7FIc52oUopGLg6rjbr7P+eYcnSYnv+/74u9U7VpH27/9iM6c+2TESk+Oaunwdoj2WwrtIwFEtRiEWHmXiGoB0VEtWVHL01VUYRdxLYMjnLPLXA6s08Vpp/LUVxvpL++toj8s/Ta83IqvK9yRlYJCbtSEi17YbDa68sor6dJLL6UWT+ic39OxjbSgj0wmM1V9Vy4ey1vdSjtf20VBW0HXb5oo2Nnz+gkA6KbD4wv3KbjYo6yJkbaO86jVVzzJObnLdc6ir5EGVJKaNle40Nv4roiWTKVNN5LjPCKqJUeNakmPdlD2gbLsFjHQjh60amni1EwHZNt9xvlnUGZG97nCBGrXib6HmWuRWP1E/s2hg68s9x6JBcYA6AueZJV99UnKiiomndrvZ7/eSKurmyMea0ekKEiA2raeUS2MNILx5L0Roop4H6TjnHUD1cgYjV0RztnooSfquJvvq5PYTa52IZoznmArfbFzbU/HedAnXOcjCQjnBotpUZ09RhpEAWO47FTHQTR5SuQGZvP7D3dUpVtNnYxQnXbpEtWysS4UWcDXF7nPaoeVY1rU80bPjHMJi3tS4OvMmkjbx11IbRkH0s5NE8LbuOo7w1EtnJFK3mZDdJ4AMCqq24uF8wjBNk0c56pwrjruJFI4ZzYp2bBGQbrNmcrC3J4CSJplnDcbuB2U7Zw6MczYDLRMejjBy7U7/VGFQYNeorZtZA1oZPNrlBtwkc/TSGZfd5SS2WQmL28HAIhgi1IYdFJxZGZyRpq032wKfH1NKJ7MbOoewxq5sDQwXlQL95PUsaoUznnymwuZ6w1nrXd0fQ+lKSIeasygJ2CsPohab+Tj7d+G7nSNrT/d8XXEttx287h7pK0Yg3BusJiWw/cYHb4Pxzno4TjvVThXHOcGHtAandZOL2kx3HUs1Mh8VCM77VScihDObsfoSRUR1aKeNwZwgahijCOriJqKDqK19efS7l2TxWPOKTmUc9p40swZpJl5RQCRyd+OAmMA9EJDR2T0hirYpktfQ3XWqctdJVNK1QKhxss536EI5+O6MjDVCQwjOQfruiZauIC0KpYXKBFwrQaNLOOIA1kbJtr5pQ5akXM+ePASb85GVYVzu2snBd0+Mne5Zh1BH3k9HWRXcs7ZtQbHOQA92ay0YenqOH9vQ1X4/kX7TwsXw+Zrr94xFcMdXnm3tiZ9TUVsYpMT+JxvrsaqqVG0RoiMU00R43uJaVHj4ow4ec/u8fBxDhTRORtL6dVHdtOln7bQytouIT3G74wkIJzrDF/QPt9WK+7zqXrYlFFpMYsMdCoO2otwnm8wATRdUcWAQsVlzo2JPMbNBhUMolFn4mVWnCwMGi4OquS/6ukC6ezspDPPPJMuv3QRBXyhfczPDJ3vzaubKaD5qDPQTDvmu0LfCZOJqub+lbbu8zhtmnYLhHMAeqG+PTJ6w6iCbW+ojmxVOJBMKVULhLYZWjgf31XoLTNiAsNvmH6pzDjn1QlmZcCq9jOMuvJKXTmlutR6LpNOj/Pe6HDbfc6PzqGbfnkTBb3dS89tHVvJ5jGLqLX2hhyymAKkdXaS1r4pwrXG2ecjrcAYAH2xtctxzpffCVGO83RZMSbHG9yCHDVtjGFqKg132DR1w8tf0G1vfEPL1u2idKSxo1MUv42OaWEKIwqE6j8e39kcSo1IRDiPzDjXN6pFjrsvvPBCcd/td4fzzTV/GV3zXj3l+Kx09rft1F63iVo7expSvIGR9T2GcK4zHKGwuyXkBJ1WXkDF2RlhR0y6DGbB0OJSOkVqZykaFkEliGrpP2qVbtVxLn7umuXu8PiFq83I8P6p15B4jnNeui7FG72jWjweD7nc3c7YvAwedJuoZXULtbnrye5ppbXWJ8jZ1eEOZJST2ZpLPs0/4paLgdTDE0ubG9rS0sFTH+U45+8954IbfeCtouZBxnKcc/+pqOsavaWhTRQkNBI7ugq9segvY/mM6Pzv8AbCbYeaby73V+6zEQassVBX3EU7zu1KVIveg9bhhNfjJa83cgBtqttImi8oRLPa70dRQ2Ym+b0Z1FzdveSbi4rx8nAuSAYA6Po+BYLhidaxBdk92js1qszIWoFs0zLtFtGXV6/HiGsZOlZVNYlxKvPO+t2U/oVBI/shanQqx6Tozfam9oQKgzJy5bpRJu953C3bbo/fI+qOMK27G8T/VlOorzp3m4uWV0e6zrlAKIvtI4n4KhxICct3hk5MZr8JZWFx1OP3psVglgWEgKaR1Yw5mKHC5Q0JgjxYlUJHLNQOCceNgP7RorjJ1WXp4ueoAqycv/bCt1tEztqvj5xDecrzRigGqFLTVWRF7axKNx5PunAHV08HiMPhoEcffZT+/cX39Gm9n0abdlBFoJZsDXZq29FM+S0uqvDX0MFb6mlj81o6KlBC2/+zndo3t1P5L8rhOAdDCndwr3/pczGxxkuOF84Ym1ZHXDqIpYtYtik8uDLywFtF7ROpkRvRrvMvt9eL97S7paPPQUwq913mzLMQIl3cRnQONioDUbXujtoO1vjcho0sU9uxXh3nBssXTVe47f77Q3+nldtXkt2hrNKr2UgUDE1ObPxuBo07ajNxj6q1dn2k4zzY5TjvORcGwIhkV4srXEM3OqaFyVRWHxupNkY0slabbOciDV7GbD+GAxzRorqhq1o7aHR+KB5uOAjnxotqCTnOuVs3piA7bRznctzd0tJCNnuoOChPZrO2Z123UWxjNWWSl9pp3x2d9Mn2r+jIiYeHf5/d6W7fyBLOoXbqTINSgGlaeSifU7p5jDKIigcPmq5+8TO6/LmPxUUZDG3GeVYvbvPoQpbokPQfdfm5KpRHH+MHP/peLINbW9MiinB+vKWGjJpvHuk4V6Jauga58n+Xxy9y5fSA3ShlZWUUyMgV9w81L6W9dv8fTV//GzLb6yjbE6SGUc00vsVH6xq/oW9+8w1t+McGqn6vmrSgNuJy1kBq2d7sDK9G+UaZ8E4XpGjLHfuirIyIwazR+xoxi4PGaQ/VnPONBso5jygMqhSPMmJWrSqcy0mWWINWnmw1Yk5t5ORwfMe5uoIB9B9ur0vLSqmwuLA7H1XTKMO5mUyaRu72TMoryaXPPzuUPnz3dGrz/yb8u+xuY7e5WpQMgJHOdiX6YVJxDOFcEd+kOG1EZN+iWziH4zwV8Lg0Xi29dKG2a8waM6pFEc71jozzB4O0u0sDG5OfHVGAvC/hXO/JeznuLi0tFW0w32xmG328fTmVbF0jtrGYuJaYifbd6aOAL7I/yMK5/L2RAoRznVG/8PldsRAyx5oHJUZeEv7amh1U2+YW0Q+fb02/i3LaCee95JsziGoZHFoU0SBaOFd/XlPdPaOvCtNGwRnVmebZe76exHKcy/+1GE71VCPFyVxTK/FClmBnkFobNVFgbH7LLirpCNDm1hVkspioqKSKZs76lEZveZqCbRt03W8wcjLCa9qN9V1PNC+S4SgTuXJJirZGEWyTEs5jRLVE55yrYrXe7FCEENUFb8Ss+YaO7jawLCqqJXoCWe9BaywiJ4dthh20DmdcbVvI49tKebYmynW6aOKpE6h266HUXDOTWrd1D77Z3RbUgohqAUBhmyqcl0Tmm/eI+DLg5CXDUWnSUSv3N0+tqaTzWGO4wu0fxwCrfJGGGg2v6I7nOJeRfNFjdj2obnWFDWey6HtvcMF1mR1gJOOBiEwLBkgjjf78+d2UsXUJufx1YpXe6rkLacnZv6e9S86KKZyPJONaUsL5HXfcQQsWLKDc3FwxQ3HqqafS+vXdS+5UWKA57rjjxCzFkiVLIp7bsWMHnXDCCZSVlSVe59prryW/P1Lkef/992nvvfcWywimTJlCjz/+OA1H1KWuUrySAynWzPVexhEP/rK/u2G3oZbKDEf4YizPAXVpXiw4LifbYe2R8Qn6/51UZ7XVya1Ynde69u4oBCM6zlmU4fcWISp0uT8iJl106sxyG/DSSy/R1x+9S8FAgArMXNxPI1t5ERWcuJ5m5n9K+bYaWltmpzZvE7kyXFRStptmzfuEyprep2B7aFkZAENBnRJ1wiI6u0zScQJWFRJlX4PzVPVaaZIMqrAcK+M8ekBlpFofqoivFo+KzDg3hmunsaO7jSjNjRXV4ohZTNsoRNfxMOoy6eECt92vvfIaffLOJ+Gx3IrGTbTjfQs1ryAKbvNRzuGTydzWTpb6Bur4tqrHa6A4KAA9Hec8x13ZVUhaJSLiy6COc3W/YkW1ION8aFhXG+k2ZzgvnwXedBTOeRFTcdTKN3XyXu+Mc1m7Ri363husi8qcc71Xvclx9+uvv06d3k4hgm9p2kLOThdNq/OS2WQhS2kh/feUS2j7+Mm0u83bQzjneJeRVGMsKeH8gw8+oMsvv5w+//xzeuutt8jn89HChQupo6NnTMdf/vKX7iV7CoFAQIjmHET/6aef0hNPPCFE8Ztvvjm8zdatW8U2RxxxBK1YsYKuuuoquvjii2np0qU0XEU6bkxkTnjk0l1jNogfba4JF54wwoVruKIWC+srqkUdJCKqpf8091JYbNaowrBb85DJFfSXMw4MZ+3WGsyFGi2cMzXt7giXR06XiCajWph2nXIHuQF/5JFHaOXbr5AW9FNhl3AecBTT2flT6aDRAcqy2mhdmZ3s3gBZ1nxGntau70SAKNBZr8t+g5EXq8Yas5oZbnR8ijDuUPoXsiiwkfsaKqo7R30fKkZdCq4OrlTHOff7bBaTgTPOezrO1ZVXajFto9DuUVdy2g1dmGs4wG33k489Sa/99zUKdB3Tr+rW0r9zi+id3XnUbtuLsmqrqKBxO1maW6hjfZOIV1NBcVAAKNwOVHeJhizExYp+iNAJDHodU/eLi4NGj6kwTh164XxGRUH4/ufbaildYPOtzDjnOivRdfT4Z2k21du4KYv4MuMTcJyrE/h6O87luPupp54Swjmzsm4l+YIaXXBuBd13xinU/D8XUGF26FhXt3rFShK1Rgl/VohqicObb75JP/nJT2jPPfekuXPnCsGb3ePLly+P2I7F7nvuuYf++c9/9niNZcuW0ffff09PP/00zZs3T7jSb7/9drr//vvDVV0feughmjhxoniNGTNm0BVXXEFnnnkm3XvvvTSc4JNNFiLMVxqTyGJRAUPu95vf74hbUBEMHupAOqur49EbsiHh84bFknSAz6fvdjfSLoMsq5fLvnIc1h4d1oq8LPrrmQeK288P3VMMymX2GgtpaoNiSOG8zRXurPLqBDkJYISYH7PZTPsdeBAVT51NmSY3Ocx+sepGsxeRY91GUUwvw1pAWQUH0IsvZNCcjbvJVxV6L1pAI/I1w7XWCx9++CGddNJJNHr06JgrwZxOp2hrx44dS5mZmTRz5kzRFqt0dnaKyfPi4mLKycmhM844g2pra5NeUZaORK8oMVo0U+IRJ9Y4bmfj9TV6c5zHi2rhyQBZQ1uvScBYbZwcXLEjXk5YRue1GyWqRQrnLOhHC889o1qM1/dT2zC1bWPsFsVxniZ9JKPDbfdBhxxEc/eZSyazSUSvrKhdScv2zqe//HgSdTzwV/JOmUiFGW1UMKqRKsZ+S++8cZ3YTgLHOQAhtjW2i9hEZnKMmBajRnwl7ThHVMuQsLYrRpS7QRfsu0f48S/SKOeczw3ZJ40VF6euCOe4OD1jjSOF874d52qRcr2Fc267DzvsMDrwwAMpSKH2eGXNSvIFNGrJslD1ngvJcuJRNLrAEV6lt7u1jTz+yH4foloSpLU1VHipqKgo/JjL5aJzzz1XCOEVFRU9fuezzz6j2bNnU3l5efixY445htra2mjNmjXhbY4++uiI3+Nt+PF4eDwe8RrqzeiExE2th4NHdYEZxYGksqqqiXa3RIoGLQZ0HQ0H1Oy6bHvkADAW+Wk4m//2+t10x7IV9L+vfkW7o3LZ9JzMUpejqxRnZ0QUKpGOPH9Q033mW0VdESKRNQminR/qfb06s3a7nc679HKadsyZVGB1hUX9gCWfbJu3ifst5WPJUXk+ubPLKdPqJV+LmUz+AGkBIpOneUTNeicLrwzjCW9um2NxzTXXiMlxntReu3atWOnFQvrLL78c3ubqq6+mV155hV544QWxAq2qqopOP/30pFaUpSt1iuOcqW5Ln0rykU5tc5zBt/G/O6rQqYr+KjwpJF3nRhmY84o8GZUzXikMGj0JYIT+HreBMuOc2zqesIymMCKqxWvwCEQ4zlPRdv/i6l/Qj376I3F/S/MWKnu9jGYvmU37f78/mfwW8lWOpaIcFx1+0Vu038KPaY+Wz2l7y/bwd9YbMN55BIAebG1sD9+fGKMwqFEjvqJxRRi/Qn0NOM6H/phv71rdxn2NCcW54Yx8fjxdDB8R+eZ5sYVzOUbn1ZR69vXkuN9iMlFxdmzdIBq5Up1jEvWE2+tf//rXdOWVVxJZiDq8HbSxaZPQM7Ito2h8QakYi4/Jt5Oj5isa/97/0v0PnkOf7/p8xArnfWc/xCEYDIqB9UEHHUSzZs2KGFjzzMUpp5wS8/dqamoiRHNG/szP9bYNi+Fut1u44WLlr9966609Hm9ubhaDeb2JJeTXtneSPxBqWOymoNhXRvN7w4/XNjZToSVgqH1f8s3G8P6ZyURB0qihPUCNTU0xB1l6kw6TKPGob24LH2vye8PnSDxsWiC8/c7aBjInuGxIr+POg/Ql324W+8zaznNfrqOLFkwivWDnhtvrExnbGWatz+PN5FkpfMw37a4ls1KcTs9jX9+inDtdbKppJGdnqJF3KNcc8nVfi2qbWqi5OUsfp01Nszj2WeZmMpFG3noP7f6uiTJa8mlMViPVjavkVBb6z1Gn0C//9Rx1dmSwWktBX5BMrmZqaGqgLJs++z7Qa81QX6d4dRff4sFC94UXXkiHH364+PnSSy+l//f//h99+eWXdPLJJ4uJ8kcffZSeffZZOvLII8U2jz32mFgVxvFt+++/f3hF2dtvvy3abF5VxivKrr/+errllltEJy0d4ZUk0dEstWkyAOnNqR05Sa9/P6kvpLjPTuje+hrsamPx1ChRLaojSY1piZ7AMIJzkAegPEFhtZhjxrSkU3FQFpeiV41FFgeF43woWFX9FR3Q6afWXTmUVVNO5uvNvLaecsblUntTLuUWt9JYk49ervmWJhZOFDmpEM4BCLGlobsvOLmkb+HcCO1GLNzeQI82jq/HvO+8z0Zpn4cTmxqc4dUKMqZl/wnltKWhPew6P2XOBDI6MqaFKc+NPaYrzFIj4zw9JslThTRFcH86VkR1LOTKN70d5yqd/k7a2LwxHGFXaJ9Go/JDEwEzqrfTkQ//k1yBOvpgZg59c/g3dNiEw8IFvj0B45gGDSuc83Lt1atX08cffxx+jJ1p7777Ln377beUam688UbhmFNFiHHjxlFhYSHl5ekvZDG8Lyo1nmayWkIfQUVhXvj5orwWslpCeb32zKwev6cHch+qWjtobUOH2O+SnAwak59NK3c3iucsGdlUEFVM0SgY4Rj2B/Pu7nOkuKD7HIlHWWETWXc0hX6wZ+j+vvv6+2trmqmp0x9+j8urWugCW6Y4t/Sa5eZ94WawJD8noeNXWeYk69YGcd9tsul+zCV+c1v4uEp2tHWGHyvJ635/o72m8OMBi37vwVPjJLPFQoWWdrJaTORp89L6D1uppXoqnTmxilomTxZLy3aOn0ymGePJ+66DTJpGpqBGNlM7OfKyqSCjO9NPD/p77CxKhIAe8IQ3t+E//elPRZwLF+jesGFDOCKNI9m4rom6Gmz69Ok0fvx4sRqMhfN4K8p+9rOfiRVle+21V8zVYnwz8kQnd8qji2fWpJHjXM0albEg0bEtRnWtqchBRrzCoBJZAJWFUf6dvrZPZb55ZSzhvCuGjV0+HLEWK9M2VdQrE0RlMQqDRhfNltFmRkK6z2R0nYpdObbIOB8actYsp1MO+YKCh5pod8tBZLadJR7PnlNOzXX5QjjP8ptoZ81nRDNOFzmpEM4BCLG5SzjnCeIxBbHNTzwBzhKdZuC2W90vdZKe22cWztNlVXQ6sbFLIGeml4fGIvtOKKNnv94k7n+ehsK5usI7Xj+E++ixiuimcmVFInXoJLJPygkz/mCwR4a7HnD8yubGzby0k67+oIncE/w0eTKb68opZ5+Z5HfkksldT3ttc9ML1SuE8ZEnCiwmy4hqv/slnPPy7VdffVVkpnIeqoRF882bN1NBQaRwwTmohxxyiBiIc3wLO9hUZEaqjHbh/6NzU/lnFsBjuc0Zh8MhbumEupxUjYWQS5rUmSyjsHTtrvD9hdPHRhREbHZ7DCucD4+olr6/rqoTLB1m89/dUBXxM2tTr67eTj/Zf5ou+6Mu95LiS1+ojbq6vExvOpT3wgICR7So2a9xo1p0Om84P/u2qy+nnc1OOuTieWTyBSnIwlernXKs1RSgALVOnhjaNtBM27OayBsoJI/bThZfkKy+thG1XGywue+++4TLnNt0q9UqJigefvhhOvTQQ8MrwdgxHt2+s0je12ox+Vws0mG1WK23vcfqjR2NLQmtSNETOQlR39S9+iTo9YT3O+jzhB+vb26h5iz9O+8q0ZMoTndofy1k6fXY2ymorLyqp2Id+iXqvm+obgjvT54l0GPfTWLFVej5mvpGUV9DL7ZWN4lVP7w3WeqqpKgVGLwNrzasa3Ua6nvQ3NJKra6Q+O8wha4jKp2ujvCxbmk31r4bdeIwkbb751f8nNpcbfSHe/9Ak7/dQrZKTYTsmqiQ3F39Itv00dT5Sg5ZTAEyk586m1ZTR1sHBdh9amqnJmtTwo69kbRaDAwtPM7+1/JNVJKdYQhRsbGjM1xTKd4kKn9PuEA2C9CGdZyrK90UUZFXhPEELcdJcltixJXq6cqG+m7hfEZ5qK9enpspIls4O59vPMaKtfLNsFEtCQjneq18Y/FYdZwnSmSR8iBZ7Wbd2u5FixaRx+uhX931K2pwN9D46k46aXUH5az7gDSrg+ig6ZSbm0Gbp0ynsat2Ub7bQ8U76qnD10E59hzhOIdw3ssJwjk4ixcvFiI4F/BUueGGG+jiiy+OeIzdZ+xY46JkzAEHHEC///3vqa6uThQOY9566y0hinMxMrnN66+/HvE6vA0/PpxoVWZbVcFTXUpttOXTn2+tDX/pj9hjNL35/c7wc80uL00s1nHnhiFqcZVEZjNVAVQ9v/Tmi221tLO5g06YNT68ZI8bmy+7ipXwZBHPunID8t6GKjp97kTKi1GUbKhRi8nJnNx0Fc5lcVCOCueZeK5NoKIW6YksDqrfecMDUH9nJ9nJQ8FOTQgzrvZMKs+qoozMPOqcUEm0zUk7XG/RJ96VVERjyO+ykskUJEvASS5knA9IOOfIFXadV1ZWiolxXlnG7vPomiMjbbXY9sbulRqSFk+AcvPzDeEU6Q0+jnZn96qewrzc8KqI4vzQ6jHG6sg0zGoZFXWfAhRaGZOb2ftqqtL8XLLWhMQsSwav2tPnPJL7WOcOHX/Oipw+blQPMaQgN5usDaH6Ho7sHCqMM1BMBe5dbWLVD+9vZXlx3ONclJMpBqvugLFW9LV2+sLndGl+do99KwmE3htjsTsMte8SI+5TX4Nvvrk6XZSZk0mBYFcfQiNyTJxBmV3ns2neDJr/xn1EtjYKWk002uwmn91HGRkZolBofkG+GITrRbquFgMD4611u+jtdbvF/dmji2hSnHiUVMArjmTts+gi0tFkGl44V8evquM8NLbid+ns9Oky1huO8PHe3twh2u8xBVkRx/WQyRVCNGf+vXwzXXv0XDIydf1wnOu1mlPWJVXP8b5QV0HyqkjVMJtqeMzFwjnXCGtyN9Hk6tCxNLFFZe40YaLgibrGeXNo/Or3KRD00LxNLmp0NQrhnFeMBbSAaMP5/nAnqU+KB9Gcb/rSSy9Rbm5u2EGWn58vnODsFI9VEJSXckuRfeHChUIgP//88+muu+4Sr3HTTTeJ15aO8csuu4z+/ve/03XXXSeWjbOT/fnnn6fXXnuNhhNqQU21qGOm3ZgFu7hBl45VFuK4UVcd5kYqjDhcUCdO1PMiHurSZKMU7eLCaH99b7XoJG1vaqdrjpwjLsKfbq0NZ4wePLlCuA54IoYfe3PtTjpr78kp31fVbZ2w41yJlamLykHWE2fXpAt/T0flZ/UQztVJFm602fTBHQC9HOd8/T/60mtpQ20TvWepIOuKebT59a+p02ujD6+20KwTLiNHS1dskX0WNeW+TaeV/4tKNCJ7QTl5gxz50UKUO0qX/U9nuHbIb37zGzEpzsU9mTlz5tCKFSvo7rvvFsI5t+1c9LOlpSXCdc6rwdTVYn2tKEvH1WJqJ17mc/J3pb69U3y30irjXOncqw4ZNc7FiLAzzeMPJhbVorSDeq+84n5TVVfR69H5sR2EqllCbxGkocvtyJTFyTiXBe1ZOOdi2kZyDUasGosx+W1XM84Nfs6nC3z9vvsvd9OqHavIardSpqV73GKdETJEMd6J40lrzxTnisMXpHHZPmpwNdC4vHHk9rvFwF1P4RyMTNRMcW7r9RTOVbNUXw7WkIvba9ioFnXFvDp+VcepbR4I54PFxrpWYTbiHsaMrpgWyVHTxtBrq3eI8fg3OxtETOqMCuNO0NZ2RSFyXy6eqGwE4TziHE/GcW6QyDhuu++//36qbawln81HmdZMmlYdILPJIoweNGd6eFvPvnuT6enQe5yzxSXc6ZUFlaJGCce8cPtttwz/SbCkpgYefPBBUSCMi4eNGjUqfHvuueeSmpnnmBf+nx3k5513Hl1wwQV02223hbdhkZ1FcnaZz507l+655x565JFHRFbqSHCcS0eu0Rzn6oBENnxFUcUZwOCifv7ZiUS1qJEbOlaZVqlvd4eLlXy9o0EUJ2He2xByeDBHTB1NJ84aH7pQd0UC6RFTFDnoTkw454E4iwhGdZzz+4i11E2dGOCBrPxZr/OGJ1MC2QWUVVxG2Q4r1eyqJVdnBgWDFpo+cwYFRpWHs4AL7VOpLSfUaeqoCZKvs4JaC/cnj687SxgkDmeX843jWVS4neZC4Mz8+fPJZrPRO++8E35+/fr1tGPHjvBqMP5/1apVYkWZJHpFWTpS7+z+Xs9UBhs1SlSZkZGCc7TorA5mjdTXiAWvRpLwEvXeUK9tegvnXBdGxuOPj7M8OsIs4ffr3l5LSnupNSJNE/ze2DVoFJye7uMXq1hYxIAVxUEHre0eN34clY8OxXIVZJu7IldMZBnfHedJdjt5iycTBUxk82tUafII4ZxdatzOsGsNgFSzuzU0sakaTvRC5iVHawGxkEJdyPUaWYPFeBnnkVEtsVb5GhGeFOboHBaleeX0+xurwlE6RmNtTUv4/vSuwqBqv+/MvSaFf+bMcyOeM1JLYoGfGZUX35gix93yd9JJOI90nOtXpJzbaTY3V4yqEHGoNxx4Gx3YPImyLBVkdtjIN7GS/K7QeyycPJpqi4vE/SnVHmpqD9ViFI7zYEDcRgJJOc778yWL9Tu8DDw6iiUaFuf1KDKaSlrdnpgXAPXLZ6SZ5DbFwSwF2siMKQjnQ9mJSmQpjzpQVD8vPYk+hx/7fL3YT1nle0JRjshfk87zDzZWi8bonfW76aTZlYZ3nMulZOy+Y5e/EYrR+QPBsHMx224TWYnRRE8M8GcSykH36tZBlZ0lnpBz1oZEcM2s0ewps8X9TFtI9DCbbGQqHk9E1dTwiZn80w+hmmPOJjOZwwVLQCROp5M2bQoVCGK2bt0qHOVFRUWi43TYYYfRtddeK1aPcRv9wQcf0JNPPkl//vOfwyvLOAuPY1X4d1gM5+g2Fsu5MGiiK8rSEbVgIi/lZscOU9PqIlJ0IaOirlxTHecRsXAGq6cSrzAok6HkQ8bCKNFTPQqDxileFfk5BAxxrnNhOtXQEY36HLd9RlluH8vgoaK2zXCcDz4Wk5kcWR4ykYm87iyyZkdOvnTuNYeCpq/IldNEo00+WtNRL1zmfs0/YgbewDjwiqBqbscNIuRGiM19jPlkbjjLLCy+qW27EVBXT6m6RsQ41UCTrtGsrmqiBz/6PjwukfAqw7tP298wq6wk62oV4bwr31zl0Cmj6PU1O2hXSwdtqm+jL7fX0X4TImsSGYE11U0xjSrRcG1AWSCXo4L1oCNCOE9cUlVXvnkMUMuJJ625zW7Y1UhTWxqJTGZyTppIO/+zgzY+vJFGHT2Kxlw5nXYUl1Nh/TYyBTXqqNlBtAe3+Rbx+yNl4nv4h9EYGFnMgK+96qBDLaJhKOFc6VDIhq9QKWoKx/ngo7oAExHOczJsoiExUsZ5tHOcO0p/emdl+OfD9xgdvn/y7Mrw/r+2ZgcFpFUvRbR7ep7jiaBmsKmOPb1wKecNR7VU5PV0nEe/Pyk2sbNTFalSRWO7i3Z99yXVrP6acu1EgabQPnhyPTQqLxS/ktXlOGdcxRW0qjKTPtwzl+omlIjlYrxUjG+gJ19//TXttdde4sawAM73b775ZvHzv//9b1qwYAH9+Mc/FuL3H//4R1GPhKPTJFyv5MQTTxQFv7loKMevvPjii0mtKEtH5EoSXgkxsWuSj6kxwHc9EdQYFlWkjXQ6G7vTq+5fn1EtXRmqRlh5Vd3WLcqMKciOuY064NLTcc6TjnJ1RUlOZq8TkGrfz0imifa+HOdRRbnAwPH7/fT2W2/TVx9/RYHGBrJmhfpRPm/PfkfTpRfQZu9s8rkLKKs5mzo7tgvHGp97I2XgDYxDbZsrvCJIXalphPFSX3WtIiO+/OnpODfIyuhoNtS10N3vrOwhmjM80SLj14wCTwJvqm8V93mFcXHUhCXDK7rP2ac7AvVfyzeL2mJGY3V1d8HuWaPjC+f8fuS5ZATHeXIZ58aIauG2e+nSpfTeu++JVcfe79aGn/NMnypEc6b67WrKbPeTqyBUm5Ivmf7qUGoAT3xzvvlIGXtDONcRmUHNzlZ15jIid9RAy6dVN6508rDgL3ddrxm/4UyywrmI3Oj6bPReoh5LxI0+r9nVdtCk7uzj0fnZNGdMcfj7keolcRGO8wSjWhg1CqXWAGKaOgue47BSqRBBIreJfn+qw16PwUNtq5M2vvMSbX1vMR3lfIgOnvs1HVi8lWY0NNO4Je9FOM4ZV8loemj+HPrIezR9uCSDnGudwrHmCxrjvDcavIqLxYno2+OPPy6eZxH8scceo927d4vM83Xr1glxXRXPuIgb5+E1NTVRR0eHEM2js8vlijKXy0X19fUiI91q1a/wzUDhgYUcPHHms7p6o0YRRY2MJ47onFaOc6UdSaeMc/VaHM/BHbnKUL8+Hx8ruWy4t5gWpkCJ6ZMmkHRznHsQ1TJog++HH3yYFj+7mPzrN4TND35/zxUWZouZOuqKyNlSQHW7xtP4nO6aJCNl4J0sXKj7pJNOEoW6uT1esmRJxPPcDvNqr+LiYvE8rySLhou38sov3iYnJ0dMfsv6IxKOXeMaJ1lZWVRWViZWoPFnO5xh961KhzLxZvS6VkZpN+Lh8sYev6oTmno7/GOxq8VFd761MtwWTirJpYUzxtLe40JjU2Z9XUikNtJ57O+aAZpWnh93u73GltCMrhgXzhHnld1Ggsck7PSX+sAeZT2d8ypFXakHstaKkWsSxHWc6ziBz9d3rin55GNPUoBXy6/b3L1f46dEbFv7US1ZK8aI+3yo/V11LiUjZcUYhHMdLw5SOOflJipqA6NHznM8VAezbPhYqJUxM3CcDz7uLsGDtStV5OgN+dkYxXGuNizzxnZ3PJgFlWU9KscXZzt0O/8jMs6TjGoxUoFQVazJdthEQbqSKAdCPMe5XvEGLL4UT5pOYyeNo4neL2nvud9T5eRqGutqp8xvV/VwnNvMeZTZkkmlG0vJuc5J3lov+TQfBt9gUGlyeUUnkSnNzRBinFwSnS7CebzioOp94zvO1biZ3gWFiGuZsorIqA7CDIMIIGph0D6Fc2USwCiFyKMzzmMVB+U+q7WrlgqiWgYHro2xz4J9aMacGeSqX0mZVjc5LF6y5cSOJtrlPpXefeUX9OU7F9GcspNG3MA7WXiSmut98aR1vOcPPvhguvPOO+O+xtVXX02vvPIKvfDCCyKGraqqik4//fTw84FAQIjmXAD8008/pSeeeEJMqssVaSNFOG9PI8e5UVYqJeI4V9s4dWxltKgWdpP/5eP14c9h1qhCuuX4feii/afRybMnhLfb2OXuNgrqeJ9Xi8WDJ9bO3adbEH1xxVZDGTRr2tzU2BEyqkwrK4hZTF1FxgXznIEeJomIqJYEzI0Sh8UYkXHcdu+3336059w96YMdH1D1ulfIE2imoBagdiqN2Lb2w1oKjptAtaOm0IZp+9LBM7vbD2akrBhLXxtYmsNfNjk7GO1CUh0xRhrMxnPj8oWL3eZ84eZoDVngEQwcOYjmDlKiuc1q5AY3iHrn3qkdweNmjhP523IpFhcFjSayOK5fl3PcYTFHzAj3hSoyGKFAqNqYy04qO2Vlfi0vV492bUbEG+jQAWnzabTnSefRJO07cti/J5fNQ+ZONxU6mskzbT+xTYbiOLdSAXmzvVRU2EonHvs+jfa8TG279ydfSfoWoQTGQ3bipeOcr8P8XdrW2C6+T5yR2lfn3rBRLeq11kCT9LFQXTl9TSJHOtoMJITEGVhFCCA6LrlXB9/Rho5o8tWYPkNFtfhiFktX4fbP7w3oEkk2HLHb7XTtjdfS8i3Laft/rqY5didZ2OxRWUixplQyy1nY0VitJXd1B9kL7CNq4J0sxx13nLjFg2uKMNu2bYv5fGtrKz366KP07LPP0pFHHike49VlM2bMoM8//1zUKFm2bBl9//339Pbbb1N5eTnNmzePbr/9drr++uvplltuEZ/xcC8MajThvC8Ha8TEt4HEz+jxGxdktiqF543qOOcJ4D8s+5baPD6yWqw0pTSPrjlqTrh/N6kkT+gbrHNsMJjjXK1pFmullcqU0nzad0IpfbmtXkxccCyNXOltpHzzWaNDhSh7Q62zx+bN3uqyDHkxX2v/olr07IfwdZ3rUH29+Wv668q/0syGKvIGPWQz55KzKfL9tK5tJf8P59KS/wlN0F48N+Q+H2kT38Ye7Q1jVIdO9BedHTGyQTR6xrk6wGJXnlFczsNNOM9OYiZTPZ/0LowWq8DpFYfNoiP3GE3n7zs1ZsOoigupdt/J4xXtgk/KcW4w4Vy+l3Il5zyWm17veAMZy5NnaiWr3UxFuRaaZq+leSV15J0+tYfj3KLlCeE8GDRTSXE92YKNZPM1w3EOBpUGJTtRfs9HdX2XuM2TmdBpE9WSro5zpR1R30MseFKQB+tGEELUPlw84dwojnO1X9pXjY9CtTioTvmisXAqbV88AcHe5fZCxvngU7U9QFsftNOOxxzkKz5aPPb19ja65dXNtHRNqKhyyeqvyLZpK9k3byPP+jrxGOece/3691eHI8uXLxf5tUcfHfo8mOnTp4ui4J999pn4mf+fPXu2EM0lxxxzDLW1tdGaNWtivq7H4xHPq7d0Y1dzpHDuTIP2wggmo0SQbVm0eStydatxHOfvbayihi5z0bjCbLr+B/MijjEL6LLGDTvTjTC+jnUcE1ktzZEt8VZd6MmqgQjnOkzgdyjXi/5GtXh1jozjfHKOOG1yN9HycRn06cRM2la5B/nbevZFHeu6r/Etru73LtrvgHG+D0MJHOcGGKCoS17VLyDPILuVjDBDRbUo+1yoZl26POHMKTBIHQ+TOaF889gVy70Roq7+DgqrEPYvOWhGgrl9qesMcj6aFFlykjjesvHmPDZfQDOIcN593cjuei8VuVm9iiKqyMCOi1QjByy51CYmD+1BE5E3h4K2LGodH5rZ5sUsdquJvH6NTJRPF2wh8u2wkc3nJ80bIJu/DcI5GFQaO7w9VpaUK98lXlrKtRnSJqpFyZvn7xk7X9jNbUTHWiI57fHgwTmvFjCK45zjQeKtTMgwiONcda315dxS+4DGyjgPHT8+r+OtGpPnjzdg7HM+HSMo/7WPg16YPpGmBYvoN/MWiGvLy9/Vk8cXpPc2NNP+kwooK99Epq6idJ0baimoTRUFxrxB45xHw4mamhrhLCwoiMwLZpGcn5PbqKK5fF4+F4s77riDbr311h6PNzc3i+gXvelLyOf6JTub2iigZCO3ON1i//WisbWd/AE/BQMB8ne6et2XgLdTbMvUN7dSc45xJB0+9u2u0P5ZyRrxPnisxe8vSBo1tjl1Pd4quxqaw8f+R7PGkM/lpOaoNL5xOXZaVxM65t9u2UVzRsUvXplKaptbw/tu8nn6PKa55kD43NlU3UDNo2PHaqWSltZWWrmjXuwXF9osMPv7fB+2oC/8PnbWNdGE7NSusG9qc4b/ftCb+LXD63aFf6+ptU3X74Av4CNPh4fqnfX0xYI8spgK6IjSX9CNR42liT+dSPUf1VPH9g4qPbCUaiscFFwZmgCva3aTuzjUDwx6gtQSbKE8LS/l+z+QCdv+/K5xrrIjDC5k0NsAhQXGZvIaahY53oxm9FIZMDhwBIAvGCSrJVnhXN+s6oE4KKJzwlIZH8ACh+w/c0HNZGABigtwVrW6qM7pFgPIRKN1UuU4r1Ac57GE88ioFh0yzttd9MU/76Yq02Y66xcmyvL7KeAKdYICpSF3RIOrgSxmfm8W0gLZdOzUg+n11zoo6LeQ5mPhvJ3aR8isN0gNDUpUC3/Ho79L6ZBzrorO0e4v7mt4/Mbqa8SiU4lqScTZw32UkHDu1fV6LI9rb/mXRiny1hKxqrB31xqfN3wu8YSLsYRzX5+OeVvXMmk4zgcHdh5f/j+XU21rLfn3DpI3y0GeSZWk5ebQiq2tQjRnuH+1uspJCyYX0Zxj36WCUU3U6fgvbWh8gCpyKsQAHqQPN954oyggrooQ48aNo8LCQsrLS72AEgvel3jsbukgk9kSIYR0BjUxwaBb/91aI2JCuNUoLy6iwsL4gmZJgVtsy1gcmb2+11TDba5XI7F/edk9960gO0NoCh7NZJj91iy7w8d+bFkxFSqF4CVzJ3jpva0h4bDaHaTDDLLvfnN1eN/HlBZTYWFOr9vPyM4lq2WDuN/gCRriM9je3EGeYOicmTuulIqL+nacjy31k9WyS9z3m20pfx+aZVf4O1hakJ/w3y/uCIR/z6bjd5fb7ksvu1S03e2HtXOAGmVaiign00aZuZniln92d7FZU6uHArSLOgLVtKLRQ3tNWUCjc0eTx+Yhq82q2/vo79+1KFnziQLh3AhRLTE6+HJwy4MSvUW4aBGWxU/VORW5VMY4A6h0J9qpnSjq+WSEZXCJZLzGX36YOhFBdSYm6zhn2NnPwjkPxvm4pzprTaVDEcHkssgxBd2u2CKlAKsk066veNPh9ZKnvYVc1E5aMJdM/gBpbisF83JJc9jFcjK+cc45X2bcviBZRuWTiVzk6cggW56PrH7niFkuBnSMasnPSivhXLrJuRshCyNG9DXc+gq2ieBR9k/GsPSGLAzJpWT4vSUz+TwU7V9vhd6M6DjvK+M8tI2danxuajVIxjk7GXnS2GKxRsQBxCvMxUukjdK/Tmf4GDbUN1B9Wz3NemUWFdUWUW5eLgUOCdBnWyKzgFftdtLB08rJWtdMhaOayG8P0HcdtTQmd4xou/F5DD4VFRWi6GdLS0uE67y2tlY8J7f58ssvI36Pn5fPxcLhcIhbuhIrooInd/ianZ1kXOOQjPv6aLMiM86NNfHtC2phI1KsiW6e2ORxEusKRvnOJ3Lsp5Z1f3+MlHPensSkN8P9IU4HaHJ5xASSET6DdXXd7t89E3Ty623cdCmru7P6WRxUz4xz/tzr6uqorq2OTGQS/eUMcxFl2mP3sQuyrHTcaw/RmK1fUIk7SF8/+X80etpospqtI6b9Rsa5TqgOnYIY0SZykKUZKHtUXpijL8p6X7iGK8kKztGCQfRAWC+ka5yvpYkIHup7VY/BUNPu6T5WyTrOGTUSp7ZdXzFNdZxn27uLg542dwLNqCig42eOM1xeYmfARHudfRld9eNJ5KvqoCZnJr2yfgH5SkNFa1w+F2XZsijbbiGNNFGgx1tUSBlWrxDOgwGNrAEn+fy4BoHBd5yzSCgnjCuiolqMjuxD8AA2ulMrv/f8nedOr1Hp9PtjCs3xiKzZoE87yMdTtmGJOs47DZNx3vfgW4rrPHj0GqCfypPf8gzuzXHOxUHVlX1gYHAMyB/u+gMd/7Pj6ZRZG+iwA1fQrFmbaFerl6pbI9vjbY1u6hhXRu6W0DXUHDSR27lNZKTKyXEwuMyfP59sNhu988474cfWr19PO3bsoAMOOED8zP+vWrVKiCiSt956SzjHZ84cngXXWTCUqGMTPXPO1Qns3iZbjdRuxELdn1jGL9k+c7ylWvjb6GNuFptLuiL7NjdwNKQx9l01yeUk0HYzYwuzw+/bCKvG1tV3C+ezRvXtNo/Wn3gSQM8abn0VrY+16k3vyDjZdp/wsxPIzJ0nTaMMS1HEtYWLfvqDXf1Ym4XKXQGqbPJTTmeQOmp3ise5/ebi3iOhwDcc5wbIC+/NcS4b0mQcx0MBDy7kzFr0gKRQcSZBOB88OrzdDWF/o1qMUKxVdgT5PSQyE6lesNVGKbWO8+TdJmVdMQ4M55zvoTgTUk1HV85rdKHTs/aeHPd31NxgPWbAXf4A5ZaPpVk2PwVr/OR2OqjFHSBfSch54PK6xJKwbEczBTUfWUwWcuUXUIbFSx6Xg7SARibOTvQ2jYhZbzD08PeA8/55SWWpMjHGgz6+TvG1LR0c59KtreabR/c1WDNnB24i+eH657QnFtUi4c8wMr03df0mdvD0FS+jHnM9nf9y8M2rEhLpc6irqnjgrXc9lURdd7I4KMPnfLwsdJAYZrOZJk+ZTPZNJjpgyjbiEbizs5Ge29rtyCzLtVNdOzvSiFZq2bSnM/T5mIJmCjh3ioxzn98nBt4Wwueh4nQ6adOmTeGft27dSitWrKCioiJR4LOpqUmI4FVVVWFRXDrF+Zafn0+LFi0SsSr8OyyGX3nllUIs33///cW2CxcuFAL5+eefT3fddZfINb/pppvo8ssvT2tXeW/sanGG708py6fvq5vDBYb1aC96up57/x6oE8hGi1qLmACI0ZZEFgj1UoZN37ZDHW/azWaymuObvPYoyxdFRHl18Y4mJ00qyTNMIkC2zdrrvqvwKuTvdjeFV1+oIrQefaWNDe3Cz8t180Yrqzr76oPwUI/bFTUCOVW4unQa7kdbolZzJu44D+redlt2W+hHS5rpqK9aqKPwPfqmeAqt+aqFCmcVkuWA0L4WBgqp7pM62r19Lo0NrCSHxU2+qt3iOR6Pe4IeIbKz+3w4A8e5TrS4+ioOaoylu7GysqMjKNSLLRcHBUMw+51MVEumwaJaus7fRN9DhPM5pY5z3wAd5yEXAlPXHqrMrhcdXZ1WbscTrfQdedxTL96wy8dEQcqjNqIAkduZQRkWN2nl5cKFxi7zwsxCys/ICDtjX2/+mnyBBupwWkljhSqokcnXCtcaGBTqnd3f47IulxHDkzK8goNp6Og0vGtVOs5Vp23svkYgPYqDJpJxHuE49xl61RjXyAjH8ynO+lQjB565dltCE49qYXgjmCbU/k5vjnMuHGqEZdLDDdO27WTVgmQNsIphpu92hYRJXvZ91j7dUuTKOg8V2fyUaXFSprWNyFUVcpwHg2LgDSL5+uuvaa+99hI3hgVwvn/zzTeLn19++WXx8wknnCB+Pvvss8XPDz30UPg17r33XjrxxBPpjDPOoEMPPVQI6i+++GJE1uurr74q/mdB/bzzzqMLLriAbrvttmH7cexucYX7yVMU8VOvFUqqAN6XeBs9gWy0tltdKR8zqkWtqaSjw19Fjjf7GjNNLc03XFyLbPtykxi7jivI6TW2KJVsqm8Vk9jMnqOKEjY+cd9Jmk/1iWpJTt+I1Qcxwmq9RncjlbT6KdOrUXldJ/k3e2nXq7to+0vbw5/Ftv9uozX3rKHO1jxq9FaKx3zV1eJ/nvjmcbp0pg9nhve0gIGRTmA+H2Mtq4kokGiABlEdeEY7eXiAKmf8jDB4SpT/fLuFVu5upHGFOWIGeVpZgSj6ZhSnarJFNWMNGPUuDqouVU/0PajbpdJFMRgZ56rjXE/kMWe3eaLns7rKJdViAguPHp+fmjasoH8HK2lBSwb5WrKpeqqD3PvtLWJasu3ZlOvIpcKsLNIoVJynOdtBcxzbKNPrpAyHR1xQLd4WuNbAoFDv7P4eR7tpua3Y2tgu2r3aNheN7aMYk57XYDn5HiviJLIwpV/X2gwJC+cJOc71LXYcvWKqr4EViyAsfugxaSnzwbsLaya24qooq3syiSeQppG+qP2d3qNaFMe5QWICollV1UT/+noT7T+hjE6eM4GMTCAQoI8++Iiqv1hHgRmacN65/NkiTo1ZUJlHYwscVJprp/p2r4hrKSjLI5vZI/IoM9w1wrE2UpZ6J8vhhx/ea4zWT37yE3HrjYyMDLr//vvFLR6VlZX0+uuv00iAz82q1pBYODo/O8LApmdUi+y7RxfxjkWkTmAswcqdYFSL3hMVsSIu+3L6s14g2VDXQsfGiL5M9fhJTpyoK4z7YkxBt6t7t7L6Qg9Wd632YGYlmG+umjd5xRvfuB/DYnqqz5lka+iofRA9J+9l2712+Vo6uCX0PWz3lxBlhd6PY6qDcu251OnvpKKDimjrM1uJbDZq6JhAozPXEdWFxuLcfvs1/4hovyGc64QspsQzZbG+5BGD2RS6bhMZkKgDUob3n+NaOF9Kj4yp/mZ5/nfFVnF/U30bvbchtMSRO09XHLanmPHUGzWnOpmLcrbdKhwUPGbRO+OcG3Q5eEr0PehVHDTyHB+gcK4IbnrAS02JTEl1oji/mQe8/HmluhPOAxUtEKDvly6mze5O2uk6gGxmG9GxRJ3770MdzlqqLKgUS8DyMrrFGnNuOZXmbaRizU92Xh5mNpHZ1xpyrWG1Nxgg6gRYqRLFxPCE62dbQ3mwS77bRlccNsuQx9sf1HqNC4ksMGbcTm/SUS3qwFwnISRCPEik0JtbP8e5k/PBu86TvATbv1JlFQYvXdcbNZqu1+KgaeA4X7xyq5iY49v4ohyaN7aEjIrP56O///XvVL1mM/n24H4EUaOvW1zad2K+mMCfMyaH3lnHUWpEVeYKmkw89tGoyNkYdqzBcQ5S1bZz28iMLsiK6Cu3K1GHerUZiawUzUwTx3msSQC1FpdeK8JUWHCVxzAzRqSdCl+PefUeT7oawXHe37HrmPxQxrkRHOc8USyZNbooaeFcmlhY84hVN3Ao4Hx7OfHeV/+u92jUoP5td3M1Te4sIKu5neq00WTqqvngmOKg8pxyqnXWkrfSS5mjM8nV4qRWXwUFNRNlNLaIdptXjPEk+EhovxHVotMFWhZhihXTYsTssoiiUTH2uaBryS43gFIoNTLsjooFz1j+e/lmSufioDxAkZ0SvaNa+nIdxBvUyrkkl25RLclnnPP7k049PR3n3JjLTmt2ku+jOy4gtY1fBx97k4kKxk2mkqIyMnc1TdkV7FoLiHM63xEaiBdkZJCZTCK6xWEpoOZsK7l3majt8yyqn3IVtWdNHBGz3mDoqVcil1SRkDlyjzGU3TVI+WRLLX23u9GQH4k6oFaznY1SFDhR1MFFIm48NePcEFEtfRZ6k0VaA7oUaY0YfCfoOJdF0owinEeujIzvOLdFZJwbs61Qj+fDn6xLaV+oPzmps+fMpn3Ky8li5v6bmRpMoRovU8uyqDQn9FnMHtO9KmdtsDy8Gq7U6yRfIPTZjYSl3sBY+eYcWaGu/Bb9UR3g675sgxMSzpVxYar77Mn0O2KNX9VVTXqPU6MNin0de47QmdwV7dPY4aHGOHqCPsJ54mM+Hh9ysVMpnOtVHJ41sS0NbeGVnMXZkX3tpCLjUphz3t843ejYRD37ILLtnj9rHo32ZFKmpZjqtXJhvuRxdPGexVSQUUA59hzyBX2UNzWPzDYraWQmTzCHCtt81OwOrRbg9nwktN8QznWA3ZVSW85XCmsauVq2KirGWsYrc875belRoCFZ1EiZgydX0I/mTw43IOxAr9U5aiP6opyd5Gym7KjomZfaX/GfL75y23SKalFd500dHt1yj9XCoMk656WTM9WrXNghb7HaaM7pF9F+ex4qln0xeRV51OHrEI0235iCTI5TMotZboe5gFpyLORtMJP7axN15u1PnfaCETHrDfSNauFIk/MWTA3//Ohn6wzpXlX3KZbgHF2I3KgkHdViAMd5Mu1fdJHWVMOmAUlegoNvdRWGWg/AGFEtiTnOjRjVwgKGalbhlZxPfbmBjIrdbqebbrmJrlgwlhzWkBjemFkq/t9vYrfzvCLPTqU5oc/lrbFzaFduMe3OyyK7w0pN7pDjEJPeIBWoDlsukhjpONenvWDxW2qXqps8HlzEWdYjNMLK9KSiWiKOt/5RLWqsWiKTFhztquZz60mbu/9jVz735bhRbXNSCf9dabhMtCioSoGio6UyLlj9zkkTTaLw5Iv87urpOJdt92U/OolsbEjTiFq9+SJJwpJvodETRlOGNUOMv3lcnT02m0x2q1gr5g7kUVG7n+pd9eF+y0hovyGcp4BHPl1Lt729mrY1csXgyPiMeI5ztaFRL+hGzY6MKBCqczxIIqhFTGdWFNKpcybQwhljw499uqWG9Cayunr/BFCedNFrFrk/nZHuba26Os6z+si4i4d0pWo6OvDUfMZse3KOc3ncUy0ASoePifykNXeQwx8km1+j0rICcvvcVJJZIpZyM7kZDrKYzOK8tlIevXhgId1zSjn97fy9yeMMiIPPojoAA0WuHGFjZCwXzGFTRtGMioJwQeAXu+K/jIRaXDyWcG60QuTxkPvGgw0WDPoicim4Pn0SdeK3rzZc7yKt/XGtsThts3QJpTq77pIpDmo3uOOcRafoyZP3N1bTyl3GXNUisWmhMQ5TnT9K/D++KLKo8uyxueJ+Y1YxtWwcRfa2bCrxmKnJGSoyhklvkAp2K8L52ILsiGueXhnn6ngnkVVV/H2S7Yrbn2bFQSNqcenvOFcNR4mMVaeWdReTXa9zXEvkarHktIKxhfrHtahit6olJYpqknCmMGYpmRo2sZAGECMUB3XUNwrnf0cgm3xaqG5hzrQcKs4qFs+zeM5kjsoUBaQDZrMQzkva/NTo6u6X+APGHUMMFhDOhxh2Lr+zvop2tbroyS7HSHMfsSdGdJxHRLX04jhn0qFAqPoZyJiZAyaWhx/7dEst6c1AlgHJThdP4socv3SKm5Hnf2od597wfvZVzT4e5bnds+W17S7Sgw6v4j5IcMl99HnDM+DciKY6z/8k83N01UFP0iXnvkp7ao107B8fFtlpeRndndRszvA3W0RUizdgoarSCZT78ilkfmw2bXxsi9hmJMx6g6GnvksMLMnOEPn/sQauFx84PSzkvrJ6O21v6haPjDaAjeXUjpykN+73Rr4PvkYlUvA40tGmvxCS1cdgPLpIq759vMRXh8kJJV6doeckfTJxM+oyaT3dXol8FqqT7R+frDV0ZIvNJoUXjXYVjBPXzNyMyPOec86ZYKaFVm+cS99+fSgtX34WVRZMGDFLvYH+SJGQm5KKvMiMc72E80iXtsXQq0STE86NXxxUbXP7aquZKaXdK2k26iycq/2bZKJajJJzPlDhPMeuT8ySOtmSbHFQtUCo3itV2z3t5N21jQKaRi3eAvKLjP+giGnJc4TG3g6rg+wWO9kqbKKGScBiIXcgl0pcJgp2ieVsbvMEjK//DRQI50OM2iCsrWmhXc3OpB3nRsgdVWeEuaBpNFwcNJ2Ec9VxLvedl+JP7aqWzQ3IjiZ9q0yrs5nZAyg8oefES3/Ff3n++wJayiJPZOcj0XzXWBRl67/yIiJyJtmMc50qffNAJeDz0lNPvk+3/reJHLntFPQRmSvyKdOWGY5pke+JM84ZlzdAeUV5wmWekdlGNtcWyurYDNcaGJRrl+wYl0bFtKiMzs+mU+dOEPdZN3zk03UpnXRKKqrF2ldxUP37Gn29j4w+CnepxY7le9PL0dafqBb9HOf9G3zLuBYWoFPp9uqt7bPzZ99L1IHaPzKC2ysaNe7w0CmjaNaownBkCxcNNRoej4fOW3Qe/fHTdWQztZAj0EmtWYWUl2ERS75VOK6lmONaTCZaUTOftm84hGrX7kmZdl4ebiZvQH8RDQxvuH2uag2JhKPyskRbwddnearqJpyrjvMEolrUsZLxMs6Dva62MprjPGKFdwJjVd7/UV2xIlyYkmtLGWGiNTdJrWCc4jhXV2Gkk3CuTi5LE1bKHef9EM5lZJwe0Xxq233hJRfSLx68n3a7tlOzN4MCFn4vGlXOqwyv9HZYHMJ1bqmwkIlM5M4tonXjD6NXzr+VDqk8RGzDEasjof2GcD7ERGeGvrV+d4SgxjmpsVAbGiPkjqpOnlgu1ojiDOkgnCsDE3Xy4qBJ3a7zT3SOa+mvWzt6EK7nbGbELH4S70HdNhUiAuerSZEs2Rn7eJ1BvfLi1E6/OhOfCHrlHcvOTktTO1U5/ZRhbqcJ5m9JqyijTGtmqGJ3F9ligMPlQ03CXZBfnM8ZL3TBOW/Q/nvcRxN2PALHORgwaoHf6MKg0Zw8e0I4m5FrZGzQ2YGkol4/Y0e1GGOStS9kO6a6hftCro4zRFRLgsVBo39Pjz5RohnnPQqEdrgN0U/lye/eViWwUCbRc9AaD3WMwMaOSw+eEY7EeW9jleHEfl5psGXbFtpRFSDnFj+11dmFMF6Y1fM84s+lrKtYqMPqIVNrO5lrGslT7x4xA2+gf9vOphwZ08LwBE9217hDr2LSke1Fgo7zru24VoORJuzd/t5NU+rEtl4rwuIL54kd+8qinPD4US0kr6s+k+T4lY0fejvOeUJYImvNJYMaSZpS4XwAdeiM4jjntrtqdxV1tvnF/c5ADvktFlHke8weYyLabTawBXODZMm0kDcnh5psJbRi/DTydV12WGQfCe03hPMhJtrB9dGm6ogIh/iOc0XAMsASLNmR4Jm9WDEWehVn6C/NrtCXm8dWalzO/hPKw66DT7fW6psP3vW583JXdaCXvOPcnzaz+HotW1fF5t6KivWF+n1u1UmsUTsOOckWB1U+I08qhXOPj8wWK/3yh+V02znFVEhWOqCCyFtcIITz6ElFbtRZPGeHY5ktnyyWTvJ12EnzBsgacJEvoH9HHKQ3db0UBo2Gr8/HzBinu3MnFp19Os6NtbotFtwOy4k8dX/7ItcRuh7zpKgeokJ/Hed6fA6Jxpz0JpzrWSCUxQspwPSWb97TcW5s4Zz7FOzq576pPJe5b2okuMDYgRfsTwePyaXPv8uhT9fPF48XZMU+52V8y5SGjWStqSNLYxN519WIgbePl5oBMISo7bMsjqiKjkbIOE9UvNW7NkZCjvM470WalNS2J53qcVUo0ZzVbfr1+SJXiyU35uNzXo5bd7Y4ddE8VOFc1ZISRe1bpTKqZSB16BhHV60Vnz+om9bEbfd+F+xHth+V0y0nl9KGI8ZQ/eGlNPa4MVRcEco3l7BwrpFG5QeXk2XfYmqbE6rv1OKW9YfMwrQ23GuMQTgfYqIbMh78fbS5ptfYk+jBoRGWYEkRMN6ARJ0llKK0kZHiPh9/dSkprwCYPbpI3Ofijht1rJYtL8rZwmHbd6ariiqQ6Hn+JFMcLV5DmIpMT9WRKMWW/qCuIDGE4zzZqBadxBsW+3k+bv54H80ZYyFzpy20bLu4UGSrqfD3lc8PLhDK58axGbNon4ZOsrbbOduHLAE3ef36F6oDw8dxXtaH45ypyMuMqG1ixD6II8aAMKKtMNDAW4XrdMhxRaJL2FUBWNNJDElmKW+k4zz1n0OrWxZo5gFS4m11qVI0V6+C2LJPJ8+RYiUyLRYc5SLxGLA4aKvi/pd9ih9M7y5e//a63WQkzGYzBSuI/vXjCrrh3DH01BkXi8djOc6ZvMyueIlCB9kyOymvooFqVr4s2nzOOB/uA2+gL7viCOeyoDRff/WI3uhPxrnetTHioRYrjTfZLfUEl8cvJj71JJl6JBIZ1cLUtOnX51Md+8m03RK56oInZfWIzWlRNCM17jRR1HFuejnOQ/0QPvP1qkXHbbe/1E8de2TRl5VZtPzEw6jt6Aqaf/18MkXVdRIFQk1Es66fRYX/swc1HVQiHm9xhY6D1WwVManDvU4JhPMhJpZoqQ5OC+IsSzGS45wzpuU+x3PjcnyL/I6pS36NCDvP5ERArDytgyZVhO9/omORUDno7s9MplGiWvpfHDS1LsiI4ioDcJynvXCuY8Z5NrWTiTSy+AOkubuWn5YWks3S8z2InHOzRQw0KqfuQ7m2IHk7HaEeSECjgM84URkgPVHds305zplyZRu9CgP3J+M8MhbOmB3eCPE/GeFcLRCqw4DQ7e29QJoRYrIksk/EfblYhXATi2rRTzhvVP62uk+9LZGWbi9jO85DfdQppXnhaIDNDW20paGNjETbmjaa9PkkGvf9eMpot/bqOJfFZ93lWXTiNYvpyEXLaLT5JRHVwgNvvgEwVMh88+jiiKroqBb+08XBmmA7p3e70VebzZ4vmeVstIntgbr9VbNEdat+fT7p2M+yW8ia5Op0ZqySc76rJfW13ZpcobbbYjL1Kyo1wnHu9aVRxrkxdJpaZ0jnspqyyWJyEL8Vri0WjSgQaraLOJZCh4m81W8Qff8X+tfiX4nHhON8BLTfEM5TOOCLNg3z4CQ7zpeNB7imGF9OPVBFxXgOeXaBShHa6FEtXJxVOpPUbHbJPuNLw3mSn2+t1WUmnJftyIY8maKaRi4OmmhnRA8xp7+F0WJ9b6WbLR2Fc71ydtklkKM104p1LvpsjYdaWrLJG7CQr6SIbOae7yHbbhUZ550+jXxZmeRwBMjXaRPVGbUAUdDTnLJ9ByPBcd63cF6ckxFu42t1dB/1GtUSK+PcIG1Fb6iDCoct8W6rOgmqR46qvIaa4hz7+M5/f8r7G7Jofbw+XkLCuY6Oc1W0L1Zc8LFQhRwjOs5VB55cRs+rDo9WXOdvrdtFRsHv91PTB/WU+04OzX5jJjkaQvtf0OUsjxfV0lxeTO720LW1wOoJu815uTcAQ0WVInKqrmG1v6xHe6GO9ftqL7q3U6Na/Ibrd/CYIt6KadWIp1cdkv7UI5GMyusWnGva9BPO5bHrK6IsHmMKQhOyeuWcS82I27pkV9fLqETp3k7lhBevlJD0R6dR6/XoJZx3ejtpx5od5N7mJpsWil7JtJlijrtlgVBPwEMVrU30x4efoZuf/4gWvLeSmt3NoYlvLTDs228I5ym8GB84sbvwpBygxLtI8OOy4dR7MNuuXBx6c+NK4ZxFSD0rTPfHzaPCM4d7jSsJv5fVVU2kR6dDivvZ/XGcGyaqJTBwx7ni2kuN47z/US38vZWuc71WXnDu9+BEtaQ24zw70EQPvNRAd77WQct3TKBdHVkULCsVy79iFYPhjHOu7u30eikj10JeN3+XWTjXiPztWO4NBkU4t5vNcYt4q3DtD1lElKNa9KyPkYxbW52kNEIsXCxUQSDTmkTGuXIt12NgLieO+bqqRsLFIuJzSHGfjz93WSRTrfmSCEVZ3RNGemacNyp/u6QP4dzelS3KGK3QphrVwseVVwCoxeulAeHTLbUpzXPtjbr2Oqrf8C01FL5C5/3oFaooqo6IauGaI7UdtVTfUS9+zu1ynDeNKaaOlpD4lGULUGtH9YhwrAHStxhel3DOkU5qm6h+1/RwQPenOGhkVItxvjdyX3p7H2ospt4FQvtj8mI9hOu+MdU6CeecCODqGiP3d+wqo1r0EM55/+WYtWAAq71zugqEpiLaNWYu/gAyzvWstbK7ZTc1vbSb6L91tPdmK+XtbKGMoEZ2iz2mvpFrzyWP30OZY0uJNDt5gxlU3O6nBleDqFHCE9+IagEDQm3IDp86OnyRZfoajEvxUO/l02qD1tuMphp7opfbNhGaFUEzXlTOgRO741pW6SCc9zfiJGaRR6NEtfSzOGgqVlyoospAioOq32vuDOgxgSQ7/DzoTsblHy2cp1K84X3ONbfRlHIrTSm1kNtpJ4fZSVppaZyoFqsQolg8b3G5KZAbII8r5Dgn7oB4WzH4BgMaXNd3FQflwXWiLpjyrmJR/N3RIxqkz6iWGNdgq9kUjuZI5aAjGTxdom68nPaEolr0cBB2Hc9EHGzqZHeq+3zSbc7kJ9n+8bkja9w0KAV1je04V5dIBw1r7mAHnjrhwufRIVNGifs80fGhUi9JT+pcdTQzx0ozysxUkdtKWf6Q+JKfaSWn10lN7iYqzgwVGuNBtYxqaSorIndLpuirmDWNOtu3UoCGv2MN6AeboeR1ebQS0xLdXugxKRUp3lqHh+O8l/GrOtbSI1tbRYrPTDLjplF5oT5fY4dHl0nYtkEYu6rCeaoL26sJBXJ1VX+QOokzlVEt3vSPauG2u9JhpqOa/HTRf1w0+aENVHDzeqr+ODT5LcbUriqirroj2fZs8rZ5ac1VK+idhh/S+vaDw8K5ZLhPfMNxPsSoDRkLaod1dXoTuUhk2i2GmEVWHay9Cefq+1GrJBsN9UJdGOczGKMs30vlhThWrn2/ZjKVJUB6rlgYjOKgqcj4H6yM8+gJMT06g1I450HAgIrKpui8YZGSo1qKrE668cQcuvnYHPI7symzxE7mjIzYUS1igGMSBUJbPZ20PrCJ3G6zaOM1X5DMficG32BAwpWPVy4I92rixYrKDVgg1NOH49xIq9sS6UephR2TimrRI+O8a78TGVSpQkmqPweZb84ksroimtKuKCPuK+p1DqkZ5xyblOgSaaM5ztUaPPkxVkT+YNqY8P231+0yxMqWRk8jXXtwCd18TDZlWDVqthZRtsNMLZ0NQiifWjyV9ijeg3IduUJIz3FYhFjut9mpyLabcq2NlGVpJ59zmwg8Hu4Db2CMfHM1psUIUS0R475+OM6N0n5zvKlcwdTbBIDeK8JU1GzsZAqQV3QJ53oVCFX7Nf2NauHzXrb7qXacq1pRQWb/x97SlMrObXaxpwJZiJTbsmTOmZj9EJ0i4+o99XTG9Fy6psBKAS2f/JaQKS2vPC+0wYb7iD48meibX4sfM6wZZMm1UKDNRwGLldyBPCp2Bqi+oy78msN94hvC+RCjLn3mQevR08aGi2iq1bxjkdG1JJkbQ+5MGyGqpbcZTdVxbuScczU/MlZx0B4Z2zq48AZadCLCOaxrVEviS9XjZ22nIKolIuO8/7Pe0RNIqpMv1cJ5SFxOjgj3ij815z07/rijvTx4ED239DB6+f196Rsqpp13/Vo00rHEf7ksTxQI9QbJVOmicZZVlGNtJnsmkdnfjsE3GJR885I4bUS6FAh1K9/jeJ17o6xui4fqxknUiad3cVC+pkk3cyKTxpFL7lP7OagrBPsz+I7MOddnwogdf7LAWF+mlIiolhQNshOFJx9kdz/W+xhbmEMzKkJZpBw58X2N/vU8qturKTsz9P3SfCaqyymnHIeJsuxZNLN0JlXkVIjItdKsUur0d3bVd+rKOTeFBuiWoEamlu0jYuAN9EON1Bjdm3Cuw0Sr6mBNVIiLXKlkjO+N2n71ttI4wnGuc1SLOlZNpji2dJzrFdcyGI5z1XXO5736mil1nA8gJpXjOyWpWjkptSHWaPqTzW6ElW81zhoqbfWJAr2dgRzyW0JaTdH4IiK/i2jrk6ENnZuE69xhdYSyzkdnUNBipc5ALll9RM763eHXRFSLwh133EELFiyg3NxcKisro1NPPZXWr18ffr6pqYmuvPJKmjZtGmVmZtL48ePpF7/4BbW2tkZ8UDt27KATTjiBsrKyxOtce+21oriMyvvvv0977703ORwOmjJlCj3++OOUjqj5zDzg4xnu646eRz/cexKdMmdCwgMpPeM2+hPVYmThPJGollQLt4MVcRI96RLtOEw1/S1wqk4WpCaqZfAc5+p3JNU55xwNI8/XZPPN9XKvSKdHq1ZAu9aNptXrJlFThp38E8ZTpjWzV3cBd1XcviDRqGzKCrSS2RQgU9BPJn8HBt9gcITz7P4K50ZxnAf7LDhmfMd57675hBxtntROYqoT1om0fxm6Os59A3Kcq5niamRKKpF/l1cR9jVJb4SiXPFoS+CzOFpxnX+wsWtJtY6MN4+nnFw+/ibq7HCQz+ag3AyTmPjOsXcXnstz5JHdbCdvwEt5XYVDq03dtZ+ymruEczjOwRBRrRQGjY5qUfvM+mSch65FpqhVw4m2G0aZ+I5cadxLxrmBioP2d6xaoawy1KNA6GA4zqP7rurqrdRGtQzAca5oBqn67sqxq/q3k8Gu9GX1Wvm2aK9FdGbhwVSUUSZE8IDZQrYsKznyHES173dvWHIgkckcLhBqH2UnstlI4zY/mE2eXaFi5WaTWdQ0Gc4k5Tj/4IMP6PLLL6fPP/+c3nrrLfL5fLRw4ULq6Agt7aiqqhK3u+++m1avXi3E7jfffJMWLVoUfo1AICBEc6/XS59++ik98cQTYrubb745vM3WrVvFNkcccQStWLGCrrrqKrr44otp6dKllG6ork05OJ07tphOnzuxT2FLdSnpmT2aqHAucy6ZBh2LRPVFSwKZWvxZmXQ89nIJUL+zswziOA93RpJ8DxEZ5ymJagl12ni83Z/jraIOdlOd9a9W+c7px/vQIy5AdnKCrS76qOF9+qj5I9IKNVFkhBvoWMhrJxcH7fRpRCUl5N5lou2P22lT47lUU3YcBt+g36hFDpMTzvVdthsLVRiMJzrL7z27b9kpbTRUN04yGed5OjoIIwtjW5KbtExxmx2ZcT5Qx3mnLkJNR1fbV5TV9/7blLifVC3rTpRWpb9dGCOqhVlQWRYW1r6ratI9rmWyt5xuf7uZfveGk9qdoXMhL9PcI2Yty5ZFeRl5Iq4lNyN0vldlTCSfxURuu5ls/pB7Ho5zkIqolmjHeUTGuQ4Rna5+rNBVx7CpNuokahiMh7q6V++6aP0dq6pxP3o7ztXzN1mKlH6uXL2VauG8P30PibrCOhWaAbe5/Z1skTgs+k/gB/wB+t0nX9Ht9e3UomWSxtGN5dkhB33V690bjj4uokCotdxKZA0dc3cgnwI1teI+Fwj1BIxxHRoqkvq0WQRXYcGbHePLly+nQw89lGbNmkX//e9/w89PnjyZfv/739N5550nHOVWq5WWLVtG33//Pb399ttUXl5O8+bNo9tvv52uv/56uuWWW8hut9NDDz1EEydOpHvuuUe8zowZM+jjjz+me++9l4455hhKJ6T4xAW41M56OmWXqVEtvbmR1BlL1bFn1MJLpl7eD3dauPPCg189ZvHdg1l0QqdzR12qHs/pGA+1IUrF8ZeiCnc8komU6TOqJcVijZrH3x/HuXrepOq8l52PYHMTtfpbxH1bcWjfeXl3LLLld8JkIv71zNIy0vwmarFaKOh2icdZeAcglY7zMgM6zmU+uKWXPkhkbQN/v2KehhJVSM5I0InH8Pvgq7mmQ2ZtpHDedxvOnw03PayBpjoermfGeXLXzlJFOFcnnVJFo/I3ExHOuY3nrHyeKEpHxzmfKzMrCunbXY1CcNrZ3EHji7qd3anGtXU9bW0IiO+Zyxs6F3Ic3J+IvHbyoLskq0QUEsvvKhD67uST6Aj7q2QNmsjW4SfNZCaPf3gPvIF+cLyRXHUSHdWpd1SLGv2QKGoh5FQKnr2hjh16G/tx4XXWRvxBjTbVtwkxsj+RF4OxUjeZWLX4Ged6COcDWy0W+zzqNFTNuUTIVj63VEx68fkiPSZZ/ewvRzjOdZrAD7rdtKmllVw+jSZ2jbezR2UTeRqJGr8MbZQ5mqhgTvh3su3Z5BjlIJPNKsxr7kAuWRqawo5zr1/fSTBDZ5zLCJaioqJet8nLyxOiOfPZZ5/R7NmzhWguYTG8ra2N1qxZE97m6KOPjngd3oYfTzek4N2fwgGRcSF6Os4VF2svFwguyCTbPCML5/JCzcvErOb4XwE9c18HWq05Ii81RVnVvS5VT9Zxbk9xxrksqDnAmBa9o1rUJWo5/Xgvergew/vc3kaHZi6ghdb5NL25jWx1jWSz2Pp0nHv9JmqfN4OuPH0fuvnwA+kLLRSBBdca6C91zv4J5zxIlIOXWh0GUbGQ3+PeIk70dDvHa//e31hFdV2CqFocVF2a3hc8WZDVFeuUese5Wuit731mwSDc50jxZzCoGec6RLWof7Mojks7GlvXBAwXEzMSqhDSW1b77NHd467vqhpJT4JVW+lXR2aLW6eWKx7LzejpOJdxLbySLMPWddx9ZvI35JPNY6VsnyYcaxzlAsBgw6tL5IQ2Z1NHm2T0Lg4qHefJ1PEoyLKHx92pFDwHI+OchcNp5QXhCVe9VumpDvlk3cP8Wck+nxoDlCrUCLqBRLWoiQFNKZyAUYuD5g9g/J2tCueKZpWSOnRJGgNjFbrXawLf1thIN42qoEW540izhM6f/LF5RNXLug0Uo44NLcd314rccza0ZYzJEFEtDEe8ZDU7hVmNn/MFfbqvghtK+p1JEAwGRYTKQQcdJJzmsWhoaBBu8ksvvTT8WE1NTYRozsif+bnetmFx3e12i/z0aDwej7hJeFsjNSDJ5HIaJWc7ugOR47D2WjSDRWjOuuQGkDsnes0e9wbvk3ScxysMqoq9fFHXI6rFOcCM80jHuT4Dw2Qdd9ENinTfDfXx5860nOAaaGFQpkAZuKd6+WGEcK4US+lXxE/KMs67KpO3OWmSVkoFngDNX/MdZdY1xxx4q4Vg+OrCfaTy7GKa9ME0ym7MpraCNqIfIScV9B858cttnioqJ5p5yd97FsC4/U9mEDwUyA55b5P3EX0N/j72Xrd8yPn38k301rrdZKYg3XJCVoS4mWxfilcR8UBKz4zzRF1sfK5xe5dqx7m63JtFAFd7Z78zzvVwnDcoA/1EHOeMw2KhDvKTN6D/RFF/8uZnjykmoo3i/qqqJjpxViXpRa61nvYdbyZ/0ERfbc8PR7WwCB4Ni+YFjgKyWUNjNH+BjZY+uojs5izK37OM9jh7+GekAv3adannRMe0MBx/ZLOYyBfQUiK+Ra/Q9fbD9czjbp5ga3Z5DeM4V8dsffV/5o4ppjXVoYimlbsbI6JPUkWECJpArFo0fC7JPh+/94HGffZ3olUYv/rZd9Dbcc59n2RXpquoqyTVlddGNTf2zDjXR6ex1NTQguwsCrbm0zZbqL9RMKaAqOrfykYZRB+cQuTeTTT3D2Qt3E9xnJupUyuhKQ6riEdlxzkXB2XjmtU0uN8D/o69uHKrKIK7R1lBhHkglfT7XXHWOeeYc4RKLFi45pzymTNnigiWoYYLl9566609Hm9ubha56nrhdHvEMiCLFtqXZAj6POQPhL6c9U3N1Jypjwjd4uokP5kow2Lt8z0U2M1UHfBTe8BPu2obhPCgN+okCk8CeHyhi2qmWev1/Vi0gDj+PP5tbGoacIRHMuxubBF/OxgIkMXfmfS54w8Ew+dOW4cr6d8fDOqbW8P7YPL7kt4Hu4mow++ntg73kO5/s9sb3k8bBcTfGsjEW9DnD79eXUt7So99jXLekM+T/DVH0ygQ8Isl16k6b2qbQvvsCbaR3eIiE9kp2+KhztwCcrY5yWvpKXj5O7s+M55YcWmUFcwin8NHc/fcSMV5RCXbnNRqXkQZ/tgZ6UPFQM4bo0z2jnR4Ik06btTolUThyLL1ta3hgXplUciBqfuqt14GJWoBLz0n6SWbG9rCy1fvfHsFTeIvdRfJrt7jAqHsZHN5A6Iv1tsqs8FEPY6JTp6EtvOk3PUvzQR8jvDEhKsfA0AezPAgXo1NSRVNquM8QeFcFghV8/ON5ziPb+4Yk58lXIJs7lhb0yyuW8nGQQ4Wo8tbSOt0ie9rS35oMJufaY4btVacVUxZ9p3iftBhIbc5j6wBE3XWdZLFZAk71oxmvAHpjZpBHV0YlOHzjU0ZfD1MdXHQRF3a8URPFs65VoWe14HYk8a9t9cRK2d2N9KxM8dRukWjcp9vbU1LOK5lUkl3fyXVq8WcXveAM85VF/hQwtd4+bf6MjL2RXZEVEsKHOeKOJ/VD5MaRRUA1i0yrjpUXLzDn0kBS+i7Omacm6jt+9DzedOJ8vck2vhA6Oemr8lafBBlFGeQJddB1eXjaGvpHpRz7jw6wGITrnNP0CNE9Hjt/0BWAi9bGypCunDG2PQSzq+44gp69dVX6cMPP6SxY8f2eL69vZ2OPfZYys3NpcWLF5Oty87PVFRU0JdfduXmdFFbWxt+Tv4vH1O34ciXWG5z5sYbb6RrrrkmQoQYN24cFRYWit/TA55BDprMZLWYKTczQ+xLMhTnd5DVEvqIrBlZSf/+YMCVflk05/0ozcvpcx/GlxbQxqZQ58RjsdO4wpD7RG/kfrc3OcPHtLwwr9f3k5+TRdbW0IAsIzs3pbmvHQES+8mX/0mjy/u1YsFhs4lzUDNbdDl3rA3t4WNdlJeb9D7kZWWQxxmatBnK/W8Ndu9naX73fvb3bxZoGmXabcK10qkN7b73oKo9fN6UFxf0629nZziE2BZM1XljbRL73LbveBq7bjHl76il0dkltLNyNJUWl4oZ7GiycgPhzyxg1qiksIT8WX7af8H3lJfjoazWRmrOvkyX876/f9PS1WkB+sJuG7nIsDSnP8J5t2OqVmfhnAcmiUS1ZHRF6ekdCydRY1XY+ceOWkmybSELuhJnp48KBjhA69eKqwSdVHJyg91HPImZqsl66TgfSHEujmth0ZcdZKkWb9SCpOqS896Q55HPaI5zRbDjGIZ4sMg3e0wRfbCxWvQ11tW26DaQDDZNpG+XlFJLsIO2HFRJVgvHDpmECB4LjmspycmioNZBZpOFChp2k6mdHSMBspj3EYPuoXCsgZFNVUt3YdB4zmaOOGThPNUrlCIdrMm1cfKax/0Wvv72Z8Jfj+KgTGVRjlhZwwLwGp0mAJNxyMdCnYSpTrFwLttu7jsM5Ljx++bzjg0GqXKccx9JOq0HLJyrxUFTEdUyCI7ziGQAnYRz7md+k5VNqz2d5Mvi88dEY/O+JZKXylHHhfLNuS3X2EX6NVmnWclmtVHO2CyqbvSQrcVHLc7QecirzFg8Z9e5gxxDV8R+AHn4A8Wc7CCMRXMWw999911RwDMaFqwXLlwoiny+/PLLlJER6fg74IADaNWqVVRXVxd+7K233hLiNrvT5TbvvPNOxO/xNvx4PBwOh3gN9aY36hehf1EtqS/U1+syoASEY1U0MGLOuZo5XdhHFqZ6MVSXcqVyIJhrt/Xr3FEH4Xpl1kY4KPqV056ajHk1y3AwMs55QCuz5vSMaulvhXXp6ExdVEton/MtE+nZjY30YJufAkWF5MjKjSmayygfLirE8FvOsmXRpKYOonYbWf1BsvicyEkF/UJtt/rrOJfU6pTZKeGiW3Jpeq+O84i+hv5CYm/5ssku583VKbe2P1FlkUVaU/M5sFAhYwnyBjAYkTnnWpQDPNUZ54kWGJMiA4vOPHg0CrLPzY74vlZXzIlya+qF95AT6LeuEvrl9lxanz+J8jMswm0WK6qF4dol4wtKwgW8M3ztZO7spECHj0zeoBDNWTwHYDCpinCcZ/XaXvB1IZVi1kDEW64vJjFCznlkfQ9Ln+MleR1jEXV9bci5nUrUYpL9cpznZepWIFT2aVSDQH8p6opr4RWXqcioVvsJg+k4T3VUi/q30604qPfEE+nmvFz686R2Wnn1FKq6ZBLlal91PWsiGrWQyJoZcp0zrh1k9TaLSfHJP59MjZdOpi2/nErtXZNl/LhfC0W1DGUR+4Hk+adUOOd4lqeffpqeffZZ4SbnLHK+ce64Kpp3dHTQo48+Kn6W28i4FH6eBfLzzz+fVq5cSUuXLqWbbrpJvDaL38xll11GW7Zsoeuuu47WrVtHDzzwAD3//PN09dVXUzoRUVlaWZKRKGrOmR452/05UctyuxtwWYTFSKhLkHpz8/TMfU1dB4pd4nI/E112HAs56NJrJjNiqXp/hPMu1wV3YHlwP1S0D8HFWM6GsnOSP089hPP+rpCQwpQ71RnnWpAmmk1UyW794gJyWOJ3pMSS2q73xzoDD8APN1eQqcMhYrE4Gsjv73YXAZAosiBlf4XzMmUQVduub4FQddK0NxFOLbipFuLUA7XmxJi8zB4CR9IZ58o1Xc3yHmrUAp8JR7VEFMX2pz7ffACD79LsTN1yzqVYxKJHojUJjOD26k0459zivqJKZo3qFs7VVRmpxjxpEo3aZz51TJlJrYUllJ9lEaJ5b0u1R+eFigIyeeOradqhK2j2sV9S9aqlooYWinuDwaZKKd5Ykde7cM6kMq5lIEaj4qwMXQo7JpYZ3vd74ZxzCeecp5pIETT5NnC0ci6lskAoR8+FJ70HYewqzyM2XKSimDrHpA6WcB5hdExFVMsAarhJHAYoDmo2m6lyQiVlFFeQP8dO9ol5ZD7wn0SzbyGaeAFRRmlow6J9wr9jaVkp2vbc2blknZz7/9l7EzhJqipd/Mt9q6x9672bbvZdUEARURBQBkXxOY4bMzCijstD/w99zFOeuAwjOoo6KjPjPso4OqOOog9hQEUBQRBkFWi6oZfq2qty3zP/v3Mzb9aJ7KyqzIh7I7La/LR/XXRlRUZFRtx7zne+8x0UQx7Ec2lkCplqzFLRM2OMCxFXGpyuG22xuV/+8pcRi8Vw9tlnY926dfU///7v/y6+//vf/x733nuvUJTv2LHD8Jq9e/fW29HJ5oX+JgX5m9/8Zrz1rW/FRz/60fr7kJL9pz/9qVCZn3jiifiHf/gHfOUrX8H555+PtQROPFlVnNulPlqRVGzhRm1sU+80SC/P1fwjG6+/nYpzarWTxV4rxLm855y6d4wer+3f/9znTyeJYFCcK7LjkfdWpeEZsnU4qGni3Fsn0OxQHchz7s9mcMOGDbh+3SgwOoiQb2XSUs5PyBTK4jz9fX7ks7XnpVBCJb94SE/27sIGxTlTcq1FxTn3b+aDf1eMNRwmETmRO9ITxN+efzKGav6b1BEUaXNuikFxbkMyqEpxblfhsnEwqFXFeaMCXDdILS6H4sn7pF1/0UKH+JwL9X8t1ulbJT6VMfnWoaoV1HPzSds73CSow/h9//daHPv6v4bb60Nv0C26xZazaiGEfEGE/fQZVBDdMIOjX/w4tp+yE6X9j3cV511owYFYqm5tslwxM+LYfmFece6EP3WrnuGt/C5kMeVysABoJEHbz1XHesP186d5KnaBrOcklBDn7D6aS2dtnk1ijTjvsbnglWbvYcbe5+DhoCXH9u7rP3N9fe/uocKRNwJs+DPgyPcsvZAR51h4AAFvQNixDD9xC47+8eU489t/jTt+/W3xbSLP6Xs6Y1UVHRZm0danvRoJQYR6K0TFli1b8LOf/WzVYz344INYy+DKrXYHWomfsYk4bNWqpZUblSv0OtKqhQUVA6uQ0jzZtfP6zySXrtuQhc2EW7U4MWjJDHGwvPquhF5Ncx55cKzCqqWRgKBijV2+uknm7dZjsgouSTQSypPywOfRe99IxUR/fAHumv1KYXgQPe6VP4vhSAj7F9OCbIhlyhgh4jxTu+7FMlyFRNcntQtL66/Yz0q5tola6RXpdPG4Vbs4vtbK59EpGIp/fq8YfHbthafi9if3iwS73eGefE2306ola2L/45+DXcr/xuFiZkFFjmae47pBvpeyq2u41mreCnxspkSuQ3zO+WfRquUMPRPPzpFBOPDoxDxetL06K8puyOIFoTfkForz5axaCJR49wa9mE6UkKosfW6e+ILoIOsqzrtQTbrI+Hg5mxYnCLimvuDtKs4Z4Wln0bKV3K8VazVZANw9lxAFQMrT7cqZVAwHJdsvssuhfY88zu3Ktw1WukqsWlgBJpXDtqVGgI5XnEv7TspZ7VGcq/Y4d654Tzk0XTe4XehZLgbsPx6gmSOVovA5D6z/C6EqP2r/FI59MCZe8ss9+8XfdP9rsWrhHudrxaqlCyubh9ui4taZwL7dpIoWEKkEdVpt1wwLzON8dcU5t2qxj0zgHnUqFOdUyxKLos3grertTok/yHdX4/U3epwrsmpxyB5Aqts9Lpcpe5yDVY9F2/wFR9LxumojPzyw6kTuHSNyjoULB2JF+AZCyGcCS8R5MdH1Se2ibciCL92LRNq2C0qYRmudV5TEUjutU8gxi6uVun642sdpj1ReyJSxBH0Or3/edhw93v7gXV7wd0px3rJViwNe84ZkxILinD8rdhLnnCjiXr/tKM7lgLJOmsHT6mfBfc6dtGuZTy/dR9GgCz63b9kZJQT6fl/IhwoqyLiXiExvKi7+7nqcd6ES3EJjucGgTlq1cCKu3Q5dvvY6vX+bVc9zu5aHbV7HrA4HJYzXRIN0LLsK9EYFrgrFOb+PcrZ016tSnFPcLQlsaf+pE/w9IqY9znkM4lzxfv+jMxi/dRpD9y4gMrcMX9Hgcx4upYQFdyazAfvSx2AicyRc0zP1lxdL6j+DGBfyrhWrli7Mq73MKc47YTho+60R0q6FFkWd3tRmsJhuvcJpHA5q36LG/UFVeJw7df9Y9ji3y6qF3eOqrFp40mtn+7Rs0ySvWrdJxQNfd+yw+ZHJyUBiAX+7fwIfmZpBpr9PDBBrhTingGkyVsKkdx7ZjEck4sVCEe5Csqta68I0cU7qGzlA0KxdCxUt7SQRG5HjxXuvtyWbDbv9qRvBk06zXTPLKQgTuXxnW7U44DXPVWt9ihTnMzaSN5woakdx7meK83wHKs5XE3ZIHDnWL9R2hIcn5hyxJ8vn8/iXT1+PP/7o2ygXC4I493tWvpdo3x6KhMT5piIR+Nw5+N1Z+D0L4vtdxXkXKkFKYIl1fZFlX8eVu0kbBVNWVM+0VshwvxM8zs0UjU/Y4NygYz4c1IzHeWMxxq4BoaqtK+wWUHDi3KrinNsspWwoXFjtUjhIce7UcNB8Htdd+1E8esd3MXL7fjx/8YfAxM+B5K6DXzx4Sv3LQPxR8ffcXT14JvUC7M8cDc9Mde+mTrN8Oa+lu5BA8Y4ZTlUVusS5TQ+WOY9zbwd4nLefVMkBoZUOtGuRih7ySV2NFOkIxbmFqhonQJ0YPGFoVTdRxbdr2IcdVi12gIpUfLCYWRjJG733jUiaa59tbmgEj4bDeNTjQXHdmFCkrYTtw33ib5dQnJcwG4yjmKnAhRJQKXUV5120DVpnZDu3mcGgneZzzoNxrm5pBBUM5fdnO0pxbn09DrNE2M4h39IrnqyuWi3AODHXRpXinJJWef6zzO5IN2aTOXPEeUcqztv/LOjekp0YC+k89i/aPxSbhnk++/STSEzsEXt6T4CS29XPfzgSFoXu+PgQgp4kgp4UQr5F8b2u4nwJd955Jy666CKsX79eFBx+9KMfGa4jXfNrrrlGzBMLhUI499xz8fTTTxteMz8/jze96U3o7e1Ff38/Lr/8ciSTScNrHn74Ybz4xS9GMBjEpk2bcP311+NQAX8u+DDHRkRY3mHnfCKj4ry9fMnjdmGgVmjrCMV5LaanNZbOrRXsGOmr56ykOKfZFXZBhe3GuAMDQo1WutYVuINs/7TDK58T5/0WRIISkdpnR4Ub3fcP5yTM8BsEErdJK1SnFOe0dz+982nM5qcQiSRx8vDPgYf/D/DUlw5+sfQ5dwfgLcbgcrvgG6ne97lyDwLzVcsW6jTLl9SvnbHaekyxkd3Wwxxd4lwj+JCt4ApJ60qtpPLe6ATFeas2Fp06IJSCS7lQyyCj9bbpoiOJoJX2JUM104HCi2WPc59NxHmtOk3+aKqqmJy4tsuqhbd591soABgU58Wi9ntExjdzL3gh3vuVG/Fnn/4wKiccu6pVCxU5xntDwlJjKl6C/7AwjgzdhR7vArzBIlzFpPBK7aK15JvwxBNP4FWvehX6+voQiUTw/Oc/H3v27Kl/P5vN4l3veheGhobQ09ODSy65BFNTU4Zj0OsvvPBChMNhjI6O4qqrrkJR832kZzCoBeK8N9QReyD3TeQFsUbQ/SAVw0R6OjlUl6vCVSjOjZ1jRdv3v3ZIECeGg8pkREXyLTsXqM3bLuLDvFWLs8KC1RXnrX8WXK15+1NVn1E74fP5cPKr/gLbz3sNPB4PyCqXPMxXw1CkmivE/WP1fwuUMtqGi61VpFIpnHjiifjiF7/Y9PtEcH/+85/HjTfeiHvvvVfs3eeff77YryWINH/sscdw22234eabbxbxwBVXXFH/fjwex3nnnSdmkD3wwAP41Kc+hY985CP453/+Z6xFPD0dwy+fmaoTyUbFeed5nBuIOBP7nvSnJjLV6U5vmS+3s/dRAfCYWgGQiudyboMd4AV1s8S5UXGeWZOKc8NwUBsKMLI7gs693dk1zcC7BXTP6uFWLWZtUfmAUKdiENq7X370RXhB70swNJCAR+b/kc0Hv3jgROC0rwDn/hLlLX8h9mnvuh4hXytVvAjPVe8ZGgpeKKldO2mOjRyGq6JIZAVd4lwjsm0OyGgE3ZSSvHXM47y2MLvaUONy0qCTFOcUmBRKlZaTErsUz42QGxZVIqM1j1czMFj9FNcicW4PiSAVjnR/q6pi8oXdLsU5b9G0pDjn5I1mhSZvkYyGgzjjRWfg+FOOR8gXWnGwGFep0GcmNtVQL0pJF3IzLsRiURR8/d127zaS72eeeQZnnnkmjjrqKPzyl78U6rMPf/jDQn0m8b73vQ8/+clP8P3vfx+/+tWvMDExgde+9rX175PnHZHm1P53991345vf/Ca+8Y1vCDXcWhsMOlLrnLKqOJ9M2KM+agYejK/mm0rDdgm0R3IlUyd4nFuBXbMyliXO29j7nLBqUaU4J8jiC63Hdu17Zq1aeBdAvkMsBQ3F7zZEE6dsHqmLbG55fB9+9fQE7ASR5aEtR2Nwx9HoDfuFypSS59Uw0hMWHWNzgWHxNyFQyYmfzRWdt5zoFLziFa/Axz/+cbzmNa856HtU5LzhhhvwoQ99CK9+9atxwgkn4Fvf+pbYm2VxnArit9xyC77yla/gtNNOE/v8F77wBXz3u98VryN85zvfEfv21772NRx77LF4wxvegPe+9734zGc+g7UGsmr42C0P4KaHnsN7vn8XPvfLR7B7NlHPq7g1Wad4nBvyJRMK1iGb1cKtFY3b4z2c8jmX+b3LJFdDGGeCwYm4PV0/xm5p62QiFTposL0dlj9UWJcz56z6mzeLF3luqQNShEG2IWYtHQmBmmWcU11vtHdvxGZsDG7D4EB8iTgPb2ry4gAwcBLgpkKHF16XF97xSPXBoWdn3iX2I8rdC+WCUgEOrcUVRXGqVXSJc43gKmUzVi28YmtnwtdsYe5pwzPZ0KbeQcQ5n+DcSlJil8c2By00UkFFgZAVIpd72jph9SM7LkjJbWZj4WS7Vo/zmsJRhS2Akx7nPFgesLCxGCyiNBdcuCqABz1EnLeCw0fIroVSbjcW3FFkD7ix/3t+PPXwFiwOvKDb7t1i8k34P//n/+CVr3ylUK+dfPLJ2L59u1Cfk2qcEIvF8NWvflUk0i972ctwyimn4Otf/7ogyH/729+K19x66614/PHH8e1vfxsnnXSSeM+PfexjgqynpLzTwf29rSjOeduuo1YtLBiXPsgteVTbaLWh2+OcCoFyG03buI9L8qAdEsSJ4aBStUYd9RGL13swHGzahm0HcU6fcTs+qTwml23S9LedFgGqFOcUc1962hH1//6Xu/+IP05VLU9st4kLV+OoVgrfw5GIiHHnooOYCQYwF/Ui1k/7uUsk3l2sjt27d2NyclLYs0hQxxgR5Pfcc4/4b/qb7FlOPbXWag+I17vdbqFQl68566yz4Pcv3XekWn/yySexsFD1rm1ELpcTSnX+pxNAeacUSdHj/Nvd0/X4eF1vZMVclhOQ0rZtLSjOuVp43kG7lureVzT1exzPBh3/cdK+9UuSrESam50NRUIL6Upjn1ULK3orUuHKPXw+ndXaeUh+1fLwKvzNuce5HQNC07W1wWyHQqNlXM7BOSuZyWqhp3c0AU/NOqap4pzB6/aKPT44GkCx1i0QigWQLqSFVQt1e6ucU8JFBSq6K6zAelbShTarliUSK6edwGqGqmoo1/aizL1hO0lxvsj9tFpISkIOKM5psZck94gF4oYQYEm4E/5ZdeLA5MZiR+GCEj4ZYKskziN+rygYFMuk3rSHMDT4xYXUWLXoLhhxRQ8RXE/+8Uk8t/c57Fi3o6Wf5wNCD3gHcP+OCBIhN/xboji6O2CsLZ+7n/70p/jABz4gkuUHH3wQ27Ztw9VXX42LL75YvIbatwuFgiFBJ3X65s2bRdJ9+umni7+PP/54jI0ttd7T8d75zneKNnEi5Jsl3/RHwsnk22DVYsHjnBIBUrbR2uLkHmgYUL4KgctVeDTQlLo5DgXFebVzzyv2cLv2cSpYlFERypR2FOfGoqW9inMVvpEDDeTN9uHq+qwTcvgu2e+16qfLW6QJ339wF77zu52CXKPn4BMXPd+RdmAu7mj3/c8/epPwcb7tj/tF7P6ZOx7Gx//s+ZbWsVYxk0gjPrEHmXwGfZuPE/+2mtUaYTQagRsuZHwBxKfH0RtOIVz2YLFS1OKReiiCSHMC33Plf8vv0d+yAC7h9XoxODhoeA3t+Y3HkN8bGKjaaHBcd911uPbaaw/6dyLaqfvMKUzOxlAsFVEuldC4ig4G3csWAgjFUln8LGEunlzxtSqxmEzV3zebSiCbMvrPr4YgSvWff3ZyFuNquEhTMUehWL327nKpresXqFQQ8riQyBfwxIFZzM3Pmyay20E8nRXXzuev3htmY9CRsA/74xnsno3h8ecmsI513+vAbDxZ/8xL2RQWChnL8XOPl56BIih03Ds1Y+jAUInnFpbu96CrbOm611HI1485OTuPQY++NSieoXumJFTX8h43c/6ucvW5TWer18Bu5JI57Jt8Dol8Hj3Di6iUyygWS0jmo6iscD7lShnFdBG+PjdKbje8pRI8mSBmJvdhoHcc2WIWc/65lmadtIL909U1nWTnmcw8JmcnESAFvMWc0czPdonzDh4OykksIlNJCWPHJiKxbzFZJxU3DSw/hbxTSYOVKlatVDi5Uswu9ZdMAhsVBGbA7zknFOdmWtXtLlzwZ7TdtsKVQAQEJb6UiK9lqxbdvmsGnzh3Bdf87TWIp+N42Rkva+nnNw/2iLUmX3Jhr2sQu4ZehJ65Hoz/bhxH/1V3wFirmJ6eFoPC/v7v/16o0j/5yU+K1m6yYfnFL36Bl7zkJSJ5JjUaKddWStCbJfDye83QScn3ntnFeuDtL+UsBfN9AQ8mE1lMLCTEYDYnhtnEUun671PIpLCwsPwaF0Sx/trnpudwZL8z7ZBEVtB5eFwuFNJ0ztbDVC+qZEginbUlOaE1X5I27nKx5ffMZzL1z2DBBtKGYsr5ZEaQ/CG331ICSPCXC/Xz3zczjx29epVBZLEyn6rGmFFfsK3ntZBZejb2LSz56U7Gkvjdzr14HvMNtwtEhNB9E/B5kYxXB221g1cdMYpnpxfwxEwcC6kiPvGz3+F/v/QY0/YDreKpvdP4w/f+RSTTLz/14yimgVQshUpmZcVimSxyKi6UXMDExAYkQxmk3UFUkjnRMk6JN6nX7IDdyfehACqsv//97zdcBxoqSiQ7DSF1Cp5YHl6PV6y/f/H8IzASCeLWP+4T4pJXn3w4BgaMMUwjeoIBkTPl4W5aMNCBkssjzpmKf6NDg1j0tPfem0fpd67a7uTdPtvOuxF0jeW17+8Jt30ex24Ywv17ZpEvAxmXHxsHyENZLwoViHPui4Tq52vm+r38mK349u+qQ3nvO5DAX25ZD53IVVzivGke3tjwks2Nlc9+3WAf/jhbVSCXfXQ9otCBXYmiOHfC+qF+S9ddYmQgDq9nWnztDix9ljripmLt2vdHjPd4u+/ZEw7Bm8wLG5K+/n5bOT7CxIEJ3DL7E1EwvKLXD5/XC28ggv6xw6ttfI1ITwD7/wtI7cU630ZMbdyGfWQ3UyggV44gFJtGeMNhKOaKiPZFEfYtP0+iHZTns+J6Vypl9ES9iEQjiAaW7k2znzVZ1bSLLnFum+LcmlWLOF6hZLktpB3sZsM5tg21HgTRg09q6YlYGlOJtGj3cXICrsRCur02WDmcldqJ7LJq4YOuVvLha7vt22bFuWxVd5P3pcmJ03YULjJtKDLbBSn4iDgnxbkdRS9qrZPot6CWMyrO7fM4Dwd8GBsfgz/pb7lKTQNlDhvuxaMTOaSzAYzuGUTfgV6hpnJDz2TvQ1VxTiCPVPIxJ5DVCtmw0MAxIs7/FJLvRKEigjMqxmxZN1p/Zs0EZZsG+zCbLoKubCUQxmAb/suq4PL66snJ8OAABgaWv57bCpQIPCe+Tlc8jiXeMhmk9ZMsB1ScByXE8XwZhYrLlt8r7UqJvY9+j8FopOX3LPlC9c+r4tVPfpC6n86TqMmRvqilBJCwKVWG17NXfJ136T9/aomvJ9+DvW0l3ye4/Ag9vLcuDpEdYgS3X1/SvVLMlCpWxOcx3Ns+4STxgQtOxTU/vV9cm+l0AQ/NpPGKY1duu7aK/MQiQgNDohAx1B9AqCeIgcGBlpLm/kgQs6kkbr/vddgIN/wDfpw+OCz27t6+Xvg89rVl25l8q8L4+Lj4m4Z0r1u3rv7v9N+0h8vXUHGcgwZ2U0FX/jz93TjoW/63fE0jAoGA+NNp4IIM6pZ+8Y514k+rIJUt5dvklW7/QE2yFms/V+DxhR2DHZcDz5XN5FRHjPYL4pzw5HRMO3HOu46t8isvOXwdvvf7Z0RB91c7D+DPT9luOv9tp1tMZXcU9xun+2jrUFR7d5Uqq5Yem6xarNoqcQT4rJViWXuRuxHJA0n0eKMouEroi87C7e4Hwpubk+aEwiLwzFfFlz2DZyE8fgSK/ghy8GDfplNwzvB2uNxeYdOicsC3tIKjJzXsrMV51+NcJ/hgPfPEuX22CSsR54cNt7d4Sp9z2pD4Aukk5CCKVhdq2eJNsKvFmyvO2xl01aqHp12QrepWNpbwGlacc59zKrxw+wF7CkNWrFqWrntON3HOPCSJYPr8lz6Pqz9+NSKh1jtcdgxXB4R6XF4Ug2Vc/Mpf4w2v+zG2P3FdlzhvEcPDw6LYcMwxxxj+/eijj8aePXvqyTP5lC8uLh6UXFtNvokg53+cABFXskOKCr9WC13jfUukkV2elyt5nK8Wg/DhYrNOepyzYc2qIPdxSmiLtSKRTnAv9XYSZ5402dElFlPsGznIuuTs8DjnQgN+/7aCDf0RfPo1Z+AjrzwF//SGF+OdLz7GkWHwXGgjB4T1hQKWyIPLTj+y/t/TNjzL8XwFp156JU540zsx1BcU3tmtWLVIix1CgcKlYhHFmaSwb1HtkdoIGj6q08PXLpC9Cu2tt99+u6H4TN7lZ5xxhvhv+pv2bbJbk7jjjjtEwZy80OVr7rzzTmHHJnHbbbfhyCOPdKyIqnp2TrteyTRvw657RK45ZvMlI+Hp3HBQw5BTE7/LkWNLFnFPTev3Oec5oJmhrI1r75nbx+v79693Nu+yVAESY8n7XGWsxAV7Ou8jHh/wuMEK+P2ms+jF44OIZY9zZy1187N5XDT6erx+28sRDFEe7Wo+GFQivLH+ZTA/Dd+QB+mefswPjuPZ9SfAvW69yMdp3SyVVXqcS36jgrDP2fGc3eGgGsE9Ks16nPMKml2WDxK7ZpfaD7cOtkecc1/FqbgzpMFKHuetVjjlQmxXIjWnUHHupFULJw7Cvs4tGvHjqlYG2D0gVCrO6Z41aw3lpMd5xF9d6yjhbkdldvgoEa0uUaYphlwYGoxheGgegcwECqXugLFWQBYsz3/+88UgMI6nnnoKW7ZsEV/TMFCfz2dI0On1RKzzBP2RRx4xqNso+SYyvJGU7zTQM0rEqor5EoT1fUvFn/2xauursx7nK68JtCdKj2g+JNXu85WfgUpvTcOgaSZosCUZN+lxboc9HB/EqkK1xuMqPqy6U+MlilOPHOtHb8hvLNTbLFIhLLbZEbkS+GfJi2e68CQbRNoXclUHh7lai0Go4EHDQA/f9yR8u56Fd9ce+mBF0q0y8eYgUv7ZxWeRKjizLrcLslF76KGHxB85EJS+pr2XSIorr7xSWKz9+Mc/FvvvW9/6Vqxfv74+n4QK4BdccAHe9ra34b777sNdd92Fd7/73XjDG94gXkd44xvfKOKAyy+/XMwj+fd//3d87nOfM3SDrRUYOhlrcWU7kHsPceZ25H5mh0k3rr2y1u/kcNC0RTESdbdT9w/hqen27araBVcmq+jof/lRS8TirX/cq63wQgIDeWRdinPexawa/B6VxVOriHDifK0ozhk36MSA0N4NvSifNIzIjgwqXldVMLTSYFBfL+CrFrd82QPwRD2o9PuRHQ8i229ca1UWvuWsOHqegn5nC95dqxaNkGQlbQJe1o7RDsZ7l1Rrk7G0LcOWCDRcaM98dTjJSCRgmFbcjuKcQCq+o8edVyzwwgMnNVeC3PjtIp45YTFM5E3R/MblZLeCio3F53GLKeXUPc2JCJXg5IRq4pwnvzHNA0JpM5EVfB74mEGIF1y0e5zzQYC0xlTEpG6fu/X1hoYYUvJIofaLns3Av92HQKQCTz6FQinXMVZRnZB879y5s/7fMvmmAWE04POqq67Cn//5n+Oss87CS1/6UuFx/pOf/AS//OUvxevJNoOSakqk6WeIDH/Pe94jyHIaDEo477zzBEH+lre8Bddff73wNf/Qhz6Ed73rXR3Z0r3c/jDcY/1cNzDFOdmWOYFcjYRuRXFOATN1OU0lMobOJzvBO3PUKs6Ne6HKY6ssyFLhQs6HsWPP5oXLHgXXJOKv2hyJTkO7iXOLHXp2dLi1OoOn1fi0lcHw2ueU5Ap4fLLqjd8bdGMwQiZpbrGPt4LBSEjsz6WgG17Kh11AaWIRpSN7lLZ6c9BxKalfK4rz+++/X+zJEpLMvvTSS/GNb3xDDPVOpVK44oorhLL8zDPPFPt3MLj0THznO98RZPk555wjOgIuueQSfP7zn69/n/b3W2+9VezVVCSnLrRrrrlGHHOtgZNmERP5B98fJhMZbNc0IJHbhVDObWUmFO3fFPuTSnjOhrV3OWQtipEo79s23Iunp2OYjGeEoMHqetiyyEsBcU7WJkeN9eOPU4vYv5jGYwcWcNx69fMyEiynJDsiVTBa/uhUnC+dv9WctVF8RUiz3LKTBXdccW5HkbsRm8/cjMT0DuQXF/Bc4ChsC5aqVi0rIbIZWHwE7vwc3JUCkv/nWEws5gRfw/NtlYXveC0/o1VS43LQErrEuUZIspUHse1iHU++bVRu719M1ZVfW9oYDCoxxqZJUyLeCZCqX6rwtbrYyY1UtniTp7IdiaCrtpkk41k1Vi2MQLEDKjYWWnzp+idzRW3qOx7kqfYW48EMb4nXAbpG0qfPql8ct2rRrzgvGgb4ffrv/gGJVAKf/LtPwuNvMfEOB4RiIVPIIOLuRSETEAo2FxEGxaRQl7WqfjuUsVry/ZrXvEb4mdOwzve+972iRfs///M/RRIu8dnPfraedOdyOZx//vn40pe+ZPB6vfnmm/HOd75TEOqRSEQc/6Mf/Sg6HcZin/VEmawgJCYWnVectxKHkGqX9mtab4kQa7dgbhWJ3FIy1RtQFx3bTYqmmaq9XSKE9kvqlLGjWM/XX97daBa0Z9P+M53I2qI4Nw5Tt0ic833PAeKcd6VZVeDxIpnu++ihfXPC3uOxm2/CcKSM3EvegZ5o677EQ2GyxXJj/JhdOOFVTyEQyuF3f3gWY0f8L21WLYI416Rm14Gzzz57RZKfnjvaY1faZ6nYfdNNN634PieccAJ+/etfY62D2zSY2cOOHuvH3buqFnO/eGq/dsGaKgEPkZ5EdlIBmsh4IqHthorf5cjRPkGcE56eieHUzSNYC+phifOO3iiIcwINpdVCnPOit8I4bYjZpuj0ypeFdRILqCjaEyLMlonHNp3cpeCkpa54z3wed970L8jkC4j9xf/C2S89u9pqsxLCmwRxTnl2ID+LgHcA8/knkC8lcN+uWZy2vdqBrHL/luJD4uT9osLuHLrEuS3DPsxf5nVMcX7AxnbvXXNLNi1bGAHQKkZZq7v0jXUa8sFrp3ptaJ3OlxANum1RnNM5Wg16jH6p9iaCZlvVm13/KnGu5/zTGhXnRqsWvZYhvKXOql8cv29yhbJtCQ5Z+jz4+wdFEi6HVbYCShrJruVAIoZCyI1CpuYtXyzDVUyIzduDLnG+WvJNuOyyy8Sf5UAKti9+8Yviz3Iga5ef/exnWGvg3Q8RBQkUtc+SvyqtX85ZtZSXut5aKPpyuwvai2wnzrUpzu0rBh6UjLdZkKX7hgYhUWJJKkRpn6NbcR414QXcDIPhoCDO6RpQ4caKbdhq4Io4y8PUnbZqMdERuRz8rP1bdzL+uz0zItFeeO4p+HtIPVtoebg3oT/sh8flBkIV9I1UyTJ/cqGqQtdEbuu0geli7SvOX7R9HN+5f6coOv3mmUm88dTDlZGqOsnb4UgATzPSk3esOyOaMrf2HzHaZ7CBso04V5QDPn/LiOg4pjX9/j0zosBrdX/S3S3GYyX63KgAorP4LWfOUaHd6jyhZopzuzzOra4Lfsbz6O4Oa4ZSqYQDO58QuWFYxg2rfR7hqh0R7dHB/Ax8rhDed+NHMJYoIb95HPjhz0UxXKVVqhQWRPweVCr2K/M5uh7nGiGVHlaUrOTBKO/hA/GMI4NBN/e3v/kaPM47gDgntbgcpNGOHxjf+HUnU6QQkIuDik3WUY9zg8+d+Y1Ft8e8sa1Qz3DQxjZs7YNWLCvO7bP4kQkOrXHRUEB4db7jHe8QgyrbweEj/aJLIx/xI5etXfdiGe5Cspsgd9HWvUiIKEqSpeqc2lKdsH+QxHmrXW/c7oIPXnSEOFdI2nNS1A7inL9Hu4nV+lqXIZHmukUHXLXGk04rGAgv7Xu67VrkPUpdhBGLz6zTVi1ccW7V49wQ+2lMxilm/cP+ObjcHhz/itfhjZe9TmSVbRHnIfJmdiPtWZqj5M+kq8PFdCrONQ4e7cJZWCW2KGc587Dx+h7662cOQCdU2YVQ0dIOtbBuUvGI0f7617p9zq3s1cuBRArnHrVBfE1alf9+ch90xqsqFee8e4t8yHXYWVFsI2M9q3sdB+Xwki/T6XGe0jYc1H5CuAgXjnj5a7Hj3IvRw9aPFRGuDg8lxXk4P4vecBQ9uQqChQpCte5asmrLl9RY1NI9KD3OIwE3yugS54ckaGGQ9hhWiENSHUv19mS8GkzaTpybsGqhxUAmUJ2gOI+ZVPMYh4oV18xg0MZ2Xbu9swztehYV54RiuSKSNNXgBQWdVi1y0deFeYXEOQV95FNriz9qTeUbIW9cn0/4b77kJS9pmzinQJs28RwR52m/8EGr0LkX4t0EuQvHWnY3sAGhEw6ozuXzu5q/+XKKc7sR5wpohSoqriSzxarFQjLOLX50dypwVZaq5JtbhekkzikWnqvdo3TfWp1jQc+IPIQTxLlU4KlQnJOCTyrZdO7hjx6YFzGU2+PBK88/B6eecQo8Xk9bw70pXnHDhYR3SWXqL1Q/12JJn8e5Lv/0LpyHVOPSvB6zalZJfBL++4/7tebeVgdqNrPZmNfoT63bqoXWPzknbfdcXEvu1/TaK+wqOOeIDfVuMWn7o09koLYbQnYtk/0nL67ruOaqCvbc3rXxPTq1o54Q8Dg7HDRfBsaPeR7Gjj4ZfaFWifPN4i+6uwOkOH8miccWLsDv5i9GYnqUZOxVxXm5oGxNkVa0Yb/z88q6inNN4MFqq0nraj7nFKAuMAJYJ+n/XI04H40GTVfUxqLV86a2Y7uHUzaCE5ftVDjtbPHmfp1WB101qgyzRQeHg1oIBMPM51pH4UL1YBiOPuZTqt2qhQXJVj3OuWLNLo9zq5YQ24Z6hff2bHAW2bRbtHKRd5urlOgqzrtwjDhfz4hzmhvi1HDQVouC3GJtNml/wZsPvFKpojIOB9WfnFhp/zYQ55rvmaSGQgVXPeokb8imRYpT+H1rJemW8Z4TVi1GxbmCPbx2z+skzu9/bqb+9UkbB8TfVMBuZ6YIWbWQ4jw+OoiQJ4GwN45w74xQrOVKeu4fUpvT7JMuDk3I9dfKPr5lMIrDa5Yh+xZTdc9q7apnix7nTnaMNf4uVsRI0q6FCDMiz3XBSOKqywH7wwFsrHcc5pQXXvjerZJ8bhRf6djD+eBO7kuuAj21a5HUOBxU5Twko+LcfuKcCjBDmMKHAn+LVyQ/Cuz9weo/FKkqzklpEMjNwFcsIZldh3SpH5l8BJ75RRED5It55bFRqEucH7pQ4fPVbEDoZEz/gFBSxsmEhAgps5AVY8LNj+7BXbsm8fD+OUOl1C7wB68tqxYb23dnFSvOSTlM3raOWLUoIqR54UJHMmtQnLepcl4NFBBIe1rdw0ENinOLHuf8uuu8b8qVSr0YEvF7ha/5rl278Oyzz7blcS4D9KFwAOlBPyKlGfg9Gbi9RXiLqW6C3IWJ9ks1icgGZnO234a9m4MSNUmateozzfcdXsi1C1zdFG1jn+6kfbyxyNuuio13KRBhs5atWnR6pO5bTDYtNliBjFWcGA66WLtWpL5WUcSQz7yuPZz2b/LvJfhcQDi7gIm9E2LvJtK7PasWF+b7h+B1F+B1FRB0pauJt6JW70aQ92rXquXQBO17ci+3ItohvPxIpjp/cj86XfXcEYpzRTadR4zZY9eiUzwli//UMa3aMovP5NFl1aLL8iepqVjBiXh6pmiP0gF+7a364gccdAYgJLN5eGYex/z0PIZyfwSyLXRH+HqB0bPh2vQ6ZEbOQmjEj1JthlKmFEZpckIozslSRcUsES58DfmcHQxK6CrONSGj0AKCDwidiOtPvnfNLlV3Dxta8h204nP+g4d24x9/9Riuu/UhvPv7v8GMzWo2s4OXwmtYcW6X6kinelO34t9Q4GLqdhWgFlF5r8nBtLrAg2Su+LN63+gkzukeqbDAjxTi//N//k9cffXV4ut20Rv0ITMYxZjrjwi60/B4CvCUUt0EuYsW78eCVsW53VYtVPyWz1erxDkV3WRn+6HkcW7nPm51xgd5nLtsumfk3Bey5iKfcOVWLRoLxnsXlq7NpoEepfeJEx2SssjQH/IpGZYmO12lCEY1ds7ERDcp4ZixKK754N/iH6/7RzHc2+v2tmVHSc/6fGhpAGDIVUAFZHepJ27KFp1R43ahH0Q+Ude0in389G1jYsA34d5npw0CLF18gRUijhOe82mnFOfsd7Fw/Y9kA0K1EucahoNK8AKoasEgL3rrJM51FL91WbXw4xFnrit/5cUWq4p5Owd5N8N8MoXbb/p3fOzbM+SwUrdhWRXP+zRw7P9GYd0rEBzxouipcU3lCLIH9oriuRjCrWCWiOTvKCYI+52nrZ0/g0MUfOhg0OJizInzAzYk38/OL/mbbxs2rzh/3qbhpsN5aQDCYwcWYCfiJgcv2dnizYkKvnGpSJ7sVpyraj00DHXLq/8d+DGtqCNW8zmngFtX9Zt7yZKnngq1mrwWlHRrq9ob1I5eoTobHBzEwMCAKb/a3pAf+dAg8nMuzN/rwc6nRxGPHq/VqiVXVN+C2cWhMxyUFNzSa9huqxa+5rfa9UZdSgM1mwjnFee6hoPaYNVS2/9cJoQT1Lorlf8TMb1zbWS7NyWbVj3Cm9kF6FQ9cv932RJvFZLoIXsAnb66jaD3kiT0gKJhaTIhzxVKWu6h3zGbludtHhH7dm9fL7wub1tWLbK7JBPuRzlXvQfD7jLS+bRIunXs36q8V7voPBjVoNbEMFTUOfvw9eJrIuN/+fQE9HcomT9nEurILleysnICi7ViqcsicU5dRNKq88mpRW37oC6P80ZCW7VXuLS5bHwfFdDduaBDpNKMyOZ2NiqRZtc+pFJxbmPMwePhwR6gP+KGh2abRVokzmsIeAIIDHpQkvEGU5wL4lzB/l23cKxUEPQ7n2+rZ4q6MJ20tmTVYovinBHnQ1EU0kstse1g61AUX/gfL8Jz80mxgD0yMY/fPDPpyPTgxaw5q5awjS3ectCVKqsWsKRddZuYzlb15QsXehXnqoeDLvmcJ0ECGHoG2rn32oFUl5DaT6VaTa5lqoObg9r1Aj4EAgF885vfxMLCgvi6XdC1zUSGUIi7sPA7L6aKfvSGt2hRnBPRsZjJYjK5B1sHNiAaMN+Z00VnwKCCUZSI0LNI+zftgVOJjLhvKBm3A7zLqFXFudx7SGVERB49+zrWxdUCZFJA0xqUXaPDQSU5TySImfWYSAMazkrXnz4LVYX0RsjEskdhkWLQpuGg+2qKc5dSqxaP4T6xOqSzVXBVXzvCjpUgn3lKM2nd4V6qVlFhNi10e5++fQNO/+qNeGDXAwiFQm1ZtdQ7zvwBPBEdxUAwgUTZi2QhiZ5Aj9i/PVB77lTw7uLQhKr5ShLnHrlBWI0Sbn9yPy46fouSGHt5a1fzsTadFxUuqeitw2JjNZDIRq7LI5GgpViHfpfDR/rxh/1zIhaZTmQwxkSEnTzbplnXXFKx4lwKj8iOVVW3WLM9XEfnISf9VXucR2zgbOrCCJd13kAKa2SR227kK258+O1H4zC4ECI1d6uK8xpoEHgk6EW+xwUskuI8jPL0lCiei8K3gvxbduwLxTn5wjmMruJcEzhRaVVxTmSYVI+QAkn3xicV5yM9QcuVTEr4SHl+1o51OHXzcP3f7bYOMSrOA505HLS2QdFCHFG0gcvkSZfqSHcwortwIQtcRNSQ2lI1+kK+pvegSlBSLAMRFYNB7ShYNCrO5UAXK6DnOh8exKdeNoi/e9U4bj5to6h6qxpQwkHq4fd87y5c85Od+Nf7dik/fhf2Q64vtBaoJLclqUfL75QNhe+m8xvaiEFo35ew265FtjNHAzQwUF2AbGcBnL+HWRKE+5zr6lSgtmBp46HSFoeeHZkM6yLOKZaRinMq9LRTGGp5poqNPufz7DlTpTg3FL8Vx9s0r2EyXrVbPGqs3yAIINK8HasWef+53G6UJwbhj4fRn/YhnUsKv/RiWe3noCqZ76IzwVWmKqw3iKw9fv2g+JqKmTpycCvWXsuphSknsDvPPhBL1/eUTWy+i9UBoTrtWuS1FySowuJi45yWRC6vp1uM1k7FhZxBW61a1BLnfFgn7z5RCWnZRPuW1SIa56OcKHbRNRrC9JJ3uX/pmWsFvlIGPa44Mn21br1yEJWJ6nBvGsCtQnEurzcxWJ2gOO8S5zYobq0uxkK1Vqu0kmqt2ObgvHYwsZiqq8G3DatVURqmBwszJfvAvenaURHZNVSMEkG5aJK/uaqNUBImUnVkF3irupWkVvdwUEkK67BpIfCEkvvsaxsMqog451V0XTY/BmsMBaoDmmJf8gVRfOo8DP3XqxD+yvFww418Wf11p6C1UiNDVaiaunAe8n5UvRaQZ7UTA0KzxaXnq501eKRnaTbJrI2zSGgPlO3MKm1a7CoENov/zHZbbRzQT5zr9EiV+xAR5zoK9lTQkfvSJnatlBZYbPQ55+3wqohzOaeEoJpA+31NbU44dfOSNzmBitXtW7XQ/efC/U8/H/fdcw5+d8+rcfjQ4ShWisqtWnLFgijady3WDk3oUBBT97SEjmKgygGVPAewe0Doc8zmdSOLe8zicEac72bH1nHt6bqrJqCjLK9JMJW1SuJcZdFbgq6FjJl0kLnGZ1Tt+few54cr21WB9g05rFJFR9pIdKlIQYU5u5HJpdHnqlonV0KbWv/BxE7g9peh967XYOvszUgPynXHBdf0UmypUnEurFq6w0H/VBTn1okVaddCOchMQt/DtXtuaXM6bMi8v3kzBGrDA5yYHiwfPGprasc6xzhUTB/ZT61o5KtJUNmWHWQtXHb6nNcVdyZb1ZsFkTquv27inG+sqj3umqnVaLifCnCFqi6bH4PivDYc9O///u9xww03mBoO2h+kwYYuwOOB31dATzgO/+I+LYrzqpKBGscqCCtuNezCGUjfRVU2LU3VwzYOCDUoztsgzocMinP7Em9a3+VQN9XJIKmgaf6DeB/NSmISNkjVnVnFo7HYktKuzIwoVn3JAdXF8lIxRCV4MWFDv5rBoE50JjQrfg+wLjVl3qmKY6cJ1jlzzPiA2K8/86nP4KZ/uQmuoqttAkoozgE8MX0ydj/9Akw9cwIi3qjYX1UrzvcuJPDpW6fxqZ/P47sPPKv02F04Dy7IUCVq4J2cixqIcz5rySpxzvNHuxWsZEknsal/qQBvFryIqGs9lvGA6sGgsnPuIJ9mBSARnORQVBe9G+8jyi9VFxmTTAkeUa04Z9dDxz1DcarkauQMMysg7qG3JhQhOyK7UY4/i3/6yTxu/Mk8CoENrf9gYBgoxMVeHyrMIjC6BbH+ccSGN6B86XvrL1NR+I5nqvcLxQMhn/N6727Gb4vHufXLzAeETsRSBt9zldg1Z/Q3VwknpwdLxTkRme0E9XYlUrwlnrfKq0yeiABVWwrR16ouwYscOq1adPn4cgJaV9skV8DIwX5qPc713Pfc45yeM2rLvuuuu1AoFMTX7WIgHBLJdzHoxmVv/hmiPRlEdj2MPSM3isBPpZqE1CPVYLKixGamC2dBFmUyeY0oDuS5/zJ1dNkFvt60s76RN6kTinNdg0EJIrj3eYQCSfdwUBUDp41WLXq6FFIah4v1h/0G1aPq2R57az66hE2K/M2d8MJvRnANsGvXqUPHpKWSjKlpv773nnuxmF5s26alXihzudBTTMMTy8ONCkoHFoEBNYo1DppNQgk4/d+jWGHahfPgggw+s0AVgatbcW41Z7JrxsRqxLkKxTnPn3R0ilEMLwstOuY48dkhKgvIhqK3JuEOibD2LaYESUznrnIP12nVYlScFzrGvWC1Lk8ST9LzauccJIInsx8PPJURHdSu0MbWf9DXB3h74CokEMxNwzvoRzocEZZHKVcUfRoU58Qh+rzO79ld4lwTjMM+PEqJ8wMafVJ3z8XrX29TrTjngbyNxDmRInLTanehMyqeNRLnjKDgir+1QNw2g7xWVoMRnR7ztEGRIk5X0HSQRYCmRNxg1RJZQ1YtDR7nXq8X73jHO5BMJsXX7aI/FKQmMYRcZRTSfrgjabhysarPWqUEr0vdZyxavWsWSBFFyVkXzoGeTampUb0WjPeGRTBJdRZbrVoK5oeDOtE6yhVZ3BtUFehzJeJctwWHse3e3NpAXQ8Uq1CSpktxbihUaLJqISxmaH+KalSc67FqscPSx47hoDoU59w7loourkoZl73tMuyc3IlwIGyKYKK9e/3iBLyzLnhcQGnXFHDKoHLFeTy7dK17gt0U+NBWnHuVK84XNFguytyAiCHZGWUWvGPOTrspgpyPRmSuCsspnofkCuq71CkHLGnMAQ3DQRWSuI3dujpAlrESNGxWJXHOi/aqrVp0ix2lTQuhV9FePRoN4pnZuMhB6FrrEsY2Q6R0AG88p0/kJ6GBw1r/QZcLCG+CK/YE/IV5eHf4MXfmCIq9XkSPWrJYKpaK6jzlab8WDD8cRTdqsM2qxVqrC3+QJjUS53sXkvXkWbXqy+hxbp9VCyXLstOo3dYasnahQIY2V50t3rPJXFPFn1LFuU1WLRSMyFYmq+oJQyKr+PrzxFj1UJhmx9VleaLH49yr/b5p9KIksvzCCy/EwsKCKeJcdpOcuS8L73Yf/IMu+IpFlIpp0S5mRgm3HKpBMKnWKsqtPbqwHwa/fcUJFKlHRqMhTMUzoluMCrlWBwppVZw7RZxrJHL5XqTbqoUf30oyToQwJQyk7qVkTbVqmyfyXBm3FoaL7V2sxqkuxcS5XTNtllOc0+/Tp+iz0Bl7yHuH1pWqOs6N819xPoZ3DSMUaN+iQZDvtKNGywj3pxAI53Dgmd8hfMr5ShJvDj6kT8c600UHEeeKRA00P0er4lyhXYhTdlO0V0mCa8tgVEmHJ49bdBQyVQ5lbQbOofAuHaXzSTR1vDYKKA4b7tVSeFVdsOD5mI7hoHW/7YYZZlZA+YEE2bXYSZz/0f18eI7rwZBrCsHRk9v74fAmuOJPiL17cEsWz/ZU5524RqrrJQ0ItTpjjPikpfXRBbfbeasW58/gT2I4qFeJak1Cx1TvpRu0ZJjMrRIB1n5ip/qZWmDMVghlizdBZ4v3XFq9T7VdxG0jeICjVnGu9vz58XRZteg8/2aBvCri3KCU13TfqByIJINUl8sNRHpQyAaqReliWbSSqW73rg8H7SrODwnoGCjWzHqDBm/b5TtqKN63URikArf0XJyzVXGuz6qFf67UZaRzULax29ACcc59zjVY/HDVWkRx8q3T2oAKT/J60GAtK8PHO8bjvDZLgIq/XkWJIbdGVB1vy2c10rBW0lBQK1Yt64+YwCvefTNedtltSE7/XBwrV8ppUwxGNHUadtEhVi2qFOdsPat20GiataTgfnTKbooPBt0yqGbuBK2FPmo/0cQZqM5BGkFxF4nvVCvOdRa9lyNzVULel5R3W+2wWMmqhRfRVCHOOk76FVq1SMzYaI9ImCoO4JHKKbjPfSEQbsOqhRCpDhOlbrFR9wymp78PPPoPuOOL74d3/wExKDxXzCnbr8MB6kZzvsO7rQjtuuuuw/Of/3xEo1GMjo7i4osvxpNPPml4TTabxbve9S4MDQ2hp6cHl1xyCaampgyv2bNnj1AWhsNhcZyrrroKxaLxBv/lL3+J5z3veQgEAtixYwe+8Y1v4E95OCgpMmQye0ATcW70nVK/GBsU5zYOB42zTcbMQic3VJ1tbzrIz4Na3ewizhWSUDpbp1We53II2HD9ZdLd2FLa6R7njWQl+Q1OTEzgwIEDpobRUJBNQVM+4kM+U33OK0ScFxNKBpRwUEAmz1GXx2AX9sF4L6rf+/iwxwlNntUrWrW0GYNItZH0XLSdONeoONdNKKgqwnAltQ6LH65ak7GlKvDiv2rinApPcjDaRoWDQQ8inWyyOaBhslKpqUs0oTL2oH1PEjfyOaV/OzBxAPPT83Cb0GNVh4O6kPQsqRoD+Uw18VZMnMcyuXqnd9eq5VBXnHuV5a7yWIvpvPJCoNyrVSvO7bSb4v7mWwbUrcuyMKrjd9FxrzQK76QIQKXinJ+3LqsW3nnIZ7CpgNw/IhquuUFxzixh9CjOfcqLFFM2DwilQmNmYRblxEL7eXe4SpxTB+2wewZH/vG3uOY/7sYrvv1rBB98RJDchXJBGXFOl5tiAqfR1hn86le/EqT4b3/7W9x2221iiNt5552HVGpJDfO+970PP/nJT/D9739fvJ6IkNe+9rX175dKJUGa0xT2u+++G9/85jcFKX7NNdfUX7N7927xmpe+9KV46KGHcOWVV+Kv//qv8fOf/xxrBSoGRDVCtm8sZvJaNhHdrdI6FTCtKs7NTEG2o8WbB2O8LVCtV7U9QRTf1K36+0urHB1kh9GqZe16nEtigghcXpzqdI9zrsCnZyyXy+Htb3873v/+94uvzYCC1ELEj1yNOEepDHcxodwntao4r2rOdRQZu7AXvKUzokNxbiBB7RkQarBq8ZojziuarDZWtVDQ4HGue9C0cuKcDQglix/VSGpVnAeaFnZVgAaWSai0aXFKcU6xn0xXhzTZ9KmMt2nfrtkC10kb2q+vfPeV+PRHPo2iiThTPu9xP3mjVuO9YDkvFOf5Yl6Dx3n1PbqDvQ89GNZfhZ2kcmgvxdtmhB12zVdxym7KqDhXN9NC2kbqyEMM4ikNVi18jaT4RtV9w0l4rrBWCa6CVq04l7m3jtyJP0NarFoMw0HVcDWjrEgxbaPinIp26UwWv/vWDfj11z7Tft4drinOXW4MYhaxSD8ypSjmc8OoTE8LqxbKva3c90vXuwLSvaq0XDWLts7glltuMfw3Ed6kGH/ggQdw1llnIRaL4atf/SpuuukmvOxlLxOv+frXv46jjz5akO2nn346br31Vjz++OP47//+b4yNjeGkk07Cxz72MXzwgx/ERz7yEfj9ftx4443Ytm0b/uEf/kEcg37+N7/5DT772c/i/PPPx1pAo7+oimeB7FqenIrVVecqPacOat3VoKIkRShxoBR0520kzhOs6mhmCnJji7eOiccLtfY/8uRT2Xps8Di3SeXPWxn7LW4sVLGn60+BgupiEe8KCWka8Kh7IC5tfJLYUtupoN/jXAatgdpAJFp9IpGIKKqaBT3f6agH+XmvILYpifeWMnqsWioVUdjhBcEu1iZ0epzbQYI2A39u2+16G4mEDIOrx5giRheSmgv3dinx0qqsWnixZVEzca443iNLPBnrqVac71tYuhabNBLnur3wJea5TZ/CPdyvaTjocjYB4UgYRRRNJbZ1cilYHSrmclUQclXvG9q7qWOMknBVHagUV9L+3e0WO/QgyTKy+FCZq1Eus38xLeZzUfFIlULZEHsoWIcjBuK8ZLvinGJ52ruS8SpXYRUydtGRh+i26FsqCqbE3C/qlFLhQGAQemhSnFN3Pj1DdN40sFLHDDQd6y+pn0kkQc+oSnscnYrzoZ6gmLVJ/PJMwj57RHqm6JPwBoIImuFAwtKqBRjALMbuOgX3zZ8k/ntzzaqFiHPaw70uryXinPbrsN+19hTnjSCinDA4OCj+JgKdVOjnnntu/TVHHXUUNm/ejHvuuUf8N/19/PHHC9JcgsjweDyOxx57rP4afgz5GnmMtQCZmNEmomrzXs98zg9oGBBqTKR8WonEXMkZqxYzxLnuFm9aEGRyaZVoXrFd16bhoPNMPa/COkSXx3xG82AYOzzOqaAgp8IPhoOavPH1WrXIaxQMBvHd735XFF/pazMgwm0xnEc2U7V+of3IU0yqt2oRA4crCPodHu/dxZpIoLhVCyXf9nuce00PhlKZNDnpcW5ciztfcU6Jqxxup8XjnJ2n6kIFJa9y71fdsbCvNhiUsHEgsuZmkjRijinyVSrOdc23MRDntfuG9uuv/+vX8Xef/ztEQu1/JpQn0b0+u3EMUd88ot55RPtnhD8qJd2qOsbKlbLYu+WuHel6nB9ykPYMyrtoNA0INSrkFSjOHbCbIjJUdtJt7I8oLVjIdYwKFiQUWkse5417K++q62SRAYGKi8M11TmpoFWp5fm567rmkr/SwdfEM9YcDJYTlcq9307FOV0fj8+PF77jQ3jz317Xft7tHwA8YTGfpK88i+QA6+rcOy+K6JR7W9m/pWME3X0hX3XgqNMwvbqVy2VhofKiF70Ixx13nPi3yclJoRjv7+83vJZIcvqefA0nzeX35fdWeg2R65lM85uKWgzo+/yPk5DVUZVDB/mkXR0+54aAWJP9gFTB5NeUVYveyd6UoEnPd1Ue1U0tNzQRoI2YZ35oKjw7ZSBIi7zSFskGqxAd0D0VngfwspVUBfg9r0txLoNWlcETtc7lhvuw3v0oIt4YvIEsPKW0csU52VpVUEbI53z1u4vO97qkQF4Wbe1SnPMOl3a7Ivg+RNZwdkC3VZwdNhxUxLx/z4yS34MS1/W1TgUiV1XvH7JQ4fe4lVl8NbuHKP5S6ZMvrVqI/JTXR+V+7bLZ5oBb2ahUnOuar7JSnkD+5mYT22jAj8XwUDVDpldxKZYAAQAASURBVDXTXUIyXy16q9q/KYGvqnBdCPqqnW5dHFqQalzValY+H0slcc7XdRWxB93T1MUpjm3TGrZ3ISnUsoTNCv3NG3Mz1bmILYpzTpwr8jnnsZIuj3PCaI04J36C8yid3N3Jj8tnUalWnFOsoDJukp7yVPizK/ZIW7XVdbmAU25A+gVfx283XI3UgA+V2paaO5AQ6nAqVlsRrnGrlqCv0hGKc9N3LXmdP/roo8JCpRNAg0uvvfbag/59YWFB+KrbjUQmi2KpCC884hxUEPmiEbJUvdF3T81jYWEAKjG9EKsfH4WsOG+CyiKEm9QjpSKIW5XH1425RKr+e5VzaSwstLcBuEqF+s9Pzs4joHhY0WQiUz9+0FU2XBer1z6XXvrdF+JJW675gfnqfVQuleAp5Cy/p6d2z1DuNzM3r0zNMLcYr1+bYjat9LpzuFEWaolEOqP8+j83tcDunZKyZzabXbrnF5Mp5edNypGk8Bqlz1fdPR9AGdnoICLFWbhdRaDsgTuTwOL8IkJFNXYTolU3l0O5VIEfFcTpPvIXLZ+708XeP2WkDR7nehIRal+mIJASEEqgdKiqObIWCoO8CMett3RCJpWU9Osgcu1QE9/86HPYNVv1eh3vCWL7SK/le2bnTHVdmIilsV2hPZ8kQHUl3pwEpvuedzFY2Tek+n4kGlRqayeV8pQQ0/1hl1pzjlu1rIHhoAa1Y8Ma5na7TXuQ0n04EY6iHHfBEwBCngrShTTC/rAyxbkgznMl4clK6rUuDi3QoF05OFj1kG9jMVmT4lwRkUjHyRXzthFwfDDo1iF1/ubNxEcqCW7Vav9m4GukKusQPvRSp90UHxBKSmgzHfsrxdqqn9FGxTkJGVTZ4zQSuarU5nxA6BOTi3VPedXP0WqWPyGzsdTg8+DKxeENLqIUDaDoTsNPtkRzBYRcnqrVmoXCtyxUVDu816DHucS73/1u3HzzzbjzzjuxcePG+r+Pj48Lf9rFxUWD6nxqakp8T77mvvvuMxyPvi+/J/+W/8Zf09vbi1CoOflx9dVXi6FynITYtGkTBgYGxM/ZjWLFBa/Hi2g4KM6BIP82i57ePvi8T4jK7kK+bPl4jah458U5E8aHBgzHV/VekVAQC9kSynArP//lkC1XPwsqjm0cGxHJUTsY7J2D1zMvvvaFwsrPeyJbqV/3dYN9Bx3fyvuNlj31Y7t8fluueab8nHhP2tq3jo8Iv1Mr6O+JwLtQ7TQJRKJKNm+C279QvzYjg/1KrztHTzAgyLKSy6P8+hem0/XfYcPwoLJnNlws1Y9b8fiUnzcFrPL4/T3VZ4psVb74xS8inU7jqquugs/XflA1PriIXLgfmUk39t7kw1PbxzB60SXoD6u79nOpLNweD1xlIBz2oLe/F9HAUpBj9n08Hudb0P5UwRMRXcoj8gl//MBC/R7STpwzsqxdgpFbhtk1HDReC5B1DAa1YzjovoUk/uPBXeJrCjH+8tTDRAuuMm/8xZQy4pySEN3EOSeayMdbBXFOtkGSGNvYr1bZyJ9/QZw7oTindu1iVsN8FXWK/2ZqR9q7/+WL/4LF5CKOu/o4uE10YtGxKm4vUqUAgt4MKl4gno9jKDykzGotXywikpvCBt8MAr5eoBAD2N7dxdpGmu3jqocmGhXn+Y7udqPj0DnaVfzjg0FVK851FQB5zNE4r0ElejQozuXeTd0F7Q5+b5fMlZhJZHD4SHUGRacrzvmzT+SwKuKcOudkXGCV11jpWlORwg7inApR5WIRT9/xX3CPRvFXLzrGVN7tcXnENS/2BlCSivME4CqXRaypQnFOfQPkcd4JaCu6oQtApPkPf/hD3HHHHWKAJ8cpp5wiLvrtt99e/7cnn3wSe/bswRlnnCH+m/5+5JFHMD09XX/NbbfdJsjtY445pv4afgz5GnmMZggEAuIY/I9TIFUMqRJVW0CQ0lZWAMnjXHULinFIiSaP8xo5RNdH9fkvh3gtmKIKYbukeeOUch1KtUUWhKn3OPc64HFeTfxocKIKckhXiz1vkdRl1cKPndUQxBpscVQOFvO4Bfmj67wNLWI1H1/qDKJ1nwqyZruEBkJBZCI9QNGD7LwbmUQBbk8QhVJBMclKo0fps3W+bawLrI1gnu2pOoYWNUKu9zTgqV1LAoO6TiFJsFLMJAk5XZ6dOoeDkrLpy795XAwQJ1x47GYcNmSdQDAMCFVo8UNEqjxXXYSB0RM4r9SmRXrp6oCM9+wbDqrHqoXbM2mzaqmpHWm//uUvfonf3f0703s3xYq0Sj0eOgwzvj5kfD2IZxeFZZEqxXksm8VpE/fh3XNfwBV7PoGpL9+g5LhdHPqWa4b1TJfiXFEespRz0GDdiq2K8y2DOhXnJW3zJYYVzpdY3uNcLXFOMSWtj7rAi90zimbd8OJWRFOszTkbLopRWWjpUxw3jfJrbdOA0Kr9bRlTTzyIJ39/n+m92+P2iM6Hcl8IpVqukSoE4I4n4ILLkuJcXnM6bNDbGcS5t117lptuugn/9V//hWg0Wvck7+vrE0pw+vvyyy8Xym8aGErk9Xve8x5BeJ9++uniteedd54gyN/ylrfg+uuvF8f40Ic+JI5N5DfhHe94B/7xH/8RH/jAB3DZZZcJkv573/sefvrTn2ItgLdIq64GruuNYDqRFe9B3qMqPbGbDf1RDR7MU/VOR0t2s4Tc5faYVirzYEaHCokHYSp9qnUmTyuhPug06FeyqRuIc4WEBw/CVLZy2TkVXlfSTZ8brV10jbRPs689X16vF3/1V3+FZDIpvjYDKjwVfQFccfGL4CpkMeAK4+NwoVAuiEKdivuRFAyVuudaZ2zkXVgDJ1J1FY15AsVbJHVBzrQwY2dBRXoKhCnpsMOqpRrAV7/WpcTX6XH+40eerVu00CDY1z9vO5LxmFriXOGAUKNPtZ7kle9HqroW9ttAnMv9iMQdZP1gtWug1eJ3b9CnZageQeUenuSq3tqapmLvFs+9y4XKTBj+UfoacBXJO7mizON8MZOFx19Bye0RwoDi+OFKjttFZ4Dvq6r3cWMxWQ9xrsp2o7FIrNMHm55PqTgfigSU799Bnz7xF3X+EUhYoCvu4MdVTZzrEhk0VZwrGlqpe55Q4+wNlbG2gTjXrDi3AzTvg7ixbWeej5dsHTa3d+fm4Z25Cxtn78Phw3mU3FV1eLoUhGcxDowGLSnOlzpRyTWiM/Lttq7Sl7/8ZfH32Wefbfj3r3/96/jLv/xL8fVnP/tZ4XN3ySWXiIGd559/Pr70pS8Z2tHJ5uWd73ynINQjkQguvfRSfPSjH62/hpTsRJK/733vw+c+9zlhB/OVr3xFHGstwKhkVUvIDfcYfSO1EeeaFrTG9lHdxDkl/qVKRdzoZhe6kK0DHgNrzteVg4ohMrFSVZHlhYuUQqVm1ibFuSTOSeFH10dlcrygiTiX10QXcW5YI2trDW3ar33ta4Xfudnkm4aDUoX7sN/vwPBerwiIM+/PINwTFsm312X9cxbBb6WqOO8S54cGpCqF1Nkqn08OnhQnFKpgVh9Qbu6eHwgFqsR5Oq+s6LQceAuzrmTQuBequ/6kgv7Ph3aLr+kSvfPFxyi7h6jDkO7JQqliUFvr9KnWYtXCOqOsYN/ikrJx40DElgJLrybrICnskEUF1fu3HcNBaaCnqr2bnntaYW555FXYMFuEtxLFS//8fMxjXlnHWCKXR0+lSvLReubpHVZy3C46AwbvZ+VWLXoGZhsH9Kk550jDGqaTOCclsswttwyqt89q9DhXCb72mulGbwXcei7BiFezoBxSWm/p9DcnjDAVPgk2VYAT2bruS35dOFFvFUuDKqE8LhipDWKVHud2gISIZDu66ZQX45znbzW3dyd3wfPotRgrpHHS4KnY4x6Cy5VHzjcIV55yh4DpwjfFR/Kah/3utUmct2KtEQwGhU8t/VkOW7Zswc9+9rMVj0Pk/IMPPoi1CJ1KVj5MQXW7tzwe3Zu81UWXAjov2kL0VkxjTC1ndpiDTqVaYxCm2qrFWKjQT5xzZRn3BOzEFnuqttpCnHO7nGJJKTEnNxVqY1JNgMjEWypXVYJ3DqgcykPFMdpcS0E3jtyxB33RLHqe+y/kj/4fouqtYrAIJWdV2rxr1XKoQAbzOtcBPnRUZQGwGag9W+5VZov3tH4TWSuG4RZK2tRBktCS0OZxrsly7TfPHKi3w//ZcVuwQ4EPqAQl8xv6e/DsXAJT8YwohqiIKbnyTdcwXOMwPbVWLZQ+rWf+7yphiDc0E+d0XWRaJfzNFYKU8lQ4pnszrzD240UulcRNtWDmQtw9iIF0RtjA5ON5ePo8yBVzyvKBcCCDss8NNyrw9neJ80NWca54XaN1l/7QGswFK514zrpz1uX8zVXbtDQOLeRzW6yCPkdZaBnSZNPSKARQwdmkbNi7JSinpGHtRNSrUpzrGIar+5o3I85VK84p3pYiCVXX2paiXWhcxGP0ZyyQxWPrNyGTLiF0eB/yR+6AOzVjuvBN5yedpiJ+F9zxJFzPzsAXdgMDeUAtXdYyugatGsCVrGbVXq0NPVC7IcpNJOL3aqu+So9zu4hcGsooYdqqRXMQwoOwfsVWLZQ40WJs1/U2qOc1EOcq73lOCOu0apEe3joUEzKJ1eF1Jwk3IplUzyNoFjzRe8zNzWF+ft70+9Gx6J4vBz148RmP4OwzH8SG+Z+hXCkra/emQGwAM3iZ9xfYUbgPrvReJcftwjnI+1GXTYvOYL4ZphLpuof1ur6wqWP0a7DaaElxrsuqRZPlGveAP2t7dcC9ShxWGxJVaSApVCXfulRfgxH1989kPF33XjVjQdT2faJ5uJ5hRgm7Xqogr5FKwqmZYlDF3i1jmPHYAYQnJ+Hdsxf4wzPCPzVfVlN4Wcxk4B/NIz8cgHdTCMOn7FBy3C46AzpsT5oVA5UqzgvqiURd3VWr+5vrUJzrKXjLWVzSYkYXVA8HtaNbTILWY6mEJlsbUgCvBasWLr7gZLdSPklxQZ2utbRrIcW5HTMASRhA75NLxpBLxs29Z2BU/OWCG4PuBRQjXhGr5mPVTlWxf5fMfQb8s6PLPXPXf6Pw5jcieslf4A+f/l9wCnr7PP5Ewa0NVFu1hNkirJrETdYCYp3kAU928rV2I53g/qzmrVqWHhMdthUyCCPyVofika55oVTUcu4rJYKqrFoimgZ9ZGqKc532DAep/hV/BjKI0qGKk+dNe6nqeQQ8AJbBE1l7keVXoVDAj370I9G91C6o4EfXojeZRCXhhy9agaeQRqmcVzZgjK75RtceXOT7KQYXPPDMDAED1cHWXaw9UDIg78eIRlW1ncNB9y4s2Xps6jeXzPKOIfJ01eUp3aiA7tUUf+hq+datlt9WI84Ju+YSOHKs3/Ixkyx21JV8UywjFZoqrFqI7JddYtwPVGdngm61Jh9Op9qqRfqc0++gMtaWz2qYCWxU7N1yOGiknIY3k4Ob1PL75uFxbRWKcxV2UeSXeua+RzAamoXL48HCzDyiPRssHbOLzgFf1yIa9nLaEw/E0mJNoz1ERb7GhyWqIhIPJcU537dV5k987VVtkcpBHJDs/FHhcc6Pwb28dWE0GhSdXqSEJq7C6j5lR9Ge5oU08yXvZMU5gYoU+xfT9Wut876URbtysYB7v/opXPtfPfh/N/+k/b3b4wf8g3AVD6DftYj9r9+Eis+NS16zQ+zXbpdbzBgzA/7ZUR008UAKd0y9FX53FsN7huAUuopzDeDqDtWqmAjbEFUm30QeyA1c52Js97DKmIIKoSGR0uhxrsraZLmKvUrV0XJYYOq7TrdqkcdS3RVil888tV+TjYKuAESnPz5Vupu9D83AoD9Wg6YjEh64EgF4Ky4Eyh54CklLA0oaFXdRV6xukVPxO7eBd7E2WkcbFXD6ifMlFdimgR4FVhtrX3FOxVHZfaWSTOAqJB0qx23DvfWvd8/GlRyTe63q9L9VqdAkL10JUpzrgp2kE1fi67ALkPG2ythvucF0Vvfu6n3oQmA0h8NPfxxHvuQh7HzuP4RiTUXHGBHviWwOQU8SlXIBhXgWt8781tIxu+gs6Lax4KQh7zSyAplTuhR2vtq5hsl1mWJhHQVNXQVvORhU935C5KFcK1UozrliW7fHOWGYeW/PKrAQsSPe5kIyHp9ZRYzFTTrEajqGsbbyWbjcbvhNziYRCJJdiws9iKHc70Ex7MFsJi72bY/Lg3zR3FopPzuyRiU6qTxDHYcu5Eoh+HoH4BS6xLkGLEcKqUBE04ZIx6rYkEjZ7bkdV1AhNAwHVRyEkHJBKsF1VRdJdeSMVYuvw61aSlq6Qpa7/vw9VcCgPNAQQPGAVXXRxeitVn0fqnSTWu3b3/62KcUaJ96KET/yGb9Y0yrFMryllDKrFrruPYgvEeeBLnF+yLR3ayTOOdGk2matEXyQpFmlOC988oKoDnBliRw4qAMyHlNZCOQKXPKVVo3NAz1inSHsVmbVYs89L4kminGsxqvTLJHUqTjXZemzqlWLJsW5ytjPILBh65mKvVseLzSSwYkvfxhHn/k4QuXdIDdy6hazWvim/T+ZLyHgq641+bwbfb0jOFSQSCRw5ZVXihlioVAIL3zhC/G73/3OUDi45pprsG7dOvH9c889F08//bThGGS186Y3vQm9vb3o7+/H5ZdfjmRyqQjb6dBNyhkHhKopJtdnkSi0SOVrmO5YY0n4pWfAJs+fVOawvGipY+3l6KmJAXh3mprhzDYozhUPrZT3I90q/LNVCU5qqyhW2KU4V32tW1l7PD4/Xvyej+Lb37GQd4fGaupyILHwVZz4g7/Cc++5CL6vfkMozmnvNbN/1zmOSgVEJ5Xml57ZyCbn5pN0iXMN4CSTau9kbqPCvQatwtg+oy+R4nYPUi2rEzy4MVsh1Km85eenS3EuixXU5qbbN4v7xqnyAOPKEZWJrCyC6BwIWD2+JosAFhDoUZzrKxjp8HXk910u4kcuXQ2GK8USPMWEOsV5rlhXnFOw0FWcr23YpTinvU8qnlW07K6EfYtVssPrdmG815zHOS/kLuhWnNvk2yk/Xx3rsE6lvOwa2L+YUkIe2JV8cyXfgZo/uVnMMsX5iMZhbmE2kySt2d5Ot+JcDvgmm4BiuaxoMLaemEPG5/HgkhVRBCXkyjmhXLNqtUY/n8umUCyFkC71YjYxioGgc6o11fjrv/5r3HbbbfjXf/1XPPLIIzjvvPMEOb5//37x/euvvx6f//znceONN+Lee+9FJBLB+eefj2x26bki0vyxxx4Tx7n55ptx55134oorrsBagbQb1aXG5TOoVM1tkPGHytjDoDjX6HFOa4rc/3QJv3T5tfOipc7hoHyPJQsOq/u37rxvpT2cd32ZheStIn6v8rlc/NmXh1Zp1SKPRQRxREOuMBJdutbTCevXejXItcHvcVsTfQSrA0Lpugx687j4kQTOfDIF34MPi44xQZybEK7JzkhSnAd9LlQWlo7Ru2U9nEKXONfuca5Pca6ykmzwhtOpOGde0vYMB7VuHaKrxfvgwaCaFOe15KliQ7HCaNXSuRPiybNbDs/TTZzL5FW1R59xSIw+j3PdinPVZGVvyI9cTwC5TI04L1WUKs7puu8ubcUfSicj1XMEKgHnKt9dWAcvQIc1e0bKRIcXqlWD1raJWJWkXBcNCX9Ny+o6O4eDaow/5FojOuwUFJGJkJRrmc7z3ioHhFbUDAg1divpO2/uebt7ztp5cwXWiE7FObeG06441+uzq3q+SlKjwIbi7IDXjVhoqYMr7K4gmUuiWCla3r+pcF7JxJBJhVCq+JBI9WIgdGgQ55lMBv/5n/8pyPGzzjoLO3bswEc+8hHx95e//GWx1t1www340Ic+hFe/+tU44YQT8K1vfQsTExOiU4DwxBNP4JZbbsFXvvIVnHbaaTjzzDPxhS98Ad/97nfF69YCuF+4DqsWg32Zgi4s+lzqg8k1Eec61zBS4MpddIAVFbTlIZo8zu0izlUooHm8qpOrkRiL6lGc63g+JajzQV5zHcQ5qc11kP4GxbkdVi219ZLbEZtCcEy0EFCqsa7gw+7ccdgZPwWzzxWF4rxcLpsSri1ZtRCPUkaZNT8NHr4VTqFLnGsAr4qqtoHgi43KQYlJm6qYPtuHg1YXOpdFcnGpxVsfcT7AiIq1EHis9PtQ4KbK318Hcc4/R9VdIXZ1LOhWDRqH4ipWnDexs6LBYpTkfe1rXxNfm0V/MIB01I9cyisSk3wuB18pLQaMqQBVwX9bOgM/cP0lnjn8A4BP/UCkLuyDwTNSo+JcHL/2nOr0OJ+IpQTBSljfG1KirlPhUd0qkaszGZRrDV2fnIL4g3+OfCCVahw21KuMgG48b53Xmw82fXbOmj87V7uNaPSk1aVwXMlnlxRyOuIQ1TZxxmL90n2jau+mGH2ufwhedx4+dw494QQS+YTInFUozpOTLvzg+lfjux99De7+xeHoZ+r2tYxisYhSqXRQqz1ZsvzmN7/B7t27MTk5KRToEn19fYIgv+eee8R/099kz3LqqafWX0Ovd7vdQqHeDDQUNh6PG/44CW4DocOCcUCxVQsX8KgUkIRsspuyQ/hl7NgtKV97qStPZ5cbgR/fauxnt1ULV5zzri/LhSLN/uz1DqZsXolIgmzKpFWLqm76RnALOlusWsRw0CKevuPH1vbu4JjwOKdiwsZ8GbuSz8ee9HFYnPIJj3OrinMK2Elx7s5U56BUXMDApnVwCvonC/wJQqdVi9HvuaClxa1HI3lgt+JcLnS0cZlV3smKHFW/VLe9cUJCV8W+0au6T8u7VDdFadWi0jNOKv6pzU1VEGjsCtFLnBsLFwqLXZrtDXQWXDghIdc0Svx+9rOfic37Xe96F3w+c79TfyiITF8YuTS165VFtdtXTiNfyitbK4uVAjweaiHvkuZrHXZ5nDdr2VU9PLzR39wKcU7JN+0d9OzzBFkHpAKf3o/We13gaz3t5VbjM7uGbHICWgVxLmNH3dd7y2CPEC1UVCjOawosIjp02QLYSTpRIi6fK10euwZrRAWFIkOBi4l4VO3d9Aw9OzKG0GIKLlcJvZEsFrOLwlLFqtUaEefeKboHK6hU3Mh5y+gL6IqG7UU0GsUZZ5yBj33sYzj66KMxNjaGf/u3fxNkOKnOiTQn0L9z0H/L79Hfo6Ojhu97vV4MDg7WX9OI6667Dtdee+1B/76wsCDuCbuxmEqjWCoi4vNicXFROZHvKmTE8QkT8zHxe1pBLFuoH89dLhqOZ+XcC5lc/bjziaTl81wOe6cX6u8TqJSUnT8HxR/yPeKptLLfZTpGtk1F9AcCiC0uGr6n+r7xlIr132H/zBz63Ob3ldlYsn6sYiaFhQXjsXQUr3yuCjLFEvYvxC1dfyp8FIrV8/WUjfeL6nMPuCviOhHNNDkzZznWo6JcvnbudOxm10HF+QfcQKpQxMRCQttzK+OPZDaHUjGP535/N/7frhDe+MY3mvI5dxci8HuHsejzIebuQ9gN+MpAPllGPpFFPpvHwvwCCoH2OMuZWEJ8hrT3+wtlhIq9yLv9cPldyBSy9etj5bqb+dkuca4BvDUqaGVSbRMQ+SuTWaVWLTa17hoDeb2BFRG5srWG7BtUDFuRLd6q2nQMFXtNinPDcBWNinO6H4kQIqhObCmZLZQKyopFzRTPdpA1+oaDalCcMxJRtVWLXCOplkX+ajJJ+4u/+AukUinxtVmQ8iUzFMUxwdvQ4y0gHfDDV0wjWyooUQjR2ruYfwr3L34dD9zsxf887X/i7ae+3fKxuzi0Pc4bn1Pac7UQ5wtqiHNpbzZZyGhXnMv4Q7dn50Et7Bb3KYM3u8ahpoKApkHEFWD3rAKrllqHYUTz/U5763hfGAdiaeyZTwpPXDNemhRzSbUbKeB0DKGz2x+YPgOpNh0M61HQqx6sl1qmWK9q76bCYtEbQDHngS9YQshbQTwXFy2jVhXnhXIR/pmlXu9ST1m0kR8qIG/zyy67DBs2bIDH48Hznvc88Zk88MAD2t7z6quvxvvf/34DCbFp0yYMDAyIAaN2I19xwevxoj8SEudAkH+rQLAnKo5PyFbclo+dXkzVjzcY7TnoeGaP7wsX6sctu71KrwFHcWrp/DeODCg7/0aCz+fxiuJrRdHvQjF8rgxx7usGok2PqfKajQ0m4PVMV//DH7R07KLLU7/mG8aGm+avqj/v9QNRPDefRCxXQl9/v+n9t5TM1s99oDei9bqP9PXgmfmqZaEnFMGARXu3FHtWR/qa3zMqzp+uNYkM4vkSon19WgbOy72cfh83XDjl3Atx7vYRjIyMmNu/B16I1PiP8PjTD+Deh+/HWe6dokCdLfnFnJJ4xItIX6Rta7RcxV09RzcQiboxnXOLWCAQBvoG+hANRC1fd9or28WhEzX8iSjOCZFaYK/SJ5Xbvmj1OPcyxblmv21R3awRuf0WW2skiUiJq0qfcE5I6Gp14x7b2VrFVAd4EUC1gkrek8oU55qfUTusWnQT50H2rKoeiisJCXquZBGKNmyqeL/uda+zlHwPhIJIh6OoZDwoxF2Iz7tR9vejUC5YbtmrrrklFMsJETzS0LKwz9zwxS46AykbifMIO74uu5Y9C0vk0IY+a/emLIBSoqnLtoKeyYRNxLlqGw7pwUjQ2e5NgoON/RHx9d7FpCjgWbne8t6zo9VbquWJJOZFnXZA94fsehphPqA6oMMabrVB6kMRG2z6FBDny8UcqvZueoYoHsjlq/F6yF/GYmZBtIHT/m3t3HMIxDJi0BhFHOU+fcUXJ7B9+3b86le/QjKZxN69e3HfffeJDoDDDjsM4+Pj4jVTU1OGn6H/lt+jv6ena+Qes4CZn5+vv6YRgUBAEOT8j1Mo22ADQfuHzGFVdGHxwpy24aB2WbVYFKctB4qzZQ6rKn/la6+ubh8Ovs+qsmoRIkoNwotmkNZoVOflczksCTQ1zxOSVi2qfM55d6Gue50PCKVUdU7BMNbV1h63x4PTz/8zy3u3x+2prrvRHpRq1EG26Id7MSa+NlP4rgs8Ah6401lkvD0UbMDX4yx13SXO19hwUN4iqWrA1UFWLRq9pwI2Ks6lTYsc5mAFYd7irTAQ4UPXdG3gBp9LjYpznYOupOKfzp8CZLVzCPSSZQarnMLamUug0+NcEvHyc1WJvlAAmVAEhXkvnv2GD/f/PIi5sQtM+6w1Bn4b8Ry+OfDvuHWTD5894iScsu4UZefehf1I82FLNg0HVT2jhGPfYpU4pwR/0KL9F08QVAxDW24tkEu6Tps4HYSCbrusZgS01QGhdL1rQmfthQrCVjYg9FmT5839zUdrieWaJ85ZvDSoizhXPJhcN/FB9yPR2XvC45iPeLDY50U2PQ2v22t5RslCKoPjoo/gtFc9gBPPfQSRcb3+9U4hEolg3bp1oo395z//uRgGum3bNkF+33777QZ1OHmXk8ULgf4mexOuUL/jjjuE1R15oXc6smwf0RFXNuY2KgZm82GmKolzIptl3qFzToMdHdM8h1WVv/LBoIOaB4MSeoLqhoPKNTjCREe6wYdxz1gYWtnMotOOYgUXOKgQOXJSXu+AUI3EORcMKVgvPa4qce7x96LkKYsOEVKcexbjpojzqqCmes3DfheKlSAKmzchf9gWZF98LJxElzjXAL64a1Gc14htUvCoUj9zEk4neWCwatGsOI+xCqFVqxZdA6MWagNmyK5Cl9e26gFRq/0uWohzqfhXNCWeH8NW4lzh9dc9JIaft0oCgQ+I4cET/Tu1etMfKwVBCmqKPj8e2TSIuw8L4/HNQaTzaeGTZtUnNZkvoscVg7cMDGez2JAYw3Bh2NIxu3AWnMC206qFqzdVxh4ziWqwvbG/x7KlBV/HVQxDc5p8Vt39Y7Rq0U2cqxkQaiA/NV/vgweEmjvvWZasD2tWnNtFOnHyZkgTeaNzOCi/d1Tt3dVnyIW9icNQzAygEu9HJTYrLFWsziiJZbMIbkhh46nPYcdZTyNzeLWD41ABkeS33HKLGAR622234aUvfSmOOuoo/NVf/ZUg2K688kp8/OMfx49//GM88sgjeOtb34r169fj4osvFj9P3ugXXHAB3va2twm1+l133YV3v/vdeMMb3iBe1+ngNo46u6blnkj7h1UiV5fiXByvttfpKtATFlgxXefciaX1WBVxrr/bhyOqMO6THZK6YyWOEbY/WRlaaWd3p2rFOT+GVSFmJwwIlTk47dcessG1uHd73B4EvF6E/WEUgtXYJlcKwB2Li+9li+0VAShekY4RNBh0ojCBP7z5D3j2fzyL1FnmOhdVoetxrgEy2KZWGh2DlyKGAaFFJT6pfEHTqUKyczioQXGuyKpFtW2FDDz6w35t1WPVqqPlMM/V84qDEd56ScGm1cCYf4Yhv952Nz7nILuGrFr6FAceEmQzUKpJHnnwlMvlRJJG7cU/+tGPTA0pkUE23fNfP+ICBPbNYuBAEM+LLWJkcMSy4pysWqKIw10oA7Eidt+cQHp+F0bfZhyq1cXaAd/7IjYS5yqHe0vsj6VEcZGwccA6McRVZLoGhBoGDuq2ajHs40Wl7btRjSokwmHDagaE8vtOd7s0YauCwabTtWIQYbTWNq4TRDrRXm2XVYu+4aBupR2eyxXrVe3d0qplPjmG2eIostkIth+2QyjOiTi3Ml9oMZNBr78Ml9sPt9uFE059FQ4lxGIx4Tm+b98+MdDzkksuwSc+8Yn6oNYPfOADghy54oorhLL8zDPPFEQ7/6y+853vCLL8nHPOgdvtFsf4/Oc/j7UAThDr7Jo2dGFlchi3YNVnjD3UrsUUW1NOpnNOgyym06winURusN5xrGbG2LwNRUsOHhvwmMHsjCW7usWakbm8+6tTZ+kRehWq/Bv5JH5srYpzjcS5jH/LxQJuvOb9+I+Qz9Le7d7zH9i66yZ8KLgfN6/bDMyNIF/2wz2/KNTo7RLnvEuALvdsfha/Df8WCAOv3fJaOIkuca4BcmHT5Z3MN9h0rqAk6OYLmi5/OLuHg6q1alHfvku/vzyWzmo9L6zoLFYsHGTVou69+PWnAHmkR2FXiGafOKNVi3rFedjvEUU61eCe+1xZYhWGooUmtb9QG+zvweCTfvg8LiTmEhgcGLQ8YIxIvqgrjrLfDc9YCP1/tgPrX9D5aqwulgcnUHUTt4bhoAqC+UbsZf7mm/otLpJiHeckgR6rFsPAQd3DQRV30fCETHciuHkgygaEVttfO/2c5XuQRyol3GQxQ1Zr7XZC8PZw3R7nBtJJI3FusAvQNd9GceyXNJCT+qxa7p45CwceOlEQFKe+6lShOKe9mwrfXlf7MQP9LCXi61xVpRq9R3CkuW/3WsXrX/968Wc5ENn40Y9+VPxZDkS433TTTViLSNvURWqMi3MY7w0r6XylOF5HkThfLJseyrwaZDGd8j2dA5tlNzbpbajTnmL6tepxboXE5bGSbltBDhrIrcKqxWAPolmkwjkf1YpznbZExiKFfsU5waPi2c1OIZTciUF3Dj3rj0X2mR6EQwGkhtfD5/EJq7V2il7yetNMElKcJwpLoovegHOzNAhdqxaNibiuzTvMiG2yDlABqUIikk/XFN+DA3n7rFosE+cafC8XMvYsxNwCRvWQx+UU57qsWlQpBe30WqOuE0lsq2z91j1QL+L31gNUbsNjFcsFTzRo6oc//CH+9V//VXxtBaR+KYV8OPuFf8Cfv+Z2vGD6BlQqpHS3qjgvIooYwul5hONTGHv81xjs17uOqcKdd96Jiy66SLRdU/BC6oLl8I53vEO85oYbbjD8Ow0Ke9Ob3iSGgPX39+Pyyy8Xw8g4Hn74Ybz4xS8WyoVNmzbh+uuvx1pQqtG9rqNDjIMr4XRYtfDhi5vWoOJc+3BQxfs4P/dezedOsdmGPjkgNGV6QKidqq9GuxaK+Q7E0pYU53JQmR33CZ2v7I7S26G3Nqxa5P1OuS+PK1Xt3dKqpdTjFd7aKBSQf25OKM5p7zZb+KafI9LJP5RHsceLSk8PvIPdod6HEoxzujRatbA90Wox2RgLK1acc1uyvPq8j8h4qQrVmb82rmMqcqjZpL2Kc1orpbbJynBQ3facy4EXq60ozvn9HrHV4zyvVIips7uCihSSWp7SatVSXRPcXh/+7stfs553h8ZFzkhch39jBAsDYyiMjCK5cbtQnNMe3M7+XS8wVYg4B7bd+jt8+N8m8N4fT2H9vPrcqR10iXMNkAGqLsU5b6/lg82sQN6kuhMp1a2jKyHGyD61Vi1qyE8+XIYr+9as4rz2+1BSpXrqtOrCBf8MZRugTsi1QJXHOan25GAhXc8sbYIyIFZJmvHWUZ580/vRVG/6Y7UVk573SsSPsdEFbFw/g2jhADzFjJLhoFHyOM8l4U4l4L75ToAS/DUAatM+8cQT8cUvfnHF1xEB8tvf/raprymR5o899pjwUL355psFGU+t33zg2HnnnYctW7aIIWOf+tSn8JGPfAT//M//jE6FLBrrnnXQqBLibdqqsG9xiTjfONCjVF2nS3FuJ5FrIBOUWLXUiEQbuhW47QmRuXtYd4F51bA9TadbLfqcz6Yy9eKWTn9RXfdJM8RrzxPNt9FVvDfY9Cm0aiFSgu/RqvZuSUisSxzA4J498O9+Dq47fi8U51ZmlBRKBUH6D3nn4fUX4R/ogSest9OwC3thFynHcxurcTG3zVLucc4U7DrsWjiRqDN/1dG1KxXnXrfLFq9wWhNlbGNFMMGFkjqdARpB96Z8P2se5/Yp5o0e5wqsWtgxdA4HJfGOtLrVadUiPwu6N3vDQet5d3BM/DwViCIjaWQ2hBA8eVCwzORxTnt3odz652BUnAO+hxMYf2oIxz8cQE9Ofwy4ErpWLYpBhBa1RmlVnHOPcwWDP8Rgn9qC3KO5CmggcXUPB1Vo1WJQbSsiPPiwNa5iWKvEuVRQUWCpum3PcM8ruP52WrXI96BnVdW9Q8eSOrhoQN8mQp0DpDCg9yOFowpFrh3tetTifdLENAIeHwLrK/DlAU8pqWA4aAFjiMFDZDl9ADk3KoMDWAt4xSteIf6shP379+M973mPGDR24YUXGr73xBNPCE/U3/3udzj11FPFv33hC1/AK1/5Snz6058WRDt5pObzeXzta1+D3+/Hsccei4ceegif+cxnDAR7J0Hej3YQnzxJ4223qq1a6LkaCPmxmGtf3cvB25h5oVcfce5dU8NB63ZZAa8Wu6xmyu3fPDNZ9wvfPtzbscrM5QaE0nm/aPt4W/GpHHhLyjdds2CWJZ1oILSG6yQTQ52fgUFxrvB+13XO8rjpoUg9vsnNpOEiFTrMK84pYc/mSxhL7IOnXER+dwzx55m3O+riT3fIN++mtUqcG2JhxXyBoUtXQ5Ge/+46rUYbxU0q1jFpk0UEpU6LGQ4iW4nAtWLV4kS3mATtvalcQuT5ZuzW7HxG5fBqOsOKKquWGp9EBQTdnalk10L3KIkcaI3Qca2U5+HBcbFP06Xxby9g7+ZtOOmEYfRs7hExXLFiTnFOn1/AV0b8qX4cWDxc/NvW0jCcRFdxrhh2EHIRv1qrFlLBynZU3YsxLbbS/kG74lxhhVB1wt3oG82VfarBz12lxzYH3T9yY9ERREUUE+d2WrXwz0BV4cIusomG1kqoUp0v50VZLBYF4frtb39bfG0FvaEAIh4/Silqe3PBW/HAW0opGQ7a41qAq+hDLhXAvHcEYMNf1zKoNf4tb3kLrrrqKkF4N+Kee+4R9iySNCece+65YojYvffeW3/NWWedJUhzifPPPx9PPvkkFhYWmr4vDZYjpTr/YxcoAZDrecSGdYDvr6qtWujelMVLsmlRQTBSwZgUsXoV5/YMJtfRuZTI5W1tm+YE9LNzcevt3jao7QjbhpYI/t3z7SnOiWzI10QWdvib67Lm46BEsq7e1vgZGGYKWRSqrDSYTtXeLUUAqaEoznrzHTjn7Tdj+Mh/x57YHpFBm92/SXFeySXgLtNwwTLmc3kxbLSLQweGocca12OVXVg6RSQ8ttaxhvHfXWf+epDi3GJuQLmfvB522LRIyHWe1mGzeaBTVi38/WjGitn7iefudnBNRJ6rU5zntavNm/mc61Kd14eDlkq4+fv/Zj3vDkni3IUopjCXexy/2fMLHJjZLXIRQZ63QZzXFeeVCsJ+NyrJpXwmvNXZ+SSHRsbfQeCEHFcpqwRXxqmwajFUAW1o/6FgvlAqalU/c4UcqeitVgi1eJwzIlK1tclyinNVViHN1PMVjcNWVKsnePFDl6USh3wP8kw1W61vRtjoTrwNitNMzrChqy5a0KZNNiGFQgFve9vbROuYWVAHx1RvELlM7fyLZfhKaTGgxArimRx6EEOx2IP5hQH8fuZsHEwxr0188pOfFNf8ve99b9PvT05OYnR01PBv9HoaKEbfk6/Ztm2b4TVjY2P17w0MHKzOv+6663Dttdce9O9EtJdKevcICuSLper96KmUmpL7Kol8MRynUkahXMZCMr1sMcEMds4m6r/LcMAjjq3i3CM+l9jzZuIppecrMRuL18+7lKVrsuQprbqIkmOf92LS2u8jPF5rXWMBV+WgY+koAPW7SyiVqt1GTx2YN3X+s4tL90kxY7zeOs896nNjIZvH05PzYlZCq4WdXXPJ+vn2eA++zlrOvViov+fU3Dz63GqJJ4o/8rUk1Yey4XdSee1zqczS/Z5IWrrfSYgij+WDca3MZrP43ve+J/bw173udWK+hZX1cXG4HwNYhD+Ygzvlw2MLM9gQ2SDe05VtP3ZaTC7CuziDSqX6s6mSC6FiCPHFOIr+ouXrbmext4vVSTmdYhiVXVjcQkW9VQsXG6nvbuP5q+4Bmyo7Z+w8bw5OdBMBzvNyM8S5HR2SHFygRc+aGeLbzuGgsvOYlMtWFefEVcn7TidXIzHKBALTyYzB6k4V5GdRKZdw+//7KdyVsrW82xsFPAF4XBlEKvvwjq99DEfMFzAytgW5n/24XsBuX3FOxLkLqXRtQDB5qA/ZI6BYDl3iXDH4wqCLkIsoVt/aXcUMeDxIoVi3tNEBCsClVYuKoV0RDUGIwapF4wau2ufS7sGgjYNzVFx/Y4HLBo/zBrscq+9pV8uecTigGoVWZpn2VNqwX/Oa1wgvbiukOaEvFEC2N4T8fDXIqRQr8JfTllVmpdwc3OWCSMCzySA8Eb3Erl0gP/LPfe5z+P3vf2+LFQLH1Vdfjfe///0GEoKGihLJTkNIdaKQyMDrqd5rg9FIU2KfsNy/m0FfOCjUWvmyS+lxY9Pp+u9y+Prh+rGtvsdIbw8WsiXkykAk2mtQsapA0eWtn/eG0eGDFD1Kr32lUn+vkstj6dhEmshjDfX2ND2WynOXGI6GxVocy5dMHb/o8tTPeyNd72USQdXnfvj4IH6/dxaFClDwhTDWYhE2t7B0nbeMDLZ0XlbPfahvEV7PjPjaGwwrvxZ83Wl276h6v6IvWH8fl9dv6biphWT9WMO9UcOxiDB//etfL/bukZERS/t3fySImUoJ2WcDgjgPhQrIuNPwRrwIRAIY6Gv/d5gtz6J3bhrJIv1sBbOzY4j0RtDb34toYImUMHt9PJ6uX7rT4NZnOv2TZRcWKYetKs5l/q5jMLnurhm7hF+qO76lTYvOoczNwAVORAqaUbtzkaPdVi38/Sj/bHX/5pC5u477vRkoltyPtOCaKO82U6ywczCoxGg0qF1xLoeDutwevPrii1HO56zl3S4XKsExeHIxDHmymCmX4S1W4I0nkavNAWuLOJfiQBoO6nWhkqted5evBLfPWbOULnGuGPvZcC4V6sxmiBg8zq2TiIYqpuaBDXxAaE6jmlAoemqtqX0KFjpe3VU11I0TkTqJc9U+l0743UU0eZzTcBg7NnDjcFnrxDn3ydMZQKn0c2w6HJR5ydKmfdlllwlVmVXifCAURGoggtzj1fMvkz97MYVMGxt3M8znfPiX3S/FGT+eRyHrhyd6sFpzLeLXv/41pqensXnz5vq/kdr7//v//j/ccMMNePbZZzE+Pi5ew0GECalH6XsE+ntqasrwGvnf8jWNoEnulqa5W4DdChj5vFLCzfddFdjHhkVuVjAYtJldE5236rjGTt9O6vQhQQOt/1b3woQDlieE4Z6QiB2o/djM3AnjgDH7zptsZog4lzYzrSbeNGNDYqTHHqJDN+lkuHc0fgYq59skVnhOVe7dZNcyk0wjkw6AyqbESWfSU/C6j0emaI5EoJ8rzFSwa9cWBMI5LCQiGLF0ll10GlI2DU4kYQHtidOJrEEwZEVEwoVBqsBFKWkNeZ+RONcbvxm6pi0OOp1LLe0nQ7UhjHZAhU2fodPYZuLcMNje5PnX5wnZwDM1xmWkOjdr9cYV67rvdQKPsXn8oxIyD3d7PHj7ZX+NRCxmee9GcAyIP4UwyphYOAeLc2GMZZI4NlUV9bSzf8eZ4txfTKNYrF53T1DPsPZ20PU4V4w9LHndpDB51Uni2uk7xTdBnR7nvEJI7TpWwYemqiI8FmqKc6q+RjSSNkZ/uLVJnKu2apEbOCe010rg1+gLrDOA4p8l75DQ4XGuEgPhIFL9EWTTbuFpms1l4SPFeTkvulHMYjFfwcyeAPY8tglTu8bg7T80as/kbf7www+LQZ7yDw37JL9zGhRKOOOMM7C4uCjU6RJ33HGH8EY/7bTT6q+58847hd2OxG233YYjjzxSi/rWKnj3io7ktRnkHkuFXZV74F5WtN/QH9Fm16Qa9QGbfq8tg7qkhZ5VQtQpv9FhluzPmkiqErUkkK6DHQNNlxsQ2ipmkkvJ1ogmMcrKpJP6RC1pU+G7sdPNCow2AV69Q91IveYtIuRJIOyJwTW3F153e4m3BA0EzxfziO0P4N4fn4I7v/tC7J5dp+Xcuzj0rVo4eUZ7CBUvrZ5zRMP58mdUz3DQvG2WJyGFOew8U5zb6nFem9/A92AreZ/ONbgVxbml+92mc+fdi1Z8zlXzSathLBqufz0V1+RxXvssqHvG61ZEBQerPudEknt9EaRLfUgWQnAvxsT+3Y5VqixWBLwu+OeJVw3B7fIg2GtvwagZDo2sv4OwV5PqiyOiWA1j96Rm2epdKFWU+D03A0/wVSx0dM6yPU/VUDfpj0e2EjrtEej6ynPPaVKc82BEt8e5inteBl+6BviuGPgp+AzsemYHWAumKsU5b7XknysR2qRyJhWz8Dq18EzQcNB0by96SrPwuXMoeMpwFRIoV8piwJjX5TXlaUznPjpPBGWf+DfvsH6/O1VIJpPYuXNn/b93794tCHLyKCel+dDQkOH1Pp9PqMSJ9CYcffTRuOCCC4QP3o033ijI8Xe/+914wxveIEh2whvf+EbhV3755Zfjgx/8IB599FFhAfPZz34WHa9Ss6mIxpMGUv8OKlqDpOKc9juVA4x02DVxyP3ULvKZ1hz6PawSojwRs2NgFFecc1J5Xd9SktXOPW93q/dWk8Q5KTvtVpyHNCvO7RrQSt0ItItWFBDnfK2MNtzvKvdu8tKln876fPC5q+tNcGG/SLxpuBgR4R5362tmoVxAulCAf7HqQy7ObLCb+h5qkPO+SCikjAhaBnztpPWhz4RVCeW+MhfQQfTrHw5azQeo9iqHMNphN2o1f5pPO6U4X/o8zHIIXOlt9/7N41Yz91OpzO93e86dx2VmixWNsV6fDbEexfDkzEAWM+RxrgN18aDPI/ZtFXt3ZfxlmCq68Jt9AcQzpDwHsqUAPAsxeMb6hHCNcnC3y93iNa+AltbwdBHe2p4/MtoPp9FVnCvG3oWq6otuel1WLUTikkpZlfrZ4A1nx3BQ1lqsy3Obe8+psGrhG5UKexxSKcjqsU6blsbAw+pEcqcGrhDBLddzFQowqfq2w9/8YMW59XveruGg/WH1pNlyivNcLic8zkn9TF9bAZFw6XAPDvM+jKAnCZ8ni3JuQSTd9MfUeeeqgUU4lqsm30RMjKkf2qIL999/P04++WTxh0C+4vT1Nddc0/IxvvOd7+Coo47COeecg1e+8pU488wz8c///M/17/f19eHWW28VpPwpp5wirF7o+FdccQU6EfxejNho1SKhYi+R+4kM7lXHHf0aimecPKDnyk4VklxzaB2m919rVi2cPJ5lbeetgH5fGTPanXhTXCBFDM+aUJwTIWZbcUUz6bSS7YlKUBIsYz/LVi2MPIg2PKsq9+5eocx0IeYarA+cDyem4fP4hEcqEeHtgH6Gzt0XX+rIcTs8XKwL9ZD5VMSGfVyFkEe3TZzhHDV0zchYgPJX3Z1iPE+wmj/NJnmuaqfHOSdxzcV9cu+mTjG7RF+qrGaciLW5aJKrxttFjJHuy82EUb1vywGhs8mMpTh1Ocg1IeAqK9u73SNnYmH9qzDbdyaStSw5VwrUFedU+G7F55zyGXrO6bcO+V2oTCzFi74BZ2w9Obpld4WgD1oa+W/sj2jdTKhiRwuBiqDeLr9FiUDN45xAFbWQTy9xrmI4qEzsydNORbHCMBjUhoWYqoq0WfNWr7Vk1UIbCQWCNBzFqj0RLcrU7dDosW2Xgk1F8cKuxDvi94oiHV0vaS20FnylSWkXiISQLvqx8KAb8z4f4mceJyaH0+YdQPv3aDJfwGbXLgxszMCzLY7EdC+8h49ireDss89uy6aGfM0bQer0m266acWfO+GEE4Rn+lqAEx7nfI9VZfvF1wMz6je77Zr49a80aWW2Mwk3+7lzBZOtVi2cOG/TqkVc79oFt5s4pz2cLITik4uiyEMDulcrXFPCKH9H+r3tGlxssIbTYdVio3KQhCoqPP3tmoVE14M+5UXP2NL7pRfgcXnqiXfQ2zrhRT+TypfgE/dR9by9Y3q6gbtwXnFuxz7OO0jNktK2EueKi3/UfSkL9XYIvwxzuizmT1JxTvOt7LDdaDoc1LTHefXnIn6vbXuhBL1nsyGlZmwR+bF0gsdlqqxa7FCcSwHMvsVUNfdO55TaCunqdvG4PUJN3hdy40CoNhC07APm41XivFRsKf+WNi2Ur4Z8LuTCgygNDdLgLXi2O59zd4lzhdi3mKwngbr8zSUifq94mFV7nNszHFSd7+JyiLEEX7XinBYy8qblv0e7WGTqXa7q1QUaikEtzxRAUQKkOlmTQ3Ko00JX4EoqMNqwrQaBPIEMWh2G0SJC7F6xOhWewAsguhVrZNVAA0rUWbU0TxhoQOR3v/tdMWBMxbBImkwe90fgvjOBdK8HewJbsalSElYtZkDPzYs9d+Ck0x+H/6Qybv7B2xB+3tGWz7OLPw1f1GYzSlTZfsUNHoxqA3tu1cL3LfUEok1rMSuW0lpkmjh3qG16hCVQ7Q6OkvZwOgos7XYvUPy6GnFOe06xXI2qR22yabHDqsVOsQop9Yk0UOlx3ni/q9y7BcHkcuHhsRNwSvQWVODCXMErknJq86bEux2QQj2br+BlL70Ho38WQz7lx6/W/Q9L59hFZ4GLYexYi1UrziM6iHONXTOcSORruh2FCqv501zNVnQwotciVYdgQsardna4NfU4ZyR4J8faPMbhAz6t8El22fLxzlES5KokzvnMg55wWNne7XaRX7oXvSE3cvUxSxXk9s+Lf6fcu5X9O8EGg9Ktnor2AZtHUEqV4DrhMDiNrlWLBpsWnf7mjck3kYBU/V1LVi0Bz9ImSL7bOsATfFVV5ahCwoOrd+0IPMZ7lxbhA/G08uNLUpXasXUFI3KzrarmKoqIW3sU50GFrYZ8uJjXhpY9WdihooWVQUiN1586T3hXDt03kUhE/FFxD23oD+EHLzwFnzxnEF94+Sji2bjYiM1atVCxotcVg6tc9UnNZ3vgH1w7HuddLK9Ss6tofLBVi5qElicFqhUxXFGmquvESfJZlferwbrCxkR2iBHIc21atcgCt93+rs2tv1a/l7iinor/dsFIOpXWtOJc2sSpJM4b73eVezfF2TRgbD4+DH+8B/54GAPzlXrM165VCw0GzRTKKPb7gWgZ/rEsgscfZ+kcu+gs8HvTDvtFFaT0cpaFa6H4p7vDWJfHOeUe8lrYORj0IP7AhPpZ2lfYGatycLLbjMWggWdywuPcAmfDbUrtuN8JoyzOk04WqmAQywZ8SvPuQDmHUezF0NbFpfc7sFD/upX9u57PVKrE+a39t+JLb/oSvvOO76D0Qj1i23bQVZwrxB42GNQOxbkEbQRWqmB2J6+kSpYg5bZ+j3O/cqUgER5D9YqaNWLfjoV4vHdpgNhkLI3DR6rDDVWAghGpAtD5u8iNm4aMUMGF+4abJc45oa1b9aVjOKhoa9asmhgIGz2OrXooy8BVt+qA7LLuiR2PgfvGECz6sPD8BWAUphXnFPhFERNf5/JBlMte+Pqdn/DdhXlw4touFUwPex9VVi2cOFfdfkxKcCrQkfJXr+LcvuGgKmw4jJ7P9hXQiGQhgQPdu+0qzjlxblcCyDHAuhda8RyV/uZ2E+e8K0GHP7CdRRcZJ5Eil1q0zVpI2pUnyEGDi6FB3P/AaSjle1E+fIuBCG8H2WIWuaILUV8RbnhQKERw0lGnYKG4lMx3sbbB1xI7bLNU7CF8XdERe9BzTnkH5Ruq1zCeX9uxj6jyOJ+vqc11zeJaCfQZ0yBVaqAyo352cjBo43NlxunACVtEVcNBpUCB4i6ey+sEz7PbjfOc/Cw2/+G9WJ9bQO+LMrjz/60X80py00nI36ZdxTmJG3fHqiR8spxET6Rn7SnO77zzTlx00UVYv369IGx+9KMfGb6fTCbx7ne/Gxs3bkQoFMIxxxyDG2+80fCabDaLd73rXRgaGkJPTw8uueQSTE1NGV6zZ88eXHjhhQiHwxgdHcVVV10lpr52MvYy4ly74tyi3xSH/HnyMuaDO3WBE576hoPm6kM0VKmKI/yam2hVWm64ox0+a+s4ca64eql7MKhExHD9zd/zvNWPtwDqBN9sVXimyvtH9zT7xs9UhcexvP6NGzat7+Sf/R//8R9K1vpNAxF43AH48h74/Tl4YgeAFlvFmiGZzSPqiqPU54HnuC04+wdnw+3rNm2tZaQcHg6qjjgvaBteRHGebHlVrTiXnTO2EueqFOe1NZh4SLsGmzbatVBCR4Vkc4pzexV3Bw2abYE4n+aK86h95+t1u+sCD97WrAryuad7RzeJEFQUb8tzpjyhUbSgcu+WBE2+J4w/PPhiHNj9AhTnjxHrELV7Z4qZtolzCtfD3mpHcL4QgcemTsMu7MGz80vD42iOgm6oUHNzIlTXGhBhXbprWnGuyOPcyf2Px1FmBlWu1PFjB8ialtb+xnMxp3K2P9Y263FOxWbeUW8XxhhxPqWYs+EcRMAFpXl3JTAkWrL7oz7EBsaxMLwBuXMurD8DueLqOUTd47w2HLQ0O41Avtrt3hdUJ/o0i7az/lQqhRNPPBFf/OIXm37//e9/P2655RZ8+9vfxhNPPIErr7xSEOk//vGP66953/veh5/85Cf4/ve/j1/96leYmJjAa1/72vr3S6WSIM3z+TzuvvtufPOb38Q3vvENXHPNNVgLxDkRobr9Iw3qZ4skrlwEqX3GDs8v7g1Ow0F1QFbESW2uakirSqsWu5V2Y5w4V2zVwtup7VCcWw0Es2zTsKOtU2XyKn9e+jnaoa7hHse8Zc3qUJLGa0+b9r/927/hP//zP5Vs4JsHokA4hNdceDfe+/Yf4g3hf4evEBMDSswgk1mEp5LD+NQe9N5xN3wfeK/lc+zCWfAAku+ptlm1WNy7myvO1ccecl0nJYgKuyYnPc5V7SNSFUOfp85B8CsNCCX3inZmT8wzaxcnFOfcqoX7rS+HGZYwjtqoOOcFFh0e50sxt1f7vaMq3uZdbo1QuXfLmKYS8sKXy8C9GEPpD7vFvxFxTkR4qyBbtnw5j2Ihj/KIB7nhAALHbLR0fl10Hp6Zide/3jHSuzY8zpmAJ6KJOJcEv+riH99z7LAa9XnIO9llWXHOrc3I49xuyO73WDYvcqFOnYuxHKTFihmxptHT357zJ+GkJOnNFCtknCfnrNhZbBlp8DhXCf5ZBDwupXl3xT8oGG+fv4JCrw8ZfwCJmt681f1bFjnIni0ScOPSf74P3/rsbnzj83vgdzvf5d32av2KV7xC/FkORHRfeumlOPvss8V/X3HFFfinf/on3HfffXjVq16FWCyGr371q6LC8bKXvUy85utf/zqOPvpo/Pa3v8Xpp5+OW2+9FY8//jj++7//G2NjYzjppJPwsY99DB/84AfxkY98BH5/5/nK0kMpP2zdanNCRKXivJa826X4CjBVe66kXnFOG5IkEvqZzYRqqxYrsLvNm6qXlJ/RXq2aOOeLOifoVSPMNlvuTWxl07Cr7YqTxFYV5wblgQ3PrMHj2OKAUB7ANxLnHo8Hr3zlK5FOp8XXVjEQDiIc8iO/6IObAp9CGf5yBrmSud+hmJkByJ+tUkYukcd0YQFd3draBu+2ouTMDvRY9LpsBp4U6Ohg4msA7a2qEogkH1LkgMe5lUFjMpF1IokdZiQy2ZlIIn018PV7yAnivGE46GrgLcojNg4HleQYCTC0WLWsQEJrtYkrltCngThXuXfL45OIZ5i6xPI+FBfL9cSbPFKJEKdhoauBussKpQI8mRgG43Mo08+401Db+N6F09g5WyXOiVrdOhRdE/E8z2F0Kc5l8Y+sLWkWGnXSqCfO7dlHyOe8mCtaIs7tVso3ok9cq6TIw2k9bUfkwBXTThHntDbTnmhuOKj++70Z6BpTnG9W7MiLLXbOhaEuC4rl6XPXSZxHgn6leTf8A+Ivd8CD3LkhTLtH4Tu+vy3iXHZz0jBwutendh+N2ZwLlXIZx9ssUmkG5ZniC1/4QqEu379/v6gW/OIXv8BTTz2F8847T3z/gQceQKFQwLnnnlv/maOOOgqbN2/GPffcI/6b/j7++OMFaS5x/vnnIx6P47HHHkPH+5sP9tg7qMECiUjqsVxNhWJX4mpUwKgnzomMkMVcldVwlS32CZvbrogUGq6RHQdiaUvDNRvB24h4e5FqRFQpztk9F1qDHuecbLPjmeXPkFWrFk5CcB9Zgs/nwzvf+U5cdtll4muroE36+YsTiMz54S8BkaIP/lIa+ZI55UEpMwl3zealmHKhNFANBrpYu5B7p13rQGObKieOrYAnVaqHgx5ksWGxeNZJHudm1zM+qMsJv1FOIvOOr9UwV/N4JfGeakuftruXWrj28ncjuz27OkIa7xP6nNtVB7Z670Q1PKsrWiOajD14l1uz+13l3k3XXeTGLhc8PXmEB+KIjE3jwP7HqsR5qdDygFBJsgfjUwhk4wilFzDz+B8snV8XnQV6nvbUrFrIpmUtDgfloqBO7K5qBO88tcu+Qn6uVoRHvDPPDqW81cLx8pyBM+JRGbtS51K7nYfcqsWu7kIeD9Oea6Zbktv7DIbtLd5LISIVK1RyZTwP7w0Hlebd8A+Jv7wBN4qn+hA/aQD54ep187g8opi9ml1q3eO8UkHAXcZCYhMmMkfgQPIIdAKUE+df+MIXhK85eZyTMvyCCy4Qti5nnXWW+P7k5KT49/5+I+lAJDl9T76Gk+by+/J7zZDL5QSxzv8cqv7mjcGrlQ3RMOnYpsXM6HGu3qqFJ/bVCq8a8MVelVWLHR6XjQNCSWVn1u9rVcW5RuJcnVWLAx7n7J63onJ0YpgvD4ytkmaGZEFzgkOKtOC6XuQztGm7gBKqivMWPNaaIjcLT76ETDmKfXPbkShsUH3KXdgMeT/aScrRWkAtpI37rxXwhFDH3ANusaGLOLdLRcXjs12zS964a8lvlCv+Z5kiajXIz47Udnbby8gYSt77rRAHkly3O2E9SFWqkHTiz3zUhphbxUwhO59TUpqTStAFFw5/8aM4729uxkveegf2PfgjQZy3knhLEMlOw8BDyelqu2WlgljFLRJyN9y2WFN2od/fXI552D6s36bloFzEJJGbsiEWNgwxVbiG8RlidsxZ4jkUFz61C77n6BAYrAZerOYDVlsBH25pJ/HMwS1W2o1djZ7+9sVMvAPTzFBWJ+19RplAgs97Uao4V7yfu8jjXOzjFbinvwfvH67Hzu/8PVy5PHweX0v7N/FTNBi0gjI8iWkUStXr7gnpmYnYLrw6iHOyXCHV+ZYtW8QwURoESsNEucpcNa677jpce+21B/37wsKC8EzXjacmZureuX3uknhfDtVEfjmXqb/f9EIMCwvmBqJMxJeO4y0XDzpvHeeez6br77kQi2NhQa29x77pJR9jf6Wo7PxL2aVrNbsYb3qtWsV8onoNevxexBarE4ObQeW17/Ohfv5P7ZvEjmE1LY1756rX2+NywZ3PYGEhq+eez2cb7nlzmxh9dvI4xWzGlnueElb5nvFU2tK9Mzk7Xz+Wu5TXf/75Yv39JhcSls59ai5RP5arWNB67pQch8YjyGeqwWq5WAFSMWT8Gcz6Z1tq9eYop6fgKdBwUT/2z25HuncAvakC4nQ/+YuWz93uYu+fOkhFKotYERtbR4msofej4FDdcNB8vfitqh2bQ3YrESbj6tpGefeMXcWL8d6QIBQoedg5ExPrRLsEmtFqbW0ozkltJQvmgw4MBiXQdSbVHSnfVyvAkLJQenI7UZxoJMdU3Z9GEtpva9HeLOlkd5cb3d/zqTSSpSUhRmB6SuzZZZDasbV1Uybo/pkJ5ANBuFBBqjSASKVq9UIKuC4OHX/zwxwgzjN5c88UJ7J1idYMyniFllNy7aa13K4CrOzapT2BYjcz78vJamc6rpbeM94mcZ7sAMU5J+zpfLigYjWk2XMSsTHe5rGDGZtB2aXnxEDZ0Qaf842KBh9rFbAFhoVOjZwvX/LUg9h0axEhXxie//EOeNaPtUic50WRmyw0C5PTqNRMUYNDBfiSu+GjW6miZz5iK1B6xTKZDP72b/8WP/zhD8VwT8IJJ5yAhx56CJ/+9KcFcT4+Pi6Gfi4uLhpU51NTU+J7BPqbPNE56Pvye81w9dVXi8GknITYtGkTBgYG0NurfzOdyZbg9XiFx9oxW9YfNHWeQOeiCmPZing/QsXrN33sqbyrfpzhvt5lj6Py3IcSxfp7eoMhpccmlGYz9eOvG+pHb2+Pkvco+0P145bcXkvHzJYhjtUfWf33V3V9to0l8Jvn5sXXKVg7fwkiHRZy1Xuf1ObDQ4OG76v8bEdihfr1d/sCpo/t9s3WjzMy2G/LPU/XyefxiinR5LNp6dhTqfr5j9lw/v1EQPt9olU7XbZ2XG+yeq8QBvuihmNls1m84Q1vEFZeNKgkGLQepIwdtR4L6WpwVylVEPYXsBjxINoXRcDbXuGlVCmjmPXAjwqyqSBCm/rhi/jQ29+LaGCpCGX2+ijxl+vCVCJi17AiCVJUqiXOC1pVVFyl/VytLV6lVY6dnVdE3pIy8ZGJeXHdKDFq1SO80YPRqSSWny/3AV8JnKi2q71+ObsWuuZExpbKlboCvRGcWHCirV6XzYHdHWN+71IhzWy792qWSqr3booln5x2IVFZyt1C8Vj1CzGupHWrFvGzCzPIjVRJh+TiOgyVS3C73G0Xz7voPDxT8zcnbLdhMGij9aIaqxa9w0FVzOWSIK90GW/Y6RPOC4AkRDJjySMFBrQm2mnPJ9FnSXHuvMe5Yc5bm/e9kx7nEvFM+/E2H6hud9zUSJyrArc78pSLuPjii5Xt3e7AIMpwwed2oxL2Il2MIFOKYH0sDs/G9cK3fLXCNz2npDcP+VzIPzdb//exHQtY/8j/wlilDH/pvcAR74QTUCpNogtPf9wNiiciBMrlanXglFNOET46t99+e/37Tz75JPbs2YMzzjhD/Df9/cgjj2B6err+mttuu00Q4GQD0wyBQEB8n/+xC1T93LeQEl+P9YaakuaqwRP9tWbV4mdD2HR4nPN2LJWDSwwe5xauud0elxLr+paU/VOKVIOUBMrfhS/yOhAx+PoXlWwadgVPRNbQcBsr7dJOTVevqgQDyoeDNgueqDtIZYfQ2GHjKGTpulfEHlTOLQjPU2rfbhe3F8/Ft7/1Enzv716N2b1DCG0eFu3kXaxNOEnMyb2WrMrM+C5y0B4q12CemKkEecdKfvM5ZktnFTIZjPi9tlqH7GAEyzOzNUJuDSWx9J6SEG3VqsXJlmMOeY9SETm2Qus0/56u+3olcCWWVXu1ZYlzG5T0KhTnnPRYjuxXuXdTLEl7a9w3WG/7DhWXCNJWFeeZQgbliht7nhnCnf92Bu798SlYcI2J/b+rOD+0iHOv22WLTSqB9ipJnpseDlr7Odr2+DPaaZYyK+XXthLnrFhhdj2WZLUThdhGYUO781XszvuaoYfxTu2KPmTOTvaodsZ6nGMxY9Vi8Di3OW7i1rfTSXXEOecMg36v0r3bHRgRf5MgIhjfjt8tvAYPLJwD90KsziespDgnPpWKfBQf0mOa37/UlV4OhpDqOwG50Ga4wpvgFNpmjZLJJHbu3Fn/7927dwtF+eDgoBjw+ZKXvARXXXUVQqGQsGr51a9+hW9961v4zGc+I17f19eHyy+/XKjD6WeI4H7Pe94jyPLTTz9dvIYGiRJB/pa3vAXXX3+98DX/0Ic+JCxfiCDvNBAJSVOrCXZt3MYBY4U1o35R5bm4EhYMGzstmhVlhL9oHSnRg11Q5HFp3wY4Hl0izg/E02vK37zxnk+rIs5trHxTwEAEl9nWTicH6lFLHikbaUMjoo+GzVq99o3EOa3t3/jGN0Q3kqp1PjwygnKG1uYKKuUKMplJkTS36pFq2MzzRfiTQKlAJB8Q2TImWr27qrW1iQUH23Z54ZueZysJKJ9XoWs/oeedyPO9CylMLKYsrQHNCDm7B2xyL9ydM3GcttU4U6fThns3gpIPss+ZiKUxl8y2ZDfTKYrzakxWxWI6t+y5GNrqg4eO4tzu/VsW7K3E2wmmVm12v6veuymWpLt5dvMoor458XU0Wr2/yeecCPFWQK8rFN3ITXuw/6n14jiRFwWE6s3rIr/9ruJ8LYOeywOxai6zebBHyZ7UTmGN4nmza4MkEqvDcPUQiRENaxjfR+wkoIOskGlmyDEp5aXqXuXsM9NWLW2SuPW5aDbP5FkuB29HvEbXfqbGFdhZbDnY47x93kYKDug5tbtLQZfinNvmDPRElO7d7mBVUEbFbn+0ulaUKh5UpqvEOcWqK+XfSeFvXrNa9bmwOT+KefekUKC7Qydhz+EXiNedOHYinELbd8H999+Pl770pfX/lvYol156qbj43/3ud4Vtypve9CbMz88L8vwTn/gE3vGOd9R/5rOf/axQpV9yySViqOf555+PL33pSwaF+s033ywmvRKhHolExPE/+tGPohOxhymwNtlFnHPFuYUWLN6+FbGJROTEufSwVIkYq+SKDbKYVehN6xMJnZUWe6cqxyPRoFA30IykSUXE+RRbzHUrzlX59XG1QsiG7pDG+z5btBbA2u032kh2UOBs9rPmwXtjEELP19DQkNgbVCUSgUAY68M7EfLEUfQAufS82JBJdd7ueRN57k176hrz8OHrUXa7RSLfxdqDk4pzvu5TIdVKMsGVuToLACQKIOKcBrHtX0xh65C1GRlk0yEDeCeJc97qv1YU59KuhYhzEm1QQriaKpt7dTpt1dKK6o4nuSo7B02JUxRZKolj2XzvGIQqJpWaRlurg/c71Xv3GA2yd7kwFVlX6+qqIFKuxpq032ZbiOmJHM+X88gXXfCm6Pyrz4d/vF/s/+GA2tlKXdiPXdymxSZ/cwki0kiNajYXkd2XugaDNh47o4E4d05xXrQ4GNSZPZsT9matWsJswPZaUZyTI0OxNsF3m8W40ZJVS5vFCsoV52txkxNdevR8UScNXTulxDkXsNFcpJC6vdsTWoddx38KJV8/fnLtP2Fz7d+T+2ZAdw9ZpK20f8vPSFi1+N1wzeVrHQou9G0aE/u63+N3tOjd9op99tlni5tpOZAH+de//vUVj0EeOl/84hfFn+VAhPvPfvYzrAXsdYA4p01EkqBWbCucUK9yz0UdinO+IVFrVEbhNGK6RlaJc+OQD/s2cBoaR4QndUgQcW5mKFojpu0kzg0Tvc3f89LWwH7FubdO3Fu59sbhYjYR5w1kh2nifAXFuQ6QIjzcn4HHVRDpdzG7sGqrWDPQM1+plOHJVa+3212CZ7hXKM+7A8bWJgxWEDYrWjkpxwupZhC36ffYMhjFXbum6j7nVolzpzqvZAfNUKTqtU3kS7vDxpxWnB80IDSVXZU4X2AktVPDQQl8oNhK5IFRAOGMHY6VFu9OEU4osWqxOU+QVi3z4WGUMy64/RX0uPNIlasWK0SIE/m9UvJMdi60z+eLbnhTS793cNNwNfl2O2PX0IUmf3ObiXOZO5D4ixS17QzlpvifK851n6NKqxa+ZjvpcW4pTnLIqoXiPsoZiEPmRH47+4ZThXorivNn2Vwcq3GjNcV5+8UKSfgPhe2PmSgmJYHEZDwjuBYVnA0vohEX18661Qpcbg8q4U0oFHMo9bPuwr1zGGmh8F2Pjyo0k8yN/NwSx+Rb1yv29IhPzZBUs7Cvr+kQBifO7fRYkxsu9x/U4V2oVQGj0M+4cWOn4gKvUquAvEZWvGm5kspupZ20a6Hzb7fivZriXLdViyq/PqlWoKq9na2d8l6kYpcVX2OD+ss2xflSgLyQzitR+zcmDMViET/4wQ/wk5/8RHytAiKx7uvD3v/wYs+/+vC1/IurivM2Pc4XEov4S99XcPirn8PgWdOIbY+ihHLXqmUNw1BgtTmR4uu+lcJ3o3JeJ4m7ZbCnaZedCgLRidZjOUiO9kJS0LeDBEvAogFnkvAhRn7PtuB/afA4d9KqhQ9IW2EviTls1WJUqhXW8HBQ6x2eqwlsVO/ddI+QLWKqJ4pMzouczw1XD127OHxuX50UXwn0/WKpCNJYbBidxcjmWUQHE4hsWSf2f7+3S5yvdTiqODeouduLJ6lLiDqudBPn4UPIqoV3qFpWnDtk1UK8jeyeaoc4p8KM/PycGEbedM5bG4KP3XNLxLmzivP29vG59FLMREILJyBFahSn8k5HK5D3Eq1hOvJun9tXzbGHlmLU5IGqVYskzpcTYMviRpm6zAJuLJ5wCgpbNqGwcT0828dF0Tvgddayu0ucK4BMIskDm4aD2oVInThfW4pzTlbqtGrRsakbKq4mlYJOXHOJdX1L96cKuxY7FedEdAdq3QpW2g5l0KW6qNKKx3kz1bvZxJuCYrta9viz1O5Qm+WtWozXnzZt6la66aab1BHnLg/cg73ITLiRWXBhcjYuKvatDheTWFzYj2Pdj+LEY3dh+5ExbDn5eULxRi1jXaxNcEWrk1YtVm0geDKgW3EusWfeOnHOi/ZOqLZ3DPeZtmvpOMV5cvU1mRc87fYZ5ehrcS8xEh1OE+fqFOfJNTkcdOVOMdV7N+3RY9EwKm4v7gtux5y3F3n0IJGeEok3keKF8srrJn2fkvdcwYXn/9mjeOnld+LMS+/GwPp1Immn5L6LtQ25blMsv77fXhVi2O8xLeThcbBW4tynmzh3xqrFTP7U2InuFKSdHnU8UqdbK+Ad1lHGQdgNbhGcMkmcb2VxpB3g8Vm73Z3SpsXJgeqcV5lSNCBUrle09ujIu/0evyC43eNLxczs7JLV2kpzxmRxgDq8ewJe7F0/g+JLvHCfFsWO4t9hxxPXYPjpz8FJdM1ZLYKUo1J1S8Oz7JwWXF3EsnX/XTPvnWQLshOKczlUVRXyRRrWUtK2qTdWXHnbcScPZJUYJ+9INiD06PEBS8cj2xe5OdlhvUH3fK6Ys9RlIRNyu1VsfLhNplhCr8XE286WPaPiXA1x3ujtSLMtzjnnHKTTafG1CtAmvevSP8d1vXGUin70xAuCTG/FI5UjmZikCSfi61ymB/4hf1W11iXO1yximYJzw0HZs2tVRWJXCzIdm9peiah/biFpuW3UMKuBJWR2YUdNcU7YORPD2Yevb/vc6de3Y99rhuGepYRqNrV6QjVfU5zTZ2hnp9VKe8mKVi3svnbC45y3eKtSejUey3aPc5PE+Wr2cDr2biIMds25UJrsg883i0rFhWJmGp6ho0RSvlrxW34/k8vDE3ajVPIi5e1DX7gf85n57mDQNQ4aLCznNpCK1c7cm8DX/XaFPIZClG9tKc6Nsyf8Dnmcm7BqcbgQW3/vWt5JnDnNu2gl9uyEDreD+Y/W7ifipsjajzAaDdreXUhWJPQc0P3P74FWwOfC8A4/OzHK4jwSKh4+siT4MAP6PGThSYjvNOzdPYnHUZr6Dc4/8lHsiowgnwqiuJird4HT0G4izn0e37LrC50n3W9fDv8bFk9YxEh4CJeVFuDN5+EN2Ntd1IgucW4RRMLJoiFXANmBSG1TpPenB8FMAicJSIo5GhWg9ijOzStvqfX4e79/Riwk5x610Zb2e57gm/Wm5cOheHJmp1ULQU6jt1I0kiQqX9xtGchjMggktTm1PDmhMjUov0xazdBmIocB21l0GVREnPMWy8b1yufz4corr8TCwoL4WgVoky7vjOKF/+90sU4+6qbhoO5V1WoHnXdqSoyzL0W88GwcQ//G6oCxgMfZlrEurCvOae+zu4BptGqxRspxZa7uNl5SnT8yMS/Iv4VM3pLlh5OdV9JrU86JeWbGnOKcSES7CRsJHm/OrDLHhfYNuW47qTZvLFivtJfI+5riUieIfq4G16E4p+45O36vgM86cS4Vj2Sfwq1fdO7d1L1LPue/3PlnePaeClDswdmXnFb/fiuKc0IxF4e7TEdyw+Uerifr3dkkaxtO+ptbtQ7hRDvvYl4LxLlB+GVj/mrV43yxAzzOCf1hv6E43BJx7nCs1NzjvLW4VQwwr+Xcdvubc55FEOdt7uOdYG9nUJwrGBBq7Pr2atm7w/P3IXzgv9Dfn8fkUC/y6QgKlepnLzrGKtWOsRAO5o3oM6rU/kdWLfFcdZ3fEKCOornqiwLDcBJdqxaVA8ZsXoxpGq7V5FsmJ5QAqhg60Aoo0SRbG6vDQW9+9DncuXMSX73nyToJzEkE7VYtJgMRJ4Y7Soz3hZUtwjPJDGSjGVWS7YAcdkPkN/m+tQvuqWo7ca7AqoXunYoDAVS/Io9zuWmTw4wdpAElxz1DPRgbWcBxR+/G+QM7kYwtiq6FlYZcN6KQnoWnUoTfm8OO40LYfma1Ldjr6dae1/reTeuA3eQntSDqsGrRXYg1+JyzgU9mYOx2s/85oqRhY629nxT07RTx610/Dtm0SAJcOnXNrkKcU1xUs9R1zKtTgtZ9+Xmv5PMqk1ynSA5Sqsl4T6XiXN47du3fxoK9NcU5b9XXDSHGcLkQ9w0hlaa41Y3c3FKhZTWPc+oqo8J5JTWLiteFiseFkndJrbfSYNG1hlKphA9/+MPYtm0bQqEQtm/fjo997GOGGIe+vuaaa7Bu3TrxmnPPPRdPP/204Tjz8/N405vehN7eXvT39+Pyyy9HMmndlutQJM6tkNI8X+cEvI6cw6V4OKhUP5NVjerBgi137Jr4XeIdYtXC37vVOWOdYA23NFvP01bcups9p9uGeh0t1lOnQjvzxUik5/RA9XFm/zylwF43bUfRLjAo/goHgsht6MOB8a145Hnnin9zu9xiL1pu/xbEOe1blQo8xSQuuG8eZz6WwElzLMbtEueHjk+q7QPGDH5T7W8kTqqQaJqvVY9zrrJ6fHLhIM9MHZ+HCm/aRC7vWPV4OBKs+2KTVYu6waBLhLxORCwqKPj9YcZmx8nA76Cii40BVMTvFWozwoICj3NKOuwo1FFyHOwP4vij9+DCl9+H1219EtnYAWGz0s6A0HJuDt5cAn2xKRS++C947uZv16vnXaw90N4nSTsnkigzLa+rEYx2KOf58HOrPudOK8454UJx+rMtFgIo8ZIEpJNt05TEyvbhWaaMWi0BdFpxzvde2o+bFTDpGkvLPSfVgTLe4wIZK6DflXcr2BlrW+nwlESfnc+pVJyvSxxAaGISvmf3oHLfk/V9l9q9VyXOXR4ECjMoDPmRGw1i+MITqt6rLvchpTj/5Cc/iS9/+cv4x3/8RzzxxBPiv6+//np84QtfqL+G/vvzn/88brzxRtx7772IRCI4//zzkc0urR1Emj/22GO47bbbcPPNN+POO+/EFVdcgY4nzpnt1logzg3klcZiFO0RoRrRqVpxbvfeZ3VGlJPD4Dn4e3P+qGV7LweJc36/tioc3M3iqm02+5s3E5S0ozrninOnBAecW5msWeNawUp2qarg8g+JDm2P14VQOCO+LsSNfBmJ15a/10lvDrjn9+Ott8/hPTdP4yWPTFRFgy7AFRiBk+gqzi3CrsFcq23cZhTndINKFZLdbSjSd9GK4pwvAH+cWjxoc9SRIEaUEOdMaWfzJkikuWz9oeplq8NJVhsMatdQXCu+go2t4QM2+6ZabTV0kmwiklv6zJK3pFnIgkEzWylK4N7whjcIlRNP5qyAkuNwMQVP3AdfuYJgAUgnY8JmZTXFGoc7Pw937fWFdAmF/mqidigl339KoOdI7n19DvgnG6xaLCvOl7rGdCvnNzPFOam01zxxzvwid7Zo18ITL6eT2OGaXQsJJ1Yqxkp/cyeVUxz9tVi5UKo0TcBjnaIOrJEc2TaVaivFrDLkcmKmkJnhoPR7S3u75RRqOvZuUpzTahZyZ+DNZuDK51F4bt7gk7r8OReQLqQR8AYwuvdBrJ/cg3XT++B7aJfY+2nfPpQU53fffTde/epX48ILL8TWrVvxute9Dueddx7uu+++esHmhhtuwIc+9CHxuhNOOAHf+ta3MDExgR/96EfiNUS433LLLfjKV76C0047DWeeeaYg3r/73e+K13US6PeRxDmRYiQIshuGwZvtDgdlxK/uGRlS0a6COC+Vl6wi7d77rHbsyj2FBEB22dI2A+905nN21oJVC+dARPdzC/zBs3wwqGNWLUvXvJ3uMSk4IJW9zs6Q1e57ec+osGpptEvVsXe7gzVi2+1C5aIodr7vCOx7w6b692k22Fx6run9Q/c6/TvpO0u79uGOqUtx5/SfIxbfKBQuVEx3yeM7hC5xbhGcqHVywJgZ25D5NE+m7FacVzeufMk8cc5V9s2Icx1WHEbCo2ip1Y02bztb3STW1UhuSlqt+FUbFef2E+dm7nndVj4rIWRxuI0Tg8WaqQRJIWuGRKDNUAbvywUhqVRKDClRSfj3lnIILPrhKbsQLAWQTSdE8kx/WgGdc6Acg5vWKrJaTbvgHh0Vx+4qztcmjIOi7E9EqIgmbTasWLXQMyWJXDvij/V9kXrHkmXFed75ZHAHa/HnCsZOXYMbwQmjlexaDC3HHaQ4X65dnSu87Y6rl0u4VficO9ExZtUasVV1rOq9mwQetMcWBtzYcfrjOOacB5BLV0le2nfz5fyye3immBGKc5pBEknvQqVShKuUw4PTz1YV5+5DS3H+whe+ELfffjueeuop8d9/+MMf8Jvf/AaveMUrxH/v3r0bk5OTwp5Foq+vTxDk99xzj/hv+pvsWU499dT6a+j1dK1Iod4MuVwO8Xjc8McOUBwv86/qrArXmlKcc9GPbuJcHt9slysHFfkrDu19Qa/XUgFQ7ikklHDifmmmOOcd0K3ORXM65uhhs/VWy2NJmCeJc4o7nFL6cxEl571Wi62l4GAw7KzYYLw3XF/3rD7HBo/z2mepPO8ODgmCmxLm/qE8SiE3FrOLdd4g4o8gmU+Kfbq5xzmtWy5kdh4Q/1YoBxHoc4t/p6M6TZx3+8wtwkkiTi5gZqvJ82xisFST2oVAPZA3r+LhKntKGukPb33S8TupsWqxv+2VY0wswtUhC+QNb3Za9BRrG+IDLHQiYpE4p4F2zlm1WPcadVJ5MMDWNyq4tPuZ50vlusq3WbIQCATwT//0T1hcXBRfq0Jg0zDy6do9Xq7AU0ighNYV5xQ8RRGHKxNEohjGb/aci63hUXhd3kNKtfanBGPbrv1EIiVu9PxSx5rZIdMygaQCqF3KXPKnJl/w5+aT2B9LiUDY7KwCngw6tRduHIgIUpHWplYHhHaK3yhXnEvruk3MSme5WK8jiHNOHqRzda/5phaIDirOeYs3FUzMxkpO7980IJTucTPEufGcm6eMOvZuWlfoXk0N9uCkk/8g/m06Ub2/fW6fUJTTHt5sDyY1OhEf9L3e4jRQe0lucIuwaDvUFOf/+3//b0FaH3XUUfB4PMLz/BOf+ISwXiEQaU4YGxsz/Bz9t/we/T06Omr4vtfrxeDgYP01jbjuuutw7bXXHvTvNGiOzkEXds0lUSxV47deb/X9mkEnkV/IpuvnMBdLLHsOzTCzGKv/bCmbbvqzqs7dUymL96JHf2Z2Dl4Ls4Um4pn6eftQWvF3Vn3ts5l8/b0Xk6m2rjcp5RdT1ZlcIXdl1Z/VWgDKZ+u/x9RCvKXfY3oxXv+Zci6DlX5Ed/HKUynVz2VienZFC5PpZBaJbHUvX9fT49h191eK9XPeMz2PbT3elva9jLQoW2GNseP8+3yon//T+6awqd+8Ne70/GL9WJV8ThDmZO1Fsyzo60zGuqq9lA0gUK6g7CqjOPPfOPHmH2A4VULe93EUzzlTvCaTzuCA5wAGQ1U/9HqxIplBqVREjw/I7iRV+oD4XnikhHKNeE/lQ0ixz8PKdTfzs13i3CJ4gM/VKXaAk09mSFyuNnZKcU4bGg15NKO8biwW/HFqwTC4UEd1k7eqcsVcq6AKrFOtbhLratVLwmQig+MsWrV43S7bfFMN7ZFmPM4NVi3+Ne1xbvdAPf4Zk1KiXeKcq2yaKc6JTFy/fr0YXKVSERIYG0UxW3u/MuAtJYRyvFWPc/IPjrgS4mfzGT/S+V5EtmwQSqyu4nxtgita7S54S0ji3OyQ6UblvO7BoNznnIhzUhztW0yaHvgk1zISsDvVOk1xBykWn5qOiQ4qIkdX25c7wWJGYoQGKK5BxTnfS5oNCOUWiE49n40+vqoV57YS514PEiiYKtinWZzLu1zt2LspxpgYHkE+E4A/lEMkkAWtlrTvktq8UC4ggIPvZ1KzETFO8XYkVUTa2wu3q4xcZPsh6XH+ve99D9/5zndw00034dhjj8VDDz2EK6+8Unwml156qbb3vfrqq/H+97/fQEJs2rQJAwMDYsCoLmQXcvXB7JtHBsX7LYeVvmcF42Xv0nB4r6+99/FO1X92bJjOv1fbuQ/0hOFdrOZqgUjUUgfPZA718x7t7131/FRe+2BPcel6u71tHZu4Dk/tZ0f6oi39rK77xhfuqf8eObhbep+S21P/mY2jw6tyG7rOnTDc1wPvZJVs9IYiGBhY3n7lj4tL9/lR60ccu+6bUyV4PXvF1zm0du/E5hL1c1832Nfyeek4/y2jg7h3X9VVIeNqc61pgHsyWf+9Rgb6RGGU/lBhQNW5l3v8yNQKdKPI4+JHqt2plV1PI3Txy8XXQU8QBX/B8J7E6bjcHrgqJUTCQGU6K2Y3EfrG3XC5XeK/B8YOB0LGczV77lRobhdd4lylx7ndVi2sbdKU4tzBZIr7LtKAUK+/PeKcAuLGNiGya5FJmEtTIcMw1K0NrywJai9canXzO9r2Q5iMmWvPocqgJM5HoiHt3roSYXb9zZDPi04qzhV7nEdtLtRxsoMXqFoF94HU3Z7K4esfBPK1xrEK4CslRRtZq4rzmWQaTxdPBh7OoZL3wuvNwhftPeSS7z8ldIKHsiShpH+yGeU2LwDYFX8In/Nnql8TgW6WOOedV062TtNgOSLOCc/MxnDSxuEVXz/JhmrbLZZYWXGeaS3W6wSPc9691KRd3UkLRBVDxTrN5kfGHqQ6t0T2axxk2AzjvRE8MdKH5DNhDIZyCAfzmC2k4feFReGbvMwbQcR4LBcTNi2ZQhnBPO31fmRTAQQ3bheEe8irluB3GldddZVQnZNXLeH444/Hc889JxThRJyPj4+Lf5+amsK6devqP0f/fdJJJ4mv6TXT09OG4xaLRczPz9d/vhHUXaCyO7BVUHeNxGjUmfVM2hxYHw5qj1ULgYr0VtbTuIOWIVZmNRjiPQf3E0LET52qLiEabFY0Xu26O12s57zTaoLN3czffJtD/uaNlnZ84OdK4K9zWmzAORuaTadsOKimtcfti5DROUnP0ePxYm/6OOTKYQw8OQPZExn2hRHPxYWlWtAbNOQElUoZEb8Hleml57xnsDoy1E3tYzR81EF0Pc4tQi581O7LiTE7wNXP5jzOmfrW5oXB71269fIWfRc5cS49w0g1Jr1YtXnTmrjmTiqGmynOD5hchOm+l0kYDXGyC2GmTDSlOK/dH6SSj9hI3jYG2mY9zg3Eud1WLWyNaDX4aGfDpiTtpz/9KW699VbxtSqQ0qRU9IhNlwo+pXzV57RVj3MipG7JnY97fnAa7vvxKXAHqvfQodbu/aeETiDmBkx4Xa5IxNlE4m4ZXEqA9lgYECpjFqftTrYzn3Oe6C2H3+5eIpeOHOuHk2g1IZRenWQXZmfRsqXupSZF2E4obB3scW5tiK+TinMZb+cKpZYGunHQTBOJ5YaD6tq7x6Nh5CJhlJJl+Nw5+D1ZJGJPi+/Rfp4vHXzvUCKeL+bFYNB0voTbv/ZifPdjr8Et/3w2hrZtFoS7z+PsmqMa1GJPHXCNSrpyuRqjb9u2TZDf5IPO1eHkXX7GGWeI/6a/yWrngQceqL/mjjvuEMcgL/ROwgybrzRsY/6xfPdre/E8z9f5cbQT/Bb9kY02ZfauyyTOCtTWsXY7ZzplPyFQwU6eQ7P5Hitdd9q7dfAapue8rdJ13wmDQRvFApz3Wgn8dVyg4AT4DDlyCbACQx7u82rbu8v+AbFH06e+K3Uq9meOQWrPEt9EZDnt1al8ytCVTj9D+XnI74E3QXGiCxWXC7Gtb8b+jW9GYsubAY+zz7DzEfQax9LACb/tCgZe+aOhHZasWmwefsCrxzkTKphmhYL9i+l6W4cuz3buTWvGHieRW9oonSIMyJaHJouTNy5Xz5keDFobNroWhoPKQMWJ59WgmDAZwCYcVB4YNm8T9w0vFjSzZqBN+8Ybb0ShUMCrX/1q4a+pAkRwFysUcFO1uoJweRalMvkatxa0zqYy8MUycNWWKXek6o8eoap6F2sS3OLEKSsIYwdHzmC70SoMQxTtIs6ZjzYpzs2AFPYy+V1p4KBd1jMS+xeXgvhm2LuQrBcLdoz02jYUeznwhO4AmznCIXwja7Ge08qpVgekGTzOnbRqafA4twqn/PFl7FGpPXvSKrEVpFpQnOvauym2pDjN5V1E0FN97tKxpzE0fCL8Hj/mMnMYjVQHdXN/c7Jwoe+n0kmUcwGg4hJ5xmB4ELFsTHzvUMJFF10kPM03b94srFoefPBBfOYzn8Fll10mvk/Xh6xbPv7xj+Pwww8XRPqHP/xhYeVy8cUXi9ccffTRuOCCC/C2t72t/lm++93vFip2el0nwaA4d4g4J2LHLCHdbECfLkQszkLjSGSdzV/J7jJXbH9AohOdeSuBzoH2ZOpiou751Tq2JdfgtMigsXiaYEXVZnHH7rl4vXPLydiDeAMSDVDM2aroq5PmwljNvTl4jEuiIV17d7H3GOQ8PTjgTqHirsBVdqE4v/Qc0p5EuTkNDR0KVxXk9FyXykUUKyVE/QGE8wNIe/wo+3wob30BZvOHIdy/FU6jS5xbALXaSLsOJ1p2IxZJRJlMEYlqt/qZFPrWFOfNkxgpptG5OVoizg2tbs5s4LRJ04DQfQspQYC3snE3Qtq0EOwkD4z2RO1df0oY5fW3u8OikSw2OxyUdznYTZwbvPGXIWnaqXQ3ghRTL3rRi8Rwkkb1lBWQKvyEk/cjlJ+DNwwMe/xIlCiRbk15QIFWdHqm/t+uKA05LQtFWxeHwnBQZ9ZhHozzIN28ksqe9YCCbbpm9N7PzSdEgtRuEdJAxjncekxtsHT6FDusRpzfvWuq/vWLDmtuX2AnyN5n00AEexdSeG4uIT6TxvuZ1l2yw+uEBFCCCxuaqe5iDlogcvC4npMva9HjnNsctEOc85hjOcW5rr2b4lSyVZst92Eb5oXdmnvqSWA7kfg9ggRPFVLiawkaGiqR2zuPSpl+1wpKEVKs+8XefagR51/4whcEEf43f/M3wm6FiO63v/3tuOaaa+qv+cAHPoBUKoUrrrhCKMvPPPNM3HLLLQgGl4pv5JNOZPk555wjPsdLLrkEn//859FpmK7ZUhEZ5lTnLs3HoE4OWlvbJXJlLEx5sNnh2q2Cx9qc+LZc+HNg36bPO5ZZ21Yt/Bwo5qCcdKVzIp6JLF6duuYr5uArcCBzqVy9W4ns/Jy2xqIhpiSupFi7lZiVZltJOB03ka0jFU3oXpkykXtzDuSxA9WhmnTPbegLC8Jcx96dOeZqPDP/DB6dfg5F113wkZI8Yex2C/lCgjgnyzXqAotls8iXCnDDhf5SAanabLJgjxseKlTmKeZ1/hnoEucWQJtQxcHFOGzRqkUqzimJcVJ9a8bvWW4khI39EexrSHh1qgi5N227g00THUIYrK8R57Qpk3qjXfKbK87bHRLplK8g9yh1QmXKrZzMWrVIwomsZmTbol2gjZsq93TdzVj88OSimV2A3+8XPp00pIS+VgWqansH+1BYdKFSBA70+BEoVkQ7dytYSKbgXVxE3uuCrwy4BtzC5oU8VLtYm5CKVtr2nFqHefGu1fbRlfYTO4v3WwZ78PD+ebEP07kPtemb7aTlVCOIuCDy/EAsjf2x1LKFZEq27to1Kb6mb5+xbQydgJM3DgvinGLRh/bN4iWHr1/B37wz1iwqIhNpRHZvzRTncQctEDm4wk/1cFA773sr8TYvci03HFTX3k1qYnoWp9wjZKRUfa+5Z6t/e/wi4U7kEgbinMh0WdTOPjEhWr8J5b6lztZDzWItGo3ihhtuEH+WA+V4H/3oR8Wf5UBD4mjAaCeD1mc5CHmkJ+goIUdD7imO5IPvW4FcS+yIPXhX0o8efhbP3zJqmqw3WsPZv2+HauuYFasWJ4dNowlf1KzYvdz663SsROCFKh5/NkKqzZ32N+euCkScU8xBfNlqzx6Pm4YctmohEEdDzx+dV77N4rfEE5MLdYvdkzYOibVT197tddeGN29dh0fcNFsMKGa8QD5PAUPd53wmNSOGeQ+EBjAZn61aqbl9CGWziPv6SLuO0ODSPdcJc8W6HudrWLVGhC1VYM1YtdCDJ8lnJ5Ip/tBLNVQ74IWCkzcdPMxL5+DHKC9YrNCqtLrVhrcjhk0ciK2sslvVqsVWxbn5YpGTg0Flm6FEu4qJxsSbEli7EwZ6v/GaLc9cMiuq16bbUzX7OjZu4J6hYez7gRe7vuHDl57dgkLJg3w5v6rfK31/W+aXuPoFX8fFH7wNoy8twPv6V9aP28Uat1gL+m0bbNwIvu9y2zSzljN2Eufc3oQI53bRKQVkCVLeEMi+jNsAcDw9E6t/77h1gx2hXCM8j8U/D+6bW6Xl2PkEUO4l/eHlfV6dtEDU6XHulNWaYTB5m6STk8NBKUamWOHh0WMxF/ZgpteHQs2yhRD0BTGdmhYqckKumEO6mK4PGxuYuQ0vvORenHTuIwhvXrrPunv32gXtlST4IZixN1MJKQBpR8RDxL/c//jwYV0golzu11Rg/d7va5O9TcDpjmmZQ9Hn307+4YSl3Urg5P1qnUxxh+y9Wuv6Xv6+390h/ubN4m1ZeFsJci4MFfntzFdb4Wx4x3874PHhSRv0Dtj0uDwidtuyZRzFmi9/thSEZ66qeCe4XVUKOpFPCK/zicRsnRgv+vpQ2LQRhcO2InjuDvgTf4Q/Nwsv2ucLVaNLnFsAV6A4NXAiUtu42yURjf7m9pOIXC2bK1kbDkqKZ2pXtk1x3sZU6RUVRw5u4OtqRMFK3qitEud2Bq8RC4rzRXbPO6E6oKEuZItkxeO87nXnENkk7VoqJqZ789Y3SZrYAVKXlQ/bhsc3RnDn9hBSriTyRZpqX/UqXwm0robKi3BXKoiEcwi4e7DjsMNF+/ihplr7UwElrlKB5CT5yUlMs4pzp1qQuVrejIUF3weXs3+wExv7V/c5v8tg09IZanPC4aN99Wv48P450QXX6YpzbtdC4gNOgnALRKeLE6TMDNe8jK3aHPD7nmoBdg5pDfjMzxTiuYXdzyol3iPRAJ7yPh+lzAAQ70dgain2JKU5qdXoD4GGjRF5LrvBPH0LWHfSBHac9TQyx/R2lGqtCwX+5lFnC4HSBoUU0BRXtALKW+RL7cgBaQ37m7OOEV2qhJ8+ukcoT81Azuii9cuJfVsKBdtVnXecVQv73Hle2gx83+kEkQE/h5X4Dz4YtBMU57wrcrV4mwRT0gu93W7KTh4QSh2J8vk9fv0gdMLr9orr6Pf7UApV75NsKQTPvHHtIdX5bHoWE4kJMWxWkulPlX+P+//yfuz6810YOfMZjD/6tzjmj1cjMHEznEaXOLcAvhiT76ezFe/2CFy+cDjh9+z3WFWcs8Tb78VRY/2G7+skRnuCrU+V7tQWdV69NDNsQlY8STHBgxk7gkA5Vbxd4nzB0K7nDIEgK9dmPM6JXMjVnhWnAihecJlo877hylR+/0nkcjlceumlwqeTvlYFSpInTz4LP1j/Svzccw4y873IFauTu6ktbCWQWjPiisNVIk88IJsNIzAUEO3fXdXa2gQRdTXBmmPrQLPhoGYgVbCUFDcbuKsLXCnHVe9myDgnC8gSG/rDKxLnROb+dneVOKfiJ6n4OgXUMSHVQ2QB9uTUouH78+nO8epcLkbjdi0kSKl0kDpQ3p9KFOes8G2nkp7PFLKiOF9ukK+uvVvacRSCA9i7ZwOee/YwLM4fXf8e7cFllIU9CyFTrMal8tpGXTm4XD7A7ce2F76u2mHmOvSsWv6UMOOQaGel/LvSRkzPVdt2KM4JWwajeP3zttfP9Yt3PmZqUKg8d8o/nOjU47lmO77ykquhOCliY8GyJauWVQqynSK2k+AFk+UEm7TO7qoR5/R6WsOdBnmct0qckzc7dR92kthAdnub5WxI7Chnkx0hxBY+rXu3P/4Ejnzq77Du/r/GMWc8Jf4tXw6icKBqd8h9zmmgN3WOeV1Lv+NMYR8e8D+A2/pug28sXvWlhwvuINm2OYsucW4BhknNDrXQSAKNHvJ2yDjHiXOv1eGgxVWI84DjFdfVN0HniHM+6LHdVnvyyJSBCA1vshOUEEVMtEc2VvYHbFQ8N/MaNWPV0gkqTU54tzukRG729Ow3I3BoY5yfnxdea6tZqLQDSpJLCyVseWADtj61Hv1TYcSyaaE2X01xPpfOogcJFHu88IwFsfXyUxBcHxRV8a5qbW37mxP6Qs6twZQESjWrVLeY7XojgtFOIs6qhYXBsqIDEtkN/ZEVifPHDszXf0/yFLdTLdwK6JwkHtw7t7xVS4ckgY12aYvpfMeqA+W9TvFGo5rf7B5ud+GbE07tdnhKcoQKc1K0YNfeLZV2xX4/bv7JG/HbO/8c08+8yvD9sDeMmfSM2MuJQOcF7TDmxd/5gh9DmzeJQrmH/tdVnB8SinOnCblQbf9uJx/hnep2EqEXHrcZR4/31wc3fvPeJ9s+hp0WM6taTrWRQ8nCrNPWX834Cb7fdeJA1mafgdwGluM/iF+Sv9dhHTAYlMBzTrIaXQk8Hu8UsQHnWnjHf7tqc8JJLF7UtXe7qasu8yy8+Tn0bfAjG4oi2TOIxcBwU2U6/V0sLd0niUJ17yYMukui4CeGgwedF610VvS/xtAJAb5hMUhlDQngSjD6Xjph1WK+dfTg9lEfNjLPVd2fB/d5lFOj15q3KwU+RNqk86W2q5f8vudVXLtApAWRGOk27U64qs15xXn7ag8+x8AxxTnbvCfa8MYnwkFu9nSMZoEUDSb53Oc+h1gspnRICSE8EEYoWMK559yN12yZhmf2X1De9K5VifOZRBpRVxy9iVmEUcH4zh8g86IrqwNHux7naxLcU9lpRSvZtaTzKaE4F4qKNhIMag1fIs7tXQ/4/tpsuGNbvskdkAyu74uIoJyCcxoQ2gg5FJTwwsPG0Wk4UQx6oiSIfCxn8eYXHN7cqqVDPM4bFecL7B4yxNUdoLDjZAUVfMwKTZzsGDPE220qzmXcsdI569y7x6JhahtDqceDcq6A3M4lAoAQ8UfEgLHF7CJShVR9MCg9DOGhFIpZHxbig+hfHxFdZlRI7yrO1y6mkx2kOGfex60qoI0Dve1bB0gh/o4zj8EH/+teIbK7c+ckLjpuy0G580rrlxTnOeFv3jgnqtXrzT3lO6EQ2+hQ0GzGRyfMxVgOFKPSeVAOvpzifOfM0mDQ7cNLFlmdY9WSbX0waIdYtYzTPmhStEZ4aBl/c117tyc4iiJcojs7tCGMhYEq4T0dGAWN/OQY7al+L1tcirNf+807ccn8DGZ7veg9gT6PKnXeCcR5V3G+xj3O+UPdjmqNJyrOWLVYU5xzEpGCFyL/uQeUVqsWpvZtV3EuN0H6/XkyYzeqgx7D9UEZ7QxacboCXrcnEpYLrVdIF5iqzakASiq/qEOkXfVawsEhXU0V521UvWcS2bqvI7d74XC73TjssMOwdetW8bVKhPvDqJSCOOrwvdjam0Q0t0v8O/mcr3jeyYxQnPvyaWAhhcx//RAlobzrJt+HQqeYk1YtfO+l9aDdIiwp3KTljN1WcX0WFeedRpzTXjxcUy/uW0wZlDcUn9z33ExddcuHcXYK6BpS+y1hIpY2zJ+QIglSC3fCcLGmPvmMPDDE1R1AdBhsiSz4nDspmuBKzXa63eg5qA8kX6HLQufeXVXauXD49JOI7t2LyhN7UUws5S/U/UUi+of27UM6n6kPBnUXYogU4/B5C4j39yEY9Ij9vtsttrbBB+PRfCsnETIxc8kpxbm8Xhcdv6XpAMf2ztuZfYTb0bVjjSO3804oxB40HHQV4pzHSk4p/Rsh96/l+I9ds0vE+WHDzvubH0ScM+HoSoNBG3/OSRDvJLmPdsWOxO88fmChfu9tGezRvnd7AiNVMQetF4M5ZDaEkDi6F+m6Ed/BkMICQvCxEAZ3jmHHU1H4y0nxDNPxPEHn5wt1iXMLiGWWFo0+hxJwmew1Dt9razioAwuDQQGjwKqFcO5RG8TfJ24Yqvs3dZpVi0yeOoEsMDvokQ8rccJzTQYdlRaCDg6pjHQ5SJiZHW7j9JAuCdq4ZfBGBE2rOBBPNVWt24Wop4SRqRRK82H05Nzor1SHia2mOJ9NZRB1JeAplVFKA+neEMo1j9Su4nxtgq8ZTgwJ5uDdXqupYFZKZO1WzvN13wyZyBU9Tn8GErJbj9Zlfn4P7purr9Uv2Doq5mx0Irhdy++ZukjeV0RUO+FL25LinF1vrsDrDOLcWpFIQg48dYJ44tZCXMG4Gohkl8U557rcqt0g5ahbnEsFZWQeqxa+CYlsEd+8J4nP/3IPfvFkor4v+5IH0LMwg8GFSYw/+oj4N2HV0i16r2mQ0EfGwE5bZnHFeasdsHwNcUJ4tIUpzA+0lfc5bxnCC4A0z6MV8I44p2bRNSLi99aHta4WP3XCdW+E5FcoLmomAOPE+faRRn0xHCu6yPx7Na6MrIw6zaqlKnYM1c+/HbEjkeb52uur3Yn640CXr0fMFiHGu29DCXv/chsOvHYjCmPLFzuzheo5ukoFHJg6Fo8sno2HZ86HJ0+kfwUVTxguX2uuGjrRmRnAGvNKpfXPKSLLQJyv4tvEwStuAw5sJn6v1eGgS0FKuHbt/+y4Lfjyn5+JD7z8ROhExCRxTuodSTp3gvLL7KBHpzdyrlRbbcgHh/RRpWu/nFenrV6jbRaMOmWwLNkaSAKyVZXNgVhmVeK8WCzi9ttvx69+9SvxtUr4+3vRW84gvdADd8WDsKsETzG96nDQWHIR/koWlZIf6WQEseCIaPem5FxO/+5ibaEThnqrGBBq+D1s3k9o/ZSdV+0ULxtjFRq06fRnILFxGZ9zsj6ROH2r822irfmcV8/52blEvZOhUxJACV685mQ5JxL6O444N684d3L/5rFeMw//Vs5Zxtl2792j0TCiQS98WyZx/t/cjFd98HvIP/GP9UT7q3dNYDFdFvvyMzNL5+ubfkaQ7JVKGRMFGhLq6u7daxw0pFmSXiMdoAQNW1ScO6EglvF7u+IXQ6exQ/lrwET+1ElCCQlai+S+slrcZ7zunXH+fC4NdX5zUBf4MzXinGaJdUrcQddcWssSb7CSnzd3b3DCjnY5SFcFOvV2+L6H9i8JKU7euGTTonXvdrlQ8fULgWOoEsdc7nFMpH6Dh3f+YlXFeTi2H7lSNWbx9BThKSyK41T8g+gEdD3OFbR80wLolJJnmAUP7TxIcrEmwp+T2HYhwIaDtjusiAcpdBwvay/hA6c6TXFOFXKp3umEyjEnMCfNBlAO/B7cp5UP/FwJtJlL5YET1kQSIS/z6KN7OGJ2OKhz989Ybwh/nFoUX08l0tg21NuW4nx8GasW2rRvuOEGFAoFXHDBBfCya2UV3nAPwuECUosRoAyUyxUEi/PIFVe+f7KpGXgKBWRLPdg3txH7Ykegt1xEpAOq3l2YQyclUnxYYztFwEblmhMtyER8EinLrW9aASUskvygbrdOUUE3Dgg9YcOQONdH9s/XSf5jxgfQqdg0EBFJHqmlnphcwJ07D+Cr9/yx/v3DOsRrtHm7enPFeScQBUarlsKatGrhHv57F6vdVq0gxUiRlWI9nXu3z+3FxScNYu9/BtE7XLWWqCxOo1iu4F/vPYADsZoVkcuDZNZdnxVR2vcMShU33Chjxt2DTTVrNp/P+di7C3MgQkvyXU7btDQS5yKeb1N45MSMlZEo7bmUE5GgxWSnsUMe53JGVDsdu502M0OiP+wXMR/tC5SfLhcH8X3DKYHmiuLBfMEgfiAbEdkN0GkxB3EH+xfTQqxJ4svl9uHn5qt7JH0kww7PUTho3kcNk4n0sranyw0Gpd/nuHWDtu3dCAyhkp1EoJzAX3/jYzhmKg9XJAqc8+oVFeeDcxMoV6rnERguwl1Kg8aDuhoGizqFzngK1yAoOJMLspPtpNx/qVWrFjp3SZw7VQ30ezzWPM7z0nfR52i1tR1fWoOvawcozrlftWnFeXBtKM7p2suihZPPa8jffqthsyTWSauf9ey+ocC7NeJ8dcU5+audeuqpyGQyyn1SqT07NOBGaiEi2INKoYxgfg65Um7FdbKUnYW7UL3fs6kgPANeoVqTPqpdrD0YrCAcHw4aaNl3cUXlmgNrmlz7KQmhJJZ306wESlhk0tsJqkGJDUyJJweEEoEu95ijxgYcERm0CiILT940jP/+435BKn7514/Xv7djpBevOXErOgl0z8qBpnz+SCcVthpjHE4erSXinKwRiTCbTmTFPd3qIGKD4nyFWFvn3k3nefR4FInDt6FSccHlqiBciuHLv9qLfQsNXudlOueSUKgnnt4Nv6tffG9+tjosl/Zuv8f5e6oLc6CZM826rZ0CJ3JbVZw7nT+R0IyKDpPxjCA5VyJtO01xztfNVvkOXtjvBOuvxtiT9j+6J5Y7N7nnhP1kD9kZXa58zhvPSwnPdOBg0GZCFSrCNduH6Tneu5Cs2xq1GtfaJVqToOe3FZANr3ztkaP9B4nudO7dCAyKfNvndiEYdMFTBnxkHZjP01TSZRXn0QM0U6h6LpFNNZ6E+JsOIc474ylcg6DkrxOIOKp4y6p3q4pz2gApsWpU7zrncW7eqsUJjzsaCCPjDEngd3ritOqgxzamNCdyzg6Jofavdu0NpE2L+HkHBwLy5LOde+fggXrO1Ty5YrzVzVt2NND9sty9TxO9/+///b/44Ac/qHS6t1SjBUb9VcU5JeO5AoKFeeRLyxMhVBTbUxjHv999Ju763ml49g+b4RsNCtXaWky+77zzTlx00UVYv369ICN+9KMf1b9HagO67scffzwikYh4zVvf+lZMTEwYjjE/P483velN6O3tRX9/Py6//HIkk0YF48MPP4wXv/jFCAaD2LRpE66//np0EmQiRWu40wVMK1YtTrd887inHQsLHqcMdQD50VxxXl2vHp6oqs3l7JROB7drkThz+ziuecUpjqgbVwKRNfKcmg0HJTugiMMexlbu804biLuxv6ceb7dKOnFLxJUU5zr3boLP48O20w9HNlFdL/qCmTpp7vW4sGWID36rXufJZ8J46r7t2P/UOsxGqs8uWbOtxb27iypm2N7RcYrzFoUwcR5/OJQHSrsW8j3mthSdTPgT+FDD3cxHu2XFeScR5y0OCJW8gVMq/2bggkVSnHNIm5ZOJM4NA0KXibd3zsTq4yuPGKsWXjsFRs6mNbGjIYZtsGnRvnf7B+ux3ELyDNw79zr8dvaVcM8tnVMzxXkPK770bKY1tka4BTojBu8S5wpUa04nJNKuhRYCqh635W/OSEg74WdWLe0qzmkogvRFd6J1iRK+SC1gaseqxdjq5uuIoE9u3hM1hV37Huf+NaE45wNiqEXOKURYoM2T0rVUeOGK8VaGC2UKxfrnxDd+O0GK88D6CBLzlGyVUcjMw5+fFcNBSYX2/7d3HuBtFOkb/2TJvduJ7RSn95BGCBAIhBogoXcIvR29HvWAHO2oR2/H/+j1jnrUECD0lpAAqaSRXp2497b/5x175JEsybJjSyv7/fGIWKuVNJrdndl555v3Ex9gQFFpxUj+mkRZv7S3FGxOl7ieDYluIjExaFlZmYwZM0aeeOKJZq+Vl5fL/Pnz5ZZbblH/vvPOO7Js2TI58sgjPfaDaL548WL57LPP5MMPP1Ri/AUXXOB+vbi4WKZMmSJ9+/aVefPmyf333y9///vf5ZlnnhG7UKxXioXRYs13ctC2W7WE4x7EjNZvjc+5Kdp1t5Fwjv5QH48NjXYWCxptWsDoXvbwVwzEyB7pbhs8nNmnjB8oF+8zwrYJTXV+nYKKKilr7N/cKznjEJEefhsf89pqTWLNQMlBwzHR5W1F1Hp7uPD1eRC7Y0f0ltiKYkl0FUpycoHECHzLRU6dkCO79ExqJpyv/z1T5n0yVr55fS8p6dFTbUOkfST23aSBbSUVtuo7PDzOg04O2phrKTY6bPcfPY3gl2DtWsJt0QnQP+sx6587SgL6VNveqsUjx4fvez/oOdpD3A7Bdhozaln3276Ec/tZtZgR577rfEVekfvvwd3tVf4cY7Iw2KC1FduKPO4PQ4mjMUIczZwrLkYq65Okoi5ZqjZt8Jm/oqau4XruVpwgTgfyiDlFcsbIuj3+I4tH3Cc1/U4TO8A7iDZiiqDhXk6KJWvrCkrViQdBvyX7FQxSNOHye96ZiHNzSVximCKS0IkhIrWtVi12EM71DRRuLCDCoF6DieAPt4BrrpII1t7AjOY0b1giIZmQrxuUcN5EIUGJ9ksN5qbbXM1g2ryEEgyW44emStzXhRIdVSlRjlpxVGxW0eMQz31FoUE4V7YsRaj3hhuW+D6ZbiE+0jjssMPUwxepqalKDDd5/PHHZffdd5d169ZJnz59ZOnSpTJz5kyZO3euWtoHHnvsMZk6dao88MADKkr91VdflerqannuuedU9MLIkSPlt99+kwcffNBDYA8XZq4DO0Qfwa5C+43uXHLQ0P+WlHaIODdztNilP8QEBvp1XP/wCge4pzKTh9oV3FddtM9I5W9+8LBeMtZHBLqdGJqdJmvyS9Vy9a9XbpZDR+Q25Q6ywfXZPDnoTgjnYe6/c43zd31BWVDnhue9dvjuOdA/1yXFiaOmSpyOOompd0huXJHstssoGdkzSRZtalr1lF/eUM81O/Bvw7FzDGkQziO17yY+Is5t4D2c0BarlsZ2IJwBd6Y3MhKEIp9HqwK/wlR2TKQO7JYs89fvUPWNiZTsFsYUnlYt9kn0aPZv/gIP0GdYYY7yb41dLYIa1+Y35KHISYm3ldjvnegz389Ki+WG0Dwky14R5xizIDACmhnyiwWDnshA1He/jGQJJVa3vWVTRakkJOXKjsqvVG9cZ7mkaPUa6Ta+YQzpSwdMLI6W+sZ+usfo4WI5nFITmy7OuO5iB6Lac7m3BgNsRKphMI5l3xMmTFCDb01lZaVccsklkpmZKUlJSXLcccfJ1q1bPT4D+0+bNk0SEhIkKytLrr322nbP1t5pIs6NmffthgecP0yxEcm5wi2cV7cyOah5gxLId7Ej0R0CyhJMlD8osYlHtUlOK6OHzRs/JEszk7yGCtiU4Lu9J4ECYRff1LYmljX3h9AWF0avXfj8aq84nDMtRX2Y55Xp0eZNVVWVElevvPJK9Xd7W7Uk79ZDRsT+IjFWmbikVqzyTUo4x8MXm4qLlHAeW9r0+9IG5kZsxHlrKSoqUn08LFnAjz/+qP7Wojk46KCDlC/ezz//7N5n33339Vjyd8ghh6jo9YKCBhEynMCP0Q4WaxpEnOnJ69ZHnIdZODcjztsqnNsgatCkd3qTuPjFso1qKTsY1SvDFtHPwbBHvyy59qAxthfNwUHDerv//uyPDSoqW3cndokORLS+9jndGasWJPMK5+S9R8R5kCsMPYMkXGHpu83+ttRq+A2uekvOHVslu/ZpiAjMSGi6r8ova7jPrilpuF4dUTWSktvL416ARL7HuR1svlobCFNVW+deLR1OIbS1q0abrzQOX9nNnEpmdHNLYz+Mm+ySXLNZcmw//Ypd6jxgxLlh1YKVejpq2G42Lc2sWnwE3UHL0RHaGB/YYVWLCe5BtWaDSSMEywYCbZIOboPNka+Vhx3ZdzsyxkpB9mFSnDFRSmObRPuiVZv92rSo9xU1tKWWwyGxg7orizUn/rNJ3x3Vnsu9wapVq2TSpEkybNgw+eqrr5TfKZZ/w+9Uc9VVV8kHH3wgb775pnz99dfKR/XYY491v15XV6dEc0Su/fDDD/Liiy/KCy+8ILfeeqvYBTv5ZplRW8H4nJuRbRlhsq1wRSHJT9PNRGswG+pweJx7C6DeyTGCm7G3Rydo3kBpH+qW0J05bFrCISYoMa9x4BlslGaBKZyHaZUFSNyJiPPSxv1x7oVbxNHnDX6DObj2hXljbi4R9QYC/ObNm9UkajBLMFsDoszqemSr5V/5vzhl0e8x8kvSvqpDRsS573IXy+CoZZI8pEyiBxVJaYJDkkb0V0nI7NKBdxSY3Ibn3SmnnKL8zMGWLVvUJLYJMrBnZGSo1/Q+2dnZHvvo53ofb3CzBosX89FRFBkTbXYR5vTkNdpVROwEixbxYHsWjgRGHgO/Vli15JWZCd7CHzVo0qvRBxp8vmyj++/RPe1v0xKJIIpfLx/G0uPv/mxqI1Lj7XGPZIoWrZkgMsEAd/X2hkg8DMbDcf8H4VzfNWwoCE44LzfutQNFnHdk321GiZdFN0YAOkRqtzYlv01PbLqv2lFWI1ZNpThUOSyxYsulZ0pTn9QVJr07K3kllW6rIzMxpz08zlu+nzcn3sIphGqP87ZYtWDsHh/GhImm/cfqHQ1taiDc1l9qdZ99Jr+DuX/ymLi0iWbgPYlqRpyvauzj7Cqce1q1NNfKYGGmcxUMyUoN+zjb34pvAM28pVwlf24vdq9Y8Hc8OrLvdkW5VGAVxtnSvWnsX7pmR7N9zYjzzVkDpbZbptRnpElc93gV3IZ7ALusFnO153Jv8Le//U0t3TYTgg0cONAjiu3ZZ5+V1157TQ444AC17fnnn5fhw4fLTz/9JHvuuafMmjVLlixZIp9//rkadI8dO1buuOMONZCHX2pHJJ9pLXbyzTKXn/jzbfIvnIdnRg0NEqLOK2uaZuCDxRSqE20gnCMSOJiBkIdVi01EG3PJXjCRB2hY9e8Ip/iPiGcs28S5AI98REEHwvSQ076q4b7Rbq3Hua53O6xWwHmzaHNDBPGW4vKAEa/mjXkgj3O06+g30Ee0dxsPsVtycsTpdMmOn51SmOWUZad2kwOsep/CubqZKC6Ww2JmybApyyXaEnn3vWvEkZUuzqrSTj34RqLQE088UdXBU0891eHfd/fdd8ttt93WbDsi1DGJ3p6s31YstXUNxztG6oKKgu9IIR8kRFnuMq3ZvE26Gf15ILYVlUltXZ1kxMX5/R0dWvbqCne5t+QXBb2iYFN+wzHAkCSqqlwKaivDVvfeJDtq3b+poDFyFeXMTXC2esVEqMvenoSy7Hv1TpXfN+Spv9+ev9Jd/9FWcNdnKMoe13iNFpXXyvYd+WrZc2tYX1gu5dUNY4beyTEBf1dH1n1qrFO2l1fJ2h1FKtFzS8LA9qJS9/GorSyTggLf9yz19fVq7IfAKjwqKoJPNh8MZVVlUltWK7+P2kMS6reK1EdJ8fbNkmR4XsdGiVQg8WlxpdQt/UGOvvZDqa1yym+/9JTqqCQpLSqVuqo6KS0ulfqK+nat90i+1iMFTCrrsatdViohghNCcm29FVQgTLjzkjR9d7Qai6DMwea3cgdMxYU3cGdAZnLQEeeIINaTneHWabwxj7/pYGB3e1fvSVTTQhSJNTUDbeYPDnDOI8gEutMOH0F3pk3L4O4N+azsRrZXglAtpPsTzlvym+/IcbfT4VRjZYyxnT0RINHQ1lRvLvMbcW5Z9VLUO1lykl3idDklNe8DqavaIQnOFHHljBE70K6jf9w8ffTRR3Ldddep5dm//vqr9O/fX2688UY5+uij1T5IGIaBOZZ4axCdDg9VLPOGcI5/R40a5RG5hs+76KKLVGKycePGSbgxZ47DHnFuWrUEkSHbXBIeLo9zEOOMUg3YzkSch8t30XPGtSYiPC5bSjYRXKJHWFtYYe/IzfMWN9Mt+dwVltsj4txzpULwVi219fXqWvFeJmcLi5+i8oBecBDWfb3PG8xMYwIVogL+bm+i4xPlyyMPlE8KPpctcYkypni72u5LOK+orVDLglOkFKZsUlMbL7EZybab+e4o0Xzt2rUye/Zsd7Q5yMnJkW3btnnsD/s0CDB4Te/jbbumn+t9vMH9wdVXX+0hQuTm5kp6errH97cH9QVV4nI2tN09MtLUdwRDsPu1hR4ZqeLa0nCDWxcdJ+npLfsqIrqtxnKo35KdlhSwfB1V9lxnrLsuaxzOoL+nuLpevQ/RP927ZYa17r0ZHp8oLudKj20DuiVLbk7bvBVDWfb2JlRln5yaJu8s2ayCPsprrTZdnx1d9m4pSbK+uOG+OTohqdX3/PO3lbt/18je2S2Wr6PqfkBWmhSu3yG1lkh9TEKLAmQdBr6N5e6V1S1glC+sN9F3d0TZnVVOia6OlqqKIRJb1XAP4arPk3jj/rVbSqxsLKyS0hpLoqpLpDbaJfUuS5b0zpC9MlMlJkEkLjpOMtIzJD7at9jQ1rI7nZ3zfsBOIEJUx0NmBRCLQk18jEuJysEI53ZZdQzhG6s/V+YVq3YXY3DTPtUbBFGUVOmkpuHVOzB+w70DdAxEnEMc9xdJrjzCLfv5mwcdcW5atdhI+PcYxxp6jBZqcTj6hthPO9jzHucOfP3hcY7z2pwEMhNpDs22p3AO73jNVmPi2BfmxNIgP8J5R467nY4oia0tlvqq7dJriGA0rajbUes34ry6vlhemPykJMc5ZZ9e+8jJeevFWfaniMMlzjE3iB1o11rCoLq0tFTuueceOfTQQ1Xk+DHHHKNsWGDJopdrY1ZD+6ZqIJJHynJv7xnC8Avn8W2yakHUTDg7b91Rt1Y4L28UEEG4PMtMwb7U6DgCYUe/MgjODh8CZ1CZ1ePsIZwH4w2sI86xxDDQDaJdI87NVRbJNvDpM1cqtJTdW0/IYGVMOOseCcbWVuwumV8dI2M+OEyK1heoSPQdFTuaLVErrS6VwooaSXaUoPeXqupEic2IbfBai2qYRe+sovmKFSvUai8IISYTJ06UwsJCNfmtgbiOCfM99tjDvQ/yoOCzNEg6OnToUL+iRGxsrBLIzUdoLNaibbd81F/CIm9Mf0bTtzFsya2CTJqIqEEddWcHj1pfUWCIxjMZ3bNlcZ+0HdyHHji0yYPabtdn8wShrbdrMQew4YzEMy0a4EcbKXlVELmGR4+BY2XuTwfL15+fKFt3TPfYJyOx4XxBV169Y5M4xClR4pS6hN0lISZK5SuBMNBZJ707O/D0tWNSaZ0gtDwIqxY7CaEePuct2LVU1ta5vavtYDOKyWyAYKJA49ZiG97vmWNB2OwFClqzbcS5MQbVudtwLDYUNkQS90lPCutYL5j7bQi13mPwZdsK1b9YRWJ66duJ1lgz6/sORNn3DFNy+wELrpT+S26RA3p9rjzL8Z+rNMavcF5ZX+C2cU5KSBJnTYHq1Oui08XRAQF1tog4B0cddZTyMQewWYFP+dNPPy2TJ0+WzrDcG+SZSxgrSqWgKnDH05FCPkQfC0nuLEs2FRS3uLx1a2PZU2JipKiwoaEIR9kd9XWqHOVVVquW5OYVFLnrvq6qosX3dkT5HbXVTcvUtxdIQWLLncSOEiytrxWnwyGVpcVSFcRyt1AsAdXLd9fvKG5x+e7G/Kbz3lUfeCl1R5Y91mpaUr9+2w7JaSGYYHtJubI1SIh3hdWeAddqfV2d1IslBSVlQZ/3m4ubbBGcLdR7SOwlrBp3eVZvy5eCAt8ewPC/Kypv6NwzYhMClhvt9Ny5c9VS7/3226/do7hqSmskPjpOXDVREhdbLbJ9vaSU5svmsmqJr42XlNimG6UNBRukurxCYuKrRLq5JG2XfjL85OFSUVwhMS60m02RCZGy3BuT2itXNkXTrl69Wn777TflUd6jRw85/vjjZf78+fLhhx+qY6EnqfE6JrsRlYAJ8fPPP1/15xDHL730Ujn55JNVsnBw6qmnqn743HPPVdZqixYtkkceeUQeeughsQOewnms7YTzAmNljN1XjUFIw6oxJNA0B6hBJwYN0pImHF7QxVua7otG96K/eUdzwJBe8s5vq5Xlgd2uT2+xqE3CeV5TJF6/MEbi5aY3efhD5GgpgWyweVXQX2ClcElJiUyZMqXd+26I3ZjkzunXS378s2GSNnpLsk/hHFjFTauedkQjUt4plbVVbgGeRB6wZtQEsicIR8Q5QMS5dwRrYKuWaFvZdPYzLFACC/7hF3BhO/HLuu3uKGdzQtBfgGM4rXF8gfMEvtNLtxSq+yKsqPAOgvBMAB/+ete4ohry6kAs13kwVu8odkf329HfXGMGbCAARUfPo651EBiuBV+JNO2AeY748mnXFJZXua2b+2cm+12V0aF9t8Mh9bHdRcrXS3JsiRSm9xDLESU1o0f5tWpJLFgv/SqqpSwzVrrHpYuzqEjUXUisfQJY2lU479atm0oWNmLECI/tGHB/99137uXaSPqJyDUz6hzLuc3l3nPmzLHtcm9QWS9qCSMsO7p7ReaFY+lrVkqiurHAMuhA34OIr8q6hrJnpyWHdal6UkKcuMprBNMaOBeC9U2zXDvcy0dzMtODKlt7lz87o9JdBomODerzqxuX1qMDhBhllyXTfbqlSuHGfKmxRJzxgZchry2rc//urPSUsC057t0dlgsNmZmro6IDfg9sDWob6z4rNbCtQSjKnpoY15AIUKKC/o5t1Q3lB92CvG478rxJSbUkxrVEJSgpDNDm5G0rdJe7b1bgaxUJKZ988kklyB5++OEeCaXbg6roKknKTZbsrEI56+RZUp1aLemlg6Skx0lS5iyT3LRcNTivqq2SooJqSYyqkKiaMum2I09qftwqsclpUn35XyQ5LnD923W59y+//CL777+/+7nuL88880yVO+T99993T3abfPnll2oiA7z66qtKLD/wwANV9N5xxx0njz76qHvf1NRUtdLskksukfHjx6t7AiT1RtZ2O2Am9zOXyoYT5GvQ5DdOMrWEGZlu5jgJJeivEXWOQV+wYiLsj+yaGFTTKzVRDWYBBoeDs+y5ZLczgXuOPftny3ertthSKPCMOA/eYk3ff6xvjO5GJF44Evmak0IaHR0YCG0n15I9HPrse++9V/2LPqbdhXMI3ljple4UV3GRWFU1UvsjEozt6t4n0xDO43tXSE1hnNRV10tBUm+1qkFFnCOxNyPOIxKz7+huo75DryKFaIioyUDXt6cQGt77D1i1BBtxXmKzyOcBRjTwnztKZNLAHkHc79lnIlYzLDvNfa+xbGuh7DUgx39yUBvUu0lijEsJ55ic+GVdnizclG9rf3OfCULLK6VPRlIzf3YkBrUrnsJ5VXCr3AJMZHR03y1x2WKVrZc4Z6XUpjiltjpGysqj/Arnk379RabO2aRWua27fatIpkrxLVZs4En+iBXOEZU2YcIEWbZsmcf25cuXS9++fdXfGExHR0fLF198oQbdAPuvW7dOLfMG+Peuu+5S1i9ZWVnu5d4QwL1FeXO5Nx6hQjfIdmmM4VUI4Ryz3niYlhD+EoOmJ4S3445pvEBxw4FIo2hncMK5nuEMp8e5GQ1VEGS0ne4Ew33D5GvJ3oKNDZ0eEsUEEs7NyINwduSeUZpVwfub20AsS2z0RGyNVYvd/PExEIXPJGboEa3iL9LGvCE3l4b6AkLsLrvsogT0jvA4h71KfKZTNucni9OyxFltiatym6TGpkp+Rb4UVhZKRnyGsmnB8s9UR6lE1VeJJfVSXlIthWk1IvV1Euu0R5vfWiB+B8qaHkxGdUz4IbF3IEaPHi3ffvut2BE7JfX2l68hUiLOtbgJ4RztGXJftJQ00czB0t2GVi3e4uLIHukquop0PFOG9/YQzu1yb+0t4rc24nzNjhLbROJhUkizsQXhHNczctqARD/jiVD23RC96x2WpBRvktLyaKksj/K470hPaCpj+urfJMYqkMr6OIlLaWjnYbPmz9ucRFbEefdk+/QdCYZQDruWQMK5nQToHilNbUFLCUI9vdnDf9/Uv9GqxTsBYiRY83kL55pl24qaC+c2tHc1x6EQbmEj+s8vFvid2IgU4dlMDBooZ1e4QfsCqxzUe6CI82CF847uu624Bg0XY4OqQ+Nlm2RLzwEpfq1a4lbEya/5UyTeVSZZ0Y2agSXiiPcdNB0Rwnmg5d5I8HnttdfKSSedJPvuu6+avZg5c6Z88MEH8tVXX7kj0rCMG9FueA/E8Msuu0yJ5UgMCrBcAAL56aefrrK9Ysn4zTffrKLYQimOB4ogqW48yHZYtuTte4SLKSGmaUmmiTkwN8XHcKD9vUB1bV3QS2PMJCz+Jgg6GjPKLxhfWvw+fc7YQfgM5Fc9PCc9ojzOWxTOG/3Nw50Y1PucraiuDZjYxq9wHqbJIm8ghON8wXkNIc+X17Lpf26eZ/4mXmG5BTuX9s7uraPWcqq3S8XmKJHyaEmOdoiraqtEO6PVa1tKt0haXJoS0PPLLOVvHlXbaAlVLhKd3UOqrHqJdYX/HCJtQy/dxSWXFGdDj/NghXMbeJybE5FWYxvVUr4X06olnOUOdjC7Z7+Gm37S8SB5FQZ4GPBhYGiXe2tvP39TzAgGJODTDOqeGvZBNyasIEIi4jyQtYSZ9K2le9aO7rtRRojnWA2W0L1CYmW7xKeWibVpmTh6DfOwanFIvcTUbJXomkqp31EqO1IwXh2uIs47Y26Szgqsm5D/CtGfWPVjepzbMeJc39NLgDGGOekW7vYNSQYdjX23P49tuwq4CD7T7VigBKGeeof97jfQH6DYmFj9w7CH8zXebmnVT6jBdbk2v3meDNwDmpZgdsOfdmMK53ZfZQi9r6yqVI0X/J37weZV6fC+Oy5btTGIqYkeJ1JSmyZFjZPZvoRz1w6XFFT3kMIakYFJTftZ8b5XlYQDV3su937hhRdUMlD4n+JAXH755Sop2Ntvvy2TJk1yvwd+p3qZN5J6HnLIIWqJvgZLBeCzetFFFylBPTExUX3+7bffLnbA9PO0QwSrjjg3o7r8NVz5NupIzOQRVXX1EmzqgjIbCOdm3QUjdthFcPaFGQnccuSBPW6gPIXzwBFgps9dug2i2PQqCavxRjuYGyLznIc9lB3IgRC+YYc7sawvIWxTcVnQEecdjfJJHZYtUY7FUlaYJLFJheKq2i5i1UlqXKrsKN8h28q2KeG8pCJKkhwl4qyMkoq6ZFm3I1eccb3UAJ4eqZGL7rsx8ApmwioUxLiaIkhMQTwQnv14+No0M/oM0V2tEc7tGnEOf8vrDx6j2ty9+nsmqScdB9rWSyePlA8XrZUJfbJsc316rxI0l/8HQ7CRX6Gid1qDrSOW2fub8PZOSJ4Ypvts7+Te5TXlEj9ohYzbs2FV847FAySqUThPS4APu0iKVSTO+oayl1Q4JC6hYSxUF8Grxbois5dv9IgIdRhjbjv5D8c3JgcNJkGop8d5mFd8u5zuleqbivyvGrXr+BVR5yg7Anc2FZZJbx+ah+cKe/td+9Av+qYnyZr8UllfUKqssczxoA6YQo4GO53z4PTdh6gVefnl1VLZGEyKPHu4Z2pp5WE48dBuGtuX2vp6dz+NayLcQaUtgT4bkxZYFQZ9w7u8uJb178FqubAmU45rihTPrPxNohe9K3VlBbI46WwZuecRzaxaXGUu5Wle53BIt+7o5Bted8Q3TyAfLlztvdwbnHPOOerhD3jXPvHEE+rhD1i7fPzxx2JHzBvnlgaK4RDOdwTItGsvqxbPiPNg0Tf0jjAK58Eul7GbxYkvcgxBc6sR1eGLkip7RExg0gXHHqsPWvIFRpIMO010mcJ3WZDCud2sWkBP47z5bcMOGdmjuW//lqKG8wn3UeGOEoLgHdurm8Q4q6WsIFHSeyDCo16cEM/jslXkeV5ZnhqcF5Rb0tdRIo4ah9TWR8vGHQNlUAYizi1GrUUouG9pslgLfztggkEd+hKsjmkpwZgZKYPdwnkPktpKQdG0ajHvWexGS0kTScfdi5y313DbVa8ZJNBaqxY9gI11RUnv9GDDQzrWiujXxgnvDQVlfoXz0lZEnIcCiN4QvytjmwSy+q1/uv92RTkkNd4lg4qWSU1NnDikTjYWJEi3xIZrGRHn6OOJ/cGEjrd/r1YdutsoMajPiPMgLE/sIoRiFaieRMPEt78VuXYJmDIZmJkic9bkudtYX8K5GWCQFma9wx/DctKUcI7ze3lekYwz7j30+WKXyQoTnL979Iu8wAIz4lxrN+vyS92OAHb2N/f3G7yFc6zO0ToZJuuDzR/YETjis92rKoYVrJQD35uvtq/r84uIl3DuqK8XqWy4H7Fi6iUhqqDxQ0SiEnqLXQh/yx2BmL5Z4Z411pgzSqYXnDdmRJutIs5bI5w33tDHxzjDGpWkGyvUKZbLRJrwaQoY2l8eHUggPG+gwnvu6/rHZFCgyTzTg95OVi3ey6GDPX/ssmRvdK9MNVgFHy1ep7xcTXBNIBIdZKcktBiFgKTRWKV0ww03qL87xOM8NlqiEuqkrDBRdeRWvSWuqm3q9ZTYFCmoaOio80oa6rumvOEcryyLk/jGBERMLhaZaB9uu/knm21ZTZ3lEd3lDz0gxPvC2Qeaor25Eq+liHNMOpuReoTYGfM+vzVWLRgr6HN+QLcUW0TRI+I8mAShOjFoMPesHd13A1ikQfyuTmsSllylTZ74ID0hWoYWLJPq+jipqk+Q5asHSfeETPUaV4tFDujXnjxpkly5/yiZNrKPDOqe4r5/3HuAvcQ6j4jzloRzm+W58kgQGsCuxQyYskvZ0Z6aCUIDBQoi0M2u9xtDDT9tJAg1x09a/LSbZhDJYPyNSWx9H11TVy+zlm5wvz44zHZqwWBOdvuyCjZXuZnXSTj6bmd8z8YwV0uSkmOkrDZN8qt7SvnqpmSy2qoltTBPquoS1O7OxGpxVec1BBKJQ6ISIjjinCCyqsbeEecBIqDN6NxwL13CUrG2COf65iQhzF7PaLzWF5SpxKYYTAVMqmmjpDDeYDDXLzNFVmwrUhHniKjyd3NkpyV7OH8x8IPYhMhtfzcXZsR5ug2uVzOhrbkcOtImXpAc9Ogx/eStX1crEfqZ75fKHYdPcA9wcNNaXVcftE1LfX29ypmB7N74u72B4B0b7ZL6ZIeU5ieoKfCK0jIlnFc1CuuwbKmrd0hBebHMsqZIxcMFElsVL1GWSF2PVHHWlkl0lD3qn7SOPI9oZ3sJ52ZfjInYQINT3OjrJd8ZYfYJb03EOQaC+t4krEtHCWklEF6wQhL9WWuE81Xbi2xl0wLMyMwNhf4DJVpj1dLRfTdw+5P36S111VESE1st6YmbpKi+RqSxT85MjJaVMwdJ4bI4yeq/TRbElsqguMa21eKkdySBPnGPflnqofs9jP3sMub2FQgTyKoFdhD6mgr32MlXgtCNReV+81t5BEzZpOz9MwMnCIXgpoVzO1tvDDUThG5t6i/ySircqyzCHaTWmcAEKrQb2BPllVbIDf/7Wf2tXhORXXr4z/FmF8z75+2lze0dVxrXAyYdwzruTugpteIQSyxJrrXk24Kj1fakZes99qusrZc+K/7AGnAV0V3X3SFVySOlxooSR22xpMY2X9EeLiictwEzssocONplBspcDh3Q7znMnYlpWWF6vwUCnaEWzhPD7LvovVwm0A2dnZLC+AKNK4RzPVtpLhfzJZxjEGmuGAh7Ur2yKv/CuUdm9fDfQJnlNL3LA1FqDGKTbOA3qjlyVD/5afU2NYGBJD0fL14nR4zqq17TNyM6EVFLIDHJHXfcIcXFxR2WHDTKESXVaSIlhWgvLaksKxVX5Vb3PvHR8bKhoKH9rK4vkajqOKmvc0psXKXUOaPEWe+kVUuEsr20yYaqm42Si/lKEAqf7UiwWzOTJrZkYYFyNwb82zYxKCH+wH0b7CNaY9ViJga1i3BuRpkGijhvjVVLR/fdpnCetld/iX5jhyQOqZea2BqpKl4slWlj1WsZiS6p+7NESgsTpPTXfvLbeYtldEzDfSoG7sxPErnAGsJuonkz4TzA/Xyph7+5PcaAsGrRbA6Q38qOgV9YeYtxxZbiClmbX6ImJlxRTSYKFTV17oSD4dY6AoGyZSfHq6A1TLRiggjn+qd/NEVBD822fxR0JIH7bYxPEXSnx6lYPX3a7oN9Wv7YjQxTe/JhVftnKyLOO3zcHZMhFia2rRqJT2tqK+vzKqW2vtbdr8Oqpc8f+VLuSBBXlCWpY3OlKPcEKakqUeP2TBsl9qZVSyfxOIeIqQVZMwGXt+i8ubGRQGePKJpwkmWIF2bG9EAg4gcR3hJGf/O2JAg1RUTzd9sFc3mSFtADRR7YIerZM0Go//qHb7DuGO2QWDN+J6xasNLbfH+4wQ3eBXsPdydueuvXP5U9y5y12+Sln5e79+uR2rK3KxJGjx07VkaNGqX+7ohIg5ioGIkakSi96n6VRFehOBxV4qpqEs7BtpKG9r26Ml8ctQ11HZ1Qpzp5eKS6I99IRGFamHW3mXBrRo631JeYr4dbgDYDB8wJykhNDEqIP/QqEIhILVnz+RTOW4j8ChWwLNArVDcWlvm1ufO0h3OFte8G6HfRhyfkDpTSVQ3f4ayqkYT8n9z7pNXXS31jMjFXTLGs7Z8hCTFO5Y2OwTf7btLeJBgWIBBr/eG5WteGVi3GGNXfuA8WF+ZK8XCjRUEIoMjXEEmJQU20MI7fAdETNllfLt/kDlI7cKh9bCo6A973zQO6JcvdR+4uhwzPlUgrv3dOQ9hRrt5e4r7PbslaqcPH3VFRsnHMw7Jk7L/k5+4zpM7Z0F4m5btk2epf3Pthkit5dYnSEhxRLul/xES1vaa+RuJccWH1afeGwnkb0EKcnfy+zOUbGFhrL1cTiNN6cGuH6BdYPbRWODcjdBPDLZz7SDLhD2TM1uTacEbTFM7NwZ4JBlh2SlbiIZwb16QJZu+1zzY6Gzs0vmbEeGutWhJjXLbwSTUZnJUqh4zIdU9sXf/ez/LQ7IXuaDaUdohNfONiXDFSPTBbkirzxGHVSV11nVQlDfXYJ69ROI/L2yaOxmbUleyQOqtODbzpcR6ZeAi3yfYSbjOMyPFAk4DenobhXoJsBg7oviGY+rdzYlBCfKHv9aEzm6KyP3C/pL1GEWFqJ3si7XMOoc/fRJ15r22HQAn0vS6HS5JSu8nCuiSxakWiqmslYcPXSFai9kn5bgXWnqu/i7PzJDm6T4Nw3th3d1bhvF+/fure1vtxySWXqNcrKyvV35mZmZKUlCTHHXecbN3qGTCwbt06mTZtmiQkJEhWVpZce+21Ulsb3P1pVwa5toKJOPdYdWyD60nfP8Q0+j1vCuBxrstuF8FfMyCzScfAile/iUFtEuDoj2HZTfYgf2wtlNnLN6mErWDfwT1spTN1Bsb0bsh7gdxup+42SG6fNiEiIs1N7cnhx2EC9mvaItUuk/VRCb2lVlySlpMgFXENul9FTaps/fy9pp1KyyW/qsGOxYqPlYxdGyy6MOmdEN2y1WsooXDeBooramzZIOvBKG7sfQ2+lxqJJ4bnNPlqhYvubRDOy80oGBt4nGu8M8B7D6DWNQrnuFGxg+jsy3ZGn8sr84p8RlRhkKXnY+xw42dOXPgb/CF6HrP4YJjhJRdOzOSeLSUT8o5Mt8MA1hcn7jrA3f7oTltbAP3t0HHSJ6Plm5K6ujqZO3euzJ8/X/3dEcS74qW0W7pIrUu2fe2UT2cnSXHOoR77bC2plhipkouyv5bRpy+VrAO2SfWkESriPNZp78gVElzEebgjtQOuXmphEtbsa8ItnKM90jfwRcZ9kS/MG3w7iYiEBINprxBMIlxYCOj+fVD3VFtM2vtKEAqbg5asJVrKJxSKvltPWiNR2Po9xsiGghjZnuySqop8iS35Q4nnu1Q/JmOm/CbZ/bfJqgH5khm7iyTERKnBN97bWYVz1P3mzZvdj88++0xtP+GEE9S/V111lXzwwQfy5ptvytdffy2bNm2SY4891v1+HDOI5kgO98MPP8iLL74oL7zwgtx6661h+00RGXEeUDg3rVrsoRugTdL5hzAGR6CRNxgL6mh5O4z7TMxxxSYvqxkzt1W4c8G0hGnFsnhzgXyyZJ36Gz3G1BF9wliyzsnE/tly39F7yGMnTFLWojovV6QAS6K0xmAbb+3pz8Zo82ADZEM17q6tr5XUtBip7NlwrpfWZkj0lwtU34w2prjWKX8mD5BtPXtL/K65EpMSpcRMJARHYnA7QeF8J6xaVMIgGy1bMsUAJD3wBjOZGjuIiBBqMePnLWoEwoyCSQiz7Yanx3aghKxV7shiO0ab6xsonUQCAvkmH96XdkoM2izi3M/ExeItBe6/R9gk6YdpMWT6iPoDq0fKq+tsLZxj+ff5ew0Tff+BxD3XHTxGbp+2m4zsEVxSDyQnuf322+X+++9Xf3cE6ICrsrpLlMMpG5e5pLhKpLjKc4XFtuJqOcD1uYyIXy8jRq6WgQPLpOf4sarjx5IxEpnoiGfcJNtt6a6n7VQLXuEV9hkQoi51X9BSclBGnJNIJtvI0/FHADs7n4lBbRL5pelj3Ic+9e0SWb6taWzg6167JcEsFH23Fs5V9PiJJ8l/KsZJfVmqlG+uktjNP0h0+RpJyNguAyetlL57/ym/jLMkydnLHXHema1aunfvLjk5Oe7Hhx9+KAMHDpTJkydLUVGRPPvss/Lggw/KAQccIOPHj5fnn39eCeQ//dRgczNr1ixZsmSJvPLKK2rZ/mGHHaZ8b5944gklphP/xAeZHNSuea60XQtipeCz7Q0m/3QclZ3KDeANrvEuuxk8aK7osyOYvNATsws35bvvAXfr293Dh560H9Bi7GK1vDN6X1FFtceE1xpjItxckRHOvjva2XBuJ8e5ZNvoHmI5GnTTbouTZfGWBcqmZWn5a/LVEfNk474Vkn3sAEnd+Lb0nnuaDFv2d4ktXSl2gsJ5G8CJaqfEoL58Q31Zh/yxpdDt9YwImHADywlt17K1pNyv12IkWLUE8qXV0eYgN71lr2db2LX4yFJuLsW3w5I9U2zyV/9LNjcJ5yNtIpx7WrW03FGZPuh2Fc7B6F6Zcs9Reyix/K4jJqgEs62JsoO/2uDBg2XAgAEd5pMaHRUtjtREufy4o+SySXvIrJQJsqO80RRVROVPyKqYLwe6PhMXmiPLIatWjJbYjFjVPsU4w3/ek7ahJ5MR7Ww3uyMMSHXUS76PZD8mZt8e7ohzM3oOUbiB+nAPj3kb5vkgJBBjjYTp89flBdwX0Y+fLFnvfj7IBtaIJnv2z3aL50g8fufM+SoviYlpR9NSPqFQ9N0QvpGjBBPYE3rvLivycqSyNFlW/rGLrF8zUJzr50l9jSUOZ4wsrhkpKX1OavBEj3G6V4vZKeq/o4DQDQH8nHPOUb933rx5ShA56KCD3PsMGzZM+vTpIz/++KN6jn/hcZudne3e55BDDlEJ4xYvXhyW3xEpBJscVPuE2yniHPQyVp+s87H6xCy33SLOscpV3zdtLa7wG2Bgt0AJb3CdDvURzHj4Low2Jy0Hypr6h2kLHMxK71D03bGVmyVnywfSd+MzkjNui1TGxglGCqWlvaTw56+ltLJKttctlBU58+TLsR9I/+P7i6tqm0TVVUhcxUZxueylm3XO6fcOBL5TunNMsdlslXkhbfeKwMWFpWdkIZojqZ8dwAB6Y2FDdmP4r7fUwZVXB798NBRRtrhpwvkQyON8fX5TQ9Y3iIYsnF7VGti17De4p60zq2PiCuMgf9ZEVbV1siKvyB0tZhd7Bk+rlpaXRpk+6HYWznd2RQUyeiMqqqCgoEOye6vvcMZIfLRT+s7pKcPXporL6ZBtedtkUEoPSd3wXymoTZBTot9Q+zrjnLKx7HCJGj5BkvomSaVUumfOSWSBCSqduMuOiSkh5KPvQ1R2i8lBy+w1IFT3QYVlyqKpsrZO9Yu+0H0kVpmZtheERAIDMpMlPSFGRQMu2pyvxgJYdep9z/He72vkg0Vr3XmGkFjTDoEqJrj/nzF1vDw0e4Es2lyg7r8fnr1QTt9jiBzWmK9Ej3OQELCl8UIo+u6GssRKSXWJpMWlyfl/uVpmXf2rMjToFpUiPSZ/6t5vVdwY6RY7UolqMU6HlFbXdRmbtffee08KCwvlrLPOUs+3bNmijklamqcwB5Ecr+l9TNFcv65f80dVVZV6aCC0dzXiXE73OKQiQMS5OX6yU/9n2jkgYGqvATker5dU2Stgyvu+CfdzsMXSwXd6csy8T0qLt/+1PzQrTeauzfMYjw/JCr8zALEnsNc1HQ+w+gLn/9pGvQlBNcHoBaHou6OrtkrO1vclOsolA3ocIGvS0yUmr0YKU8bKlNEHyKxNv0qd1dDO9E8ZoybJnVXb3NezI8FeyXEpnLeSuUZURq4xU2sHzIRb272sWv4wLCvsYNPiK0EoIgJbEgLsFHGuo84xwEAnbXba/iLO+6Qni13pn5nivgFcsc1XxLm9rFowKILdDwayvsSmZVsL3YPXkTn2iDb3jlAxo8n9YUZ+JYXZnijSgfCNZE61adEia0XNepetXy09S/5PnDWFEl1dLzukob4LqvtIdnI/6T65Qip7R8sWq7LTLvXuUv7mNhTO9Y0uhHNMlKE98xdNricJMfi2wwR4qpf3sy/hHH2j9ji3S5JmQloDztnxud3l82UbldC8YNMO2b1vQwIrgMCUu2bO92hrcE9+4aThLUZshwOU6bqDx8oz3y+V71ZtUX3hSz8vl26JsTKhb5Zb6At3LiETWKXBdgVkjcqS2G5xUr2hUIpnLpG0Q/6U+qx4ydseI2t79lP7wN8cx01FnNvMJ7WjgC0LrFZ69vQMfOkI7r77brntttuabYcI01F+ua0hVEJ+jEOkrLZWisoq1G/3xbbCYqmtaxi/1leWS0FBcz/xcJQ902W5y7VkQ54UDO7u8fqmvAL36866Gr+/L1zlT42Jkg11tVJbJ7JuS557UmJrYYm73FJVLgW1wdnBhmsCqEe8o6m8IjK5b3rQdW0SyZNXkVz2UJc/Xurc58uaLdslJ7YhQLa4ouE8z05IbNX505Flr6lJkvh6kTqrXrq5CuXHMSOl/PdE6ZkSI+tWW1Iz9z/yyseb5deesVJxZA+pKKmQqLLNUl9fL7VRCVJdLlJdVdAhZW/Le+13N2dzvlyxyf33voN6iJ0wE25tL/UUEpcZnox2Es49PMqKK1qcYbWbcJ6ZECcbCsqUxQOSv/jyzFrbKJxjRZmd/coQPYXlu5ixXF9YqqInTBGk1GYR5wATLRDOIdhAJDeTfCyxob+5jpJA1DMiYM069Yfpg26nQWwkAquWpJgYqU1Du1OolPPNG2ulpmcfcRYVSk3jRMum+p6y/K0FMnHpjyqxibzzlkSlx1A4j1DMxJRZNrUJGZGTLssb++nZyzbK8eMGNNsHSXT0JKEdbFqA2efB5zy7MdmYd7+NCF3AxKAkUhnfp5sSzsH8dds9hHOIzlo0x33IEbv0laPH9JNYG+VB8gYTbxfvM0Jdk+8tWKO2/ffXP2V8n+7u1Z12WuWGiW9tB+WIcsgAa6WsWlcl3fpsE8e6DRKXlSjlfcZLZWJDUBNsWsz3dnbWrl0rn3/+ubzzzjvubfA8h30LotDNqPOtW7eq1/Q+c+bM8fgsvK5f88eNN94oV199tYcIkZubK+np6ZKSYg97IpSlo0lOiJOq0kqpsRx+v69anOJyNoyncnO6+12ZFeqy4xt6pSer8fem0ipJSvFakb69wl3unIzUVpUpFOXv0y1Nlm1vyMlV5YyR9PSGc7ysTlS5EeSV1S2z1Z8birKbJKemSspPf6pAPOgi+48c0GZLwVCXvT2J5LKHsvy5WTXicjbokZUOl/re1aXb3dfq4B7dWl2Wjip7TfwwqY5yqEnsrJhiKRuaKilLSiVtYpYkZCVI3Ny1IlsGyv47tsqqk/tJfGK0xNQVSl2UQ+oScqR7ZvcWg23aWnans/X3Z+EPV4ogthaXy9JGn/CclHgZYlhb2IGGCLSGk8vbOmRpo4iIc8+05Ag35rL5bT4SmnpTbthW2CGKx2O5jA9vWiRt0Ik24SVnhwjBQOgEoRib/Onlc25GnNtlMKVXKGAoVWh42uns5Bo7RZybdi2BPBE1pTas944Ag7vrrrtOZsyY0WEJqdD5psTFSrrUSnZpnfQqrJOevxZKfr9zVcKS2rp6KZNEeabqFEkpxrHBKhKR6vQUJZoz4jwyyTMSR5krs+zEgUN7qXMNfLFso9TW1/vMr6JtxMOdGNSXXysmj33BxKCkM4BE19qeZf6G7WoiC2woKJX567erv2Hncu9Re8hJ4wfaWjQ3+8QTdx3gvvdDIMj3q7aoqHptNWOHvht497/Zhw9U9349h6+Tqoo6qd+4UWLe/0lq6ss9hHOHOLpE342kn1lZWTJt2jT3NiQDjY6Oli+++MK9bdmyZbJu3TqZOHGieo5/Fy5cKNu2Na2o/uyzz5T4PWLECL/fFxsbq/YxH12RhEYRPKBVS2OOKIzRYe9iJ3QOBlzz5grp5iuN7WXVArKTmybqtc85Jtf0yrz0CLBpAa6oKLlk35EysX+WXL7fLrbLw0PshTmO0Xqfh7tBkLbAoei7XTHpYjnha25JuqNQygYmyaqrh0j21SNkR8/tkrY8XVaUTpQ5hcdKSWF3cVUj71i9uo6tuGzbrVC1t4pnM75eudn99/6De9ruYKI82scZEXY6MgNRresLGsTbfhnJthCcfVq1lLS8lKrMZtG3HgkavHzldZKoxiBWdzImO2N6ca7IK/brdWeX5DamtY/pc44b2FWNwn+vtARJs0l0piax8RpEJGZLSXG7ilULlmUtXbpUli9frv7uKDISkiQlOkoyKp2SUhMlAwoypTYhV7YPvloW1I+XZ6oulrU1+ZJa4lQipZWYKDUxTgrnEUyeMZFsR49zfSM8PrchASHyffxi+F1qTEsqMzl1ODH7ApQ7UGJWwMSgJFJB4MPonhluQUmvEPlw8Tr3PtNG9vVIuBcJYOxw3Nj+7udvzFvVqsn6UPXd3uL3in2TpPfwJTJgt+VSU+9S99rv15WJJQ1lSGwUzjFg7+zCOeodwvmZZ54pLlfTb01NTZVzzz1XRYZ/+eWXKlno2WefrcTyPffcU+0zZcoUJZCffvrp8vvvv8unn34qN998s1xyySVKHCeB0WNqCM/+VpHqSeXk2BjbaQfmuA/5rUyKGwV/u3mza5C/SqPzuKFt1jaddsgDEyy75naTy/cbJQNslkya2A9z5eaORu3JTO7bN0i9KRR9tyMqSupjuyutI0UKRJwiVnSUrNqxQV559gqpLmlYuVcdEyuJozLEWdWw2kkFrsXby98cdO47iXYEkSVfr2gQztHn7WMzmxZNj5QElSgDy6JhK3PAkF7K61kzLMc+Ni3ewnlQEeemVYsNRERTvDDtADTrPBKD2tffXDM4wA2U3TzOve0KTFHpj62F7shMWCDY9UYbN3dVtfXNkoyZlBrnfGeOOEdU1N/+9je13Bd/dxSION/RM0N6y1YVrVa+saHdKcvYU56t6C61liUVNd/K0hUnyHKrTnpGVUnf+lpl89LZB9+dlUiJeJ4yPFd+WdcQuTrrjw2yZ3/PhG3mSjK7WLUgz4SvQbaJmazcXKVFSKQBG5M5jZNa89blqYk4eITrfv2AoR3vLd0RjOmVqRIFIuDAvJdKDCLQJlR9N/pfp8MpdfV14oxySq/sQfJ7VNO4obo2VraNHCX9MzJlU1GVjO6dpPZFsrHO3nfDogVR5Oecc06z1x566CGJioqS4447TiXzPOSQQ+TJJ5/0WK7+4YcfykUXXaQE9cTERCXA33777SH+FZFJ/27JaswBvlm5WaaO7OPxOgQjHXFul7GTiV5tAlbmFcshw8VnQJIdxx8edq+NwrnZfmEFECGdDbQjrijk77Dc4wIdIIvtOUHaAoeq75a4LJHydRLjqJF4qZAKSZCh/3xQJv+yVb6qacirUJMWJ4k5CeKqari/gobjSKRwHrEs3JjvbozH9e5m21nMQ0fkyq8bsMxB5NW5K2Rs727uDt1u/uYAPm9oACDK6mVWwXqc2yFy3kO49RFxbi6dyU23fxQSPNhRr5igWLGtyCPhqZkV3i43UP4izpeYNi028jf3VX9YRRFQOLdhvXcEGLwhAgoJTdriOxYsqXHxsr1npkRHVUudFSPbVtZIv6V5UpmbKrV1llhWvUSvWStR1gixxCnRiSkquVhitP2vX+Ib7T2Mpiwjwb7C+S490lUbvLmowRZufUGp5BqRI2YbZxfh3MOqxU/EuZmsnBHnJJIZ2zvTnUR9XqM9i45unDKsd1DexbaNOh/XX+777HeP7cHcc4Sq71bCeZRTJQh1ilPS49Nl454psuzHwdJvzDr58ZPdJfvBveXwXXKlvLpOrdCrrqvuEqvFEDXub/ViXFycPPHEE+rhj759+8rHH3/cgSXsvCBA7ZPF69Xfn/2xQY3DTasNjFv1ymO7rNY1QVCXFuHMgCkEDC7YlK/+jnFFeYjUdgy+08K5adtpV62GkJ0B7QsCN7eVVCrhHLbAG4uabIFh/RMMoeq7JS5H3TM5HQ5JcxRIhZUgdd2HSlH1Wqm3nMpOrWRghsS5osRVqSPOxZYR57RqCZKvVjYlBd1vsD2jzcHoXpmyz8CGZC64cXz+xz/cvuxgWAvJN8OBTtYGUQAXfzDCOe5J7OAT52HVYogavoXzpIhojBF1pJcWasHJjDiPcUbZxrvTFI9MUcn0Nx9u44jzYHzOy7qIcB4qUuPjpC4mVpxDGtuaunpZedNHsrWkQfQrrlkr+36S474esgbFqqg1lSSURCTaKgTttZlA2I7i1cHDerufI+rcRC/JtJXHeRAR51gFFwkR/4S0BIQvHYCCCa5Pl653exdDMItkxvbKlAHdkn3mY7EDWgDHRLYm/fj9ZMUPWfLeA4fL2qoSGd93ouq3ta0d9oXY3tmFcxI+eqclugN00NctahSbfa3WtaPdCSyo+mUmu8uvg3X+2FLoLjvahhibjPtMMBbVq96Qhw4UlFfbLsCAkI7Sn6D1YcLLsrEtsCOuYfUsxl9pjgZNcsGOIbKwaKpEOZwS5XDJtuHZEhcd5Y44V/sn2u+eisJ5EKDj0H6j6PTGNfqQ2pXTdx/i7pyx7Nv0ejYHuXYhK7nh4re8ltT7orzR4zwxxmULnzjTqsU7IStAxKC2lYmUDtxMHmtGHxTbcKmhr4h/3PSt2VHi7kDsGOGRaAjn/jwRm17vGlYt8FdDgqolS5Z0qE9qSmy8ajt+OmFPcUZVqIi0jT8XyPo5f6jXE35fLDlrM9XfydGVknXFPlJv1UusKzKuX+IJ8h2UNV5DdvU3N5k8qIfEuhpuzb5dudljYs2OEeepQXic60gYRK1ROCeRDrxoNTqJ5r6DekiqDe+vd8brHCQFsbIzVH23adWi2WPAPvLfy5fIuuM/ly8vd0qvFM8INfTdeA8ehHQU5oT3p0s3+M0PZccEm83tWhrGfXPWNiWL3b1fgw+xHdE+5wj2wv1efnnTWJwR56SzYgZu6uTkrUkMGsq+2xHfEIzmjBLJcDa0L1syMsURE6uE8/L4RCnrlyyx0VFSmHuybB56g2zKPU2iEiicRyTf/blFLWEC+wzsEfQSiHABYfOsPYc22243mxbfS60aZoxbiji3Q2JQgGW5CY0JiLytWiA065lvCLh2EPqDYbBxA6UThGIJqBZ4k20k3ppJP7WopPzNG7eNsKFNi3cUV0sR57reHTaxJ+ookNH7pptukjvuuKPDsnuDuOgYJUxuz+om5bvkS1rMZpmQ8ZFkznoOo2zZ8/0KiWps74cdkii1Y4ara5cRa53A39wmUdqBwDU+qXHVGPIfwDPVp8e5TbzCYTMFQdw7sk5TXVsn2xojznulJnosYSckEhmf2+DJqXE0JgXtDMCKsn9j9KkEec8Rqr4bYAK7pr6pnclMyJS99z9N5k/pI8dPPLfZ/og4x3si5f6bRCa79enunsz+dcN2j4TYOjGo3QKPTAZ1MwOmipVNy9zGgEHYuKBdsCumhcy2kgqPAAMK56SzYo5nfmu0aG5txHmo+u6opAFSmjxcSrrtK9UxDdHneQOSpaZ3TynJ6CYFKelSkx6txuZ1cdlSmjJGirKnSHS0/aLn7a0A2wAIhl8tjwybFpM9+2V5RMWA4dnpthfOTWsQb9CRa5HRTgKiXjKPWW7TY1BHm9t16Yw/Bho3UMu3NSypqaipc3v02SliIjHGpaxjwIbCMnl5znJ5Y95KW/ubg0Tj/DV9+31R2rjKIiHW1alFJwxsc3NzpVevXh06yEWisMQYl7pW55+yjwzP/kzSYvJk1E/zpPeXCyWuMF4pISlxhZJx92nqPdiXwnlkYvYp3W3o0emLKcOaoizgmYq+z7QDi4922spLWUedF/mwatlUVO6eyMSSdkIiHeQhwApOzYR+3dW2zgD63ukTBqkl1WhngkmuHqq+G8S54jwizsHJu5wsTx/+tOzea/dm+2PfWKc9JhlJ5wXXy4HDGlY7oLv+YtlGnxZmdlwB6ytBqJkkeFTPDFuNub3JTk7w8Dk3rVoonJPOSqaxghb6h5mzwHbj7szxsm7IDbJ1wMWSn7CL2laaES3JY7pJeUKibN8nS3kww+McYHIcY+5op/0mGu3bEtoEnIzapxodS+8IEUBxAZwzcags3VKgRE8wLMemEeeNHud6ttgflTV1bg+nRBt14pkJcbKhoEwt2UXEnbbDWZdf2qalM+EGERFIZIoMzau3l0hRRbVU1ph2IS5bnee4McLNEmwCPm5M0NPwmn1XWSSawnmLVi0NryfZZJVFRxEbGytPPvmkSlKCvzuSpNgYFb1bmTJYvtknVw6ZvVaqa2ok9/Mtsj4zQTLLK6XbSbFiZaSrgTfEdgrnkR9xHglWLbq/GJ6TpvKTQHhGNMm43plS0LiqyVyiaQdgUYEJitLKGpUo0fSRXx9heT4ICYa9BuTIm/P/VPcZR47q16kqbWSPDHnshL3V6tpgImRD2Xcjehz2K8GiIs4pnJMQcMDgnvLOb6tVH/jl8k1y3NgByj/cXIllpxW73gFsuNZR1lXbi2TOmsiwafGOON9a3BRxjrY50u2zCPFHpo9Vp7Bpbs05H6q+29WYn6SmrkbSE9AGVqgLtO+MMbLgyw2yI6pB3INVi+63E1wJauxtN+xXIpuBCKl/HDlBpgzvLYcMb/IwiwQwuL508i6qUzlyVF/bDbZ9RZwHEs5NSwtE39oFc8m8mSB0XYRGnAO9LM9qXHZYbIi7doo41xmkfXUep08YbFtPcNOqJVDEuVpl0ejPDJ980j4kxcSocxu5iAuOO0IK45yyqH6UlEQlSHpCT+lz4gQZfMslat86q862M9+kZcwl05Fg1aI5zEg0+OGitaqdqG5Mnm23KCodRYdrqsQr6tyMhPHVVhMSiRw1qp+cvedQuXHKOHdC9c4E2hg72kq0ZQKbfTcJlXXk7n27u+1Zflq91UfEuf2uKR2ENKixHUNepdmNK+0hPnuvXrcbWY0e51pD0ONwJA3tzKt0SdfG13jGrlqTq1E4x3g6Nb6pDy+urZfy9BjV0MS4HBJTtU0S876W2OKlkuAIvBo/XFCJCaIz6Z+Zoh6RCDo8u3d6EPTRtyGaPJBwbiZRtIvHufesH6JYdXZyLZyj2+6dZs/GzB9IgPv+wrXqbwjn5gym3SImMHjtmZqgxBt0Gn0zkjy8z+1IYpBWLZgs0jYHdp0EiETMiYuRffaSG45+W2rWxsv4n2ulT7RTxlw0RhLjEt0z384oJyPOI5TtZZEXcQ7G9+kuOSnxsqW4QkWea79RO/mb+xIDYNditr+mcE6rFtJZwKoKBNQQ+wvn6L8JCQVThufKj6sborXfXbBGJcP29Di3V+CRyaDuqfJro1eyDlSDVZNd7WV8RZxvKi6X4sYk5XYLMCCkPfEVDGtnd4M4V5xUVFZIurpcoWw4pLC8RiprGgKC4G8eV7RAMv58SlLqaqV62NUiWWPFbjDinNhiAKIFjW1GdKA3yJatSbSTVYvReO1oXEqPSGHYt+jZcCRQiyQGd091Rzgv2JjvkWzFblFIuDGdPmGwHDGqr4zpnWl70dx74ieQVYs5WWS3CYv2BolJbrnlFrnrrrs6PMEYbG8cakrLkh5JA2RVeob0XJMjVk21DDyzryTmNkXGwqrF6XCqB4k88koahHOHkY8iEkCk1OG7NCUc/M/8Ve6/dRIyu2AOULcUeyb43tgonCOBKNpqQkjnIpR9N4RzBDSZ+YSCeQ8hoWBoVqo76nNzUbnc/sl8+eHPLbaPOPf2Odfs3tfeNi16bIR8DGBVXpE72IjCOenMIO+At7bU2ojzUPbdWetelKELLpdp68+VBGkYFxRV1EpVbYNwDn9zZ1WTRZQjoSFnhN2gcE5s5XNeXl3nIRaamJG5dkpUYooYiDjXPmt6WX1fmy6daWkyY2yvTLe3/Lx1eV1GwA0FiUFGnHussujk9V5fXy+//fabLFq0SP3dkaTExTXI5pYly7dWSnrMKPlq2i/yyXlfSsmUhoS4GkScY6a8o5OekY61asEgCn6jkcS+g3q4B9rINaGx2wTA4KymhNJLtjRdP1W1de5VZIg257JpQjofoey71ZJvh0slD2sJPelN4ZyECtwnXrTPCI9J4vpGJRe3kHZeOerLcmq3RusZu9e5jjrXIhygcE66ml1LaxKDhrzvtmrFVVOs9KU0R8M4oaC8STiHv7nLEM6dCU12lXYiskaRpNMSjM95mU2tWkwRQ3ur/bSmwdsukhOiwa5Fo5fv2X2pYaRgiuCmd38g4dzON9ztQXR0tFxzzTVyySWXqL87EpzDKmpNLPl5TZF0jx0tltOSqLRaefbXZz32hScbhHMSeVTX1rmXSZsZ6CMFCP2HGl7ndo04H5qVpkQBgITkmk1FZe7oL9q0ENI5CWXfDREc1isQxVsCfTdt1kiogV3nQ8dNlEv3HamsIzU5KUh257D1uKRHaoLHhLjd7jWC8TnXUDgnnR3T8QBNS2vzCIWy75b4HPWPKZznlVQrm2aA6HkI53iO3xIV31PsiH3CdkmXxls4H9AtRWX3fmPeSnUxTR3ZxyMyN9FGEefeHuco9weN/uC4+Pce0NBYRBpjemW6vefNVbGMOG8fQSzGGaVWJZRW+4+cKmtMDAqSbHTOdwROp1P2228/ld0bf3ckSbEQzqOkvr5OdpTWSFr0QIl3ZkhcTJmMzh7tsS8G6LGuyBg8EP/+5lkRKJyDg4f1lv8tWOMRSWW3wSxWgPXLSJbVO0pkfUGZSoYGX1RtVyYRmOeDEGK/vtstnFstC+fMT0LChSsqSvYemCN7DciWhZvy1SMSxoJIEAqLGaATnUYC2clNgr9d75MI6Uj9qVdqYqtX1Yay746K6yEYxUQ5RLKii2RJlcjWkqaVtLBqcVUi4tyS2ug0iYux55ihcysxJGIwk7bllVYqC4Unvlksv29siHT+/I+Nkm3MKCc0+m/bgfhol/JXq6ipk/yyKnlvwWr1N9h/cE+PGfxIAhHOQ7JSZdnWIo/tdvM4j1QgNlVXVEu5IY57U9KFIs5DXfdRjiipk4br1OFwyvGDr5VRfQtkQq8JzfbnUu/I9jcHkeqvjet+/yG9ZOaS9bZNDqqTiEE4B39sLVTeqGZi0NZGwhBCiDfot2OiYqS81jOXgi+Yn4SEG6xsHN0rUz0iAYj7367aonJcRYLQ7ytBqCY93n73SYS0J+ZK2tx0e99jR8XniKXyk4jkxBSLVInU1jVFZSa6asVZUyD1EM5jsyQ6yp6aB61aiC3ITmkSl7eWlMtXKza7RXOAyFxEsmkSbRZ9q5fLIMLxsz82qL+jnQ45dmx/iWR27d1k16KhgNs+6HoMFHHe1TzOV6xYIatWrepwr7XERuEcVi2avfv3k8n9JktCdPOJLgrnkUmeEXHerTGPRiQydWSu2woF/YodV/0Mz0lz/7200efcFM5p1UJI5ySUfTfACrBgrFoQcY59mZ+EkOAY0ztTHjl+L3n4uL0iyurEXLWuiaTyE9IWuhtWLX1a6W8e6r7bmdCzIbeYWNLdVdzs9RzZpP5F4GxdXI5aWdYphPNvvvlGjjjiCOnZs6e6GXnvvff87nvhhReqfR5++GGP7fn5+TJ9+nRJSUmRtLQ0Offcc6W0tNRjnwULFsg+++wjcXFxkpubK/fdd19ri0oiMDkoWLatSF6as9z9fI9+WcoTySTBRh7nZgRgXb0lNY0zaIeO6OPhPxXpPucgxhUlsS57NmaRhk5wW11bLzWNiWS9KTNE9SQbrbLoCJDR++qrr5abb765w7N7I0eCQ3XhDbicDhneo/mysHqrXvVhdp35JoHZ3pgY1PsGM9LonhQv00b2cScMtaMQNCw7zX1FaZ/zDYUN93WxrqiIjfgnhNin7wbIOQJRvCVg5xLrpHhGSGtF6EgLkPIZcU7hnHRykLy3f2ayCkzZb1APW/fdroReyr8Ywnl21JZmr/etX6j+hYJWlzZW7EqrlZiysjIZM2aMnHPOOXLsscf63e/dd9+Vn376SQns3kA037x5s3z22WdSU1MjZ599tlxwwQXy2muvqdeLi4tlypQpctBBB8nTTz8tCxcuVN8HkR37kc4HREF4mVfW1Hn4ok4e3EMunDRCtpZUyH/mrZQfV29Tmb972sz+xFsghyh65Ki+EumgMYaH1o6yhqSndox0jFQSjVUTSBCaGt886Sr88jWRdiPbWiAGZmVlSVVVVYcLg7g+8R0N32LJ4KxEiYtuPo+MwbnL4WLEeYSyvbQp4rx7cmQLt9MnDJajR/ez7coTlAuJ0Nbkl8q6/FKV70Nb5cCmxc5J0QghkdF3g2hncG0gotKZ2JuQzg8m5hFgh+A1vTKvswcbEQKr4LuOmNDmfjeUfXdUdJLUJPST6LLVklW/XtIdO6TAarKw6l29QMTZEHFuddtL7EqrW5XDDjtMPQKxceNGueyyy+TTTz+VadOmeby2dOlSmTlzpsydO1d22203te2xxx6TqVOnygMPPKCE9ldffVXNfDz33HMSExMjI0eOlN9++00efPBBCuedFFywmDFem1/qkdjjjN2HqL/x2uX7jZLz98bSS6ftBuHeSUggcHQGoRPHZdfcbvLZHxvVcwrn7YcpgCGy3Fs4r62vVwmFAG4IO3v0RGxsrDz77LMqSQn+7nDhvPE/3GaP6ZXk3yM1yknhPELZZgjn3SI44lxjV9FcM7xHuhLOcU3NXr7JbYREmxZCOi+h7LtbY52G+1farBHS+YEmgFxpW4obVhmmxdOiiXQNdkbwDnXfXd19srjKVitNY4zzN/mq9kC13SH1Uhw/RGod1VIhLnEltD56PmI9zuGRc/rpp8u1116rBG9vfvzxRxU5rkVzgMjyqKgo+fnnn9377Lvvvko01xxyyCGybNkydXBJ58Tbo+wvk4a77SzM2TW7iebeEecQ0Q8Z3ls6C6ZdS3Jc86ho0jYSjXO7rLr5suN567ZLcWPE+W59uqtzn7QPmNRSEecOh0Q5LJ82LXqpNwbeHHxHJjsahfOUuGiJocVUSBKEar5Y1jDZCnqn+b6+CCGktQTbHyNyjX03IV2D7OSmleidPdCIkEikPvsA1S9DOO/h2OzebkmUrMo6UzaNe0pWDrg66FVl4aDdlZh7771XXC6XXH755T5f37Jli1oW4FEIl0syMjLUa3qf/v09kypmZ2e7X0tPbxqcabDMAA8N7F6IRKxH2YFDe0ZMFnIwOCvV/ffJ4wd2KpFmZE66msnPK62UYUYCOLJzmJNCZUYSUM2XyxsSZYADhjS3vCJtJz7aqfIqrCvEOR3j06ZFW7UkxSTZ0lOaBAZ5AwrKq9we4SR0PueINC+qaPJL7J2eyOonhLQLEMOdDqd7RVhL+xJCOj/ZKfEijfP1FM4JsR9RSX1lfe5pUpW8p7z+VZNmC2Kjo6SmvlYcMam2zk3SrncU8+bNk0ceeUTmz58fcqHh7rvvlttuu63ZdkSo19W1nH29o4lkIT9UZd+jR7LMWxMjGQkxMm1Q93ZbXRCK8kMWuG7fIVJVVy9DM+MiquzBcM2kwbKlpEIGdUsO+rfZpextIRRld9RWS21dQ6T51vxCKUhsGgDml1fJ/HVblQCVmRArveIdrTqnIrHuYc/16KOPqglQrFgyVxx1BJdNHCg/rVsr3ZPLpKKkKYmkSUVlhcTHx4fknI/EY2ZnsDLptsN3k7ySik41kWn3lRx9MpI8LNcArVoI6byg777vvvukoqJCZsyY0eF9txLOo5xqRZgTpqh+Jr0hrlM4J6TrBd9lJNpXeCOkK/fdOzInS7eELHE4VoplmUm/o6SqrkpiXbHq0SWE82+//Va2bdsmffr0cW+DaH3NNdfIww8/LGvWrJGcnBy1j0ltba3k5+er1wD+3bp1q8c++rnex5sbb7xRZYY1RYjc3FwVnZ6SkiJ2wFekfKQQirLjKx48ISeCy5/eac8blKBvj8gse1vp6LJ3T68Ql7OhCY6KifP4vtlr/xRn42sHj+gjmRkZnb7uKysrZcGCBSphdGpqqsTFdawnNaonPcMlS/OWSryfiORiR7FkpGdIekp6h9e700lxtz3BUsDB3VPVg4SO4TnpHsJ5rCuqWfJsQkjnARadsNpE342/OxptnwZxPMbpe6Bfb9Urcb2liHRCSOegT3qTJVzPlCbbFkKIffpuh7JJtSQlziVFFbWSKoVSL1FqrFBdVyGpsam2nvBu15LB2xx+5SbwJsf2s88+Wz2fOHGiFBYWquj08ePHq22zZ89WB2yPPfZw7/O3v/1NHcjo6Aafm88++0yGDh3qV5SAqX0ojO0JIaQ9SDSsWkoNq5Z6y3LbtGDdzn6Du4ZNCyy7Lr30UikpKVF/h4J4V7zqoGvqavx6qtnZa40QuzE8J01mLlnvft4rLdGWeUkIIZHZd5tWLf6AqM78JIR0HUb2SJfjx/WXksoamTTQvskFCemqfXd0VLS4HA2T3qnxEM5r5MDoz2Si83tJ/XO4rOk7XVJT9xE70+paKi0tlZUrV7qfr169Wn777TflUY5I88xMT19qCN+IEofoDYYPHy6HHnqonH/++fL0008rcRwH7eSTT5aePRsEolNPPVXZrpx77rly/fXXy6JFi5QFzEMPPbTzv5gQQmwAbA005UZy0EWb8mVHWYP3F3z+u0q0JjptTLTCFiVUwnmcK04tCcPyMH8CuZ1nvgmxG8OzPYMbcpkYlJBOTTj6bvTbRVVFfl9X/ue0aiGky4BI1uPGDgh3MQiJGELdd7uwWszpEqsyTw5yfiTdYn+W7o4GF5KEyjVS7UpT43I74zsjWgB++eUXGTdunHoA2KPg71tvvTXoz3j11Vdl2LBhcuCBB8rUqVNl0qRJ8swzz7hfxzL9WbNmKVEeUemwesHnX3DBBa0tLiGE2D45qBlxPttMCjq0a0Sbhwss406OSZbK2spmr1XXVatl4HbvxIPlm2++kSOOOEJNUGOA8d5773m8jkzn6Gd79OihfN2xemzFihUe+8BSbfr06cr+LC0tTU1uYzLdBHY7++yzj7LagV0a/PNI1yE5LlpyjWSgiDgnhJB2bWdiktVKMX8gos3OPqmEEEJIV8LVuFrMqquU3Ws+dovmoCJ5mDijU20/5m719MJ+++2nBtjBAl9zbxCd/tprrwV83+jRo5VnOiGEdJWI8+KKavllXZ76OyUuWnbN7SZdBfQr69evl6KiIiXKhirBdEpsimwu3dxse3lNuSRGJyo7l85AWVmZjBkzRs455xw59thjm70OgRvJWV988UXp37+/3HLLLSoSYcmSJW6/eYjmmzdvVtZpWC0GCzZMaOv+HLlFpkyZokR3rChbuHCh+j4cT058dx1G5KTL+oKyZr6jhJDORzj67qTYJPU98DKPcjSPAUPiUArnhBBCiD36bofDoYTxouh0KYkdIFLZEJyFry1JHS0xLvsHq3ENOiGEhDnivKxROP9i+Uapq2+YmNx3UA9xRbV6UVDEUlVVJZdccokSZBEN3dHJQTXx0fFq4K2WdhuJxKpqq6RXcq+QCfgdzWGHHaYe/m6ekMD75ptvlqOOOkpte+mllyQ7O1sdC1ipLV26VGbOnClz586V3XbbTe3z2GOPqVVjDzzwgIpkx2oyZGl/7rnnVHb2kSNHKiu3Bx98kMJ5F+LwXfrKirwi6ZYUJ6N6tT6xMSEkcghH350Uk6QmtTHBjb+9QX8e62TEOSGEEGKXvjs5JlnyyvOkMG1PSSvSwrlD8pNGSEpMss+JcDth79IRQkgnJc7lVLOsoKyqRnaUVcr/FjSs0MH2A4b0kq4GLECSk5ND+p0YfGOGGz7nGpV0zNEwOO8KwBZty5YtHsm9YZmGhN0//vijeo5/EZGgRXOA/aOiolRWdr3Pvvvuq0RzDaLWly1bpjz0/N24IVLdfJDIBoL5XUfsLlftP5qJQQnpAoS678aS77S4NKmoqQi4DyGEEELs0XenxqWqBKFlGXs0bXQ4pTy6myTHhnb83xZ4V0EIIWEAM6ywa0EGeEScv/7LSqmqrVevHTi0l/RITehSxwUz3YhYhsAaqmhzgKSgsGRBorGE6IY6r6itkARXgiTGdA1/ZojmABHmJniuX8O/WVlZHq8jmQys18x9YPPi/Rn6tfR0z8SR4O6771bJwL3BeVBXVyfhJpKF/Egue6SXn2VnvXel8+bxxx9XZa+oqFCPUBBVFSXVpdVSXlfebGVYbXmtVMRUSEGN7wnb9qz3SD1mhBBCui7hGHcnxiQqgXx7gkO+qd1PdnX9It85T5IxjTYudofCOSGEhNGuBcL59rJKyfuzIUFlYqxLThw3kMckhCBybXv5dvdzRLFlJ2UzYi0E3HjjjSrJuClCIKkoRHZEQtgBX4J/pBDJZY/08rPsrHeeNx1HfG285NXnSZQrymPArfOTZHfLVlZsHX29Op1NFm+EEEII8Q2sWLondpf8inz5Of5E+aDkaBmfnSgxTvv7mwMK54QQEiYSG33OzXzLJ+86UJLjmhKHko5Hd9bw+tYJx1JjU7tM1efk5Kh/t27dKj169HBvx/OxY8e699m2rSkDOqitrZX8/Hz3+/Ev3mOin+t9vImNjVUPQgghpDX9NvxSsVrMHHCXVJVIbmpuq0RzQgghhHQ8yTHJKgfJGRNTZVNhnfRKr5M4V2xE5CWhxzkhhISJxBhPgbxfRpIcMLTreZsDJJVEkkkknMTfoQQD7FhXrPI5r66rVjPfXcWmBcBeBcL2F1984RH5De/yiRMnquf4t7CwUObNm+feZ/bs2VJfX6+80PU+33zzjUo0o/nss89k6NChER19SwghxH59d0Z8huqzzaTe0VHR0i2hm3QlNm7cKKeddppkZmZKfHy8jBo1Sn755Rf36wgKuPXWW9XEOF5HfpIVKxoSs2kwCT59+nS10gv5TM4991wpLS0Nw68hhBDSWfvuhOgESYlLEaezUsb0ThZLapSY7m25ZkconBNCSJiALYvJWXsO7bLJ9CDAfv311/LDDz+ov0MJZrmRJLSytlIt80anjuedCQyAf/vtN/XQCUHx97p169TNypVXXil33nmnvP/++7Jw4UI544wzpGfPnnL00Uer/YcPHy6HHnqonH/++TJnzhz5/vvv5dJLL5WTTz5Z7QdOPfVUlRgUA+7FixfLf/7zH3nkkUc8rFgIIYR0HsLZd2OCG0K5Fs+LKoukW2K3LpPYG8Cfdu+995bo6Gj55JNPZMmSJfLPf/7TY7L6vvvuk0cffVSefvppNSGemJioEndXVjZYBAKI5ui3Mdn94YcfqknwCy64IEy/ihBCSGfsux0Oh5rc1v02VnlHSrAarVoIISRMJDZatYBJA3NkaHZalz0WSDR53nnnKYEXf4cSdOLwOUdENeiV3CsiZr5bA6LP9t9/f/dzLWafeeaZ8sILL8h1110nZWVlaqCMepg0aZLMnDnTI2EMkshALD/wwAMlKipKjjvuODUY16SmpsqsWbPkkksukfHjx0u3bt1UlBsH34QQ0jkJZ9+NSW48kJfEgf8cDume0F26Evfee6/KC/L888+7t5lJuhFt/vDDD8vNN98sRx11lNr20ksvqcTd7733npr8Xrp0qerv586dK7vttpvaB1GIU6dOVRGJenKcEEJI5yCcfXdKbIqyWEOwmjiaLFPtDoVzQggJE+N6d5PZyzdJenysnLrboC59HNBpY1CH6KlQd+AAg29LLJW4pDNGq+23335qAO0PCA633367evgjIyNDXnvttYDfM3r0aPn22293qqyEEEIig3D23eivM+MzZXXRaqmpr5H0+HQ1IO9KYJUYosdPOOEEFT3Yq1cvufjii9XqML26bMuWLcqexZzkhsXajz/+qIRz/At7Fi2aA+yPCXJEqB9zzDHNvreqqko9THs3QgghkUE4++44V5ykxabJhpINasxN4ZwQQkhAxvfpLo+dsLfER7skwYg+J6EHPucxUTES7YxWIjohhBBC7E1ybLL6t66+TnKScjrdarGW+PPPP+Wpp55Sq8huuukmFTV++eWXK9s0rCiDaA4QYW6C5/o1/JuVleXxOoQUTJbrfby5++675bbbbmu2HSJMXV2dhJtIFvIjueyRXn6WnfXO8yY0RNdES31FvTjqHVJWXCbljvKQXq9teS+VGkIICSOZiZGxPKmjQTR0Xl6esglB5FOoB7+Y7UaCUAzCIZ4TQgghxN59N7xRE6MTxRXlktTYVOlqwJsWkeL/+Mc/1PNx48bJokWLlJ85hPOO4sYbb/TIXwIRApYx8FZHglE7EMlJySO57JFefpad9d4Vzhvdd2PlUFj67rpE2WHtUPZqGWkZIa93p9PZ6vdQOCeEEBJ20HEjqWRNTY3y3TS9tUO15BudNyLPCSGEEGL/vhuCue67nVGtHwhHOj169JARI0Z4bEMy77ffflv9nZOTo/7dunWr2leD52PHjnXvs23bNo/PqK2tlfz8fPf7vYmNjVUPQgghkUe4++4YZ4xkJ2ZHTGJQEBXuAhBCCCEAgzAsLw4XvVN7S2ZCJg8GIYQQEiF9d25qrnRL6CZdkb333luWLVvmsW358uXSt29fd6JQiN9ffPGFR3Q4vMsnTpyonuNfrBiYN2+ee5/Zs2eraHZ4oRNCCOl82KHvzohve7R5qGHEOSGEkLCDme633npL+WOGetabEEIIIa2HfXd4ueqqq2SvvfZSVi0nnniizJkzR5555hn1AFh+f+WVV8qdd94pgwcPVkL6LbfcIj179pSjjz7aHaF+6KGHqoSisHhBBOKll16qEodiP0IIIZ0L9t2th8I5IYQQQgghhBASQUyYMEHeffdd5Tl+++23K2H84YcflunTp7v3ue6666SsrEwuuOACFVk+adIkmTlzpkeQwquvvqrE8gMPPFCioqLkuOOOk0cffTRMv4oQQgixFxTOCSGEEEIIIYSQCOPwww9XD38g6hyiOh7+yMjIkNdee62DSkgIIYRENvQ4J4QQEnawNPixxx5Ty4vxNyGEEELsDftuQgghJLJg3916KJwTQggJO3V1dTJr1iz58ssv1d+EEEIIsTfsuwkhhJDIgn1366FVCyGEkLDjcrnk9NNPl9LSUvU3IYQQQuwN+25CCCEksmDf3XqoThBCCAk76MBPPPFEKSgooHBOCCGERADsuwkhhJDIgn1366FVCyGEEEIIIYQQQgghhBBiQOGcEEJI2LEsS4qKiqS4uFj9TQghhBB7w76bEEIIiSzYd7ceWrUQQggJO1VVVXLaaaepLN/vvfeexMXFhbtIhBBCCAkA+25CCCEksmDf3Xo6rXCuIxYRvWgHUA6n0ymRSCSXPdLLz7Kz3rvKeVNZWalE89raWlX+6upq6Ur1rvuqrh5tz767a7cDnaX8LDvrvaucN+y72XcD9t1dux3oLOVn2VnvXeW8Yd9d3Opxd6cVzktKStS/ubm54S4KIYSQVpCdnd1l6wt9V2pqqnRV2HcTQkhkwr6bfTfH3YQQElmw704Nqp4cVicNb6uvr5dNmzZJcnKyOByOsM9C4UZi/fr1kpKSIpFEJJc90svPsrPeed50nesVXTFE4549e0pUVNdNP8K+u32I5P4j0svPsrPeed5EDuy72wf23e1DJPcfkV5+lp31zvMmcigOw7i700acowJ69+4tdgIHNdI6kc5Q9kgvP8vOeud50zWu164caa5h392+RHL/EenlZ9lZ7zxvIgf23TsH++72JZL7j0gvP8vOeud5EzmkhHDc3XXD2gghhBBCCCGEEEIIIYQQH1A4J4QQQgghhBBCCCGEEEIMKJyHgNjYWJkxY4b6N9KI5LJHevlZdtY7z5vIIZKvV9L5jmkklz3Sy8+ys9553kQOkXy9ks53TCO57JFefpad9c7zJnKIDUNb02mTgxJCCCGEEEIIIYQQQgghbYER54QQQgghhBBCCCGEEEKIAYVzQgghhBBCCCGEEEIIIcSAwjkhhBBCCCGEEEIIIYQQYkDhnBBCCCGEEEIIIYQQQggxoHBuM5irlRDCtoaQyIJ9NyGEbQ0hkQX7bkII2xoSDBTObURhYaHU1tZGZGe+atUq9QDmb4gEFi1aJG+//bbU1dVJJLJixQp54IEHZNmyZRJprFy5Uvbdd195+eWXI+6c37Jli2zatEkqKirU8/r6eokUSkpKPJ5HUr0DXeeRTKTVOfEP++7wwL47fLDvDg/su8MP++7OA/vu8MC+O3yw7w4P7Ls7R99N4dwGVFdXyyWXXCKHHXaYTJs2Te69914lwjkcDokEZs+eLYMHD5bjjz9ePXe5XBIp9X7uuefK6NGj5ddff5WoqMi6HCD047wZNWqULF26VPLy8iRSQN2fccYZMmzYMPnuu+9k8eLFansknPM1NTXyl7/8RSZOnChHHHGEum4rKysj4vxB2S+88EKZOnWqul5feumliKl3Xf6LLrpIjj32WHX+/PTTTxEziEXZMcH17rvvRlSdE/+w7w4P7LvDB/vu8MC+O3yw7+58sO8OX71z3B2+uue4O/Sw7+5cfbf9lZ5OzmuvvSYDBw5UwuF1110nvXr1kjfeeENefPFFiRQQ6YyoYQi3//d//xcRUeePPfaYZGZmyh9//KFE8zvvvDPihKwHH3xQfv/9d/n666/l2WeflUmTJqntdhcS77nnHklPT5e1a9eqmW+Iz4jeBnaP+t+4caM61xHlj2v3iiuukPXr18sNN9wgdufPP/+UCRMmqHMebU1qaqo6FhDSIwGcI3vssYcsWLBAnTP4F2W///77bR/x/8knn8iYMWNUvWN1C1YqRMK1SvzDvjs8sO8OH+y7wwP77vDBvrvzwb47PLDvDh/su8MD++5O2HdbJGxs3brVmj59unXfffe5t+3YscMaNWqU9corr9j+yNTX16t/r7/+euv888+3br31Vqt3795WVVWVx+t2o6ioyMrIyLAOOOAA97alS5daK1eutIqLiy27g3otLS21Jk6caP3f//2f2vbDDz9Y//rXv6xvv/3WKikpsezKv//9b2v06NHWf//7X/e22267zRo4cKAVCbz++uvWmDFjrM2bN7u3nXHGGdbNN99s2Z3HH3/c2m+//ayysjL3efTUU09ZDofDevvtt626ujrLzrz11lvWyJEjrQ0bNqjnhYWF1t///ncrLi7OWrRokW3bHFyraB8vv/xy6+6777Z2220368knnwx3schOwL47PLDvDh/su8MH++7wwL6788G+Ozyw7w4f7LvDB/vuztd3M+I8DOjZDkTdYibk7LPPdr+2bds2SUtLU48dO3aIndER2og0h8XMCSecINHR0TJjxgy1vby8XOxY7ykpKWrpxm+//SafffaZnHjiiXL44YfLoYceKgcddJA8//zzYvd6x8wZZjJR5muuuUaOO+44tUoB/x5zzDFSXFwsdkJHA6NsqHecK5rExESJj493e+Tb3Q8R0eY5OTnq+ebNm1Xkc0ZGhrKcsTOI7sdKkISEBHUt4DzS18Q//vEP27Y3+txBO1NQUKBW5QBEzMMyByst8C+w46oR1PdZZ50lF198sVqZ0KdPHzUTjvPG7pHyxBP23eGBfXf4YN8dfth3hwf23Z0H9t3hrXeOu0MP++7ww7678/XdFM5DyJw5czw6EojM8Nfu1q2ben7jjTfKyJEjleAMH98DDzxQPv74Y1uIK7rsZjn074CYWFZWJkOGDFG/4amnnpLp06erv+0gxnnXO8AFNWjQIDnkkENUh/7cc8/JI488ovzCb775ZuXbbhd81X3v3r2V1QzKCsuTL774Qt5//33177x585T1jB1sILzrHgKzFjf1NthvLFmyROLi4jy227He4WsOwRZlhkc4GmM8/+ijj5Rv+O233648texY9uTkZFXHaFP0Mfj+++/ltttuU4l6Zs6c2ew94eKtt96Szz//XE1MaO94p9OpJiy+/fZb9354jk5x7ty5ahLMDuePWXaAut5rr71k6NCh6jnsZTZs2KA811DWSPDG7+qw77ZHvQP23eGpe/bdoa139t2hh31354N9tz3qHbDvDk/ds+8Obb2z7+7kfXe7xK2TgLz77rtWz549rczMTGv16tVqmy9bhCuvvNL6/PPPldXJ8uXLrb/85S9Wbm6urcteWVlpDR48WC1/07YbsE6IjY215s2bF1brBF9lr62tdb8+d+5c64YbbrC2b9/u3ob9jj76aGvq1KlWuAlU/vz8fOvcc8+1kpOTrWOPPVYdE31csCwrNTXVKi8vt1XZ/VmBwCKnT58+1vPPP2/ZAV9lr6mpcb+ObZ988ok1YsQI66WXXnJvh71SYmKitX79estOZdfWSUuWLFHnNs6Nk046yUpKSrJ23313a+PGjer5EUccYYUb1GdWVpYqV/fu3a29995b2ciA+fPnqzq/55573L8JbNmyxTryyCOt008/3XZlx/HQ577ZFl588cXW5MmTVXtvV4sZwr47XLDvZt/dXucN++7QwL6b2AmOu+1T7xx3h6/uOe4OT71z3N15x90MdetgXn31VWWDgISCw4cPVwkagDnboWenkOwRUeYxMTEyePBg2W233dRrSOZnx7KjbJi52XXXXVWylXHjxsnjjz8uJ510klomUVRUpGZ9wpEo1F/ZEbWqGT9+vPztb39Tkduafv36SVJSkvpdiKIPFy2VHzY/+lxBQk0cEz27vMsuu6jtS5cutVXZ/c3wIQo6NjZWKioqJNz4K7vL5fI4R2AZgmNx2mmnua9fWIYga7leCmSXsuNcwLmBbY8++qg89NBDapXLK6+8Ij///LP07NlTlRvR8+ECbQRWfNx9993qNyCq/L333lOJk//973+rcwPtC+r4nXfekR9++MH93uzsbLV6J1yR24HK/swzz0hVVZUqG9pCfa5cdtllUllZKf/73/9UO4Pjs3z58rCUn/iGfTf77rbAvjs8sO8OD+y72XfbDfbd7Lvb87zhuLtjYd8dHth3W20ad1M47yAgZgLYgUDgvPfee+XII4+Ur776Sj3MfbTg4+3PO3/+fNWADxs2zJZlR7lLS0uV+ANbFohasNyAf/jBBx8sp556ajPR0S5l1/UNkdwEAh38wyE+w3s71ARTfoicANtPP/10ZdGCJSq6c4fX9tixY9XDbmXX+2ggGMKzGuLnTz/9FDarkNaWXS/1QU4Cff3CrgWTSLvvvrtty56bm6tyKmCC66ijjlLbtmzZIuvWrVPvDxcQj+FhfuaZZ6ryQezHMqsRI0Yov359zsNWBlY4EKQ3btzocd1iOaIdy25OHOoJLrTp8Pv/5ZdflK3ShAkTlL2V9/VBQg/7bvbdHXXesO9uf9h3s+9uK+y7Oxfsu9l3d9R5w767/WHfzb47IvvudoqWJ43AYsU7/F8vFV20aJGyFDBtQLz3LSwstNauXWudd955Vv/+/a0PP/zQ5352KHt1dbX694MPPlC2Jyaffvqpdccdd6jPs2PZfdX7unXrrHPOOccaPny4spkJJa0tv1769ueff1pnnHGGsgiBZcspp5xiZWRkWP/617/U65FQ93h+xRVXWHvttZfKhBxKWlt2vezts88+U0t+dtllF+vpp5+2zj77bFXvDz30kG3L7r3vmjVrrA0bNljTp0+3xo0bp9qdUOJd/l9//dV9Xut6fvXVV62xY8d6WLO8+eab1j777GP17dvX+uc//6ksWrBU69tvv7V92c3X0WZGR0dbDofDuuCCC5rtR0IL++4G2Hd37HnDvrt9YN/Nvrs9zhv23ZEP++4G2Hd37HnDvrt9YN/NvjuS+24K5+3Ef/7zH6tfv37W0KFDldfOs88+637NPNDPPfec8unFv97+U7NmzVI+5zk5OdZ+++2nThI7l930jvTeP1Seve1R7/Crvuiii5Q/Fep9xYoVISl7e9Y9xNtrr71WCbh//PGHrcvuy3PtwgsvVMcgVOJhe9T7999/r3zBDznkEOuoo46KqHqH//3NN9+sxH6I0PCZDxXe5Ycnv4lZzlNPPdU666yz1N/muQHBH52ezkcQrroPtuze1+tTTz2lOu4pU6ZYq1atCknZiW/YdzfAvrt1sO9ugH11la1vAABMp0lEQVR3x58z7Lt3HvbdnQ/23Q2w7w7NecNxd3iuV467dx6Ou612G3dTOG8HIHijMXjiiSesmTNnWldffbWa0XjmmWfcCRr1hQ/BB0kdJ0yYYJWUlHgIQoh4RkMye/bsiCm7jjqP5HpH5C0+QycMiJTyR3Ld67Lr2cJQ/padLTsS4mowoMVqhUgpu1nPv/32m/X111+HrOwtlb+iokLto1ep4Pno0aOtl19+2e/n6fdEWtl///13dSNDwgv77siud/bdoa979t3hOefZd7cd9t2dD/bdkV3v7LtDX/fsu8NzzrPv7lx9N4XznUDPkN12223W+PHjPS4OZG/dbbfdrHfeeafZ+2C/gtdmzJihDuThhx+uRPNQwrL/bk2bNi3k9c665znf1c6ZtpZ/48aNqsPUK2/w71VXXRXikkd22Unnu55YdvbdPG94vdq5vbFL/xfJZSe+Yf8X2fcdHHeHr+4j+bxh2btW/1dv47IzOehOoJN5IiHmwIEDJTo6WiWuAzCej4uLU4kzkXwPaAP6/fffXyUQvP3222X8+PHqPVlZWTtTFJa9DfWO5AGhrnfA84bnfFc6Z9pSfoCEt0hm2qNHD7niiitU0o+1a9eq92HSl2UnXfF6YtnZd/O84fVq1/YGsO8mdjkf2Xfbq9457g5f3fN+tevUe1vKD9h3B0G7S/GdGCwZuOyyy1QCwJ9//tm9HUsGkpOTm9lOYPuQIUOsr776yr0vkh/i/U6nU/lpL1iwgGXvxPUe6eVn2VnvoTxvvvzyS/ds8wknnGClp6er3AMjR45sloC4o4jkshPfsB1jO9YWeN7wvOlK50yk93+RXHbS+a4nlj3y2jGeN6z3SDxnIr3/mxVBZadwHgSbNm1Sy0SysrKs6dOnW6NGjbJSU1PdB3fZsmVWr169rFtuuaVZEjsk+sSJoFm8eLG1xx57WC+99FL7H02W3Tb1DnjehKfuWe/hqff2rPuysjL1Ob1797beeOMNlp2E9XwEbMe6Rr1HevlZdtZ7OM8b9t2kPWA71gD7D543vO/oWCK5rWnP8rPvDg4K5y2AE+nMM8+0TjrpJOvPP/90b0dG4LPOOkv9XVxcbN15551WfHy82z9K+/NMnjzZOu+886xwwLKHp95Z9zznu9o50xHl/+WXX1h2YpvzMZSw7J2nHWPZWe92PmcA++7w3HcQ37ANZv/RFnje8LzpSucMYN99Xsj7bnqct0BCQoLExsbKWWedJf3791f+XGDq1KmydOlS5bWbnJwsp556quy6665y4oknKh9eeAutW7dOtm3bJkcffbSEA5Y9PPXOuuc539XOmY4oP7zhWHZil/MxlLDsnacdY9lZ73Y+ZwD77vDcdxDfsA1m/9EWeN7wvOlK50xHlJ/j7iAIiTwf4ZjZXOvq6tS/p556qnX++ed77LdhwwZr0KBBKqvr8ccfb/Xs2dM64IADrC1btljhgmUPH6x71ntXOmcivfyRXHbS+Y4pyx4+WPes9650zkR6+SO57KTzHVOWPXyw7lnvXemcifTyV0dg2R34XzACO/Fk0qRJcv7558uZZ54p9fX1altUVJSsXLlS5s2bJz///LOMGTNGvW43WHbWPc8bXq9sbzp3W0k63zFl2Vn3PG94vbK96dxtJel8x5RlZ93zvOH1yvamE7SVIZfqOwGrVq2ysrOzPfx0TLN9O8Oys+553vB6ZXvTudtK0vmOKcvOuud5w+uV7U3nbitJ5zumLDvrnucNr1e2N52jraTHeesmGdS/3333nSQlJbm9gG677Ta54oorlFeQXWHZWfc8b3i9sr3p3G0l6XzHlGVn3fO84fXK9qZzt5Wk8x1Tlp11z/OG1yvbm87VVrrCXYBIAmb6YM6cOXLcccfJZ599JhdccIGUl5fLyy+/LFlZWWJXWHbWPc8bXq9sbzp3W0k63zFl2Vn3PG94vbK96dxtJel8x5RlZ93zvOH1yvamk7WV4Q55jzQqKiqUQb3D4bBiY2Ote+65x4oUWHbWPc8bXq9sbzp3W0k63zFl2Vn3PG94vbK96dxtJel8x5RlZ93zvOH1yvam87SVTA7aBg4++GAZPHiwPPjggxIXFyeRBMvOug/HeVNXVyc1NTVhqfxzzjlH+vbtKzfccIPExsZKJBHJZY/08kdy2UnnO6YsO+ue503kEMnXa6SXP5LLTjrfMWXZQ0d0dLQ4nU73c2oe4YH1Hj5Y9x0LhfM2ABHQbJgjCZaddR/K8wa+VVu2bJHCwsKwVTzKoJcBRRqRXPZIL38kl510vmPKsrPued5EDpF8vUZ6+SO57KTzHVOWPbSkpaVJTk6OOl+oeYQH1nv4YN13LBTOCSEdxubNm5VoDn+qhISEiL3xJYQQQgghhBBivwkKeCIjkSDE8x49eoS7SISQTgaTgxJCOmzWU4vmmZmZrGVCCCGEEEIIIe1KfHy8+hfiOcaekeoOQAixJ1HhLgAhpHOiPc0RaU4IIYQQQgghhHQEeswZrrxahJDOC4VzQkiHQnsWQgghhBBCCCEccxJCIg0K54QQQgghhBBCCCGEEEKIAYVzQghpBWPGjFFR9N9++22r6+3vf/+7/PDDDx1a3yjbAw880KHfQQjpeNBeJCUltfo1u7Dffvup9ijQ46yzzgpL2fC9vspz+OGHt9t3fPXVV/KPf/xD2pNRo0bJQQcd5Pf1a665RhITE6W0tNS97YorrlC/7Y477vD5nn79+rl/v8vlUs/PPPNMWb9+fYvleeGFF3zWo93PTbscz52htr5WqmqrQv7A97aF999/X/bYYw9JTk5WiftOPPFE+fPPP5vt9+yzz8qQIUMkLi5O3W99+OGHHq/j3D711FMlNTVVxo4dK3PnzvV4HRYNw4YNk3fffTfodqot1z3O/ddee03sSmv6iGuvvVZOOOGEZtf19u3bm+0b6DW74K99Nx847h2N/q7Zs2d7bEf+J2xHXbaGNWvWqOO6adOmgPsdccQRMnjwYL+vP/bYY+r7V61a5d720EMPqW3nnntui/15VFSU9O7dW4499lhZsmSJz/2feOIJmTBhgt8yHH300T7HKwcffLDcddddHtvq6+tl6NCh8uqrr/r9PEII6WiYHJQQQoJk8eLFsmDBAvU3Bkz77LNPq+rutttuUwOZvfbai3VOCOnUPPnkk1JcXOx+fvHFFyv/UXOg3L179zCVTmTAgAHNBuLp6entKrTit950003t9pkQDG+55RbZsmWL5OTkNBMX/vOf/8iRRx7pFsyQpBvbdJ+F9/ri+OOPV6I7REcIkTNmzJBff/1V5s2bJ9HR0S2Wa+bMmUrI1HTGpGwdcTzbCsTrJXlLpKKmIuTfHR8dLyO6jxBXlKtVdXfMMcfIGWecoUSxHTt2yK233ipTpkyRhQsXupP6vfHGG3L++efL3/72NznggAPUuYv3IVBhzz33VPtg8gJi3X//+18lPEKAX758ufs8ffjhhyU3N1e9ryPBd+M6wzUZyUCEhcjZlmAQu4J27sILL3Q/x6ThH3/84dHep6SkhKw8t99+uzqfdxYI5xhHYKKnZ8+efvfDOYkH2nJf4vXrr7+urqeBAwe6t+m6eeedd1TfHRsb2+x9e++9t2oD0a/gGrz55pvVRC7GRmbfWV5eLnfeeac8/vjjPsv3ySefyE8//eTzNbSvEORxv6A/E0L9DTfcoPqlk046SU3wEkJIqGHLQwghQYIbS9zATZ48Wd5880159NFHgxIVCCGks1NVVaXaQ7SRYMSIER6vQ6iA0KQFMF9UVFS4RbSOBt8TqCx2A3UDMQSiIgRFRJKbfPPNN7Jx40YPIe+LL76QrVu3KnHj888/l/nz58uuu+7a7LOzs7PddYEJ4crKSvU9v/zyi0ycOLHFso0fP166desmHXUuEU/q6uuUaB7tjJYYZ0zIqqe6rlp9L76/NcI5BPG+ffvKc8895857k5WVpcREnGM6CAHC2Mknn+xeHbH//vurYAUIjx9//LHa9tlnn6lz85BDDlER55hAWrFihWpvMKF0zz33RKQIHMq2z+Rf//qXik7GNRypQMjFxKG+H4cgbIrCmKBdu3ZtWPoenMNffvmlOidbG2zTVo466ijV12Ky1Fs4h/j+448/qvGLBhNPmCTV/cRHH32kxGtv0tLS3HUIER2rm6ZPn64mTk855RT3fuifMAmLcvhq2y+//HK5++675ZxzzvFZXxDMX3zxRbnyyivd2yGYX3bZZWoFCqLVCSEk1PCOlBBCgsCyLBWlgYHe1VdfrSKmcLNosnTpUnWzmZGRoSIrscwY7wF6sIglsXq5I6KwcBOLv9966y2Pz8INI5bMazZv3qxuMhEliZt7DHQQmYGbUEJI1waiBwaw3lx//fUqMg3Cgm5rMCDFcmxECKOtQntWW+tpv7BhwwY57bTTlBiK9mbfffdVA2sTtE+XXnqp3HfffUoUw375+flBlxntH8qDQToiniGsa7sALGVHxBksHRD5ht83a9asZp+B98L+Ad8NceSiiy6SsrIy2RmCbWsh1Dz44IMyfPhwVUYIeCh/UVGRWk6PyECUxZc1AERurDzC56OO8X1m3eljhahWROBmZmbK7rvvruoZ79P9igm2Yb9DDz3UvQ3CCawx8DkQlYJd6j5u3Dj177p162RngWCF44vzDUILBE9EGQd7LqHso0ePVtYdvXr1UqIpzmcTTBggmhkTAHgvrDoeeeQR9+svvfSSTJo0SZ3vEGVwLObMmdPsnEf0Mj4D39W/f3+56qqr1GstHc9wAdE81I+2ABEN56GZLF6vUMC9FYBtCwQ8HAMTCOmYANLXH/7VAifus/Q23d7hPPCetGsN2ooEKy4OO+wwdc6iDcA5pMGx//rrr1X7o88HnCPBtkv+2j5YjOyyyy7NygSxEPsvW7Ys6PM5WPBZKENbOe6445SI6s1TTz2lriN9HaP8mNS47rrrVJ3gfMDvLSkp8XhfMG2/ttdBXwYLD+z3+++/B11mf+2rPpfQ3qMdwueiffdlyQPxGeMBnB84lzFhuW3btmb7TZ06Vf0GTP60RKDzBucMRGUAMVyfd77AdQHRGqsy0E959xNYEQQhWoPfh8965plnVPu3s/0Ejgu+31dkOCLWcc4GsmnDtYDP8P5N06ZNa7adEEJCBSPOCSEkCOBNjpttLC/GwB832rjZhJcgQMQTIvOwRBiRHBBRFi1a5L6hxE02XkfEhI4IxOAuWKEJfpIYJEGowU0nBpgYqEHkef7553kMCemkeIvawHswjME/BHCItlqQgrj48ssvK79q0zoDogAsEjCoRgQy2rSYmBglaoCCggIlyiBiDV6o+Dz8C5EA7RwiRTVvv/22EpUgUuI7ICK0lgsuuECJ9PAkxmdUV1crn1NESsPWAWLpK6+8ogbNKC98vgEmGzH4P/vss5WoibYQy7lRfkS4tqVuMdAPtq1FW45oTYirKC8EIAgf8GA+77zzlBCLPkL722prAExAYH+IP1i5hN+JcmO5O/oZ81jdeOON6ndD7NDHHP3HJZdcooRGiPtamER9QHTUUZeIGseye1hWoA4hqKNe7r///hYjuSF2A4jHwYBzzaxL/AYIMagT/E5839NPP62ENBxTTMQgkhj9ZaBzCccAQhvq+J///KeanNbCuT5fMYmto+Lx2agTnKemfy/6bgiqiELF+YX61GWAnzbA67CtQP8N8Qh9N6KhQaDjSVoGIhkEWlhAYIIPxwztEIQ3LbrCSgNg0sMEwiWO2erVq9VrEA3/7//+Tx1ziLNon3AMYf3w6aefusXlnQXl1O0qvg+/Ad+N8uB3oM0yrafg+dzadsm77YNoC2EQ946mgI7zFStFIBIHez4Hw8qVK9Vn+RK+fV3X/voeTDCg3nX5AFYXoO1BW6pBP4Lfgd+I44l6QTul6yXYth/g2kTZIUijnTbbkmDx1b6iDf3uu+/U6gcca6x0wDHCd+B36vt5tGsQxRFdDXEbtiUQi/GaL+sYREnjHPUX9d7SeYN6g6UO2n70Q97XiTfoJyCAQ3A3bWLQhqGOzX5cW0+ivcfvh4Bu3ku0pp9A5D76MZyf3qBNRaQ5otr9if4Ak8OYRM3Ly/Owc8N23K/gWHE1EiEk5FiEENIBVFRUWEuWLFH/dgYuvvhiKy4uziosLFTP//KXv1gJCQlWSUmJen7qqada3bt3t4qKivx+Bprc+++/32Pb6tWr1fY333zTY/sVV1xh9e3b1+9n1dTUWK+++qrlcrmssrKygN9BCIk8ZsyYoa5nf4/ExET3vmh30B49+eST7m3vv/++2m/58uUebc0+++zj8T233HKLem9+fr56fuutt1qpqanW1q1b3ftUVlZaffr0sa699lr3NrRPmZmZVmlpaVC/Z/Lkyda0adPcz7/88ktVngsvvNBjv+eee061a4sXL/bYvscee1gnnHCC+ru+vl59/ymnnOKxzyeffGI5HA5r0aJFActy5pln+qzTb7/9Nqi2dtmyZep7/vGPfwQ8fuYx0hxzzDGqLqurq93bPv30U/X9OGbmsTr00EObvT8vL0+V5c4773Rv++CDD9T+33zzjXvbf//7X7Vt5syZ6vnrr7+unn/xxRcen4d6RP+G31leXq4+Izc315o6darVEs8//7zPerzjjjvU64888oiqJ9wLaHbs2KHq5eqrrw54LhUXF1tJSUnWjTfe6PGdTz31lBUfH29t375dPb/pppus2NhYVWfBUFdXp37r0KFDPT4bZXr00UdbfTzDQWVNpfXT+p+shVsXWsu2LwvZA9+H78X3txaco8nJye5zZOzYsdaWLVvcr7/yyitq++bNmz3eN3fuXLX9+++/V8/XrFljDRkyRG2LiYlR70N7MGHCBOvf//53q8vl3S7pc/qJJ55wb8N5iTZSn9e+3teadslf24fzEveROKc1aHNwHfi7r/N3Pgdzvr722muqHGhTgrmuzYd+D74f7dl1113nfv/ChQvVPrNmzXJvw/P+/ftbtbW17m3PPvusqpelS5cG3fbruo+OjrbWrVtnBQPa+5EjR7qf+2tfZ8+erbajPTY56aST1Pml2Xfffa299tpLHW8Nyozf8tFHH3n8Zhw37Dd69GjrsMMOU9sLCgrUa6jntpw3uCZaQp9L5513XrPj8tJLL7m3zZkzR217+umn1fMff/xRPcexMUGdo0/A51ZVVVnz589Xv2ncuHHq/kDzww8/+C3jsccea51++unN6scbfXw+/PBDj+369wfq3zvb2JMQYh8YcU4ICSk3vT9Hiiqqw1brqfEx8o8jG5ZkBgsibhAZiOgSHYGBaA5EGyJS6PTTT1dLifWS244A95iIxEMkCCJ1EKWjQeShr+W9hJAmFry6QBa+6mkR4Ytuw7rJIQ8e4rHt06s/le1/bG/xvaOmj5LR00e7n9eU18h/j/9vs+3BgiXbsPXwBu2AuXwc7Q6i1RDlh+XdAFFpiCJDFK+Jd9I8tFvwFIZ9BqIWsSweS8IRKaijDRENidwOSDZmgqg7M8rcjE5ERFkwSSIR8WeC70dkISInzc9DlByiDwGiwBHthkSA5j4oIyLREI04cuRIFTGprSCAuXQc0ZreEaCI4gumrUXUMfaD5U1rgdct/GDN/BhYAQD/WEQ66lVMvuoGwNoF+yNKEtHXAH/36dNHrRTQ4PxAVCF8a4FOGoooRO9EdYigxUODuvdlB+MPRBCa0YmIFNW/FfWFyE0NziscS/zWQOcSohYRvY9l++Yxxu9BVCOicnG80ffi95jWZt4gUh0RzvhM004B55EGEZ2IHsY5gvINGjQo6N9PAoN6x30SopNhsYGIc7Q5OL9xjrTGWxoWGkhMiGsRK/tg+fHss8+q6xGWRz///LN7RQbav3//+99tSkKMa0yD8xLfi1UHgQi2XfJ3fePcw/mOKGZEXGubFkQ0w7KmNedzMCCqGeXCCspgrmtdHkREa/B+tIOI/keZ8RvQD6G+DjzwQI/3om0z+wT0PXgvbGbQ9gbT9mtg32RGmftaPdSWvgftE9oT7+9HslH0J1gV8P3337uTZGpQZpQHfSTGCiboCxGRjmhunAPebUtrzxtvfPVz+lxCO45Idawqw99YJWHeA6CfQF+kbdIQEa8TZ3t7kCP63uy30L/g95qJRHFOAe9rDnWLRzArQnS+DP1ZvrYHqg9CCOkIKJwTQkIKRPP88sjy5cbNHpYM4qYf/osAN/fwYMRNJwaEGAgGynK/s+CG+q9//atatq6T5+CGFQNEU9ghhPimpqxGyra17H+dlJ3UbFtlQWVQ78V3mGAwi/d5bw8WDJp32223ZtshXngDUQpLmbFcH20T9oH46425RBvAlsIcpMKqBEvKfSU+NhOume8FWDZvLtmGcIJtLWF+hv5++Av7+n4tumAfX5MAmvXr17vLq5eTAwjhWmCFbYivun3ooYdabGvR3kOY8K7LYMDSe+/fDLDN27rL13564hb2ATjWEGH+97//KesYvfwd/RREDvRNpocwbMZg3wKR3BQ7IOgg/wZ+H96H5fR/+ctfghbPkc/DV3LQQL8Vwneg36qPsa9kpuYxxrEINHGM3w8RFEIOrF9wXuLYw37F7DshVmIiAg94LMN24h//+IfPJHmkdSAZIMRI2O1oINBhsgd2UrAswXUGYBEBQdw8h4Bp+YF2QE8IYn8cs/fee09ZFsFzG4lzcQxht4Lvbs0kkAYTWSYQHlu61wq2XdL4ujYwqYbrE2IyfLdRdkwAaCuYYM/nYMD+aGf92Wb4uq69r1sAgVUncIWdCURu1L+3nYZ3e4kJX5Td7Htaavv91Z33e0wh2R++2hy0wb6+H6CcqCsI1bCP0jkQAh1jDc5L2DNiwsjbp7u15403/vo59BM4l5CPCROnOJf0BCqA5Qkmj7Wdlh7fwHIGk8ewrjLHNZiYRf+I6wyTB7jucL5i8ksfa30Omv0LwHWIB4R7/T16fzw3rzf9XkyQmvjbTgghoYDCOSEk5BHfkfb9OrIT3oN4mEBQR8QPInZwk9laMGjQ3o4merCoQcQ7bnghaGgQdUUICY7oxGhJzGrZgzsuPc7ntmDei+8wwSAb7/Pe3hHA8xdRWIj2gyCFtkVHkZl4JzCDnyyA2K4FKvhhY4Dvjfdg2BRcMMA2I9K99/WHt2iD70c0IaJI/aFFtMcff1wlU/NGD/Y/+OADj6SewUxuBtPWor1HZCDqsrXiOcruK4kcjoMpDgJ/ghb8chEFCyEE4hYiUnXuDO2Xiz4FdeirHuHFbgrCEOH0JALEEUR6w48YSap91W9rfquvCMNgfqt+HUK/L/9iPUnTUt8Lz2FECmMiCXWlgeCqxUh9/uPaQYQyfOjvvPNOtYoD5dde8qRt4PqBGGeCuocoq73otWczvM5Nr2w8h2jt7xgg/wAmhCDEYyIJiWKx6gbXB8TkQEkI25tg26VA1zf8xnG+Q9BEPXzyyScqcKK153Ow5UX7CPFS34u2BXwv+gxcP2gXIQR73ysD73avuLhYfbfZ97TU9vurO+/VUMHgq81BW4gJAF+grYdojPch4h/tsDe+JhABhGUIzZjw9E5k2trzxht//Rwm0iGgo59A2SGom0mTsXJqy5Yt6qEnrkxwDsLjX4PVB7qfwP0GftM111yj+kydbFT/Fojh5gQY2lFMROLh7f+OB8RwfQ5qYd17JYS/7YQQEgoonBNCQkprbVLCTXl5uYrmww0yophMcLOJaAtEqmH5OMSKe++9Vy0d9gWiWLwjgnAzi+1YequB4PH111977IebSgweTbCUkhASHLBKaYtdCvC2bgmW6IRomf7xdAkViDqH4Id2BQNZX8k6YS9lRsqh3UIUmE68hrYMEYOw12hNsk+0T74iuFsLvh/CBQb//gQDiGwQa2DHgEhwf5jJ5IIlmLYW0bMQT2CHc/311/v8HHyGKWZoIEwjOhbRt9pO4LPPPlOigGm1EggcF4j7EDYQAYrfaf5WTPZCMPGVOBqWD/g9gSKpIUYiKhIiB/q/toLfg/PLTByISWFYQCDKOBAQZnBeQiT0F4mpzxfYJiDxHCaMvNHRieYx1cm+fS33hxiEJJC4jt5//32VQBGirb/jSVoGUdFI7miCCFkIrHoFCOoYdhcQ4UyRHfdXsPzwviYB7puQdBSJdb3v2xBViwmlYCKP24KvCPRg26VAoF3BNYprGCspEN0MS5O2ns+B0NckBFXTTqmtfQ/KCXEcxwvH3JfAiyh5HT2OtgG/F9dbsG2/P9qr70FSStQtBHxf4DW0TTj30Ea0BvTJaFsRnd+W80Yfc+/zzl8/h7rFGAViOdpSCM6Y4NDgHENfgjbeO6Ifk6boJ0zh3BuscoLYj0lmLZyb55SZwPTLL79s9n6s6IIFDt5rns96pZo5gWZub00CXEIIaS8onBNCSABwQ4noOywxxHJGb3CTjZtPDN4QAQShAEv8EUGDKCsM4PAcYGCCz8OyW9ys4qYQIjsEDNx8Ysk9olXwNwZ7ZjQMPBZx84vXcNMIYQsDekII0cCaA0IuBCl/UXuI8EQ0IMQZiFkY9EJI1xFnGChjwAx/VUwWQozEyhp4B0PM8LU8vT0544wzVP4ItLewTEF7B1EZS/gxqYjyom2EAIMoa4hj8KpFmwoxDtHUEHzbOrgOpq3Fdgz44VuLpf0QitDW47shjMDjG+09oi/xWYj8gy0B2nxEHeI5vJ4hPCD6+oYbblC2DN7euIHAb0ckIX6zGR2PiFtMvKJsvvosvXwfEare/sUaRA2ibKhHCERtFdVwnmFpP44PRCZEFGofZAgzgcDSfQhM6D8hnuO3QNyBuIR+9O2331ZiEM5H9L/w50fkIgRY7APfYExkIxIZIioEKdQz6mfGjBluH3aAukDUMq4fHCOcZ4i4Rxm0VYy/40laBtcKjjfaE1jewV5HT/DBJkiDawf2KrCegKgG0Rztjq88DwCfh2OqI1txPNBGoQ1DO4Lrwttnu73A+YDJJYjBuN/TYm97tEsQO++//351PsOWxYxiDuZ8Dha0ObgWscJiZ4Vz/FZEayMi3p81DiaeEIQCGxcIq+irILbr7w6m7e9I0Pbj/IS4jHYH4jmOIyZm0AdgNQrAscHkKQRf9KPoO9FGYQIUbZ6vdheg/UKkunc0frD9GR74DET2ax/zliYM8JmoN0yiwn5L29BAfMdqHljI+LpGYL+D68uc9PQGn4Xfg0kT2MGg3rASCNcDzinY9mj81Qmude/X4OmOc3zs2LHNtuNc8RfVTwghHUq4s5MSQjonnSWz+eGHH2716dNHZb33xcMPP6yyvK9cudJavHixdeSRR1opKSlWQkKCNXbsWOuNN95w7/vtt99au+66qxUfH6/egwzxYNu2bdbRRx+t3terVy/1mVdccYXVt29f93tLSkqss846y0pPT1eP888/3/rggw+aZa/3l6WeEBJZzJgxw0pMTGz1a1OmTLFGjBjRbPvq1atV+/D8889bZ555ppWcnGylpaWptqa6utpj382bN1vnnnuu1aNHDysmJsbq3bu3dfzxx1vff/+9ex+0T5dccknQv2fy5MnWtGnT3M/R/nm3X5qioiLrqquuUm1vdHS0KsfUqVOtDz/80GO/WbNmqc9FXeAxcuRI65prrrEKCwsDlgW/H/v6Iti2tq6uzrrvvvuswYMHqzLm5ORYJ510kio7qKmpsS6++GIrOzvbcjgcqpyar776ypo4caIVGxtrZWRkqO/bsWNHs2P15ptv+v0NOGaZmZnqs9euXeve/sADD7j7JF/89ttv6vVnn3024HFEedAnoa78gXMJn5WXl+d3nzVr1ljHHnusOt/QLx588MHWggULPPYJdC69/vrr1oQJE1S/ifKMGzfOuuWWW1T9atatW2dNnz5d1WVcXJw1bNgw69FHH3W//sknn6jjjddGjx5tffzxxx7nY2VlpXXeeedZQ4cOVd+Dz8F1NGfOHPdnBDqeoaayptL6af1P1rxN86yFWxeG7IHvw/fi+1sD7p+eeuopVfe4TnGtHHPMMdbSpUub7fvvf//bGjRokGp3Ro0apa49X7zzzjvWkCFDmrVdaKPGjBljJSUlWYcddpi1adOmVrVL/s5pfKZ5LWzYsEG1SWhDsT/a5GDbpUBtnwbnIvZ5+eWXm73W0vncUh9hcsQRR1innnpq0Nd1oNcuuOAC1WbievIG77n77rutq6++Wl1fOD6nn366u71sTdvv/Vtbwru9D9S+VlVVWbfddptq13EOdu/e3dp///2tl156yWM/HDuUKzU1VbUZ2P/CCy+01q9fH/B+HO1I//793X1xa/uzp59+2howYIDlcrnUZwQDzhHs+80337i3vfXWW2rb559/7vM9OL6of7S1geoc1x9+j9keXnbZZdZee+3VYrn8jVdwTuLc8AbtgS5PZx97EkLshwP/61hpnhDSFUE0AyJKEH2wM96JhBBCggOesYg8ROQmvEdNdPJOWCGYS/8JIZFFbX2tLMlbIhU1oU+SFx8dLyO6jxBXFBctdwYQMY+oZKx8wQqOtoJEk4gexkoarNbwBlHViNRGJDnp3CDXwLhx49TKH1+WPYGAlRdWkCB6H6uINIj6h6f/ihUrPJKQe8OxJyGko+BdDyGEEEJIBFNSUqKsoWDBAYHCV2I2QkjnAKI1xOu6+rqQf7czyknRvBMBoRsWILAhgSVha4GFCpJdwq98/fr1cumll3ZIOUnkAIsb5OCArRUsaFoDJl2QINcUzQFygsDKJ5BoTgghHQmFc0IIIYSQCAZ+ovAEzs3NVb678KgmhHRu8ZxR32RnwUTr008/rcTvtrBp0ybllQ5/c+SFoO8/0fmf2pJYGvcujz76aLPVDMgBBeGcEELCBa1aCCEdApfLEUIIIYQQQgjpaDj2JIR0FFEd9smEEEIIIYQQQgghhBBCSARC4ZwQQgghhBBCCCGEEEIIMaBwTgghhBBCCCGEEEIIIYQYUDgnhBBCCCGEEEIIIYQQQgwonBNCCCGEEEIIIYQQQgghBhTOCSGEEEIIIYQQQgghhBADCueEEEIIIYQQQgghhBBCiAGFc0IIaYG///3vkpSU1OrX7MJ+++0nDocj4OOss87q0DKsWbNGfY/L5ZIVK1Z4vPbbb7+p17766qtWfSbeh/ovLy8PuN+oUaPkoIMO8vv6NddcI4mJiVJaWuredsUVV6gy3XHHHT7f069fP3fd4Tfh+Zlnninr169vsdwvvPCCz2Ng9/OoLeCY/uMf/wh3MSIStjsdz7vvvquuvQMPPNDn62gXzWs0OztbpkyZIj/++GPQbZ6vxy+//CKdCfxWnK+bNm0Kd1GIzWjp3gMP9Ik7c39z+OGHt/n9N910k7r+gylna+9RdrY/fPXVV2X33XeX1NRUSUlJkeHDh8t5550n27Zta9X36rborbfecm/DPcull17qfv7ee+/Jk08+2arPJYQQQroKrnAXgBBCSMeCwVBxcbH7+cUXXywJCQnywAMPuLd17949JIehrq5O7rrrrp0aKJvC+W233aYGf/g9/jj11FPllltukS1btkhOTo7Ha/X19fKf//xHjjzySLdwjTJiG3jttdfUe31x/PHHK9G9pqZG5s6dKzNmzJBff/1V5s2bJ9HR0S2Wf+bMmWpArHE6ndLZgFCA8wziBOla2KndCSRM6fMUom/Pnj2b7TNgwAC1n2VZ8ueff6rrHBNxCxcuVK+1BISy/fff32MbBLDOBIQ5tMUQMH3VIem6eE8yTZw4US677DLVL2sGDhy4U+3MzvSdH3zwgZx22mnqvkSzefNmOfbYY5tduyNGjJBQ9Yf33Xef3HDDDXLVVVfJ7bffrtqfRYsWqbYIbVVWVtZOTxqmp6d7COeY0EM7TQghhBBPKJwTQkgno6qqSgm3UVFRPgd7iFyCSLznnnv6/YyKigqJj49v97JhEIqBH8Sn/v37SyjAAP1vf/ubEsMRSW7yzTffyMaNGz0G8V988YVs3bpViWOff/65zJ8/X3bddddmn4voU12H++yzj1RWVqrvweAT4kBLjB8/Xrp16yYdddwJCSV2bnd8AVH/o48+cl/nb7zxhlx99dXN9kN5dJlxXaPd2nvvvVV7cuONN7b4PYMHDw74m1sLBLTq6mqJjY1tt88kpKPwde736dOn3dqBnRGz165dq8RoiOS4Ts2JoI64dlvDo48+qla8/POf/3RvO+yww+Taa69VE/47y7hx43b6MwghhJCuAkfXhBDSjkAMnT59erPt119/vYrEQzSzXjb74osvyrnnnquijjMyMpRoU1tb6/G+DRs2qGgoCKwYSO67774qotlEL7lFhFLfvn3Vfvn5+a2KgkJ5ICIhihoC1wknnKBeKywsVBFIPXr0UEINft+sWbOafQbeu8cee6jvRhTpRRddJGVlZc32w+9FpFQwy5URlT569GiJi4uTXr16KVEa9adfO/vss9Xf+D6UH/XgC9TJXnvtJa+//nqz17AtMzNTDj30UPc2RJknJyer74AQqKNSgx2Irlu3TnYWDOhxLHBuwEbmkEMOURGuwR73QHWnwYTBGWecoSYA8N5hw4bJI4884n79pZdekkmTJqlzE5FpWBI/Z86cZufniSeeqD4D3wVRERFyANYNiELFeaCXuuMzSPvDdidwu+OLd955R0124TxF/YXjOsf3o91H34DrZ+zYsSoS1ATi2S677CIff/yxjBkzRrXDiJLV0bwHHHCAaiPQVmAC0NvGARMaN998s4qOx3t79+7tYc2Fz8CKG5QBn4MyvPzyyx6fgVU1EOwgeOIz0B8cccQRUlRUpPoPHZU7YcIE97VOSGssqdC3YGIK18ETTzyhXkPENazW8Dr6sVNOOUVFgweyatGfh/4S/RdWueD6+fTTT5t994cffihDhw71EM0DTVghWnzIkCHqGsD19NBDD3VYf1hQUKCuM1+Yk+P6PuD+++9XdYTfe9RRRzWrJ29Mqxa0B7gfXbx4ccjs+wghhJBIghHnhBASJN6iNvCO/Dn//POVEAJBQdtwQLCEEAEPbHNJMZbrwi/3v//9r4pqvvXWWyUmJkbuuece98AJAz8MAh977DH1efgXQgl8ws2lum+//bYa/EH4xHdAAGktF1xwgRLpIdzgMxDVePDBB6voayxjxqDslVdekWnTpqnyYkAL4Jt50kknKSEbA0MM2DDgRfkRxWmCAed1112nRBhYoECI8cWDDz6o9sOgExFXS5cudYu/qB+UAWLQnXfe6bY8CRSBCUHpkksuUVYL2l4BYhDKjoGutlaBkAVB7ZhjjlG/F4I6fgMGpS1FckPsBsFG0uO3mOcU6hwD1pKSEjWgxvc9/fTTagCO+sekyYIFCyQ3NzfgcW+p7sCOHTvcUfH4bNQJzqlVq1a5PxsTPBDWsYwe5wImGXQZIB4AvI5l44iOg1gAMVH7N8OLFUICJiJmz56ttmFShrQOtjs73+74AkI5xCNMqqF9gO3SsmXLlJDWntc5+gjzGOK61m0JJlnRfuEaxMQVJquOO+44ZZsAMVuDa+zyyy9XbR7aTDwgeKOdmDp1qop+hyCH1yGamfYY+Dxcf+hvED2bl5en2jjz9yCC/sILL1Rtzffff68mOFFu9Fng7rvvVm3RvffeKyNHjpTt27erCVSI8liNA6ET7evzzz+vfgfpeNBXAfR7eqIC5xke6AtMu7BA++JcxH1HW/ZtL9C/4BpEn4VJdUxmA0wC4bzFpA7OW/RnkydPliVLlqjcIv5A345rC9cM7jNw3uI6wLmuPxtgAipYf3SsVvv3v/+t+lIECfzwww8qIAKTzrh22rs/xGQerjm0Myijt82cCe7ZMHn+1FNPqfYP5UIUfTC5GADqCPX7xx9/uCcQw22jRQghhNgKixBCOoCKigpryZIl6l9vXvn9FeuwVw5r8XHVzKuavRfbgnkvvsOkrLqs2bZgmTFjhoXm0t8jMTHRvW9RUZGVkJBgPfnkk+5t77//vtpv+fLl6vnq1avV83322cfje2655Rb13vz8fPX81ltvtVJTU62tW7e696msrLT69OljXXvtte5tffv2tTIzM63S0tKgfs/kyZOtadOmuZ9/+eWXqjwXXnihx37PPfec5XK5rMWLF3ts32OPPawTTjhB/V1fX6++/5RTTvHY55NPPrEcDoe1aNEij9/85ptvWuXl5VZ2drZ10UUXqdd+/fVX9RrKAYqLi62kpCTrxhtv9PjMp556yoqPj7e2b9+unj///PPqfXl5eS3+ZuyD33LnnXe6t33wwQfq/d98841723//+1+1bebMmer566+/rp5/8cUXHp+H33zxxRdbNTU16vfgM3Jzc62pU6e2WBZdbu/HHXfcoV5/5JFHVN3h+tHs2LFDnWdXX311wOMebN3ddNNNVmxsrDouwVBXV6d+69ChQz0+G2V69NFHA1475vURVv58xbJmH9by45fm7Y7aFsx78R0mNWW+twcB252db3f8sXnzZsvpdFo33HCDer5x40YrKipKtcEmZ555pjVy5Eh17ldXV1vLli2z9t9/f/Xd27ZtC/gdus3zfhx44IHq9d9//109f/rppz3eN3HiRGvXXXf1KAP2++mnnzz223fffa299tpL1YUGbTV+/0cffaSez5o1S733tddes4IBn4XfesEFF6hyaNBfHHvssX7fp/uQuXPnBvU9ZOc5/PDD1aOwsNC97T//+Y/a5t0mH3fccWq7eS/x3nvvqW3333+/x76nnnqq2r527Vr3NvSH2Kb7qJ0B54n5nbqde+ONNwK+r7a21tqwYYPa99NPP/V7P6M/T18D5rX48ssvu7eh30QfqO87TMz7FbBy5Up1Xf3rX//y2O/666+3cnJyVP/Y3v3hwoULrUGDBrnbjf79+1uXX355sz4bbVFycrLHeYD7FfM+xvv36Pddcsklzdo6Qjrr2JMQQnYGWrUQQkJOWU2ZbCvb1uKjoLKg2XuxLZj34jtMMF7z3tYaEFWEBJDeD0SYmyCCCFGQzz33nHsbovDgge29HBhRzSaw5igvL3dbciCiD0vgYZVhRpIh4grfbYLIQzPKXO+Ph7dFhz8Q0WmC70d0J6KLzc9DFLr+/uXLl6soLkRtm/ugjIhO09FW3nWJ6E7UEexCvEEkV2lpqbKLMT8TXsTwPoUnaSDM9+hIT1jdILrftGvB34jcRFS/BtFgiOTHdwGdNNSXjQOSkiGqD0ujEYmN3+XLDsYf8FU2zyVEeYJvv/1WLS03EwjiHEC9f/fddwGPe7B1Bx93rFzwZ28DEKmOcxSRczqCERG5OOYaRJti+Toi3VauXCm2pq5MpGpby4+a5u2O2hbMe/EdHlh+tgcH252da3f0qg6zLQCI0MZrOrcBIlrxXlz/3sC+AOc+Im0Rjf7zzz+riO1gIzIR7Wpe52g39HUOtC2WBv0HkgybljOIkkWUqwb9BCLD8V7zN6KtxooUXU+4ztE+nXzyyX7LhwhVROYiYhW/E49nnnmm2XUOqxjYTeCz28NjmZBA9x/gk08+UStCsKIMEeawGQLmuekLtAG6Dwfo59CWIuJb89lnn6ltZv8fqK8GiFr37leRdHz9+vXt3h/iHgBtD2zwEO2OOkAkOyzYkBjdBPeJZqJx9O24Z0BbRQghhJCdh1YthJCQkxidKFmJTTYj/kiPS/e5LZj34jtMsOTYe1trwEBst9128+mR6Q3EdAz2YGkBj0rsAyHCG9NqBUCgBNqbEsvhf/rpJ48l1xrYZ/h6r7bYMG0EIIjoZFeBMD9Dfz8EHF/fry1nsI+vSQCNHlB6Ay9iCErw59Ze5eb3Al8JOQN9psa7vA1Bbg12LbCiwXEZNGiQ/O9//5PLLrvMvRwdfu4Qh04//XRll6KBvziEMgheph0MRDtYzmBpO94HO4O//OUvQYvn8Cv2lRwUQpb3sQDY5j1p4OuYBVN3sGrBwNwf+P2YaIA4COsXnEOwccByc72UXwuQWLqOB7zwISxiqT2WidsOZ6JIbMtth0Sn+94WzHvxHR44Gt7XbHtwsN3ZuXYH7aS2VgGrV69WIhomwnCuQmTGda8nyWAVAbHJFKnxGbB+gUD9+++/KxskXPtoRyBKtwRskHz1HbjO0VZB4PK+ptFmoVx6Usz7Osd7UR6UV3so+7vO0QcF8hyHlzEm3GAVBhsWTP5C+MO1rcH1jXMRPsiwxUG7AGsWvId+5uHhzTffVP+afRLaXZzHpiUcgNWR974QqtG3eVuQPfvss832PfDAA92TUh0BriNMUJtggga/BdZDsGDC/RLONdgNmX2QLyCIe1vK4Ln5Pti0wIotkOWLBu0Nrkl/ybxxvaGPbO/+EGWGFRMeAD7tOG633367h92S972k3taSzzkhhBBCgoPCOSEk5EwfPV092sKDhzzYpvclRCe0+TtbC7yjIUAgohoRzRAcvaMKgXcSN3iJA50QCoIKBnZ33HFHs/d6+3mb4gWiJ82I9EDe3/4+Q38/opv0QNoXWvR5/PHHPcQmsyy+wCAZXvDwKD/ssMN8fiYGhqaft6Ylb2HvaHzN0UcfrYQoCNsQrRHRqSNOtWcyvFbxe339ZkR+mQNgiEdaEEPUGiK94UF/5ZVX+qyLYMHvR2S3Nzg/vEU2X8csmLpDBCu8WP0Bb1RE52HSB3WlgXe/jvrT5yrOc3i/ImktjiciZlF+7SVvG/pPb3i0hfFta3fElSCy/8cSCtjuNG93II7Bh9vcjkhQ3UYg6a03ENXN6xftt77OsR3iGSJPca3DS7it4FqFFzNEcLMcuM5xXaelpfm9zvEatsH/Ge2aN1rgw3UO8Qyiny+BG0IirnFMjmESUeMdUY4+BNHmeKD+cM3jb1zjmGgkoQfnpTcQgX0JwR21b3vh69yEbzeiqJEDRgv25iTYzoDrAZPdiA4P9lpFGbHiy5fHu86L0NH9ISY60B9jNVige0m9zV9yUUIIIYS0DgrnhBDSASDqHIMmRP1g4OQrWScGhma0IIRbRF7p5HdYBoxIMVh2tCbZJwZ2viIcWwu+H4NLiE3+BHAkgoOQiqSbiEBsDZdeeqlKuomHtwCIeoBw6y+iFOgBrHf0mb/fjjpEBBuiRxG5jXrWdQ1g04BoVFjreAOrAwhqgSLHICQhIhMRZohmbysQ4XEumIkKIa5huTgSuAYi2LrDsYVogORlvhK0wtYFmCIBolKxcgGTQt5A2JgwYYI6599//30lrkEowPtN4ZJ0LGx3PDGvb/M6hwiGySVTnAZInouo0YceeqhZ1K4GbQCSaWIfWCj4EhqDQVtEIHLYvK7xfNy4cQHbfLyGax0CGq65QNc5VvZAfEQ/5A2uTYjk5nWO1Sa4hv2B1Tpo4/71r3+5BTx/bTEhbQV9EFZkmKK6L8u0tgArJyTD9J609wci7vUKjiOOOKLF/dujP8QEmvdKE9QJotu9++Avv/zSIyE9ko/m5+e3agLfOyKfEEIIIU1QOCeEkA4AUXiIRsQSX38R26tWrVI2JRBl58+fr6w+IKTr6ENEZGOgiCXSEGggcGKwBysBCNm+lui3J2eccYYSR+Cj/de//lX558I+APYtiMxGeTGoRbQiIrcRwY1lxBB1EBmGCG0ILHifL2AJgN+Fpf8mELOwFBmWCBCA8f0QsSDOQ5B+++23lTisPcCfeOIJFXVpTjr4A+VExDnKh/Jr4LX+9ddfy80336y+z9f7YNViDk59RaUhahO/GYKS6VHeGnBOQJRDXWLgDWHurrvuUhF/iGYPRLB1h3PnpZdeUt7st9xyixrUYx94x0Jow3J4rArAZAiWyaN+ZsyYIb169XJ/F+oCEXA41yHw45xAFC7KoK1iUAfwgn3kkUeUfRGOuZ4MIO0P252W2x0I58g54StSu7i4WFlDYJIK53agSTJ4rr/wwgty4YUXtulYYTUPRHi08xDEcF1gohQTVMFMvGHCEV7GEMTRh6DfwDUP72a0Ibj2IZzD5uGcc85R/Q2ENAhqmJjDBAHaMgh8mDDAChq0Mfgb280oVtTV+PHj3YI+IvkxmYfvB6hrtDOIttXRye0xeUu6Lri+Hn74YdWnYhIYq6BefvnldvlsnL/oj7xXcPkD5zf6QrSvsGfDdYTVIugvIVq/99577d4f4l4GIj0+E5Hj6IOxsg/3lLhvMklOTlaTAOircY+Ge8/dd989YBvmDcqG6xf3R8jHg1UrgXKgEEIIIV2KnUotSgghXSCz+YwZM6zExMRWvzZlyhRrxIgRzbavXr0aptvW888/b5155plWcnKylZaWZl1xxRVWdXW1x76bN2+2zj33XKtHjx5WTEyM1bt3b+v444+3vv/+e/c+ffv2tS655JKgf8/kyZOtadOmuZ9/+eWXqjxz585ttm9RUZF11VVXWX369LGio6NVOaZOnWp9+OGHHvvNmjVLfS7qAo+RI0da11xzjVVYWOjxm998802P9+Xn51spKSnqNZTD5PXXX7cmTJhgxcfHq33GjRtn3XLLLVZNTY17n7///e+qTqKiolQ9tATqNzMz03I4HNbatWvd2x944AFVhpUrV/p832+//aZef/bZZwPW+Y4dO1RZcVz9geOOz8rLy/O7z5o1a6xjjz1WnRsJCQnWwQcfbC1YsMBjn0DHPZi6W7dunTV9+nQrIyPDiouLs4YNG2Y9+uij7tc/+eQTdRzx2ujRo62PP/7Y49yprKy0zjvvPGvo0KHqe/A5OOfnzJnj/gx838UXX2xlZ2erOsf7SXCw3dn5dsebX375RV17//73v/22D927d7dOP/109RzXMT7TF5MmTbIGDhxo1dbW+nzdX5tnUl5ebl155ZVWTk6Oat9xnb399tse+wQqA9ps1Etqaqq6BgcPHmxdeOGF1vr16937oA++4YYb3HWJ9vKcc85xv75ixQrrgAMOUO1Mbm6udf/99zc79+677z5rt912U9+D7bvuuqv12muveZTl6aeftgYMGGC5XC71uwnxBc4NnGPBtHP33nuvOl91H7h8+fJm7/e+n/H3eTh38RpAf4jP9oeva7e+vt567LHHrF122UVdq+jvJk6caD344IMd0h8+8cQT1qGHHmr16tVLfV/Pnj3V89mzZ/u8D7j77rtVO4n++ogjjrA2btwY8Pd43z/gXu/kk09W90fYN9A9DCF2pTONPQkh9sKB/4VbvCeEdD6w5BOJ2OCp3Nal7JEMIhcRnYvIxGuuucbjNZ28E0vyjz/++LCVkRDSuWC7Qwgh/kFuD9ybLV68WEaMGBHxVYWo8MMPP1xFoxPS1enqY09CSMdBqxZCCGlH4A+7ZMkSZesBGxMsmSeEkI6E7Q4hhLQMbO4YM0YIIYSQ1kDhnBBC2pF58+bJ/vvvL7m5uSpRZLAemoQQwnaHEEIIIYQQQuwDrVoIIR0Cl8sRQgghhBBCCOloOPYkhHQUUR32yYQQQgghhBBCCCGEEEJIBELhnBBCCCGEEEIIIYQQQggxoHBOCCGEEEIIIYQQQgghhBhQOCeEEEIIIYQQQgghhBBCDCicE0IIIYQQQgghhBBCCCEGFM4JIYQQQgghhBBCCCGEEAMK54QQQgghhBBCCCGEEEIIhXNCCAmev//975KUlNTq1+zCfvvtJw6HI+DjrLPOCmmZjjjiCBk8eLDf1x977DFVrlWrVrm3PfTQQ2rbueee2+LvjIqKkt69e8uxxx4rS5YsabE8X331ld+62b59u3QmfvvtN3XelpeXh7sohBBCCCGEEEKIbXGFuwCEEEI6lieffFKKi4vdzy+++GJJSEiQBx54wL2te/fuIT0Mp556qnrMnTtXJkyY0Oz1119/Xfbcc08ZOHCge9urr76q/n3nnXfUb4qNjW32vr333lv9rrq6OiWY33zzzXLQQQfJ4sWLJT09vcVyPf/88zJs2DCPbWlpadLZhPPbbrtNLr30UnUeEEIIIYQQQgghpDkUzgkhpJNRVVUl0dHRKuoajBgxwuP1lJQUFSUPYdofFRUVEh8f32FlPOqoo1QZXnvttWbC+Zo1a+THH3+URx991L1t+fLlMm/ePCWCf/755/LRRx+paHJvIHLr3wURPTExUaZPny4zZ86UU045pcVy7bLLLrLbbrtJewEBv76+Xh0PQgghhBBCCCGERA70OCeEkHZk/PjxSqj15vrrr5eePXsqIRXCMCxAXnzxRWU7kpqaKhkZGXL11VdLbW2tx/s2bNggp512mnTr1k0J2fvuu68SkE369eunoofvu+8+6du3r9ovPz8/6DJrmxKI0ccff7wS1k844QT1WmFhoYpQ79Gjh4rwxu+bNWtWs8/Ae/fYYw/13Yhev+iii6SsrMzvdyLSGeL5f//7XyUse0ebO51OOemkk9zbILCjjM8884xkZ2e7o89bYty4cerfdevWyc6COj3nnHPcx2KvvfaSb775ppldzOGHH66O7dChQ1Wd/f7770HXEer7sssuUzYzeG///v3lxhtvdL+Ozzj44IMlKytLHSd8HiYFvD/j/PPPl169eklcXJzk5ubKySefrF574YUX5Oyzz1Z/owyoU5w/hBBCCCGEEEII8YQR54QQEiTeojbwFn0hWEIALyoqUoI4gFj+8ssvy5lnnqkEYc1NN90kU6ZMUeLx/Pnz5dZbb5WYmBi555571OsFBQUyadIkFZkNz298Hv494IADZMWKFUo81bz99tvKM/yRRx5R34FI69ZywQUXKJH+3XffVZ9RXV2tRNqtW7fKXXfdpYTYV155RaZNm6bKO2rUKPW+t956S4ncEGRhAbJ582a54YYbVPnfeOMNv98HqxYI4BDu8ZtMkVyLw+a2ffbZRwnJJ554ohLQzTr2x9q1a9W/eF8w4FiZxxlR+3hg+2GHHSZ//vmn3HvvvUq8R0Q8yvnDDz+oCQXNL7/8oiZHbr/9dmUPA+E6mDrCSgHUA947Y8YMVb/r16+X7777zv3Zq1evVv7wf/3rX1W5PvnkE5k6darMnj1bifYA5x+24zyCKI7vwnOAYwf7mjvvvFMJ7qg/X5Y3hBBCCCGEEEJIl8cihJAOoKKiwlqyZIn6txmvvGJZhx3W8uOqq5q/F9uCeS++w6SsrPm2IJkxY4YlIn4fiYmJ7n2LioqshIQE68knn3Rve//999V+y5cvV89Xr16tnu+zzz4e33PLLbeo9+bn56vnt956q5Wammpt3brVvU9lZaXVp08f69prr3Vv69u3r5WZmWmVlpYG9XsmT55sTZs2zf38yy+/VOW58MILPfZ77rnnLJfLZS1evNhj+x577GGdcMIJ6u/6+nr1/aeccorHPp988onlcDisRYsW+S1HTU2N1b17d+u8885zb1u4cKEqy0svveTeNmfOHLXt6aefVs9//PFH9fzZZ59t9rumTp2qPreqqsqaP3++NXr0aGvcuHGq3gKh68D7ce6556rX//e//6nnM2fOdL+nurpaHYtjjz3WowzR0dHWunXr3NuCraNnnnlGfccPP/xgBUNdXZ36rVOmTPH47JEjR1pXX3213/c9//zz6nvy8vKC+h5CCCGEEEIiduxJCCE7Aa1aCCGhB/YU27a1/CgoaP5ebAvmvd42IZbVfFsrgL0GEll6PxBhbgL7DEQWP/fccx4JJxEtjYhwk2OOOcbjOWxSysvLZeHCheo5LFH2339/ZeOCKGg8EAk+efJk9d0miDY2o8z1/nggWjoYEI1sgu9H1POQIUM8Pg9R1vr74T2OqG5EgZv7oIyIiEb0tRnJrR/A5XIpSxhEyyO6Xdu0wMbFrBtEm8MjXNvHwMN8wIABPu1aPv74Y7Uvoqh33XVX2bJli7z33ntBR1W/9NJLHsf3lltuUdu//fZbdWwPOeQQ9774HvismxHhYPTo0SrKXBNsHX3xxRcyfPhwmThxot/ywboHKxcQ/Y/6QxlwnPAdGvxuWLIgSeqiRYuC+t2EEEIIIYQQQgjxhFYthJDQA4HXsOHwS3q6723BvNfbqsThaL6tFUDg9JU08sMPP2y2DWI6/K8XLFigvMGxD6xFvDGtSADsPwCsNcD27dvlp59+8plYcuDAgT7fC2D1YVqTwPcc21rC/Az9/b/++qvP79eWM9jH1ySABlYjurzaNkVbjsBGBHYtTz75pLINOfLII5Vwjn9hT6OtcGBlgokBHAP4dwP4o8OWZtOmTco7XgNrm4ceekhqamrk+++/l7/97W8qKSiEb50sNRAQrn0dZ1iqeB8vXWfefvK+6jGYOtqxY4fHb/EGdYG6gUUNbGAGDRqkJktg8WN6uMPOB5Mt//znP+Xaa69VIj580uGpTgghhBBCCCGEkOCgcE4ICT1InukjgWZQPPhg296XkND272wliBgeOXKkijrv06ePStCoo6VNtiEy3gBe4gBiO4D4eeihh8odd9zR7L3eEdRI8qiB+GpGpAcbbW1+hv5+RE8/++yzft+DfcDjjz+uElV6o4XgDz74QHl4e2/HBAMEdAjmEKYhqEMQ18C7G1HjeMAv3BuI6vD01sCzWwvfOA4Qy6+55hp58803PZKNthb8Tu/jpY+ZroNA9RhMHWVmZqrJFn+sXLlSTWQggh4TB5qKigqP/VAHDz/8sHpg9QLqEwled9llF7XygRBCCCGEEEIIIS1D4ZwQQjoARJ0jASPEYAi2vpJ1IgnnVVdd5X6OBJKwKdFJNw866CCVjBNR0K1J9okEo76iplsLvh/WJxB2/UVCDxs2THr37q2SZl5yySV+P0v/Jm8gMiMiHOIufjvEY0wWmDYt+O3/+9//PBKrgiuvvFLZtZjCuTeXXXaZEqzvvvvunRLOEcl+//33K1sUJHQFsFvBMcRrgQi2jlDf//nPf+Tnn3/2KbBrgRzHV4MofkTWw07HX70jAh+TH0uXLlXCuX5/ZWVlkL+eEEIIIYQQQgjpelA4J4SQDuD000+X66+/Xtl0+IvYXrVqlZx99tly8skny/z585W4CyFdR1ZDEIYwDC/sK664QkWv5+XlKWEVQrYpuncEZ5xxhvzrX/9SNil//etflTgLqxREPcOTHOWF8P3ggw8qy5WysjLlkw6hG4LuRx99JP/4xz/8iroavBefBS/4v/zlL25rGAi777zzjhx33HFy4IEHNnvfOeeco+pl2bJlMnToUJ+fjc+66aab1EQG7GBMUb414Hftvvvuctppp8k999yj7FhgiQJbHXx+IIKtI5wzsK3B6zNmzFAR4hs3bpRvvvlGWf1oAf6GG25QnvGlpaVqP/idm+y9997KFgbvx2QDfNshlutoc0zEgCeeeEKOPvpoj8kaQgghhBBCCCGENMDkoIQQ0gHAngOC94gRI1QyS1/cddddYlmWsnG57777VDQytmkQfQ2P87FjxyoRHpHOEMvhV+4rIrm9gcULrFIOP/xwVS58Pyw/kMzSjLJG+RGZ/scff6jocfhww18bFizeft++gMALSxjUBcRlDURl+HlDwPcF9oUw7itJqAmSacLzHYJ3W4EAjd8IURu+4RDzi4uLVQT6+PHjW3x/MHWE+kaCUCQRhZgOkR/CuPZWx+uYSMC/+Dx4m8PDHeeZt3AOsRz7IOEs7G9glaMF83Hjxsnf//53tZoBVjlHHHFEm+uFEEIIIYQQQgjprDgsKBWEENLOIFoYgh0ES3h8dzUgqiISGAIlPLZNdPJO+G5D2CSEEEIIIYQQ0ja6+tiTENJx0KqFEELakZKSElmyZImy3IBFB6xYCCGEEEIIIYQQQkhkQeGcEELakXnz5sn+++8vubm58uKLLyrLFkIIIYQQQgghhBASWVA4J4SQdgSJNFtywIKvNV2yCCGEEEIIIYQQQuwLk4MSQgghhBBCCCGEEEIIIQYUzgkhHQojqwkhhBBCCCGEcMxJCIk0KJwTQjqE6Oho9W95eTlrmBBCCCGEEEJIh6DHnHoMSggh7QU9zgkhHYLT6ZS0tDTZtm2bep6QkCAOh4O1TQghhBBCCCGkXVY3QzTHmBNjT4xBCSGkPXFY9FEghHQQaF62bNkihYWFrGNCCCGEEEIIIe0ORPOcnBwGahFC2h0K54SQDqeurk5qampY04QQQgghhBBC2g3YszDSnBDSUVA4J4QQQgghhBBCCCGEEEIMmByUEEIIIYQQQgghhBBCCDGgcE4IIYQQQgghhBBCCCGEGFA4J4QQQgghhBBCCCGEEEIMKJwTQgghhBBCCCGEEEIIIQYUzgkhhBBCCCGEEEIIIYQQaeL/ASiNxAlGkrJDAAAAAElFTkSuQmCC"
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "execution_count": 7
+ },
+ {
+ "cell_type": "markdown",
+ "id": "var-ex-params-md",
+ "metadata": {},
+ "source": [
+ "## Inspecting the Time-Varying Coefficients\n",
+ "\n",
+ "`forecast(type=\"parameters\")` returns every VAR coefficient as a column. A VAR(p) is conventionally written as\n",
+ "\n",
+ "$$y_t = A_1 y_{t-1} + A_2 y_{t-2} + \\dots + A_p y_{t-p}$$\n",
+ "\n",
+ "so with `var_p = 4` there are four coefficient matrices `A1, ..., A4` (one per lag), each of size k x k, where k is the number of series in the panel (here `k = 10`). Each DataFrame row holds the equation of its `series_id`, i.e. one row of every lag matrix, which is why a row carries `k * var_p = 40` coefficient columns. Reading a cell:\n",
+ "\n",
+ "- the row's `series_id` selects **which equation** (the matrix row: the series being forecast),\n",
+ "- `A{j}` selects **which lag** (the j-th matrix, j months back),\n",
+ "- `(Series m)` selects **which source series** the coefficient multiplies (the matrix column).\n",
+ "\n",
+ "Below we look at the equation of Series 6: `A1(Series 6)` and `A2(Series 6)` are its own-lag effects (the multivariate analog of the `AR(1), ..., AR(p)` columns of `HyperTreeAR`), while `A1(Series 5)` is the weight that Series 5's previous month receives in Series 6's forecast. In the simulation that is exactly the lead/lag link (Series 6 is driven by Series 5), and `A1(Series 1)` is an unrelated series for comparison. Stacking the k rows that share a date recovers the complete time-varying matrices for that time step. Note that the coefficients live in the scaled space (per-series scaling)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "id": "var-ex-params",
+ "metadata": {
+ "execution": {
+ "iopub.execute_input": "2026-06-11T09:51:29.947717Z",
+ "iopub.status.busy": "2026-06-11T09:51:29.947717Z",
+ "iopub.status.idle": "2026-06-11T09:51:29.979925Z",
+ "shell.execute_reply": "2026-06-11T09:51:29.978920Z"
+ },
+ "ExecuteTime": {
+ "end_time": "2026-06-11T13:43:54.020521800Z",
+ "start_time": "2026-06-11T13:43:53.958499900Z"
+ }
+ },
+ "source": [
+ "params = htnet_var.forecast(test_data=test, type=\"parameters\")\n",
+ "params[params[\"series_id\"] == \"Series 6\"][[\"date\", \"A1(Series 6)\", \"A1(Series 5)\", \"A2(Series 6)\", \"A1(Series 1)\"]].head()"
+ ],
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ " date A1(Series 6) A1(Series 5) A2(Series 6) A1(Series 1)\n",
+ "60 2020-01-01 0.045522 0.026507 0.056337 0.020276\n",
+ "61 2020-02-01 0.043268 0.027609 0.052135 0.022095\n",
+ "62 2020-03-01 0.037825 0.030270 0.041986 0.026486\n",
+ "63 2020-04-01 0.030708 0.033750 0.028717 0.032228\n",
+ "64 2020-05-01 0.023783 0.037752 0.015588 0.037502"
+ ],
+ "text/html": [
+ "
"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "execution_count": 9
+ },
+ {
+ "cell_type": "markdown",
+ "id": "var-ex-notes-md",
+ "metadata": {},
+ "source": "## Practical Notes\n\n- **Aligned panel required**: all series must have the same length and identical dates, because the VAR design vector stacks the lags of every series at the same time points.\n- **Series identity feature**: include one (like `series_num` above) so the global GBDT can produce equation-specific coefficients, and cast it to pandas `category` dtype for true categorical splits.\n- **Scaling**: per-series mean scaling is on by default (`scaling=\"mean\"`). VAR coefficients multiply *other* series' values, so unscaled heterogeneous panels force the model to learn scale conversions while the loss is dominated by the largest series. `\"standard\"` and `None` are also available.\n- **Which model**: prefer `HyperTreeNetVAR`. Use the direct `HyperTreeVAR` for small panels where coefficient-level interpretability matters, or `type=\"factor\"` when the panel is large and cross-series dynamics plausibly run through a common factor rather than dense pairwise links.\n- **When to prefer the univariate AR instead**: if your series do not genuinely lead or lag each other, a VAR pays a large estimation-variance bill (`k * p` coefficients per equation) for little information gain."
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/quickstart_forecast.png b/examples/quickstart_forecast.png
index 5dda77c..0e53b44 100644
Binary files a/examples/quickstart_forecast.png and b/examples/quickstart_forecast.png differ
diff --git a/examples/utils.py b/examples/utils.py
index 54e3074..9bc27cf 100644
--- a/examples/utils.py
+++ b/examples/utils.py
@@ -1,5 +1,8 @@
import numpy as np
import pandas as pd
+import re
+import matplotlib.pyplot as plt
+import matplotlib.dates as mdates
def calculate_metrics(
@@ -148,19 +151,242 @@ def load_air_passengers() -> pd.DataFrame:
def plot_example_forecast(
actuals: pd.DataFrame,
forecasts: pd.DataFrame,
+ levels: list = None,
) -> None:
- """Plot actuals vs. Hyper-Tree-AR forecast for the air passengers example.
+ """Plot actuals vs. Hyper-Tree-AR forecast, shading any conformal intervals.
+
+ If ``forecasts`` contains conformal interval columns named
+ ``-lo-`` / ``-hi-`` (produced by
+ ``forecast(..., level=[...])``), each band is shaded. Pass ``levels`` to
+ restrict which are drawn; by default all detected levels are shown.
+ """
+ try:
+ import matplotlib.pyplot as plt
+ except ImportError as e:
+ raise ImportError(
+ "matplotlib is required for plotting. "
+ "Install it with: pip install hypertrees[plot]"
+ ) from e
+
+ plt.figure(figsize=(12, 5))
+ plt.plot(actuals["date"], actuals["value"], label="Actual",
+ color="#2E86AB", linestyle="-", linewidth=2, alpha=0.8)
+ plt.plot(forecasts["date"], forecasts["fcst"], label="Hyper-Tree-AR Forecast",
+ color="green", linestyle="--", linewidth=2, alpha=0.8)
+
+ # Detect and shade conformal interval bands, if present.
+ model = forecasts["model"].iloc[0] if "model" in forecasts.columns else None
+ if model is not None:
+ detected = sorted(
+ int(m.group(1))
+ for c in forecasts.columns
+ for m in [re.fullmatch(rf"{re.escape(model)}-lo-(\d+)", c)]
+ if m
+ )
+ if levels is not None:
+ detected = [lv for lv in detected if lv in set(levels)]
+ # Shade widest band first so narrower bands sit on top.
+ alphas = [0.15, 0.28, 0.40]
+ shade = {lv: a for lv, a in zip(sorted(detected, reverse=True), alphas)}
+ for lv in sorted(detected, reverse=True):
+ plt.fill_between(
+ forecasts["date"],
+ forecasts[f"{model}-lo-{lv}"],
+ forecasts[f"{model}-hi-{lv}"],
+ color="green", alpha=shade.get(lv, 0.2), label=f"{lv}% interval",
+ )
+
+ plt.axvline(x=forecasts["date"].min(), color="black", linestyle=":", alpha=0.7, label="Train/Test Split")
+ plt.title("Forecasting Results - Air Passengers Dataset", fontsize=16)
+ plt.xlabel("Date", fontsize=12)
+ plt.ylabel("Number of Passengers", fontsize=12)
+ plt.legend(fontsize=11)
+ plt.grid(True, alpha=0.3)
+ plt.tight_layout()
+ plt.show()
+
+
+def simulate_var_panel(
+ k: int = 10,
+ n_train: int = 240,
+ fcst_h: int = 12,
+ seed: int = 42,
+) -> tuple:
+ """Simulate an aligned panel from a stable VAR(1) with a lead/lag chain.
+
+ Each series follows its own past and its neighbor's previous month
+ (``A[i, i-1] = 0.3``), so the panel carries genuine cross-series
+ structure that a vector autoregression can exploit. A common monthly
+ seasonal profile and strongly heterogeneous per-series scales are
+ applied on top. Columns: ``series_id``, ``date``, ``value`` plus the
+ features ``month``, ``quarter``, and ``series_num`` (pandas ``category``
+ dtype, so LightGBM applies true categorical splits).
Parameters
----------
- actuals : pd.DataFrame
- Full series with ``date`` and ``value`` columns.
- forecasts : pd.DataFrame
- Forecasts with ``date`` and ``fcst`` columns.
+ k : int, default 10
+ Number of series.
+ n_train : int, default 240
+ Training observations per series.
+ fcst_h : int, default 12
+ Test observations per series (the forecast horizon).
+ seed : int, default 42
+ Seed for the random number generator.
Returns
-------
- None
+ tuple
+ ``(df, train, test)`` where ``df`` is the full panel and
+ ``train`` / ``test`` are the per-series head/tail splits.
+ """
+ rng = np.random.RandomState(seed)
+ dates = pd.date_range("2000-01-01", periods=n_train + fcst_h, freq="MS")
+
+ # Stable VAR(1) with a lead/lag chain
+ A = 0.5 * np.eye(k)
+ for i in range(1, k):
+ A[i, i - 1] = 0.3
+
+ const = 10.0 * (np.eye(k) - A).sum(axis=1)
+ Y = np.full((len(dates), k), 10.0)
+ for t in range(1, len(dates)):
+ Y[t] = const + A @ Y[t - 1] + 0.5 * rng.randn(k)
+
+ # Common monthly seasonality and heterogeneous series scales
+ season = 1.0 + 0.25 * np.sin(2 * np.pi * np.arange(1, 13) / 12)
+ scales = rng.uniform(20, 2000, size=k)
+ Y *= season[dates.month - 1][:, None] * scales[None, :] / 10.0
+
+ df = pd.concat(
+ [
+ pd.DataFrame({
+ "series_id": f"Series {i + 1}",
+ "date": dates,
+ "value": Y[:, i],
+ "month": dates.month,
+ "quarter": dates.quarter,
+ "series_num": i,
+ })
+ for i in range(k)
+ ],
+ ignore_index=True,
+ )
+ df["series_num"] = df["series_num"].astype("category")
+
+ train = df.groupby("series_id", sort=False).head(n_train).reset_index(drop=True)
+ test = df.groupby("series_id", sort=False).tail(fcst_h).reset_index(drop=True)
+
+ return df, train, test
+
+
+def simulate_intermittent_panel(
+ k: int = 20,
+ n_train: int = 156,
+ fcst_h: int = 12,
+ seed: int = 42,
+) -> tuple:
+ """Simulate an aligned panel of intermittent (zero-inflated) demand series.
+
+ Each SKU's weekly demand is the product of two feature-driven components,
+ which is exactly the structure the TSB method targets:
+
+ * a demand *probability* (occurrence) that rises during promotions and
+ follows a mild monthly seasonal cycle, and
+ * a demand *size* (units when demand occurs) that is larger during
+ promotions.
+
+ Both components depend on a binary ``promo`` feature and on ``month``, so a
+ Hyper-Tree-TSB whose smoothing rates are functions of features has real
+ structure to exploit. Columns: ``series_id``, ``date``, ``value`` plus the
+ features ``month``, ``promo``, and ``series_num`` (pandas ``category``
+ dtype, so LightGBM applies true categorical splits). All series share the
+ same length, as TSB requires.
+
+ Parameters
+ ----------
+ k : int, default 20
+ Number of SKUs (series).
+ n_train : int, default 156
+ Training observations per series (weeks).
+ fcst_h : int, default 12
+ Test observations per series (the forecast horizon).
+ seed : int, default 42
+ Seed for the random number generator.
+
+ Returns
+ -------
+ tuple
+ ``(df, train, test)`` where ``df`` is the full panel and
+ ``train`` / ``test`` are the per-series head/tail splits.
+ """
+ rng = np.random.RandomState(seed)
+ T = n_train + fcst_h
+ dates = pd.date_range("2015-01-05", periods=T, freq="W-MON")
+ month = dates.month.to_numpy()
+
+ # Per-SKU baselines for occurrence probability (logit) and size (log),
+ # plus a shared monthly seasonal effect on the occurrence probability.
+ base_logit = rng.uniform(-2.2, -0.4, size=k) # mostly low occurrence
+ base_logsize = rng.uniform(0.5, 2.0, size=k) # heterogeneous sizes
+ season_logit = 0.5 * np.sin(2 * np.pi * month / 12)
+
+ frames = []
+ for i in range(k):
+ # Each SKU has its own promotion calendar: short recurring bursts.
+ promo = np.zeros(T, dtype=int)
+ start = rng.randint(2, 8)
+ while start < T:
+ promo[start:start + rng.randint(1, 3)] = 1
+ start += rng.randint(6, 14)
+
+ # Probability of demand and mean demand size, both functions of features.
+ p = 1.0 / (1.0 + np.exp(-(base_logit[i] + season_logit + 1.3 * promo)))
+ mu = np.exp(base_logsize[i] + 0.6 * promo)
+
+ occurrence = (rng.uniform(size=T) < p).astype(float)
+ size = rng.poisson(mu) + 1.0 # at least one unit when demand occurs
+ value = occurrence * size
+
+ frames.append(pd.DataFrame({
+ "series_id": f"SKU {i + 1}",
+ "date": dates,
+ "value": value.astype(float),
+ "month": month,
+ "promo": promo,
+ "series_num": i,
+ }))
+
+ df = pd.concat(frames, ignore_index=True)
+ df["series_num"] = df["series_num"].astype("category")
+
+ train = df.groupby("series_id", sort=False).head(n_train).reset_index(drop=True)
+ test = df.groupby("series_id", sort=False).tail(fcst_h).reset_index(drop=True)
+
+ return df, train, test
+
+
+def plot_forecasts(
+ datasets: list,
+ split_date=None,
+ title: str = "Forecasting Results",
+ xlabel: str = "Date",
+ ylabel: str = "Value",
+ level: int = None,
+) -> None:
+ """Plot actuals and model forecasts on a single axis.
+
+ Parameters
+ ----------
+ datasets : list of tuple
+ Entries ``(data, x_col, y_col, label, color, style)``, one per line.
+ split_date : optional
+ If given, a dotted vertical line marks the train/test split.
+ title, xlabel, ylabel : str
+ Axis annotations.
+ level : int, optional
+ If given, every forecast DataFrame carrying
+ ``-lo-`` / ``-hi-`` columns gets its
+ conformal interval shaded in the line's color.
"""
try:
import matplotlib.pyplot as plt
@@ -171,19 +397,303 @@ def plot_example_forecast(
) from e
plt.figure(figsize=(12, 5))
- datasets = [
- (actuals, "date", "value", "Actual", "#2E86AB", "-"),
- (forecasts, "date", "fcst", "Hyper-Tree-AR Forecast", "green", "--"),
- ]
for data, x_col, y_col, label, color, style in datasets:
plt.plot(data[x_col], data[y_col], label=label, color=color,
linestyle=style, linewidth=2, alpha=0.8)
- plt.axvline(x=forecasts["date"].min(), color="black", linestyle=":", alpha=0.7,
- label="Train/Test Split")
- plt.title("Forecasting Results - Air Passengers Dataset", fontsize=16)
- plt.xlabel("Date", fontsize=12)
- plt.ylabel("Number of Passengers", fontsize=12)
+ if level is not None and "model" in data.columns:
+ model = data["model"].iloc[0]
+ lo, hi = f"{model}-lo-{level}", f"{model}-hi-{level}"
+ if lo in data.columns and hi in data.columns:
+ plt.fill_between(data[x_col], data[lo], data[hi], color=color,
+ alpha=0.15, label=f"{level}% Interval ({model})")
+ if split_date is not None:
+ plt.axvline(x=split_date, color="black", linestyle=":", alpha=0.7, label="Train/Test Split")
+
+ plt.title(title, fontsize=16)
+ plt.xlabel(xlabel, fontsize=12)
+ plt.ylabel(ylabel, fontsize=12)
plt.legend(fontsize=11)
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
+
+
+def plot_panel_forecasts(
+ datasets: list,
+ series_ids: list,
+ split_date=None,
+ interval_fcst: pd.DataFrame = None,
+ level: int = 80,
+ history: int = 100,
+ title: str = "Forecasting Results - Simulated VAR Panel",
+) -> None:
+ """Plot actuals and model forecasts for several panel series side by side.
+
+ One subplot per entry of ``series_ids`` with a single global legend
+ below the panels. Each DataFrame in ``datasets`` must carry a
+ ``series_id`` column.
+
+ Parameters
+ ----------
+ datasets : list of tuple
+ Entries ``(data, x_col, y_col, label, color, style)``, one per line.
+ Actual-value entries (``y_col == "value"``) are truncated to the
+ last ``history`` rows per series.
+ series_ids : list
+ Series to plot, one subplot each.
+ split_date : optional
+ If given, a dotted vertical line marks the train/test split.
+ interval_fcst : pd.DataFrame, optional
+ Forecast output with ``-lo-`` / ``-hi-``
+ columns; the band is shaded in each subplot.
+ level : int, default 80
+ Confidence level of the shaded interval.
+ history : int, default 72
+ Number of trailing actual observations to show per series.
+ title : str
+ Figure title.
+ """
+ try:
+ import matplotlib.pyplot as plt
+ except ImportError as e:
+ raise ImportError(
+ "matplotlib is required for plotting. "
+ "Install it with: pip install hypertrees[plot]"
+ ) from e
+
+ fig, axes = plt.subplots(1, len(series_ids), figsize=(5 * len(series_ids), 5), sharex=True)
+ axes = np.atleast_1d(axes)
+ model = interval_fcst["model"].iloc[0] if interval_fcst is not None else None
+
+ for ax, sid in zip(axes, series_ids):
+ for data, x_col, y_col, label, color, style in datasets:
+ series = data[data["series_id"] == sid]
+ if y_col == "value":
+ series = series.tail(history)
+ ax.plot(series[x_col], series[y_col], label=label, color=color,
+ linestyle=style, linewidth=2, alpha=0.8)
+ if interval_fcst is not None:
+ band = interval_fcst[interval_fcst["series_id"] == sid]
+ ax.fill_between(band["date"], band[f"{model}-lo-{level}"], band[f"{model}-hi-{level}"],
+ color="green", alpha=0.15, label=f"{level}% Interval ({model})")
+ if split_date is not None:
+ ax.axvline(x=split_date, color="black", linestyle=":", alpha=0.7, label="Train/Test Split")
+ ax.set_title(sid, fontsize=12)
+ ax.grid(True, alpha=0.3)
+ ax.tick_params(axis="x", rotation=30)
+
+ fig.suptitle(title, fontsize=16)
+ handles, labels = axes[0].get_legend_handles_labels()
+ fig.legend(handles, labels, loc="lower center", ncol=3, fontsize=11)
+ plt.tight_layout(rect=(0, 0.12, 1, 0.96))
+ plt.show()
+
+
+def plot_model_intervals(
+ actuals: pd.DataFrame,
+ forecasts: dict,
+ levels: list = None,
+ title: str = None,
+) -> None:
+ """Plot actuals, forecast, and conformal interval bands, one subplot per model.
+
+ Interval columns named ``-lo-`` / ``-hi-``
+ (produced by ``forecast(..., level=[...])``) are detected automatically
+ and shaded, widest level first. A single global legend sits below the
+ panels.
+
+ Parameters
+ ----------
+ actuals : pd.DataFrame
+ Realized values with ``date`` and ``value`` columns.
+ forecasts : dict
+ Mapping of subplot title to a forecast DataFrame containing
+ ``date``, ``fcst``, and a ``model`` column.
+ levels : list of int, optional
+ Restrict which detected levels are shaded; by default all are shown.
+ title : str, optional
+ Figure title.
+ """
+ try:
+ import matplotlib.pyplot as plt
+ except ImportError as e:
+ raise ImportError(
+ "matplotlib is required for plotting. "
+ "Install it with: pip install hypertrees[plot]"
+ ) from e
+
+ fig, axes = plt.subplots(1, len(forecasts), figsize=(6 * len(forecasts), 5), sharey=True)
+ axes = np.atleast_1d(axes)
+
+ for ax, (name, fcst_df) in zip(axes, forecasts.items()):
+ model = fcst_df["model"].iloc[0]
+ ax.plot(actuals["date"], actuals["value"], label="Actual",
+ color="#2E86AB", linewidth=2, alpha=0.8)
+ ax.plot(fcst_df["date"], fcst_df["fcst"], label="Forecast",
+ color="green", linestyle="--", linewidth=2)
+
+ detected = sorted(
+ int(m.group(1))
+ for c in fcst_df.columns
+ for m in [re.fullmatch(rf"{re.escape(model)}-lo-(\d+)", c)]
+ if m
+ )
+ if levels is not None:
+ detected = [lv for lv in detected if lv in set(levels)]
+ # Shade widest band first so narrower bands sit on top.
+ alphas = [0.15, 0.28, 0.40]
+ shade = {lv: a for lv, a in zip(sorted(detected, reverse=True), alphas)}
+ for lv in sorted(detected, reverse=True):
+ ax.fill_between(
+ fcst_df["date"],
+ fcst_df[f"{model}-lo-{lv}"],
+ fcst_df[f"{model}-hi-{lv}"],
+ color="green", alpha=shade.get(lv, 0.2), label=f"{lv}% Interval",
+ )
+
+ ax.axvline(x=fcst_df["date"].min(), color="black", linestyle=":",
+ alpha=0.7, label="Train/Test Split")
+ ax.set_title(name, fontsize=14)
+ ax.grid(True, alpha=0.3)
+
+ if title is not None:
+ fig.suptitle(title, fontsize=16)
+ handles, labels = axes[0].get_legend_handles_labels()
+ fig.legend(handles, labels, loc="lower center", ncol=len(labels), fontsize=11)
+ plt.tight_layout(rect=(0, 0.08, 1, 0.96 if title is not None else 1.0))
+ plt.show()
+
+
+def coverage(
+ df: pd.DataFrame,
+ level: int,
+ model: str,
+ value_col: str = "value",
+) -> float:
+ r"""Empirical coverage of a conformal prediction interval.
+
+ Computes the fraction of realized values that fall within the
+ ``[-lo-, -hi-]`` band, reported in percent.
+ For a well-calibrated ``level``% interval this should be close to ``level``.
+
+ Parameters
+ ----------
+ df : pd.DataFrame
+ Forecast output containing ``value_col`` plus the interval columns
+ ``f"{model}-lo-{level}"`` and ``f"{model}-hi-{level}"``.
+ level : int
+ Nominal confidence level (e.g. ``90``).
+ model : str
+ Model-name prefix used for the interval columns (the ``model`` column
+ value, e.g. ``"Hyper-Tree-AR(12)"``).
+ value_col : str, default "value"
+ Column holding the realized values.
+
+ Returns
+ -------
+ float
+ Empirical coverage in percent, in ``[0, 100]``.
+ """
+ lo_col, hi_col = f"{model}-lo-{level}", f"{model}-hi-{level}"
+ missing = {value_col, lo_col, hi_col} - set(df.columns)
+ if missing:
+ raise KeyError(f"Missing columns: {sorted(missing)}.")
+
+ true = df[value_col].to_numpy(dtype=np.float64)
+ lo = df[lo_col].to_numpy(dtype=np.float64)
+ hi = df[hi_col].to_numpy(dtype=np.float64)
+ inside = (true >= lo) & (true <= hi)
+ return float(np.mean(inside) * 100)
+
+
+def mean_interval_width(
+ df: pd.DataFrame,
+ level: int,
+ model: str,
+) -> float:
+ """Mean width of a conformal prediction interval.
+
+ Parameters
+ ----------
+ df : pd.DataFrame
+ Forecast output containing ``f"{model}-lo-{level}"`` and
+ ``f"{model}-hi-{level}"``.
+ level : int
+ Nominal confidence level (e.g. ``90``).
+ model : str
+ Model-name prefix used for the interval columns.
+
+ Returns
+ -------
+ float
+ Mean of ``hi - lo`` across all rows.
+ """
+ lo_col, hi_col = f"{model}-lo-{level}", f"{model}-hi-{level}"
+ missing = {lo_col, hi_col} - set(df.columns)
+ if missing:
+ raise KeyError(f"Missing columns: {sorted(missing)}.")
+
+ lo = df[lo_col].to_numpy(dtype=np.float64)
+ hi = df[hi_col].to_numpy(dtype=np.float64)
+
+ return float(np.mean(hi - lo))
+
+
+def plot_stl(
+ df,
+ date_col="date",
+ cols=("trend", "seasonality"),
+ group_col="model",
+ base_size=14,
+ figsize=(10, 5),
+):
+ """Stacked plot of the trend and seasonality columns, one panel each.
+
+ Expects a long DataFrame with a date column, one numeric column per
+ component, and an optional grouping column (``model``) that gets one line
+ per level. Complexity is O(n) per column, trivial next to the upstream
+ decomposition.
+
+ Assumes a single ``series_id`` per call. If several series are stacked in
+ the frame, filter to one first or the lines will connect unrelated points.
+
+ Returns
+ -------
+ (fig, axes)
+ """
+ df = df.copy()
+ df[date_col] = pd.to_datetime(df[date_col])
+
+ has_groups = group_col in df.columns and df[group_col].nunique() > 1
+
+ fig, axes = plt.subplots(
+ nrows=len(cols), ncols=1, sharex=True,
+ figsize=figsize, constrained_layout=True,
+ )
+ axes = [axes] if len(cols) == 1 else list(axes)
+
+ for ax, col in zip(axes, cols):
+ if has_groups:
+ for key, g in df.groupby(group_col):
+ g = g.sort_values(date_col)
+ ax.plot(g[date_col], g[col], linewidth=1.4, label=str(key))
+ else:
+ g = df.sort_values(date_col)
+ ax.plot(g[date_col], g[col], linewidth=1.4, color="#1f77b4")
+
+ ax.set_ylabel(col.capitalize(), fontsize=base_size)
+ ax.grid(True, color="grey", alpha=0.12, linewidth=0.6)
+ ax.tick_params(labelsize=base_size * 0.8)
+ for spine in ax.spines.values():
+ spine.set_edgecolor("black")
+ spine.set_linewidth(0.8)
+
+ if has_groups:
+ axes[0].legend(frameon=True, fontsize=base_size * 0.7)
+
+ x = df[date_col]
+ span_years = max(1.0, (x.max() - x.min()).days / 365.0)
+ step = max(1, round(span_years / 8))
+ axes[-1].xaxis.set_major_locator(mdates.YearLocator(base=step))
+ axes[-1].xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
+ fig.suptitle("Hyper-Tree-STL Decomposition", fontsize=base_size * 1.1)
diff --git a/experiments/models.py b/experiments/models.py
index 0ea852d..b5e09e8 100644
--- a/experiments/models.py
+++ b/experiments/models.py
@@ -12,6 +12,7 @@
HyperTreeNetAR,
)
from hypertrees.models.mlp import MLP
+from hypertrees.utils import NoDeepcopyObjective
from sklearn.preprocessing import StandardScaler
from chronos import ChronosPipeline
@@ -921,13 +922,16 @@ def HyperTreeETSForecast(
season_length=htets_params["season_length"],
freq=freq,
fcst_h=fcst_h,
- loss_fn=loss_fn
+ loss_fn=loss_fn,
+ # Experiments reproduce the paper benchmarks, which were produced
+ # with the pre-0.2.0 state initialization.
+ seasonal_init=htets_params.get("seasonal_init", "legacy"),
)
# Train model
if manual_param is None:
ht_ets.train(
- lgb_params={k: v for k, v in htets_params.items() if k not in ["num_boost_round", "season_length", "ets_type", "manual_param", "scaling", "train"]},
+ lgb_params={k: v for k, v in htets_params.items() if k not in ["num_boost_round", "season_length", "ets_type", "manual_param", "scaling", "train", "seasonal_init"]},
num_iterations=htets_params["num_boost_round"],
train_data=train[["series_id", "date", "value"] + features],
seed=123,
@@ -1389,7 +1393,6 @@ class HyperTreeNetDirectForecasting:
message="Using backward\\(\\) with create_graph=True will create a reference cycle.*"
)
- _network_states = {} # Store network states for each instance
def __init__(
self,
freq: str = "M",
@@ -1483,9 +1486,6 @@ def get_embeds_loss(
network_loss.backward()
self.optimizer.step()
- # Store network state
- HyperTreeNetDirectForecasting._network_states = self.network.state_dict()
-
# Calculate loss for GBDT
self.network.eval()
fcst_gbdt = self.network(gbdt_embed)
@@ -1580,10 +1580,11 @@ def train(
).to(self.device)
self.optimizer = torch.optim.Adam(self.network.parameters(), lr=network_params["learning_rate"])
- # GBDT parameters
+ # GBDT parameters. The objective wrapper stops lgb.train's params
+ # deepcopy from cloning this instance (see NoDeepcopyObjective).
self.lgb_params = {
"num_class": self.embedding_dim,
- "objective": self.objective_fn,
+ "objective": NoDeepcopyObjective(self.objective_fn),
"metric": "None",
"random_seed": seed,
"verbose": verbose
@@ -1639,9 +1640,8 @@ def forecast(
device=self.device
).reshape(-1, self.embedding_dim)
- # Load saved network state
- self.network.load_state_dict(HyperTreeNetDirectForecasting._network_states)
-
+ # self.network holds this instance's trained weights (boosting
+ # updated it in place; see NoDeepcopyObjective).
self.network.eval()
with torch.no_grad():
forecasts = (self.network(gbdt_embeds)
diff --git a/hypertrees/__init__.py b/hypertrees/__init__.py
index 159ebe2..e6d7966 100644
--- a/hypertrees/__init__.py
+++ b/hypertrees/__init__.py
@@ -1 +1,5 @@
-"""Forecasting with Hyper-Trees"""
\ No newline at end of file
+"""Forecasting with Hyper-Trees"""
+
+from .conformal import ForecastIntervals
+
+__all__ = ["ForecastIntervals"]
diff --git a/hypertrees/conformal.py b/hypertrees/conformal.py
new file mode 100644
index 0000000..df4e517
--- /dev/null
+++ b/hypertrees/conformal.py
@@ -0,0 +1,379 @@
+"""Conformal prediction intervals for Hyper-Tree models.
+
+Acknowledgement
+---------------
+The conformal-interval approach in this module is adapted from Nixtla's
+open-source forecasting libraries:
+
+- statsforecast: https://github.com/Nixtla/statsforecast (Apache-2.0)
+- mlforecast: https://github.com/Nixtla/mlforecast (Apache-2.0)
+- neuralforecast: https://github.com/Nixtla/neuralforecast (Apache-2.0)
+
+The calibration procedure, the two interval construction methods
+(``conformal_distribution`` and ``conformal_error``), the per-horizon-step
+quantile logic, and the output column naming convention (``-lo-``
+/ ``-hi-``) follow Nixtla's design. See the individual
+repositories for the original implementations.
+
+Description
+-----------
+1. **Calibration** runs a rolling-window cross-validation over the training data
+ and collects the *absolute residuals* ``|y_hat - y|`` (the conformity score)
+ for each window, series, and forecast-horizon step.
+2. **At prediction time**, for each confidence ``level`` the intervals are built
+ from per-horizon quantiles of the conformity scores, using one of two methods:
+
+ - ``conformal_distribution`` (Nixtla's default): build synthetic forecast paths
+ ``[y_hat - scores, y_hat + scores]`` and take the symmetric
+ ``[alpha/200, 1 - alpha/200]`` quantiles, where ``alpha = 100 - level``.
+ - ``conformal_error``: take the ``level/100`` quantile of the absolute
+ residuals and form ``y_hat +/- q``.
+
+Quantiles are computed independently per horizon step and per series, matching
+Nixtla's implementation.
+
+The module is intentionally model-agnostic: it only relies on a ``model_factory``
+that returns a fresh model exposing the standard Hyper-Tree ``train`` / ``forecast``
+interface, so it can be reused for the other Hyper-Tree models in the future.
+"""
+
+import warnings
+from dataclasses import dataclass
+from typing import Callable, Dict, List, Sequence, Tuple
+
+import numpy as np
+import pandas as pd
+
+_VALID_METHODS = ("conformal_distribution", "conformal_error")
+
+
+@dataclass
+class ForecastIntervals:
+ """Configuration for conformal prediction intervals.
+
+ Parameters
+ ----------
+ n_windows : int
+ Number of rolling cross-validation windows used to collect conformity
+ scores. Must be at least 2. More windows give a more stable calibration
+ at the cost of additional refits.
+ method : str
+ Interval construction method, either ``"conformal_distribution"`` (default)
+ or ``"conformal_error"``.
+ step_size : int
+ Step (in time steps) between consecutive cross-validation windows. The
+ default of 1 produces maximally overlapping windows and therefore the most
+ conformity scores for a given series length.
+ refit : bool
+ If ``True`` (default), a fresh model is trained for every CV window
+ (rolling-origin evaluation). If ``False``, a single model is trained on
+ the oldest window's training split and reused to forecast all windows,
+ matching Nixtla's ``mlforecast`` behaviour. ``refit=False`` is
+ substantially faster but may under-estimate errors for later windows
+ whose training data the model never saw.
+
+ Notes
+ -----
+ Calibration is always performed at the model's own forecast horizon
+ (``fcst_h``), yielding per-horizon-step intervals.
+ """
+
+ n_windows: int = 5
+ method: str = "conformal_distribution"
+ step_size: int = 1
+ refit: bool = True
+
+ def __post_init__(self):
+ if not isinstance(self.n_windows, int) or self.n_windows < 2:
+ raise ValueError("n_windows must be an integer >= 2.")
+ if self.method not in _VALID_METHODS:
+ raise ValueError(f"method must be one of {_VALID_METHODS}.")
+ if not isinstance(self.step_size, int) or self.step_size < 1:
+ raise ValueError("step_size must be a positive integer.")
+ if not isinstance(self.refit, bool):
+ raise ValueError("refit must be a boolean.")
+
+
+def validate_calibration_length(
+ train_data: pd.DataFrame,
+ fcst_h: int,
+ forecast_intervals: ForecastIntervals,
+ min_train: int,
+) -> None:
+ """Validate that every series is long enough for the rolling-window calibration.
+
+ Each series needs enough observations so that, in the oldest window, the
+ training portion still has at least ``min_train`` rows after carving out the
+ cross-validation test blocks.
+
+ Parameters
+ ----------
+ train_data : pd.DataFrame
+ Training data with a ``series_id`` column.
+ fcst_h : int
+ Forecast horizon (length of each cross-validation test block).
+ forecast_intervals : ForecastIntervals
+ Calibration configuration.
+ min_train : int
+ Minimum number of training rows required by the model (e.g. ``p + 1`` for
+ an AR(p) model, so at least one training sample remains after lagging).
+
+ Raises
+ ------
+ ValueError
+ If any series is too short.
+ """
+ pi = forecast_intervals
+ needed = fcst_h + (pi.n_windows - 1) * pi.step_size + min_train
+ lengths = train_data.groupby("series_id", sort=False).size()
+ bad = lengths[lengths < needed]
+ if len(bad) > 0:
+ raise ValueError(
+ f"Conformal calibration with n_windows={pi.n_windows}, "
+ f"step_size={pi.step_size}, fcst_h={fcst_h} requires at least "
+ f"{needed} observations per series, but these series are too short: "
+ f"{bad.to_dict()}. Reduce n_windows/step_size or provide longer series."
+ )
+
+
+def rolling_origin_residuals(
+ model_factory: Callable[[], object],
+ train_data: pd.DataFrame,
+ fcst_h: int,
+ forecast_intervals: ForecastIntervals,
+ train_kwargs: dict,
+) -> Tuple[np.ndarray, List]:
+ """Collect absolute-residual conformity scores via rolling-window CV.
+
+ For each window ``w = 0, ..., n_windows - 1`` the test block is the ``fcst_h``
+ observations ending at ``L - w * step_size`` (per series), and the model is
+ trained on all earlier observations.
+
+ When ``forecast_intervals.refit`` is ``True`` (default) a fresh model is
+ trained for every window. When ``False``, a single model is trained on the
+ oldest window's training split and reused to forecast all windows. Before
+ each window's forecast the model's forecast seed (lags, states, etc.) is
+ re-anchored to the window's own history via ``set_forecast_origin``, so
+ that residuals reflect the correct origin — only the GBDT refit is skipped.
+
+ Parameters
+ ----------
+ model_factory : Callable[[], object]
+ Zero-argument callable returning a fresh, untrained model exposing
+ ``train(train_data=..., **train_kwargs)``,
+ ``forecast(test_data=..., type="forecast")``, and
+ ``set_forecast_origin(history: pd.DataFrame)`` (required for
+ ``refit=False``; re-anchors the forecast seed without retraining).
+ train_data : pd.DataFrame
+ Full training data (``series_id``, ``date``, ``value`` + features).
+ fcst_h : int
+ Forecast horizon / length of each CV test block.
+ forecast_intervals : ForecastIntervals
+ Calibration configuration.
+ train_kwargs : dict
+ Keyword arguments forwarded to each fresh model's ``train`` call (e.g.
+ ``lgb_params``, ``num_iterations``, ``seed``). Must not contain
+ ``train_data`` or ``forecast_intervals``.
+
+ Returns
+ -------
+ scores : np.ndarray
+ Absolute residuals with shape ``(n_windows, n_series, fcst_h)``. If the
+ data carries a ``mask`` column, residuals at padded rows (``mask == 0``)
+ are NaN and are excluded from the interval quantiles downstream.
+ series_order : list
+ Series ids in first-appearance order (axis 1 of ``scores``).
+ """
+ pi = forecast_intervals
+ series_order = list(dict.fromkeys(train_data["series_id"].tolist()))
+ grouped = {sid: g for sid, g in train_data.groupby("series_id", sort=False)}
+
+ scores = np.empty((pi.n_windows, len(series_order), fcst_h), dtype=float)
+
+ # When refit=False, train once on the oldest window (w = n_windows - 1,
+ # which has the least training data) so the model never sees any of the
+ # test observations across all windows.
+ shared_model = None
+ if not pi.refit:
+ oldest_offset = (pi.n_windows - 1) * pi.step_size
+ train_parts = []
+ for sid in series_order:
+ g = grouped[sid]
+ start = len(g) - oldest_offset - fcst_h
+ train_parts.append(g.iloc[:start])
+ oldest_train_df = pd.concat(train_parts, ignore_index=True)
+ shared_model = model_factory()
+ shared_model.train(train_data=oldest_train_df, **train_kwargs)
+
+ for w in range(pi.n_windows):
+ offset = w * pi.step_size
+ train_parts, test_parts = [], []
+ for sid in series_order:
+ g = grouped[sid]
+ end = len(g) - offset
+ start = end - fcst_h
+ train_parts.append(g.iloc[:start])
+ test_parts.append(g.iloc[start:end])
+
+ test_df = pd.concat(test_parts, ignore_index=True)
+
+ if pi.refit:
+ train_df = pd.concat(train_parts, ignore_index=True)
+ model = model_factory()
+ model.train(train_data=train_df, **train_kwargs)
+ else:
+ model = shared_model
+ window_train_df = pd.concat(train_parts, ignore_index=True)
+ model.set_forecast_origin(window_train_df)
+
+ fcst = model.forecast(test_data=test_df, type="forecast")
+
+ # Residuals are computed positionally; enforce the row-order contract
+ # (one forecast row per input row, in input order) so a model that
+ # reorders or reshapes its output fails loudly instead of silently
+ # mis-assigning residuals across series.
+ if not (
+ np.array_equal(
+ fcst["series_id"].to_numpy(), test_df["series_id"].to_numpy()
+ )
+ and np.array_equal(
+ pd.to_datetime(fcst["date"]).to_numpy(),
+ pd.to_datetime(test_df["date"]).to_numpy(),
+ )
+ ):
+ raise RuntimeError(
+ "model.forecast() returned rows in a different order than "
+ "test_data. rolling_origin_residuals computes residuals "
+ "positionally and requires one forecast row per input row, "
+ "in input order."
+ )
+
+ resid = np.abs(fcst["fcst"].to_numpy() - test_df["value"].to_numpy())
+ if "mask" in test_df.columns:
+ # Padded pseudo-observations (mask == 0, used by HyperTreeETS for
+ # uniform series lengths) carry no information about real forecast
+ # errors; mark them NaN so the NaN-aware interval quantiles ignore
+ # them.
+ resid = np.where(test_df["mask"].to_numpy().astype(bool), resid, np.nan)
+ scores[w] = resid.reshape(len(series_order), fcst_h)
+
+ return scores, series_order
+
+
+def _align_scores(
+ scores: np.ndarray, cal_order: Sequence, target_order: Sequence
+) -> np.ndarray:
+ """Reorder the series axis of ``scores`` to match ``target_order``."""
+ cal_order = list(cal_order)
+ target_order = list(target_order)
+ if cal_order == target_order:
+ return scores
+ missing = set(target_order) - set(cal_order)
+ if missing:
+ raise ValueError(
+ f"Series {missing} were not seen during conformal calibration."
+ )
+ idx = [cal_order.index(s) for s in target_order]
+ return scores[:, idx, :]
+
+
+def _distribution_bands(
+ point: np.ndarray, scores: np.ndarray, levels: List[int]
+) -> Dict[int, Tuple[np.ndarray, np.ndarray]]:
+ """``conformal_distribution`` intervals (synthetic-path symmetric quantiles).
+
+ NaN scores (residuals at padded pseudo-observations) are excluded via
+ NaN-aware quantiles; a cell whose scores are all NaN yields NaN bounds.
+ """
+ # Synthetic forecast paths: (2 * n_windows, n_series, h)
+ paths = np.concatenate([point[None] - scores, point[None] + scores], axis=0)
+ bands = {}
+ for lv in levels:
+ alpha = 100 - lv
+ lo = np.nanquantile(paths, (alpha / 2) / 100.0, axis=0)
+ hi = np.nanquantile(paths, 1.0 - (alpha / 2) / 100.0, axis=0)
+ bands[lv] = (lo, hi)
+ return bands
+
+
+def _error_bands(
+ point: np.ndarray, scores: np.ndarray, levels: List[int]
+) -> Dict[int, Tuple[np.ndarray, np.ndarray]]:
+ """``conformal_error`` intervals (symmetric ``y_hat +/- quantile``).
+
+ NaN scores (residuals at padded pseudo-observations) are excluded via
+ NaN-aware quantiles; a cell whose scores are all NaN yields NaN bounds.
+ """
+ bands = {}
+ for lv in levels:
+ q = np.nanquantile(scores, lv / 100.0, axis=0)
+ bands[lv] = (point - q, point + q)
+ return bands
+
+
+def interval_columns(
+ point: np.ndarray,
+ scores: np.ndarray,
+ levels: List[int],
+ method: str,
+ model_name: str,
+ cal_order: Sequence,
+ target_order: Sequence,
+) -> "Dict[str, np.ndarray]":
+ """Build the ``-lo-`` / ``-hi-`` columns.
+
+ Parameters
+ ----------
+ point : np.ndarray
+ Point forecasts shaped ``(n_series, fcst_h)`` in ``target_order``.
+ scores : np.ndarray
+ Conformity scores shaped ``(n_windows, n_series, fcst_h)`` in ``cal_order``.
+ levels : list of int
+ Confidence levels in ``(0, 100)``.
+ method : str
+ ``"conformal_distribution"`` or ``"conformal_error"``.
+ model_name : str
+ Prefix for the interval columns (the model's ``model`` string).
+ cal_order, target_order : sequence
+ Series order of ``scores`` and of the desired output, respectively.
+
+ Returns
+ -------
+ dict of str -> np.ndarray
+ Ordered mapping of column name to a flat ``(n_series * fcst_h,)`` array,
+ flattened series-major to match the forecast DataFrame's row order.
+ """
+ levels = sorted(int(lv) for lv in levels)
+ for lv in levels:
+ if not 0 < lv < 100:
+ raise ValueError(f"level values must be in (0, 100); got {lv}.")
+
+ # With few calibration windows, high-level tail quantiles sit at the
+ # extremes of the available scores and the intervals will undercover.
+ n_windows = scores.shape[0]
+ for lv in levels:
+ if n_windows * (100 - lv) < 100:
+ warnings.warn(
+ f"level={lv} requires tail quantiles beyond the resolution of "
+ f"n_windows={n_windows} conformity scores per series and "
+ f"horizon step; the bounds then sit at the extremes of the "
+ f"scores and the interval will likely undercover. Increase "
+ f"ForecastIntervals(n_windows=...) or request a lower level."
+ )
+
+ scores = _align_scores(scores, cal_order, target_order)
+
+ if method == "conformal_distribution":
+ bands = _distribution_bands(point, scores, levels)
+ elif method == "conformal_error":
+ bands = _error_bands(point, scores, levels)
+ else:
+ raise ValueError(f"method must be one of {_VALID_METHODS}.")
+
+ columns: Dict[str, np.ndarray] = {}
+ # Mirror Nixtla column ordering: lower bounds (widest first), then upper bounds.
+ for lv in sorted(levels, reverse=True):
+ columns[f"{model_name}-lo-{lv}"] = bands[lv][0].reshape(-1)
+ for lv in sorted(levels):
+ columns[f"{model_name}-hi-{lv}"] = bands[lv][1].reshape(-1)
+ return columns
diff --git a/hypertrees/models/HyperTreeAR.py b/hypertrees/models/HyperTreeAR.py
index a374a30..5d135fb 100644
--- a/hypertrees/models/HyperTreeAR.py
+++ b/hypertrees/models/HyperTreeAR.py
@@ -6,12 +6,18 @@
import torch.nn as nn
from torch.autograd import grad as autograd
import lightgbm as lgb
-from typing import Tuple, Callable, Optional
+from typing import Tuple, Callable, Optional, List
import time
from ..utils import CustomLogger
lgb.register_logger(CustomLogger())
-from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order, GaussNewtonHessian
+from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order, extract_forecast_lags, GaussNewtonHessian, NoDeepcopyObjective
+from ..conformal import (
+ ForecastIntervals,
+ validate_calibration_length,
+ rolling_origin_residuals,
+ interval_columns,
+)
class HyperTreeAR:
"""
@@ -87,7 +93,7 @@ def __init__(
freq: str = "M",
fcst_h: int = 1,
loss_fn: Callable = nn.MSELoss(),
- hessian_method: str = "exact",
+ hessian_method: str = "analytic",
n_hessian_probes: int = 5,
):
"""
@@ -104,10 +110,24 @@ def __init__(
Forecast horizon (number of periods to forecast ahead).
loss_fn : Callable
Loss function for optimization. Must be a PyTorch loss function.
- Default is MSE loss, but can be changed for different error metrics.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
hessian_method : str
Method for computing the Hessian diagonal. Options:
- - "exact": Exact diagonal Hessian via per-parameter second-order autograd.
+ - "exact": Exact diagonal Hessian via per-parameter second-order autograd
+ (one backward pass per lag and iteration).
+ - "analytic": Closed-form gradients and exact diagonal Hessians,
+ exploiting that the AR fit is linear in its parameters
+ (dL/dtheta_j = l'(y_hat) * lag_j and d2L/dtheta_j2 = l''(y_hat) * lag_j**2,
+ the second-order fit term vanishing exactly). Produces the same
+ values as "exact" for any loss that is a mean/sum of
+ per-observation terms -- which covers all standard PyTorch
+ regression losses -- at a fraction of the cost: at most one
+ small double-backward through loss(fit, target) instead of one
+ backward per lag. nn.MSELoss uses a fully closed-form fast path
+ with no autograd at all.
- "gn": Gauss-Newton approximation estimated via Hutchinson probing.
Guarantees positive semi-definite Hessians. Avoids second-order
differentiation at the cost of Hutchinson estimation variance.
@@ -125,8 +145,21 @@ def __init__(
raise TypeError("freq must be a string.")
if not isinstance(loss_fn, nn.Module):
raise TypeError("loss_fn must be a PyTorch loss function.")
- if hessian_method not in ("exact", "gn"):
- raise ValueError("hessian_method must be either 'exact' or 'gn'.")
+ if isinstance(loss_fn, nn.L1Loss):
+ raise ValueError(
+ "nn.L1Loss is not supported: its second derivative is zero almost "
+ "everywhere, so LightGBM's Newton boosting receives all-zero Hessians "
+ "and cannot grow trees. Use nn.HuberLoss or nn.SmoothL1Loss for an "
+ "MAE-like loss with usable curvature."
+ )
+ if getattr(loss_fn, "reduction", "mean") == "none":
+ raise ValueError(
+ "loss_fn must use a scalar reduction ('mean' or 'sum'); "
+ "reduction='none' returns per-element losses that the "
+ "boosting objective cannot consume."
+ )
+ if hessian_method not in ("exact", "analytic", "gn"):
+ raise ValueError("hessian_method must be one of 'exact', 'analytic', or 'gn'.")
if not isinstance(n_hessian_probes, int) or n_hessian_probes <= 0:
raise ValueError("n_hessian_probes must be a positive integer.")
@@ -154,10 +187,20 @@ def __init__(
self._iter_count = 0
self._fit = None
self._target = None
+ self._lags = None
+
+ # Conformal prediction interval state (populated when train() is called
+ # with forecast_intervals).
+ self._is_calibrated = False
+ self._cs_scores = None # conformity scores (n_windows, n_series, fcst_h)
+ self._cs_series_order = None # series order along axis 1 of _cs_scores
+ self._pi_config = None # ForecastIntervals configuration
# Bind Hessian computation strategy
if hessian_method == "exact":
self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_exact
+ elif hessian_method == "analytic":
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_analytic
else:
self._gn_hessian = GaussNewtonHessian(loss_fn, n_hessian_probes, self.dtype)
self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_gn
@@ -276,14 +319,29 @@ def get_params_loss(
# Calculate loss between forecasts and actual values
loss = self.loss_fn(fcst, target)
- if self.hessian_method == "gn":
+ if self.hessian_method in ("gn", "analytic"):
self._fit = fcst
self._target = target
+ self._lags = lags
return params, loss
def _calculate_gradients_and_hessians_exact(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
- """Exact diagonal Hessian via per-parameter second-order autograd."""
+ """Exact diagonal Hessian via per-parameter second-order autograd.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ params : torch.Tensor
+ Model parameters (AR coefficients as an ``nn.Parameter``,
+ shape ``(n_samples, p)``).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
loss.backward(create_graph=True)
grad = params.grad
hess = [
@@ -297,13 +355,82 @@ def _calculate_gradients_and_hessians_exact(self, loss: torch.Tensor, params: to
return grad, hess
+ def _calculate_gradients_and_hessians_analytic(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
+ """Closed-form gradients and exact diagonal Hessians via model linearity.
+
+ Since the AR fit is linear in its parameters, ``grad = l'(y_hat) * lags``
+ and ``hess = l''(y_hat) * lags**2``, matching the "exact" method for any
+ per-observation loss. MSELoss uses closed-form derivatives; other losses
+ use one small double-backward through ``loss(fit, target)``.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model (unused; derivatives come from the
+ fit/target/lags stored by ``get_params_loss``).
+ params : torch.Tensor
+ Model parameters (unused, kept for a uniform dispatch signature).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ fit = self._fit.detach()
+ target = self._target
+ lags = self._lags
+
+ if isinstance(self.loss_fn, nn.MSELoss) and self.loss_fn.reduction in ("mean", "sum"):
+ # MSE fast path: l' = scale * (y_hat - y), l'' = scale
+ scale = 2.0 / fit.numel() if self.loss_fn.reduction == "mean" else 2.0
+ g = scale * (fit - target)
+ h = torch.full_like(fit, scale)
+ else:
+ # Generic path: per-element first and second loss derivatives via
+ # a double-backward through the (tiny) loss(fit, target) graph.
+ # Requires the loss to have a well-defined double-backward (true
+ # for HuberLoss/SmoothL1Loss and other standard smooth losses).
+ # nn.L1Loss is rejected in __init__: its zero curvature breaks
+ # Newton boosting, and torch 2.8.0's l1_loss double-backward can
+ # return an uninitialized buffer instead of zeros.
+ fit_leaf = fit.requires_grad_(True)
+ loss_local = self.loss_fn(fit_leaf, target)
+ g = autograd(loss_local, fit_leaf, create_graph=True)[0]
+ h = autograd(g.sum(), fit_leaf)[0].detach()
+ g = g.detach()
+
+ # Broadcast (N, 1) loss derivatives over the (N, p) lag matrix.
+ grad = (g * lags).cpu().numpy().ravel(order="F")
+ hess = (h * lags ** 2).cpu().numpy().ravel(order="F")
+
+ self._fit = None
+ self._target = None
+ self._lags = None
+
+ return grad, hess
+
def _calculate_gradients_and_hessians_gn(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
- """Gauss-Newton Hessian diagonal estimated via Hutchinson probing."""
+ """Gauss-Newton Hessian diagonal estimated via Hutchinson probing.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ params : torch.Tensor
+ Model parameters (AR coefficients as an ``nn.Parameter``,
+ shape ``(n_samples, p)``).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
grad = autograd(loss, params, retain_graph=True)[0]
rng = torch.Generator().manual_seed(self._iter_count)
hess = self._gn_hessian.estimate(self._fit, self._target, params, rng)
self._fit = None
self._target = None
+ self._lags = None
grad = grad.cpu().detach().numpy().ravel(order="F")
hess = hess.cpu().detach().numpy().ravel(order="F")
@@ -320,6 +447,7 @@ def train(
seed: int = 123,
verbose: int = -1,
deterministic: bool = True,
+ forecast_intervals: Optional[ForecastIntervals] = None,
) -> TrainingResult:
"""
Train the Hyper-Tree-AR model on time series data.
@@ -356,6 +484,12 @@ def train(
If True, sets LightGBM's ``deterministic`` and ``force_row_wise`` parameters to ensure
reproducible results. May slow down training. See
https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic
+ forecast_intervals : ForecastIntervals, optional
+ If provided, calibrate conformal prediction intervals via rolling-window
+ cross-validation after the main model is trained. The collected conformity
+ scores are then used by ``forecast(..., level=[...])`` to produce
+ ``-lo-`` / ``-hi-`` columns. See
+ :class:`hypertrees.conformal.ForecastIntervals`.
Returns
-------
@@ -383,6 +517,8 @@ def train(
raise TypeError("validation must be a boolean.")
if not isinstance(deterministic, bool):
raise TypeError("deterministic must be a boolean.")
+ if forecast_intervals is not None and not isinstance(forecast_intervals, ForecastIntervals):
+ raise TypeError("forecast_intervals must be a ForecastIntervals instance.")
if early_stopping_round is not None and not validation:
raise ValueError("early_stopping_round can only be used when validation is True.")
if validation and early_stopping_round is None:
@@ -401,10 +537,30 @@ def train(
# monotonic dates so the training reshape and fcst_lags extraction align.
validate_series_order(train_data, name="train_data")
- # General model parameters
+ # Each series must keep at least one training row after lagging; a
+ # shorter series would silently contribute nothing while leaving a
+ # ragged forecast seed behind.
+ lengths = train_data.groupby("series_id", sort=False).size()
+ bad = lengths[lengths <= self.p]
+ if len(bad) > 0:
+ raise ValueError(
+ f"Each series needs at least p + 1 = {self.p + 1} observations "
+ f"so that one training row remains after lagging, but these "
+ f"series are too short: {bad.to_dict()}."
+ )
+
+ # Fail fast if any series is too short for the requested conformal calibration.
+ # An AR(p) model needs at least p + 1 rows to retain one training sample.
+ if forecast_intervals is not None:
+ validate_calibration_length(
+ train_data, self.fcst_h, forecast_intervals, min_train=self.p + 1
+ )
+
+ # General model parameters. The objective wrapper stops lgb.train's
+ # params deepcopy from cloning this instance (see NoDeepcopyObjective).
self.lgb_params = {
"num_class": self.p,
- "objective": self.objective_fn,
+ "objective": NoDeepcopyObjective(self.objective_fn),
"metric": "None",
"random_seed": seed,
"verbose": verbose
@@ -417,10 +573,15 @@ def train(
self._iter_count = 0
self._fit = None
self._target = None
+ self._lags = None
self.model = None
self.dataset_references = {}
self.is_trained = False
self.features = None
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
try:
# Initialize TimeSeriesPreprocessor for creating lagged dataframe
@@ -459,11 +620,7 @@ def train(
self.lags_eval = lags_eval
# Store lagged train values to be used in the forecast method
- self.fcst_lags = (
- train_data.groupby(["series_id"], sort=False)
- .apply(lambda x: x["value"][-self.p:][::-1].values)
- .to_dict()
- )
+ self.set_forecast_origin(train_data)
# Train LightGBM model
start_time = time.time()
@@ -481,11 +638,43 @@ def train(
# Set trained flag to True
self.is_trained = True
+ # Calibrate conformal prediction intervals via rolling-window CV.
+ # Fresh model instances are trained per window (no forecast_intervals
+ # passed, so there is no recursion) using the same hyper-parameters.
+ if forecast_intervals is not None:
+ def _model_factory():
+ return HyperTreeAR(
+ p=self.p,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ hessian_method=self.hessian_method,
+ n_hessian_probes=self.n_hessian_probes,
+ )
+
+ cal_train_kwargs = dict(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+ self._cs_scores, self._cs_series_order = rolling_origin_residuals(
+ model_factory=_model_factory,
+ train_data=train_data,
+ fcst_h=self.fcst_h,
+ forecast_intervals=forecast_intervals,
+ train_kwargs=cal_train_kwargs,
+ )
+ self._pi_config = forecast_intervals
+ self._is_calibrated = True
+
# Return results
result = TrainingResult(
train_metrics=evals_result["train"] if validation else {"loss": []},
validation_metrics=evals_result["validation"] if validation else None,
- best_iteration=self.model.best_iteration-1 if hasattr(self.model, 'best_iteration') else num_iterations,
+ best_iteration=self.model.best_iteration if self.model.best_iteration > 0 else num_iterations,
training_time=training_time
)
@@ -494,12 +683,25 @@ def train(
except Exception as e:
self.is_trained = False
- raise RuntimeError(f"Training failed: {str(e)}")
+ raise RuntimeError(f"Training failed: {str(e)}") from e
+
+ def set_forecast_origin(self, history: pd.DataFrame) -> None:
+ """Re-anchor the AR lag seed to the end of *history* without retraining.
+
+ Parameters
+ ----------
+ history : pd.DataFrame
+ DataFrame with ``series_id``, ``date``, ``value`` columns, ordered
+ by ``(series_id, date)`` with each series in a contiguous block.
+ """
+ validate_series_order(history, name="history")
+ self.fcst_lags = extract_forecast_lags(history, self.p)
def forecast(
self,
test_data: pd.DataFrame,
- type: str = "forecast"
+ type: str = "forecast",
+ level: Optional[List[int]] = None
) -> pd.DataFrame:
"""
Generate forecasts using the trained model.
@@ -523,6 +725,11 @@ def forecast(
Type of forecast to generate. Options:
- "forecast": Generate forecasted values
- "parameters": Return the AR(p) coefficients used for forecasting
+ level : list of int, optional
+ Confidence levels (in ``(0, 100)``, e.g. ``[80, 90]``) for conformal
+ prediction intervals. Only valid with ``type="forecast"`` and requires
+ the model to have been trained with ``forecast_intervals=...``. Adds
+ ``-lo-`` / ``-hi-`` columns to the output.
Returns
-------
@@ -533,6 +740,8 @@ def forecast(
- fcst: Forecasted value (if type="forecast")
- model: Model name identifier
- AR(i): AR coefficient values (if type="parameters")
+ - -lo- / -hi-: prediction interval bounds
+ (if type="forecast" and level is provided)
"""
# Check if model is trained
if not self.is_trained or self.model is None:
@@ -581,6 +790,22 @@ def forecast(
if type not in ["forecast", "parameters"]:
raise ValueError("Parameter 'type' must be either 'forecast' or 'parameters'")
+ # Validate conformal interval request
+ if level is not None:
+ if type != "forecast":
+ raise ValueError("level is only supported with type='forecast'.")
+ if not self._is_calibrated:
+ raise RuntimeError(
+ "Prediction intervals were requested via level, but the model "
+ "was not calibrated. Pass forecast_intervals=ForecastIntervals(...) "
+ "to train() before forecasting with level."
+ )
+ if not isinstance(level, (list, tuple)) or len(level) == 0:
+ raise ValueError("level must be a non-empty list of integers.")
+ for lv in level:
+ if not isinstance(lv, (int, np.integer)) or not 0 < lv < 100:
+ raise ValueError(f"level values must be integers in (0, 100); got {lv}.")
+
try:
if type == "forecast":
@@ -603,19 +828,34 @@ def forecast(
lags = np.concatenate([next_val, lags[:, :-1]], axis=1)
# Create output dataframe based on requested type
+ model_name = f"Hyper-Tree-AR({self.p})"
out_df = pd.DataFrame({
"series_id": test_data["series_id"].to_numpy().flatten(),
"date": test_data["date"].to_numpy().flatten(),
"fcst": np.hstack(forecasts).flatten(),
- "model": f"Hyper-Tree-AR({self.p})",
+ "model": model_name,
})
+ # Append conformal prediction intervals if requested.
+ if level is not None:
+ point = np.hstack(forecasts) # (n_series_test, fcst_h)
+ columns = interval_columns(
+ point=point,
+ scores=self._cs_scores,
+ levels=level,
+ method=self._pi_config.method,
+ model_name=model_name,
+ cal_order=self._cs_series_order,
+ target_order=list(test_series_ids),
+ )
+ for col_name, values in columns.items():
+ out_df[col_name] = values
+
elif type == "parameters":
params_fcst = np.asarray(self.model.predict(test_data[self.features]))
- # LightGBM may return 1D (column-major) or 2D depending on version/objective.
- # Normalize to (n_test, p) before indexing.
+ # Booster.predict returns (n_test, p) for multi-class output
if params_fcst.ndim == 1:
- params_fcst = params_fcst.reshape(-1, self.p, order="F")
+ params_fcst = params_fcst.reshape(-1, self.p)
out_df = pd.DataFrame({
"series_id": test_data["series_id"].to_numpy().flatten(),
"date": test_data["date"].to_numpy().flatten(),
@@ -628,4 +868,4 @@ def forecast(
return out_df
except Exception as e:
- raise RuntimeError(f"Forecasting not successful: {str(e)}")
+ raise RuntimeError(f"Forecasting not successful: {str(e)}") from e
diff --git a/hypertrees/models/HyperTreeARMA.py b/hypertrees/models/HyperTreeARMA.py
new file mode 100644
index 0000000..9dc71a1
--- /dev/null
+++ b/hypertrees/models/HyperTreeARMA.py
@@ -0,0 +1,1124 @@
+import warnings
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+from torch.autograd import grad as autograd
+import lightgbm as lgb
+from typing import Tuple, Callable, Optional, List
+import time
+from ..utils import CustomLogger
+lgb.register_logger(CustomLogger())
+
+from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order, extract_forecast_lags, GaussNewtonHessian, NoDeepcopyObjective
+from ..conformal import (
+ ForecastIntervals,
+ validate_calibration_length,
+ rolling_origin_residuals,
+ interval_columns,
+)
+from .HyperTreeAR import HyperTreeAR
+
+
+class HyperTreeARMA:
+ """
+ Class that implements a Hyper-Tree-ARMA(p, q) model for time series forecasting.
+
+ The Hyper-Tree-ARMA(p, q) model extends the Hyper-Tree-AR(p) model with a
+ moving-average block, so that
+
+ y_t = sum_{j=1..p} phi_j(x_t) * y_{t-j} + sum_{i=1..q} theta_i(x_t) * eps_{t-i} + eps_t
+
+ where the AR coefficients phi_j and the MA coefficients theta_i are
+ time-varying and estimated by gradient boosted trees. The MA block is an
+ error-correction mechanism: it regresses on the model's own past one-step
+ forecast errors, adjusting the forecast when recent periods were over- or
+ under-predicted.
+
+ Because the innovations eps_t are latent, an exact ARMA fit would require
+ reconstructing them recursively from the parameters (eps_t depends on all
+ earlier eps), which makes the fit nonlinear in its parameters and forces
+ an O(T) sequential autograd graph per boosting iteration (the
+ Hyper-Tree-ETS situation). This implementation avoids the recursion with
+ the classical two-stage Hannan-Rissanen approach:
+
+ 1. **Stage 1**: a long autoregression (a ``HyperTreeAR`` of order
+ ``stage1_p``, by default the Gomez-Maravall proposal
+ ``max(floor(log(T)**2), 2 * max(p, q))`` used by statsmodels'
+ ``hannan_rissanen`` and RATS' ``@HannanRissanen``) is fitted to the
+ training data and its in-sample one-step residuals
+ ``eps_hat_t = y_t - y_hat_t`` are extracted.
+ 2. **Stage 2**: the lagged residuals are treated as *observed* regressors,
+ so the ARMA fit becomes linear in its parameters -- structurally
+ identical to the AR design with a widened lag matrix
+ ``[y_{t-1..t-p}, eps_hat_{t-1..t-q}]`` -- and trains with the same
+ closed-form analytic gradients and exact diagonal Hessians at AR speed.
+
+ The classical third Hannan-Rissanen stage (a one-step Gauss-Newton bias
+ correction of the stage-2 estimates) is intentionally omitted: its
+ derivative series are themselves recursive filters in the estimated
+ coefficients, which would reintroduce the sequential graph this
+ estimator exists to avoid. Unlike the classical procedure, both stages
+ here are feature-driven GBDTs, so the residual extractor and the ARMA
+ coefficients are time-varying.
+
+ The stage-1 residuals are used both as training regressors and as the
+ forecast seed, so the coefficients are applied at forecast time to the
+ same quantities they were trained on. Beyond the forecast origin, future
+ innovations are unobserved with expectation zero, so the MA terms
+ contribute to the first ``q`` horizon steps (multiplying the known last
+ residuals) and then vanish, leaving the pure AR recursion.
+
+ Key features:
+ - Combines tree-based models (LightGBM) with ARMA time series modeling
+ - Allows AR and MA coefficients to vary based on features
+ - Recursion-free estimation via Hannan-Rissanen residual proxies:
+ analytic gradients/Hessians and AR-level training speed
+ - MA block corrects the first q forecast steps using the latest
+ observed forecast errors
+
+ Use this model when:
+ - The series has short-memory error-correction structure that a pure
+ AR(p) of moderate order does not capture
+ - You have relevant features that might influence the autoregressive
+ or error-correction structure
+
+ Note that training fits two GBDTs (the stage-1 AR and the stage-2 ARMA),
+ roughly doubling the training cost relative to ``HyperTreeAR``. A pure
+ ``HyperTreeAR`` with a longer lag order approximates the same conditional
+ mean (every invertible ARMA has an AR(infinity) representation) and is the
+ natural baseline to compare against.
+
+ References
+ ----------
+ [1] Hannan, E. J., & Rissanen, J. (1982). Recursive Estimation of Mixed
+ Autoregressive-Moving Average Order. Biometrika, 69(1), 81-94.
+ [2] Gomez, V., & Maravall, A. (2001). Automatic Modeling Methods for
+ Univariate Series. In Pena, Tiao & Tsay (eds.), A Course in Time
+ Series Analysis. Wiley. (Default order of the stage-1 long AR.)
+
+ Example usage:
+ ```python
+ # Imports
+ from hypertrees.models import HyperTreeARMA
+ import pandas as pd
+ import matplotlib.pyplot as plt
+
+ # Initialize model
+ lag_p = 2
+ lag_q = 1
+ frequency = 'M'
+ fcst_h = 12
+ model = HyperTreeARMA(p=lag_p, q=lag_q, freq=frequency, fcst_h=fcst_h)
+
+ # Data
+ # The data needs to have the following columns: 'date', 'series_id', 'value'. All other columns are automatically treated as features.
+ # You don't have to add lag-values or residuals yourself, this happens automatically during training.
+ df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/air-passengers.csv', parse_dates=['ds'])
+ df.rename(columns={'unique_id': 'series_id', 'ds': 'date', 'y': 'value'}, inplace=True)
+ df['month'] = df['date'].dt.month
+ df["quarter"] = df['date'].dt.quarter
+ test = df.tail(fcst_h)
+ train = df.drop(test.index)
+
+ # Train model
+ model.train(
+ lgb_params={'learning_rate': 0.1},
+ num_iterations=100,
+ train_data=train
+ )
+
+ # Generate forecasts and inspect the time-varying ARMA coefficients
+ forecasts = model.forecast(test_data=test)
+ coefficients = model.forecast(test_data=test, type="parameters")
+
+ # Plot results
+ datasets = [
+ (df, 'date', 'value', 'Actual', '#2E86AB', '-'),
+ (forecasts, 'date', 'fcst', 'Forecast', '#F18F01', '--')
+ ]
+
+ for data, x_col, y_col, label, color, style in datasets:
+ plt.plot(data[x_col], data[y_col], label=label, color=color,
+ linestyle=style, linewidth=2, alpha=0.8)
+
+ plt.title('AirPassengers - Forecast', fontsize=14)
+ plt.legend(frameon=True, fancybox=True)
+ plt.grid(True, alpha=0.3)
+ plt.tight_layout()
+ ```
+ """
+
+ def __init__(
+ self,
+ p: int = 2,
+ q: int = 1,
+ freq: str = "M",
+ fcst_h: int = 1,
+ loss_fn: Callable = nn.MSELoss(),
+ hessian_method: str = "analytic",
+ n_hessian_probes: int = 5,
+ stage1_p: Optional[int] = None,
+ ):
+ """
+ Initialize the Hyper-Tree-ARMA(p, q) model.
+
+ Arguments
+ ----------
+ p : int
+ Number of AR lags. Must be a positive integer.
+ q : int
+ Number of MA terms (lagged residual regressors). Must be a
+ positive integer; for q = 0 use ``HyperTreeAR`` directly.
+ freq : str
+ Frequency of the time series (e.g., 'D' for daily, 'M' for monthly,
+ 'Q' for quarterly, 'Y' for yearly).
+ fcst_h : int
+ Forecast horizon (number of periods to forecast ahead).
+ loss_fn : Callable
+ Loss function for optimization. Must be a PyTorch loss function.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
+ hessian_method : str
+ Method for computing the Hessian diagonal. Options:
+ - "exact": Exact diagonal Hessian via per-parameter second-order
+ autograd (one backward pass per coefficient and iteration).
+ - "analytic" (default): Closed-form gradients and exact diagonal
+ Hessians, exploiting that -- with the stage-1 residuals frozen
+ as observed regressors -- the ARMA fit is linear in its
+ parameters (dL/dtheta_j = l'(y_hat) * z_j and
+ d2L/dtheta_j2 = l''(y_hat) * z_j**2, the second-order fit term
+ vanishing exactly). Produces the same values as "exact" for any
+ loss that is a mean/sum of per-observation terms -- which covers
+ all standard PyTorch regression losses -- at a fraction of the
+ cost. nn.MSELoss uses a fully closed-form fast path with no
+ autograd at all.
+ - "gn": Gauss-Newton approximation estimated via Hutchinson
+ probing. Guarantees positive semi-definite Hessians. Because
+ the fit is linear in its parameters, this estimates the same
+ diagonal as "analytic", with Hutchinson sampling variance.
+ n_hessian_probes : int
+ Number of Hutchinson probes for Gauss-Newton Hessian diagonal estimation.
+ Only used when hessian_method="gn". More probes reduce variance but
+ increase computation. Default is 5.
+ stage1_p : int, optional
+ Lag order of the stage-1 autoregression used to extract the
+ residual proxies (the Hannan-Rissanen "long AR"). If None
+ (default), it is resolved at training time via the
+ Gomez-Maravall (2001) proposal used by statsmodels and RATS:
+ ``max(floor(log(T)**2), 2 * max(p, q))``, with ``T`` the
+ shortest series length. Larger values give cleaner residual
+ proxies at the cost of dropping more training rows: stage-2
+ training uses rows from ``max(p, stage1_p + q) + 1`` onward per
+ series. Pass a smaller value explicitly for short series.
+ """
+ # Validate inputs
+ if not isinstance(p, int) or p <= 0:
+ raise ValueError("Parameter 'p' must be a positive integer.")
+ if not isinstance(q, int) or q <= 0:
+ raise ValueError(
+ "Parameter 'q' must be a positive integer. For q = 0 (no MA "
+ "terms) use HyperTreeAR directly."
+ )
+ if fcst_h <= 0:
+ raise ValueError("Forecast horizon 'fcst_h' must be a positive integer.")
+ if not isinstance(freq, str):
+ raise TypeError("freq must be a string.")
+ if not isinstance(loss_fn, nn.Module):
+ raise TypeError("loss_fn must be a PyTorch loss function.")
+ if isinstance(loss_fn, nn.L1Loss):
+ raise ValueError(
+ "nn.L1Loss is not supported: its second derivative is zero almost "
+ "everywhere, so LightGBM's Newton boosting receives all-zero Hessians "
+ "and cannot grow trees. Use nn.HuberLoss or nn.SmoothL1Loss for an "
+ "MAE-like loss with usable curvature."
+ )
+ if getattr(loss_fn, "reduction", "mean") == "none":
+ raise ValueError(
+ "loss_fn must use a scalar reduction ('mean' or 'sum'); "
+ "reduction='none' returns per-element losses that the "
+ "boosting objective cannot consume."
+ )
+ if hessian_method not in ("exact", "analytic", "gn"):
+ raise ValueError("hessian_method must be one of 'exact', 'analytic', or 'gn'.")
+ if not isinstance(n_hessian_probes, int) or n_hessian_probes <= 0:
+ raise ValueError("n_hessian_probes must be a positive integer.")
+ if stage1_p is not None and (not isinstance(stage1_p, int) or stage1_p <= 0):
+ raise ValueError("stage1_p must be a positive integer.")
+
+ if hessian_method == "gn" and not isinstance(loss_fn, nn.MSELoss):
+ warnings.warn(
+ f"Loss {loss_fn.__class__.__name__} is not nn.MSELoss. The Gauss-Newton "
+ "Hessian requires a twice-differentiable loss; non-smooth losses "
+ "(e.g., L1Loss, quantile loss, HuberLoss/SmoothL1Loss outside the quadratic "
+ "region) have zero or undefined second derivatives at kinks, "
+ "causing degenerate Hessians."
+ )
+
+ self.p = p
+ self.q = q
+ self.n_params = p + q
+ self._stage1_p_arg = stage1_p
+ self.stage1_p = stage1_p # resolved at training time when None
+ self.freq = freq
+ self.fcst_h = fcst_h
+ self.loss_fn = loss_fn
+ self.loss_name = self.loss_fn.__class__.__name__
+ self.dtype = torch.float32
+ self.model = None
+ self.features = None # Stores feature names after training
+ self.is_trained = False # Flag to track if model has been trained
+ self.dataset_references = {} # Store references to LightGBM datasets
+ self.hessian_method = hessian_method
+ self.n_hessian_probes = n_hessian_probes
+ self._stage1 = None # Trained stage-1 HyperTreeAR (residual extractor)
+ self.fcst_lags = None # {series_id: last p values, newest first}
+ self.fcst_eps = None # {series_id: last q stage-1 residuals, newest first}
+ self._iter_count = 0
+ self._fit = None
+ self._target = None
+ self._design = None
+
+ # Conformal prediction interval state (populated when train() is called
+ # with forecast_intervals).
+ self._is_calibrated = False
+ self._cs_scores = None # conformity scores (n_windows, n_series, fcst_h)
+ self._cs_series_order = None # series order along axis 1 of _cs_scores
+ self._pi_config = None # ForecastIntervals configuration
+
+ # Bind Hessian computation strategy
+ if hessian_method == "exact":
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_exact
+ elif hessian_method == "analytic":
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_analytic
+ else:
+ self._gn_hessian = GaussNewtonHessian(loss_fn, n_hessian_probes, self.dtype)
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_gn
+
+ def objective_fn(self, predt: np.ndarray, data: lgb.Dataset) -> Tuple[np.ndarray, np.ndarray]:
+ """
+ Custom objective function for LightGBM training.
+
+ This function defines the gradients and hessians for the LightGBM model
+ based on the PyTorch loss function. It converts the raw LightGBM outputs to
+ ARMA coefficients, computes the loss, and then derives gradients and
+ Hessians via the bound ``hessian_method`` strategy.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM, representing the p AR and q MA
+ coefficients per training row.
+ data : lgb.Dataset
+ LightGBM dataset containing the target values.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians for LightGBM optimization.
+ """
+ self._iter_count += 1
+
+ target = torch.tensor(data.get_label().reshape(-1, 1), dtype=self.dtype)
+ params, loss = self.get_params_loss(predt, target, self.design_train, requires_grad=True)
+ grad, hess = self.calculate_gradients_and_hessians(loss, params)
+
+ return grad, hess
+
+
+ def eval_fn(self, predt: np.ndarray, eval_data: lgb.Dataset) -> Tuple[str, float, bool]:
+ """
+ Custom evaluation function for evaluating forecast accuracy on an evaluation dataset.
+
+ This function computes the loss value to be monitored during evaluation.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM.
+ eval_data : lgb.Dataset
+ LightGBM dataset containing the evaluation data.
+
+ Returns
+ -------
+ Tuple[str, float, bool]
+ Name of the metric, value of the metric, and whether to maximize it.
+ """
+ # Use appropriate design rows based on dataset name
+ dataset_name = self.dataset_references.get(id(eval_data), "unknown")
+ if dataset_name == "train":
+ design = self.design_train
+ elif dataset_name == "validation":
+ design = self.design_eval
+ else:
+ # Default to training design if unknown
+ design = self.design_train
+ warnings.warn("Unknown dataset in metric_fn. Using training design.")
+
+ # Calculate loss
+ is_higher_better = False # Lower loss is better, so we don't maximize
+ target = torch.tensor(eval_data.get_label().reshape(-1, 1), dtype=self.dtype)
+ _, loss = self.get_params_loss(predt, target, design)
+
+ return self.loss_name, loss.item(), is_higher_better
+
+ def get_params_loss(
+ self,
+ predt: np.ndarray,
+ target: torch.Tensor,
+ design: torch.Tensor = None,
+ requires_grad: bool = False
+ ) -> Tuple[
+ torch.Tensor, torch.Tensor]:
+ """
+ Transform LightGBM outputs into ARMA parameters and calculate loss.
+
+ This function:
+ 1. Reshapes the raw outputs into the coefficient matrix
+ 2. Multiplies the coefficients with the joint design rows
+ ``[y-lags, residual-lags]``
+ 3. Computes the fit by summing the weighted design entries
+ 4. Calculates the loss between fitted and actual values
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM.
+ target : torch.Tensor
+ Target values (actual time series values).
+ design : torch.Tensor
+ Joint design rows ``[y_{t-1..t-p}, eps_hat_{t-1..t-q}]``,
+ shape ``(n_samples, p + q)``.
+ requires_grad : bool
+ Whether to compute gradients (True during training).
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor]
+ Parameters tensor and loss value.
+ """
+ # Reshape outputs into parameter matrix (samples × n_params)
+ # The 'F' order means Fortran-style ordering (column-major)
+ params = nn.Parameter(
+ torch.tensor(
+ predt.reshape(-1, self.n_params, order="F"),
+ dtype=self.dtype
+ ),
+ requires_grad=requires_grad
+ )
+
+ # Forward pass: Compute the fit by multiplying coefficients with the
+ # design rows and summing
+ fcst = torch.sum(params * design, dim=1, dtype=torch.float32).unsqueeze(1)
+
+ # Calculate loss between fitted and actual values
+ loss = self.loss_fn(fcst, target)
+
+ if self.hessian_method in ("gn", "analytic"):
+ self._fit = fcst
+ self._target = target
+ self._design = design
+
+ return params, loss
+
+ def _calculate_gradients_and_hessians_exact(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
+ """Exact diagonal Hessian via per-parameter second-order autograd.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ params : torch.Tensor
+ Model parameters (ARMA coefficients as an ``nn.Parameter``,
+ shape ``(n_samples, p + q)``).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ loss.backward(create_graph=True)
+ grad = params.grad
+ hess = [
+ autograd(grad[:, i].sum(), params, retain_graph=True)[0][:, i:(i + 1)]
+ for i in range(self.n_params)
+ ]
+
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = torch.cat(hess, dim=1).cpu().detach().numpy().ravel(order="F")
+ params.grad = None
+
+ return grad, hess
+
+ def _calculate_gradients_and_hessians_analytic(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
+ """Closed-form gradients and exact diagonal Hessians via model linearity.
+
+ With the stage-1 residuals frozen as observed regressors, the ARMA fit
+ is linear in its parameters, so ``grad = l'(y_hat) * z`` and
+ ``hess = l''(y_hat) * z**2``, matching the "exact" method for any
+ per-observation loss. MSELoss uses closed-form derivatives; other
+ losses use one small double-backward through ``loss(fit, target)``.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model (unused; derivatives come from the
+ fit/target/design stored by ``get_params_loss``).
+ params : torch.Tensor
+ Model parameters (unused, kept for a uniform dispatch signature).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ fit = self._fit.detach()
+ target = self._target
+ design = self._design
+
+ if isinstance(self.loss_fn, nn.MSELoss) and self.loss_fn.reduction in ("mean", "sum"):
+ # MSE fast path: l' = scale * (y_hat - y), l'' = scale
+ scale = 2.0 / fit.numel() if self.loss_fn.reduction == "mean" else 2.0
+ g = scale * (fit - target)
+ h = torch.full_like(fit, scale)
+ else:
+ # Generic path: per-element first and second loss derivatives via
+ # a double-backward through the (tiny) loss(fit, target) graph.
+ # Requires the loss to have a well-defined double-backward (true
+ # for HuberLoss/SmoothL1Loss and other standard smooth losses).
+ fit_leaf = fit.requires_grad_(True)
+ loss_local = self.loss_fn(fit_leaf, target)
+ g = autograd(loss_local, fit_leaf, create_graph=True)[0]
+ h = autograd(g.sum(), fit_leaf)[0].detach()
+ g = g.detach()
+
+ # Broadcast (N, 1) loss derivatives over the (N, p + q) design matrix.
+ grad = (g * design).cpu().numpy().ravel(order="F")
+ hess = (h * design ** 2).cpu().numpy().ravel(order="F")
+
+ self._fit = None
+ self._target = None
+ self._design = None
+
+ return grad, hess
+
+ def _calculate_gradients_and_hessians_gn(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
+ """Gauss-Newton Hessian diagonal estimated via Hutchinson probing.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ params : torch.Tensor
+ Model parameters (ARMA coefficients as an ``nn.Parameter``,
+ shape ``(n_samples, p + q)``).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ grad = autograd(loss, params, retain_graph=True)[0]
+ rng = torch.Generator().manual_seed(self._iter_count)
+ hess = self._gn_hessian.estimate(self._fit, self._target, params, rng)
+ self._fit = None
+ self._target = None
+ self._design = None
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = hess.cpu().detach().numpy().ravel(order="F")
+
+ return grad, hess
+
+ def _stage1_residual_frame(self, data: pd.DataFrame) -> pd.DataFrame:
+ """Attach the stage-1 in-sample residuals to a sorted copy of *data*.
+
+ Runs the trained stage-1 AR over *data* to compute its one-step
+ in-sample residuals ``eps_hat_t = y_t - y_hat_t``. The first
+ ``stage1_p`` rows of each series have no stage-1 fit and carry NaN.
+
+ Parameters
+ ----------
+ data : pd.DataFrame
+ DataFrame with ``series_id``, ``date``, ``value`` and the
+ training feature columns, ordered by ``(series_id, date)``.
+
+ Returns
+ -------
+ pd.DataFrame
+ Copy of *data*, sorted by ``(series_id, date)``, with an added
+ ``resid`` column (NaN for the first ``stage1_p`` rows per series).
+ """
+ preprocessor = TimeSeriesPreprocessor(
+ freq=self.freq,
+ lags=[i for i in range(1, self.stage1_p + 1)],
+ )
+ lagged = preprocessor.create_lags(data)
+ lagged_dict = preprocessor.extract(lagged)
+
+ # Predict the stage-1 AR coefficients on the lagged rows; enforce the
+ # stage-1 model's training feature order for the Booster.
+ params = np.asarray(
+ self._stage1.model.predict(lagged_dict["features"][self._stage1.features])
+ )
+ # Booster.predict returns (n_rows, stage1_p) for multi-class output
+ if params.ndim == 1:
+ params = params.reshape(-1, self.stage1_p)
+ fit = (params * lagged_dict["lags_target"]).sum(axis=1)
+ resid = lagged_dict["target"].ravel() - fit
+
+ # Align back: `lagged` equals the sorted frame minus the first
+ # stage1_p rows of each series, in the same row order.
+ work = data.sort_values(["series_id", "date"]).reset_index(drop=True).copy()
+ occ = work.groupby("series_id", sort=False).cumcount()
+ work["resid"] = np.nan
+ work.loc[occ >= self.stage1_p, "resid"] = resid
+
+ return work
+
+ def train(
+ self,
+ lgb_params: dict = None,
+ num_iterations: int = 100,
+ train_data: pd.DataFrame = None,
+ validation: bool = False,
+ early_stopping_round: Optional[int] = None,
+ seed: int = 123,
+ verbose: int = -1,
+ deterministic: bool = True,
+ forecast_intervals: Optional[ForecastIntervals] = None,
+ ) -> TrainingResult:
+ """
+ Train the Hyper-Tree-ARMA model on time series data.
+
+ This method:
+ 1. Trains the stage-1 long autoregression (a ``HyperTreeAR`` of order
+ ``stage1_p``) with the same LightGBM hyper-parameters and extracts
+ its in-sample one-step residuals (Hannan-Rissanen)
+ 2. Builds the joint design ``[y-lags, residual-lags]`` and sets up
+ LightGBM datasets
+ 3. Trains the stage-2 ARMA model using gradient boosting
+
+ The training data must contain columns:
+ - 'series_id': Identifier for each time series
+ - 'date': Timestamp for each observation
+ - 'value': Target value to forecast
+ - Additional feature columns used for forecasting
+
+ Each series must have at least ``max(p, stage1_p + q) + 1`` rows so
+ that one stage-2 training row remains. Note that the stage-1 model is
+ fitted on the full training data, so with ``validation=True`` the
+ validation metric shares stage-1 information through the residual
+ regressors.
+
+ Parameters
+ ----------
+ lgb_params : dict
+ LightGBM parameters like 'learning_rate', 'num_leaves', etc.
+ Used for both the stage-1 and the stage-2 GBDT.
+ num_iterations : int
+ Number of boosting rounds for training (both stages)
+ train_data : pd.DataFrame
+ Training data containing series_id, date, value and feature columns
+ validation : bool
+ If True, a validation set will be created for evaluation. It splits the last fcst_h values of each
+ series for validation.
+ early_stopping_round : int, optional
+ If provided, training will stop if the validation loss does not improve for this many rounds.
+ seed : int
+ Random seed for reproducibility
+ verbose : int
+ Verbosity level for LightGBM training
+ deterministic : bool
+ If True, sets LightGBM's ``deterministic`` and ``force_row_wise`` parameters to ensure
+ reproducible results. May slow down training. See
+ https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic
+ forecast_intervals : ForecastIntervals, optional
+ If provided, calibrate conformal prediction intervals via rolling-window
+ cross-validation after the main model is trained. The collected conformity
+ scores are then used by ``forecast(..., level=[...])`` to produce
+ ``-lo-`` / ``-hi-`` columns. See
+ :class:`hypertrees.conformal.ForecastIntervals`.
+
+ Returns
+ -------
+ TrainingResult
+ Object containing evaluation results and training information
+ for the stage-2 model.
+ """
+ # Validate inputs
+ if train_data is None:
+ raise ValueError("train_data must be provided.")
+ if lgb_params is None:
+ raise ValueError("lgb_params must be provided.")
+ if not isinstance(train_data, pd.DataFrame):
+ raise TypeError("train_data must be a pandas DataFrame.")
+ if not isinstance(lgb_params, dict):
+ raise TypeError("lgb_params must be a dictionary.")
+ if not isinstance(num_iterations, int) or num_iterations <= 0:
+ raise ValueError("num_iterations must be a positive integer.")
+ if not isinstance(seed, int):
+ raise TypeError("seed must be an integer.")
+ if not isinstance(verbose, int):
+ raise TypeError("verbose must be an integer.")
+ if early_stopping_round is not None and (not isinstance(early_stopping_round, int) or early_stopping_round <= 0):
+ raise ValueError("early_stopping_round must be a positive integer.")
+ if not isinstance(validation, bool):
+ raise TypeError("validation must be a boolean.")
+ if not isinstance(deterministic, bool):
+ raise TypeError("deterministic must be a boolean.")
+ if forecast_intervals is not None and not isinstance(forecast_intervals, ForecastIntervals):
+ raise TypeError("forecast_intervals must be a ForecastIntervals instance.")
+ if early_stopping_round is not None and not validation:
+ raise ValueError("early_stopping_round can only be used when validation is True.")
+ if validation and early_stopping_round is None:
+ raise ValueError("early_stopping_round must be provided when validation is True.")
+
+ if deterministic:
+ lgb_params = {**lgb_params, "deterministic": True, "force_row_wise": True}
+
+ # Check required columns
+ required_columns = ['series_id', 'date', 'value']
+ for col in required_columns:
+ if col not in train_data.columns:
+ raise ValueError(f"Required column '{col}' not found in training data.")
+
+ # Validate row ordering: each series must be a contiguous block with
+ # monotonic dates so the training reshape and forecast seeds align.
+ validate_series_order(train_data, name="train_data")
+
+ # Resolve the stage-1 long-AR order. The default follows the
+ # Gomez-Maravall (2001) proposal used by statsmodels'
+ # hannan_rissanen and RATS' @HannanRissanen: the long AR grows with
+ # the sample so the residual proxies stay consistent.
+ lengths = train_data.groupby("series_id", sort=False).size()
+ if self._stage1_p_arg is not None:
+ self.stage1_p = self._stage1_p_arg
+ else:
+ t_min = int(lengths.min())
+ self.stage1_p = max(
+ int(np.floor(np.log(t_min) ** 2)), 2 * max(self.p, self.q)
+ )
+
+ # Each series must keep at least one stage-2 training row.
+ needed = max(self.p, self.stage1_p + self.q) + 1
+ bad = lengths[lengths < needed]
+ if len(bad) > 0:
+ raise ValueError(
+ f"Series too short for stage1_p={self.stage1_p} and q={self.q}: "
+ f"each series needs at least max(p, stage1_p + q) + 1 = {needed} "
+ f"rows, but these series are shorter: {bad.to_dict()}. Pass a "
+ f"smaller stage1_p to HyperTreeARMA for short series."
+ )
+
+ # Fail fast if any series is too short for the requested conformal
+ # calibration. The stage-2 ARMA needs max(p, stage1_p + q) + 1 rows to
+ # retain one training sample.
+ if forecast_intervals is not None:
+ validate_calibration_length(
+ train_data, self.fcst_h, forecast_intervals,
+ min_train=max(self.p, self.stage1_p + self.q) + 1,
+ )
+
+ # General model parameters. The objective wrapper stops lgb.train's
+ # params deepcopy from cloning this instance (see NoDeepcopyObjective).
+ self.lgb_params = {
+ "num_class": self.n_params,
+ "objective": NoDeepcopyObjective(self.objective_fn),
+ "metric": "None",
+ "random_seed": seed,
+ "verbose": verbose
+ }
+
+ # Update with user-provided LightGBM parameters
+ self.lgb_params.update(lgb_params)
+
+ # Reset state for re-training
+ self._iter_count = 0
+ self._fit = None
+ self._target = None
+ self._design = None
+ self.model = None
+ self._stage1 = None
+ self.fcst_lags = None
+ self.fcst_eps = None
+ self.dataset_references = {}
+ self.is_trained = False
+ self.features = None
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
+
+ try:
+ # Stage 1 (Hannan-Rissanen): fit the long autoregression and
+ # extract its in-sample one-step residuals as MA-term proxies.
+ self._stage1 = HyperTreeAR(
+ p=self.stage1_p,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ hessian_method="analytic",
+ )
+ self._stage1.train(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ train_data=train_data,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+ work = self._stage1_residual_frame(train_data)
+
+ # Stage 2: build the joint design. The y-lags come from the
+ # standard preprocessor; the residual lags are appended as
+ # lag{p+1}..lag{p+q} so the shared extract()/prepare_datasets
+ # machinery picks up the joint [y-lags | eps-lags] design as one
+ # (n_samples, p + q) tensor while keeping the residual columns
+ # out of the GBDT feature set.
+ preprocessor = TimeSeriesPreprocessor(
+ freq=self.freq,
+ lags=[i for i in range(1, self.p + 1)],
+ )
+ full_ts = preprocessor.create_lags(work.drop(columns=["resid"]))
+
+ resid_grouped = work.groupby("series_id", sort=False)["resid"]
+ elag_names = []
+ elags = {}
+ for i in range(1, self.q + 1):
+ name = f"lag{self.p + i}"
+ elags[name] = resid_grouped.shift(i)
+ elag_names.append(name)
+ occ = work.groupby("series_id", sort=False).cumcount()
+ elag_df = pd.DataFrame(elags)[(occ >= self.p).to_numpy()].reset_index(drop=True)
+ full_ts = pd.concat([full_ts, elag_df], axis=1)
+ # Drop rows without q valid residual lags (the head of each
+ # series up to stage1_p + q observations).
+ full_ts = full_ts.dropna(subset=elag_names).reset_index(drop=True)
+
+ full_dict = preprocessor.extract(full_ts)
+
+ # Store feature names for later use
+ self.features = full_dict["features"].columns.tolist()
+
+ # Prepare datasets
+ (valid_sets,
+ valid_names,
+ callbacks,
+ evals_result,
+ design_train,
+ design_eval,
+ self.dataset_references) = (
+ prepare_datasets(
+ full_ts=full_ts,
+ preprocessor=preprocessor,
+ fcst_h=self.fcst_h,
+ dtype=self.dtype,
+ validation=validation,
+ early_stopping_round=early_stopping_round
+ )
+ )
+
+ # Store design rows for training and evaluation
+ self.design_train = design_train
+ self.design_eval = design_eval
+
+ # Store the value and residual seeds to be used in the forecast method
+ self.set_forecast_origin(train_data)
+
+ # Train LightGBM model
+ start_time = time.time()
+ self.model = lgb.train(
+ self.lgb_params,
+ valid_sets[0],
+ num_boost_round=num_iterations,
+ feval=self.eval_fn if validation else None,
+ valid_sets=valid_sets,
+ valid_names=valid_names,
+ callbacks=callbacks
+ )
+ training_time = time.time() - start_time
+
+ # Set trained flag to True
+ self.is_trained = True
+
+ # Calibrate conformal prediction intervals via rolling-window CV.
+ # Fresh model instances are trained per window (no forecast_intervals
+ # passed, so there is no recursion) using the same hyper-parameters.
+ if forecast_intervals is not None:
+ def _model_factory():
+ return HyperTreeARMA(
+ p=self.p,
+ q=self.q,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ hessian_method=self.hessian_method,
+ n_hessian_probes=self.n_hessian_probes,
+ stage1_p=self.stage1_p,
+ )
+
+ cal_train_kwargs = dict(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+ self._cs_scores, self._cs_series_order = rolling_origin_residuals(
+ model_factory=_model_factory,
+ train_data=train_data,
+ fcst_h=self.fcst_h,
+ forecast_intervals=forecast_intervals,
+ train_kwargs=cal_train_kwargs,
+ )
+ self._pi_config = forecast_intervals
+ self._is_calibrated = True
+
+ # Return results
+ result = TrainingResult(
+ train_metrics=evals_result["train"] if validation else {"loss": []},
+ validation_metrics=evals_result["validation"] if validation else None,
+ best_iteration=self.model.best_iteration if self.model.best_iteration > 0 else num_iterations,
+ training_time=training_time
+
+ )
+
+ return result
+
+ except Exception as e:
+ self.is_trained = False
+ raise RuntimeError(f"Training failed: {str(e)}") from e
+
+ def set_forecast_origin(self, history: pd.DataFrame) -> None:
+ """Re-anchor the ARMA value and residual seeds to the end of *history*.
+
+ Recomputes the last ``p`` observed values and the last ``q`` stage-1
+ residuals per series without retraining either GBDT. Used by conformal
+ calibration with ``refit=False``.
+
+ Parameters
+ ----------
+ history : pd.DataFrame
+ DataFrame with ``series_id``, ``date``, ``value`` and the training
+ feature columns, ordered by ``(series_id, date)`` with each series
+ in a contiguous block. Each series must have at least
+ ``max(p, stage1_p + q)`` observations so that the residual seed
+ exists.
+ """
+ if self._stage1 is None or self._stage1.model is None:
+ raise RuntimeError("set_forecast_origin requires a trained model.")
+ validate_series_order(history, name="history")
+
+ needed = max(self.p, self.stage1_p + self.q)
+ lengths = history.groupby("series_id", sort=False).size()
+ bad = lengths[lengths < needed]
+ if len(bad) > 0:
+ raise ValueError(
+ f"history must contain at least max(p, stage1_p + q) = {needed} "
+ f"observations per series. Series too short: {bad.to_dict()}."
+ )
+
+ # Value seed: last p observations per series, newest first.
+ self.fcst_lags = extract_forecast_lags(history, self.p)
+
+ # Residual seed: last q stage-1 residuals per series, newest first.
+ # The stage-1 residuals are the same quantities the MA coefficients
+ # multiplied during training, keeping train and forecast consistent.
+ work = self._stage1_residual_frame(history)
+ tail = work.groupby("series_id", sort=False).tail(self.q)
+ self.fcst_eps = {
+ sid: grp["resid"].to_numpy()[::-1]
+ for sid, grp in tail.groupby("series_id", sort=False)
+ }
+
+ def forecast(
+ self,
+ test_data: pd.DataFrame,
+ type: str = "forecast",
+ level: Optional[List[int]] = None
+ ) -> pd.DataFrame:
+ """
+ Generate forecasts using the trained model.
+
+ This method:
+ 1. Uses the trained model to forecast ARMA coefficients for each test point
+ 2. Recursively generates forecasts using the forecasted coefficients
+
+ The forecasting process implements an ARMA model where:
+ y_t = φ₁(x)y_{t-1} + ... + φₚ(x)y_{t-p} + θ₁(x)ε_{t-1} + ... + θ_q(x)ε_{t-q}
+
+ Past residuals at the forecast origin are known (stage-1 in-sample
+ errors); future innovations are unobserved with expectation zero, so
+ the MA terms correct the first q horizon steps and then vanish,
+ leaving the pure AR recursion.
+
+ Parameters
+ ----------
+ test_data : pd.DataFrame
+ Test data for which to generate forecasts. Must contain the same
+ feature columns used during training.
+ type : str
+ Type of forecast to generate. Options:
+ - "forecast": Generate forecasted values
+ - "parameters": Return the ARMA coefficients used for forecasting
+ level : list of int, optional
+ Confidence levels (in ``(0, 100)``, e.g. ``[80, 90]``) for conformal
+ prediction intervals. Only valid with ``type="forecast"`` and requires
+ the model to have been trained with ``forecast_intervals=...``. Adds
+ ``-lo-`` / ``-hi-`` columns to the output.
+
+ Returns
+ -------
+ pd.DataFrame
+ Forecasted data with columns:
+ - series_id: Identifier for each time series
+ - date: Forecast date/time
+ - fcst: Forecasted value (if type="forecast")
+ - model: Model name identifier
+ - AR(j) / MA(i): coefficient values (if type="parameters")
+ - -lo- / -hi-: prediction interval bounds
+ (if type="forecast" and level is provided)
+ """
+ # Check if model is trained
+ if not self.is_trained or self.model is None:
+ raise RuntimeError("Model has not been trained. Call train() before forecasting.")
+
+ # Validate input data
+ required_cols = ['series_id', 'date']
+ for col in required_cols:
+ if col not in test_data.columns:
+ raise ValueError(f"Required column '{col}' not found in test_data")
+
+ # Validate row ordering: each series must be a contiguous block with
+ # monotonic dates so the forecast reshape aligns forecasts with seeds.
+ validate_series_order(test_data, name="test_data")
+
+ # Validate series IDs match training data
+ test_series_ids = test_data["series_id"].unique()
+ train_series_ids = set(self.fcst_lags.keys())
+ missing = set(test_series_ids) - train_series_ids
+ extra = train_series_ids - set(test_series_ids)
+ if missing or extra:
+ parts = []
+ if missing:
+ parts.append(f"Missing series in training: {missing}")
+ if extra:
+ parts.append(f"Extra series not in test_data: {extra}")
+ raise ValueError(". ".join(parts))
+
+ # Validate rows per series matches fcst_h (forecast only; parameters
+ # can be requested for arbitrary-length input).
+ if type == "forecast":
+ rows_per_series = test_data.groupby("series_id", sort=False).size()
+ bad = rows_per_series[rows_per_series != self.fcst_h]
+ if not bad.empty:
+ raise ValueError(
+ f"Each series must have exactly fcst_h={self.fcst_h} rows in test_data. "
+ f"Series with wrong counts: {bad.to_dict()}"
+ )
+
+ # Check that all features used during training exist in test_data
+ missing_features = [f for f in self.features if f not in test_data.columns]
+ if missing_features:
+ raise ValueError(f"Missing features in test_data: {missing_features}")
+
+ # Validate type parameter
+ if type not in ["forecast", "parameters"]:
+ raise ValueError("Parameter 'type' must be either 'forecast' or 'parameters'")
+
+ # Validate conformal interval request
+ if level is not None:
+ if type != "forecast":
+ raise ValueError("level is only supported with type='forecast'.")
+ if not self._is_calibrated:
+ raise RuntimeError(
+ "Prediction intervals were requested via level, but the model "
+ "was not calibrated. Pass forecast_intervals=ForecastIntervals(...) "
+ "to train() before forecasting with level."
+ )
+ if not isinstance(level, (list, tuple)) or len(level) == 0:
+ raise ValueError("level must be a non-empty list of integers.")
+ for lv in level:
+ if not isinstance(lv, (int, np.integer)) or not 0 < lv < 100:
+ raise ValueError(f"level values must be integers in (0, 100); got {lv}.")
+
+ model_name = f"Hyper-Tree-ARMA({self.p},{self.q})"
+
+ try:
+
+ if type == "forecast":
+ # Get ARMA coefficient forecasts from the LightGBM model
+ # Shape: (n_series, fcst_h, n_params)
+ n_series_test = len(test_series_ids)
+ params_fcst = self.model.predict(test_data[self.features]).reshape(n_series_test, self.fcst_h, self.n_params)
+
+ # Reconstruct the seed states in the same order as test data
+ lags = np.array([self.fcst_lags[series_id] for series_id in test_series_ids])
+ eps = np.array([self.fcst_eps[series_id] for series_id in test_series_ids])
+
+ # Generate multi-step forecasts
+ forecasts = []
+ for h in range(self.fcst_h):
+ # Compute next value using the ARMA equation:
+ # y_t = φ₁y_{t-1} + ... + φₚy_{t-p} + θ₁ε_{t-1} + ... + θ_qε_{t-q}
+ next_val = (
+ np.sum(params_fcst[:, h, :self.p] * lags, axis=1)
+ + np.sum(params_fcst[:, h, self.p:] * eps, axis=1)
+ ).reshape(-1, 1)
+ forecasts.append(next_val)
+
+ # Update the value lags with the new forecast; future
+ # innovations are unobserved with expectation zero, so the
+ # residual state is shifted with zeros (the MA terms die
+ # out after q steps).
+ lags = np.concatenate([next_val, lags[:, :-1]], axis=1)
+ eps = np.concatenate([np.zeros((n_series_test, 1)), eps[:, :-1]], axis=1)
+
+ # Create output dataframe based on requested type
+ out_df = pd.DataFrame({
+ "series_id": test_data["series_id"].to_numpy().flatten(),
+ "date": test_data["date"].to_numpy().flatten(),
+ "fcst": np.hstack(forecasts).flatten(),
+ "model": model_name,
+ })
+
+ # Append conformal prediction intervals if requested.
+ if level is not None:
+ point = np.hstack(forecasts) # (n_series_test, fcst_h)
+ columns = interval_columns(
+ point=point,
+ scores=self._cs_scores,
+ levels=level,
+ method=self._pi_config.method,
+ model_name=model_name,
+ cal_order=self._cs_series_order,
+ target_order=list(test_series_ids),
+ )
+ for col_name, values in columns.items():
+ out_df[col_name] = values
+
+ elif type == "parameters":
+ params_fcst = np.asarray(self.model.predict(test_data[self.features]))
+ # Booster.predict returns (n_test, p + q) for multi-class output
+ if params_fcst.ndim == 1:
+ params_fcst = params_fcst.reshape(-1, self.n_params)
+ out_df = pd.DataFrame({
+ "series_id": test_data["series_id"].to_numpy().flatten(),
+ "date": test_data["date"].to_numpy().flatten(),
+ "model": model_name,
+ })
+ # Add the AR and MA coefficients to the dataframe
+ for j in range(self.p):
+ out_df[f"AR({j + 1})"] = params_fcst[:, j].flatten()
+ for i in range(self.q):
+ out_df[f"MA({i + 1})"] = params_fcst[:, self.p + i].flatten()
+
+ return out_df
+
+ except Exception as e:
+ raise RuntimeError(f"Forecasting not successful: {str(e)}") from e
diff --git a/hypertrees/models/HyperTreeETS.py b/hypertrees/models/HyperTreeETS.py
index df5d265..a67490b 100644
--- a/hypertrees/models/HyperTreeETS.py
+++ b/hypertrees/models/HyperTreeETS.py
@@ -7,10 +7,17 @@
from typing import Tuple, List, Callable, Optional
import time
import random
+import warnings
from ..utils import CustomLogger
lgb.register_logger(CustomLogger())
-from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order, GaussNewtonHessian
+from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order, GaussNewtonHessian, NoDeepcopyObjective
+from ..conformal import (
+ ForecastIntervals,
+ validate_calibration_length,
+ rolling_origin_residuals,
+ interval_columns,
+)
class HyperTreeETS:
"""
@@ -28,7 +35,8 @@ class HyperTreeETS:
- Combines tree-based models (LightGBM) with exponential smoothing time series modeling
- Allows ETS parameters to vary based on features
- Provides ETS parameters that can vary over time
- - Supports both triple exponential smoothing (with seasonality) and trend-only models
+ - Supports triple exponential smoothing with multiplicative ("triple") or
+ additive ("additive") seasonality, as well as trend-only models
Use this model when:
- You have relevant features that might influence the smoothing structure
@@ -92,6 +100,7 @@ def __init__(
fcst_h: int = 12,
loss_fn: Callable = nn.MSELoss(),
n_hessian_probes: int = 5,
+ seasonal_init: str = "classical",
):
"""
Initialize the Hyper-Tree-ETS model.
@@ -99,13 +108,21 @@ def __init__(
Arguments
----------
ets_type : str
- Type of ETS model to use. Either "triple" (with seasonality) or "trend" (linear trend-only).
+ Type of ETS model to use. Options:
+ - "triple": Holt-Winters with *multiplicative* seasonality and a
+ damped trend. Requires strictly positive series.
+ - "additive": Holt-Winters with *additive* seasonality and a
+ damped trend. Use when seasonal swings are roughly constant in
+ absolute size, or when series contain zeros or negative values
+ (where multiplicative seasonality breaks down).
+ - "trend": linear trend-only (no seasonality).
season_length : int
Seasonal length of the time series (e.g., 12 for monthly data, 4 for quarterly).
seasonality_feature : str
Feature name for seasonality. This is used to create seasonal indices. Must be present in the dataset.
For example, "month" for monthly data, "quarter" for quarterly data, etc. This is required when
- ets_type is "triple".
+ ets_type is "triple" or "additive". Values must be 1-based season positions in [1, season_length];
+ shift 0-based features (e.g. pandas dayofweek in 0..6) by +1.
freq : str
Frequency of the time series (e.g., 'D' for daily, 'M' for monthly,
'Q' for quarterly, 'Y' for yearly).
@@ -113,16 +130,28 @@ def __init__(
Forecast horizon (number of periods to forecast ahead).
loss_fn : Callable
Loss function for optimization. Must be a PyTorch loss function.
- Default is MSE loss. Must be twice-differentiable for the
- Gauss-Newton Hessian; non-smooth losses (e.g., L1Loss) have
- zero or undefined second derivatives, causing degenerate Hessians.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
n_hessian_probes : int
Number of Hutchinson probes for Gauss-Newton Hessian diagonal estimation.
More probes reduce variance but increase computation. Default is 5.
+ seasonal_init : str
+ Initialization of the seasonal/level/trend states for the triple
+ ETS forward pass (ignored for ets_type="trend"; ets_type="additive"
+ always uses its classical additive estimator). Options:
+ - "classical" (default): decomposition-based estimator following
+ R's forecast::ets and statsforecast (centered 2 x m moving-average
+ detrending, slot-aligned seasonal indices, OLS level/trend seed).
+ - "legacy": the exact pre-0.2.0 initialization, kept verbatim for
+ reproducing earlier results (including the paper benchmarks).
"""
# Validate inputs
- if ets_type not in ["triple", "trend"]:
- raise ValueError("ets_type must be either 'triple' or 'trend'.")
+ if ets_type not in ["triple", "additive", "trend"]:
+ raise ValueError("ets_type must be one of 'triple', 'additive', or 'trend'.")
+ if seasonal_init not in ["classical", "legacy"]:
+ raise ValueError("seasonal_init must be either 'classical' or 'legacy'.")
if season_length <= 0:
raise ValueError("season_length must be a positive integer.")
if not isinstance(season_length, int):
@@ -131,8 +160,20 @@ def __init__(
raise ValueError("Forecast horizon 'fcst_h' must be a positive integer.")
if not isinstance(loss_fn, nn.Module):
raise TypeError("loss_fn must be a PyTorch loss function.")
+ if isinstance(loss_fn, nn.L1Loss):
+ raise ValueError(
+ "nn.L1Loss is not supported: its second derivative is zero almost "
+ "everywhere, so LightGBM's Newton boosting receives all-zero Hessians "
+ "and cannot grow trees. Use nn.HuberLoss or nn.SmoothL1Loss for an "
+ "MAE-like loss with usable curvature."
+ )
+ if getattr(loss_fn, "reduction", "mean") == "none":
+ raise ValueError(
+ "loss_fn must use a scalar reduction ('mean' or 'sum'); "
+ "reduction='none' returns per-element losses that the "
+ "boosting objective cannot consume."
+ )
if not isinstance(loss_fn, nn.MSELoss):
- import warnings
warnings.warn(
f"Loss {type(loss_fn).__name__} is not nn.MSELoss. The Gauss-Newton "
"Hessian requires a twice-differentiable loss; non-smooth losses "
@@ -142,14 +183,15 @@ def __init__(
)
if not isinstance(freq, str):
raise TypeError("freq must be a string representing the frequency of the time series.")
- if seasonality_feature is None and ets_type == "triple":
- raise ValueError("seasonality_feature must be provided for triple ETS type.")
+ if seasonality_feature is None and ets_type in ("triple", "additive"):
+ raise ValueError(f"seasonality_feature must be provided for {ets_type} ETS type.")
self.ets_type = ets_type
self.season_length = season_length
self.seasonality_feature = seasonality_feature
+ self.seasonal_init = seasonal_init
self.freq = freq
- self.n_params = 4 if ets_type == "triple" else 2 # alpha, beta, gamma, phi OR alpha, beta
+ self.n_params = 4 if ets_type in ("triple", "additive") else 2 # alpha, beta, gamma, phi OR alpha, beta
self.fcst_h = fcst_h
self.loss_fn = loss_fn
self.loss_name = self.loss_fn.__class__.__name__
@@ -162,16 +204,31 @@ def __init__(
self.fcst_states = None # Store final ETS states for forecasting
self.n_hessian_probes = n_hessian_probes
self._iter_count = 0 # Iteration counter for seeding Hessian probes
+ self._init_cache = {} # Per-dataset cache of _init_triple_states results
+ # Recursive h-step validation metric: the terminal level/trend/seasonality
+ # states from the "train" eval call are stashed in _eval_boundary and
+ # consumed by the "validation" eval call of the same boosting iteration
+ # (valid_sets order is [train, validation]); see eval_fn.
+ self._last_states = None
+ self._eval_boundary = None
# Shared Gauss-Newton Hessian estimator
self._gn_hessian = GaussNewtonHessian(loss_fn, n_hessian_probes, self.dtype)
+ # Conformal prediction interval state
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
+
# Activation function for parameter bounds
self.sigmoid_fn = nn.Sigmoid()
# Set the appropriate forward function based on ETS type
if self.ets_type == "triple":
self.forward = self._forward_triple
+ elif self.ets_type == "additive":
+ self.forward = self._forward_additive
elif self.ets_type == "trend":
self.forward = self._forward_trend
@@ -206,6 +263,224 @@ def _create_mask_from_data(self, data: pd.DataFrame) -> torch.Tensor:
return mask
+ def _seasonal_positions(self, values) -> torch.Tensor:
+ """Convert 1-based seasonal feature values to 0-based tensor indices.
+
+ Validates values lie in ``[1, season_length]``; a 0-based feature
+ (e.g. pandas ``dayofweek``) would silently wrap into the wrong slot.
+
+ Parameters
+ ----------
+ values : array-like
+ Raw values of the ``seasonality_feature`` column.
+
+ Returns
+ -------
+ torch.Tensor
+ 0-based seasonal positions as a flat ``torch.long`` tensor.
+ """
+ idx = torch.tensor(np.asarray(values), dtype=torch.long) - 1
+ if idx.numel() > 0:
+ lo = int(idx.min())
+ hi = int(idx.max())
+ if lo < 0 or hi >= self.season_length:
+ raise ValueError(
+ f"seasonality_feature '{self.seasonality_feature}' must contain "
+ f"1-based season positions in [1, {self.season_length}]; got values "
+ f"in [{lo + 1}, {hi + 1}]. Shift 0-based features (e.g. pandas "
+ f"dayofweek) by +1."
+ )
+ return idx
+
+ def _init_triple_states(
+ self,
+ target: torch.Tensor,
+ mask: torch.Tensor,
+ seasonality_idxs: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Initial seasonal indices and level/trend states for triple ETS.
+
+ ``seasonal_init="legacy"`` reproduces the pre-0.2.0 initialization
+ verbatim (flat seasonal profile), required to reproduce earlier
+ results including the paper benchmarks. ``"classical"`` (default)
+ follows R's ``forecast::ets`` / statsforecast: centered 2 x m
+ moving-average detrending, per-slot ratio averages normalized to mean
+ one, and an OLS level/trend seed on the seasonally-adjusted head.
+ Indices are assigned via ``seasonality_idxs`` so they land in the
+ slots the recursion reads; short series and empty slots fall back to
+ a simple per-slot average. All statistics are mask-aware.
+
+ Parameters
+ ----------
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding),
+ shape ``(n_series, T)``.
+ seasonality_idxs : torch.Tensor
+ 0-based seasonal slot per observation, shape ``(n_series, T)``.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor, torch.Tensor]
+ Seasonal indices ``(n_series, season_length)``, initial level
+ ``(n_series,)``, and initial trend ``(n_series,)``.
+ """
+ N, T = target.shape
+ m = self.season_length
+
+ if self.seasonal_init == "legacy":
+ # Pre-0.2.0 initialization, kept verbatim (including its positional
+ # slot assignment and flat resulting profile) so that results from
+ # earlier releases and the paper remain reproducible.
+ init_len = min(m, T)
+ seasonal_avg = torch.zeros((N, init_len), dtype=self.dtype)
+ for i in range(init_len):
+ valid_obs = mask[:, i::init_len]
+ seasonal_avg[:, i] = (target[:, i::init_len] * valid_obs.float()).sum(
+ 1) / valid_obs.float().sum(1).clamp(min=1)
+
+ seasonality = target[:, :init_len] / (seasonal_avg + self.eps)
+ if init_len < m:
+ # Pad with ones for missing season positions
+ pad = torch.ones((N, m - init_len), dtype=self.dtype)
+ seasonality = torch.cat([seasonality, pad], dim=1)
+
+ season_adj = target[:, :init_len] / seasonality[:, :init_len]
+ level0 = season_adj[:, 0]
+ trend0 = season_adj[:, min(1, init_len - 1)] - season_adj[:, 0]
+ return seasonality, level0, trend0
+
+ slot_ids = torch.arange(N, dtype=torch.long).unsqueeze(1) * m + seasonality_idxs # (N, T)
+ ym = target * mask
+
+ def slot_sum(ids: torch.Tensor, values: torch.Tensor) -> torch.Tensor:
+ """Sum ``values`` into their (series, slot) cells -> (N, m)."""
+ return torch.zeros(N * m, dtype=self.dtype).index_add_(
+ 0, ids.reshape(-1), values.reshape(-1)
+ ).reshape(N, m)
+
+ # --- Fallback estimator: per-slot average of y over the series mean --
+ cnts = slot_sum(slot_ids, mask)
+ slot_mean = slot_sum(slot_ids, ym) / cnts.clamp(min=1.0)
+ grand_mean = ym.sum(dim=1) / mask.sum(dim=1).clamp(min=1.0)
+ simple_idx = torch.where(
+ cnts > 0,
+ slot_mean / (grand_mean.unsqueeze(1) + self.eps),
+ torch.ones_like(slot_mean),
+ )
+ seasonality = simple_idx
+
+ # --- Detrended estimator: centered 2 x m MA, then per-slot ratios ----
+ L = m + 1 if m % 2 == 0 else m # always odd
+ if T >= L:
+ kernel = torch.full((1, 1, L), 1.0 / m, dtype=self.dtype)
+ if m % 2 == 0:
+ kernel[0, 0, 0] = 0.5 / m
+ kernel[0, 0, -1] = 0.5 / m
+ trend_ma = torch.nn.functional.conv1d(
+ ym.unsqueeze(1), kernel
+ ).squeeze(1) # (N, T - L + 1), centered at offset L // 2
+ window_valid = torch.nn.functional.conv1d(
+ mask.unsqueeze(1), torch.ones((1, 1, L), dtype=self.dtype)
+ ).squeeze(1) >= L - 0.5 # full window unmasked (implies a valid center)
+
+ half = L // 2
+ y_c = target[:, half:T - half] # window centers, length T - L + 1
+ valid = window_valid & (trend_ma > self.eps)
+ ratios = torch.where(
+ valid, y_c / trend_ma.clamp(min=self.eps), torch.zeros_like(y_c)
+ )
+
+ r_cnts = slot_sum(slot_ids[:, half:T - half], valid.to(self.dtype))
+ detr_idx = slot_sum(slot_ids[:, half:T - half], ratios) / r_cnts.clamp(min=1.0)
+ seasonality = torch.where(r_cnts > 0, detr_idx, simple_idx)
+
+ # Normalize to mean one and guard against tiny indices
+ # (statsforecast clips initial seasonal states at 1e-2 as well).
+ seasonality = seasonality / seasonality.mean(dim=1, keepdim=True).clamp(min=self.eps)
+ seasonality = seasonality.clamp(min=1e-2)
+
+ # --- Level/trend: OLS on the seasonally-adjusted head (as in ets) ----
+ s_t = seasonality.gather(1, seasonality_idxs)
+ y_sa = target / s_t.clamp(min=self.eps)
+ maxn = min(max(10, 2 * m), T)
+ t_idx = torch.arange(1, maxn + 1, dtype=self.dtype).unsqueeze(0)
+ w = mask[:, :maxn]
+ sw = w.sum(dim=1).clamp(min=1.0)
+ mean_t = (w * t_idx).sum(dim=1) / sw
+ mean_y = (w * y_sa[:, :maxn]).sum(dim=1) / sw
+ dev_t = t_idx - mean_t.unsqueeze(1)
+ var_t = (w * dev_t ** 2).sum(dim=1)
+ cov_ty = (w * dev_t * (y_sa[:, :maxn] - mean_y.unsqueeze(1))).sum(dim=1)
+ trend0 = torch.where(
+ var_t > self.eps,
+ cov_ty / var_t.clamp(min=self.eps),
+ torch.zeros_like(cov_ty),
+ )
+ # Evaluate the line at the first observation (t = 1 on the OLS axis)
+ # so level0 matches this recursion's timing, which seeds the state at
+ # the first observation rather than one step before it.
+ level0 = mean_y + trend0 * (1.0 - mean_t)
+
+ return seasonality, level0, trend0
+
+ def _cached_init_triple_states(
+ self,
+ data: lgb.Dataset,
+ target: torch.Tensor,
+ mask: torch.Tensor,
+ seasonality_idxs: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Cache wrapper around :meth:`_init_triple_states`.
+
+ The initialization depends only on the data, which is constant across
+ boosting iterations, so results are cached per lgb.Dataset. Entries
+ pin the dataset object, so a key hit proves it is that exact,
+ still-alive dataset. The seasonal indices are cloned on every return
+ because the recursion updates them in place; the cache is cleared by
+ ``train()``.
+
+ Parameters
+ ----------
+ data : lgb.Dataset
+ Dataset whose identity serves as the cache key.
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding), shape ``(n_series, T)``.
+ seasonality_idxs : torch.Tensor
+ 0-based seasonal slot per observation, shape ``(n_series, T)``.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor, torch.Tensor]
+ Seasonal indices ``(n_series, season_length)``, initial level
+ ``(n_series,)``, and initial trend ``(n_series,)``.
+ """
+ key = id(data)
+ entry = self._init_cache.get(key)
+ if entry is not None:
+ return entry["seasonality"].clone(), entry["level0"], entry["trend0"]
+
+ seasonality, level0, trend0 = self._init_triple_states(
+ target, mask, seasonality_idxs
+ )
+ if len(self._init_cache) > 8:
+ # Bound growth from one-shot datasets (e.g. _store_final_states,
+ # conformal re-anchoring); the persistent train/eval entries are
+ # simply recomputed once after a clear.
+ self._init_cache.clear()
+ # Storing the dataset pins its id: a key hit therefore implies this
+ # exact dataset, and with it identical target/mask/positions.
+ self._init_cache[key] = {
+ "data": data,
+ "seasonality": seasonality,
+ "level0": level0,
+ "trend0": trend0,
+ }
+ return seasonality.clone(), level0, trend0
+
def objective_fn(self, predt: np.ndarray, data: lgb.Dataset) -> Tuple[np.ndarray, np.ndarray]:
"""
Custom objective function for LightGBM training.
@@ -255,17 +530,110 @@ def eval_fn(self, predt: np.ndarray, eval_data: lgb.Dataset) -> Tuple[str, float
-------
Tuple[str, float, bool]
Name of the metric, value of the metric, and whether to maximize it.
+
+ Notes
+ -----
+ The validation metric is the **recursive h-step forecast** loss, not the
+ in-sample one-step fit. Rolling the deployment recursion from the
+ training-window terminal states and scoring against the holdout measures
+ the quantity the model is actually used for. The naive in-sample
+ validation loss is degenerate for the seasonal variants when
+ ``fcst_h <= season_length`` (every state update becomes a
+ parameter-independent fixed point, so the loss is ~0 for *any*
+ parameters and early stopping selects noise). The training metric
+ remains the in-sample one-step fit.
"""
- # Calculate loss
is_higher_better = False # Lower loss is better, so we don't maximize
+ dataset_name = self.dataset_references.get(id(eval_data), "unknown")
target = torch.tensor(
eval_data.get_label().reshape(self.n_series, -1),
dtype=self.dtype
)
+
+ if dataset_name == "validation" and self._eval_boundary is not None:
+ # Recursive h-step forecast metric. The terminal states were stashed
+ # during this same iteration's "train" eval call, so the boundary
+ # states and the horizon parameters come from the identical model
+ # state (no off-by-one tree).
+ loss = self._recursive_eval_loss(predt, eval_data, target)
+ loss_val = loss.item()
+ if not np.isfinite(loss_val):
+ # A diverged rollout early in boosting would otherwise feed NaN
+ # to early stopping; report a large finite value (worst) instead.
+ loss_val = float(np.finfo(np.float32).max)
+ return self.loss_name, loss_val, is_higher_better
+
+ # Train metric (and validation fallback before the first boundary is
+ # stashed): the teacher-forced in-sample one-step loss.
_, loss = self.get_params_loss(predt, target, eval_data)
+ if dataset_name == "train":
+ # Stash terminal states for the validation rollout that follows in
+ # this same iteration.
+ self._eval_boundary = self._last_states
return self.loss_name, loss.item(), is_higher_better
+ def _recursive_eval_loss(
+ self,
+ predt: np.ndarray,
+ eval_data: lgb.Dataset,
+ target: torch.Tensor,
+ ) -> torch.Tensor:
+ """Recursive h-step forecast loss for the validation split.
+
+ Mirrors deployment: the predicted parameters over the validation window
+ drive the same rolled recursion as :meth:`forecast` (via the shared
+ :meth:`_roll_forecast` helper), starting from the training-window
+ terminal states stored in ``self._eval_boundary``. Padded holdout rows
+ (mask == 0) are excluded from the loss.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw LightGBM outputs over the validation rows (class-major order).
+ eval_data : lgb.Dataset
+ Validation dataset (provides the seasonality feature and mask).
+ target : torch.Tensor
+ Holdout observations, shape ``(n_series, fcst_h)``.
+
+ Returns
+ -------
+ torch.Tensor
+ Scalar loss between the rolled forecasts and the holdout.
+ """
+ params = torch.clamp(
+ self.sigmoid_fn(
+ torch.tensor(
+ predt.reshape(-1, self.n_params, order="F"),
+ dtype=self.dtype,
+ ).reshape(self.n_series, -1, self.n_params)
+ ),
+ min=self.eps,
+ max=1 - self.eps,
+ )
+
+ level_h, trend_h, seasonality = self._eval_boundary
+ if self.ets_type in ("triple", "additive"):
+ seasonality_idxs = self._seasonal_positions(
+ eval_data.data[self.seasonality_feature].values
+ ).reshape(self.n_series, -1)
+ else:
+ seasonality_idxs = None
+
+ fcsts = self._roll_forecast(
+ level_h, trend_h, seasonality, params, seasonality_idxs
+ )
+
+ if "mask" in eval_data.data.columns:
+ mask = torch.tensor(
+ eval_data.data["mask"].values.reshape(self.n_series, -1),
+ dtype=self.dtype,
+ )
+ else:
+ mask = torch.ones_like(target)
+
+ return self.loss_fn(fcsts * mask, target * mask)
+
def get_params_loss(
self,
predt: np.ndarray,
@@ -325,8 +693,12 @@ def get_params_loss(
series_len = target.shape[1]
mask = torch.ones((self.n_series, series_len), dtype=self.dtype)
- # Forward pass to compute fitted values
- _, _, _, fit = self.forward(params, data, target, mask)
+ # Forward pass to compute fitted values. Keep the terminal level/
+ # trend/seasonality states so the recursive validation metric can roll
+ # the deployment recursion forward from the training-window boundary
+ # (see eval_fn / _recursive_eval_loss).
+ last_level, last_trend, seasonality, fit = self.forward(params, data, target, mask)
+ self._last_states = (last_level, last_trend, seasonality)
# Stack fitted values and compute loss with masking
fit = torch.stack(fit, dim=1)
@@ -355,6 +727,17 @@ def _forward_triple(
- Seasonality: s_t = γ(y_t/(l_{t-1} + φb_{t-1})) + (1-γ)s_{t-m}
- Fitted: ŷ_t = (l_{t-1} + φb_{t-1}) * s_{t-m}
+ Parameters
+ ----------
+ params : torch.Tensor
+ Sigmoid-transformed ETS parameters, shape ``(n_series, T, n_params)``.
+ data : lgb.Dataset
+ LightGBM dataset whose raw DataFrame provides the seasonality feature.
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding), shape ``(n_series, T)``.
+
Returns
-------
Tuple[torch.Tensor, torch.Tensor, torch.Tensor, List[torch.Tensor]]
@@ -370,59 +753,330 @@ def _forward_triple(
gamma_t = gamma.unbind(dim=1)
phi_t = phi.unbind(dim=1)
- # Initialize seasonality
- init_len = min(self.season_length, series_len)
- seasonal_avg = torch.zeros((self.n_series, init_len), dtype=self.dtype)
-
- # Compute initial seasonal averages
- for i in range(init_len):
- valid_obs = mask[:, i::init_len]
- seasonal_avg[:, i] = (target[:, i::init_len] * valid_obs.float()).sum(
- 1) / valid_obs.float().sum(1).clamp(min=1)
-
- # Initialize seasonality as ratios since we are using multiplicative seasonality
- seasonality = target[:, :init_len] / (seasonal_avg + self.eps)
- if init_len < self.season_length:
- # Pad with ones for missing season positions
- pad = torch.ones((self.n_series, self.season_length - init_len), dtype=self.dtype)
- seasonality = torch.cat([seasonality, pad], dim=1)
-
- # ETS initialization using season-adjusted initialization
- season_adj = target[:, :init_len] / seasonality[:, :init_len]
- level_prev = season_adj[:, 0]
- trend_prev = season_adj[:, min(1, init_len - 1)] - season_adj[:, 0]
+ # Get seasonal indices from features first: the state initialization
+ # assigns initial indices to the same slots the recursion reads, so a
+ # series that does not start at season position 1 is not rotated.
+ seasonality_idxs = self._seasonal_positions(
+ data.data[self.seasonality_feature].values
+ ).reshape(self.n_series, -1)
+ batch_idx = torch.arange(self.n_series, dtype=torch.long)
+
+ # Initialize seasonal indices and level/trend states via the classical
+ # decomposition-based estimator. The result depends only on the data
+ # (never on the boosted parameters), so it is cached per lgb.Dataset
+ # and reused across boosting iterations.
+ seasonality, level_prev, trend_prev = self._cached_init_triple_states(
+ data, target, mask, seasonality_idxs
+ )
fits = [target[:, 0]]
- # Get seasonal indices from features
- seasonality_idxs = torch.tensor(
- data.data[self.seasonality_feature].values - 1,
- dtype=torch.long
+ # Pre-unbind the data tensors so the loop indexes Python tuples
+ # instead of slicing tensors at every step.
+ target_t = target.unbind(dim=1)
+ mask_t = mask.unbind(dim=1)
+ idxs_t = seasonality_idxs.unbind(dim=1)
+
+ # Triple ETS updates with masking for padded values. Shared
+ # subexpressions are hoisted: every node created here is re-traversed
+ # by each backward pass (gradient plus the Hutchinson probes), so a
+ # smaller graph speeds up the forward and every backward.
+ for t in range(1, series_len):
+ valid_mask = mask_t[t]
+ invalid_mask = 1 - valid_mask
+ y_t = target_t[t]
+ s_prev = seasonality[batch_idx, idxs_t[t]] # s_{t-m}
+ phi_trend = phi_t[t] * trend_prev # phi_t * b_{t-1}
+ pred_base = level_prev + phi_trend # l_{t-1} + phi_t * b_{t-1}
+
+ fit_t = valid_mask * (pred_base * s_prev) + invalid_mask * fits[-1]
+
+ level_new = valid_mask * (
+ alpha_t[t] * (y_t / s_prev) +
+ (1 - alpha_t[t]) * pred_base
+ ) + invalid_mask * level_prev
+
+ trend_new = valid_mask * (
+ beta_t[t] * (level_new - level_prev) +
+ (1 - beta_t[t]) * phi_trend
+ ) + invalid_mask * trend_prev
+
+ seasonality[batch_idx, idxs_t[t]] = valid_mask * (
+ gamma_t[t] * (y_t / pred_base) +
+ (1 - gamma_t[t]) * s_prev
+ ) + invalid_mask * s_prev
+
+ fits.append(fit_t)
+ level_prev = level_new
+ trend_prev = trend_new
+
+ return level_prev, trend_prev, seasonality, fits
+
+ def _init_additive_states(
+ self,
+ target: torch.Tensor,
+ mask: torch.Tensor,
+ seasonality_idxs: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Initial seasonal indices and level/trend states for additive ETS.
+
+ Additive counterpart of the classical ``_init_triple_states``
+ estimator, following R's ``forecast::ets`` / statsmodels' heuristic
+ initialization [1, 2]: the trend is removed with a centered 2 x m
+ moving average, the detrended *differences* ``y - trend`` are averaged
+ per seasonal slot and normalized to mean zero, and level/trend are
+ seeded by an OLS fit on the seasonally adjusted head ``y - s``.
+ Indices are assigned via ``seasonality_idxs`` so they land in the
+ slots the recursion reads; short series and empty slots fall back to a
+ simple per-slot deviation from the series mean. All statistics are
+ mask-aware.
+
+ References
+ ----------
+ [1] Hyndman, R. J., Koehler, A. B., Ord, J. K., & Snyder, R. D.
+ (2008). Forecasting with Exponential Smoothing: The State Space
+ Approach. Springer. (Initialization heuristic, Section 2.6.1)
+ [2] Hyndman, R. J., & Athanasopoulos, G. (2021). Forecasting:
+ Principles and Practice (3rd ed.). OTexts.
+ https://otexts.com/fpp3/
+
+ Parameters
+ ----------
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding),
+ shape ``(n_series, T)``.
+ seasonality_idxs : torch.Tensor
+ 0-based seasonal slot per observation, shape ``(n_series, T)``.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor, torch.Tensor]
+ Seasonal indices ``(n_series, season_length)``, initial level
+ ``(n_series,)``, and initial trend ``(n_series,)``.
+ """
+ N, T = target.shape
+ m = self.season_length
+
+ slot_ids = torch.arange(N, dtype=torch.long).unsqueeze(1) * m + seasonality_idxs # (N, T)
+ ym = target * mask
+
+ def slot_sum(ids: torch.Tensor, values: torch.Tensor) -> torch.Tensor:
+ """Sum ``values`` into their (series, slot) cells -> (N, m)."""
+ return torch.zeros(N * m, dtype=self.dtype).index_add_(
+ 0, ids.reshape(-1), values.reshape(-1)
+ ).reshape(N, m)
+
+ # --- Fallback estimator: per-slot deviation from the series mean ----
+ cnts = slot_sum(slot_ids, mask)
+ slot_mean = slot_sum(slot_ids, ym) / cnts.clamp(min=1.0)
+ grand_mean = ym.sum(dim=1) / mask.sum(dim=1).clamp(min=1.0)
+ simple_idx = torch.where(
+ cnts > 0,
+ slot_mean - grand_mean.unsqueeze(1),
+ torch.zeros_like(slot_mean),
+ )
+ seasonality = simple_idx
+
+ # --- Detrended estimator: centered 2 x m MA, then per-slot diffs ----
+ L = m + 1 if m % 2 == 0 else m # always odd
+ if T >= L:
+ kernel = torch.full((1, 1, L), 1.0 / m, dtype=self.dtype)
+ if m % 2 == 0:
+ kernel[0, 0, 0] = 0.5 / m
+ kernel[0, 0, -1] = 0.5 / m
+ trend_ma = torch.nn.functional.conv1d(
+ ym.unsqueeze(1), kernel
+ ).squeeze(1) # (N, T - L + 1), centered at offset L // 2
+ window_valid = torch.nn.functional.conv1d(
+ mask.unsqueeze(1), torch.ones((1, 1, L), dtype=self.dtype)
+ ).squeeze(1) >= L - 0.5 # full window unmasked (implies a valid center)
+
+ half = L // 2
+ y_c = target[:, half:T - half] # window centers, length T - L + 1
+ diffs = torch.where(
+ window_valid, y_c - trend_ma, torch.zeros_like(y_c)
+ )
+
+ d_cnts = slot_sum(slot_ids[:, half:T - half], window_valid.to(self.dtype))
+ detr_idx = slot_sum(slot_ids[:, half:T - half], diffs) / d_cnts.clamp(min=1.0)
+ seasonality = torch.where(d_cnts > 0, detr_idx, simple_idx)
+
+ # Normalize to mean zero (additive seasonal indices sum to ~0)
+ seasonality = seasonality - seasonality.mean(dim=1, keepdim=True)
+
+ # --- Level/trend: OLS on the seasonally-adjusted head (as in ets) ----
+ s_t = seasonality.gather(1, seasonality_idxs)
+ y_sa = target - s_t
+ maxn = min(max(10, 2 * m), T)
+ t_idx = torch.arange(1, maxn + 1, dtype=self.dtype).unsqueeze(0)
+ w = mask[:, :maxn]
+ sw = w.sum(dim=1).clamp(min=1.0)
+ mean_t = (w * t_idx).sum(dim=1) / sw
+ mean_y = (w * y_sa[:, :maxn]).sum(dim=1) / sw
+ dev_t = t_idx - mean_t.unsqueeze(1)
+ var_t = (w * dev_t ** 2).sum(dim=1)
+ cov_ty = (w * dev_t * (y_sa[:, :maxn] - mean_y.unsqueeze(1))).sum(dim=1)
+ trend0 = torch.where(
+ var_t > self.eps,
+ cov_ty / var_t.clamp(min=self.eps),
+ torch.zeros_like(cov_ty),
+ )
+ # Evaluate the line at the first observation (t = 1 on the OLS axis)
+ # so level0 matches this recursion's timing, which seeds the state at
+ # the first observation rather than one step before it.
+ level0 = mean_y + trend0 * (1.0 - mean_t)
+
+ return seasonality, level0, trend0
+
+ def _cached_init_additive_states(
+ self,
+ data: lgb.Dataset,
+ target: torch.Tensor,
+ mask: torch.Tensor,
+ seasonality_idxs: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """Cache wrapper around :meth:`_init_additive_states`.
+
+ Identical caching semantics to :meth:`_cached_init_triple_states`:
+ results are cached per lgb.Dataset (pinning the dataset object), the
+ seasonal indices are cloned on every return because the recursion
+ updates them in place, and the cache is cleared by ``train()``.
+
+ Parameters
+ ----------
+ data : lgb.Dataset
+ Dataset whose identity serves as the cache key.
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding), shape ``(n_series, T)``.
+ seasonality_idxs : torch.Tensor
+ 0-based seasonal slot per observation, shape ``(n_series, T)``.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor, torch.Tensor]
+ Seasonal indices ``(n_series, season_length)``, initial level
+ ``(n_series,)``, and initial trend ``(n_series,)``.
+ """
+ key = id(data)
+ entry = self._init_cache.get(key)
+ if entry is not None:
+ return entry["seasonality"].clone(), entry["level0"], entry["trend0"]
+
+ seasonality, level0, trend0 = self._init_additive_states(
+ target, mask, seasonality_idxs
+ )
+ if len(self._init_cache) > 8:
+ # Bound growth from one-shot datasets (e.g. _store_final_states,
+ # conformal re-anchoring); the persistent train/eval entries are
+ # simply recomputed once after a clear.
+ self._init_cache.clear()
+ # Storing the dataset pins its id: a key hit therefore implies this
+ # exact dataset, and with it identical target/mask/positions.
+ self._init_cache[key] = {
+ "data": data,
+ "seasonality": seasonality,
+ "level0": level0,
+ "trend0": trend0,
+ }
+ return seasonality.clone(), level0, trend0
+
+ def _forward_additive(
+ self,
+ params: torch.Tensor,
+ data: lgb.Dataset,
+ target: torch.Tensor,
+ mask: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, List[torch.Tensor]]:
+ """
+ Forward pass for additive triple exponential smoothing.
+
+ This implements the ETS state space equations for the additive
+ damped-trend Holt-Winters method (ETS(A,Ad,A)), in the component form
+ of Hyndman & Athanasopoulos, Forecasting: Principles and Practice
+ (https://otexts.com/fpp3/):
+ - Level: l_t = α(y_t - s_{t-m}) + (1-α)(l_{t-1} + φb_{t-1})
+ - Trend: b_t = β(l_t - l_{t-1}) + (1-β)φb_{t-1}
+ - Seasonality: s_t = γ(y_t - l_{t-1} - φb_{t-1}) + (1-γ)s_{t-m}
+ - Fitted: ŷ_t = l_{t-1} + φb_{t-1} + s_{t-m}
+
+ Parameters
+ ----------
+ params : torch.Tensor
+ Sigmoid-transformed ETS parameters, shape ``(n_series, T, n_params)``.
+ data : lgb.Dataset
+ LightGBM dataset whose raw DataFrame provides the seasonality feature.
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding), shape ``(n_series, T)``.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor, torch.Tensor, List[torch.Tensor]]
+ Final level (n_series,), final trend (n_series,),
+ seasonality matrix (n_series, season_length), and list of fitted values.
+ """
+ series_len = target.shape[1]
+
+ # Unpack and pre-unbind parameters
+ alpha, beta, gamma, phi = params.unbind(dim=2)
+ alpha_t = alpha.unbind(dim=1)
+ beta_t = beta.unbind(dim=1)
+ gamma_t = gamma.unbind(dim=1)
+ phi_t = phi.unbind(dim=1)
+
+ # Get seasonal indices from features first: the state initialization
+ # assigns initial indices to the same slots the recursion reads, so a
+ # series that does not start at season position 1 is not rotated.
+ seasonality_idxs = self._seasonal_positions(
+ data.data[self.seasonality_feature].values
).reshape(self.n_series, -1)
batch_idx = torch.arange(self.n_series, dtype=torch.long)
- # Triple ETS updates with masking for padded values
+ # Initialize seasonal indices and level/trend states via the classical
+ # decomposition-based estimator (additive form). The result depends
+ # only on the data, so it is cached per lgb.Dataset and reused across
+ # boosting iterations.
+ seasonality, level_prev, trend_prev = self._cached_init_additive_states(
+ data, target, mask, seasonality_idxs
+ )
+ fits = [target[:, 0]]
+
+ # Pre-unbind the data tensors so the loop indexes Python tuples
+ # instead of slicing tensors at every step.
+ target_t = target.unbind(dim=1)
+ mask_t = mask.unbind(dim=1)
+ idxs_t = seasonality_idxs.unbind(dim=1)
+
+ # Additive ETS updates with masking for padded values (shared
+ # subexpressions hoisted; see _forward_triple).
for t in range(1, series_len):
- s_idx = seasonality_idxs[:, t]
- valid_mask = mask[:, t]
+ valid_mask = mask_t[t]
+ invalid_mask = 1 - valid_mask
+ y_t = target_t[t]
+ s_prev = seasonality[batch_idx, idxs_t[t]] # s_{t-m}
+ phi_trend = phi_t[t] * trend_prev # phi_t * b_{t-1}
+ pred_base = level_prev + phi_trend # l_{t-1} + phi_t * b_{t-1}
- fit_t = valid_mask * (
- (level_prev + phi_t[t] * trend_prev) * seasonality[batch_idx, s_idx]
- ) + (1 - valid_mask) * fits[-1]
+ fit_t = valid_mask * (pred_base + s_prev) + invalid_mask * fits[-1]
level_new = valid_mask * (
- alpha_t[t] * (target[:, t] / seasonality[batch_idx, s_idx]) +
- (1 - alpha_t[t]) * (level_prev + phi_t[t] * trend_prev)
- ) + (1 - valid_mask) * level_prev
+ alpha_t[t] * (y_t - s_prev) +
+ (1 - alpha_t[t]) * pred_base
+ ) + invalid_mask * level_prev
trend_new = valid_mask * (
beta_t[t] * (level_new - level_prev) +
- (1 - beta_t[t]) * phi_t[t] * trend_prev
- ) + (1 - valid_mask) * trend_prev
+ (1 - beta_t[t]) * phi_trend
+ ) + invalid_mask * trend_prev
- seasonality[batch_idx, s_idx] = valid_mask * (
- gamma_t[t] * (target[:, t] / (level_prev + phi_t[t] * trend_prev)) +
- (1 - gamma_t[t]) * seasonality[batch_idx, s_idx]
- ) + (1 - valid_mask) * seasonality[batch_idx, s_idx]
+ seasonality[batch_idx, idxs_t[t]] = valid_mask * (
+ gamma_t[t] * (y_t - pred_base) +
+ (1 - gamma_t[t]) * s_prev
+ ) + invalid_mask * s_prev
fits.append(fit_t)
level_prev = level_new
@@ -445,6 +1099,17 @@ def _forward_trend(
- Trend: b_t = β(l_t - l_{t-1}) + (1-β)b_{t-1}
- Fitted: ŷ_t = l_{t-1} + b_{t-1}
+ Parameters
+ ----------
+ params : torch.Tensor
+ Sigmoid-transformed ETS parameters, shape ``(n_series, T, n_params)``.
+ data : lgb.Dataset
+ Unused; kept for a uniform forward signature with ``_forward_triple``.
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding), shape ``(n_series, T)``.
+
Returns
-------
Tuple[torch.Tensor, torch.Tensor, None, List[torch.Tensor]]
@@ -458,27 +1123,40 @@ def _forward_trend(
alpha_t = alpha.unbind(dim=1)
beta_t = beta.unbind(dim=1)
- # Initialize states for trend model
+ # Initialize states for the trend model. With back-appended padding
+ # the slope endpoint must be the last *valid* observation inside the
+ # window, not a padded value, so the endpoint index is mask-aware.
level_prev = target[:, 0]
- last_idx = min(2 * self.season_length - 1, series_len - 1)
- trend_prev = (target[:, last_idx] - target[:, 0]) / max(last_idx, 1)
+ cap = min(2 * self.season_length - 1, series_len - 1)
+ n_valid = mask[:, :cap + 1].sum(dim=1).to(torch.long)
+ last_idx = (n_valid - 1).clamp(min=0)
+ endpoint = target.gather(1, last_idx.unsqueeze(1)).squeeze(1)
+ trend_prev = (endpoint - level_prev) / last_idx.to(self.dtype).clamp(min=1.0)
fits = [target[:, 0]]
- # Trend-only updates with masking for padded values
+ # Pre-unbind the data tensors so the loop indexes Python tuples
+ # instead of slicing tensors at every step.
+ target_t = target.unbind(dim=1)
+ mask_t = mask.unbind(dim=1)
+
+ # Trend-only updates with masking for padded values (shared
+ # subexpressions hoisted; see _forward_triple).
for t in range(1, series_len):
- valid_mask = mask[:, t]
+ valid_mask = mask_t[t]
+ invalid_mask = 1 - valid_mask
+ pred_base = level_prev + trend_prev # l_{t-1} + b_{t-1}
- fit_t = valid_mask * (level_prev + trend_prev) + (1 - valid_mask) * fits[-1]
+ fit_t = valid_mask * pred_base + invalid_mask * fits[-1]
level_new = valid_mask * (
- alpha_t[t] * target[:, t] +
- (1 - alpha_t[t]) * (level_prev + trend_prev)
- ) + (1 - valid_mask) * level_prev
+ alpha_t[t] * target_t[t] +
+ (1 - alpha_t[t]) * pred_base
+ ) + invalid_mask * level_prev
trend_new = valid_mask * (
beta_t[t] * (level_new - level_prev) +
(1 - beta_t[t]) * trend_prev
- ) + (1 - valid_mask) * trend_prev
+ ) + invalid_mask * trend_prev
fits.append(fit_t)
level_prev = level_new
@@ -555,6 +1233,7 @@ def train(
seed: int = 123,
verbose: int = -1,
deterministic: bool = True,
+ forecast_intervals: Optional[ForecastIntervals] = None,
) -> TrainingResult:
"""
Train the Hyper-Tree-ETS model on time series data.
@@ -593,6 +1272,12 @@ def train(
If True, sets LightGBM's ``deterministic`` and ``force_row_wise`` parameters to ensure
reproducible results. May slow down training. See
https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic
+ forecast_intervals : ForecastIntervals, optional
+ If provided, calibrate conformal prediction intervals via rolling-window
+ cross-validation after the main model is trained. The collected conformity
+ scores are then used by ``forecast(..., level=[...])`` to produce
+ ``-lo-`` / ``-hi-`` columns. See
+ :class:`hypertrees.conformal.ForecastIntervals`.
Returns
-------
@@ -621,6 +1306,8 @@ def train(
raise TypeError("validation must be a boolean.")
if not isinstance(deterministic, bool):
raise TypeError("deterministic must be a boolean.")
+ if forecast_intervals is not None and not isinstance(forecast_intervals, ForecastIntervals):
+ raise TypeError("forecast_intervals must be a ForecastIntervals instance.")
if early_stopping_round is not None and not validation:
raise ValueError("early_stopping_round can only be used when validation is True.")
if validation and early_stopping_round is None:
@@ -639,15 +1326,22 @@ def train(
# monotonic dates so the ETS reshape to (n_series, T, n_params) aligns.
validate_series_order(train_data, name="train_data")
+ if forecast_intervals is not None:
+ validate_calibration_length(
+ train_data, self.fcst_h, forecast_intervals,
+ min_train=self.season_length + 1,
+ )
+
# Check if all series in train_data have the same length
unique_lengths = train_data.groupby('series_id')['date'].nunique()
if len(unique_lengths.unique()) > 1:
raise ValueError("All series in train_data must have the same length. Found multiple lengths.")
- # General model parameters
+ # General model parameters. The objective wrapper stops lgb.train's
+ # params deepcopy from cloning this instance (see NoDeepcopyObjective).
self.lgb_params = {
"num_class": self.n_params,
- "objective": self.objective_fn,
+ "objective": NoDeepcopyObjective(self.objective_fn),
"metric": "None",
"random_seed": seed,
"verbose": verbose
@@ -655,6 +1349,7 @@ def train(
# Reset states
self._iter_count = 0
+ self._init_cache = {}
self._fit = None
self._mask = None
self._target = None
@@ -663,6 +1358,12 @@ def train(
self.is_trained = False
self.fcst_states = None
self.features = None
+ self._last_states = None
+ self._eval_boundary = None
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
# Set random seeds for reproducibility
torch.manual_seed(seed)
@@ -729,11 +1430,42 @@ def train(
# Set trained flag to True
self.is_trained = True
+ if forecast_intervals is not None:
+ def _model_factory():
+ return HyperTreeETS(
+ ets_type=self.ets_type,
+ season_length=self.season_length,
+ seasonality_feature=self.seasonality_feature,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ n_hessian_probes=self.n_hessian_probes,
+ seasonal_init=self.seasonal_init,
+ )
+
+ cal_train_kwargs = dict(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+ self._cs_scores, self._cs_series_order = rolling_origin_residuals(
+ model_factory=_model_factory,
+ train_data=train_data,
+ fcst_h=self.fcst_h,
+ forecast_intervals=forecast_intervals,
+ train_kwargs=cal_train_kwargs,
+ )
+ self._pi_config = forecast_intervals
+ self._is_calibrated = True
+
# Return results
result = TrainingResult(
train_metrics=evals_result["train"] if validation else {"loss": []},
validation_metrics=evals_result["validation"] if validation else None,
- best_iteration=self.model.best_iteration-1 if hasattr(self.model, 'best_iteration') else num_iterations,
+ best_iteration=self.model.best_iteration if self.model.best_iteration > 0 else num_iterations,
training_time=training_time
)
@@ -741,14 +1473,15 @@ def train(
except Exception as e:
self.is_trained = False
- raise RuntimeError(f"Training failed: {str(e)}")
+ raise RuntimeError(f"Training failed: {str(e)}") from e
def _store_final_states(self, train_data: pd.DataFrame):
"""
Store final ETS states after training for use in forecasting.
Runs a full forward pass on the training data to obtain the final
- level, trend, and (for triple ETS) seasonality states per series.
+ level, trend, and (for the seasonal variants, triple/additive)
+ seasonality states per series.
Also stores the series ordering to ensure consistent state access.
Parameters
@@ -791,15 +1524,167 @@ def _store_final_states(self, train_data: pd.DataFrame):
'last_trend': last_trend[i]
}
- # Store seasonality for triple ETS
- if self.ets_type == "triple" and seasonality is not None:
+ # Store seasonality for the seasonal ETS variants
+ if self.ets_type in ("triple", "additive") and seasonality is not None:
for i, series_id in enumerate(self.series_order):
self.fcst_states[series_id]['seasonality'] = seasonality[i]
+ def set_forecast_origin(self, history: pd.DataFrame) -> None:
+ """Re-anchor ETS states to the end of *history* without retraining.
+
+ Recomputes the terminal ``{level, trend, seasonality}`` states by
+ running the full ETS forward recurrence over *history* using the
+ already-trained GBDT parameters. Used by conformal calibration with
+ ``refit=False``.
+
+ Parameters
+ ----------
+ history : pd.DataFrame
+ DataFrame with the same columns as the training data (including
+ any ``mask`` / ``seasonality_feature`` columns), ordered by
+ ``(series_id, date)`` with each series in a contiguous block and
+ all series of equal length.
+ """
+ validate_series_order(history, name="history")
+ self._store_final_states(history)
+
+ def _roll_forecast(
+ self,
+ level_h: torch.Tensor,
+ trend_h: torch.Tensor,
+ seasonality: Optional[torch.Tensor],
+ params: torch.Tensor,
+ seasonality_idxs: Optional[torch.Tensor],
+ ) -> torch.Tensor:
+ """Roll the ETS deployment recursion forward from terminal states.
+
+ Shared by :meth:`forecast` and the recursive validation metric
+ (:meth:`_recursive_eval_loss`) so the deployed forecast and the
+ early-stopping metric can never diverge. Each step's own one-step fit
+ serves as the pseudo-observation in the state updates (future
+ innovations are zero in expectation), exactly mirroring the training
+ forward passes. The caller's ``seasonality`` is cloned, not mutated.
+
+ Parameters
+ ----------
+ level_h : torch.Tensor
+ Terminal level state at the forecast origin, shape ``(n_series,)``.
+ trend_h : torch.Tensor
+ Terminal trend state at the forecast origin, shape ``(n_series,)``.
+ seasonality : torch.Tensor or None
+ Terminal seasonal states, shape ``(n_series, season_length)``;
+ ``None`` for ``ets_type="trend"``.
+ params : torch.Tensor
+ Sigmoid-transformed ETS parameters per horizon step,
+ shape ``(n_series, H, n_params)``.
+ seasonality_idxs : torch.Tensor or None
+ 0-based seasonal slot per horizon step, shape ``(n_series, H)``;
+ ``None`` for ``ets_type="trend"``.
+
+ Returns
+ -------
+ torch.Tensor
+ Point forecasts, shape ``(n_series, H)``.
+ """
+ H = params.shape[1]
+ n_series = params.shape[0]
+ batch_idx = torch.arange(n_series, dtype=torch.long)
+ level_h = level_h.clone()
+ trend_h = trend_h.clone()
+ if seasonality is not None:
+ seasonality = seasonality.clone()
+ fcsts = []
+
+ if self.ets_type == "triple":
+ for h in range(H):
+ alpha = params[:, h, 0]
+ beta = params[:, h, 1]
+ gamma = params[:, h, 2]
+ phi = params[:, h, 3]
+ s_idx = seasonality_idxs[:, h].long()
+ s_h = seasonality[batch_idx, s_idx]
+
+ # One-step-ahead fit (pseudo-observation),
+ # structurally identical to _forward_triple.
+ pseudo_y = (level_h + phi * trend_h) * s_h
+ fcsts.append(pseudo_y.reshape(-1, 1))
+
+ # State updates exactly as in _forward_triple.
+ level_new = (
+ alpha * (pseudo_y / s_h)
+ + (1 - alpha) * (level_h + phi * trend_h)
+ )
+ trend_new = (
+ beta * (level_new - level_h)
+ + (1 - beta) * phi * trend_h
+ )
+ seasonality[batch_idx, s_idx] = (
+ gamma * (pseudo_y / (level_h + phi * trend_h))
+ + (1 - gamma) * s_h
+ )
+ level_h = level_new
+ trend_h = trend_new
+
+ elif self.ets_type == "additive":
+ for h in range(H):
+ alpha = params[:, h, 0]
+ beta = params[:, h, 1]
+ gamma = params[:, h, 2]
+ phi = params[:, h, 3]
+ s_idx = seasonality_idxs[:, h].long()
+ s_h = seasonality[batch_idx, s_idx]
+
+ # One-step-ahead fit (pseudo-observation),
+ # structurally identical to _forward_additive.
+ pred_base = level_h + phi * trend_h
+ pseudo_y = pred_base + s_h
+ fcsts.append(pseudo_y.reshape(-1, 1))
+
+ # State updates exactly as in _forward_additive.
+ level_new = (
+ alpha * (pseudo_y - s_h)
+ + (1 - alpha) * pred_base
+ )
+ trend_new = (
+ beta * (level_new - level_h)
+ + (1 - beta) * phi * trend_h
+ )
+ seasonality[batch_idx, s_idx] = (
+ gamma * (pseudo_y - pred_base)
+ + (1 - gamma) * s_h
+ )
+ level_h = level_new
+ trend_h = trend_new
+
+ elif self.ets_type == "trend":
+ for h in range(H):
+ alpha = params[:, h, 0]
+ beta = params[:, h, 1]
+
+ # One-step-ahead fit (pseudo-observation),
+ # structurally identical to _forward_trend.
+ pseudo_y = level_h + trend_h
+ fcsts.append(pseudo_y.reshape(-1, 1))
+
+ # State updates exactly as in _forward_trend.
+ level_new = (
+ alpha * pseudo_y
+ + (1 - alpha) * (level_h + trend_h)
+ )
+ trend_new = (
+ beta * (level_new - level_h)
+ + (1 - beta) * trend_h
+ )
+ level_h = level_new
+ trend_h = trend_new
+
+ return torch.cat(fcsts, dim=1)
+
def forecast(
self,
test_data: pd.DataFrame,
- type: str = "forecast"
+ type: str = "forecast",
+ level: Optional[List[int]] = None,
) -> pd.DataFrame:
"""
Generate forecasts using the trained model.
@@ -812,6 +1697,14 @@ def forecast(
The forecasting process rolls the ETS state-space recursion forward,
mirroring the training forward pass.
+ Note that the pseudo-observation rollout collapses algebraically: the
+ predicted alpha/beta/gamma at the forecast horizon cancel out of the
+ trajectory (they multiply innovations that are zero in expectation),
+ so only the damping phi and the seasonal-slot rotation shape the
+ h-step forecast -- and for ``ets_type="trend"`` no horizon parameter
+ affects it at all. Horizon features therefore do not alter the point
+ forecasts beyond phi; they do affect ``type="parameters"``.
+
Parameters
----------
test_data : pd.DataFrame
@@ -821,6 +1714,11 @@ def forecast(
Type of forecast to generate. Options:
- "forecast": Generate forecasted values
- "parameters": Return the ETS parameters used for forecasting
+ level : list of int, optional
+ Confidence levels (in ``(0, 100)``, e.g. ``[80, 90]``) for conformal
+ prediction intervals. Only valid with ``type="forecast"`` and requires
+ the model to have been trained with ``forecast_intervals=...``. Adds
+ ``-lo-`` / ``-hi-`` columns to the output.
Returns
-------
@@ -828,9 +1726,11 @@ def forecast(
Forecasted data with columns:
- series_id: Identifier for each time series
- date: Forecast date/time
- - model: Model name identifier
- fcst: Forecasted value (if type="forecast")
+ - model: Model name identifier
- alpha, beta, gamma, phi: ETS parameter values (if type="parameters")
+ - -lo- / -hi-: prediction interval bounds
+ (if type="forecast" and level is provided)
"""
# Check if model is trained and states are stored
if not self.is_trained or self.model is None:
@@ -876,6 +1776,21 @@ def forecast(
if type not in ["forecast", "parameters"]:
raise ValueError("Parameter 'type' must be either 'forecast' or 'parameters'")
+ if level is not None:
+ if type != "forecast":
+ raise ValueError("level is only supported with type='forecast'.")
+ if not self._is_calibrated:
+ raise RuntimeError(
+ "Prediction intervals were requested via level, but the model "
+ "was not calibrated. Pass forecast_intervals=ForecastIntervals(...) "
+ "to train() before forecasting with level."
+ )
+ if not isinstance(level, (list, tuple)) or len(level) == 0:
+ raise ValueError("level must be a non-empty list of integers.")
+ for lv in level:
+ if not isinstance(lv, (int, np.integer)) or not 0 < lv < 100:
+ raise ValueError(f"level values must be integers in (0, 100); got {lv}.")
+
try:
# If mask was a training feature but is absent from test_data, add it (all test obs are valid)
if 'mask' in self.features and 'mask' not in test_data.columns:
@@ -901,94 +1816,49 @@ def forecast(
last_level = torch.stack([self.fcst_states[series_id]['last_level'] for series_id in test_series_ids])
last_trend = torch.stack([self.fcst_states[series_id]['last_trend'] for series_id in test_series_ids])
- # Generate forecasts by rolling the ETS state forward, mirroring
- # the training forward pass and using each step's forecast as
- # the pseudo-observation in the state updates.
- if self.ets_type == "triple":
+ # Roll the ETS state forward via the shared recursion (also
+ # used by the recursive validation metric in eval_fn, so the
+ # two can never diverge).
+ if self.ets_type in ("triple", "additive"):
# Extract seasonality in test series order
seasonality = torch.stack(
[self.fcst_states[series_id]['seasonality'] for series_id in test_series_ids]
)
-
- seasonality_idxs = torch.tensor(
- test_data[self.seasonality_feature].values - 1
+ seasonality_idxs = self._seasonal_positions(
+ test_data[self.seasonality_feature].values
).reshape(self.n_series, self.fcst_h)
- batch_idx = torch.arange(self.n_series, dtype=torch.long)
- alpha_fcst = fcst_params[:, :, 0]
- beta_fcst = fcst_params[:, :, 1]
- gamma_fcst = fcst_params[:, :, 2]
- phi_fcst = fcst_params[:, :, 3]
-
- fcsts = []
- level_h = last_level
- trend_h = last_trend
- for h in range(self.fcst_h):
- alpha = alpha_fcst[:, h]
- beta = beta_fcst[:, h]
- gamma = gamma_fcst[:, h]
- phi = phi_fcst[:, h]
- s_idx = seasonality_idxs[:, h].long()
- s_h = seasonality[batch_idx, s_idx]
-
- # One-step-ahead fit (pseudo-observation),
- # structurally identical to _forward_triple
- pseudo_y = (level_h + phi * trend_h) * s_h
- fcsts.append(pseudo_y.reshape(-1, 1))
-
- # State updates exactly as in _forward_triple.
- level_new = (
- alpha * (pseudo_y / s_h)
- + (1 - alpha) * (level_h + phi * trend_h)
- )
- trend_new = (
- beta * (level_new - level_h)
- + (1 - beta) * phi * trend_h
- )
- seasonality[batch_idx, s_idx] = (
- gamma * (pseudo_y / (level_h + phi * trend_h))
- + (1 - gamma) * s_h
- )
-
- level_h = level_new
- trend_h = trend_new
-
- elif self.ets_type == "trend":
- alpha_fcst = fcst_params[:, :, 0]
- beta_fcst = fcst_params[:, :, 1]
-
- fcsts = []
- level_h = last_level
- trend_h = last_trend
- for h in range(self.fcst_h):
- alpha = alpha_fcst[:, h]
- beta = beta_fcst[:, h]
-
- # One-step-ahead fit (pseudo-observation),
- # structurally identical to _forward_trend
- pseudo_y = level_h + trend_h
- fcsts.append(pseudo_y.reshape(-1, 1))
-
- # State updates exactly as in _forward_trend
- level_new = (
- alpha * pseudo_y
- + (1 - alpha) * (level_h + trend_h)
- )
- trend_new = (
- beta * (level_new - level_h)
- + (1 - beta) * trend_h
- )
-
- level_h = level_new
- trend_h = trend_new
+ else:
+ seasonality = None
+ seasonality_idxs = None
+
+ fcsts_mat = self._roll_forecast(
+ last_level, last_trend, seasonality, fcst_params, seasonality_idxs
+ )
# Create output dataframe
+ model_name = f"Hyper-Tree-ETS({self.ets_type})"
out_df = pd.DataFrame({
"series_id": test_data["series_id"].to_numpy().flatten(),
"date": test_data["date"].to_numpy().flatten(),
- "fcst": torch.cat(fcsts, dim=1).flatten().numpy(),
- "model": f"Hyper-Tree-ETS({self.ets_type})",
+ "fcst": fcsts_mat.flatten().numpy(),
+ "model": model_name,
})
+ if level is not None:
+ point = fcsts_mat.numpy() # (n_series, fcst_h)
+ test_series_ids = test_data["series_id"].unique()
+ columns = interval_columns(
+ point=point,
+ scores=self._cs_scores,
+ levels=level,
+ method=self._pi_config.method,
+ model_name=model_name,
+ cal_order=self._cs_series_order,
+ target_order=list(test_series_ids),
+ )
+ for col_name, values in columns.items():
+ out_df[col_name] = values
+
elif type == "parameters":
fcst_params = torch.clamp(
self.sigmoid_fn(torch.tensor(self.model.predict(test_data[self.features]), dtype=self.dtype)
@@ -1001,11 +1871,11 @@ def forecast(
"date": test_data["date"].to_numpy().flatten(),
"model": f"Hyper-Tree-ETS({self.ets_type})",
})
- param_names = ["alpha", "beta", "gamma", "phi"] if self.ets_type == "triple" else ["alpha", "beta"]
+ param_names = ["alpha", "beta", "gamma", "phi"] if self.ets_type in ("triple", "additive") else ["alpha", "beta"]
for i, param_name in enumerate(param_names):
out_df[param_name] = fcst_params[:, :, i].flatten().numpy()
return out_df
except Exception as e:
- raise RuntimeError(f"Forecasting not successful: {str(e)}")
+ raise RuntimeError(f"Forecasting not successful: {str(e)}") from e
diff --git a/hypertrees/models/HyperTreeNetAR.py b/hypertrees/models/HyperTreeNetAR.py
index 0c9d7ae..e8913e5 100644
--- a/hypertrees/models/HyperTreeNetAR.py
+++ b/hypertrees/models/HyperTreeNetAR.py
@@ -4,9 +4,15 @@
import torch.nn as nn
from torch.autograd import grad as autograd
import lightgbm as lgb
-from typing import Tuple, Callable, Optional
+from typing import Tuple, Callable, Optional, List
import time
-from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, CustomLogger, validate_series_order, GaussNewtonHessian
+from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, CustomLogger, validate_series_order, extract_forecast_lags, GaussNewtonHessian, NoDeepcopyObjective
+from ..conformal import (
+ ForecastIntervals,
+ validate_calibration_length,
+ rolling_origin_residuals,
+ interval_columns,
+)
from .mlp import MLP
import warnings
lgb.register_logger(CustomLogger())
@@ -21,7 +27,7 @@ class HyperTreeNetAR:
Class that implements a Hyper-TreeNet-AR(p) model for time series forecasting.
It combines LightGBM with a neural network, where the LightGBM first creates embeddings from the input data which
- are then mapped as parameters to the target time series model. The HyperTree-AR(p) model extends traditional
+ are then mapped as parameters to the target time series model. The Hyper-TreeNet-AR(p) model extends traditional
autoregressive models by allowing the AR coefficients to be time-varying and estimated by a
combination of neural network and gradient boosted trees. This creates a non-linear, adaptive autoregressive model
that can capture complex temporal dependencies.
@@ -102,7 +108,6 @@ class HyperTreeNetAR:
plt.tight_layout()
```
"""
- _network_states = {} # Store network states for each instance
def __init__(
self,
p: int = 2,
@@ -127,13 +132,18 @@ def __init__(
Forecast horizon (number of periods to forecast ahead).
loss_fn : Callable
Loss function for optimization. Must be a PyTorch loss function.
- Default is MSE loss, but can be changed for different error metrics.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
device : str
Device to run the model on. Default is 'cpu'.
This allows for GPU acceleration of network training if available.
hessian_method : str
Method for computing the Hessian diagonal. Options:
- - "exact": Exact diagonal Hessian via per-parameter second-order autograd.
+ - "exact": Exact diagonal Hessian via per-embedding-dimension
+ second-order autograd (cheap, since the embedding is
+ low-dimensional).
- "gn": Gauss-Newton approximation estimated via Hutchinson probing.
Guarantees positive semi-definite Hessians. Avoids second-order
differentiation at the cost of Hutchinson estimation variance.
@@ -151,6 +161,19 @@ def __init__(
raise TypeError("freq must be a string.")
if not isinstance(loss_fn, nn.Module):
raise TypeError("loss_fn must be a PyTorch loss function.")
+ if isinstance(loss_fn, nn.L1Loss):
+ raise ValueError(
+ "nn.L1Loss is not supported: its second derivative is zero almost "
+ "everywhere, so LightGBM's Newton boosting receives all-zero Hessians "
+ "and cannot grow trees. Use nn.HuberLoss or nn.SmoothL1Loss for an "
+ "MAE-like loss with usable curvature."
+ )
+ if getattr(loss_fn, "reduction", "mean") == "none":
+ raise ValueError(
+ "loss_fn must use a scalar reduction ('mean' or 'sum'); "
+ "reduction='none' returns per-element losses that the "
+ "boosting objective cannot consume."
+ )
if hessian_method not in ("exact", "gn"):
raise ValueError("hessian_method must be either 'exact' or 'gn'.")
if not isinstance(n_hessian_probes, int) or n_hessian_probes <= 0:
@@ -182,6 +205,12 @@ def __init__(
self._fit = None
self._target = None
+ # Conformal prediction interval state
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
+
if hessian_method == "gn":
self._gn_hessian = GaussNewtonHessian(loss_fn, n_hessian_probes, self.dtype)
@@ -244,7 +273,7 @@ def eval_fn(self, predt: np.ndarray, eval_data: lgb.Dataset) -> Tuple[str, float
warnings.warn("Unknown dataset in metric_fn. Using training lags.")
# Calculate loss
- is_higher_better = False
+ is_higher_better = False # Lower loss is better, so we don't maximize
target = torch.tensor(eval_data.get_label().reshape(-1, 1), dtype=self.dtype, device=self.device)
# For evaluation, we need to compute loss without any backward pass or gradient computation
@@ -253,8 +282,8 @@ def eval_fn(self, predt: np.ndarray, eval_data: lgb.Dataset) -> Tuple[str, float
dtype=self.dtype
).to(self.device)
- # Load the most recent network state
- self.network.load_state_dict(HyperTreeNetAR._network_states)
+ # self.network is the live module the objective updated in this same
+ # boosting iteration (guaranteed by NoDeepcopyObjective).
# Compute loss without gradients
self.network.eval()
@@ -316,9 +345,6 @@ def get_embeds_loss_separate(
network_loss.backward()
self.optimizer.step()
- # Store network state
- HyperTreeNetAR._network_states = self.network.state_dict()
-
# Calculate loss for GBDT
self.network.eval()
ar_params_gbdt = self.network(gbdt_embed)
@@ -420,32 +446,28 @@ def _calculate_gradients_and_hessians_separate(self, loss: torch.Tensor, embeds:
for i in range(self.embedding_dim)
]
- # Batched Hessian-diagonal computation: replaces the per-dim Python loop with a single batched autograd call.
- # n, k = grad.shape
- # grad_outputs = (
- # torch.eye(k, device=grad.device, dtype=grad.dtype)
- # .unsqueeze(1)
- # .expand(k, n, k)
- # )
- # hess_full = autograd(
- # grad,
- # embeds,
- # grad_outputs=grad_outputs,
- # is_grads_batched=True,
- # retain_graph=True,
- # )[0] # (k, n, k)
- # hess = torch.diagonal(hess_full, dim1=0, dim2=2) # (n, k)
-
# Convert to numpy arrays and reshape as expected by LightGBM
grad = grad.cpu().detach().numpy().ravel(order="F")
hess = torch.cat(hess, dim=1).cpu().detach().numpy().ravel(order="F")
- # hess = hess.cpu().detach().numpy().ravel(order="F") # only for the batched version
return grad, hess
def _calculate_gradients_and_hessians_separate_gn(self, loss: torch.Tensor, embeds: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
- """Gauss-Newton Hessian for separate gradient mode via Hutchinson probing."""
+ """Gauss-Newton Hessian for separate gradient mode via Hutchinson probing.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ embeds : torch.Tensor
+ GBDT embeddings, shape ``(n_samples, embedding_dim)``.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
grad = autograd(loss, inputs=embeds, retain_graph=True)[0]
rng = torch.Generator().manual_seed(self._iter_count)
hess = self._gn_hessian.estimate(self._fit, self._target, embeds, rng)
@@ -488,9 +510,6 @@ def _calculate_gradients_and_hessians_shared(self, loss: torch.Tensor, embeds: t
# Update network parameters
self.optimizer.step()
- # Store network state
- HyperTreeNetAR._network_states = self.network.state_dict()
-
# Clear existing gradients to prevent accumulation
embeds.grad = None
self.optimizer.zero_grad()
@@ -498,7 +517,21 @@ def _calculate_gradients_and_hessians_shared(self, loss: torch.Tensor, embeds: t
return grad, hess
def _calculate_gradients_and_hessians_shared_gn(self, loss: torch.Tensor, embeds: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
- """Gauss-Newton Hessian for shared gradient mode via Hutchinson probing."""
+ """Gauss-Newton Hessian for shared gradient mode via Hutchinson probing.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model (already backpropagated by
+ ``get_embeds_loss_shared``, which populates ``embeds.grad``).
+ embeds : torch.Tensor
+ GBDT embeddings, shape ``(n_samples, embedding_dim)``.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
grad = embeds.grad
rng = torch.Generator().manual_seed(self._iter_count)
hess = self._gn_hessian.estimate(self._fit, self._target, embeds, rng)
@@ -507,7 +540,6 @@ def _calculate_gradients_and_hessians_shared_gn(self, loss: torch.Tensor, embeds
grad = grad.cpu().detach().numpy().ravel(order="F")
hess = hess.cpu().detach().numpy().ravel(order="F")
self.optimizer.step()
- HyperTreeNetAR._network_states = self.network.state_dict()
embeds.grad = None
self.optimizer.zero_grad()
@@ -525,6 +557,7 @@ def train(
seed: int = 123,
verbose: int = -1,
deterministic: bool = True,
+ forecast_intervals: Optional[ForecastIntervals] = None,
) -> TrainingResult:
"""
Train the Hyper-TreeNet-AR model on time series data.
@@ -532,7 +565,7 @@ def train(
This method:
1. Preprocesses the time series data to create lag features
2. Sets up LightGBM datasets
- 3. Train the models
+ 3. Trains the models
The training data must contain columns:
- 'series_id': Identifier for each time series
@@ -574,10 +607,16 @@ def train(
If True, sets LightGBM's ``deterministic`` and ``force_row_wise`` parameters to ensure
reproducible results. May slow down training. See
https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic
+ forecast_intervals : ForecastIntervals, optional
+ If provided, calibrate conformal prediction intervals via rolling-window
+ cross-validation after the main model is trained. The collected conformity
+ scores are then used by ``forecast(..., level=[...])`` to produce
+ ``-lo-`` / ``-hi-`` columns. See
+ :class:`hypertrees.conformal.ForecastIntervals`.
Returns
-------
- TrainingResult
+ TrainingResult
Object containing evaluation results and training information.
"""
# Validate inputs
@@ -607,11 +646,26 @@ def train(
raise TypeError("validation must be a boolean.")
if not isinstance(deterministic, bool):
raise TypeError("deterministic must be a boolean.")
+ if forecast_intervals is not None and not isinstance(forecast_intervals, ForecastIntervals):
+ raise TypeError("forecast_intervals must be a ForecastIntervals instance.")
if early_stopping_round is not None and not validation:
raise ValueError("early_stopping_round can only be used when validation is True.")
if validation and early_stopping_round is None:
raise ValueError("early_stopping_round must be provided when validation is True.")
+ required_net_keys = (
+ "learning_rate", "embedding_dimension", "hidden_dim",
+ "dropout", "use_random_projection",
+ )
+ missing_keys = [key for key in required_net_keys if key not in network_params]
+ if missing_keys:
+ raise ValueError(f"network_params is missing required keys: {missing_keys}")
+ if network_params.get("use_random_projection") and "rp_embed_dim" not in network_params:
+ raise ValueError(
+ "network_params is missing required keys: ['rp_embed_dim'] "
+ "(required when use_random_projection=True)."
+ )
+
if deterministic:
lgb_params = {**lgb_params, "deterministic": True, "force_row_wise": True}
@@ -628,10 +682,32 @@ def train(
# monotonic dates so the training reshape and fcst_lags extraction align.
validate_series_order(train_data, name="train_data")
+ # Each series must keep at least one training row after lagging; a
+ # shorter series would silently contribute nothing while leaving a
+ # ragged forecast seed behind.
+ lengths = train_data.groupby("series_id", sort=False).size()
+ bad = lengths[lengths <= self.p]
+ if len(bad) > 0:
+ raise ValueError(
+ f"Each series needs at least p + 1 = {self.p + 1} observations "
+ f"so that one training row remains after lagging, but these "
+ f"series are too short: {bad.to_dict()}."
+ )
+
+ if forecast_intervals is not None:
+ validate_calibration_length(
+ train_data, self.fcst_h, forecast_intervals, min_train=self.p + 1
+ )
+
# Set the network and optimizer
gbdt_params = lgb_params.copy()
self.embedding_dim = network_params["embedding_dimension"]
+ # Seed torch before constructing the MLP so initialization (and dropout
+ # draws during training) are reproducible even when the random
+ # projection layer -- whose constructor reseeds torch -- is disabled.
+ torch.manual_seed(seed)
+
self.network = MLP(
tree_embed_dim=self.embedding_dim,
output_dim=self.p,
@@ -651,6 +727,10 @@ def train(
self.dataset_references = {}
self.is_trained = False
self.features = None
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
# Select objective function based on training mode and Hessian method
if self.gradient_mode == "separate":
@@ -666,10 +746,11 @@ def train(
else:
self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_shared_gn
- # GBDT parameters
+ # GBDT parameters. The objective wrapper stops lgb.train's params
+ # deepcopy from cloning this instance (see NoDeepcopyObjective).
self.lgb_params = {
"num_class": self.embedding_dim,
- "objective": self.objective_fn,
+ "objective": NoDeepcopyObjective(self.objective_fn),
"metric": "None",
"random_seed": seed,
"verbose": verbose
@@ -715,11 +796,7 @@ def train(
self.lags_eval = lags_eval.to(self.device) if lags_eval is not None else None
# Store lagged train values to be used in the forecast method
- self.fcst_lags = (
- train_data.groupby(["series_id"], sort=False)
- .apply(lambda x: x["value"][-self.p:][::-1].values)
- .to_dict()
- )
+ self.set_forecast_origin(train_data)
# Train model
start_time = time.time()
@@ -737,11 +814,43 @@ def train(
# Set trained flag to True
self.is_trained = True
+ if forecast_intervals is not None:
+ def _model_factory():
+ return HyperTreeNetAR(
+ p=self.p,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ device=self.device,
+ hessian_method=self.hessian_method,
+ n_hessian_probes=self.n_hessian_probes,
+ )
+
+ cal_train_kwargs = dict(
+ lgb_params=lgb_params,
+ network_params=network_params,
+ gradient_mode=gradient_mode,
+ num_iterations=num_iterations,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+ self._cs_scores, self._cs_series_order = rolling_origin_residuals(
+ model_factory=_model_factory,
+ train_data=train_data,
+ fcst_h=self.fcst_h,
+ forecast_intervals=forecast_intervals,
+ train_kwargs=cal_train_kwargs,
+ )
+ self._pi_config = forecast_intervals
+ self._is_calibrated = True
+
# Return results
result = TrainingResult(
train_metrics=evals_result["train"] if validation else {"loss": []},
validation_metrics=evals_result["validation"] if validation else None,
- best_iteration=self.model.best_iteration-1 if hasattr(self.model, 'best_iteration') else num_iterations,
+ best_iteration=self.model.best_iteration if self.model.best_iteration > 0 else num_iterations,
training_time=training_time
)
@@ -751,12 +860,25 @@ def train(
except Exception as e:
self.is_trained = False
- raise RuntimeError(f"Training failed: {str(e)}")
+ raise RuntimeError(f"Training failed: {str(e)}") from e
+
+ def set_forecast_origin(self, history: pd.DataFrame) -> None:
+ """Re-anchor the AR lag seed to the end of *history* without retraining.
+
+ Parameters
+ ----------
+ history : pd.DataFrame
+ DataFrame with ``series_id``, ``date``, ``value`` columns, ordered
+ by ``(series_id, date)`` with each series in a contiguous block.
+ """
+ validate_series_order(history, name="history")
+ self.fcst_lags = extract_forecast_lags(history, self.p)
def forecast(
self,
test_data: pd.DataFrame,
- type: str = "forecast"
+ type: str = "forecast",
+ level: Optional[List[int]] = None,
) -> pd.DataFrame:
"""
Generate forecasts using the trained model.
@@ -781,6 +903,11 @@ def forecast(
- "forecast": Generate forecasted values
- "parameters": Return the AR(p) coefficients used for forecasting
- "tree_embeddings": Return the tree embeddings
+ level : list of int, optional
+ Confidence levels (in ``(0, 100)``, e.g. ``[80, 90]``) for conformal
+ prediction intervals. Only valid with ``type="forecast"`` and requires
+ the model to have been trained with ``forecast_intervals=...``. Adds
+ ``-lo-`` / ``-hi-`` columns to the output.
Returns
-------
@@ -788,10 +915,12 @@ def forecast(
Forecasted data with columns:
- series_id: Identifier for each time series
- date: Forecast date/time
- - model: Model name identifier
- fcst: Forecasted value (if type="forecast")
+ - model: Model name identifier
- AR(i) for i=1..p: AR coefficient values (if type="parameters")
- tree_embedding_{i} for i=1..embedding_dim: GBDT tree-embedding dimensions (if type="tree_embeddings")
+ - -lo- / -hi-: prediction interval bounds
+ (if type="forecast" and level is provided)
"""
# Check if model is trained
if not self.is_trained or self.model is None:
@@ -840,16 +969,33 @@ def forecast(
if type not in ["forecast", "parameters", "tree_embeddings"]:
raise ValueError("Parameter 'type' must be either 'forecast', 'parameters' or 'tree_embeddings'.")
+ if level is not None:
+ if type != "forecast":
+ raise ValueError("level is only supported with type='forecast'.")
+ if not self._is_calibrated:
+ raise RuntimeError(
+ "Prediction intervals were requested via level, but the model "
+ "was not calibrated. Pass forecast_intervals=ForecastIntervals(...) "
+ "to train() before forecasting with level."
+ )
+ if not isinstance(level, (list, tuple)) or len(level) == 0:
+ raise ValueError("level must be a non-empty list of integers.")
+ for lv in level:
+ if not isinstance(lv, (int, np.integer)) or not 0 < lv < 100:
+ raise ValueError(f"level values must be integers in (0, 100); got {lv}.")
+
try:
# Get tree embeddings
+ # Predict on the DataFrame (not .values) so pandas ``category``
+ # dtype features keep their categorical encoding at forecast time.
gbdt_embeds = torch.tensor(
- self.model.predict(test_data[self.features].values),
+ self.model.predict(test_data[self.features]),
dtype=self.dtype,
device=self.device
).reshape(-1, self.embedding_dim)
- # Load saved network state
- self.network.load_state_dict(HyperTreeNetAR._network_states)
+ # self.network holds this instance's trained weights (boosting
+ # updated it in place; see NoDeepcopyObjective).
self.network.eval()
if type == "forecast":
@@ -876,13 +1022,28 @@ def forecast(
lags = np.concatenate([next_val, lags[:, :-1]], axis=1)
# Create output dataframe based on requested type
+ model_name = f"Hyper-TreeNet-AR({self.p})"
out_df = pd.DataFrame({
"series_id": test_data["series_id"].to_numpy().flatten(),
"date": test_data["date"].to_numpy().flatten(),
"fcst": np.hstack(forecasts).flatten(),
- "model": f"Hyper-TreeNet-AR({self.p})",
+ "model": model_name,
})
+ if level is not None:
+ point = np.hstack(forecasts) # (n_series, fcst_h)
+ columns = interval_columns(
+ point=point,
+ scores=self._cs_scores,
+ levels=level,
+ method=self._pi_config.method,
+ model_name=model_name,
+ cal_order=self._cs_series_order,
+ target_order=list(test_series_ids),
+ )
+ for col_name, values in columns.items():
+ out_df[col_name] = values
+
elif type == "parameters":
with torch.no_grad():
params_fcst = (self.network(gbdt_embeds)
@@ -913,4 +1074,4 @@ def forecast(
return out_df
except Exception as e:
- raise RuntimeError(f"Forecasting not successful: {str(e)}")
+ raise RuntimeError(f"Forecasting not successful: {str(e)}") from e
diff --git a/hypertrees/models/HyperTreeNetARMA.py b/hypertrees/models/HyperTreeNetARMA.py
new file mode 100644
index 0000000..2adf158
--- /dev/null
+++ b/hypertrees/models/HyperTreeNetARMA.py
@@ -0,0 +1,1182 @@
+import warnings
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+from torch.autograd import grad as autograd
+import lightgbm as lgb
+from typing import Tuple, Callable, Optional, List
+import time
+from ..utils import CustomLogger
+lgb.register_logger(CustomLogger())
+
+from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order, extract_forecast_lags, GaussNewtonHessian, NoDeepcopyObjective
+from ..conformal import (
+ ForecastIntervals,
+ validate_calibration_length,
+ rolling_origin_residuals,
+ interval_columns,
+)
+from .HyperTreeAR import HyperTreeAR
+from .mlp import MLP
+
+warnings.filterwarnings(
+ "ignore",
+ message="Using backward\\(\\) with create_graph=True will create a reference cycle.*"
+)
+
+
+class HyperTreeNetARMA:
+ """
+ Class that implements a Hyper-TreeNet-ARMA(p, q) model for time series forecasting.
+
+ It combines LightGBM with a neural network, where the LightGBM first
+ creates embeddings from the input data which are then mapped as parameters
+ to the target ARMA model
+
+ y_t = sum_{j=1..p} phi_j(x_t) * y_{t-j} + sum_{i=1..q} theta_i(x_t) * eps_{t-i} + eps_t
+
+ so that the AR coefficients phi_j and the MA coefficients theta_i are
+ time-varying. The MA block is an error-correction mechanism: it regresses
+ on the model's own past one-step forecast errors, adjusting the forecast
+ when recent periods were over- or under-predicted.
+
+ As in ``HyperTreeARMA``, the latent innovations are obtained recursion-free
+ via the two-stage Hannan-Rissanen approach:
+
+ 1. **Stage 1**: a long autoregression (a direct ``HyperTreeAR`` of order
+ ``stage1_p``, by default the Gomez-Maravall proposal
+ ``max(floor(log(T)**2), 2 * max(p, q))`` used by statsmodels'
+ ``hannan_rissanen`` and RATS' ``@HannanRissanen``) is fitted to the
+ training data and its in-sample one-step residuals
+ ``eps_hat_t = y_t - y_hat_t`` are extracted. The residual extractor
+ stays a direct (one tree per lag) AR by design: it is fitted once,
+ analytic Hessians keep it cheap, and ``HyperTreeARMA`` /
+ ``HyperTreeNetARMA`` then share identical residual proxies.
+ 2. **Stage 2**: the lagged residuals are treated as *observed* regressors
+ in the widened design ``[y_{t-1..t-p}, eps_hat_{t-1..t-q}]``, and the
+ GBDT encoder + MLP decoder maps features to the ``p + q`` coefficients
+ applied to it.
+
+ Training uses separated gradient flows (Option 2 in the paper, which the
+ ablations found indistinguishable from the shared-flow Option 1): per
+ boosting iteration the MLP takes one optimizer step on the current GBDT
+ embeddings, then gradients and Hessians for the GBDT are computed through
+ the updated network in inference mode. As in HyperTreeNetAR, the MLP
+ decoder lives on the instance (``self.network``) and is updated in place
+ during boosting; there is deliberately no shared/class-level network state.
+ Unlike ``HyperTreeNetAR``, this model exposes no ``gradient_mode`` option;
+ only the separated flow (Option 2) is implemented.
+
+ Beyond the forecast origin, future innovations are unobserved with
+ expectation zero, so the MA terms contribute to the first ``q`` horizon
+ steps (multiplying the known last residuals) and then vanish, leaving the
+ pure AR recursion.
+
+ Key features:
+ - Combines LightGBM and a neural network for ARMA time series modeling
+ - Allows AR and MA coefficients to vary based on features
+ - Recursion-free estimation via Hannan-Rissanen residual proxies
+ - Boosting cost of the stage-2 model is independent of ``p + q``
+
+ Use this model when:
+ - The series has short-memory error-correction structure that a pure
+ AR(p) of moderate order does not capture
+ - The lag structure implies many coefficients, since GBDTs do not scale
+ well with the number of parameters
+ - You want to leverage LightGBM for feature encoding and a neural network
+ for the mapping from embeddings to ARMA coefficients
+
+ References
+ ----------
+ [1] Hannan, E. J., & Rissanen, J. (1982). Recursive Estimation of Mixed
+ Autoregressive-Moving Average Order. Biometrika, 69(1), 81-94.
+ [2] Gomez, V., & Maravall, A. (2001). Automatic Modeling Methods for
+ Univariate Series. In Pena, Tiao & Tsay (eds.), A Course in Time
+ Series Analysis. Wiley. (Default order of the stage-1 long AR.)
+
+ Example usage:
+ ```python
+ # Imports
+ from hypertrees.models import HyperTreeNetARMA
+ import torch
+ import pandas as pd
+ import matplotlib.pyplot as plt
+
+ # Initialize model
+ lag_p = 2
+ lag_q = 1
+ frequency = 'M'
+ fcst_h = 12
+ model = HyperTreeNetARMA(
+ p=lag_p,
+ q=lag_q,
+ freq=frequency,
+ fcst_h=fcst_h,
+ device="cuda" if torch.cuda.is_available() else "cpu"
+ )
+
+ # Data
+ # The data needs to have the following columns: 'date', 'series_id', 'value'. All other columns are automatically treated as features.
+ # You don't have to add lag-values or residuals yourself, this happens automatically during training.
+ df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/air-passengers.csv', parse_dates=['ds'])
+ df.rename(columns={'unique_id': 'series_id', 'ds': 'date', 'y': 'value'}, inplace=True)
+ df['month'] = df['date'].dt.month
+ df["quarter"] = df['date'].dt.quarter
+ test = df.tail(fcst_h)
+ train = df.drop(test.index)
+
+ # Train model
+ model.train(
+ lgb_params={'learning_rate': 1e-1},
+ network_params={
+ 'learning_rate': 1e-3, # learning rate for the neural network optimizer
+ 'embedding_dimension': 1, # embedding dimension for tree-embeddings
+ 'hidden_dim': 128, # hidden dimension for the MLP network
+ 'dropout': 0.1, # dropout rate for the MLP network
+ 'use_random_projection': True, # whether to use random projections for the embeddings
+ 'rp_embed_dim': lag_p + lag_q, # dimension of the random projections (if used)
+ },
+ num_iterations=100,
+ train_data=train,
+ seed=123,
+ verbose=-1
+ )
+
+ # Generate forecasts and inspect the time-varying ARMA coefficients
+ forecasts = model.forecast(test_data=test)
+ coefficients = model.forecast(test_data=test, type="parameters")
+
+ # Plot results
+ datasets = [
+ (df, 'date', 'value', 'Actual', '#2E86AB', '-'),
+ (forecasts, 'date', 'fcst', 'Forecast', '#F18F01', '--')
+ ]
+
+ for data, x_col, y_col, label, color, style in datasets:
+ plt.plot(data[x_col], data[y_col], label=label, color=color,
+ linestyle=style, linewidth=2, alpha=0.8)
+
+ plt.title('AirPassengers - Forecast', fontsize=14)
+ plt.legend(frameon=True, fancybox=True)
+ plt.grid(True, alpha=0.3)
+ plt.tight_layout()
+ ```
+ """
+
+ def __init__(
+ self,
+ p: int = 2,
+ q: int = 1,
+ freq: str = "M",
+ fcst_h: int = 1,
+ loss_fn: Callable = nn.MSELoss(),
+ device: str = "cpu",
+ hessian_method: str = "exact",
+ n_hessian_probes: int = 5,
+ stage1_p: Optional[int] = None,
+ ):
+ """
+ Initialize the Hyper-TreeNet-ARMA(p, q) model.
+
+ Arguments
+ ----------
+ p : int
+ Number of AR lags. Must be a positive integer.
+ q : int
+ Number of MA terms (lagged residual regressors). Must be a
+ positive integer; for q = 0 use ``HyperTreeNetAR`` directly.
+ freq : str
+ Frequency of the time series (e.g., 'D' for daily, 'M' for monthly,
+ 'Q' for quarterly, 'Y' for yearly).
+ fcst_h : int
+ Forecast horizon (number of periods to forecast ahead).
+ loss_fn : Callable
+ Loss function for optimization. Must be a PyTorch loss function.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
+ device : str
+ Device to run the model on. Default is 'cpu'.
+ This allows for GPU acceleration of network training if available.
+ hessian_method : str
+ Method for computing the Hessian diagonal. Options:
+ - "exact": Exact diagonal Hessian via per-embedding-dimension
+ second-order autograd (cheap, since the embedding is
+ low-dimensional).
+ - "gn": Gauss-Newton approximation estimated via Hutchinson probing.
+ Guarantees positive semi-definite Hessians. Avoids second-order
+ differentiation at the cost of Hutchinson estimation variance.
+ n_hessian_probes : int
+ Number of Hutchinson probes for Gauss-Newton Hessian diagonal estimation.
+ Only used when hessian_method="gn". More probes reduce variance but
+ increase computation. Default is 5.
+ stage1_p : int, optional
+ Lag order of the stage-1 autoregression used to extract the
+ residual proxies (the Hannan-Rissanen "long AR"). If None
+ (default), it is resolved at training time via the
+ Gomez-Maravall (2001) proposal used by statsmodels and RATS:
+ ``max(floor(log(T)**2), 2 * max(p, q))``, with ``T`` the
+ shortest series length. Larger values give cleaner residual
+ proxies at the cost of dropping more training rows: stage-2
+ training uses rows from ``max(p, stage1_p + q) + 1`` onward per
+ series. Pass a smaller value explicitly for short series.
+ """
+ # Validate inputs
+ if not isinstance(p, int) or p <= 0:
+ raise ValueError("Parameter 'p' must be a positive integer.")
+ if not isinstance(q, int) or q <= 0:
+ raise ValueError(
+ "Parameter 'q' must be a positive integer. For q = 0 (no MA "
+ "terms) use HyperTreeNetAR directly."
+ )
+ if fcst_h <= 0:
+ raise ValueError("Forecast horizon 'fcst_h' must be a positive integer.")
+ if not isinstance(freq, str):
+ raise TypeError("freq must be a string.")
+ if not isinstance(loss_fn, nn.Module):
+ raise TypeError("loss_fn must be a PyTorch loss function.")
+ if isinstance(loss_fn, nn.L1Loss):
+ raise ValueError(
+ "nn.L1Loss is not supported: its second derivative is zero almost "
+ "everywhere, so LightGBM's Newton boosting receives all-zero Hessians "
+ "and cannot grow trees. Use nn.HuberLoss or nn.SmoothL1Loss for an "
+ "MAE-like loss with usable curvature."
+ )
+ if getattr(loss_fn, "reduction", "mean") == "none":
+ raise ValueError(
+ "loss_fn must use a scalar reduction ('mean' or 'sum'); "
+ "reduction='none' returns per-element losses that the "
+ "boosting objective cannot consume."
+ )
+ if hessian_method not in ("exact", "gn"):
+ raise ValueError("hessian_method must be either 'exact' or 'gn'.")
+ if not isinstance(n_hessian_probes, int) or n_hessian_probes <= 0:
+ raise ValueError("n_hessian_probes must be a positive integer.")
+ if stage1_p is not None and (not isinstance(stage1_p, int) or stage1_p <= 0):
+ raise ValueError("stage1_p must be a positive integer.")
+
+ if hessian_method == "gn" and not isinstance(loss_fn, nn.MSELoss):
+ warnings.warn(
+ f"Loss {loss_fn.__class__.__name__} is not nn.MSELoss. The Gauss-Newton "
+ "Hessian requires a twice-differentiable loss; non-smooth losses "
+ "(e.g., L1Loss, quantile loss, HuberLoss/SmoothL1Loss outside the quadratic "
+ "region) have zero or undefined second derivatives at kinks, "
+ "causing degenerate Hessians."
+ )
+
+ self.p = p
+ self.q = q
+ self.n_params = p + q
+ self._stage1_p_arg = stage1_p
+ self.stage1_p = stage1_p # resolved at training time when None
+ self.freq = freq
+ self.fcst_h = fcst_h
+ self.loss_fn = loss_fn
+ self.loss_name = self.loss_fn.__class__.__name__
+ self.dtype = torch.float32
+ self.device = device
+ self.model = None
+ self.features = None # Stores feature names after training
+ self.is_trained = False # Flag to track if model has been trained
+ self.dataset_references = {} # Store references to LightGBM datasets
+ self.hessian_method = hessian_method
+ self.n_hessian_probes = n_hessian_probes
+ self.network = None
+ self.optimizer = None
+ self.embedding_dim = None
+ self._stage1 = None # Trained stage-1 HyperTreeAR (residual extractor)
+ self.fcst_lags = None # {series_id: last p values, newest first}
+ self.fcst_eps = None # {series_id: last q stage-1 residuals, newest first}
+ self._iter_count = 0
+ self._fit = None
+ self._target = None
+
+ # Conformal prediction interval state (populated when train() is called
+ # with forecast_intervals).
+ self._is_calibrated = False
+ self._cs_scores = None # conformity scores (n_windows, n_series, fcst_h)
+ self._cs_series_order = None # series order along axis 1 of _cs_scores
+ self._pi_config = None # ForecastIntervals configuration
+
+ if hessian_method == "gn":
+ self._gn_hessian = GaussNewtonHessian(loss_fn, n_hessian_probes, self.dtype)
+
+ def objective_fn(self, predt: np.ndarray, data: lgb.Dataset) -> Tuple[np.ndarray, np.ndarray]:
+ """
+ Custom objective function for LightGBM training.
+
+ This function defines the gradients and hessians for the LightGBM model
+ based on the PyTorch loss function. It converts the raw LightGBM outputs
+ to embeddings, updates the MLP, computes the loss through the updated
+ network, and then backpropagates to get gradients.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM, representing the GBDT embeddings.
+ data : lgb.Dataset
+ LightGBM dataset containing the target values.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians for LightGBM optimization.
+ """
+ self._iter_count += 1
+
+ target = torch.tensor(data.get_label().reshape(-1, 1), dtype=self.dtype, device=self.device)
+ embeds, loss = self.get_embeds_loss_separate(predt, target, self.design_train)
+ grad, hess = self.calculate_gradients_and_hessians(loss, embeds)
+
+ return grad, hess
+
+ def eval_fn(self, predt: np.ndarray, eval_data: lgb.Dataset) -> Tuple[str, float, bool]:
+ """
+ Custom evaluation function for evaluating forecast accuracy on an evaluation dataset.
+
+ This function computes the loss value to be monitored during evaluation.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM.
+ eval_data : lgb.Dataset
+ LightGBM dataset containing the evaluation data.
+
+ Returns
+ -------
+ Tuple[str, float, bool]
+ Name of the metric, value of the metric, and whether to maximize it.
+ """
+ # Use appropriate design rows based on dataset name
+ dataset_name = self.dataset_references.get(id(eval_data), "unknown")
+ if dataset_name == "train":
+ design = self.design_train
+ elif dataset_name == "validation":
+ design = self.design_eval
+ else:
+ # Default to training design if unknown
+ design = self.design_train
+ warnings.warn("Unknown dataset in metric_fn. Using training design.")
+
+ # Calculate loss
+ is_higher_better = False # Lower loss is better, so we don't maximize
+ target = torch.tensor(eval_data.get_label().reshape(-1, 1), dtype=self.dtype, device=self.device)
+
+ # For evaluation, we need to compute loss without any backward pass or gradient computation
+ gbdt_embed = torch.tensor(
+ predt.reshape(-1, self.embedding_dim, order="F"),
+ dtype=self.dtype
+ ).to(self.device)
+
+ # self.network is the live module the objective updated in this same
+ # boosting iteration (guaranteed by NoDeepcopyObjective).
+
+ # Compute loss without gradients
+ self.network.eval()
+ with torch.no_grad():
+ arma_params = self.network(gbdt_embed)
+ fcst = torch.sum(arma_params * design, dim=1).unsqueeze(1)
+ loss = self.loss_fn(fcst, target)
+
+ return self.loss_name, loss.item(), is_higher_better
+
+ def get_embeds_loss_separate(
+ self,
+ predt: np.ndarray,
+ target: torch.Tensor,
+ design: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
+ """
+ Transform LightGBM outputs into embeddings and calculate loss for separate gradients (Option 2).
+
+ This function:
+ 1. Reshapes the raw outputs into tree embeddings
+ 2. Maps embeddings to ARMA coefficients via the MLP and takes one optimizer step
+ 3. Recomputes the coefficients through the updated network in inference mode
+ 4. Calculates the GBDT loss between fitted and actual values
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM.
+ target : torch.Tensor
+ Target values (actual time series values).
+ design : torch.Tensor
+ Joint design rows ``[y_{t-1..t-p}, eps_hat_{t-1..t-q}]``,
+ shape ``(n_samples, p + q)``.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor]
+ GBDT embeddings and loss value.
+ """
+ # Reshape outputs into embedding matrix (samples × embedding_dim)
+ # The 'F' order means Fortran-style ordering (column-major)
+ gbdt_embed = torch.tensor(
+ predt.reshape(-1, self.embedding_dim, order="F"),
+ requires_grad=True,
+ dtype=self.dtype,
+ device=self.device
+ )
+
+ # Train network (forward pass)
+ self.network.train()
+ arma_params_net = self.network(gbdt_embed)
+ fcst_net = torch.sum(arma_params_net * design, dim=1).unsqueeze(1)
+ network_loss = self.loss_fn(fcst_net, target)
+ self.optimizer.zero_grad()
+ network_loss.backward()
+ self.optimizer.step()
+
+ # Calculate loss for GBDT
+ self.network.eval()
+ arma_params_gbdt = self.network(gbdt_embed)
+ fcst_gbdt = torch.sum(arma_params_gbdt * design, dim=1).unsqueeze(1)
+ gbm_loss = self.loss_fn(fcst_gbdt, target)
+
+ if self.hessian_method == "gn":
+ self._fit = fcst_gbdt
+ self._target = target
+
+ return gbdt_embed, gbm_loss
+
+ def _calculate_gradients_and_hessians_separate(
+ self, loss: torch.Tensor, embeds: torch.Tensor,
+ ) -> Tuple[np.ndarray, np.ndarray]:
+ """
+ Compute gradients and hessians for LightGBM optimization using separate gradients (Option 2).
+
+ This function computes first and second-order derivatives needed for
+ gradient boosting optimization in LightGBM.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ embeds : torch.Tensor
+ GBDT embeddings.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ # Compute gradients
+ grad = autograd(loss, inputs=embeds, create_graph=True)[0]
+
+ # Compute hessians. We compute the diagonal of the Hessian matrix for each parameter separately
+ hess = [
+ autograd(grad[:, i].sum(), embeds, retain_graph=True)[0][:, i:(i + 1)]
+ for i in range(self.embedding_dim)
+ ]
+
+ # Convert to numpy arrays and reshape as expected by LightGBM
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = torch.cat(hess, dim=1).cpu().detach().numpy().ravel(order="F")
+
+ return grad, hess
+
+ def _calculate_gradients_and_hessians_separate_gn(
+ self, loss: torch.Tensor, embeds: torch.Tensor,
+ ) -> Tuple[np.ndarray, np.ndarray]:
+ """Gauss-Newton Hessian for separate gradient mode via Hutchinson probing.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ embeds : torch.Tensor
+ GBDT embeddings, shape ``(n_samples, embedding_dim)``.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ grad = autograd(loss, inputs=embeds, retain_graph=True)[0]
+ rng = torch.Generator().manual_seed(self._iter_count)
+ hess = self._gn_hessian.estimate(self._fit, self._target, embeds, rng)
+ self._fit = None
+ self._target = None
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = hess.cpu().detach().numpy().ravel(order="F")
+
+ return grad, hess
+
+ def _stage1_residual_frame(self, data: pd.DataFrame) -> pd.DataFrame:
+ """Attach the stage-1 in-sample residuals to a sorted copy of *data*.
+
+ Runs the trained stage-1 AR over *data* to compute its one-step
+ in-sample residuals ``eps_hat_t = y_t - y_hat_t``. The first
+ ``stage1_p`` rows of each series have no stage-1 fit and carry NaN.
+
+ Parameters
+ ----------
+ data : pd.DataFrame
+ DataFrame with ``series_id``, ``date``, ``value`` and the
+ training feature columns, ordered by ``(series_id, date)``.
+
+ Returns
+ -------
+ pd.DataFrame
+ Copy of *data*, sorted by ``(series_id, date)``, with an added
+ ``resid`` column (NaN for the first ``stage1_p`` rows per series).
+ """
+ preprocessor = TimeSeriesPreprocessor(
+ freq=self.freq,
+ lags=[i for i in range(1, self.stage1_p + 1)],
+ )
+ lagged = preprocessor.create_lags(data)
+ lagged_dict = preprocessor.extract(lagged)
+
+ # Predict the stage-1 AR coefficients on the lagged rows; enforce the
+ # stage-1 model's training feature order for the Booster.
+ params = np.asarray(
+ self._stage1.model.predict(lagged_dict["features"][self._stage1.features])
+ )
+ # Booster.predict returns (n_rows, stage1_p) for multi-class output
+ if params.ndim == 1:
+ params = params.reshape(-1, self.stage1_p)
+ fit = (params * lagged_dict["lags_target"]).sum(axis=1)
+ resid = lagged_dict["target"].ravel() - fit
+
+ # Align back: `lagged` equals the sorted frame minus the first
+ # stage1_p rows of each series, in the same row order.
+ work = data.sort_values(["series_id", "date"]).reset_index(drop=True).copy()
+ occ = work.groupby("series_id", sort=False).cumcount()
+ work["resid"] = np.nan
+ work.loc[occ >= self.stage1_p, "resid"] = resid
+
+ return work
+
+ def train(
+ self,
+ lgb_params: dict = None,
+ network_params: dict = None,
+ num_iterations: int = 100,
+ train_data: pd.DataFrame = None,
+ validation: bool = False,
+ early_stopping_round: Optional[int] = None,
+ seed: int = 123,
+ verbose: int = -1,
+ deterministic: bool = True,
+ forecast_intervals: Optional[ForecastIntervals] = None,
+ ) -> TrainingResult:
+ """
+ Train the Hyper-TreeNet-ARMA model on time series data.
+
+ This method:
+ 1. Trains the stage-1 long autoregression (a direct ``HyperTreeAR`` of
+ order ``stage1_p``) with the same LightGBM hyper-parameters and
+ extracts its in-sample one-step residuals (Hannan-Rissanen)
+ 2. Builds the joint design ``[y-lags, residual-lags]`` and sets up
+ LightGBM datasets
+ 3. Trains the stage-2 GBDT encoder and MLP decoder jointly
+
+ The training data must contain columns:
+ - 'series_id': Identifier for each time series
+ - 'date': Timestamp for each observation
+ - 'value': Target value to forecast
+ - Additional feature columns used for forecasting
+
+ Each series must have at least ``max(p, stage1_p + q) + 1`` rows so
+ that one stage-2 training row remains. Note that the stage-1 model is
+ fitted on the full training data, so with ``validation=True`` the
+ validation metric shares stage-1 information through the residual
+ regressors.
+
+ Parameters
+ ----------
+ lgb_params : dict
+ LightGBM parameters like 'learning_rate', 'num_leaves', etc.
+ Used for both the stage-1 and the stage-2 GBDT.
+ network_params : dict
+ Network parameters. Available parameters are:
+ - "learning_rate": Learning rate for the neural network optimizer
+ - "hidden_dim": Dimension of the hidden layer in the MLP
+ - "embedding_dimension": Dimension of the tree embeddings from LightGBM
+ - "use_random_projection": Whether to use random projection for embeddings
+ - "rp_embed_dim": Dimension of the random projection embeddings (if used)
+ - "dropout": Dropout rate for regularization
+ num_iterations : int
+ Number of boosting rounds for training (both stages)
+ train_data : pd.DataFrame
+ Training data containing series_id, date, value and feature columns
+ validation : bool
+ If True, a validation set will be created for evaluation. It splits the last fcst_h values of each
+ series for validation.
+ early_stopping_round : int, optional
+ If provided, training will stop if the validation loss does not improve for this many rounds.
+ seed : int
+ Random seed for reproducibility
+ verbose : int
+ Verbosity level for LightGBM training
+ deterministic : bool
+ If True, sets LightGBM's ``deterministic`` and ``force_row_wise`` parameters to ensure
+ reproducible results. May slow down training. See
+ https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic
+ forecast_intervals : ForecastIntervals, optional
+ If provided, calibrate conformal prediction intervals via rolling-window
+ cross-validation after the main model is trained. The collected conformity
+ scores are then used by ``forecast(..., level=[...])`` to produce
+ ``-lo-`` / ``-hi-`` columns. See
+ :class:`hypertrees.conformal.ForecastIntervals`.
+
+ Returns
+ -------
+ TrainingResult
+ Object containing evaluation results and training information
+ for the stage-2 model.
+ """
+ # Validate inputs
+ if train_data is None:
+ raise ValueError("train_data must be provided.")
+ if lgb_params is None:
+ raise ValueError("lgb_params must be provided.")
+ if network_params is None:
+ raise ValueError("network_params must be provided.")
+ if not isinstance(train_data, pd.DataFrame):
+ raise TypeError("train_data must be a pandas DataFrame.")
+ if not isinstance(lgb_params, dict):
+ raise TypeError("lgb_params must be a dictionary.")
+ if not isinstance(network_params, dict):
+ raise TypeError("network_params must be a dictionary.")
+ if not isinstance(num_iterations, int) or num_iterations <= 0:
+ raise ValueError("num_iterations must be a positive integer.")
+ if not isinstance(seed, int):
+ raise TypeError("seed must be an integer.")
+ if not isinstance(verbose, int):
+ raise TypeError("verbose must be an integer.")
+ if early_stopping_round is not None and (not isinstance(early_stopping_round, int) or early_stopping_round <= 0):
+ raise ValueError("early_stopping_round must be a positive integer.")
+ if not isinstance(validation, bool):
+ raise TypeError("validation must be a boolean.")
+ if not isinstance(deterministic, bool):
+ raise TypeError("deterministic must be a boolean.")
+ if forecast_intervals is not None and not isinstance(forecast_intervals, ForecastIntervals):
+ raise TypeError("forecast_intervals must be a ForecastIntervals instance.")
+ if early_stopping_round is not None and not validation:
+ raise ValueError("early_stopping_round can only be used when validation is True.")
+ if validation and early_stopping_round is None:
+ raise ValueError("early_stopping_round must be provided when validation is True.")
+
+ required_net_keys = (
+ "learning_rate", "embedding_dimension", "hidden_dim",
+ "dropout", "use_random_projection",
+ )
+ missing_keys = [key for key in required_net_keys if key not in network_params]
+ if missing_keys:
+ raise ValueError(f"network_params is missing required keys: {missing_keys}")
+ if network_params.get("use_random_projection") and "rp_embed_dim" not in network_params:
+ raise ValueError(
+ "network_params is missing required keys: ['rp_embed_dim'] "
+ "(required when use_random_projection=True)."
+ )
+
+ if deterministic:
+ lgb_params = {**lgb_params, "deterministic": True, "force_row_wise": True}
+
+ # Check required columns
+ required_columns = ['series_id', 'date', 'value']
+ for col in required_columns:
+ if col not in train_data.columns:
+ raise ValueError(f"Required column '{col}' not found in training data.")
+
+ # Validate row ordering: each series must be a contiguous block with
+ # monotonic dates so the training reshape and forecast seeds align.
+ validate_series_order(train_data, name="train_data")
+
+ # Resolve the stage-1 long-AR order. The default follows the
+ # Gomez-Maravall (2001) proposal used by statsmodels'
+ # hannan_rissanen and RATS' @HannanRissanen: the long AR grows with
+ # the sample so the residual proxies stay consistent.
+ lengths = train_data.groupby("series_id", sort=False).size()
+ if self._stage1_p_arg is not None:
+ self.stage1_p = self._stage1_p_arg
+ else:
+ t_min = int(lengths.min())
+ self.stage1_p = max(
+ int(np.floor(np.log(t_min) ** 2)), 2 * max(self.p, self.q)
+ )
+
+ # Each series must keep at least one stage-2 training row.
+ needed = max(self.p, self.stage1_p + self.q) + 1
+ bad = lengths[lengths < needed]
+ if len(bad) > 0:
+ raise ValueError(
+ f"Series too short for stage1_p={self.stage1_p} and q={self.q}: "
+ f"each series needs at least max(p, stage1_p + q) + 1 = {needed} "
+ f"rows, but these series are shorter: {bad.to_dict()}. Pass a "
+ f"smaller stage1_p to HyperTreeNetARMA for short series."
+ )
+
+ # Fail fast if any series is too short for the requested conformal
+ # calibration. The stage-2 ARMA needs max(p, stage1_p + q) + 1 rows to
+ # retain one training sample.
+ if forecast_intervals is not None:
+ validate_calibration_length(
+ train_data, self.fcst_h, forecast_intervals,
+ min_train=max(self.p, self.stage1_p + self.q) + 1,
+ )
+
+ # Set the embedding dimension and select the gradient computation
+ # based on the Hessian method
+ self.embedding_dim = network_params["embedding_dimension"]
+ if self.hessian_method == "exact":
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_separate
+ else:
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_separate_gn
+
+ # General model parameters. The objective wrapper stops lgb.train's
+ # params deepcopy from cloning this instance (see NoDeepcopyObjective).
+ self.lgb_params = {
+ "num_class": self.embedding_dim,
+ "objective": NoDeepcopyObjective(self.objective_fn),
+ "metric": "None",
+ "random_seed": seed,
+ "verbose": verbose
+ }
+
+ # Update with user-provided LightGBM parameters
+ self.lgb_params.update(lgb_params)
+
+ # Reset state for re-training
+ self._iter_count = 0
+ self._fit = None
+ self._target = None
+ self.model = None
+ self.network = None
+ self.optimizer = None
+ self._stage1 = None
+ self.fcst_lags = None
+ self.fcst_eps = None
+ self.dataset_references = {}
+ self.is_trained = False
+ self.features = None
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
+
+ try:
+ # Stage 1 (Hannan-Rissanen): fit the long autoregression and
+ # extract its in-sample one-step residuals as MA-term proxies.
+ self._stage1 = HyperTreeAR(
+ p=self.stage1_p,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ hessian_method="analytic",
+ )
+ self._stage1.train(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ train_data=train_data,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+ work = self._stage1_residual_frame(train_data)
+
+ # Stage 2: build the joint design. The y-lags come from the
+ # standard preprocessor; the residual lags are appended as
+ # lag{p+1}..lag{p+q} so the shared extract()/prepare_datasets
+ # machinery picks up the joint [y-lags | eps-lags] design as one
+ # (n_samples, p + q) tensor while keeping the residual columns
+ # out of the GBDT feature set.
+ preprocessor = TimeSeriesPreprocessor(
+ freq=self.freq,
+ lags=[i for i in range(1, self.p + 1)],
+ )
+ full_ts = preprocessor.create_lags(work.drop(columns=["resid"]))
+
+ resid_grouped = work.groupby("series_id", sort=False)["resid"]
+ elag_names = []
+ elags = {}
+ for i in range(1, self.q + 1):
+ name = f"lag{self.p + i}"
+ elags[name] = resid_grouped.shift(i)
+ elag_names.append(name)
+ occ = work.groupby("series_id", sort=False).cumcount()
+ elag_df = pd.DataFrame(elags)[(occ >= self.p).to_numpy()].reset_index(drop=True)
+ full_ts = pd.concat([full_ts, elag_df], axis=1)
+ # Drop rows without q valid residual lags (the head of each
+ # series up to stage1_p + q observations).
+ full_ts = full_ts.dropna(subset=elag_names).reset_index(drop=True)
+
+ full_dict = preprocessor.extract(full_ts)
+
+ # Store feature names for later use
+ self.features = full_dict["features"].columns.tolist()
+
+ # Prepare datasets
+ (valid_sets,
+ valid_names,
+ callbacks,
+ evals_result,
+ design_train,
+ design_eval,
+ self.dataset_references) = (
+ prepare_datasets(
+ full_ts=full_ts,
+ preprocessor=preprocessor,
+ fcst_h=self.fcst_h,
+ dtype=self.dtype,
+ validation=validation,
+ early_stopping_round=early_stopping_round
+ )
+ )
+
+ # Store design rows for training and evaluation
+ self.design_train = design_train.to(self.device) if design_train is not None else None
+ self.design_eval = design_eval.to(self.device) if design_eval is not None else None
+
+ # Store the value and residual seeds to be used in the forecast method
+ self.set_forecast_origin(train_data)
+
+ # Seed torch before constructing the MLP so initialization (and dropout
+ # draws during training) are reproducible even when the random
+ # projection layer -- whose constructor reseeds torch -- is disabled.
+ torch.manual_seed(seed)
+
+ self.network = MLP(
+ tree_embed_dim=self.embedding_dim,
+ output_dim=self.n_params,
+ hidden_dim=network_params["hidden_dim"],
+ use_random_projection=network_params["use_random_projection"],
+ rp_embed_dim=network_params["rp_embed_dim"] if network_params["use_random_projection"] else None,
+ dropout_rate=network_params["dropout"],
+ seed=seed
+ ).to(self.device)
+ self.optimizer = torch.optim.Adam(self.network.parameters(), lr=network_params["learning_rate"])
+
+ # Train LightGBM model
+ start_time = time.time()
+ self.model = lgb.train(
+ self.lgb_params,
+ valid_sets[0],
+ num_boost_round=num_iterations,
+ feval=self.eval_fn if validation else None,
+ valid_sets=valid_sets,
+ valid_names=valid_names,
+ callbacks=callbacks
+ )
+ training_time = time.time() - start_time
+
+ # Set trained flag to True
+ self.is_trained = True
+
+ # Calibrate conformal prediction intervals via rolling-window CV.
+ # Fresh model instances are trained per window (no forecast_intervals
+ # passed, so there is no recursion) using the same hyper-parameters.
+ if forecast_intervals is not None:
+ def _model_factory():
+ return HyperTreeNetARMA(
+ p=self.p,
+ q=self.q,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ device=self.device,
+ hessian_method=self.hessian_method,
+ n_hessian_probes=self.n_hessian_probes,
+ stage1_p=self.stage1_p,
+ )
+
+ cal_train_kwargs = dict(
+ lgb_params=lgb_params,
+ network_params=network_params,
+ num_iterations=num_iterations,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+ self._cs_scores, self._cs_series_order = rolling_origin_residuals(
+ model_factory=_model_factory,
+ train_data=train_data,
+ fcst_h=self.fcst_h,
+ forecast_intervals=forecast_intervals,
+ train_kwargs=cal_train_kwargs,
+ )
+ self._pi_config = forecast_intervals
+ self._is_calibrated = True
+
+ # Return results
+ result = TrainingResult(
+ train_metrics=evals_result["train"] if validation else {"loss": []},
+ validation_metrics=evals_result["validation"] if validation else None,
+ best_iteration=self.model.best_iteration if self.model.best_iteration > 0 else num_iterations,
+ training_time=training_time
+
+ )
+
+ return result
+
+ except Exception as e:
+ self.is_trained = False
+ raise RuntimeError(f"Training failed: {str(e)}") from e
+
+ def set_forecast_origin(self, history: pd.DataFrame) -> None:
+ """Re-anchor the ARMA value and residual seeds to the end of *history*.
+
+ Recomputes the last ``p`` observed values and the last ``q`` stage-1
+ residuals per series without retraining either GBDT. Used by conformal
+ calibration with ``refit=False``.
+
+ Parameters
+ ----------
+ history : pd.DataFrame
+ DataFrame with ``series_id``, ``date``, ``value`` and the training
+ feature columns, ordered by ``(series_id, date)`` with each series
+ in a contiguous block. Each series must have at least
+ ``max(p, stage1_p + q)`` observations so that the residual seed
+ exists.
+ """
+ if self._stage1 is None or self._stage1.model is None:
+ raise RuntimeError("set_forecast_origin requires a trained model.")
+ validate_series_order(history, name="history")
+
+ needed = max(self.p, self.stage1_p + self.q)
+ lengths = history.groupby("series_id", sort=False).size()
+ bad = lengths[lengths < needed]
+ if len(bad) > 0:
+ raise ValueError(
+ f"history must contain at least max(p, stage1_p + q) = {needed} "
+ f"observations per series. Series too short: {bad.to_dict()}."
+ )
+
+ # Value seed: last p observations per series, newest first.
+ self.fcst_lags = extract_forecast_lags(history, self.p)
+
+ # Residual seed: last q stage-1 residuals per series, newest first.
+ # The stage-1 residuals are the same quantities the MA coefficients
+ # multiplied during training, keeping train and forecast consistent.
+ work = self._stage1_residual_frame(history)
+ tail = work.groupby("series_id", sort=False).tail(self.q)
+ self.fcst_eps = {
+ sid: grp["resid"].to_numpy()[::-1]
+ for sid, grp in tail.groupby("series_id", sort=False)
+ }
+
+ def forecast(
+ self,
+ test_data: pd.DataFrame,
+ type: str = "forecast",
+ level: Optional[List[int]] = None
+ ) -> pd.DataFrame:
+ """
+ Generate forecasts using the trained model.
+
+ This method:
+ 1. Uses the trained model to forecast ARMA coefficients for each test point
+ 2. Recursively generates forecasts using the forecasted coefficients
+
+ The forecasting process implements an ARMA model where:
+ y_t = φ₁(x)y_{t-1} + ... + φₚ(x)y_{t-p} + θ₁(x)ε_{t-1} + ... + θ_q(x)ε_{t-q}
+
+ Past residuals at the forecast origin are known (stage-1 in-sample
+ errors); future innovations are unobserved with expectation zero, so
+ the MA terms correct the first q horizon steps and then vanish,
+ leaving the pure AR recursion.
+
+ Parameters
+ ----------
+ test_data : pd.DataFrame
+ Test data for which to generate forecasts. Must contain the same
+ feature columns used during training.
+ type : str
+ Type of forecast to generate. Options:
+ - "forecast": Generate forecasted values
+ - "parameters": Return the ARMA coefficients used for forecasting
+ - "tree_embeddings": Return the tree embeddings
+ level : list of int, optional
+ Confidence levels (in ``(0, 100)``, e.g. ``[80, 90]``) for conformal
+ prediction intervals. Only valid with ``type="forecast"`` and requires
+ the model to have been trained with ``forecast_intervals=...``. Adds
+ ``-lo-`` / ``-hi-`` columns to the output.
+
+ Returns
+ -------
+ pd.DataFrame
+ Forecasted data with columns:
+ - series_id: Identifier for each time series
+ - date: Forecast date/time
+ - fcst: Forecasted value (if type="forecast")
+ - model: Model name identifier
+ - AR(j) / MA(i): coefficient values (if type="parameters")
+ - tree_embedding_{i}: GBDT tree-embedding dimensions (if type="tree_embeddings")
+ - -lo- / -hi-: prediction interval bounds
+ (if type="forecast" and level is provided)
+ """
+ # Check if model is trained
+ if not self.is_trained or self.model is None:
+ raise RuntimeError("Model has not been trained. Call train() before forecasting.")
+
+ # Validate input data
+ required_cols = ['series_id', 'date']
+ for col in required_cols:
+ if col not in test_data.columns:
+ raise ValueError(f"Required column '{col}' not found in test_data")
+
+ # Validate row ordering: each series must be a contiguous block with
+ # monotonic dates so the forecast reshape aligns forecasts with seeds.
+ validate_series_order(test_data, name="test_data")
+
+ # Validate series IDs match training data
+ test_series_ids = test_data["series_id"].unique()
+ train_series_ids = set(self.fcst_lags.keys())
+ missing = set(test_series_ids) - train_series_ids
+ extra = train_series_ids - set(test_series_ids)
+ if missing or extra:
+ parts = []
+ if missing:
+ parts.append(f"Missing series in training: {missing}")
+ if extra:
+ parts.append(f"Extra series not in test_data: {extra}")
+ raise ValueError(". ".join(parts))
+
+ # Validate rows per series matches fcst_h (forecast only; parameters
+ # and embeddings can be requested for arbitrary-length input).
+ if type == "forecast":
+ rows_per_series = test_data.groupby("series_id", sort=False).size()
+ bad = rows_per_series[rows_per_series != self.fcst_h]
+ if not bad.empty:
+ raise ValueError(
+ f"Each series must have exactly fcst_h={self.fcst_h} rows in test_data. "
+ f"Series with wrong counts: {bad.to_dict()}"
+ )
+
+ # Check that all features used during training exist in test_data
+ missing_features = [f for f in self.features if f not in test_data.columns]
+ if missing_features:
+ raise ValueError(f"Missing features in test_data: {missing_features}")
+
+ # Validate type parameter
+ if type not in ["forecast", "parameters", "tree_embeddings"]:
+ raise ValueError("Parameter 'type' must be either 'forecast', 'parameters' or 'tree_embeddings'.")
+
+ # Validate conformal interval request
+ if level is not None:
+ if type != "forecast":
+ raise ValueError("level is only supported with type='forecast'.")
+ if not self._is_calibrated:
+ raise RuntimeError(
+ "Prediction intervals were requested via level, but the model "
+ "was not calibrated. Pass forecast_intervals=ForecastIntervals(...) "
+ "to train() before forecasting with level."
+ )
+ if not isinstance(level, (list, tuple)) or len(level) == 0:
+ raise ValueError("level must be a non-empty list of integers.")
+ for lv in level:
+ if not isinstance(lv, (int, np.integer)) or not 0 < lv < 100:
+ raise ValueError(f"level values must be integers in (0, 100); got {lv}.")
+
+ model_name = f"Hyper-TreeNet-ARMA({self.p},{self.q})"
+
+ try:
+ # Get tree embeddings
+ # Predict on the DataFrame (not .values) so pandas ``category``
+ # dtype features keep their categorical encoding at forecast time.
+ gbdt_embeds = torch.tensor(
+ self.model.predict(test_data[self.features]),
+ dtype=self.dtype,
+ device=self.device
+ ).reshape(-1, self.embedding_dim)
+
+ # self.network holds this instance's trained weights (boosting
+ # updated it in place; see NoDeepcopyObjective).
+ self.network.eval()
+
+ if type == "forecast":
+ # Forecast coefficients: (n_series, fcst_h, n_params)
+ n_series_test = len(test_series_ids)
+ with torch.no_grad():
+ params_fcst = (self.network(gbdt_embeds)
+ .cpu()
+ .detach()
+ .numpy()
+ .reshape(n_series_test, self.fcst_h, self.n_params))
+
+ # Reconstruct the seed states in the same order as test data
+ lags = np.array([self.fcst_lags[series_id] for series_id in test_series_ids])
+ eps = np.array([self.fcst_eps[series_id] for series_id in test_series_ids])
+
+ # Generate multi-step forecasts
+ forecasts = []
+ for h in range(self.fcst_h):
+ # Compute next value using the ARMA equation:
+ # y_t = φ₁y_{t-1} + ... + φₚy_{t-p} + θ₁ε_{t-1} + ... + θ_qε_{t-q}
+ next_val = (
+ np.sum(params_fcst[:, h, :self.p] * lags, axis=1)
+ + np.sum(params_fcst[:, h, self.p:] * eps, axis=1)
+ ).reshape(-1, 1)
+ forecasts.append(next_val)
+
+ # Update the value lags with the new forecast; future
+ # innovations are unobserved with expectation zero, so the
+ # residual state is shifted with zeros (the MA terms die
+ # out after q steps).
+ lags = np.concatenate([next_val, lags[:, :-1]], axis=1)
+ eps = np.concatenate([np.zeros((n_series_test, 1)), eps[:, :-1]], axis=1)
+
+ # Create output dataframe based on requested type
+ out_df = pd.DataFrame({
+ "series_id": test_data["series_id"].to_numpy().flatten(),
+ "date": test_data["date"].to_numpy().flatten(),
+ "fcst": np.hstack(forecasts).flatten(),
+ "model": model_name,
+ })
+
+ # Append conformal prediction intervals if requested.
+ if level is not None:
+ point = np.hstack(forecasts) # (n_series_test, fcst_h)
+ columns = interval_columns(
+ point=point,
+ scores=self._cs_scores,
+ levels=level,
+ method=self._pi_config.method,
+ model_name=model_name,
+ cal_order=self._cs_series_order,
+ target_order=list(test_series_ids),
+ )
+ for col_name, values in columns.items():
+ out_df[col_name] = values
+
+ elif type == "parameters":
+ with torch.no_grad():
+ params_fcst = (self.network(gbdt_embeds)
+ .cpu()
+ .detach()
+ .numpy())
+
+ out_df = pd.DataFrame({
+ "series_id": test_data["series_id"].to_numpy().flatten(),
+ "date": test_data["date"].to_numpy().flatten(),
+ "model": model_name,
+ })
+ # Add the AR and MA coefficients to the dataframe
+ for j in range(self.p):
+ out_df[f"AR({j + 1})"] = params_fcst[:, j].flatten()
+ for i in range(self.q):
+ out_df[f"MA({i + 1})"] = params_fcst[:, self.p + i].flatten()
+
+ elif type == "tree_embeddings":
+ out_df = pd.DataFrame({
+ "series_id": test_data["series_id"].to_numpy().flatten(),
+ "date": test_data["date"].to_numpy().flatten(),
+ "model": model_name,
+ })
+ # Add tree embeddings to the dataframe
+ for i in range(self.embedding_dim):
+ out_df[f"tree_embedding_{i + 1}"] = gbdt_embeds[:, i].cpu().numpy().flatten()
+
+ return out_df
+
+ except Exception as e:
+ raise RuntimeError(f"Forecasting not successful: {str(e)}") from e
diff --git a/hypertrees/models/HyperTreeNetVAR.py b/hypertrees/models/HyperTreeNetVAR.py
new file mode 100644
index 0000000..7f4892f
--- /dev/null
+++ b/hypertrees/models/HyperTreeNetVAR.py
@@ -0,0 +1,684 @@
+import warnings
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+from torch.autograd import grad as autograd
+import lightgbm as lgb
+from typing import Callable, Optional, Tuple
+
+from ..utils import TrainingResult, GaussNewtonHessian
+from ..conformal import ForecastIntervals
+from ._var_base import _HyperTreeVARBase
+from .mlp import MLP
+
+warnings.filterwarnings(
+ "ignore",
+ message="Using backward\\(\\) with create_graph=True will create a reference cycle.*"
+)
+
+
+class HyperTreeNetVAR(_HyperTreeVARBase):
+ """
+ Class that implements a Hyper-TreeNet-VAR(p) model for multivariate time series forecasting.
+
+ It combines LightGBM with a neural network, where the LightGBM first creates
+ embeddings from the input data which are then mapped as parameters to the
+ target vector autoregression. The model learns the full time-varying
+ coefficient matrices A_1(x), ..., A_p(x) over an aligned panel of k series
+ so that y_{i,t} = sum_j A_j[i, :](x_{i,t}) . y_{t-j}, with cross-series
+ dependence captured by the off-diagonal coefficients (see ``_var_base.py``
+ for the formulation, data requirements, coefficient ordering, and the
+ restricted ``type="factor"`` design).
+
+ Training uses separated gradient flows (Option 2 in the paper, which the
+ ablations found indistinguishable from the shared-flow Option 1): per
+ boosting iteration the MLP takes one optimizer step on the current GBDT
+ embeddings, then gradients and Hessians for the GBDT are computed through
+ the updated network in inference mode. As in HyperTreeNetAR, the MLP
+ decoder lives on the instance (``self.network``) and is updated in place
+ during boosting; there is deliberately no shared/class-level network state.
+ Unlike ``HyperTreeNetAR``, this model exposes no ``gradient_mode`` option;
+ only the separated flow (Option 2) is implemented.
+
+ Key features:
+ - Combines LightGBM and a neural network for multivariate forecasting
+ - Learns the time-varying lag matrices A_1, ..., A_p as functions of
+ features (full k x k, or own + factor lags with type="factor")
+ - Captures cross-series (Granger-causal) lead/lag dependencies via the
+ off-diagonal coefficients
+ - Boosting cost is independent of the number of VAR coefficients
+
+ Use this model when:
+ - Your series influence each other and forecasts should exploit
+ cross-series lead/lag structure
+ - The panel implies many coefficients (k * p beyond a few dozen), since
+ GBDTs do not scale well with the number of parameters
+ - You want to leverage LightGBM for feature encoding and a neural network
+ for the mapping from embeddings to VAR coefficients
+
+ Example usage:
+ ```python
+ # Imports
+ from hypertrees.models.HyperTreeNetVAR import HyperTreeNetVAR
+ import numpy as np
+ import pandas as pd
+ import torch
+ import matplotlib.pyplot as plt
+
+ # Initialize model
+ lag_p = 4
+ frequency = 'M'
+ fcst_h = 12
+ model = HyperTreeNetVAR(
+ p=lag_p,
+ freq=frequency,
+ fcst_h=fcst_h,
+ device="cuda" if torch.cuda.is_available() else "cpu"
+ )
+
+ # Data
+ # The data needs to be an aligned panel (equal lengths and identical dates
+ # across series) with the columns: 'date', 'series_id', 'value'. All other
+ # columns are automatically treated as features. You don't have to add
+ # lag-values yourself, this happens automatically during training.
+ rng = np.random.RandomState(1)
+ dates = pd.date_range("2010-01-01", periods=120 + fcst_h, freq="MS")
+ df = pd.concat(
+ [
+ pd.DataFrame({
+ "series_id": f"s{i}",
+ "date": dates,
+ "value": base + np.cumsum(rng.randn(len(dates))),
+ "month": dates.month,
+ "quarter": dates.quarter,
+ "series_num": i, # identifies the series, so equations can differ
+ })
+ for i, base in enumerate([100.0, 150.0])
+ ],
+ ignore_index=True,
+ )
+ test = df.groupby("series_id", sort=False).tail(fcst_h)
+ train = df.drop(test.index)
+
+ # Train model
+ model.train(
+ lgb_params={'learning_rate': 1e-1},
+ network_params={
+ 'learning_rate': 1e-3, # learning rate for the neural network optimizer
+ 'embedding_dimension': 1, # embedding dimension for tree-embeddings
+ 'hidden_dim': 128, # hidden dimension for the MLP network
+ 'dropout': 0.1, # dropout rate for the MLP network
+ 'use_random_projection': True, # whether to use random projections for the embeddings
+ 'rp_embed_dim': 8, # dimension of the random projections (if used)
+ },
+ num_iterations=100,
+ train_data=train,
+ seed=123,
+ verbose=-1
+ )
+
+ # Generate forecasts
+ forecasts = model.forecast(test_data=test)
+
+ # Plot results
+ for sid, group in df.groupby("series_id", sort=False):
+ plt.plot(group["date"], group["value"], label=f"Actual {sid}",
+ color='#2E86AB', linewidth=2, alpha=0.8)
+ for sid, group in forecasts.groupby("series_id", sort=False):
+ plt.plot(group["date"], group["fcst"], label=f"Forecast {sid}",
+ color='#F18F01', linestyle='--', linewidth=2, alpha=0.8)
+
+ plt.title('Aligned Panel - VAR Forecasts', fontsize=14)
+ plt.legend(frameon=True, fancybox=True)
+ plt.grid(True, alpha=0.3)
+ plt.tight_layout()
+ ```
+ """
+
+ _model_label = "Hyper-TreeNet-VAR"
+ _valid_forecast_types = ("forecast", "parameters", "tree_embeddings")
+
+ def __init__(
+ self,
+ p: int = 2,
+ freq: str = "M",
+ fcst_h: int = 1,
+ loss_fn: Callable = nn.MSELoss(),
+ scaling: Optional[str] = "mean",
+ type: str = "full",
+ device: str = "cpu",
+ hessian_method: str = "exact",
+ n_hessian_probes: int = 5,
+ ):
+ """
+ Initialize the Hyper-TreeNet-VAR(p) model.
+
+ Arguments
+ ----------
+ p : int
+ VAR lag order. Must be a positive integer.
+ freq : str
+ Frequency of the time series (e.g., 'D' for daily, 'M' for monthly,
+ 'Q' for quarterly, 'Y' for yearly).
+ fcst_h : int
+ Forecast horizon (number of periods to forecast ahead).
+ loss_fn : Callable
+ Loss function for optimization. Must be a PyTorch loss function.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
+ scaling : str, optional
+ Per-series scaling applied internally before training; forecasts
+ (and prediction intervals) are transformed back to the original
+ scale automatically. Options: "mean" (default; divide by the
+ mean absolute training value), "standard" (z-score), or None.
+ Strongly recommended for heterogeneous panels: VAR coefficients
+ multiply *other* series' values, so unscaled panels force the
+ model to learn scale conversions while the loss is dominated by
+ the largest series. Coefficients returned by
+ ``forecast(type="parameters")`` live in the scaled space.
+ type : str
+ Structure of the VAR design vector. Options:
+ - "full" (default): unrestricted VAR; every equation regresses on
+ the lags of all k series (``k * p`` coefficients per equation,
+ decoded by the MLP).
+ - "factor": restricted GVAR-style design; every equation
+ regresses on its own lags plus the lags of the equal-weighted
+ cross-sectional average of the scaled panel (``2 * p``
+ coefficients per equation, independent of k). Recommended for
+ larger panels, where the unrestricted design overparameterizes.
+ Parameter columns are named ``A{j}(own)`` / ``A{j}(factor)``
+ and the model name becomes ``Hyper-TreeNet-FactorVAR(p)``.
+ device : str
+ Device to run the model on. Default is 'cpu'.
+ This allows for GPU acceleration of network training if available.
+ hessian_method : str
+ Method for computing the Hessian diagonal. Options:
+ - "exact": Exact diagonal Hessian via per-embedding-dimension
+ second-order autograd (cheap, since the embedding is
+ low-dimensional).
+ - "gn": Gauss-Newton approximation estimated via Hutchinson probing.
+ Guarantees positive semi-definite Hessians. Avoids second-order
+ differentiation at the cost of Hutchinson estimation variance.
+ n_hessian_probes : int
+ Number of Hutchinson probes for Gauss-Newton Hessian diagonal estimation.
+ Only used when hessian_method="gn". More probes reduce variance but
+ increase computation. Default is 5.
+ """
+ super().__init__(
+ p=p,
+ freq=freq,
+ fcst_h=fcst_h,
+ loss_fn=loss_fn,
+ scaling=scaling,
+ type=type,
+ )
+ if hessian_method not in ("exact", "gn"):
+ raise ValueError("hessian_method must be either 'exact' or 'gn'.")
+ if not isinstance(n_hessian_probes, int) or n_hessian_probes <= 0:
+ raise ValueError("n_hessian_probes must be a positive integer.")
+ if hessian_method == "gn" and not isinstance(loss_fn, nn.MSELoss):
+ warnings.warn(
+ f"Loss {loss_fn.__class__.__name__} is not nn.MSELoss. The Gauss-Newton "
+ "Hessian requires a twice-differentiable loss; non-smooth losses "
+ "(e.g., L1Loss, quantile loss, HuberLoss/SmoothL1Loss outside the quadratic "
+ "region) have zero or undefined second derivatives at kinks, "
+ "causing degenerate Hessians."
+ )
+
+ self.device = device
+ self.hessian_method = hessian_method
+ self.n_hessian_probes = n_hessian_probes
+ self.network = None
+ self.optimizer = None
+ self.embedding_dim = None
+ self._network_params = None
+ self._fit = None
+ self._target = None
+ if hessian_method == "gn":
+ self._gn_hessian = GaussNewtonHessian(loss_fn, n_hessian_probes, self.dtype)
+
+ def objective_fn(self, predt: np.ndarray, data: lgb.Dataset) -> Tuple[np.ndarray, np.ndarray]:
+ """
+ Custom objective function for LightGBM training.
+
+ This function defines the gradients and hessians for the LightGBM model
+ based on the PyTorch loss function. It converts the raw LightGBM outputs
+ to embeddings, updates the MLP, computes the loss through the updated
+ network, and then backpropagates to get gradients.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM, representing the GBDT embeddings.
+ data : lgb.Dataset
+ LightGBM dataset containing the target values.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians for LightGBM optimization.
+ """
+ self._iter_count += 1
+
+ target = torch.tensor(
+ data.get_label().reshape(self.k, -1), dtype=self.dtype, device=self.device
+ )
+ embeds, loss = self.get_embeds_loss_separate(predt, target, self._Z_train)
+ grad, hess = self.calculate_gradients_and_hessians(loss, embeds)
+
+ return grad, hess
+
+ def get_embeds_loss_separate(
+ self,
+ predt: np.ndarray,
+ target: torch.Tensor,
+ Z: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
+ """
+ Transform LightGBM outputs into embeddings and calculate loss for separate gradients (Option 2).
+
+ This function:
+ 1. Reshapes the raw outputs into tree embeddings
+ 2. Maps embeddings to VAR coefficients via the MLP and takes one optimizer step
+ 3. Recomputes the coefficients through the updated network in inference mode
+ 4. Calculates the GBDT loss between fitted and actual values
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM.
+ target : torch.Tensor
+ Target values (actual time series values), shape ``(k, T_r)``.
+ Z : torch.Tensor
+ VAR design matrix, shape ``(T_r, k * p)`` for the full design or
+ ``(k * T_r, 2 * p)`` for the factor design.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor]
+ GBDT embeddings and loss value.
+ """
+ # Reshape outputs into embedding matrix (samples × embedding_dim)
+ # The 'F' order means Fortran-style ordering (column-major)
+ gbdt_embed = torch.tensor(
+ predt.reshape(-1, self.embedding_dim, order="F"),
+ requires_grad=True,
+ dtype=self.dtype,
+ device=self.device
+ )
+
+ # Train network (forward pass)
+ self.network.train()
+ var_params_net = self.network(gbdt_embed)
+ fit_net = self._compute_fit(var_params_net, Z)
+ network_loss = self.loss_fn(fit_net, target)
+ self.optimizer.zero_grad()
+ network_loss.backward()
+ self.optimizer.step()
+
+ # Calculate loss for GBDT
+ self.network.eval()
+ var_params_gbdt = self.network(gbdt_embed)
+ fit_gbdt = self._compute_fit(var_params_gbdt, Z)
+ gbm_loss = self.loss_fn(fit_gbdt, target)
+
+ if self.hessian_method == "gn":
+ self._fit = fit_gbdt
+ self._target = target
+
+ return gbdt_embed, gbm_loss
+
+ def _calculate_gradients_and_hessians_separate(
+ self, loss: torch.Tensor, embeds: torch.Tensor,
+ ) -> Tuple[np.ndarray, np.ndarray]:
+ """
+ Compute gradients and hessians for LightGBM optimization using separate gradients (Option 2).
+
+ This function computes first and second-order derivatives needed for
+ gradient boosting optimization in LightGBM.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ embeds : torch.Tensor
+ GBDT embeddings.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ # Compute gradients
+ grad = autograd(loss, inputs=embeds, create_graph=True)[0]
+
+ # Compute hessians. We compute the diagonal of the Hessian matrix for each parameter separately
+ hess = [
+ autograd(grad[:, i].sum(), embeds, retain_graph=True)[0][:, i:(i + 1)]
+ for i in range(self.embedding_dim)
+ ]
+
+ # Convert to numpy arrays and reshape as expected by LightGBM
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = torch.cat(hess, dim=1).cpu().detach().numpy().ravel(order="F")
+
+ return grad, hess
+
+ def _calculate_gradients_and_hessians_separate_gn(
+ self, loss: torch.Tensor, embeds: torch.Tensor,
+ ) -> Tuple[np.ndarray, np.ndarray]:
+ """Gauss-Newton Hessian for separate gradient mode via Hutchinson probing.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ embeds : torch.Tensor
+ GBDT embeddings, shape ``(n_samples, embedding_dim)``.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ grad = autograd(loss, inputs=embeds, retain_graph=True)[0]
+ rng = torch.Generator().manual_seed(self._iter_count)
+ hess = self._gn_hessian.estimate(self._fit, self._target, embeds, rng)
+ self._fit = None
+ self._target = None
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = hess.cpu().detach().numpy().ravel(order="F")
+
+ return grad, hess
+
+ def _fit_from_predt(self, predt: np.ndarray, Z: torch.Tensor) -> torch.Tensor:
+ """Compute the fitted values for raw LightGBM embedding outputs.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM (flattened Fortran-order).
+ Z : torch.Tensor
+ Design matrix for the dataset being evaluated, shape
+ ``(T_r, k * p)`` for the full design or ``(k * T_r, 2 * p)`` for
+ the factor design.
+
+ Returns
+ -------
+ torch.Tensor
+ Fitted values, shape ``(k, T_r)``.
+ """
+ gbdt_embed = torch.tensor(
+ predt.reshape(-1, self.embedding_dim, order="F"),
+ dtype=self.dtype
+ ).to(self.device)
+
+ # self.network is the live module the objective updated in this same
+ # boosting iteration (guaranteed by NoDeepcopyObjective).
+ self.network.eval()
+ with torch.no_grad():
+ var_params = self.network(gbdt_embed)
+ fit = self._compute_fit(var_params, Z)
+
+ return fit
+
+ def _num_class(self) -> int:
+ """LightGBM output dimension: the tree-embedding dimension."""
+
+ return self.embedding_dim
+
+ def _reset_training_state(self) -> None:
+ """Reset per-training state, including the network-specific parts."""
+ super()._reset_training_state()
+ self._fit = None
+ self._target = None
+ self.network = None
+ self.optimizer = None
+
+ def _post_datasets_setup(self, seed: int) -> None:
+ """Construct the MLP decoder and optimizer once the panel dimensions are known.
+
+ Called after ``_build_panel_datasets`` has set ``self.n_params``
+ (``k * p`` for type="full", ``2 * p`` for type="factor"), which is
+ the MLP output dimension.
+
+ Parameters
+ ----------
+ seed : int
+ Random seed forwarded from ``train()``.
+ """
+ network_params = self._network_params
+
+ # Seed torch before constructing the MLP so initialization (and dropout
+ # draws during training) are reproducible even when the random
+ # projection layer -- whose constructor reseeds torch -- is disabled.
+ torch.manual_seed(seed)
+
+ self.network = MLP(
+ tree_embed_dim=self.embedding_dim,
+ output_dim=self.n_params,
+ hidden_dim=network_params["hidden_dim"],
+ use_random_projection=network_params["use_random_projection"],
+ rp_embed_dim=network_params["rp_embed_dim"] if network_params["use_random_projection"] else None,
+ dropout_rate=network_params["dropout"],
+ seed=seed
+ ).to(self.device)
+ self.optimizer = torch.optim.Adam(self.network.parameters(), lr=network_params["learning_rate"])
+
+ def train(
+ self,
+ lgb_params: dict = None,
+ network_params: dict = None,
+ num_iterations: int = 100,
+ train_data: pd.DataFrame = None,
+ validation: bool = False,
+ early_stopping_round: Optional[int] = None,
+ seed: int = 123,
+ verbose: int = -1,
+ deterministic: bool = True,
+ forecast_intervals: Optional[ForecastIntervals] = None,
+ ) -> TrainingResult:
+ """
+ Train the Hyper-TreeNet-VAR model on an aligned panel of time series.
+
+ This method:
+ 1. Pivots the panel and builds the VAR design matrix
+ 2. Sets up LightGBM datasets (one row per series and time step)
+ 3. Trains the models
+
+ The training data must contain columns:
+ - 'series_id': Identifier for each time series
+ - 'date': Timestamp for each observation
+ - 'value': Target value to forecast
+ - Additional feature columns used for forecasting
+
+ All series must have the same length and identical dates (aligned
+ panel); see ``_var_base.py``.
+
+ Parameters
+ ----------
+ lgb_params : dict
+ LightGBM parameters
+ network_params : dict
+ Network parameters. Available parameters are:
+ - "learning_rate": Learning rate for the neural network optimizer
+ - "hidden_dim": Dimension of the hidden layer in the MLP
+ - "embedding_dimension": Dimension of the tree embeddings from LightGBM
+ - "use_random_projection": Whether to use random projection for embeddings
+ - "rp_embed_dim": Dimension of the random projection embeddings (if used)
+ - "dropout": Dropout rate for regularization
+ num_iterations : int
+ Number of boosting rounds for training
+ train_data : pd.DataFrame
+ Training data containing series_id, date, value and feature columns
+ validation : bool
+ If True, a validation set will be created for evaluation. It splits
+ the last fcst_h time steps of each series for validation.
+ early_stopping_round : int, optional
+ If provided, training will stop if the validation loss does not
+ improve for this many rounds.
+ seed : int
+ Random seed for reproducibility
+ verbose : int
+ Verbosity level for LightGBM training
+ deterministic : bool
+ If True, sets LightGBM's ``deterministic`` and ``force_row_wise``
+ parameters to ensure reproducible results. May slow down training.
+ See https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic
+ forecast_intervals : ForecastIntervals, optional
+ If provided, calibrate conformal prediction intervals via
+ rolling-window cross-validation after the main model is trained.
+ The collected conformity scores are then used by
+ ``forecast(..., level=[...])`` to produce ``-lo-`` /
+ ``-hi-`` columns. See
+ :class:`hypertrees.conformal.ForecastIntervals`.
+
+ Returns
+ -------
+ TrainingResult
+ Object containing evaluation results and training information.
+ """
+ if network_params is None:
+ raise ValueError("network_params must be provided.")
+ if not isinstance(network_params, dict):
+ raise TypeError("network_params must be a dictionary.")
+ required_net_keys = (
+ "learning_rate", "embedding_dimension", "hidden_dim",
+ "dropout", "use_random_projection",
+ )
+ missing_keys = [key for key in required_net_keys if key not in network_params]
+ if missing_keys:
+ raise ValueError(f"network_params is missing required keys: {missing_keys}")
+ if network_params.get("use_random_projection") and "rp_embed_dim" not in network_params:
+ raise ValueError(
+ "network_params is missing required keys: ['rp_embed_dim'] "
+ "(required when use_random_projection=True)."
+ )
+
+ self.embedding_dim = network_params["embedding_dimension"]
+ self._network_params = network_params
+
+ # Select gradient computation based on Hessian method
+ if self.hessian_method == "exact":
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_separate
+ else:
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_separate_gn
+
+ def _model_factory():
+ return HyperTreeNetVAR(
+ p=self.p,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ scaling=self.scaling,
+ type=self.type,
+ device=self.device,
+ hessian_method=self.hessian_method,
+ n_hessian_probes=self.n_hessian_probes,
+ )
+
+ cal_train_kwargs = dict(
+ lgb_params=lgb_params,
+ network_params=network_params,
+ num_iterations=num_iterations,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+
+ return self._train_core(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ train_data=train_data,
+ validation=validation,
+ early_stopping_round=early_stopping_round,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ forecast_intervals=forecast_intervals,
+ model_factory=_model_factory,
+ cal_train_kwargs=cal_train_kwargs,
+ )
+
+ def _tree_embeddings(self, features_df: pd.DataFrame) -> torch.Tensor:
+ """Compute the GBDT tree embeddings for a feature frame.
+
+ Parameters
+ ----------
+ features_df : pd.DataFrame
+ Feature frame (training feature columns only).
+
+ Returns
+ -------
+ torch.Tensor
+ Tree embeddings, shape ``(n_rows, embedding_dim)``.
+ """
+ # Predict on the DataFrame (not .values) so pandas ``category``
+ # dtype features keep their categorical encoding at forecast time.
+ gbdt_embeds = torch.tensor(
+ self.model.predict(features_df),
+ dtype=self.dtype,
+ device=self.device
+ ).reshape(-1, self.embedding_dim)
+
+ return gbdt_embeds
+
+ def _forecast_params(self, features_df: pd.DataFrame) -> np.ndarray:
+ """Forecast the ``(n_rows, n_params)`` coefficient matrix via GBDT + MLP.
+
+ Parameters
+ ----------
+ features_df : pd.DataFrame
+ Feature frame (training feature columns only).
+
+ Returns
+ -------
+ np.ndarray
+ VAR coefficients per row, shape ``(n_rows, n_params)``.
+ """
+ gbdt_embeds = self._tree_embeddings(features_df)
+
+ # self.network holds this instance's trained weights (boosting
+ # updated it in place; see NoDeepcopyObjective).
+ self.network.eval()
+ with torch.no_grad():
+ params_fcst = (self.network(gbdt_embeds)
+ .cpu()
+ .detach()
+ .numpy())
+
+ return params_fcst
+
+ def _forecast_tree_embeddings(self, test_data: pd.DataFrame, model_name: str) -> pd.DataFrame:
+ """Build the ``type="tree_embeddings"`` output DataFrame.
+
+ Parameters
+ ----------
+ test_data : pd.DataFrame
+ Validated forecast input (features already injected).
+ model_name : str
+ Model name identifier for the ``model`` column.
+
+ Returns
+ -------
+ pd.DataFrame
+ DataFrame with one ``tree_embedding_{i}`` column per embedding
+ dimension.
+ """
+ gbdt_embeds = self._tree_embeddings(test_data[self.features])
+
+ out_df = pd.DataFrame({
+ "series_id": test_data["series_id"].to_numpy().flatten(),
+ "date": test_data["date"].to_numpy().flatten(),
+ "model": model_name,
+ })
+ # Add tree embeddings to the dataframe
+ for i in range(self.embedding_dim):
+ out_df[f"tree_embedding_{i + 1}"] = gbdt_embeds[:, i].cpu().numpy().flatten()
+
+ return out_df
diff --git a/hypertrees/models/HyperTreeSTL.py b/hypertrees/models/HyperTreeSTL.py
index 2581696..bb3e78f 100644
--- a/hypertrees/models/HyperTreeSTL.py
+++ b/hypertrees/models/HyperTreeSTL.py
@@ -11,7 +11,7 @@
from ..utils import CustomLogger
lgb.register_logger(CustomLogger())
-from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order
+from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order, NoDeepcopyObjective
class HyperTreeSTL:
"""
@@ -112,7 +112,10 @@ def __init__(
Forecast horizon (number of periods to forecast ahead).
loss_fn : Callable
Loss function for optimization. Must be a PyTorch loss function.
- Default is MSE loss, but can be changed for different error metrics.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
type : str
Type of model variant to use. Currently, "default" and "paper" are supported:
- "paper" uses the original method from the paper
@@ -127,6 +130,19 @@ def __init__(
raise ValueError("Forecast horizon 'fcst_h' must be a positive integer.")
if not isinstance(loss_fn, nn.Module):
raise TypeError("loss_fn must be a PyTorch loss function.")
+ if isinstance(loss_fn, nn.L1Loss):
+ raise ValueError(
+ "nn.L1Loss is not supported: its second derivative is zero almost "
+ "everywhere, so LightGBM's Newton boosting receives all-zero Hessians "
+ "and cannot grow trees. Use nn.HuberLoss or nn.SmoothL1Loss for an "
+ "MAE-like loss with usable curvature."
+ )
+ if getattr(loss_fn, "reduction", "mean") == "none":
+ raise ValueError(
+ "loss_fn must use a scalar reduction ('mean' or 'sum'); "
+ "reduction='none' returns per-element losses that the "
+ "boosting objective cannot consume."
+ )
if not isinstance(freq, str):
raise TypeError("freq must be a string representing the frequency of the time series.")
if type not in ["default", "paper"]:
@@ -156,6 +172,7 @@ def __init__(
self.features = None # Stores feature names after training
self.is_trained = False # Flag to track if model has been trained
self.dataset_references = {} # Store references to LightGBM datasets
+ self._seasonal_offset = None # Training-window seasonal centering (set in train)
def objective_fn(
self,
@@ -350,7 +367,8 @@ def _calculate_gradients_and_hessians(
def _forward_paper(
self,
params: torch.Tensor,
- time_idx: torch.Tensor
+ time_idx: torch.Tensor,
+ seasonal_offset: Optional[torch.Tensor] = None,
) -> Tuple[torch.Tensor, torch.Tensor]:
"""
Forward pass to compute the trend and seasonality from STL parameters.
@@ -362,6 +380,12 @@ def _forward_paper(
STL decomposition parameters.
time_idx : torch.Tensor
Time indices for the observations.
+ seasonal_offset : torch.Tensor, optional
+ Per-series centering constant, shape ``(n_series,)``. When None
+ (training), the seasonal component is re-centered over the given
+ window; when provided (forecasting), this stored training offset
+ is subtracted instead so the decomposition continues the trained
+ one (see ``_compute_seasonal_offset``).
Returns
-------
@@ -394,26 +418,43 @@ def _forward_paper(
dim=2
)
- # Center the seasonal component (remove mean)
- seasonality = seasonality - torch.mean(seasonality, dim=0, keepdim=True)
+ # Center the seasonal component for identifiability. During training
+ # (seasonal_offset=None) the mean over the given window is removed;
+ # at forecast time the stored training offset is subtracted instead,
+ # so the decomposition continues the trained one rather than
+ # re-centering over the (typically partial-cycle) forecast window.
+ if seasonal_offset is None:
+ seasonality = seasonality - torch.mean(seasonality, dim=0, keepdim=True)
+ else:
+ seasonality = seasonality - seasonal_offset
return trend, seasonality
def _forward_default(
self,
params: torch.Tensor,
- time_idx: torch.Tensor
+ time_idx: torch.Tensor,
+ seasonal_offset: Optional[torch.Tensor] = None,
) -> Tuple[torch.Tensor, torch.Tensor]:
"""
Forward pass to calculate the trend and seasonality from STL parameters.
This implementation includes an updated trend smoothing method and more efficient seasonality calculation.
+ The trend smoothing window (params[:, :, 2]) is learnable: it enters
+ through a differentiable soft-boxcar kernel so gradients reach the GBDT.
+
Parameters
----------
params : torch.Tensor
STL decomposition parameters.
time_idx : torch.Tensor
Time indices for the observations.
+ seasonal_offset : torch.Tensor, optional
+ Per-series centering constant, shape ``(n_series,)``. When None
+ (training), the seasonal component is re-centered per cycle over
+ the given window; when provided (forecasting), this stored
+ training offset is subtracted instead so the decomposition
+ continues the trained one (see ``_compute_seasonal_offset``).
Returns
-------
@@ -429,21 +470,37 @@ def _forward_default(
slope = params[:, :, 1]
trend_raw = intercept + slope * time_idx
- # Map logit -> odd window size W in [3, max_w]
- max_w = min(2 * m + 1, 101)
+ # Map logit -> effective window width in [3, max_w_model] per series.
+ # The width enters through a *soft* boxcar kernel so gradients flow back
+ # to the window parameter; a hard int(median(...).item()) cut would
+ # sever the autograd graph, giving zero gradient AND zero Hessian, and
+ # LightGBM would grow zero-valued trees for it (the window would stay
+ # frozen at its sigmoid(0) midpoint forever).
+ max_w_model = min(2 * m + 1, 101)
w_logit = params[:, :, 2]
- w_float = (max_w - 3) * torch.sigmoid(w_logit) + 3.0
- W = int(torch.median(torch.round(w_float)).clamp(3, max_w).item())
- if W % 2 == 0:
- W += 1
-
- # Grouped conv expects channels divisible by groups.
- # Put series in the *channel* dimension: input (1, N, T), weight (N, 1, W), groups=N.
- k = torch.ones((N, 1, W), dtype=dtype) / W # (N,1,W)
- xin = trend_raw.T.contiguous().unsqueeze(0) # (1,N,T)
- pad = W // 2
- xpad = torch.nn.functional.pad(xin, (pad, pad), mode="reflect") # (1,N,T+2p)
- trend = torch.nn.functional.conv1d(xpad, k, groups=N).squeeze(0).T # (T,N)
+ w_eff = (max_w_model - 3.0) * torch.sigmoid(w_logit.mean(dim=0)) + 3.0 # (N,)
+
+ # Kernel support: reflect padding requires pad <= T - 1, so cap the
+ # support at 2T - 1 (short forecast horizons used to crash here when
+ # W // 2 exceeded T - 1). Both arguments are odd, so K stays odd.
+ K = min(max_w_model, 2 * T - 1)
+
+ if K >= 3:
+ w_eff = torch.clamp(w_eff, max=float(K))
+ half = K // 2
+ offsets = torch.arange(-half, half + 1, dtype=dtype).abs().view(1, -1) # (1,K)
+ # Soft boxcar: weight ~ 1 inside +-w_eff/2, smoothly decaying outside.
+ k = torch.sigmoid(w_eff.view(-1, 1) / 2.0 - offsets) # (N,K)
+ k = (k / k.sum(dim=1, keepdim=True)).unsqueeze(1) # (N,1,K)
+
+ # Grouped conv expects channels divisible by groups.
+ # Put series in the *channel* dimension: input (1, N, T), weight (N, 1, K), groups=N.
+ xin = trend_raw.T.contiguous().unsqueeze(0) # (1,N,T)
+ xpad = torch.nn.functional.pad(xin, (half, half), mode="reflect") # (1,N,T+2*half)
+ trend = torch.nn.functional.conv1d(xpad, k, groups=N).squeeze(0).T # (T,N)
+ else:
+ # Series too short to smooth (T == 1); keep the raw linear trend.
+ trend = trend_raw
# Seasonality: Fourier with per-cycle zero-mean centering
H = (self.n_params - 3) // 2
@@ -459,11 +516,22 @@ def _forward_default(
angle = time_idx.unsqueeze(-1) * k_h * (2.0 * torch.pi / m) # (T,N,H)
seasonality = (wsin * torch.sin(angle) + wcos * torch.cos(angle)).sum(dim=-1) # (T,N)
+ # At forecast time, continue the training decomposition by
+ # subtracting the stored training offset instead of re-centering
+ # over the forecast window (see _compute_seasonal_offset).
+ if seasonal_offset is not None:
+ return trend, seasonality - seasonal_offset
+
# Per-cycle centering (sum over a cycle ≈ 0)
C = (T + m - 1) // m
pad_T = C * m - T
if pad_T > 0:
- tail = torch.flip(seasonality[-min(T, pad_T):, :], dims=[0])
+ # Extend by reflection. When the series is shorter than the padding
+ # (T < pad_T, i.e. less than half a seasonal cycle observed), keep
+ # ping-ponging the reflection until a full cycle can be assembled.
+ tail = torch.flip(seasonality, dims=[0])
+ while tail.shape[0] < pad_T:
+ tail = torch.cat([tail, torch.flip(tail, dims=[0])], dim=0)
S_ext = torch.cat([seasonality, tail[:pad_T]], dim=0) # (C*m,N)
else:
S_ext = seasonality
@@ -474,6 +542,49 @@ def _forward_default(
return trend, seasonality
+ def _compute_seasonal_offset(self, full_ts: pd.DataFrame) -> torch.Tensor:
+ """Seasonal centering offset implied by the training-window fit.
+
+ The forward passes enforce the seasonal identifiability constraint by
+ re-centering over whatever window they are given. Re-centering over a
+ (typically partial-cycle) forecast window would subtract a different
+ constant than the training fit removed, leaking a phase-dependent
+ level offset between trend and seasonality across the train/test
+ boundary. This computes the constant the training fit removed -- the
+ mean raw seasonal value over the training window ("paper" variant)
+ or over the last full cycle ("default" variant, whose training
+ centering is per cycle) -- so ``forecast`` subtracts the same one.
+
+ Parameters
+ ----------
+ full_ts : pd.DataFrame
+ Preprocessed training data (features and ``time`` column).
+
+ Returns
+ -------
+ torch.Tensor
+ Per-series centering offset, shape ``(n_series,)``.
+ """
+ params = torch.tensor(
+ self.model.predict(
+ full_ts[self.features]
+ ).reshape(-1, self.n_series, self.n_params, order="F"),
+ dtype=self.dtype,
+ )
+ time_idx = torch.tensor(
+ full_ts["time"].to_numpy().reshape(-1, self.n_series), dtype=self.dtype
+ )
+ # A zero offset returns the *uncentered* seasonal component.
+ zero = torch.zeros(self.n_series, dtype=self.dtype)
+ _, seasonality_raw = self._forward(params, time_idx, seasonal_offset=zero)
+
+ if self.forward_type == "paper" or seasonality_raw.shape[0] < self.period:
+ window = seasonality_raw
+ else:
+ window = seasonality_raw[-self.period:]
+
+ return window.mean(dim=0)
+
def train(
self,
lgb_params: dict = None,
@@ -557,6 +668,7 @@ def train(
self.dataset_references = {}
self.is_trained = False
self.features = None
+ self._seasonal_offset = None
if deterministic:
lgb_params = {**lgb_params, "deterministic": True, "force_row_wise": True}
@@ -576,10 +688,11 @@ def train(
raise NotImplementedError(f"You have provided {self.n_series} series. Currently, HyperTreeSTL only supports univariate training (1 series at a time). Please train separate models for each series.")
self.train_series_id = train_data['series_id'].unique()[0]
- # General model parameters
+ # General model parameters. The objective wrapper stops lgb.train's
+ # params deepcopy from cloning this instance (see NoDeepcopyObjective).
self.lgb_params = {
"num_class": self.n_params,
- "objective": self.objective_fn,
+ "objective": NoDeepcopyObjective(self.objective_fn),
"metric": "None",
"random_seed": seed,
"verbose": verbose
@@ -654,6 +767,11 @@ def train(
)
training_time = time.time() - start_time
+ # Anchor the seasonal identifiability constraint to the training
+ # window so forecasts continue the trained decomposition (see
+ # _compute_seasonal_offset).
+ self._seasonal_offset = self._compute_seasonal_offset(full_ts)
+
# Set trained flag to True
self.is_trained = True
@@ -661,7 +779,7 @@ def train(
result = TrainingResult(
train_metrics=evals_result["train"] if validation else {"loss": []},
validation_metrics=evals_result["validation"] if validation else None,
- best_iteration=self.model.best_iteration-1 if hasattr(self.model, 'best_iteration') else num_iterations,
+ best_iteration=self.model.best_iteration if self.model.best_iteration > 0 else num_iterations,
training_time=training_time
)
@@ -669,7 +787,7 @@ def train(
except Exception as e:
self.is_trained = False
- raise RuntimeError(f"Training failed: {str(e)}")
+ raise RuntimeError(f"Training failed: {str(e)}") from e
def forecast(
self,
@@ -762,8 +880,10 @@ def forecast(
time_idx = torch.tensor(test_data["time"].to_numpy().reshape(-1, n_series_test), dtype=self.dtype)
- # Forward pass to calculate trend and seasonal components
- trend, seasonality = self._forward(params_fcst, time_idx)
+ # Forward pass to calculate trend and seasonal components; the
+ # stored training offset continues the trained decomposition
+ # instead of re-centering over the forecast window.
+ trend, seasonality = self._forward(params_fcst, time_idx, self._seasonal_offset)
# Combine components to get forecasted values
fcsts_stl = trend + seasonality
@@ -774,7 +894,7 @@ def forecast(
"series_id": test_data["series_id"].to_numpy().flatten(),
"date": test_data["date"].to_numpy().flatten(),
"fcst": fcsts_stl.detach().numpy().flatten(),
- "model": f"Hyper-Tree-STL(period={self.period})",
+ "model": f"Hyper-Tree-STL({self.period})",
})
elif type == "components":
out_df = pd.DataFrame({
@@ -782,13 +902,13 @@ def forecast(
"date": test_data["date"].to_numpy().flatten(),
"trend": trend.detach().numpy().flatten(),
"seasonality": seasonality.detach().numpy().flatten(),
- "model": f"Hyper-Tree-STL(period={self.period})",
+ "model": f"Hyper-Tree-STL({self.period})",
})
elif type == "parameters":
out_df = pd.DataFrame({
"series_id": test_data["series_id"].to_numpy().flatten(),
"date": test_data["date"].to_numpy().flatten(),
- "model": f"Hyper-Tree-STL(period={self.period})",
+ "model": f"Hyper-Tree-STL({self.period})",
})
out_df["trend_intercept"] = params_fcst[:,:, 0].detach().numpy().flatten()
out_df["trend_slope"] = params_fcst[:,:, 1].detach().numpy().flatten()
@@ -803,4 +923,4 @@ def forecast(
return out_df
except Exception as e:
- raise RuntimeError(f"Forecasting not successful: {str(e)}")
+ raise RuntimeError(f"Forecasting not successful: {str(e)}") from e
diff --git a/hypertrees/models/HyperTreeTSB.py b/hypertrees/models/HyperTreeTSB.py
new file mode 100644
index 0000000..bd1f6ac
--- /dev/null
+++ b/hypertrees/models/HyperTreeTSB.py
@@ -0,0 +1,1081 @@
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+from torch.autograd import grad as autograd
+import lightgbm as lgb
+from typing import Tuple, List, Callable, Optional
+import time
+import random
+import warnings
+from ..utils import CustomLogger
+lgb.register_logger(CustomLogger())
+
+from ..utils import TimeSeriesPreprocessor, prepare_datasets, TrainingResult, validate_series_order, GaussNewtonHessian, NoDeepcopyObjective
+from ..conformal import (
+ ForecastIntervals,
+ validate_calibration_length,
+ rolling_origin_residuals,
+ interval_columns,
+)
+
+class HyperTreeTSB:
+ """
+ Class that implements a Hyper-Tree-TSB model for intermittent demand forecasting.
+
+ The Teunter-Syntetos-Babai (TSB) method forecasts intermittent demand as the
+ product of two exponentially smoothed components: the demand *probability*
+ ``p_t`` (smoothed occurrence indicator, updated every period) and the demand
+ *size* ``z_t`` (smoothed over nonzero demands only). The Hyper-Tree variant
+ makes both smoothing parameters time-varying functions of features, so the
+ responsiveness of probability and size estimates can adapt to e.g.
+ promotions, listings, or seasonality. The recursion follows the reference
+ implementation in Nixtla's statsforecast:
+
+ - Occurrence: d_t = 1 if y_t != 0 else 0
+ - Probability: p_t = p_{t-1} + alpha_p,t * (d_t - p_{t-1})
+ - Size: z_t = z_{t-1} + alpha_d,t * (y_t - z_{t-1}) if d_t = 1 else z_{t-1}
+ - Fitted: y_hat_t = p_{t-1} * z_{t-1}
+ - Forecast: y_hat_{T+h} = p_T * z_T (flat over the horizon, as in the
+ classical method: future demand occurrence is unobserved and the expected
+ states propagate unchanged)
+
+ States are initialized as in statsforecast: ``p_0`` with the first
+ occurrence indicator and ``z_0`` with the first nonzero demand (0 for
+ all-zero series, which therefore forecast 0). For cross-series learning,
+ all series must have the same length; datasets with varying lengths should
+ be padded and carry a ``mask`` column (1 = valid observation, 0 = padding).
+
+ Key features:
+ - Designed for intermittent (sporadic, zero-inflated) demand
+ - Combines tree-based models (LightGBM) with the TSB method
+ - Allows the smoothing parameters to vary based on features
+ - Handles obsolescence: the probability estimate decays during zero-demand periods
+
+ Use this model when:
+ - Series contain frequent zeros (intermittent demand), where AR and ETS
+ target models are structurally misspecified
+ - You have features that signal when demand probability or size shifts
+
+ References
+ ----------
+ [1] Teunter, R. H., Syntetos, A. A., & Babai, M. Z. (2011). Intermittent
+ demand: Linking forecasting to inventory obsolescence. European
+ Journal of Operational Research, 214(3), 606-615.
+ [2] Recursion and state-initialization conventions follow the TSB
+ implementation in Nixtla's statsforecast (Apache-2.0):
+ https://github.com/Nixtla/statsforecast
+
+ Example usage:
+ ```python
+ # Imports
+ from hypertrees.models.HyperTreeTSB import HyperTreeTSB
+ import numpy as np
+ import pandas as pd
+
+ # Initialize model
+ frequency = 'W'
+ fcst_h = 8
+ model = HyperTreeTSB(freq=frequency, fcst_h=fcst_h)
+
+ # Data: intermittent demand with columns 'date', 'series_id', 'value'.
+ # All other columns are automatically treated as features.
+ rng = np.random.RandomState(1)
+ dates = pd.date_range("2022-01-03", periods=104 + fcst_h, freq="W-MON")
+ demand = rng.binomial(1, 0.3, len(dates)) * rng.poisson(5, len(dates))
+ df = pd.DataFrame({
+ "series_id": "sku_1",
+ "date": dates,
+ "value": demand.astype(float),
+ "month": dates.month,
+ })
+ test = df.tail(fcst_h)
+ train = df.drop(test.index)
+
+ # Train model
+ model.train(
+ lgb_params={'learning_rate': 0.1},
+ num_iterations=100,
+ train_data=train
+ )
+
+ # Generate forecasts and inspect the time-varying smoothing parameters
+ forecasts = model.forecast(test_data=test)
+ parameters = model.forecast(test_data=test, type="parameters")
+ ```
+ """
+
+ def __init__(
+ self,
+ freq: str = "M",
+ fcst_h: int = 12,
+ loss_fn: Callable = nn.MSELoss(),
+ n_hessian_probes: int = 5,
+ ):
+ """
+ Initialize the Hyper-Tree-TSB model.
+
+ Arguments
+ ----------
+ freq : str
+ Frequency of the time series (e.g., 'D' for daily, 'W' for weekly,
+ 'M' for monthly).
+ fcst_h : int
+ Forecast horizon (number of periods to forecast ahead).
+ loss_fn : Callable
+ Loss function for optimization. Must be a PyTorch loss function.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
+ n_hessian_probes : int
+ Number of Hutchinson probes for Gauss-Newton Hessian diagonal estimation.
+ More probes reduce variance but increase computation. Default is 5.
+ """
+ # Validate inputs
+ if fcst_h <= 0:
+ raise ValueError("Forecast horizon 'fcst_h' must be a positive integer.")
+ if not isinstance(loss_fn, nn.Module):
+ raise TypeError("loss_fn must be a PyTorch loss function.")
+ if isinstance(loss_fn, nn.L1Loss):
+ raise ValueError(
+ "nn.L1Loss is not supported: its second derivative is zero almost "
+ "everywhere, so LightGBM's Newton boosting receives all-zero Hessians "
+ "and cannot grow trees. Use nn.HuberLoss or nn.SmoothL1Loss for an "
+ "MAE-like loss with usable curvature."
+ )
+ if getattr(loss_fn, "reduction", "mean") == "none":
+ raise ValueError(
+ "loss_fn must use a scalar reduction ('mean' or 'sum'); "
+ "reduction='none' returns per-element losses that the "
+ "boosting objective cannot consume."
+ )
+ if not isinstance(loss_fn, nn.MSELoss):
+ warnings.warn(
+ f"Loss {type(loss_fn).__name__} is not nn.MSELoss. The Gauss-Newton "
+ "Hessian requires a twice-differentiable loss; non-smooth losses "
+ "(e.g., L1Loss, quantile loss, HuberLoss/SmoothL1Loss outside the quadratic "
+ "region) have zero or undefined second derivatives at kinks, "
+ "causing degenerate Hessians."
+ )
+ if not isinstance(freq, str):
+ raise TypeError("freq must be a string representing the frequency of the time series.")
+
+ self.freq = freq
+ self.n_params = 2 # alpha_p (probability), alpha_d (demand size)
+ self.fcst_h = fcst_h
+ self.loss_fn = loss_fn
+ self.loss_name = self.loss_fn.__class__.__name__
+ self.dtype = torch.float32
+ self.model = None
+ self.features = None # Stores feature names after training
+ self.is_trained = False # Flag to track if model has been trained
+ self.dataset_references = {} # Store references to LightGBM datasets
+ self.eps = 1e-6 # Small constant to prevent numerical issues in sigmoid
+ self.fcst_states = None # Store final TSB states for forecasting
+ self.n_hessian_probes = n_hessian_probes
+ self._iter_count = 0 # Iteration counter for seeding Hessian probes
+ # Recursive h-step validation metric: the terminal (p, z) states from
+ # the "train" eval call are stashed in _eval_boundary and consumed by
+ # the "validation" eval call of the same boosting iteration
+ # (valid_sets order is [train, validation]); see eval_fn.
+ self._last_states = None
+ self._eval_boundary = None
+
+ # Shared Gauss-Newton Hessian estimator
+ self._gn_hessian = GaussNewtonHessian(loss_fn, n_hessian_probes, self.dtype)
+
+ # Conformal prediction interval state
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
+
+ # Activation function for parameter bounds
+ self.sigmoid_fn = nn.Sigmoid()
+
+ def _create_mask_from_data(self, data: pd.DataFrame) -> torch.Tensor:
+ """
+ Create a mask for valid observations from the data.
+
+ Parameters
+ ----------
+ data : pd.DataFrame
+ DataFrame containing the time series data
+
+ Returns
+ -------
+ torch.Tensor
+ Mask indicating valid observations (1 = valid, 0 = padding).
+ """
+ if 'mask' in data.columns:
+ mask = torch.tensor(
+ data['mask'].values.reshape(self.n_series, -1),
+ dtype=self.dtype
+ )
+ else:
+ data_shape = data.shape[0]
+ mask = torch.ones((data_shape, 1), dtype=self.dtype).reshape(self.n_series, -1)
+
+ return mask
+
+ def _init_states(
+ self,
+ target: torch.Tensor,
+ mask: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
+ """Initial probability and demand-size states.
+
+ Follows statsforecast's TSB: the probability state starts at the first
+ (valid) occurrence indicator and the size state at the first (valid)
+ nonzero demand. All-zero series get a size state of 0 and therefore
+ forecast 0. The initialization depends only on the data, never on the
+ boosted parameters.
+
+ Parameters
+ ----------
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding),
+ shape ``(n_series, T)``.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor]
+ Initial probability ``p_0`` and size ``z_0``, each ``(n_series,)``.
+ """
+ occurrence = ((target != 0) & (mask > 0)).to(self.dtype)
+
+ # p_0: occurrence indicator at the first valid observation
+ first_valid = torch.argmax((mask > 0).to(torch.long), dim=1)
+ p0 = occurrence.gather(1, first_valid.unsqueeze(1)).squeeze(1)
+
+ # z_0: first valid nonzero demand (0 if the series is all zeros)
+ has_demand = occurrence.any(dim=1)
+ first_demand = torch.argmax(occurrence.to(torch.long), dim=1)
+ z0 = target.gather(1, first_demand.unsqueeze(1)).squeeze(1)
+ z0 = torch.where(has_demand, z0, torch.zeros_like(z0))
+
+ return p0, z0
+
+ def objective_fn(self, predt: np.ndarray, data: lgb.Dataset) -> Tuple[np.ndarray, np.ndarray]:
+ """
+ Custom objective function for LightGBM training.
+
+ This function defines the gradients and hessians for the LightGBM model
+ based on the PyTorch loss function. It converts LightGBM outputs to
+ PyTorch tensors, computes the TSB forward pass, and then backpropagates
+ to get gradients.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Outputs from LightGBM, representing the TSB smoothing parameters.
+ data : lgb.Dataset
+ LightGBM dataset containing the target values.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians for LightGBM optimization.
+ """
+ self._iter_count += 1
+
+ target = torch.tensor(
+ data.get_label().reshape(self.n_series, -1),
+ dtype=self.dtype
+ )
+
+ params, loss = self.get_params_loss(predt, target, data, requires_grad=True)
+ grad, hess = self.calculate_gradients_and_hessians(loss, params)
+
+ return grad, hess
+
+ def eval_fn(self, predt: np.ndarray, eval_data: lgb.Dataset) -> Tuple[str, float, bool]:
+ """
+ Custom evaluation function for evaluating forecast accuracy on an evaluation dataset.
+
+ This function computes the loss value to be monitored during evaluation.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Outputs of LightGBM Model.
+ eval_data : lgb.Dataset
+ LightGBM dataset containing the evaluation data.
+
+ Returns
+ -------
+ Tuple[str, float, bool]
+ Name of the metric, value of the metric, and whether to maximize it.
+
+ Notes
+ -----
+ The validation metric is the **recursive h-step forecast** loss: the
+ classical flat TSB forecast ``p_T * z_T`` (from the training-window
+ terminal states) scored against the holdout. The naive in-sample
+ validation loss is degenerate -- the TSB fitted value at every
+ validation row collapses to a parameter-independent fixed point, so it
+ is ~0 for *any* parameters and early stopping selects noise. The
+ training metric remains the in-sample one-step fit.
+ """
+ is_higher_better = False # Lower loss is better, so we don't maximize
+ dataset_name = self.dataset_references.get(id(eval_data), "unknown")
+ target = torch.tensor(
+ eval_data.get_label().reshape(self.n_series, -1),
+ dtype=self.dtype
+ )
+
+ if dataset_name == "validation" and self._eval_boundary is not None:
+ # Recursive h-step forecast metric. The terminal states were stashed
+ # during this same iteration's "train" eval call, so they come from
+ # the identical (post-update) model state.
+ loss = self._recursive_eval_loss(eval_data, target)
+ loss_val = loss.item()
+ if not np.isfinite(loss_val):
+ # A diverged state early in boosting would otherwise feed NaN to
+ # early stopping; report a large finite value (worst) instead.
+ loss_val = float(np.finfo(np.float32).max)
+ return self.loss_name, loss_val, is_higher_better
+
+ # Train metric (and validation fallback before the first boundary is
+ # stashed): the teacher-forced in-sample one-step loss.
+ _, loss = self.get_params_loss(predt, target, eval_data)
+ if dataset_name == "train":
+ # Stash terminal states for the validation rollout that follows in
+ # this same iteration.
+ self._eval_boundary = self._last_states
+
+ return self.loss_name, loss.item(), is_higher_better
+
+ def _recursive_eval_loss(
+ self,
+ eval_data: lgb.Dataset,
+ target: torch.Tensor,
+ ) -> torch.Tensor:
+ """Recursive h-step forecast loss for the validation split.
+
+ Mirrors deployment: the flat TSB forecast ``p_T * z_T`` (via the shared
+ :meth:`_roll_forecast` helper), starting from the training-window
+ terminal states stored in ``self._eval_boundary``, is scored against
+ the holdout. The forecast is independent of the horizon parameters, as
+ in the classical method. Padded holdout rows (mask == 0) are excluded.
+
+ Parameters
+ ----------
+ eval_data : lgb.Dataset
+ Validation dataset (provides the mask).
+ target : torch.Tensor
+ Holdout observations, shape ``(n_series, fcst_h)``.
+
+ Returns
+ -------
+ torch.Tensor
+ Scalar loss between the flat forecast and the holdout.
+ """
+ last_p, last_z = self._eval_boundary
+ point = self._roll_forecast(last_p, last_z, target.shape[1])
+
+ if "mask" in eval_data.data.columns:
+ mask = torch.tensor(
+ eval_data.data["mask"].values.reshape(self.n_series, -1),
+ dtype=self.dtype,
+ )
+ else:
+ mask = torch.ones_like(target)
+
+ return self.loss_fn(point * mask, target * mask)
+
+ def get_params_loss(
+ self,
+ predt: np.ndarray,
+ target: torch.Tensor,
+ data: lgb.Dataset = None,
+ requires_grad: bool = False
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
+ """
+ Transform LightGBM outputs into TSB parameters and calculate loss.
+
+ This function:
+ 1. Reshapes the raw outputs into TSB parameters
+ 2. Applies sigmoid transformation to ensure parameter bounds
+ 3. Runs the TSB forward pass to compute fitted values
+ 4. Calculates the loss between fitted values and actual values
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Outputs of LightGBM Model.
+ target : torch.Tensor
+ Target values (actual time series values).
+ data : lgb.Dataset
+ LightGBM dataset containing additional information.
+ requires_grad : bool
+ Whether to compute gradients (True during training).
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor]
+ Parameters tensor and loss value.
+ """
+ # Gradients must be w.r.t. raw (pre-sigmoid) outputs
+ # differentiating w.r.t. post-sigmoid params would miss the sigmoid(predt) factor
+ predt = nn.Parameter(
+ torch.tensor(
+ predt.reshape(-1, self.n_params, order="F"),
+ dtype=self.dtype
+ ),
+ requires_grad=requires_grad
+ )
+
+ # Apply sigmoid transformation and reshape for TSB computation; clamp to avoid numerical issues
+ params = torch.clamp(
+ self.sigmoid_fn(predt.reshape(self.n_series, -1, self.n_params)),
+ min=self.eps,
+ max=1-self.eps
+ )
+
+ # Get mask
+ if "mask" in data.data.columns:
+ mask = torch.tensor(
+ data.data["mask"].values.reshape(self.n_series, -1),
+ dtype=self.dtype
+ )
+ else:
+ series_len = target.shape[1]
+ mask = torch.ones((self.n_series, series_len), dtype=self.dtype)
+
+ # Forward pass to compute fitted values. Keep the terminal probability/
+ # size states so the recursive validation metric can roll the flat
+ # deployment forecast from the training-window boundary (see eval_fn).
+ last_p, last_z, fit = self.forward(params, target, mask)
+ self._last_states = (last_p, last_z)
+
+ # Stack fitted values and compute loss with masking
+ fit = torch.stack(fit, dim=1)
+ loss = self.loss_fn(fit * mask, target * mask)
+
+ # Store for Gauss-Newton Hessian estimation
+ self._fit = fit
+ self._mask = mask
+ self._target = target
+
+ return predt, loss
+
+ def forward(
+ self,
+ params: torch.Tensor,
+ target: torch.Tensor,
+ mask: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor, List[torch.Tensor]]:
+ """
+ Forward pass for the TSB recursion.
+
+ This implements the TSB updates:
+ - Probability: p_t = p_{t-1} + α_p(d_t - p_{t-1}), updated every period
+ - Size: z_t = z_{t-1} + α_d(y_t - z_{t-1}) if demand occurs, else unchanged
+ - Fitted: ŷ_t = p_{t-1} * z_{t-1}
+
+ The occurrence indicator ``d_t`` is data (constant w.r.t. the
+ parameters), so the size-update gating does not break differentiability.
+
+ Parameters
+ ----------
+ params : torch.Tensor
+ Sigmoid-transformed TSB parameters, shape ``(n_series, T, n_params)``.
+ target : torch.Tensor
+ Observations, shape ``(n_series, T)``.
+ mask : torch.Tensor
+ Validity mask (1 = real observation, 0 = padding), shape ``(n_series, T)``.
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor, List[torch.Tensor]]
+ Final probability state (n_series,), final size state (n_series,),
+ and list of fitted values.
+ """
+ series_len = target.shape[1]
+
+ # Unpack and pre-unbind parameters
+ alpha_p, alpha_d = params.unbind(dim=2)
+ alpha_p_t = alpha_p.unbind(dim=1)
+ alpha_d_t = alpha_d.unbind(dim=1)
+
+ # Occurrence indicators (data, constant w.r.t. the parameters)
+ occurrence = ((target != 0) & (mask > 0)).to(self.dtype)
+
+ # Initialize states from the data
+ p_prev, z_prev = self._init_states(target, mask)
+ fits = [target[:, 0]]
+
+ # Pre-unbind the data tensors so the loop indexes Python tuples
+ # instead of slicing tensors at every step.
+ target_t = target.unbind(dim=1)
+ mask_t = mask.unbind(dim=1)
+ occurrence_t = occurrence.unbind(dim=1)
+
+ # TSB updates with masking for padded values.
+ for t in range(1, series_len):
+ valid_mask = mask_t[t]
+ invalid_mask = 1 - valid_mask
+ y_t = target_t[t]
+ d_t = occurrence_t[t]
+
+ fit_t = valid_mask * (p_prev * z_prev) + invalid_mask * fits[-1]
+
+ p_new = valid_mask * (
+ p_prev + alpha_p_t[t] * (d_t - p_prev)
+ ) + invalid_mask * p_prev
+
+ z_new = valid_mask * (
+ d_t * (z_prev + alpha_d_t[t] * (y_t - z_prev)) +
+ (1 - d_t) * z_prev
+ ) + invalid_mask * z_prev
+
+ fits.append(fit_t)
+ p_prev = p_new
+ z_prev = z_new
+
+ return p_prev, z_prev, fits
+
+ def calculate_gradients_and_hessians(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[
+ np.ndarray, np.ndarray]:
+ """
+ Compute gradients and Generalized Gauss-Newton Hessians for LightGBM.
+
+ Uses exact first-order gradients and the Generalized Gauss-Newton (GGN)
+ approximation for the Hessian diagonal, estimated via Hutchinson probing.
+ As for ``HyperTreeETS``, the TSB recurrence makes exact second
+ derivatives propagate through the full recursion, so the
+ residual-curvature term is dropped, retaining only H_GN = J^T B J,
+ which guarantees positive semi-definite Hessians.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ params : torch.Tensor
+ Model parameters (pre-sigmoid LightGBM outputs, nn.Parameter).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ grad = autograd(loss, params, retain_graph=True)[0]
+
+ fit_masked = self._fit * self._mask
+ target_masked = self._target * self._mask
+ rng = torch.Generator().manual_seed(self._iter_count)
+ hess = self._gn_hessian.estimate(fit_masked, target_masked, params, rng)
+
+ # Release graph references to prevent accumulation between iterations
+ self._fit = None
+ self._mask = None
+ self._target = None
+
+ # Convert to numpy arrays and reshape as expected by LightGBM
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = hess.cpu().detach().numpy().ravel(order="F")
+
+ return grad, hess
+
+ def train(
+ self,
+ lgb_params: dict = None,
+ num_iterations: int = 100,
+ train_data: pd.DataFrame = None,
+ validation: bool = False,
+ early_stopping_round: Optional[int] = None,
+ seed: int = 123,
+ verbose: int = -1,
+ deterministic: bool = True,
+ forecast_intervals: Optional[ForecastIntervals] = None,
+ ) -> TrainingResult:
+ """
+ Train the Hyper-Tree-TSB model on time series data.
+
+ This method:
+ 1. Preprocesses the time series data to create features and handle variable lengths
+ 2. Sets up LightGBM datasets with proper masking
+ 3. Trains the model using gradient boosting
+ 4. Stores final TSB states for future forecasting
+
+ The training data must contain columns:
+ - 'series_id': Identifier for each time series
+ - 'date': Timestamp for each observation
+ - 'value': Target value to forecast
+ - Additional feature columns used for forecasting
+
+ Parameters
+ ----------
+ lgb_params : dict
+ LightGBM parameters like 'learning_rate', 'num_leaves', etc.
+ num_iterations : int
+ Number of boosting rounds for training
+ train_data : pd.DataFrame
+ Training data containing series_id, date, value and feature columns. All series must have the same length.
+ The data should be preprocessed to ensure that all series are of the same length and padded with 1
+ in the 'mask' column for valid observations. Padded values should have a mask value of 0.
+ validation : bool
+ If True, a validation set will be created for evaluation.
+ early_stopping_round : int, optional
+ If provided, training will stop if the validation loss does not improve for this many rounds.
+ seed : int
+ Random seed for reproducibility
+ verbose : int
+ Verbosity level for LightGBM training
+ deterministic : bool
+ If True, sets LightGBM's ``deterministic`` and ``force_row_wise`` parameters to ensure
+ reproducible results. May slow down training. See
+ https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic
+ forecast_intervals : ForecastIntervals, optional
+ If provided, calibrate conformal prediction intervals via rolling-window
+ cross-validation after the main model is trained. The collected conformity
+ scores are then used by ``forecast(..., level=[...])`` to produce
+ ``-lo-`` / ``-hi-`` columns. See
+ :class:`hypertrees.conformal.ForecastIntervals`.
+
+ Returns
+ -------
+ TrainingResult
+ Object containing evaluation results and training information.
+ """
+ # Validate inputs
+ if train_data is None:
+ raise ValueError("train_data must be provided.")
+ if lgb_params is None:
+ raise ValueError("lgb_params must be provided.")
+ if not isinstance(train_data, pd.DataFrame):
+ raise TypeError("train_data must be a pandas DataFrame.")
+ if not isinstance(lgb_params, dict):
+ raise TypeError("lgb_params must be a dictionary.")
+ if not isinstance(num_iterations, int) or num_iterations <= 0:
+ raise ValueError("num_iterations must be a positive integer.")
+ if not isinstance(seed, int):
+ raise TypeError("seed must be an integer.")
+ if not isinstance(verbose, int):
+ raise TypeError("verbose must be an integer.")
+ if early_stopping_round is not None and (
+ not isinstance(early_stopping_round, int) or early_stopping_round <= 0):
+ raise ValueError("early_stopping_round must be a positive integer.")
+ if not isinstance(validation, bool):
+ raise TypeError("validation must be a boolean.")
+ if not isinstance(deterministic, bool):
+ raise TypeError("deterministic must be a boolean.")
+ if forecast_intervals is not None and not isinstance(forecast_intervals, ForecastIntervals):
+ raise TypeError("forecast_intervals must be a ForecastIntervals instance.")
+ if early_stopping_round is not None and not validation:
+ raise ValueError("early_stopping_round can only be used when validation is True.")
+ if validation and early_stopping_round is None:
+ raise ValueError("early_stopping_round must be provided when validation is True.")
+
+ if deterministic:
+ lgb_params = {**lgb_params, "deterministic": True, "force_row_wise": True}
+
+ # Check required columns first
+ required_columns = ['series_id', 'date', 'value']
+ for col in required_columns:
+ if col not in train_data.columns:
+ raise ValueError(f"Required column '{col}' not found in training data.")
+
+ # Validate row ordering: each series must be a contiguous block with
+ # monotonic dates so the TSB reshape to (n_series, T, n_params) aligns.
+ validate_series_order(train_data, name="train_data")
+
+ if forecast_intervals is not None:
+ validate_calibration_length(
+ train_data, self.fcst_h, forecast_intervals, min_train=2,
+ )
+
+ # Check if all series in train_data have the same length
+ unique_lengths = train_data.groupby('series_id')['date'].nunique()
+ if len(unique_lengths.unique()) > 1:
+ raise ValueError("All series in train_data must have the same length. Found multiple lengths.")
+
+ # General model parameters. The objective wrapper stops lgb.train's
+ # params deepcopy from cloning this instance (see NoDeepcopyObjective).
+ self.lgb_params = {
+ "num_class": self.n_params,
+ "objective": NoDeepcopyObjective(self.objective_fn),
+ "metric": "None",
+ "random_seed": seed,
+ "verbose": verbose
+ }
+
+ # Reset states
+ self._iter_count = 0
+ self._fit = None
+ self._mask = None
+ self._target = None
+ self.model = None
+ self.dataset_references = {}
+ self.is_trained = False
+ self.fcst_states = None
+ self.features = None
+ self._last_states = None
+ self._eval_boundary = None
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
+
+ # Set random seeds for reproducibility
+ torch.manual_seed(seed)
+ np.random.seed(seed)
+ random.seed(seed)
+
+ # Copy to avoid modifying the caller's DataFrame across repeated train() calls
+ train_data = train_data.copy()
+
+ # Update with user-provided LightGBM parameters
+ self.lgb_params.update(lgb_params)
+
+ try:
+ # Initialize TimeSeriesPreprocessor for TSB-specific preprocessing
+ preprocessor = TimeSeriesPreprocessor(
+ freq=self.freq,
+ lags=[], # TSB doesn't need lag features like AR models
+ )
+
+ # Process dataset, including masking for variable-length series
+ full_ts = preprocessor.create_lags(train_data)
+ full_dict = preprocessor.extract(full_ts)
+
+ # Store feature names and dataset dimensions
+ self.features = full_dict["features"].columns.tolist()
+ self.n_series = len(train_data['series_id'].unique())
+
+ # Prepare datasets (adapted for TSB)
+ (valid_sets,
+ valid_names,
+ callbacks,
+ evals_result,
+ _, # No lags for TSB
+ _, # No lags for TSB
+ self.dataset_references) = (
+ prepare_datasets(
+ full_ts=full_ts,
+ preprocessor=preprocessor,
+ fcst_h=self.fcst_h,
+ dtype=self.dtype,
+ validation=validation,
+ early_stopping_round=early_stopping_round,
+ free_raw_data=False,
+ )
+ )
+
+ # Train LightGBM model
+ start_time = time.time()
+ self.model = lgb.train(
+ self.lgb_params,
+ valid_sets[0],
+ num_boost_round=num_iterations,
+ feval=self.eval_fn if validation else None,
+ valid_sets=valid_sets,
+ valid_names=valid_names,
+ callbacks=callbacks
+ )
+ training_time = time.time() - start_time
+
+ # Store final TSB states for forecasting
+ self._store_final_states(train_data)
+
+ # Set trained flag to True
+ self.is_trained = True
+
+ if forecast_intervals is not None:
+ def _model_factory():
+ return HyperTreeTSB(
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ n_hessian_probes=self.n_hessian_probes,
+ )
+
+ cal_train_kwargs = dict(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+ self._cs_scores, self._cs_series_order = rolling_origin_residuals(
+ model_factory=_model_factory,
+ train_data=train_data,
+ fcst_h=self.fcst_h,
+ forecast_intervals=forecast_intervals,
+ train_kwargs=cal_train_kwargs,
+ )
+ self._pi_config = forecast_intervals
+ self._is_calibrated = True
+
+ # Return results
+ result = TrainingResult(
+ train_metrics=evals_result["train"] if validation else {"loss": []},
+ validation_metrics=evals_result["validation"] if validation else None,
+ best_iteration=self.model.best_iteration if self.model.best_iteration > 0 else num_iterations,
+ training_time=training_time
+ )
+
+ return result
+
+ except Exception as e:
+ self.is_trained = False
+ raise RuntimeError(f"Training failed: {str(e)}") from e
+
+ def _store_final_states(self, train_data: pd.DataFrame):
+ """
+ Store final TSB states after training for use in forecasting.
+
+ Runs a full forward pass on the training data to obtain the final
+ probability and size states per series. Also stores the series
+ ordering to ensure consistent state access.
+
+ Parameters
+ ----------
+ train_data : pd.DataFrame
+ Training data used to compute final TSB states.
+ """
+ # Store series ordering from training data
+ self.series_order = train_data['series_id'].unique().tolist()
+
+ # Get fitted parameters for the full training period
+ params = torch.clamp(
+ self.sigmoid_fn(torch.tensor(self.model.predict(train_data[self.features]), dtype=self.dtype)),
+ min=self.eps,
+ max=1-self.eps
+ ).reshape(self.n_series, -1, self.n_params)
+
+ # Create mask and target tensors
+ train_mask = self._create_mask_from_data(train_data)
+ target = torch.tensor(
+ train_data["value"].values.reshape(self.n_series, -1),
+ dtype=self.dtype
+ )
+
+ # Forward pass to get final states
+ last_p, last_z, fit = self.forward(params, target, train_mask)
+
+ # Store final states as dictionary with series_id as keys
+ self.fcst_states = {}
+ for i, series_id in enumerate(self.series_order):
+ self.fcst_states[series_id] = {
+ 'last_p': last_p[i],
+ 'last_z': last_z[i],
+ }
+
+ def set_forecast_origin(self, history: pd.DataFrame) -> None:
+ """Re-anchor TSB states to the end of *history* without retraining.
+
+ Recomputes the terminal ``{probability, size}`` states by running the
+ full TSB forward recurrence over *history* using the already-trained
+ GBDT parameters. Used by conformal calibration with ``refit=False``.
+
+ Parameters
+ ----------
+ history : pd.DataFrame
+ DataFrame with the same columns as the training data (including
+ any ``mask`` column), ordered by ``(series_id, date)`` with each
+ series in a contiguous block and all series of equal length.
+ """
+ validate_series_order(history, name="history")
+ self._store_final_states(history)
+
+ def _roll_forecast(
+ self,
+ last_p: torch.Tensor,
+ last_z: torch.Tensor,
+ h: int,
+ ) -> torch.Tensor:
+ """Classical TSB forecast: flat ``p_T * z_T`` over ``h`` steps.
+
+ Shared by :meth:`forecast` and the recursive validation metric
+ (:meth:`_recursive_eval_loss`) so the deployed forecast and the
+ early-stopping metric cannot diverge. Future demand occurrence is
+ unobserved and the expected states propagate unchanged, so the forecast
+ is constant over the horizon and independent of the horizon parameters.
+
+ Parameters
+ ----------
+ last_p : torch.Tensor
+ Terminal probability state, shape ``(n_series,)``.
+ last_z : torch.Tensor
+ Terminal size state, shape ``(n_series,)``.
+ h : int
+ Number of horizon steps.
+
+ Returns
+ -------
+ torch.Tensor
+ Point forecasts, shape ``(n_series, h)``.
+ """
+ return (last_p * last_z).reshape(-1, 1).repeat(1, h)
+
+ def forecast(
+ self,
+ test_data: pd.DataFrame,
+ type: str = "forecast",
+ level: Optional[List[int]] = None,
+ ) -> pd.DataFrame:
+ """
+ Generate forecasts using the trained model.
+
+ Following the classical TSB method, the point forecast is flat over
+ the horizon: ŷ_{T+h} = p_T * z_T, where p_T and z_T are the final
+ probability and size states after the training period. Future demand
+ occurrence is unobserved, and the expected one-step-ahead forecast
+ propagates unchanged, so horizon features do not alter the point
+ forecasts (they do affect ``type="parameters"``).
+
+ Parameters
+ ----------
+ test_data : pd.DataFrame
+ Test data for which to generate forecasts. Must contain the same
+ feature columns used during training.
+ type : str
+ Type of forecast to generate. Options:
+ - "forecast": Generate forecasted values
+ - "parameters": Return the TSB smoothing parameters
+ level : list of int, optional
+ Confidence levels (in ``(0, 100)``, e.g. ``[80, 90]``) for conformal
+ prediction intervals. Only valid with ``type="forecast"`` and requires
+ the model to have been trained with ``forecast_intervals=...``. Adds
+ ``-lo-`` / ``-hi-`` columns to the output.
+
+ Returns
+ -------
+ pd.DataFrame
+ Forecasted data with columns:
+ - series_id: Identifier for each time series
+ - date: Forecast date/time
+ - fcst: Forecasted value (if type="forecast")
+ - model: Model name identifier
+ - alpha_p, alpha_d: TSB parameter values (if type="parameters")
+ - -lo- / -hi-: prediction interval bounds
+ (if type="forecast" and level is provided)
+ """
+ # Check if model is trained and states are stored
+ if not self.is_trained or self.model is None:
+ raise RuntimeError("Model has not been trained. Call train() before forecasting.")
+ if self.fcst_states is None or self.series_order is None:
+ raise RuntimeError("Final states not found. This should not happen after training.")
+
+ # Validate input data
+ required_cols = ['series_id', 'date']
+ for col in required_cols:
+ if col not in test_data.columns:
+ raise ValueError(f"Required column '{col}' not found in test_data")
+
+ # Validate row ordering: each series must be a contiguous block with
+ # monotonic dates so the forecast reshape aligns with stored states.
+ validate_series_order(test_data, name="test_data")
+
+ # Validate that test_data series_ids match training series_ids
+ test_series_ids = set(test_data['series_id'].unique())
+ train_series_ids = set(self.series_order)
+ if test_series_ids != train_series_ids:
+ missing_in_test = train_series_ids - test_series_ids
+ extra_in_test = test_series_ids - train_series_ids
+ error_msg = []
+ if missing_in_test:
+ error_msg.append(f"Missing series in test_data: {missing_in_test}")
+ if extra_in_test:
+ error_msg.append(f"Extra series in test_data: {extra_in_test}")
+ raise ValueError(". ".join(error_msg))
+
+ # Validate rows per series matches forecast horizon (forecast only;
+ # parameters can be requested for arbitrary-length input).
+ if type == "forecast":
+ rows_per_series = test_data.groupby("series_id", sort=False).size()
+ bad = rows_per_series[rows_per_series != self.fcst_h]
+ if not bad.empty:
+ raise ValueError(
+ f"Each series must have exactly fcst_h={self.fcst_h} rows in test_data. "
+ f"Series with wrong counts: {bad.to_dict()}"
+ )
+
+ # Validate type parameter
+ if type not in ["forecast", "parameters"]:
+ raise ValueError("Parameter 'type' must be either 'forecast' or 'parameters'")
+
+ if level is not None:
+ if type != "forecast":
+ raise ValueError("level is only supported with type='forecast'.")
+ if not self._is_calibrated:
+ raise RuntimeError(
+ "Prediction intervals were requested via level, but the model "
+ "was not calibrated. Pass forecast_intervals=ForecastIntervals(...) "
+ "to train() before forecasting with level."
+ )
+ if not isinstance(level, (list, tuple)) or len(level) == 0:
+ raise ValueError("level must be a non-empty list of integers.")
+ for lv in level:
+ if not isinstance(lv, (int, np.integer)) or not 0 < lv < 100:
+ raise ValueError(f"level values must be integers in (0, 100); got {lv}.")
+
+ try:
+ # If mask was a training feature but is absent from test_data, add it (all test obs are valid)
+ if 'mask' in self.features and 'mask' not in test_data.columns:
+ test_data = test_data.copy()
+ test_data['mask'] = np.ones_like(test_data['series_id'], dtype=np.int32)
+
+ # Check that all features used during training exist in test_data
+ missing_features = [f for f in self.features if f not in test_data.columns]
+ if missing_features:
+ raise ValueError(f"Missing features in test_data: {missing_features}")
+
+ model_name = "Hyper-Tree-TSB"
+
+ if type == "forecast":
+ # Extract stored final states in the correct order of the test data series
+ test_series_ids = test_data['series_id'].unique()
+ last_p = torch.stack([self.fcst_states[series_id]['last_p'] for series_id in test_series_ids])
+ last_z = torch.stack([self.fcst_states[series_id]['last_z'] for series_id in test_series_ids])
+
+ # Classical TSB: flat forecast p_T * z_T over the horizon (via
+ # the shared recursion, also used by the validation metric).
+ point = self._roll_forecast(last_p, last_z, self.fcst_h)
+
+ # Create output dataframe
+ out_df = pd.DataFrame({
+ "series_id": test_data["series_id"].to_numpy().flatten(),
+ "date": test_data["date"].to_numpy().flatten(),
+ "fcst": point.flatten().numpy(),
+ "model": model_name,
+ })
+
+ if level is not None:
+ columns = interval_columns(
+ point=point.numpy(),
+ scores=self._cs_scores,
+ levels=level,
+ method=self._pi_config.method,
+ model_name=model_name,
+ cal_order=self._cs_series_order,
+ target_order=list(test_series_ids),
+ )
+ for col_name, values in columns.items():
+ out_df[col_name] = values
+
+ elif type == "parameters":
+ fcst_params = torch.clamp(
+ self.sigmoid_fn(torch.tensor(self.model.predict(test_data[self.features]), dtype=self.dtype)
+ ),
+ min=self.eps,
+ max=1-self.eps
+ ).reshape(self.n_series, -1, self.n_params)
+ out_df = pd.DataFrame({
+ "series_id": test_data["series_id"].to_numpy().flatten(),
+ "date": test_data["date"].to_numpy().flatten(),
+ "model": model_name,
+ })
+ for i, param_name in enumerate(["alpha_p", "alpha_d"]):
+ out_df[param_name] = fcst_params[:, :, i].flatten().numpy()
+
+ return out_df
+
+ except Exception as e:
+ raise RuntimeError(f"Forecasting not successful: {str(e)}") from e
diff --git a/hypertrees/models/HyperTreeVAR.py b/hypertrees/models/HyperTreeVAR.py
new file mode 100644
index 0000000..c30bfa3
--- /dev/null
+++ b/hypertrees/models/HyperTreeVAR.py
@@ -0,0 +1,597 @@
+import warnings
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+from torch.autograd import grad as autograd
+import lightgbm as lgb
+from typing import Callable, Optional, Tuple
+
+from ..utils import TrainingResult, GaussNewtonHessian
+from ..conformal import ForecastIntervals
+from ._var_base import _HyperTreeVARBase
+
+
+class HyperTreeVAR(_HyperTreeVARBase):
+ """
+ Class that implements a Hyper-Tree-VAR(p) model for multivariate time series forecasting.
+
+ The Hyper-Tree-VAR(p) model extends the Hyper-Tree-AR(p) model to a vector
+ autoregression over an aligned panel of k series, learning the full
+ time-varying coefficient matrices A_1(x), ..., A_p(x) with gradient boosted
+ trees so that y_{i,t} = sum_j A_j[i, :](x_{i,t}) . y_{t-j}. Cross-series
+ dependence is captured by the off-diagonal coefficients of the lag matrices
+ (see ``_var_base.py`` for the formulation, data requirements, coefficient
+ ordering, and the restricted ``type="factor"`` design).
+
+ Key features:
+ - Combines tree-based models (LightGBM) with vector autoregressive modeling
+ - Learns the time-varying lag matrices A_1, ..., A_p as functions of
+ features (full k x k, or own + factor lags with type="factor")
+ - Captures cross-series (Granger-causal) lead/lag dependencies via the
+ off-diagonal coefficients
+ - Provides VAR coefficients that can vary over time
+
+ Use this model when:
+ - Your series influence each other and forecasts should exploit
+ cross-series lead/lag structure
+ - The panel is small (k * p up to a few dozen coefficients) and
+ coefficient-level interpretability (including SHAP values per
+ coefficient) is desired
+ - For larger panels, use HyperTreeNetVAR, whose boosting cost is
+ independent of the number of coefficients
+
+ Example usage:
+ ```python
+ # Imports
+ from hypertrees.models.HyperTreeVAR import HyperTreeVAR
+ import numpy as np
+ import pandas as pd
+ import matplotlib.pyplot as plt
+
+ # Initialize model
+ lag_p = 4
+ frequency = 'M'
+ fcst_h = 12
+ model = HyperTreeVAR(p=lag_p, freq=frequency, fcst_h=fcst_h)
+
+ # Data
+ # The data needs to be an aligned panel (equal lengths and identical dates
+ # across series) with the columns: 'date', 'series_id', 'value'. All other
+ # columns are automatically treated as features. You don't have to add
+ # lag-values yourself, this happens automatically during training.
+ rng = np.random.RandomState(1)
+ dates = pd.date_range("2010-01-01", periods=120 + fcst_h, freq="MS")
+ df = pd.concat(
+ [
+ pd.DataFrame({
+ "series_id": f"s{i}",
+ "date": dates,
+ "value": base + np.cumsum(rng.randn(len(dates))),
+ "month": dates.month,
+ "quarter": dates.quarter,
+ "series_num": i, # identifies the series, so equations can differ
+ })
+ for i, base in enumerate([100.0, 150.0])
+ ],
+ ignore_index=True,
+ )
+ test = df.groupby("series_id", sort=False).tail(fcst_h)
+ train = df.drop(test.index)
+
+ # Train model
+ model.train(
+ lgb_params={'learning_rate': 0.1},
+ num_iterations=100,
+ train_data=train
+ )
+
+ # Generate forecasts and inspect the time-varying VAR coefficients
+ forecasts = model.forecast(test_data=test)
+ coefficients = model.forecast(test_data=test, type="parameters")
+
+ # Plot results
+ for sid, group in df.groupby("series_id", sort=False):
+ plt.plot(group["date"], group["value"], label=f"Actual {sid}",
+ color='#2E86AB', linewidth=2, alpha=0.8)
+ for sid, group in forecasts.groupby("series_id", sort=False):
+ plt.plot(group["date"], group["fcst"], label=f"Forecast {sid}",
+ color='#F18F01', linestyle='--', linewidth=2, alpha=0.8)
+
+ plt.title('Aligned Panel - VAR Forecasts', fontsize=14)
+ plt.legend(frameon=True, fancybox=True)
+ plt.grid(True, alpha=0.3)
+ plt.tight_layout()
+ ```
+ """
+
+ _model_label = "Hyper-Tree-VAR"
+ _valid_forecast_types = ("forecast", "parameters")
+
+ def __init__(
+ self,
+ p: int = 2,
+ freq: str = "M",
+ fcst_h: int = 1,
+ loss_fn: Callable = nn.MSELoss(),
+ scaling: Optional[str] = "mean",
+ type: str = "full",
+ hessian_method: str = "analytic",
+ n_hessian_probes: int = 5,
+ ):
+ """
+ Initialize the Hyper-Tree-VAR(p) model.
+
+ Arguments
+ ----------
+ p : int
+ VAR lag order. Must be a positive integer.
+ freq : str
+ Frequency of the time series (e.g., 'D' for daily, 'M' for monthly,
+ 'Q' for quarterly, 'Y' for yearly).
+ fcst_h : int
+ Forecast horizon (number of periods to forecast ahead).
+ loss_fn : Callable
+ Loss function for optimization. Must be a PyTorch loss function.
+ Default is MSE loss. Losses other than nn.MSELoss are not
+ recommended, as they have not been systematically tested yet.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
+ scaling : str, optional
+ Per-series scaling applied internally before training; forecasts
+ (and prediction intervals) are transformed back to the original
+ scale automatically. Options: "mean" (default; divide by the
+ mean absolute training value), "standard" (z-score), or None.
+ Strongly recommended for heterogeneous panels: VAR coefficients
+ multiply *other* series' values, so unscaled panels force the
+ model to learn scale conversions while the loss is dominated by
+ the largest series. Coefficients returned by
+ ``forecast(type="parameters")`` live in the scaled space.
+ type : str
+ Structure of the VAR design vector. Options:
+ - "full" (default): unrestricted VAR; every equation regresses on
+ the lags of all k series (``k * p`` coefficients per equation,
+ one boosted tree each).
+ - "factor": restricted GVAR-style design; every equation
+ regresses on its own lags plus the lags of the equal-weighted
+ cross-sectional average of the scaled panel (``2 * p``
+ coefficients per equation, independent of k). Recommended for
+ larger panels, where the unrestricted design overparameterizes.
+ Parameter columns are named ``A{j}(own)`` / ``A{j}(factor)``
+ and the model name becomes ``Hyper-Tree-FactorVAR(p)``.
+ hessian_method : str
+ Method for computing the Hessian diagonal. Options:
+ - "exact": Exact diagonal Hessian via per-coefficient second-order
+ autograd (one backward pass per coefficient, i.e. k * p per
+ iteration -- costly for larger panels).
+ - "analytic" (default): Closed-form gradients and exact diagonal
+ Hessians, exploiting that the VAR fit is linear in its
+ parameters (dL/dA_q = l'(y_hat) * z_q and
+ d2L/dA_q2 = l''(y_hat) * z_q**2, the second-order fit term
+ vanishing exactly). Produces the same values as "exact" for any
+ loss that is a mean/sum of per-observation terms -- which covers
+ all standard PyTorch regression losses -- at a fraction of the
+ cost: at most one small double-backward through
+ loss(fit, target) instead of one backward per coefficient.
+ nn.MSELoss uses a fully closed-form fast path with no autograd
+ at all.
+ - "gn": Gauss-Newton approximation estimated via Hutchinson
+ probing. Guarantees positive semi-definite Hessians. Because
+ the VAR fit is linear in its parameters, this estimates the
+ same diagonal as "analytic", with Hutchinson sampling variance.
+ n_hessian_probes : int
+ Number of Hutchinson probes for Gauss-Newton Hessian diagonal estimation.
+ Only used when hessian_method="gn". More probes reduce variance but
+ increase computation. Default is 5.
+ """
+ super().__init__(
+ p=p,
+ freq=freq,
+ fcst_h=fcst_h,
+ loss_fn=loss_fn,
+ scaling=scaling,
+ type=type,
+ )
+ if hessian_method not in ("exact", "analytic", "gn"):
+ raise ValueError("hessian_method must be one of 'exact', 'analytic', or 'gn'.")
+ if not isinstance(n_hessian_probes, int) or n_hessian_probes <= 0:
+ raise ValueError("n_hessian_probes must be a positive integer.")
+ if hessian_method == "gn" and not isinstance(loss_fn, nn.MSELoss):
+ warnings.warn(
+ f"Loss {loss_fn.__class__.__name__} is not nn.MSELoss. The Gauss-Newton "
+ "Hessian requires a twice-differentiable loss; non-smooth losses "
+ "(e.g., L1Loss, quantile loss, HuberLoss/SmoothL1Loss outside the quadratic "
+ "region) have zero or undefined second derivatives at kinks, "
+ "causing degenerate Hessians."
+ )
+
+ self.hessian_method = hessian_method
+ self.n_hessian_probes = n_hessian_probes
+ self._fit = None
+ self._target = None
+ self._lags = None
+
+ # Bind Hessian computation strategy
+ if hessian_method == "exact":
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_exact
+ elif hessian_method == "analytic":
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_analytic
+ else:
+ self._gn_hessian = GaussNewtonHessian(loss_fn, n_hessian_probes, self.dtype)
+ self.calculate_gradients_and_hessians = self._calculate_gradients_and_hessians_gn
+
+ def objective_fn(self, predt: np.ndarray, data: lgb.Dataset) -> Tuple[np.ndarray, np.ndarray]:
+ """
+ Custom objective function for LightGBM training.
+
+ This function defines the gradients and hessians for the LightGBM model
+ based on the PyTorch loss function. It converts the raw LightGBM outputs
+ to VAR coefficients, computes the loss, and then derives gradients and
+ Hessians via the bound ``hessian_method`` strategy.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM, representing the VAR coefficients per
+ training row.
+ data : lgb.Dataset
+ LightGBM dataset containing the target values.
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians for LightGBM optimization.
+ """
+ self._iter_count += 1
+
+ target = torch.tensor(
+ data.get_label().reshape(self.k, -1), dtype=self.dtype
+ )
+ params, loss = self.get_params_loss(predt, target, self._Z_train, requires_grad=True)
+ if not torch.isfinite(loss):
+ raise RuntimeError(
+ f"Training diverged at boosting iteration {self._iter_count}: the loss "
+ "is no longer finite. With one boosted tree per VAR coefficient, "
+ "strongly correlated series make the per-coefficient Newton steps "
+ "overshoot: reduce learning_rate (a rule of thumb is eta / k) and "
+ "keep per-series scaling enabled."
+ )
+ grad, hess = self.calculate_gradients_and_hessians(loss, params)
+
+ return grad, hess
+
+ def get_params_loss(
+ self,
+ predt: np.ndarray,
+ target: torch.Tensor,
+ Z: torch.Tensor,
+ requires_grad: bool = False,
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
+ """
+ Transform LightGBM outputs into VAR coefficients and calculate loss.
+
+ This function:
+ 1. Reshapes the raw outputs into the coefficient matrix
+ 2. Computes the fitted values via the VAR forward pass
+ 3. Calculates the loss between fitted and actual values
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM (flattened Fortran-order).
+ target : torch.Tensor
+ Target values (actual time series values), shape ``(k, T_r)``.
+ Z : torch.Tensor
+ VAR design matrix, shape ``(T_r, k * p)`` for the full design or
+ ``(k * T_r, 2 * p)`` for the factor design.
+ requires_grad : bool
+ Whether to compute gradients (True during training).
+
+ Returns
+ -------
+ Tuple[torch.Tensor, torch.Tensor]
+ Parameters tensor and loss value.
+ """
+ params = nn.Parameter(
+ torch.tensor(
+ predt.reshape(-1, self.n_params, order="F"),
+ dtype=self.dtype
+ ),
+ requires_grad=requires_grad
+ )
+
+ fit = self._compute_fit(params, Z)
+ loss = self.loss_fn(fit, target)
+
+ if self.hessian_method in ("gn", "analytic"):
+ self._fit = fit
+ self._target = target
+ self._lags = Z
+
+ return params, loss
+
+ def _calculate_gradients_and_hessians_exact(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
+ """Exact diagonal Hessian via per-coefficient second-order autograd.
+
+ One backward pass per VAR coefficient (k * p per iteration); identical
+ values to "analytic" for any per-observation loss, at much higher cost.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ params : torch.Tensor
+ Model parameters (VAR coefficients as an ``nn.Parameter``,
+ shape ``(k * T_r, n_params)``).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ loss.backward(create_graph=True)
+ grad = params.grad
+ hess = [
+ autograd(grad[:, i].sum(), params, retain_graph=True)[0][:, i:(i + 1)]
+ for i in range(self.n_params)
+ ]
+
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = torch.cat(hess, dim=1).cpu().detach().numpy().ravel(order="F")
+ params.grad = None
+
+ return grad, hess
+
+ def _calculate_gradients_and_hessians_analytic(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
+ """Closed-form gradients and exact diagonal Hessians via model linearity.
+
+ Since the VAR fit is linear in its parameters, ``grad = l'(y_hat) * z``
+ and ``hess = l''(y_hat) * z**2``, matching the "exact" method for any
+ per-observation loss. MSELoss uses closed-form derivatives; other
+ losses use one small double-backward through ``loss(fit, target)``.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model (unused; derivatives come from the
+ fit/target/design stored by ``get_params_loss``).
+ params : torch.Tensor
+ Model parameters (unused, kept for a uniform dispatch signature).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ fit = self._fit.detach()
+ target = self._target
+ Z = self._lags
+
+ if isinstance(self.loss_fn, nn.MSELoss) and self.loss_fn.reduction in ("mean", "sum"):
+ # MSE fast path: l' = scale * (y_hat - y), l'' = scale
+ scale = 2.0 / fit.numel() if self.loss_fn.reduction == "mean" else 2.0
+ g = scale * (fit - target)
+ h = torch.full_like(fit, scale)
+ else:
+ # Generic path: per-element first and second loss derivatives via
+ # a double-backward through the (tiny) loss(fit, target) graph.
+ # Requires the loss to have a well-defined double-backward (true
+ # for HuberLoss/SmoothL1Loss and other standard smooth losses).
+ fit_leaf = fit.requires_grad_(True)
+ loss_local = self.loss_fn(fit_leaf, target)
+ g = autograd(loss_local, fit_leaf, create_graph=True)[0]
+ h = autograd(g.sum(), fit_leaf)[0].detach()
+ g = g.detach()
+
+ # Broadcast the per-observation loss derivatives over the design
+ # matrix (shared per time step for the full design, per row for the
+ # factor design), then Fortran-ravel as expected by LightGBM.
+ if self.type == "factor":
+ grad = g.reshape(-1, 1) * Z
+ hess = h.reshape(-1, 1) * Z ** 2
+ else:
+ grad = (g.unsqueeze(2) * Z.unsqueeze(0)).reshape(-1, self.n_params)
+ hess = (h.unsqueeze(2) * Z.unsqueeze(0) ** 2).reshape(-1, self.n_params)
+ grad = grad.cpu().numpy().ravel(order="F")
+ hess = hess.cpu().numpy().ravel(order="F")
+
+ self._fit = None
+ self._target = None
+ self._lags = None
+
+ return grad, hess
+
+ def _calculate_gradients_and_hessians_gn(self, loss: torch.Tensor, params: torch.Tensor) -> Tuple[np.ndarray, np.ndarray]:
+ """Gauss-Newton Hessian diagonal estimated via Hutchinson probing.
+
+ Parameters
+ ----------
+ loss : torch.Tensor
+ Loss value from the model.
+ params : torch.Tensor
+ Model parameters (VAR coefficients as an ``nn.Parameter``,
+ shape ``(k * T_r, n_params)``).
+
+ Returns
+ -------
+ Tuple[np.ndarray, np.ndarray]
+ Gradients and hessians as numpy arrays in the format expected by LightGBM.
+ """
+ grad = autograd(loss, params, retain_graph=True)[0]
+ rng = torch.Generator().manual_seed(self._iter_count)
+ hess = self._gn_hessian.estimate(self._fit, self._target, params, rng)
+ self._fit = None
+ self._target = None
+ self._lags = None
+ grad = grad.cpu().detach().numpy().ravel(order="F")
+ hess = hess.cpu().detach().numpy().ravel(order="F")
+
+ return grad, hess
+
+ def _fit_from_predt(self, predt: np.ndarray, Z: torch.Tensor) -> torch.Tensor:
+ """Compute the fitted values for raw LightGBM coefficient outputs.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM (flattened Fortran-order).
+ Z : torch.Tensor
+ Design matrix for the dataset being evaluated, shape
+ ``(T_r, k * p)`` for the full design or ``(k * T_r, 2 * p)`` for
+ the factor design.
+
+ Returns
+ -------
+ torch.Tensor
+ Fitted values, shape ``(k, T_r)``.
+ """
+ params = torch.tensor(
+ predt.reshape(-1, self.n_params, order="F"), dtype=self.dtype
+ )
+
+ return self._compute_fit(params, Z)
+
+ def _num_class(self) -> int:
+ """LightGBM output dimension: one tree per VAR coefficient."""
+
+ return self.n_params
+
+ def _reset_training_state(self) -> None:
+ """Reset per-training state, including the Hessian-strategy buffers."""
+ super()._reset_training_state()
+ self._fit = None
+ self._target = None
+ self._lags = None
+
+ def _post_datasets_setup(self, seed: int) -> None:
+ """Warn when the one-vs-all strategy becomes the runtime bottleneck."""
+ if self.type == "full" and self.n_params > 50:
+ warnings.warn(
+ f"HyperTreeVAR grows num_class = k * p = {self.n_params} trees per "
+ f"boosting iteration, which scales linearly in runtime. For panels "
+ f"of this size consider HyperTreeNetVAR (GBDT cost independent of "
+ f"the number of coefficients) or type='factor' (2 * p "
+ f"coefficients per equation, independent of k)."
+ )
+
+ def train(
+ self,
+ lgb_params: dict = None,
+ num_iterations: int = 100,
+ train_data: pd.DataFrame = None,
+ validation: bool = False,
+ early_stopping_round: Optional[int] = None,
+ seed: int = 123,
+ verbose: int = -1,
+ deterministic: bool = True,
+ forecast_intervals: Optional[ForecastIntervals] = None,
+ ) -> TrainingResult:
+ """
+ Train the Hyper-Tree-VAR model on an aligned panel of time series.
+
+ This method:
+ 1. Pivots the panel and builds the VAR design matrix
+ 2. Sets up LightGBM datasets (one row per series and time step)
+ 3. Trains the model using gradient boosting
+
+ The training data must contain columns:
+ - 'series_id': Identifier for each time series
+ - 'date': Timestamp for each observation
+ - 'value': Target value to forecast
+ - Additional feature columns used for forecasting
+
+ All series must have the same length and identical dates (aligned
+ panel); see ``_var_base.py``.
+
+ Parameters
+ ----------
+ lgb_params : dict
+ LightGBM parameters like 'learning_rate', 'num_leaves', etc.
+ num_iterations : int
+ Number of boosting rounds for training. Note that each round grows
+ one tree per coefficient (``k * p`` for type="full", ``2 * p`` for
+ type="factor").
+ train_data : pd.DataFrame
+ Training data containing series_id, date, value and feature columns
+ validation : bool
+ If True, a validation set will be created for evaluation. It splits
+ the last fcst_h time steps of each series for validation.
+ early_stopping_round : int, optional
+ If provided, training will stop if the validation loss does not
+ improve for this many rounds.
+ seed : int
+ Random seed for reproducibility
+ verbose : int
+ Verbosity level for LightGBM training
+ deterministic : bool
+ If True, sets LightGBM's ``deterministic`` and ``force_row_wise``
+ parameters to ensure reproducible results. May slow down training.
+ See https://lightgbm.readthedocs.io/en/latest/Parameters.html#deterministic
+ forecast_intervals : ForecastIntervals, optional
+ If provided, calibrate conformal prediction intervals via
+ rolling-window cross-validation after the main model is trained.
+ The collected conformity scores are then used by
+ ``forecast(..., level=[...])`` to produce ``-lo-`` /
+ ``-hi-`` columns. See
+ :class:`hypertrees.conformal.ForecastIntervals`.
+
+ Returns
+ -------
+ TrainingResult
+ Object containing evaluation results and training information.
+ """
+ def _model_factory():
+ return HyperTreeVAR(
+ p=self.p,
+ freq=self.freq,
+ fcst_h=self.fcst_h,
+ loss_fn=self.loss_fn,
+ scaling=self.scaling,
+ type=self.type,
+ hessian_method=self.hessian_method,
+ n_hessian_probes=self.n_hessian_probes,
+ )
+
+ cal_train_kwargs = dict(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ validation=False,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ )
+
+ return self._train_core(
+ lgb_params=lgb_params,
+ num_iterations=num_iterations,
+ train_data=train_data,
+ validation=validation,
+ early_stopping_round=early_stopping_round,
+ seed=seed,
+ verbose=verbose,
+ deterministic=deterministic,
+ forecast_intervals=forecast_intervals,
+ model_factory=_model_factory,
+ cal_train_kwargs=cal_train_kwargs,
+ )
+
+ def _forecast_params(self, features_df: pd.DataFrame) -> np.ndarray:
+ """Forecast the ``(n_rows, n_params)`` coefficient matrix from the GBDT.
+
+ Parameters
+ ----------
+ features_df : pd.DataFrame
+ Feature frame (training feature columns only).
+
+ Returns
+ -------
+ np.ndarray
+ VAR coefficients per row, shape ``(n_rows, n_params)``.
+ """
+ params_fcst = np.asarray(self.model.predict(features_df))
+ # Booster.predict returns (n_rows, n_params) for multi-class output
+ if params_fcst.ndim == 1:
+ params_fcst = params_fcst.reshape(-1, self.n_params)
+
+ return params_fcst
diff --git a/hypertrees/models/__init__.py b/hypertrees/models/__init__.py
index e70251c..0cd2dd9 100644
--- a/hypertrees/models/__init__.py
+++ b/hypertrees/models/__init__.py
@@ -1,6 +1,16 @@
-"""Forecasting with Hyper-Trees"""
-
+# Autoregressive, univariate
from .HyperTreeAR import HyperTreeAR
-from .HyperTreeETS import HyperTreeETS
from .HyperTreeNetAR import HyperTreeNetAR
-from .HyperTreeSTL import HyperTreeSTL
+from .HyperTreeARMA import HyperTreeARMA
+from .HyperTreeNetARMA import HyperTreeNetARMA
+
+# Autoregressive, multivariate (aligned panels)
+from .HyperTreeVAR import HyperTreeVAR
+from .HyperTreeNetVAR import HyperTreeNetVAR
+
+# Exponential smoothing state-space recursions
+from .HyperTreeETS import HyperTreeETS
+from .HyperTreeTSB import HyperTreeTSB
+
+# Decomposition
+from .HyperTreeSTL import HyperTreeSTL
\ No newline at end of file
diff --git a/hypertrees/models/_var_base.py b/hypertrees/models/_var_base.py
new file mode 100644
index 0000000..e94631e
--- /dev/null
+++ b/hypertrees/models/_var_base.py
@@ -0,0 +1,1146 @@
+"""Shared base for the Hyper-Tree VAR models (``HyperTreeVAR`` / ``HyperTreeNetVAR``).
+
+The Hyper-Tree VAR models extend the univariate AR variants to a
+multivariate VAR(p) target model. For a panel of ``k`` aligned series:
+
+ y_{i,t} = sum_{j=1..p} sum_{m=1..k} A_j[i, m](x_{i,t}) * y_{m,t-j}
+
+i.e. every series' forecast is a feature-dependent linear combination of the
+lagged values of *all* series. Cross-series dependence in the conditional
+mean is carried entirely by the off-diagonal elements of the lag matrices
+A_j (Granger-causal lead/lag effects). A multivariate innovation
+distribution (e.g. the multivariate Gaussian used by GluonTS' DeepVAR) only
+models the *contemporaneous covariance of the residuals*; it does not enter
+the conditional mean and is therefore not needed for point forecasts.
+Because all k equations share the same regressors, minimizing a
+per-observation loss equation-by-equation is equivalent to the joint
+(GLS/SUR) estimate in the classical linear case (Zellner, 1962), so nothing
+is lost by training with the standard element-wise losses used across the
+Hyper-Tree models. Conformal prediction intervals are supported exactly as
+for the other models (per-series, per-horizon-step marginal intervals).
+
+The two concrete models live in their own modules, mirroring the repo's
+one-class-per-file convention:
+
+- ``HyperTreeVAR`` (``HyperTreeVAR.py``): one boosted tree per VAR
+ coefficient, closed-form analytic gradients/Hessians. For small panels;
+ fully interpretable.
+- ``HyperTreeNetVAR`` (``HyperTreeNetVAR.py``): GBDT encoder + MLP decoder
+ (Hyper-TreeNet architecture); boosting cost independent of the number of
+ coefficients, for larger panels.
+
+Both accept a ``type`` argument: ``"full"`` (default; unrestricted,
+``k * p`` coefficients per equation) or ``"factor"``, a restricted VAR in
+the spirit of the Global VAR (GVAR) literature [1, 2], where every equation
+regresses on its **own lags** plus the lags of the equal-weighted
+cross-sectional average of the scaled panel (the GVAR "star variable"),
+i.e. ``2 * p`` coefficients per equation independent of k -- the principled
+remedy for the overparameterization of unrestricted VARs on larger panels.
+
+Data requirements
+-----------------
+- Standard Hyper-Tree layout: columns ``series_id``, ``date``, ``value``,
+ sorted by ``(series_id, date)`` with each series in a contiguous block.
+- The panel must be *aligned*: every series must have the same length and
+ identical dates, because each equation's design vector stacks the lags of
+ all series at the same time points.
+- Because the GBDT learns a single global mapping from features to
+ coefficient vectors, the equations can only differ if some feature
+ identifies the series. As with the other global Hyper-Tree models,
+ include a series-identity feature (e.g. an integer-coded series id)
+ in the feature set -- ideally with pandas ``category`` dtype, so
+ LightGBM applies true categorical splits (an integer-coded identity
+ treated as numeric can only separate series by intervals of an
+ arbitrary coding).
+- Series scales matter more than for the univariate AR models: VAR
+ coefficients multiply *other* series' values, so on an unscaled
+ heterogeneous panel the model must learn scale conversions between every
+ pair of series while the loss is dominated by the largest series.
+ Per-series scaling is therefore built in (``scaling="mean"`` by default)
+ and forecasts are transformed back to the original scale automatically.
+
+Coefficient ordering
+--------------------
+For the full design, the flat coefficient vector of length ``k * p``
+produced per row is ordered lag-major: position ``(j - 1) * k + m`` is the
+coefficient on lag ``j`` of the ``m``-th series (series in training
+first-appearance order), matching the statsmodels VAR design-vector
+convention ``z_t = [y'_{t-1}, y'_{t-2}, ..., y'_{t-p}]'``. For the factor
+design, the own-lag block (j = 1..p) comes first, then the factor-lag block.
+
+References
+----------
+[1] Pesaran, M. H., Schuermann, T., & Weiner, S. M. (2004). Modeling
+ Regional Interdependencies Using a Global Error-Correcting
+ Macroeconometric Model. Journal of Business & Economic Statistics,
+ 22(2), 129-162. (Global VAR; the "star variable" construction)
+[2] Chudik, A., & Pesaran, M. H. (2016). Theory and Practice of GVAR
+ Modelling. Journal of Economic Surveys, 30(1), 165-197.
+[3] Bernanke, B. S., Boivin, J., & Eliasz, P. (2005). Measuring the
+ Effects of Monetary Policy: A Factor-Augmented Vector Autoregressive
+ (FAVAR) Approach. The Quarterly Journal of Economics, 120(1), 387-422.
+"""
+
+import time
+import warnings
+from typing import Callable, Dict, List, Optional, Tuple
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+import lightgbm as lgb
+
+from ..utils import CustomLogger
+lgb.register_logger(CustomLogger())
+
+from ..utils import TrainingResult, validate_series_order, NoDeepcopyObjective, DatasetReferences
+from ..conformal import (
+ ForecastIntervals,
+ validate_calibration_length,
+ rolling_origin_residuals,
+ interval_columns,
+)
+
+# Columns that are never features.
+_RESERVED_COLUMNS = ("series_id", "date", "value")
+
+
+def _pivot_panel(data: pd.DataFrame, name: str) -> Tuple[np.ndarray, List, np.ndarray]:
+ """Pivot an aligned long panel into a ``(T, k)`` value matrix.
+
+ Validates that all series have the same length and identical dates
+ (required so that the VAR design vector ``z_t`` exists for every row).
+
+ Parameters
+ ----------
+ data : pd.DataFrame
+ Long-format panel with ``series_id``, ``date``, ``value`` columns,
+ sorted by ``(series_id, date)`` with contiguous series blocks.
+ name : str
+ Label used in error messages.
+
+ Returns
+ -------
+ Tuple[np.ndarray, List, np.ndarray]
+ Value matrix ``Y`` of shape ``(T, k)`` with columns in
+ first-appearance series order, the series order, and the shared
+ date vector of length ``T``.
+ """
+ lengths = data.groupby("series_id", sort=False).size()
+ if lengths.nunique() != 1:
+ raise ValueError(
+ f"{name}: a VAR model requires an aligned panel where all series have "
+ f"the same length. Found lengths: {lengths.to_dict()}."
+ )
+ T = int(lengths.iloc[0])
+ series_order = list(dict.fromkeys(data["series_id"].tolist()))
+ k = len(series_order)
+
+ dates = pd.to_datetime(data["date"]).to_numpy().reshape(k, T)
+ if k > 1 and not (dates == dates[0]).all():
+ raise ValueError(
+ f"{name}: all series must share identical dates (aligned panel). "
+ f"The VAR design vector stacks the lags of all series at the same "
+ f"time points, which requires aligned observations."
+ )
+
+ Y = data["value"].to_numpy(dtype=np.float64).reshape(k, T).T # (T, k)
+
+ return Y, series_order, dates[0]
+
+
+def _validate_aligned_dates(data: pd.DataFrame, name: str) -> None:
+ """Validate equal lengths and identical dates across series.
+
+ Same alignment check as :func:`_pivot_panel` but without requiring a
+ ``value`` column, so it can be applied to forecast inputs.
+
+ Parameters
+ ----------
+ data : pd.DataFrame
+ Long-format panel with ``series_id`` and ``date`` columns, sorted by
+ ``(series_id, date)`` with contiguous series blocks.
+ name : str
+ Label used in error messages.
+
+ Raises
+ ------
+ ValueError
+ If series lengths differ or dates are not identical across series.
+ """
+ lengths = data.groupby("series_id", sort=False).size()
+ if lengths.nunique() != 1:
+ raise ValueError(
+ f"{name}: all series must have the same number of rows. "
+ f"Found lengths: {lengths.to_dict()}."
+ )
+ T = int(lengths.iloc[0])
+ k = lengths.size
+ dates = pd.to_datetime(data["date"]).to_numpy().reshape(k, T)
+ if k > 1 and not (dates == dates[0]).all():
+ raise ValueError(f"{name}: all series must share identical dates (aligned panel).")
+
+
+def _build_var_lags(Y: np.ndarray, p: int) -> np.ndarray:
+ """Build the VAR design matrix ``Z`` from the value matrix ``Y``.
+
+ Parameters
+ ----------
+ Y : np.ndarray
+ Value matrix of shape ``(T, k)``.
+ p : int
+ VAR lag order.
+
+ Returns
+ -------
+ np.ndarray
+ Design matrix of shape ``(T - p, k * p)`` where row ``r``
+ corresponds to time ``t = r + p`` and holds
+ ``[y'_{t-1}, y'_{t-2}, ..., y'_{t-p}]`` (lag-major ordering).
+ """
+ T = Y.shape[0]
+
+ return np.concatenate([Y[p - j: T - j] for j in range(1, p + 1)], axis=1)
+
+
+class _HyperTreeVARBase:
+ """Shared plumbing for the Hyper-Tree VAR variants.
+
+ Handles panel validation/pivoting, design-matrix construction, dataset
+ preparation, the training loop, the evaluation function, the multi-step
+ forecast recursion, conformal interval wiring, and forecast-origin
+ re-anchoring. Subclasses provide the LightGBM objective
+ (``objective_fn``), the fitted values for a raw prediction vector
+ (``_fit_from_predt``), the feature -> coefficient-matrix mapping
+ (``_forecast_params``), and the LightGBM output dimension
+ (``_num_class``).
+ """
+
+ _model_label = "Hyper-Tree-VAR"
+ _valid_forecast_types = ("forecast", "parameters")
+
+ def __init__(
+ self,
+ p: int = 2,
+ freq: str = "M",
+ fcst_h: int = 1,
+ loss_fn: Callable = nn.MSELoss(),
+ scaling: Optional[str] = "mean",
+ type: str = "full",
+ ):
+ """
+ Initialize the shared VAR state and validate the common arguments.
+
+ Arguments
+ ----------
+ p : int
+ VAR lag order. Must be a positive integer.
+ freq : str
+ Frequency of the time series (e.g., 'D' for daily, 'M' for monthly,
+ 'Q' for quarterly, 'Y' for yearly).
+ fcst_h : int
+ Forecast horizon (number of periods to forecast ahead).
+ loss_fn : Callable
+ Loss function for optimization. Must be a PyTorch loss function.
+ nn.L1Loss is rejected (zero second derivative almost everywhere
+ breaks Newton boosting).
+ scaling : str, optional
+ Per-series scaling applied internally before training; forecasts
+ (and prediction intervals) are transformed back to the original
+ scale automatically. Options:
+ - "mean" (default): divide each series by its mean absolute
+ training value. Location-free, so it introduces no implicit
+ intercept, and per-equation least squares is equivariant under
+ it (forecasts match manual pre-scaling exactly).
+ - "standard": z-score per series (subtract the training mean,
+ divide by the training standard deviation).
+ - None: use the series as provided.
+ Coefficients returned by ``forecast(type="parameters")`` live in
+ the scaled space.
+ type : str
+ Structure of the VAR design vector. Options:
+ - "full" (default): unrestricted VAR; every equation regresses on
+ the lags of *all* k series (``k * p`` coefficients per equation).
+ - "factor": restricted VAR in the spirit of the Global VAR (GVAR)
+ literature; every equation regresses on its **own lags** plus
+ the lags of the equal-weighted cross-sectional average of the
+ scaled panel (the GVAR "star variable"), i.e. ``2 * p``
+ coefficients per equation, independent of k. The principled
+ remedy for the overparameterization of unrestricted VARs on
+ larger panels.
+ """
+ if p <= 0:
+ raise ValueError("Parameter 'p' must be a positive integer.")
+ if fcst_h <= 0:
+ raise ValueError("Forecast horizon 'fcst_h' must be a positive integer.")
+ if not isinstance(freq, str):
+ raise TypeError("freq must be a string.")
+ if not isinstance(loss_fn, nn.Module):
+ raise TypeError("loss_fn must be a PyTorch loss function.")
+ if isinstance(loss_fn, nn.L1Loss):
+ raise ValueError(
+ "nn.L1Loss is not supported: its second derivative is zero almost "
+ "everywhere, so LightGBM's Newton boosting receives all-zero Hessians "
+ "and cannot grow trees. Use nn.HuberLoss or nn.SmoothL1Loss for an "
+ "MAE-like loss with usable curvature."
+ )
+ if getattr(loss_fn, "reduction", "mean") == "none":
+ raise ValueError(
+ "loss_fn must use a scalar reduction ('mean' or 'sum'); "
+ "reduction='none' returns per-element losses that the "
+ "boosting objective cannot consume."
+ )
+ if scaling not in (None, "mean", "standard"):
+ raise ValueError("scaling must be one of None, 'mean', or 'standard'.")
+ if type not in ("full", "factor"):
+ raise ValueError("type must be either 'full' or 'factor'.")
+
+ self.p = p
+ self.freq = freq
+ self.fcst_h = fcst_h
+ self.loss_fn = loss_fn
+ self.loss_name = self.loss_fn.__class__.__name__
+ self.scaling = scaling
+ self.type = type
+ self.dtype = torch.float32
+ self.device = "cpu"
+
+ self.model = None
+ self.features = None # Stores feature names after training
+ self.is_trained = False # Flag to track if model has been trained
+ self.dataset_references = {} # Store references to LightGBM datasets
+ self.k = None # Number of series (set during training)
+ self.n_params = None # k*p (full) or 2*p (factor), set during training
+ self.series_order_ = None # Training series order (axis/coefficient order)
+ self._Z_train = None # design tensor: (T_train, k*p) full, (N, 2*p) factor
+ self._Z_eval = None # design tensor for the validation split
+ self._fcst_state = None # lag state at the forecast origin
+ self._scale_loc = None # (k,) per-series location (training order)
+ self._scale_scale = None # (k,) per-series scale (training order)
+ self._iter_count = 0
+
+ # Conformal prediction interval state (populated when train() is
+ # called with forecast_intervals).
+ self._is_calibrated = False
+ self._cs_scores = None # conformity scores (n_windows, n_series, fcst_h)
+ self._cs_series_order = None # series order along axis 1 of _cs_scores
+ self._pi_config = None # ForecastIntervals configuration
+
+ # ------------------------------------------------------------------
+ # Subclass hooks
+ # ------------------------------------------------------------------
+ def objective_fn(self, predt: np.ndarray, data: lgb.Dataset) -> Tuple[np.ndarray, np.ndarray]:
+ """Custom objective function for LightGBM training (subclass hook)."""
+ raise NotImplementedError
+
+ def _fit_from_predt(self, predt: np.ndarray, Z: torch.Tensor) -> torch.Tensor:
+ """Compute the (gradient-free) fitted values for raw LightGBM outputs.
+
+ Used by :meth:`eval_fn` to monitor the loss on the train/validation
+ datasets without building an autograd graph.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM (flattened Fortran-order).
+ Z : torch.Tensor
+ Design matrix for the dataset being evaluated, shape ``(T_r, k * p)``.
+
+ Returns
+ -------
+ torch.Tensor
+ Fitted values, shape ``(k, T_r)``.
+ """
+ raise NotImplementedError
+
+ def _forecast_params(self, features_df: pd.DataFrame) -> np.ndarray:
+ """Map a feature frame to the ``(n_rows, n_params)`` coefficient matrix."""
+ raise NotImplementedError
+
+ def _num_class(self) -> int:
+ """LightGBM output dimension; called after the datasets are built."""
+ raise NotImplementedError
+
+ def _post_datasets_setup(self, seed: int) -> None:
+ """Model-specific setup that requires the panel dimensions.
+
+ Called by :meth:`_train_core` after ``_build_panel_datasets`` has set
+ ``self.k`` / ``self.n_params`` and before LightGBM training starts.
+ The default is a no-op.
+
+ Parameters
+ ----------
+ seed : int
+ Random seed forwarded from ``train()``.
+ """
+
+ def _forecast_tree_embeddings(self, test_data: pd.DataFrame, model_name: str) -> pd.DataFrame:
+ """Build the ``type="tree_embeddings"`` output (HyperTreeNetVAR only)."""
+ raise NotImplementedError
+
+ # ------------------------------------------------------------------
+ # Training plumbing
+ # ------------------------------------------------------------------
+ def _validate_train_args(
+ self, lgb_params, num_iterations, train_data, validation,
+ early_stopping_round, seed, verbose, deterministic, forecast_intervals,
+ ) -> None:
+ """Common train() argument validation (mirrors the other models)."""
+ if train_data is None:
+ raise ValueError("train_data must be provided.")
+ if lgb_params is None:
+ raise ValueError("lgb_params must be provided.")
+ if not isinstance(train_data, pd.DataFrame):
+ raise TypeError("train_data must be a pandas DataFrame.")
+ if not isinstance(lgb_params, dict):
+ raise TypeError("lgb_params must be a dictionary.")
+ if not isinstance(num_iterations, int) or num_iterations <= 0:
+ raise ValueError("num_iterations must be a positive integer.")
+ if not isinstance(seed, int):
+ raise TypeError("seed must be an integer.")
+ if not isinstance(verbose, int):
+ raise TypeError("verbose must be an integer.")
+ if early_stopping_round is not None and (
+ not isinstance(early_stopping_round, int) or early_stopping_round <= 0):
+ raise ValueError("early_stopping_round must be a positive integer.")
+ if not isinstance(validation, bool):
+ raise TypeError("validation must be a boolean.")
+ if not isinstance(deterministic, bool):
+ raise TypeError("deterministic must be a boolean.")
+ if forecast_intervals is not None and not isinstance(forecast_intervals, ForecastIntervals):
+ raise TypeError("forecast_intervals must be a ForecastIntervals instance.")
+ if early_stopping_round is not None and not validation:
+ raise ValueError("early_stopping_round can only be used when validation is True.")
+ if validation and early_stopping_round is None:
+ raise ValueError("early_stopping_round must be provided when validation is True.")
+
+ required_columns = ["series_id", "date", "value"]
+ for col in required_columns:
+ if col not in train_data.columns:
+ raise ValueError(f"Required column '{col}' not found in training data.")
+ validate_series_order(train_data, name="train_data")
+
+ def _reset_training_state(self) -> None:
+ """Reset per-training state so instances can be retrained safely."""
+ self.model = None
+ self.dataset_references = {}
+ self.is_trained = False
+ self.features = None
+ self._iter_count = 0
+ self._Z_train = None
+ self._Z_eval = None
+ self._fcst_state = None
+ self._scale_loc = None
+ self._scale_scale = None
+ self._is_calibrated = False
+ self._cs_scores = None
+ self._cs_series_order = None
+ self._pi_config = None
+
+ def _fit_scaling(self, Y: np.ndarray) -> np.ndarray:
+ """Fit per-series scaling statistics on the training panel and transform it.
+
+ Stores ``self._scale_loc`` / ``self._scale_scale`` (training series
+ order) and returns the scaled panel. With ``scaling=None`` the
+ statistics are identity, so downstream code can always apply
+ ``(Y - loc) / scale`` unconditionally.
+
+ Parameters
+ ----------
+ Y : np.ndarray
+ Raw value matrix, shape ``(T, k)``, columns in training series order.
+
+ Returns
+ -------
+ np.ndarray
+ Scaled value matrix, shape ``(T, k)``.
+ """
+ k = Y.shape[1]
+ if self.scaling == "mean":
+ loc = np.zeros(k)
+ scale = np.abs(Y).mean(axis=0)
+ elif self.scaling == "standard":
+ loc = Y.mean(axis=0)
+ scale = Y.std(axis=0)
+ else:
+ loc = np.zeros(k)
+ scale = np.ones(k)
+
+ self._scale_loc = loc
+ # Guard against constant / all-zero series.
+ self._scale_scale = np.where(scale > 1e-8, scale, 1.0)
+
+ return (Y - self._scale_loc) / self._scale_scale
+
+ def _build_panel_datasets(
+ self,
+ train_data: pd.DataFrame,
+ validation: bool,
+ early_stopping_round: Optional[int],
+ free_raw_data: bool = True,
+ ):
+ """Pivot the panel, build the VAR design matrix, and create lgb datasets.
+
+ Sets ``self.k``, ``self.n_params``, ``self.series_order_``,
+ ``self.features``, ``self._Z_train`` / ``self._Z_eval`` and
+ ``self.dataset_references``. When ``validation`` is True, the last
+ ``fcst_h`` time steps of every series form the validation set.
+
+ Parameters
+ ----------
+ train_data : pd.DataFrame
+ Aligned panel of training data.
+ validation : bool
+ Whether to split off a validation set.
+ early_stopping_round : int, optional
+ Number of rounds for early stopping (validation only).
+ free_raw_data : bool
+ Whether to free raw data in the LightGBM datasets.
+
+ Returns
+ -------
+ Tuple[List[lgb.Dataset], List[str], Optional[List], Optional[dict]]
+ ``(valid_sets, valid_names, callbacks, evals_result)`` in the
+ layout expected by ``lgb.train``.
+ """
+ Y, series_order, _ = _pivot_panel(train_data, name="train_data")
+ T, k = Y.shape
+ if k == 1:
+ warnings.warn(
+ "Only one series found. With k=1 the multivariate structure adds "
+ "nothing (and the factor equals the series itself); consider "
+ "HyperTreeAR instead."
+ )
+ if T <= self.p:
+ raise ValueError(
+ f"Series length ({T}) must exceed the lag order p={self.p}; "
+ f"no training rows remain after lagging."
+ )
+
+ self.series_order_ = series_order
+ self.k = k
+ self.n_params = 2 * self.p if self.type == "factor" else k * self.p
+
+ # Scale the panel; targets, design matrix, and the forecast recursion
+ # all live in the scaled space, forecasts are transformed back.
+ Y = self._fit_scaling(Y)
+
+ if self.type == "factor":
+ # Per-series design blocks [own lags j=1..p, factor lags j=1..p],
+ # where the factor is the equal-weighted star variable of the
+ # scaled panel; row r of a block corresponds to time t = r + p.
+ factor = Y.mean(axis=1)
+ factor_lags = np.column_stack(
+ [factor[self.p - j: T - j] for j in range(1, self.p + 1)]
+ )
+ designs = [
+ np.concatenate(
+ [
+ np.column_stack([Y[self.p - j: T - j, i] for j in range(1, self.p + 1)]),
+ factor_lags,
+ ],
+ axis=1,
+ )
+ for i in range(k)
+ ] # k blocks of shape (T - p, 2p)
+
+ def design_slice(lo, hi):
+ # Per-row design rows in dataset (series-major) order.
+ return np.concatenate([d[lo:hi] for d in designs], axis=0)
+ else:
+ Z = _build_var_lags(Y, self.p) # (T - p, k*p), row r <-> time t = r + p
+
+ def design_slice(lo, hi):
+ # One shared design row per time step.
+ return Z[lo:hi]
+
+ # Feature frame: drop the first p time steps of each series so feature
+ # rows align 1:1 with the design-matrix rows.
+ occ = train_data.groupby("series_id", sort=False).cumcount()
+ feats = train_data[occ >= self.p].reset_index(drop=True)
+ self.features = [c for c in feats.columns if c not in _RESERVED_COLUMNS]
+
+ n_avail = T - self.p
+ if validation:
+ if n_avail <= self.fcst_h:
+ raise ValueError(
+ f"Validation requires more than fcst_h={self.fcst_h} usable time "
+ f"steps per series, but only {n_avail} remain after lagging."
+ )
+ t_train = n_avail - self.fcst_h
+ else:
+ t_train = n_avail
+
+ # Targets in dataset row order (series-major, time within series).
+ label_train = Y[self.p: self.p + t_train].T.ravel()
+ self._Z_train = torch.tensor(
+ design_slice(0, t_train), dtype=self.dtype, device=self.device
+ )
+
+ row_t = feats.groupby("series_id", sort=False).cumcount()
+ X_train = feats[row_t < t_train]
+ dtrain = lgb.Dataset(
+ data=X_train[self.features],
+ label=label_train,
+ free_raw_data=free_raw_data,
+ )
+
+ # Pinned mapping: a key hit proves it is that exact, still-alive
+ # dataset (see utils.DatasetReferences).
+ self.dataset_references = DatasetReferences([(dtrain, "train")])
+
+ if validation:
+ label_eval = Y[self.p + t_train:].T.ravel()
+ self._Z_eval = torch.tensor(
+ design_slice(t_train, n_avail), dtype=self.dtype, device=self.device
+ )
+ X_eval = feats[row_t >= t_train]
+ deval = lgb.Dataset(
+ data=X_eval[self.features],
+ label=label_eval,
+ free_raw_data=free_raw_data,
+ )
+ self.dataset_references = DatasetReferences(
+ [(dtrain, "train"), (deval, "validation")]
+ )
+
+ evals_result = {}
+ callbacks = [lgb.record_evaluation(evals_result)]
+ if early_stopping_round is not None:
+ callbacks.append(
+ lgb.early_stopping(stopping_rounds=early_stopping_round, verbose=False)
+ )
+
+ return [dtrain, deval], ["train", "validation"], callbacks, evals_result
+
+ return [dtrain], ["train"], None, None
+
+ def _compute_fit(self, params: torch.Tensor, Z: torch.Tensor) -> torch.Tensor:
+ """VAR forward pass: per-equation inner product with the design vector.
+
+ Deliberately computed with element-wise ops (broadcast multiply +
+ sum) rather than ``torch.einsum``: einsum lowers to matmul kernels
+ that honor ``torch.set_float32_matmul_precision`` -- which the
+ experiments pipeline sets to ``"medium"`` -- and the reduced-precision
+ backward/double-backward through this op systematically destabilizes
+ HyperTreeNetVAR training (verified ~3x worse forecast errors across
+ seeds on the ausretail benchmark). Element-wise ops always run in
+ full float32, at the cost of materializing the ``(k, T_r, k * p)``
+ product tensor.
+
+ Parameters
+ ----------
+ params : torch.Tensor
+ Coefficient matrix in dataset row order, shape ``(k * T_r, n_params)``.
+ Z : torch.Tensor
+ Design matrix: one shared row per time step ``(T_r, k * p)`` for
+ the full design, or one row per observation ``(k * T_r, 2 * p)``
+ for the factor design.
+
+ Returns
+ -------
+ torch.Tensor
+ Fitted values, shape ``(k, T_r)``.
+ """
+ if self.type == "factor":
+ return (params * Z).sum(dim=1).reshape(self.k, -1)
+
+ P = params.reshape(self.k, -1, self.n_params) # (k, T_r, k*p)
+
+ return (P * Z.unsqueeze(0)).sum(dim=2)
+
+ def _train_core(
+ self,
+ lgb_params: dict,
+ num_iterations: int,
+ train_data: pd.DataFrame,
+ validation: bool,
+ early_stopping_round: Optional[int],
+ seed: int,
+ verbose: int,
+ deterministic: bool,
+ forecast_intervals: Optional[ForecastIntervals],
+ model_factory: Callable[[], object],
+ cal_train_kwargs: Dict,
+ ) -> TrainingResult:
+ """Shared training loop for both VAR variants.
+
+ Validates the inputs, builds the panel datasets, runs the
+ model-specific post-dataset setup (:meth:`_post_datasets_setup`),
+ trains LightGBM with the subclass objective, and optionally
+ calibrates conformal prediction intervals.
+
+ Parameters
+ ----------
+ lgb_params, num_iterations, train_data, validation,
+ early_stopping_round, seed, verbose, deterministic, forecast_intervals
+ Forwarded verbatim from the public ``train()`` methods.
+ model_factory : Callable[[], object]
+ Zero-argument callable returning a fresh, untrained model with
+ this instance's constructor configuration. Used by the conformal
+ calibration to train per-window models.
+ cal_train_kwargs : dict
+ Keyword arguments forwarded to each calibration model's
+ ``train()`` call.
+
+ Returns
+ -------
+ TrainingResult
+ Object containing evaluation results and training information.
+ """
+ self._validate_train_args(
+ lgb_params, num_iterations, train_data, validation,
+ early_stopping_round, seed, verbose, deterministic, forecast_intervals,
+ )
+
+ # Fail fast if any series is too short for the requested conformal
+ # calibration. A VAR(p) needs at least p + 1 rows to retain one
+ # training sample.
+ if forecast_intervals is not None:
+ validate_calibration_length(
+ train_data, self.fcst_h, forecast_intervals, min_train=self.p + 1
+ )
+
+ if deterministic:
+ run_lgb_params = {**lgb_params, "deterministic": True, "force_row_wise": True}
+ else:
+ run_lgb_params = dict(lgb_params)
+
+ self._reset_training_state()
+
+ try:
+ valid_sets, valid_names, callbacks, evals_result = self._build_panel_datasets(
+ train_data, validation, early_stopping_round
+ )
+ self._post_datasets_setup(seed)
+
+ # General model parameters. The objective wrapper stops lgb.train's
+ # params deepcopy from cloning this instance (see NoDeepcopyObjective).
+ self.lgb_params = {
+ "num_class": self._num_class(),
+ "objective": NoDeepcopyObjective(self.objective_fn),
+ "metric": "None",
+ "random_seed": seed,
+ "verbose": verbose,
+ }
+ self.lgb_params.update(run_lgb_params)
+
+ # Anchor the forecast lag state at the end of the training panel.
+ self.set_forecast_origin(train_data)
+
+ start_time = time.time()
+ self.model = lgb.train(
+ self.lgb_params,
+ valid_sets[0],
+ num_boost_round=num_iterations,
+ feval=self.eval_fn if validation else None,
+ valid_sets=valid_sets,
+ valid_names=valid_names,
+ callbacks=callbacks,
+ )
+ training_time = time.time() - start_time
+ self.is_trained = True
+
+ # Calibrate conformal prediction intervals via rolling-window CV.
+ # Fresh model instances are trained per window (no forecast_intervals
+ # passed, so there is no recursion) using the same hyper-parameters.
+ if forecast_intervals is not None:
+ self._cs_scores, self._cs_series_order = rolling_origin_residuals(
+ model_factory=model_factory,
+ train_data=train_data,
+ fcst_h=self.fcst_h,
+ forecast_intervals=forecast_intervals,
+ train_kwargs=cal_train_kwargs,
+ )
+ self._pi_config = forecast_intervals
+ self._is_calibrated = True
+
+ return TrainingResult(
+ train_metrics=evals_result["train"] if validation else {"loss": []},
+ validation_metrics=evals_result["validation"] if validation else None,
+ best_iteration=self.model.best_iteration
+ if self.model.best_iteration > 0 else num_iterations,
+ training_time=training_time,
+ )
+
+ except Exception as e:
+ self.is_trained = False
+ raise RuntimeError(f"Training failed: {str(e)}") from e
+
+ def eval_fn(self, predt: np.ndarray, eval_data: lgb.Dataset) -> Tuple[str, float, bool]:
+ """
+ Custom evaluation function for evaluating forecast accuracy on an evaluation dataset.
+
+ This function computes the loss value to be monitored during evaluation,
+ selecting the design matrix that matches the dataset being evaluated.
+
+ Parameters
+ ----------
+ predt : np.ndarray
+ Raw outputs from LightGBM.
+ eval_data : lgb.Dataset
+ LightGBM dataset containing the evaluation data.
+
+ Returns
+ -------
+ Tuple[str, float, bool]
+ Name of the metric, value of the metric, and whether to maximize it.
+ """
+ # Use the appropriate design matrix based on dataset name
+ dataset_name = self.dataset_references.get(id(eval_data), "unknown")
+ if dataset_name == "validation":
+ Z = self._Z_eval
+ else:
+ # Default to the training design matrix if unknown
+ if dataset_name == "unknown":
+ warnings.warn("Unknown dataset in metric_fn. Using training design matrix.")
+ Z = self._Z_train
+
+ is_higher_better = False # Lower loss is better, so we don't maximize
+ target = torch.tensor(
+ eval_data.get_label().reshape(self.k, -1), dtype=self.dtype, device=self.device
+ )
+ fit = self._fit_from_predt(predt, Z)
+ loss = self.loss_fn(fit, target)
+
+ return self.loss_name, loss.item(), is_higher_better
+
+ # ------------------------------------------------------------------
+ # Forecast plumbing
+ # ------------------------------------------------------------------
+ def set_forecast_origin(self, history: pd.DataFrame) -> None:
+ """Re-anchor the VAR lag state to the end of *history* without retraining.
+
+ Parameters
+ ----------
+ history : pd.DataFrame
+ Aligned panel with ``series_id``, ``date``, ``value`` columns,
+ ordered by ``(series_id, date)`` with contiguous series blocks.
+ Must contain exactly the training series with at least ``p``
+ observations each.
+ """
+ if self.series_order_ is None:
+ raise RuntimeError("set_forecast_origin requires a trained model.")
+ validate_series_order(history, name="history")
+ Y, hist_order, _ = _pivot_panel(history, name="history")
+ if set(hist_order) != set(self.series_order_):
+ raise ValueError(
+ f"history must contain exactly the training series. "
+ f"Missing: {set(self.series_order_) - set(hist_order)}. "
+ f"Extra: {set(hist_order) - set(self.series_order_)}."
+ )
+ if Y.shape[0] < self.p:
+ raise ValueError(
+ f"history must contain at least p={self.p} observations per series."
+ )
+ # Reorder columns to the training series order.
+ idx = [hist_order.index(sid) for sid in self.series_order_]
+ Y = Y[:, idx]
+ # Scale with the training statistics so the lag state lives in the
+ # same (scaled) space as the learned coefficients.
+ Y = (Y - self._scale_loc) / self._scale_scale
+ if self.type == "factor":
+ # Own-lag state (k, p) and factor-lag state (p,), newest first.
+ factor = Y.mean(axis=1)
+ own_state = np.stack([Y[-j] for j in range(1, self.p + 1)], axis=1)
+ factor_state = np.array([factor[-j] for j in range(1, self.p + 1)])
+ self._fcst_state = (own_state, factor_state)
+ else:
+ # Lag state z = [y'_{T-1}, y'_{T-2}, ..., y'_{T-p}] (lag-major).
+ self._fcst_state = np.concatenate([Y[-j] for j in range(1, self.p + 1)])
+
+ def _validate_forecast_args(self, test_data, type, level) -> None:
+ """Common forecast() validation.
+
+ Parameters
+ ----------
+ test_data : pd.DataFrame
+ Forecast input passed to ``forecast()``.
+ type : str
+ Requested output type.
+ level : list of int, optional
+ Requested conformal interval levels.
+ """
+ if not self.is_trained or self.model is None:
+ raise RuntimeError("Model has not been trained. Call train() before forecasting.")
+ for col in ["series_id", "date"]:
+ if col not in test_data.columns:
+ raise ValueError(f"Required column '{col}' not found in test_data")
+ validate_series_order(test_data, name="test_data")
+
+ # Validate series IDs match training data
+ test_series_ids = list(dict.fromkeys(test_data["series_id"].tolist()))
+ missing = set(test_series_ids) - set(self.series_order_)
+ extra = set(self.series_order_) - set(test_series_ids)
+ if missing or extra:
+ parts = []
+ if missing:
+ parts.append(f"Missing series in training: {missing}")
+ if extra:
+ parts.append(f"Extra series not in test_data: {extra}")
+ raise ValueError(
+ ". ".join(parts) + ". A VAR forecast advances all series "
+ "jointly, so test_data must contain exactly the training series."
+ )
+
+ if type not in self._valid_forecast_types:
+ raise ValueError(f"Parameter 'type' must be one of {self._valid_forecast_types}.")
+
+ if type == "forecast":
+ rows_per_series = test_data.groupby("series_id", sort=False).size()
+ bad = rows_per_series[rows_per_series != self.fcst_h]
+ if not bad.empty:
+ raise ValueError(
+ f"Each series must have exactly fcst_h={self.fcst_h} rows in test_data. "
+ f"Series with wrong counts: {bad.to_dict()}"
+ )
+ _validate_aligned_dates(test_data, name="test_data")
+
+ if level is not None:
+ if type != "forecast":
+ raise ValueError("level is only supported with type='forecast'.")
+ if not self._is_calibrated:
+ raise RuntimeError(
+ "Prediction intervals were requested via level, but the model "
+ "was not calibrated. Pass forecast_intervals=ForecastIntervals(...) "
+ "to train() before forecasting with level."
+ )
+ if not isinstance(level, (list, tuple)) or len(level) == 0:
+ raise ValueError("level must be a non-empty list of integers.")
+ for lv in level:
+ if not isinstance(lv, (int, np.integer)) or not 0 < lv < 100:
+ raise ValueError(f"level values must be integers in (0, 100); got {lv}.")
+
+ missing_features = [f for f in self.features if f not in test_data.columns]
+ if missing_features:
+ raise ValueError(f"Missing features in test_data: {missing_features}")
+
+ def _model_name(self) -> str:
+ """Model name identifier, reflecting the design variant.
+
+ Returns
+ -------
+ str
+ ``"
|
-| **Package** | [](https://pypi.org/project/hypertrees-forecasting/) [](https://www.python.org/) |
-| **Downloads** |  |
-| **Paper** | [](https://arxiv.org/pdf/2405.07836) |
-
-
