Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AutoNHITS with prediction_intervals creates second optimization study #1232

Open
willadamskeane opened this issue Dec 18, 2024 · 0 comments
Open
Labels

Comments

@willadamskeane
Copy link

willadamskeane commented Dec 18, 2024

What happened + What you expected to happen

When using AutoNHITS with prediction_intervals, I noticed it creates a new optimization study and reruns the entire hyperparameter search process a second time. Example:

model = AutoNHITS(h=24, num_samples=2, backend="optuna")
nf = NeuralForecast(models=[model], freq="H")
nf.fit(df=data, prediction_intervals=PredictionIntervals(n_windows=2))

The logs show "A new study created in memory" after the initial training completes, and it runs another complete optimization cycle.

Is this intended behavior? I would have expected it to run one hyperparameter optimization to find the best configuration, and then use the best model to generate prediction intervals.

Versions / Dependencies

Python 3.11, Neural Forecast 1.77

Reproduction script

import numpy as np
import pandas as pd
from neuralforecast import NeuralForecast
from neuralforecast.auto import AutoNHITS
from neuralforecast.utils import PredictionIntervals

# Create mock data
np.random.seed(42)
n_samples = 1000
dates = pd.date_range("2020-01-01", periods=n_samples, freq="h")
data = pd.DataFrame(
    {"ds": dates, "unique_id": "A", "y": np.random.normal(0, 1, n_samples)}
)

# Configure model
model = AutoNHITS(
    h=24,  # Forecast horizon
    num_samples=2,  # Number of trials
    backend="optuna",
)

# Create NeuralForecast wrapper
nf = NeuralForecast(models=[model], freq="H")

# Fit model - this will show sampling running twice
print("Starting model fit...")
nf.fit(df=data, prediction_intervals=PredictionIntervals(n_windows=2))
print("Finished model fit")

Issue Severity

Medium: It is a significant difficulty but I can work around it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant