Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training fails when more than one Kepler is running #519

Open
5 tasks
vprashar2929 opened this issue Oct 22, 2024 · 0 comments
Open
5 tasks

Training fails when more than one Kepler is running #519

vprashar2929 opened this issue Oct 22, 2024 · 0 comments
Labels
kind/bug Something isn't working

Comments

@vprashar2929
Copy link
Contributor

What happened?

When more than one kepler is running locally and training is triggered it fails to train the model with below errors:

failed to load json /home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/util/../models/stress-ng/node_type_index.json: [Errno 2] No such file or directory: '/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/util/../models/stress-ng/node_type_index.json'
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
valid feature group:  [<FeatureGroup.CounterOnly: 3>, <FeatureGroup.BPFOnly: 4>, <FeatureGroup.BPFIRQ: 8>, <FeatureGroup.CounterIRQCombined: 6>, <FeatureGroup.Basic: 7>]
stress-ng pipeline: CounterOnly extraction done.
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
  if (arr.astype(int) == arr).all():
stress-ng pipeline: CounterOnly isolation done.
stress-ng pipeline: Replace Node Type: 0
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
stress-ng pipeline: Waiting for 20 trainers to complete...
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
stress-ng pipeline: 20/100 trainers are trained from CounterOnly to rapl-sysfs
Traceback (most recent call last):
  File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/bin/kepler-model", line 8, in <module>
    sys.exit(run())
  File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/cmd/main.py", line 1059, in run
    getattr(sys.modules[__name__], args.command)(args)
  File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/cmd/main.py", line 491, in train
    success, abs_data, dyn_data = pipeline.process_multiple_query(
  File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/train/pipeline.py", line 156, in process_multiple_query
    self.print_pipeline_process_end(energy_source, feature_group, abs_data, dyn_data)
  File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/train/pipeline.py", line 176, in print_pipeline_process_end
    node_types = pd.unique(abs_metadata_df[node_info_column])
  File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/frame.py", line 4102, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/indexes/range.py", line 417, in get_loc
    raise KeyError(key)
KeyError: 'node_type'

What did you expect to happen?

Training should work fine even if more than one Kepler is running on the system

How can we reproduce it (as minimally and precisely as possible)?

  • Setup Kepler using either compose or using k8s. With compose you will get two kepler being deployed by default
  • Gather the query data
  • Run the trainer locally

Anything else we need to know?

No response

Kepler image tag

Deployment

  • Model server
  • Estimator
  • Online trainer
  • Offline trainer
  • Profiler

Kepler model server image tag if deployed

Kepler estimator image tag if deployed

Kepler online trainer image tag if deployed

Kepler offline trainer image tag if deployed

Kepler profiler image tag if deployed

Kubernetes version

$ kubectl version
# paste output here

Install tools

Kepler deployment config

For on kubernetes:

$ KEPLER_NAMESPACE=kepler

# provide kepler configmap
$ kubectl get configmap kepler-cfm -n ${KEPLER_NAMESPACE}
# paste output here
# provide kepler model server configmap if Kepler Model Server is deployed
$ kubectl get configmap kepler-model-server-cfm -n ${KEPLER_NAMESPACE}
# paste output here
# provide kepler deployment description
$ kubectl describe deployment kepler-exporter -n ${KEPLER_NAMESPACE}

For standalone:

put your Kepler command argument here

@vprashar2929 vprashar2929 added the kind/bug Something isn't working label Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant