You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When more than one kepler is running locally and training is triggered it fails to train the model with below errors:
failed to load json /home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/util/../models/stress-ng/node_type_index.json: [Errno 2] No such file or directory: '/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/util/../models/stress-ng/node_type_index.json'
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
valid feature group: [<FeatureGroup.CounterOnly: 3>, <FeatureGroup.BPFOnly: 4>, <FeatureGroup.BPFIRQ: 8>, <FeatureGroup.CounterIRQCombined: 6>, <FeatureGroup.Basic: 7>]
stress-ng pipeline: CounterOnly extraction done.
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1056: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/dtypes/cast.py:1080: RuntimeWarning: invalid value encountered in cast
if (arr.astype(int) == arr).all():
stress-ng pipeline: CounterOnly isolation done.
stress-ng pipeline: Replace Node Type: 0
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
PolynomialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
stress-ng pipeline: Waiting for 20 trainers to complete...
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LinearRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
KNeighborsRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
GradientBoostingRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (package)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (core)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
ExponentialRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
LogarithmicRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LogisticRegressionTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
KNeighborsRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
XgboostFitTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SGDRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LogisticRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
PolynomialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
SVRRegressorTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LinearRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
SGDRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (uncore)
ExponentialRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
SVRRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
LogarithmicRegressionTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
GradientBoostingRegressorTrainer trainer (Dyn/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): Newly initialize model (dram)
XgboostFitTrainer trainer (Abs/FeatureGroup.CounterOnly/rapl-sysfs): fit scaler to latest data CounterOnly for node_type=0
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
Exception occurred: Input X contains infinity or a value too large for dtype('float64').
stress-ng pipeline: 20/100 trainers are trained from CounterOnly to rapl-sysfs
Traceback (most recent call last):
File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/bin/kepler-model", line 8, in <module>
sys.exit(run())
File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/cmd/main.py", line 1059, in run
getattr(sys.modules[__name__], args.command)(args)
File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/cmd/main.py", line 491, in train
success, abs_data, dyn_data = pipeline.process_multiple_query(
File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/train/pipeline.py", line 156, in process_multiple_query
self.print_pipeline_process_end(energy_source, feature_group, abs_data, dyn_data)
File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/kepler_model/train/pipeline.py", line 176, in print_pipeline_process_end
node_types = pd.unique(abs_metadata_df[node_info_column])
File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/frame.py", line 4102, in __getitem__
indexer = self.columns.get_loc(key)
File "/home/fserver/.local/share/hatch/env/virtual/kepler-model/_v8KwWUN/kepler-model/lib/python3.10/site-packages/pandas/core/indexes/range.py", line 417, in get_loc
raise KeyError(key)
KeyError: 'node_type'
What did you expect to happen?
Training should work fine even if more than one Kepler is running on the system
How can we reproduce it (as minimally and precisely as possible)?
Setup Kepler using either compose or using k8s. With compose you will get two kepler being deployed by default
Gather the query data
Run the trainer locally
Anything else we need to know?
No response
Kepler image tag
Deployment
Model server
Estimator
Online trainer
Offline trainer
Profiler
Kepler model server image tag if deployed
Kepler estimator image tag if deployed
Kepler online trainer image tag if deployed
Kepler offline trainer image tag if deployed
Kepler profiler image tag if deployed
Kubernetes version
$ kubectl version
# paste output here
Install tools
Kepler deployment config
For on kubernetes:
$ KEPLER_NAMESPACE=kepler
# provide kepler configmap
$ kubectl get configmap kepler-cfm -n ${KEPLER_NAMESPACE}
# paste output here
# provide kepler model server configmap if Kepler Model Server is deployed
$ kubectl get configmap kepler-model-server-cfm -n ${KEPLER_NAMESPACE}
# paste output here
# provide kepler deployment description
$ kubectl describe deployment kepler-exporter -n ${KEPLER_NAMESPACE}
For standalone:
put your Kepler command argument here
The text was updated successfully, but these errors were encountered:
What happened?
When more than one kepler is running locally and training is triggered it fails to train the model with below errors:
What did you expect to happen?
Training should work fine even if more than one Kepler is running on the system
How can we reproduce it (as minimally and precisely as possible)?
Anything else we need to know?
No response
Kepler image tag
Deployment
Kepler model server image tag if deployed
Kepler estimator image tag if deployed
Kepler online trainer image tag if deployed
Kepler offline trainer image tag if deployed
Kepler profiler image tag if deployed
Kubernetes version
Install tools
Kepler deployment config
For on kubernetes:
For standalone:
put your Kepler command argument here
The text was updated successfully, but these errors were encountered: