Not sure if RandomizedsearchCV() is using loky

[myilmazhaubtech]

Hello I was training my classifier using XGBoost with the following code on GCP and eventually I built my own server and now I am having difficulty. On GCP I used to see the message 'backend: using loky backend ...' but not on my own server which makes me think if this code is really taking advantage of parallelism. The server I built is far superior to what I was using on GCP. Below is some information about my code and the specs of the server. I wonder is there is something wrong with my environment.

If I use n_jobs=-1 inside the RandomizedSearchCV() code then I start getting the error below. However if I use n_jobs=-1 inside XGBClassifier() then things work albeit it is slow which makes me think that I am not taking advantage of parallelism. I also do not see the 'Backend: using loky backend ...' message either.

  warnings.warn("Estimator fit failed. The score on this train-test"
/home/michael/.local/lib/python3.8/site-packages/sklearn/model_selection/_validation.py:610: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details: 
Traceback (most recent call last):
  File "/home/michael/.local/lib/python3.8/site-packages/sklearn/model_selection/_validation.py", line 593, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/home/michael/.local/lib/python3.8/site-packages/xgboost/core.py", line 422, in inner_f
    return f(**kwargs)
  File "/home/michael/.local/lib/python3.8/site-packages/xgboost/sklearn.py", line 913, in fit
    self._Booster = train(xgb_options, train_dmatrix,
  File "/home/michael/.local/lib/python3.8/site-packages/xgboost/training.py", line 206, in train
    bst = _train_internal(params, dtrain,
  File "/home/michael/.local/lib/python3.8/site-packages/xgboost/training.py", line 101, in _train_internal
    bst.update(dtrain, i, obj)
  File "/home/michael/.local/lib/python3.8/site-packages/xgboost/core.py", line 1280, in update
    _check_call(_LIB.XGBoosterUpdateOneIter(self.handle,
  File "/home/michael/.local/lib/python3.8/site-packages/xgboost/core.py", line 189, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [17:24:00] ../src/c_api/../data/../common/common.h:44: ../src/c_api/../data/../common/device_helpers.cuh: 128: out of memory
Stack trace:
  [bt] (0) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x8d264) [0x7fccd8733264]
  [bt] (1) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x2fc730) [0x7fccd89a2730]
  [bt] (2) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x2fcde9) [0x7fccd89a2de9]
  [bt] (3) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x32da14) [0x7fccd89d3a14]
  [bt] (4) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x339bc5) [0x7fccd89dfbc5]
  [bt] (5) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x33aa2b) [0x7fccd89e0a2b]
  [bt] (6) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x431dc3) [0x7fccd8ad7dc3]
  [bt] (7) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x4384bd) [0x7fccd8ade4bd]
  [bt] (8) /home/michael/.local/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x17255b) [0x7fccd881855b]

score='precision_weighted'
# score = true_edge_scorer
param_comb = 1 #number of combination to be tried
params = {
        'min_child_weight': [1,3, 5,7, 10, 20, 40],
        'gamma':[0, 0.25, 0.5, 1.0], 
        'subsample': [0.6,0.8, 1.0],
        'colsample_bytree': [0.3,0.4,0.6,0.8], 
        'max_depth':[3,4,6,8] , #'max_depth': [2], 
        'eta':[ 0.001,0.01,0.1] , 
        'n_estimators': [50,70, 100, 200], #'n_estimators': [20],
        'reg_alpha': [0.1, 1.0, 5.0, 10.0, 50.0, 100.0],
        'reg_lambda': [0.1, 1.0, 5.0, 10.0, 50.0, 100.0],
        'tree_method' : ["gpu_hist"]
       }

#params={'subsample': [1.0], 'min_child_weight': [5], 'max_depth': [2], 'gamma': [0.5], 'eta': [0.01], 'colsample_bytree': [0.8],'n_estimators':[100]}
model = XGBClassifier(objective='multi:softmax',n_jobs=-1, num_class=3, eval_metric='mlogloss') #base model

    random_search = RandomizedSearchCV(model, param_distributions=params,n_iter=param_comb,return_train_score=True, scoring=score,cv=ps,verbose=3, random_state=1001 ) 
    random_search.fit(Train,Train_Labels)         

MY HARDWARE:

• AMD Ryzen ThreadRipper Professional Video Editing Custom PC System
• Motherboard: Asus ROG ZENITH II EXTREME Socket sTRX4/ AMD TRX40/ DDR4/ 3-Way CrossFireX & 3-Way SLI/ SATA3&USB3.2/ M.2/ WiFi/ Bluetooth
• Processor: AMD Threadripper 3990X 64-Core 2.9GHz up to 4.3GHz
• Memory: 128GB DDR4-2400/2666 PC4-19200/21300
• Video card: NVIDIA TITAN RTX 24GB GDDR6 PCI-Express

michael@haub-ubuntu:~$ pip3 list
Package                 Version             
----------------------- --------------------
altair                  4.1.0               
anyio                   2.0.2               
apturl                  0.5.2               
argon2-cffi             20.1.0              
async-generator         1.10                
attrs                   20.3.0              
Babel                   2.9.0               
backcall                0.2.0               
bcrypt                  3.1.7               
bleach                  3.2.1               
blinker                 1.4                 
Brlapi                  0.7.0               
certifi                 2019.11.28          
cffi                    1.14.4              
chardet                 3.0.4               
chrome-gnome-shell      0.0.0               
Click                   7.0                 
cloudpickle             1.6.0               
colorama                0.4.3               
command-not-found       0.3                 
cryptography            2.8                 
cupshelpers             1.0                 
cx-Freeze               6.1                 
cycler                  0.10.0              
dbus-python             1.2.16              
decorator               4.4.2               
defer                   1.0.6               
defusedxml              0.6.0               
distro                  1.4.0               
distro-info             0.23ubuntu1         
duplicity               0.8.12.0            
entrypoints             0.3                 
fasteners               0.14.1              
future                  0.18.2              
httplib2                0.14.0              
idna                    2.8                 
ipykernel               5.4.3               
ipynb-py-convert        0.4.6               
ipython                 7.19.0              
ipython-genutils        0.2.0               
jedi                    0.17.2              
Jinja2                  2.11.2              
joblib                  1.0.0               
json5                   0.9.5               
jsonschema              3.2.0               
jupyter-client          6.1.11              
jupyter-core            4.7.0               
jupyter-server          1.2.1               
jupyterlab              3.0.5               
jupyterlab-pygments     0.1.2               
jupyterlab-server       2.1.2               
keyring                 18.0.1              
kiwisolver              1.3.1               
language-selector       0.1                 
launchpadlib            1.10.13             
lazr.restfulclient      0.14.2              
lazr.uri                1.0.3               
lockfile                0.12.2              
loky                    2.9.0               
louis                   3.12.0              
macaroonbakery          1.3.1               
Mako                    1.1.0               
MarkupSafe              1.1.0               
matplotlib              3.3.3               
mistune                 0.8.4               
monotonic               1.5                 
nbclassic               0.2.6               
nbclient                0.5.1               
nbconvert               6.0.7               
nbformat                5.1.0               
nest-asyncio            1.4.3               
netifaces               0.10.4              
notebook                6.2.0               
numpy                   1.19.5              
oauthlib                3.1.0               
olefile                 0.46                
packaging               20.8                
pandas                  1.2.0               
pandas-ta               0.2.28b0            
pandocfilters           1.4.3               
paramiko                2.6.0               
parso                   0.7.1               
pexpect                 4.6.0               
pickleshare             0.7.5               
Pillow                  7.0.0               
pip                     20.0.2              
prometheus-client       0.9.0               
prompt-toolkit          3.0.10              
protobuf                3.6.1               
ptyprocess              0.7.0               
pyaml                   20.4.0              
pycairo                 1.16.2              
pycparser               2.20                
pycups                  1.9.73              
Pygments                2.7.4               
PyGObject               3.36.0              
PyJWT                   1.7.1               
pymacaroons             0.13.0              
PyNaCl                  1.3.0               
pyparsing               2.4.7               
pyRFC3339               1.1                 
pyrsistent              0.17.3              
python-apt              2.0.0+ubuntu0.20.4.3
python-dateutil         2.7.3               
python-debian           0.1.36ubuntu1       
pytz                    2019.3              
pyxdg                   0.26                
PyYAML                  5.3.1               
pyzmq                   20.0.0              
reportlab               3.5.34              
requests                2.22.0              
requests-unixsocket     0.2.0               
scikit-learn            0.24.0              
scipy                   1.6.0               
screen-resolution-extra 0.0.0               
seaborn                 0.11.1              
SecretStorage           2.3.1               
Send2Trash              1.5.0               
setuptools              45.2.0              
simplejson              3.16.0              
six                     1.14.0              
sklearn                 0.0                 
sniffio                 1.2.0               
ssh-import-id           5.10                
systemd-python          234                 
TA-Lib                  0.4.19              
terminado               0.9.2               
testpath                0.4.4               
threadpoolctl           2.1.0               
toolz                   0.11.1              
tornado                 6.1                 
traitlets               5.0.5               
ubuntu-advantage-tools  20.3                
ubuntu-drivers-common   0.0.0               
ufw                     0.36                
unattended-upgrades     0.1                 
urllib3                 1.25.8              
usb-creator             0.3.7               
wadllib                 1.3.3               
wcwidth                 0.2.5               
webencodings            0.5.1               
wheel                   0.34.2              
xgboost                 1.3.1               
xkit                    0.0.0               
zoom                    0.0.15