Saltclass performance

[max-arnold]

Description of Issue/Question

As noted in the #42349 (comment), the performance is significantly degraded

Setup

Amount of reclass files:

(salt-2018) user@otg ~/work/salt % find reclass/classes -type f | wc -l
     302
(salt-2018) user@otg ~/work/salt % find reclass/nodes -type f | wc -l
      74

Steps to Reproduce Issue

Salt 2017.7.0, reclass:

time salt-call --id example.com pillar.data > /dev/null
salt-call --id example.com pillar.data > /dev/null  2,09s user 0,74s system 84% cpu 3,349 total

Salt 2018.3.1, saltclass

time salt-call --id example.com pillar.data > /dev/null
salt-call --id example.com pillar.data > /dev/null  3,85s user 1,63s system 89% cpu 6,136 total

Almost 50% of the time is spent in get_class doing this over and over for each class

for root, dirs, files in salt.utils.path.os_walk(os.path.join(saltclass_path, 'classes')):
    for l_file in files:
        l_files.append(os.path.join(root, l_file))

By applying dirty hack (not suitable for PR) I was able to cut run time from 6,136 to 3,456 seconds:

How to get these graphs:

python -m cProfile -o salt.profile ~/.virtualenvs/salt-2018/bin/salt-call --id example.com pillar.data
pyprof2calltree -i salt.profile -o callgrind.calltree

Then use qcachegrind or kcachegrind to open callgrind.calltree

Versions Report

salt --versions-report

Salt Version: Salt: 2018.3.1

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.15 (default, May 2 2018, 00:53:27)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 17.0.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.6

System Versions:
dist:
locale: UTF-8
machine: x86_64
release: 17.6.0
system: Darwin
version: 10.13.5 x86_64

[Ch3LL]

ping @olivier-mauras mind taking a look here?

[olivier-mauras]

Yep got benchmarks and PR coming up let me clean it all up

[ketzacoatl]

great to see the improvements, definitely interested in using saltclass when its ready for action!

[twangboy]

Closing this issue due to age and lack of activity. Please test this on version 3006.2 and create a new issue if the problem persists. The new issue template has more information and will allow us to track and reproduce the issue more effectively. Thanks!