Reduce import-time loads of external modules

[effigies]

Beyond numpy, nibabel loads the following modules on import:

{'PIL',
 'PIL.Image',
 'PIL.ImageMode',
 'PIL._binary',
 'PIL._imaging',
 'PIL._util',
 'PIL.version',
 '__mp_main__',
 '_cython_0_28_2',
 '_cython_0_28_5',
 '_elementtree',
 '_opcode',
 'array',
 'base64',
 'binascii',
 'calendar',
 'cffi',
 'cffi.api',
 'cffi.error',
 'cffi.lock',
 'cffi.model',
 'configparser',
 'ctypes.macholib',
 'ctypes.macholib.dyld',
 'ctypes.macholib.dylib',
 'ctypes.macholib.framework',
 'ctypes.util',
 'dis',
 'distutils',
 'distutils.version',
 'email',
 'email._parseaddr',
 'email._policybase',
 'email.base64mime',
 'email.charset',
 'email.encoders',
 'email.errors',
 'email.feedparser',
 'email.header',
 'email.parser',
 'email.quoprimime',
 'email.utils',
 'getpass',
 'gzip',
 'h5py',
 'h5py._conv',
 'h5py._errors',
 'h5py._hl',
 'h5py._hl.attrs',
 'h5py._hl.base',
 'h5py._hl.compat',
 'h5py._hl.dataset',
 'h5py._hl.datatype',
 'h5py._hl.files',
 'h5py._hl.filters',
 'h5py._hl.group',
 'h5py._hl.selections',
 'h5py._hl.selections2',
 'h5py._objects',
 'h5py._proxy',
 'h5py.defs',
 'h5py.h5',
 'h5py.h5a',
 'h5py.h5ac',
 'h5py.h5d',
 'h5py.h5ds',
 'h5py.h5f',
 'h5py.h5fd',
 'h5py.h5g',
 'h5py.h5i',
 'h5py.h5l',
 'h5py.h5o',
 'h5py.h5p',
 'h5py.h5py_warnings',
 'h5py.h5r',
 'h5py.h5s',
 'h5py.h5t',
 'h5py.h5z',
 'h5py.utils',
 'h5py.version',
 'imp',
 'indexed_gzip',
 'indexed_gzip.indexed_gzip',
 'inspect',
 'mmap',
 'mock',
 'mock.mock',
 'multiprocessing',
 'multiprocessing.context',
 'multiprocessing.process',
 'multiprocessing.reduction',
 'nibabel',
 'nibabel.affines',
 'nibabel.analyze',
 'nibabel.arrayproxy',
 'nibabel.arraywriters',
 'nibabel.batteryrunners',
 'nibabel.brikhead',
 'nibabel.casting',
 'nibabel.cifti2',
 'nibabel.cifti2.cifti2',
 'nibabel.cifti2.parse_cifti2',
 'nibabel.dataobj_images',
 'nibabel.deprecated',
 'nibabel.deprecator',
 'nibabel.ecat',
 'nibabel.eulerangles',
 'nibabel.externals',
 'nibabel.externals.netcdf',
 'nibabel.externals.oset',
 'nibabel.filebasedimages',
 'nibabel.fileholders',
 'nibabel.filename_parser',
 'nibabel.fileslice',
 'nibabel.freesurfer',
 'nibabel.freesurfer.io',
 'nibabel.freesurfer.mghformat',
 'nibabel.funcs',
 'nibabel.gifti',
 'nibabel.gifti.gifti',
 'nibabel.gifti.giftiio',
 'nibabel.gifti.parse_gifti_fast',
 'nibabel.gifti.util',
 'nibabel.imageclasses',
 'nibabel.imageglobals',
 'nibabel.info',
 'nibabel.keywordonly',
 'nibabel.loadsave',
 'nibabel.minc1',
 'nibabel.minc2',
 'nibabel.mriutils',
 'nibabel.nifti1',
 'nibabel.nifti2',
 'nibabel.openers',
 'nibabel.optpkg',
 'nibabel.orientations',
 'nibabel.parrec',
 'nibabel.pkg_info',
 'nibabel.py3k',
 'nibabel.pydicom_compat',
 'nibabel.quaternions',
 'nibabel.spatialimages',
 'nibabel.spm2analyze',
 'nibabel.spm99analyze',
 'nibabel.streamlines',
 'nibabel.streamlines.array_sequence',
 'nibabel.streamlines.header',
 'nibabel.streamlines.tck',
 'nibabel.streamlines.tractogram',
 'nibabel.streamlines.tractogram_file',
 'nibabel.streamlines.trk',
 'nibabel.streamlines.utils',
 'nibabel.testing',
 'nibabel.testing.np_features',
 'nibabel.trackvis',
 'nibabel.tripwire',
 'nibabel.viewers',
 'nibabel.volumeutils',
 'nibabel.wrapstruct',
 'nibabel.xmlutils',
 'nose',
 'nose.case',
 'nose.config',
 'nose.core',
 'nose.exc',
 'nose.failure',
 'nose.importer',
 'nose.loader',
 'nose.plugins',
 'nose.plugins.base',
 'nose.plugins.deprecated',
 'nose.plugins.errorclass',
 'nose.plugins.manager',
 'nose.plugins.plugintest',
 'nose.plugins.skip',
 'nose.proxy',
 'nose.pyversion',
 'nose.result',
 'nose.selector',
 'nose.suite',
 'nose.tools',
 'nose.tools.nontrivial',
 'nose.tools.trivial',
 'nose.util',
 'numpy.testing.decorators',
 'opcode',
 'optparse',
 'pbr',
 'pbr.version',
 'pkg_resources',
 'pkg_resources._vendor',
 'pkg_resources._vendor.packaging.__about__',
 'pkg_resources._vendor.six',
 'pkg_resources._vendor.six.moves',
 'pkg_resources.extern',
 'pkg_resources.extern.appdirs',
 'pkg_resources.extern.packaging',
 'pkg_resources.extern.packaging._compat',
 'pkg_resources.extern.packaging._structures',
 'pkg_resources.extern.packaging.markers',
 'pkg_resources.extern.packaging.requirements',
 'pkg_resources.extern.packaging.specifiers',
 'pkg_resources.extern.packaging.version',
 'pkg_resources.extern.pyparsing',
 'pkg_resources.extern.six',
 'pkg_resources.extern.six.moves',
 'pkg_resources.extern.six.moves.urllib',
 'pkg_resources.py31compat',
 'pkgutil',
 'platform',
 'plistlib',
 'pydicom',
 'pydicom._dicom_dict',
 'pydicom._private_dict',
 'pydicom._uid_dict',
 'pydicom._version',
 'pydicom.charset',
 'pydicom.compat',
 'pydicom.config',
 'pydicom.datadict',
 'pydicom.dataelem',
 'pydicom.dataset',
 'pydicom.dicomdir',
 'pydicom.dicomio',
 'pydicom.encaps',
 'pydicom.errors',
 'pydicom.filebase',
 'pydicom.filereader',
 'pydicom.fileutil',
 'pydicom.filewriter',
 'pydicom.misc',
 'pydicom.multival',
 'pydicom.pixel_data_handlers',
 'pydicom.pixel_data_handlers.numpy_handler',
 'pydicom.pixel_data_handlers.pillow_handler',
 'pydicom.pixel_data_handlers.rle_handler',
 'pydicom.sequence',
 'pydicom.tag',
 'pydicom.uid',
 'pydicom.util',
 'pydicom.util.fixes',
 'pydicom.util.hexutil',
 'pydicom.valuerep',
 'pydicom.values',
 'pyexpat',
 'pyexpat.errors',
 'pyexpat.model',
 'quopri',
 'scipy',
 'scipy.__config__',
 'scipy._distributor_init',
 'scipy._lib',
 'scipy._lib._ccallback',
 'scipy._lib._ccallback_c',
 'scipy._lib._testutils',
 'scipy._lib._version',
 'scipy._lib.six',
 'scipy.version',
 'six',
 'six.moves',
 'termios',
 'uuid',
 'xml',
 'xml.etree',
 'xml.etree.ElementPath',
 'xml.etree.ElementTree',
 'xml.parsers',
 'xml.parsers.expat',
 'xml.parsers.expat.errors',
 'xml.parsers.expat.model',
 'zipfile'}

At minimum, I would say nibabel.testing does not need to be loaded when I run import nibabel, but I also think PIL, h5py, pydicom and XML probably don't need to be loaded unless you use a file format that requires them.

Particularly given nibabel's position as a basic requirement for practically the entire Python neuroimaging ecosystem, we should try to be a lightweight import.

[matthew-brett]

Sure - but - how much time would we be saving, do you think?

[effigies]

I frequently get about a 1s import for nibabel. Which isn't too bad in itself, but as I'm trying to pare down import times for libraries with more dependencies, nibabel is one of several that add noticeably to the load time, and typically the hardest to isolate.

I wrote a profiler to time successive imports and count the number of modules and virtual memory allocations after each successive import. Here it shows about a 450ms load time (after loading sys, importlib, psutil and time), but fully half of that is nose:

numpy: 136 modules; 93.05ms, 115.6MiB
PIL: 2 modules; 0.65ms, 0.0MiB
xml: 8 modules; 2.77ms, 0.5MiB
h5py: 46 modules; 49.15ms, 7.2MiB
nose: 79 modules; 234.92ms, 5.6MiB
pydicom: 42 modules; 39.98ms, 7.2MiB
nibabel: 94 modules; 35.96ms, 2.4MiB
Final: 407 modules; 456.49ms, 138.6MiB

[matthew-brett]

I think it's a good idea to avoid import nibabel.testing - maybe we can avoid h5py and pydicom easily enough?

[effigies]

I think it shouldn't be too much trouble. I'll have a look maybe tomorrow. And I think it's probably fine to ignore PIL and xml, looking at those numbers.