WIP: Sklearn kdtree #395

kain88-de · 2015-08-16T16:36:21Z

Fixes #383

I have completely removed the old KDTree library. Functions that have used the old class
directly have been adopted to use the scikit-learn implementation. The AtomNeighborSearch class
was a nice abstraction that worked directly with AtomGroups, it is still available but internally it
uses the new KDTree implementation. I tried to make the commits self contained which should make it easier to review this.

The tests have been updated for the new AtomNeighborSearch API.

TODO:

write Doc strings for AtomNeighborSearch
add removal of NeighborSearch to changelog
deprecate KDTree completely with notice to use scikit-learn instead
fix deprecation test failure

kain88-de · 2015-08-17T07:12:07Z

Btw what are is the purpose of the depcreation test that is failing? Should it test that depcreated imports are still available? In that case I would remove that specific test because I have removed the module. AtomNeighborSearch now lives in it's own module, where I'm thinking if I should move it to the analysis sub-module since it is only used there.

richardjgowers · 2015-08-19T12:14:35Z

I think the failing deprecation test is that KDTree used to live in MDAnalysis.KDTree and we moved it to MDAnalysis.lib.KDTree, but we want the old address to work (for now), and issue a deprecation warning. So there's a stub module which issues the warning then imports the other module.

kain88-de · 2015-08-19T12:24:24Z

I guess then I could readd the stub module and load the new atomneighbors instead and issue a
warning about the move and rename. I think I will also move the atomneighbors module into MDAnalysis.analysis since it is only used there in MDAnalysis and I figure it is something people want to use it in their analysis.

richardjgowers · 2015-08-19T12:29:58Z

The submodule .analysis is complete tools which can have MDA dependencies,
whilst .lib is building blocks without MDA dependencies. I need to properly
catch up on this to figure out where this should live.

On 1:24PM, Wed, 19 Aug 2015 kain88-de [email protected] wrote:

I guess then I could readd the stub module and load the new atomneighbors
instead and issue a
warning about the move and rename. I think I will also move the
atomneighbors module into MDAnalysis.analysis since it is only used there
in MDAnalysis and I figure it is something people want to use it in their
analysis.

—
Reply to this email directly or view it on GitHub
#395 (comment)
.

kain88-de · 2015-08-19T19:17:01Z

Once you found out where it should live I will finish my TODO-list to have this mergeable.

Btw it is ok that I removed the NeighborSearch class? I didn't see any particular value for a
KDTree wrapper in MDAnalysis and I figured if people want to use a KDTree on arbitrary coordinates
it might be better if they use one of the common ML-python packages

We switch from the old copied and here maintained Biopython version of KDTree to the scikit-learn. For one this will result in a speed increase and secondly it will allow us to remove the old python 3 incompatible SWIG module. Plus less code to test and maintain. For spherical selections the tests fail right now because they are not ported yet.

Continue to get rid of the old KDTree module

This removes all appearences of the KDTree Module and it's Neighbor classes as well. I will reimplement the AtomNeighbor class again later.

This is a refactored implementation of this class based on using the sklearn KDTree. The search_all method is currently not implemented. The tests aren't currently passing that will be fixed in later commits.

PEP8-tify the functions I touched as well.

This way the KDTree removal doesn't require any new dependencies.

Explain that the KDTRee has been removed and that this now only loads the NeighborSearch package.

kain88-de · 2015-08-23T20:35:07Z

Tests are all passing now and I fixed the doc string for the atomneihgborsearch class. The I didn't test the density analysis because a unit-test for it doesn't exists.

Anything else missing?

richardjgowers · 2015-08-24T11:21:30Z

package/MDAnalysis/lib/NeighborSearch.py

@@ -0,0 +1,75 @@
+import numpy


This file will need a header comment for the rst docs won't it?

I added a header comment. I'm not very familiar with the rst docs. If I missed defining some rst specific stuff please tell me. I would also appreciate a link to the appropriate sphinx documentation for this.

[ci skip]

richardjgowers · 2015-08-24T13:32:31Z

Ok header looks fine. What needs doing now is for the .rst stubs to be updated. If you go to doc/sphinx/source/documentation_pages/ there is a lot of .rst files that mirror the layout of the package. You need to update these (KDTree and lib) so that they reflect the new structure. I think this will be, deleting the KDTree directory, add a new .rst for NeighborSearch (copy another submodule, should just be 2 lines) and add a link to lib_modules.rst

Link to links:
https://github.com/MDAnalysis/mdanalysis/wiki/WritingDocumentation#online-docs-format-sphinx-restructuredtext

I also added a page for the new NeighborSearch module

kain88-de · 2015-08-24T20:50:37Z

I removed the old KDTree rst files and added a short one for the new NeighborSearch module. The formatting is different then from the other include in lib_modules.rst. But I have no idea why.

Plus I suddenly have all these import error where I have no real idea what might have caused them.

This is an example for a failed import.
https://travis-ci.org/MDAnalysis/mdanalysis/builds/77053307#L459

I assume loading NeighborSearch in lib/__init__.py somehow has caused circular imports with AtomGroup. Removing that import fixes the import-errors I'm seeing. @richardjgowers
Do you have any idea how to fix this properly? Currently I can only think of using

from ..exceptions import SelectionError, .....

in AtomgGourp.py

The problem here is that an import statement actually EXECUTES all the code that is defined inside of the modules __init__.py. The problem now is that the lib module was loaded before the execptions were loaded, note that lib/__init__.py now loads `NeighborSearch.py` which in return loads `AtomGroup.py` which loads `SelectionError` which is defined later int the top `__init__.py`. This means we have a circular import and everything just stops working. By importing the exeptions first everything will work fine.

kain88-de · 2015-08-24T21:12:04Z

Plus I suddenly have all these import error where I have no real idea what might have caused them

Yup it was a circular import see 80078fe.

richardjgowers · 2015-08-24T22:13:33Z

Awesome

WIP: Sklearn kdtree

- changed "Maintained by Oliver Beckstein and Elizabeth Denning" to "Maintained by the MDAnalysis Core Devs Team" - added Wouter Boomsma (for PR #976) - removed explicit mentions for KDTree (Hamelryck) because since 0.11.0 we do no longer include the actual code (see PR #395 and #383) - removed explicit mentions for helanal (Hall) because Ben Hall is a contributing author

JmhX7IYP+2pnu4iv7mzp+xy8W5x1pzFOZhzfS4LslB2Ac+4SENM1eAZebiZQYSYl qyYkj9bLS2RqC40dQVfAA4dMrAuWpNfgZkTeCCU7pm29TodN0hmljNG/rLoGG/iH ieA+pSultKwyryY0UZXHAMmNRIViEfy1VsjE2Hswc/AIiY9W1yfLONXVjtELjFsU jv4u1ljR+lhuP6ZllEwucKeD5ojQDyXBU9WXPi2bUwFEFpNdn5zXVRTV/kntb86d sbOE1TQpy+WeSNTyqmEtdjQRuLH21Jm9kTnWFjkKjw2lL+odUOAoOEC4tLSin30i 6KK2lVvSfXJi9e5XOL+nXWqvwr+jU5jqNV+PC60aEe9N1xhVapYroY84bIJwcuYQ 2Axu0vJWNLOpnLGAH7kYYaeEFJdsZt283kDy1paagWmXqyy3p3hh1VD8NEWWcXQ4 Da07VzZh61RvLHFWIzyXkizPjaCMCfCPIpwRACht5OJdRH0LwEhGRhJhSHEUdh7p WGNEQr5XVnd2Xb+dHfpBeq6a0vPmOv8oDXzRCnGpDeTjGE3kEQy8PGeDqm4dsD+B wOmeCYN/NmUnurpZrW2x151bKgCbcjwiTksCsIhZmN1kRHqAR+NkoHSUOiji6INw LvpyU91X7ji1yyYlhPTu =N0P/ -----END PGP SIGNATURE----- updated AUTHORS - changed "Maintained by Oliver Beckstein and Elizabeth Denning" to "Maintained by the MDAnalysis Core Devs Team" - added Wouter Boomsma (for PR MDAnalysis#976) - removed explicit mentions for KDTree (Hamelryck) because since 0.11.0 we do no longer include the actual code (see PR MDAnalysis#395 and MDAnalysis#383) - removed explicit mentions for helanal (Hall) because Ben Hall is a contributing author

kain88-de mentioned this pull request Aug 20, 2015

Replace KDTree library with alternative #383

Closed

kain88-de added 12 commits August 23, 2015 20:12

Port Sphericallayer selection for new KDTree

ad3b14e

Continue to get rid of the old KDTree module

port sphericalzoneselection for new KDTree

33fd533

Remove KDTree module

d22da53

This removes all appearences of the KDTree Module and it's Neighbor classes as well. I will reimplement the AtomNeighbor class again later.

Re add AtomNeighborsearch class

5291c97

This is a refactored implementation of this class based on using the sklearn KDTree. The search_all method is currently not implemented. The tests aren't currently passing that will be fixed in later commits.

Use new AtomNeighborSearch in hbond_analysis

91a1789

PEP8-tify the functions I touched as well.

Use new AtomNeighborSearch in pdbqt test

f6e1306

Switch to BioPython KDTree in AtomNeighborSearch

885fa28

Use Bio.KDTree for selections

7ed58b8

This way the KDTree removal doesn't require any new dependencies.

Add MDAnalysis.KDTree again

ed81f17

Explain that the KDTRee has been removed and that this now only loads the NeighborSearch package.

Add doc-string for AtomNeighborSearch

975b4b2

Add KDTree removal to CHANGELOG

318c20a

kain88-de force-pushed the sklearn-kdtree branch from 0ecfb56 to 318c20a Compare August 23, 2015 18:15

Wrong import in test_pdbqt.py

e93f249

richardjgowers reviewed Aug 24, 2015
View reviewed changes

richardjgowers self-assigned this Aug 24, 2015

richardjgowers added this to the 0.11 milestone Aug 24, 2015

kain88-de added 2 commits August 24, 2015 14:11

Add NeighborSearch to lib init

8fdd717

[ci skip]

Add more documentation for NeighborSearch

c3f375d

[ci skip]

Update sphinx documentation that KDTree is removed

3023fa9

I also added a page for the new NeighborSearch module

richardjgowers added a commit that referenced this pull request Aug 24, 2015

Merge pull request #395 from kain88-de/sklearn-kdtree

248b489

WIP: Sklearn kdtree

richardjgowers merged commit 248b489 into MDAnalysis:develop Aug 24, 2015

kain88-de deleted the sklearn-kdtree branch September 12, 2015 17:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Sklearn kdtree #395

WIP: Sklearn kdtree #395

kain88-de commented Aug 16, 2015

kain88-de commented Aug 17, 2015

richardjgowers commented Aug 19, 2015

kain88-de commented Aug 19, 2015

richardjgowers commented Aug 19, 2015

kain88-de commented Aug 19, 2015

kain88-de commented Aug 23, 2015

richardjgowers Aug 24, 2015

kain88-de Aug 24, 2015

richardjgowers commented Aug 24, 2015

kain88-de commented Aug 24, 2015

kain88-de commented Aug 24, 2015

richardjgowers commented Aug 24, 2015

WIP: Sklearn kdtree #395

WIP: Sklearn kdtree #395

Conversation

kain88-de commented Aug 16, 2015

TODO:

kain88-de commented Aug 17, 2015

richardjgowers commented Aug 19, 2015

kain88-de commented Aug 19, 2015

richardjgowers commented Aug 19, 2015

kain88-de commented Aug 19, 2015

kain88-de commented Aug 23, 2015

richardjgowers Aug 24, 2015

Choose a reason for hiding this comment

kain88-de Aug 24, 2015

Choose a reason for hiding this comment

richardjgowers commented Aug 24, 2015

kain88-de commented Aug 24, 2015

kain88-de commented Aug 24, 2015

richardjgowers commented Aug 24, 2015