keyword-selection mechanism is too fuzzy

[pytestbot]

Originally reported by: BitBucket: jenisys, GitHub: jenisys

---

VERSION: pytest-2.4.2
RELATED: #357

It is currently not possible with py.test to select the exact TestClass/TestMethod (or py.test cannot garantuee it in a general case).

EXAMPLE:

class TestAlice(object):
    def test_foo(self): pass

class TestAliceAndBob(object):
    def test_foo(self): pass

Keyword-selection py.test -k 'TestAlice and test_foo' will select both tests and not one. Currently, you have no mean to specify/require and exact match instead of using the "keyword-contained-in" operator.

DESIRED:

• A solution that allows the user to select a specific test exactly (without changing the code).
• A solution that gives the user/tester the control to specify what he/she means (keyword-contained-in, keyword-startswith, keyword-endswith, exact-match, ...).

POTENTIAL SOLUTIONS:

1. Extend keyword-selection mechanism to something like py.test -k '^TestAlice$ and test_foo' … (exact-match, startswith/endswith for example by using regex-related syntax).
2. Extend the keyword-selection mechanism to fnmatch patterns, like py.test -k 'TestAlice* and *_foo' (but that will break the existing mechanism)
3. Allow/Support keyword-matching strategies, like py.test -k '@fnmatch:TestAlice* and *_foo' (where: "@fnmatch:" prefix selects the matching strategy of the keyword expression after the colon). Candidates for keyword-matching strategies: fnmatch, re (regular expressions), ...
4. Provide another mechanism instead of using keyword-selection

From all proposed solutions above, SOLUTION 3 is probably the most appealing (at least to me), because it gives the user the control to specify what should be done and is extensible for the future (if needed).

IMPACT:
Most of the functionality could easily provided/implemented by extending the pytest.mark.KeywordMapping class that is used by the boolean expression evaluator.

WORKAROUND:
Normally, you would temporarily add a marker, like "@pytest.mark.wip", to the test of interest. This will allow you to select only the test(s) of interest by using the marker-selection mechanism. But if you cannot/should not modify the test code, you cannot use this solution.

I could implement this extension/enhancement and provide a pull-requests after it is agreed what the desired solution should be (if any).

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/387

[pytestbot]

Original comment by Floris Bruynooghe (BitBucket: flub, GitHub: flub):

---

In this specific case one can already select just a single test by using the node ID. Assuming these live in the test_foo file:

py.test -v test_foo::TestAlice::test_foo

or

py.test -v test_foo::TestAlice

work fine.

And to find out the node IDs one can use --collectonly.

[pytestbot]

Original comment by Floris Bruynooghe (BitBucket: flub, GitHub: flub):

---

Having said that I agree that e.g. -k /regex/ could be a nice syntax. I'm not sure if there's a common convention for fnmatch as well, it's the first time I see @ being used for it but it could work.

[pytestbot]

Original comment by BitBucket: jenisys, GitHub: jenisys:

---

Thanks, I also found your suggested solution a while ago (on a newsgroup or somewhere). Actually, that solution is sufficient for me.

BUT: It would be even better to document it. This is currently still missing AFAIK.

[pytestbot]

Original comment by BitBucket: jenisys, GitHub: jenisys:

---

Resolved for me.
But the implemented selection mechanism should be documented.

[pytestbot]

Original comment by Alfredo Deza (BitBucket: alfredodeza, GitHub: alfredodeza):

---

Specifying a node id like Floris mentions is doable, but only when there are methods/functions that aren't parametrized.

As soon as it is parametrized it is impossible as the node ID changes with the values of parametrization (see issue #649 )

If there was a way to tell -k to stop matching or alter how the string is matched (with a regex maybe) this would help.

[mnieber]

Though the problem is not a big one, I've been running into this issue many times over the years, and my current solution will be to add two underscores to each test name (so that no test name is a prefix for another test name).