Add sources for the definitions of strategies

[marcharper]

Right now we have a lot of strategies (expecting more, I presume), and there are a few issues that make sense to stay on top of: (1) identifying the original source of a strategy, and (2) tracking multiple names for a given strategy.

Ideally we would have a list of sources that we can cite in various places -- the strategy overview docs as well as the doc strings for the various strategies. I would also help to have an index of strategies of sources indicating what has been implemented, including strategies that have multiple names. That way if someone is either looking for an existing strategy (and doesn't know that WSLS and Pavlov are the same thing) or wants to add a strategy, they can tell if it's already present without digging through all the code (as it's not necessarily obvious which file a strategy would even be in).

I'm imaging an documentation *.rst file that has an alphabetical list of strategy names and alternates along with the library name and citations to the original source of the strategy. Of course I'm open to other suggestions.

Right now the overview of strategy doc is fine but it is organized around existing tournaments, and there are many strategies in the library already that are not in one of these tournaments (and many still from the tournaments that need to implemented).

[drvinceknight]

Ideally we would have a list of sources that we can cite in various places -- the strategy overview docs as well as the doc strings for the various strategies. I would also help to have an index of strategies of sources indicating what has been implemented, including strategies that have multiple names. That way if someone is either looking for an existing strategy (and doesn't know that WSLS and Pavlov are the same thing) or wants to add a strategy, they can tell if it's already present without digging through all the code (as it's not necessarily obvious which file a strategy would even be in).

I'm not too sure I understand how this differs from the current list of strategies that we're working on. #199 was brought in (unless I'm misunderstanding) with the same goal that you're suggesting. I think it was there decided that indexing by previous tournaments followed by the newer ones would be the way to go.

I'm imaging an documentation *.rst file that has an alphabetical list of strategy names and alternates along with the library name and citations to the original source of the strategy. Of course I'm open to other suggestions.

I think that including the sources in the docstrings and using the semi-automatically generated index would take care of this would it not?

Perhaps I'm missing something but I also don't want to fragment the documentation of strategies too much.

For example for Djaxelrod we have meatballs/DjAxelrod#35 which is going to enable a hover box (or similar) which would read the docstrings...

I just think that keeping the index short and snappy (ie semi automatically generated) as well as the curated context list is enough no? Delighted to discuss though, am perhaps overlooking something. :) 👍

(I certainly agree that we can and should include sources for the context in docstrings or perhaps elsewhere)

[meatballs]

I like the sound of this. (Or, at least, I like the sound of what I think it is):

• We create a strategy index document - say 'strategies.rst'
• It lists all strategies alphabetically
• It includes alternate names as separate entries in the list
• The 'master' entry where alternates exist is the name by which it's known in our library
• The description for an alternative name is simply a statement that it's an alternate name and link to elsewhere in the list
• The description for a 'master' entry includes a description of the strategy, links to any external sources and the relevant file name within our library
• The existing 'overview.rst' is renamed to something like 'tournaments.rst'
• It describes the original Axelrod and subsequent tournaments
• It no longer describes the strategies in each but rather lists them with links to the correct locations in strategies.rst

@marcharper Does that sound anything like your suggestion?

[drvinceknight]

I like the sound of this. (Or, at least, I like the sound of what I think it is):

I'm not sure I do like the sound of this.

We create a strategy index document - say 'strategies.rst'
It lists all strategies alphabetically

This already exists right? It's index.rst which is semi automatically generated from the docstrings.

It includes alternate names as separate entries in the list

This is worth just including in the docstrings for each strategy ('also known as...'). This would be helpful for Djaxelrod.

The 'master' entry where alternates exist is the name by which it's known in our library

This is just the class name. I don't see why we need to have a duplicate list.

The description for an alternative name is simply a statement that it's an alternate name and link to elsewhere in the list
The description for a 'master' entry includes a description of the strategy, links to any external sources and the relevant file name within our library

What do you mean by description where would this be? Does this not already exist.

The existing 'overview.rst' is renamed to something like 'tournaments.rst'
It describes the original Axelrod and subsequent tournaments
It no longer describes the strategies in each but rather lists them with links to the correct locations in strategies.rst

This feels rather over complicated and I'm not sure I see what problem it solves.

The problem that the current overview.rst file aimed to solve was that it was thought that people might not be contributing strategies as it's difficult to know what is and is not implemented. When we started working on that we thought that grouping via known tournaments was just as arbitrary as any other way of doing it.

Furthermore I think that having a specific tournaments document leaves the documentation extremely open ended: we could write a lot with regards to analysing the tournaments as opposed to just giving an overview of the strategies in our library (which happen to be organised by previous tournaments).

I'm worried about duplication but also about complication. Does continuing along the plan for #199 and ensuring 'also known as' appear in the docstrings (as well as the overview document) not just take care of whatever this is trying to address?

Attempting to capture all potential names for strategies is not necessarily possible. As and when we'll get PRs that we'll notice are duplicates we can add a note (although this is very much less likely to happen as time goes on).

I'm perhaps just missing the problem we're trying to fix?

[meatballs]

The problem, as I see it, is something like this:

I know of a strategy called 'Pavlov.' Where is it in this library?

The current index.rst includes the fact that WinStayLoseShift is also known as Pavlov but, if you didn't already know that, you may not find it.

Instead, 'Pavlov' needs to have its own entry in the index which states that it's a pseudonym for WinStayLoseShift and links to the WinStayLoseShift entry.

[drvinceknight]

I would prefer to see us addressing that via the names of the strategies and simply going for the most popular one. Your particular example: we really should have called WinStayLoseShift Pavlov for example.

If that is indeed the problem we're trying to solve then it's most simply addressed by changing index.rst by adding these 'dummy names'. I think overview.rst is sound and serving a needed purpose.

Having said that if we just included 'also known as ...' in the doctstring a simple search on index.rst would answer the question.

I would really prefer doing that as it just keep everything in one spot and has less of a chance for a mistake and/or fragmentation.

[meatballs]

If that is indeed the problem we're trying to solve then it's most simply addressed by changing index.rst by adding these 'dummy names'. I think overview.rst is sound and serving a needed purpose.

agreed. if we can add to index.rst without screwing up its autogeneration from docstrings, that sounds like a plan

[meatballs]

I think overview.rst is sound and serving a needed purpose.

I think it's doing too much. Having code in there just risks having it out of step with the actual code if that changes

[meatballs]

Having said that if we just included 'also known as ...' in the doctstring a simple search on index.rst would answer the question.

true

[meatballs]

I would really prefer doing that as it just keep everything in one spot and has less of a chance for a mistake and/or fragmentation.

also agreed - and an argument for getting rid of the code from overview too!

[marcharper]

I agree with both of you -- having it all in one spot is preferred of course but I don't think the existing approach is adequate.

WSLS is the more standard name rather than PAVLOV but both are in common use in the literature. Worse still some call Win-Stay-Lose-Shift Win-Stay-Lose-Switch (at least those two are very similar).

ALLC and ALLD are definitely more common than Cooperator and Defector. They are not better names (AlwaysC or AlwaysCooperate might be better though).

Which is more common -- TitForTat or TFT? Tit-For-Two-Tats or TitFor2Tats (as it's named in the library). Grudger or GRIM? What's the difference between Generous TFT (common) and Forgiving TFT? GTFT is often described as forgiving. Some of the strategies in Axelrod's tournaments are referred to by author name and later by another name. Class names have rules -- no dashes, camelCase -- but the strategy names do not. GoByMajority or HARD_MAJOR (or whatever S&P called it)?

It's just getting confusing, especially when one goes to add a new strategy. When one does discover that there are duplicate names, it's not clear where to add that to the library or documentation.

[drvinceknight]

Re @meatballs:

agreed. if we can add to index.rst without screwing up its autogeneration from docstrings, that sounds like a plan

Certainly possibly, we're just writing index.rst with a python script so we can grab whatever we want I guess although we'd still need to populate the names of duplicates.

I think it's doing too much. Having code in there just risks having it out of step with the actual code if that changes

I think it's nice to have an example showing what each piece of code does. It just lowers the entry point to understanding what is going on. Without reading the whole documentation someone can find a particular strategy and copy the lines of code and play with it.

With regards to actual errors, I think that sphinx can be used to doctest things so the code in there could actually be tested.

We all seem to be in agreement with trying to reduce fragmentation.

To go back to your original 2 points @marcharper:

1. identifying the original source of a strategy,

I don't think this is an architectural issue so to speak. We just need to include sources in the documentation and docstrings. Sphinx has referencing functionality but we can also just have hyperlinks through.

1. tracking multiple names for a given strategy.

That's tricky and ultimately what we've been talking about here.

I am in two minds:

1. I think just including 'aka's where appropriate takes care of things as far as I can see it. Simply searching through the index page would highlight whatever we need to know. It also means that for DjAxelrod we'd have the 'single source of truth'. It's good if the classes have everything we need.
2. It is not ideal that there are multiple names for strategies.

Now I'm not sure what we can do about 2, although you're both suggesting very good ideas but I think that just the same as in research papers etc... Certain names get used at certain times there's just nothing more we can do to be truly exhaustive. I really like the idea of 'aka's...

I think that having a default for every strategy to look like could be one option. Perhaps something like:

class TitForTat:
    """
    A player that...

    ALSO REFERRED TO AS:
         - some name
         - some other name

    SOURCE:
         - Axelrod...
    """

Another possibility (I'm completely diverging right now) would be to include in each strategy an attribute of known_names, then one possibility would be for the DjAxelrod site to index all those for us. I have no idea if that's a good idea or not... It does have the advantage of the instances being in control...

[drvinceknight]

One of the main things I don't like with some of the suggestions is that I don't think it's possible to do them '100% correctly'. This is also certainly a weakness of the overview.rst document.

[marcharper]

So maybe this will help -- when someone comes upon the project, where are they likely to go first to see if a particular strategy is already in the library? Not to the code probably, rather to one of the documentation pages. Ideally there is a page that's easy to navigate to that you can CTRL-F a strategy name and find out if it's in the library. So whatever ultimately achieves that goal would be a "good enough" solution IMO.

Unfortunately the sourcing issue isn't totally independent of the naming issue -- with the name you'd often like to know the source.

In the documentation we could have, for each strategy, something like:

Names and sources

• Name_1, Title of source, or other identifier
• Name_2, Title of source,

Alternatively, if we want this information to be programmatically accessible, I like the suggestion of something like:

names_and_sources = [("Name_1", source_index_1), ("Name_2", source_index_2)]

as a class variable (with the sources defined elsewhere in a bibtex-like manner as @drvinceknight suggests with Sphinx). Then I presume we could generate a master index from the code.

A tangential benefit of doing this well is that the documentation becomes an aggregator of strategies and names for the prisoner's dilemma, ultimately leading more people to discover the project. For example, if you search ZDGTFT2, the Axelrod source file for memory one strategies is on the first page of Google results (as well as one of the earlier issues). Ideally we'd have a single place where searches end up pointing to.

[drvinceknight]

I think I need to think about this before commenting properly but in my usual speak first think later manner 😞 :

Ideally there is a page that's easy to navigate to that you can CTRL-F a strategy name and find out if it's in the library. So whatever ultimately achieves that goal would be a "good enough" solution IMO.

So just to be clear: I believe this is already in place although the extra names are missing from the docstrings. Just checking if we're on the same page.

Unfortunately the sourcing issue isn't totally independent of the naming issue -- with the name you'd often like to know the source.

True although certain naming conventions might be difficult to find a source for ('Folk' like names). We need to word contributions guidelines in such a way that not knowing a source would not stop someone from contributing.

I like the sound of this idea: I'm right in understanding that we're talking about having everything 'in the class'...

I think that it all sounds straightforward enough from a technical point of view even if we're just writing our own python script to write the relevant rst files (as is the case for index.rst right now).

Going to think about this properly now...

[drvinceknight]

Has any one had any more thoughts on this? I've thought about it a bit more and think that the main problem (names and sources) is 'solved' not by implementing anything new but by improving what is already in place. Arguably the docstrings should have had from the start sources and other names (this is probably a mistake on my part for not insisting on them at the start).

I would suggest that this issue becomes a PR to go through and improve the docstrings (and potentially include a note in the contribution documentation about what to do to make sure your strategy is not already implemented etc...).

If I'm wrong and something more sophisticated is needed then we will soon realise it and can think about it again?

[marcharper]

Adding all the strategy names to the docstrings or docs is probably ok for now, but as I add strategies, I see that more and more of them have alternate names, e.g. Grudger, Friedman, Grim, and Grim_trigger all appear to describe the same strategy, and more of them have alternate names and variants than I anticipated. So we can table for now and see if just trying to be more diligent about it now helps.

At least I think we need a "standard" way of adding the names and source (as you suggested above):

class TitForTat:
"""
A player that...

Alternate names:
     - some name, source_1
     - some other name, source_2
"""

Are you ok with a "master list" of Bibtex-style names (maybe whatever Google Scholar uses) somewhere that can be referred to as needed? I wouldn't want to include titles or URLs in the docstrings.

[drvinceknight]

Are you ok with a "master list" of Bibtex-style names (maybe whatever Google Scholar uses) somewhere that can be referred to as needed? I wouldn't want to include titles or URLs in the docstrings.

Sure although would it be that bad to have it in the docstring?

Could include the sphinx style referencing in there and then it automatically builds up references at the end. Here's what this looks like in Sage which uses sphinx (although don't look too closely at the Sage documentation, it is a bloated beast): http://doc.sagemath.org/html/en/reference/game_theory/sage/game_theory/normal_form_game.html (ctrl+f for REFERENCES):

https://www.dropbox.com/s/8jmoj5cda2tj6sg/Screenshot%202015-07-24%2021.42.43.png?dl=0

Just googled this which might be helpful: http://build-me-the-docs-please.readthedocs.org/en/latest/Using_Sphinx/UsingBibTeXCitationsInSphinx.html

[marcharper]

Sphinx-style references are ok with me.

[drvinceknight]

I'm going to pick this one up.

[drvinceknight]

@Nikoleta-v3 I think you've worked on some of these: could you open a PR for whatever you have done? No need to PR these all in one go, getting them in, even if it's one strategy at a time is good :)

[drvinceknight]

I'm going to start working on this now.