Packaging

[gavinfernandes2012]

Packaging:

• Write a setup.py
• Write the install instructions
• Move and arrange everything under semantic_db
• Update the hardcoded paths in sdb-console.py
• End user console execution

[GarryMorrison]

A quick test that the console is running:

sa: load fred-sam-friends.sw
sa: common[friends] split |Fred Sam>
|Jack> + |Emma> + |Charlie>

A more comprehensive test is to run the walking ant:

sa: load walking-ant.swc
sa: walk^20

Also, "full-parsley-grammar.txt" was intended for documentation purposes, so belongs under docs/ not semantic_db/

[gavinfernandes2012]

Okay thanks, I'll move it there now, I thought it was configuration for parsley as I hadn't looked into the file and parsley at all.

[GarryMorrison]

It occurred to me that there might be issues with moving sdb-console to semantic-db/main.py, and being able to invoke from anywhere. Where would the graph-examples, sw-examples and console history go?

Also, in the old github repo we had a bunch of scripts that also used the code.py backend. See: https://github.com/GarryMorrison/Feynman-knowledge-engine/tree/master/create-sw-file-tools
If we were to copy any of these over, where would they live in the new code tree hierarchy?
Not that I'm currently planing on doing that.

[gavinfernandes2012]

We could store the users history in the home directory of the user, the same way bash does with .bash_history. With graph examples, I'm thinking of putting them in docs, unless you envision the end user, utilizing the examples every day. sw-examples could be put somewhere in the package, as an examples sub package maybe?
The thing about create-sw-file-tools, is that I'm confused as to whether they're meant to create new sw files from other data sources, or to manipulate sw files? Where we put them depends on what they're for.

Do tell me what you think and whether you have any suggestions.

[GarryMorrison]

I don't like the idea of putting console history in the home dir like .bash_history does. Eg, I'm mostly on cygwin, and putting them in home dir would be a long way from where I work on the console.

In my opinion, graph-examples, and sw-examples should be in essentially the same spot. Something else to consider is that the console has a 'cd' function, that changes the default location for loading and saving sw files.

So I'm not certain about what to do. Options:

1. keep sw-examples, graph-examples, sa-console.txt with respect to location of sdb-console, and don't move it to main.py
2. maybe define directory at invoke time:

python3 -m Semantic-DB working-dir

3. some other suggestion

And if we did (2), I would probably get lazy and write a wrapper:

#!/bin/sh

python3 -m Semantic-DB my-favourite-location

which kind of defeats the purpose.

Would it be too ugly to do:

semantic-db/resources/sw-examples/
semantic-db/resources/graph-examples/
semantic-db/resources/sa-console-history.txt

create-sw-files tools are indeed for converting other data formats to sw. Not that I have written any in a long time.

[gavinfernandes2012]

I actually really like the idea of moving them to a resources folder (we could shorten it to res if you're into that).
The only problem with console history being in resources is that we don't want to package the console history do we? (Do we actually? Is console history meant to be like bash history? or is it meant to be something entirely different)

What we could do alternatively, for sw-examples and graph-examples is put them under an examples folder, under the sw and graph subdirectories respectively.

[gavinfernandes2012]

To be honest, I only moved sdb-console.py to __main__.py as a way to put sdb-console in the pip package, that said, I do feel like it is inelegant to some extent, although I haven't yet been struck by a better idea. Could really use some help with that.

[GarryMorrison]

If we use 'resources' I think we should spell the full word, since 'res' isn't clear what it means.
'examples' is another option:

semantic-db/examples/sw/
semantic-db/examples/graph/

At a stretch, console-history could be considered an example? But yeah, we should probably ship with an empty console history, unless we want to include some toy illustrative examples in there? Also, currently console history is save only, so you need to be on the command line to look things up. Going forward, maybe we can get the console to load the last x lines from it, and pre-populate the console history. That should be pretty easy to implement.

[GarryMorrison]

As for sdb-console.py vs main.py I am currently torn. I need some time to think and work out my preference.

[gavinfernandes2012]

Regarding the console history as examples, how feasible do you think it is to be able to run a script using the console like so:

python3 sdb-console.py some-example-script

That way we could shift from the examples being in the console history, to the examples being in a shell script. Now I haven't looked into these specific examples, so I don't know how suitable they would be as scripts, but do let me know what you think.

[GarryMorrison]

yeah, I like that idea! Also, we should allow multiple scripts to be run at once (note that the order matters).

python3 sdb-console.py script1 script2 script3 ...

And we have two options at the completion of the scripts. Either return to the command line, or remain in the console. I currently think it should be the console.

That shouldn't be too hard to implement. I'll take a look tomorrow. It is approaching my bed time right now. So we will have these options at invoke time:

--info script1 ... scriptn
--debug script1 ... scriptn
script1 ... scriptn

So run scripts in info mode, debug mode, or silent mode respectively.

Not sure if going forward we want other command line options....

[gavinfernandes2012]

Alright lovely, we could put the scripts in semantic_db/examples/scripts if you like, hopefully tonight I'll be able to get either a few commits or some planning in before I go off. (PS: Good night :)

[gavinfernandes2012]

Okay, I'm gonna use this PR (Pull Request) to get packaging setup. I'll open another PR for code organisation, separation, refactoring etc.

[gavinfernandes2012]

What I can't seems to decide right now, is whether I should put INSTALL.md under docs, or whether I should leave it outside for visibility / discoverability. What do you think I should do?

[gavinfernandes2012]

Gonna turn in for tonight, hopefully I'll be able to finish this PR tomorrow morning before work.

[GarryMorrison]

I think INSTALL.md should be outside for visibility, along with its brothers LICENSE and README.

[GarryMorrison]

Also, 'semantic_db' vs 'semantic-db' for our directory name. I kind of prefer dash over underscore. What do you think?

[GarryMorrison]

Also, in install instructions for graphviz, I think you should mention you need to manually add the graphviz binary to your path. I know a pain right!

[gavinfernandes2012]

Well the pythonic way seems to be underscores for package names according to PEP8, but honestly, I feel a dash would be better, although I didn't want to bring it up. So it's your choice, you can either do it pythonically or (IMO) a bit more elegantly.

Ohh yeah graphviz, I'll get that done in a bit. Aren't there any python only alternatives to graphviz? Or is there too much code to make porting feasible?

[gavinfernandes2012]

Also @GarryMorrison, have you decided on whether we're keeping __main__.py?

[GarryMorrison]

Can we specify examples/ and docs/ using pip?
Something like this:

pip install semantic_db --install-examples=$PWD --install-docs=$PWD

And if not specified, then don't install examples and or docs.

Yup, looks possible, using --install-option:
https://stackoverflow.com/questions/47210058/passing-command-line-arguments-to-pip-install
https://stackoverflow.com/questions/18725137/how-to-obtain-arguments-passed-to-setup-py-from-pip-with-install-option/33200591#33200591

pip install semantic_db --install-option="--install-examples=$PWD" --install-option="--install-docs=$PWD"

[gavinfernandes2012]

Could we separate the examples from the code? That way we can have a git repository for the examples and/or docs, that the user could clone wherever he wants, and we don't have to write and maintain a more complicated setup system (We have to consider uninstalling the files, which will necessitate inelegant code).

[GarryMorrison]

Not my preference. I'm still in favour of the --install-option approach. And as for uninstalling, if someone chose to install examples/docs and they decided the location (they can of course use something other than $PWD), then they should be fine with manually deleting those directories if/when they uninstall. If you really want we can include an UNINSTALL file.

I don't see how manually deleting the examples/docs is any different than if we had separate repo for examples/docs. We just need to spend a little effort on writing the appropriate setup.py.

[gavinfernandes2012]

Okay fine, I'll throw something together for this.

[GarryMorrison]

thanks :)

[gavinfernandes2012]

Okay this looks impossible to me, I'll explain why.
The way that setuptools and distutils works, you call a setup function, and (one of) its arguments specify where each (data ie sw) file goes. Now since that install options command is also specified in the setup function, we can't change what directories we pass to the setup function based on a global variable that only gets created somewhere within the setup function call.

I also tried conda, flit and pbr, and since all of them are based on setuptools, they have the same or similar limitations.

[GarryMorrison]

well, that's annoying.
But thanks for trying.

Hrmm... seems to work for me:

$ pip3 install . --install-option="--examples=$PWD/../examples" --install-option="--docs=$PWD/../docs"

Or am I missing something key here?

[gavinfernandes2012]

Wait a second, do you mean you just run that and it installs examples and docs to the PWD?

[GarryMorrison]

Check your email. I updated the setup.py, and in testing on my linux box it seems to do what we want.

[GarryMorrison]

at some stage we need to tidy up the print statements inside setup.py too.
They are currently a bit noisy/messy. They are set up for testing ATM.
I can do that tomorrow if you like.

[gavinfernandes2012]

I can tidy them up for you, but I've hit a snag. The way pypi works, it tries to automatically build wheels for all packages uploaded to it, and when I manually build a wheel and try to pip install it, it doesn't recognize --install-options, so really our install time directory creation probably won't work on pypi. Unless of course we specify --no-use-wheel.
[Source]

[gavinfernandes2012]

So do you want to go with wheels? or do you want to go with install time directory creation?

[GarryMorrison]

I don't know how pypi works, or wheels for that matter.
But can we do a mixed approach?
If they install from pypi examples and docs are not installed.
But if they install from github, they can specify example and docs directories?
Or does it not work like that?

Though would be kind of useless without examples.....

[gavinfernandes2012]

We can do that if you want, I have no problems with it really. The way I've seen other projects do it though is to have a separate git repo for examples, or just leave examples in the root directory really. The way you have it set up currently works as a mixed approach. I'll clean up the prints now.

One more thing, the examples/sw directory is important because currently, thats where sw files are loaded from. I think we should reimplement load to load sw files from the current directory. Do you want me to go ahead and do that? If you do, I can remove examples from site-packages safely.

[GarryMorrison]

As for examples/sw/ dir and loading, my thoughts was we put a config file in .sdb/config in users home dir. And we save the sw/ graph/ dirs given to setup.py in that config file. And if we don't have a config file, or those directories are not specified in it, we default to files in the current directory.

And we have a bunch of other settings in the shell that would benefit from a config file. Like history length, where/if we save shell history etc.

Hrm... maybe you are right. Maybe current dir is way to go. It does mean things like 'ls' and 'cd' in the shell become kind of pointless. And it will also mean our .sw .dot and .png will be all in the same directory. Maybe that is not an issue either?

Hrm... we have a web-load function, so why not a web-files function in the shell?

web-files http://semantic-db.org/sw/
web-files https://github.com/GarryMorrison/Semantic-DB/sw/

We just need to be sure there are no security issues from loading remote sw files.
And to save all that typing, have it similar to history in the shell, where you can specify the file to load using a number. If we ever have a GUI shell, then you could just click a link.

[gavinfernandes2012]

We can still separate files into directories and do a load myswdir/something.sw relative to the correct directory, if we implement it correctly

[gavinfernandes2012]

Wait a second, can you explain to me what exactly create_index is supposed to do? Cause when I run, it just creates an index.html with "Some sample sw files." for me.

[GarryMorrison]

You invoke with:

python3 -m semantic_db.create_index path-to-your-sw-files

And it spits out an index.html and a sw-index.txt file that you upload to your host along with your sw files. See here for example:

http://semantic-db.org/sw/index.html
http://semantic-db.org/sw/sw-index.txt

sw-index.txt is the backend web-files uses, if that doesn't exist it looks for .sw and .swc links in the html page instead.

[gavinfernandes2012]

Ahh alright, so it's for the server side of things for web load, right?
Do we see users using this functionality? Or would it be more useful to devs/us while developing?
And does your semantic-db server (The one you host your blog on) double up as a web load server as well?

[GarryMorrison]

• Yes, it is for the sever side of web-files.
• Power users, yeah. The goal is for others to write interesting sw files, and host them themselves.
• Yep. At least for now. If I ever get too much traffic there might be issues!

[gavinfernandes2012]

Alright, so is there anything else left for us to do before we merge? (I'll resolve the merge conflicts soon, besides that I mean)
We should probably upload to pypi and test if installation with install options works. Do you want to go ahead and do that? I could work on the history in your config-file branch in the meantime.
(Do remember to upload this branch, and not any other branch XP)

[GarryMorrison]

Maybe we should finish config-file branch first, agree on that, merge that in. Then look at merging this one in? As for pypi I will have to go create an account and so on.

[gavinfernandes2012]

Alright fine, I'm working on the implement-config-file branch as we speak.