TODO.txt

- `vee commit` makes sure you haven't forked
- `vee status` tries to fetch if the revision is not in the local repo

- Should we auto-detect dev directories on VEE_DEV_PATH if we can?
  `vee dev find` or `vee dev rescan` could be adjusted to auto-detect
  things too. It likely isn't a great idea to have the discovery run on every
  dev, since you can easily have repos that you put quite a bit of work into
  auto-discovering, only to have them fail. However, that failure could also
  be cached in the JSON document.

- --build-sh, --install-sh, etc., need to execute within the build/install
  path, not where they are sitting (since the repo:xxx ones are sitting
  in the repo)

- vee-rc command for communicating with parent vee via $VEE_RC_PORT
  - name: vee-rc, subvee, __vee__, vee-agent, vee-build
  - this is to replace the wacky envvar stuff for the build environment
    - this is used by key_base
  - also to replace the develop callback functions
    vee_develop_setenv -> vee-agent setenv
  - call the build/install/develop script if they are executable, otherwise
    source them
  - these are used in minimal places:
    $ find ~/dev -name 'vee-*.sh' -maxdepth 3
    /home/mboers/dev/key_base/vee-develop.sh
    /home/mboers/dev/key_base/vee-build.sh
    /home/mboers/dev/pyqb/vee-build.sh

- git is not respecting the umask. We could set core.sharedRepository
  (see `man git-config`) in all of the repos, or use --shared when we init/clone.

- dev tags should have the platform in the name


- can we bootstrap patchelf?
- can we include vee in the PYTHONPATH?

- platform keys: darwin-10 and fedora-20 (instead of just linux)

- detect that a package is from homebrew via existence or "INSTALL_RECEIPT.json"?
    - perhaps only when our metadata indicates that it is in the same path

- `--checksum sha1:xxx` for static packages: we are getting TONS of read errors either 
  across the NFS or across HTTP. This should be calculated during repackage,
  before it even hits the disk (with a passthrough file object).

  - Also support #md5=xxx in URL fragments.

- add lots of cascading to foreign key contraints

- requirements functions: $FFMPEG_PREFIX or $FFMPEG_INSTALL_PATH or ${packages['ffmpeg'].install_path}
  $(prefix ffmpeg)
  

- pipeline_log table containing the stdout/err and metadata from each pipeline
  step. This would allow failed installs to tell you a quick `vee cat log:123`
  in order to see what happened.

- normalize_dev_path(xxx) -> (name, path) so that all the stuff dealing with
  dev packages is normalized

- summary of results at end of upgrade in a manner that ksfab can parse and
  aggregate

- `%include PATH` for moving the platform dependencies into other files

- `vee upgrade --clean` deletes the environment

- shell setup fails really badly if kscore isn't installed

- local PyPI mirror would be a good idea
    - this could be some caching proxy server or anything
    - could verify the checksum itself

- parse `make -pn` to determine if there is even an 'install' target or not.

- automatically relocate generic packages? Homebrew, Python, and self packages
  would not do it.

- remaining relocation problems:
    
    /usr/local/vee/installs/imagemagick/6.9.1-0+5df00e1f/lib/ImageMagick/modules-Q16HDRI/coders/jp2.so
      Warning: No relocation targets for /usr/local/lib/libopenjp2.7.dylib
    /usr/local/vee/installs/openexr/2.1.0+5df00e1f/lib/libIlmImf-Imf_2_1.21.dylib
      Warning: No relocation targets for /usr/local/lib/libImath-2_1.11.dylib
      Warning: Best target has 1 collisions for /usr/local/lib/libIex-2_1.11.dylib
      Warning: No relocation targets for /usr/local/lib/libIexMath-2_1.11.dylib
      Warning: Best target has 1 collisions for /usr/local/lib/libIlmThread-2_1.11.dylib
    /usr/local/vee/installs/gobject-introspection/1.44.0+5df00e1f/bin/g-ir-compiler
      Warning: No relocation targets for /usr/local/Cellar/gobject-introspection/1.44.0/lib/libgirepository-1.0.1.dylib
    /usr/local/vee/installs/gettext/0.19.4+5df00e1f/bin/msginit
      Warning: Best target has 2 collisions for /usr/local/Cellar/gettext/0.19.4/lib/libgettextlib-0.19.4.dylib
      Warning: Best target has 1 collisions for /usr/local/Cellar/gettext/0.19.4/lib/libintl.8.dylib
    /usr/local/vee/installs/libtiff/4.0.3+5df00e1f/lib/libtiffxx.5.dylib
      Warning: Best target has 1 collisions for /usr/local/Cellar/libtiff/4.0.3/lib/libtiff.5.dylib


- should we be doing initial relocations on packages BEFORE repackacing them,
  replacing their install_path with @@VEE_INSTALL_PATH@@ ?


WesternX
--------

- Linux clean build issues:
    - key_base did not see metatools in vee-build.sh (for entryponts, I think)
    - docs could not find sphinx-build, likely since the "make" step doesn't
      include the python bootstrap
    - cythonmagick returned exit status 1 during egg-info, without saying what
    - pyav failed due to corrupt gitvee

- OS X clean build issues:
    - pyav failed due to corrupt gitvee


Homebrew
--------

- `--relocate` signals that we only want to use Homebrew to build, but we
  immediately copy the artifact out and relocate it to use pinned dependencies.

- document how cavalier we are towards it
  - when you ask for a version, we don't even try to get it, but just warn when
    it isn't the one that you wanted.
  - Homebrew is generally for small installs, or for you to build your own
    packages with repackage


Relocation
----------

- relocate spec "DEPS" pulls in dependencies, and "@" is the default of "DEPS,SELF"

- "SELF" adapts to "BUILD:INSTALL", which is a mapping, where it searches in one prefix
  and then rewrites it to be in the second (e.g. "BUILD:INSTALL").

- should deferred dependencies include the name, such that they can resolve
  against other things in the PackageSet?

- we aren't relocating against packages that are in the same packageset??


Pipeline
--------

- uninstall after "init" in chain
    - this only makes sense if things can immediately tell if they are installed,
      which they can only do if the URL is the same and revision is provided...
      which seems sane.
    - putting it after "inspect" seems like a better fallback

- rewrite docs to talk about the progression of names

- Package(pipeline='develop') to select the pipeline. Each pipeline is supposed
  to start with init, which is indempodent.

- should pipelines not be a property of the Package at all, and instead you
  make a pipeline when you want do perform an operation?



Dev
---


- `vee upgrade --dev` uses dev repos in their current state instead of having
  to push them out; this is a way of testing dev-install.sh (when it exists)

- `vee dev build` runs the build command in the pipeline
- translate https to scp git URLs for dev purposes
  - a DevPackage could expose a bunch of this stuff
- Requirements.iter_scm_packages()
- Requirements.iter_dev_packages()


Server
------

- IMAP-like protocol, in that one can send several packets related to a single
  request. Can also send unsolicited ones.
- no end is specifically the server, they just use different tags to identify
  themselves





---

- should build and install names be tons simpler, given that they all
  exist in the same namespace anyways?
- package names should still be huge and gross

- Package.assert_unique_install() makes sure there isn't anything else
  that uses the same install path.

- remove `vee init`, as it is largely unnessesary?

- install dependencies during `vee install` as well
  - pull in default (or specified repos) to resolve them
  - --no-deps stops this, and uses PackageSet.install(ignore_deps=True)

- development_packages should record which user they are for. Perhaps a
  "dev_set" or "namespace", which defaults to the user, would be better.


- bash completions

- `vee commit --auto` should recate a commit message out of all constituent commit
    messages

- vee-install.sh
    gets VEE_INSTALL_PATH, also $1

- remove write permissions from packages once installed?

- `vee selfupdate` should be as close to just calling the installer as possible

- create timestamped commit environments too?
    by_hash -> real environments
    by_time -> links timestamped by commit time
    $branch -> links

  
- dependencies

    - vee-inspect.sh or vee-requirements.txt to specify requirements, we can
      then immediately create our own "bottles" from Homebrew

        - `vee package-installed name`
            - if there are dependencies, then package those too
            - this requires there to be dependency links for homebrew
              packages

- `--link-prefix` so that it doesn't have to install it elsewhere, just link
  it elsewhere.

- need to populate environment.repo_id and repo_revision in the database

- PyQt for Maya/Nuke in Linux
  - perhaps we should just do this with envvars?
  - just package up what Mark has preppared previousely in /opt

- add for loops and string formatting to requirements files
  
  % for x in ...:
  % endfor

  - do not allow git-based packages to be found inside of for loops
  - if statements are ok though

- check config/environ against database when deciding if resolve_existing
  matches. use this for build_subdir and install_prefix too

- it could be nice for packages to register envvars (or scripts to run at source time)
  but it certainly isn't nessesary for us to do up front

- We *could* isolate the various homebrew packages. You must make a
  $PREFIX/bin/brew link to the origin command, and a $PREFIX/Cellar directory.
  Then you can symlink dependencies into $PREFIX/Cellar/$NAME/$VERSIONISH and
  do whatever you want. OR, we can copy things out of the Cellar when they
  are built, and relocate them to their new environment.

- move save/delete/whatever to the Database or a Session?

- `vee upgrade` after a `vee add` without a push wont find the commit
    - `vee upgrade --dirty` should rewrite GIT urls to pull from dev

- s/Home/Context

- put `vee.__about__.__revision__` into every database row
- `vee server` and `vee client`
    - make a super simple socket protocol. just send JSON back and forth

- setting envvars in requirements.txt should allow for @ or $KEY to take
  in previous values that were set within that file. So, we need some syntax for:
    1. Immediately resolution of variables.
    2. Deferred resolution of variables.
    3. Deferred resolution of previous value.

- `vee dev checkout --repo REPO` to fastforward everything to match the given repo
- `vee dev ff` to `pull --ff-only` everything to their origin/master

- use difflib to compare the old and new requirements. This will allow for us
  to detect re-orderings. Put dev-only packages at the end. Put dev information
  with the new destination.

- EnvRepo.iter_requirement_versions() -> yields (work, head, dev)

    - Home.get_dev_repos()
    - zip_matching(dev_repos, work_reqs, head_reqs, key=lambda x: x.name)
    - zip_matching(home.get_dev_repos(), env_repo.iter_requirements(), env_repo.iter_requirements(revision='HEAD'), sorted=True, key=lambda x: x.name)

- `vee status` summary at bottom, like `git status`
    Your branch is up-to-date with 'origin/master'.
    nothing to commit, working directory clean

- should dev_packages track remotes (and branches), should that be in the
  git repo itself, or do we do it ad-hoc? We could pick an ideal remote given
  a matching requirement, such that it picks whatever remote we are installing
  from. Everything gets a little easier once we have a DevPackage (which is
  actually a GitRepo).

    - DevPackage.pick_remote_for(Requirement)
        - picks one that matches, and saves it

- `vee push` should also push tools themselves (as long as they match up with
  those in the requirements)

- `vee status` should warn if there aren't remotes that seem related to those
  in the requirements

- server and client: start as a super simple notification server that says when there is
  a new version of any repos


COMMANDS TO WRITE
-----------------

  - vee gc [--installs] [--environs]
      - delete installs (and their DB records) which are not linked to
      - delete installs in DB that don't exist on disk
      - delete anything on disk that isn't referred to by the index
          build a set of relative paths (and all their ancestors), then walk the
          root looking for directories which aren't mentioned, then delete them

  - vee uninstall (NAME|REQUIREMENT [ARGS])
  - vee unlink ENVIRON (NAME|REQUIREMENT [ARGS])
      - this may not be possible, or desireable

  - vee list
      - list packages, environments, etc..

  - vee freeze [-e ENV] [-R req] [-r repo] [DST_REPO]
      - freeze all requirements into a repository
      - this uses much of the same code from `vee exec` and `vee add`


---

- there are a ton of times we need to guess which dev remote to use; perhaps
  we should have a remote/branch to track in the database, OR figure out how
  to interpret that from the repo itself. Git seems to store a repo per branch,
  so we could always look at the active branch

- RequirementSet elements should be their own class:
    - el.path is path to file that contained it

- permissions

    - install_vee.py
        - just warn if they are running as root, but don't stop them
        - default to assuming a single user
        - --multi-user signals that it should try to setup a group setup, OR we
          can just let them figure that out themselves

    - `vee doctor` should do the main permission checks

    - do we need to set our umask, or can permission bits somehow handle that?
        - document the result
    - do we need to chgrp, or can setgid handle that?
        - document the result

- test python packages (for each of source, sdist, bdist, bdist_wheel):
    - they import
    - they import each other
    - console_scripts entrypoints work
    - scripts work
    - their install_requires doesn't matter

- install data that comes with Python wheels
  $NAME-$VERSION.data dir at top-level, beside $NAME-$VERSION.dist-info

- install Python commands listed in egg-info

- build-subdir and install-prefix should be used to invalidate matches in the
  resolve_existing

- log everything about the different steps, and stuff it into the database
- CLI IO/API/logging package

    - name:
        x clio (on PyPI)
        - clout

    - styles (copied straight from what we have is cool)
    - io indenting model
        - replaces sys.stdout and sys.stderr, and those are pushed through
        - can spawn reader threads which can (1) buffer the output, (2) echo it, (3) push it to a callback
        - with clout.io.indent(), or clout.io.push_indent() and clout.io.pop_indent()
    - event log
        - format: "$datetime $stream $string-escaped-content"
        - events:
            - exec:$pid -> nth executable
            - arg:$pid:I -> argument I of nth exe
            - out:$pid -> stdout of nth exe
            - err:$pid -> stderr of nth exe

- record stdout/stderr from build process in the database. Use a timestamped
  format: each line starts with "out" or "err" and the timestamp

    out 2015-02-18T15:02:01 sdflkjsdf

- Http/File/Base could do checksums of files to see if they have changed
  - memoize the caches based on inode,size,mtime


LATER
-----

- switch to click (after vendoring it?)

- support multiple python versions at the same time, e.g. 2.6, 2.7, 3.4.

- custom managers
  - e.g. `vee install PyAV.py`, where PyAV.py contains:
        - REQUIREMENT = 'https://pypi.python.org/packages/source/a/av/av-0.2.2.tar.gz#md5=ec0198f28d9294d20b54b0ac3a9ff77d'
        - DEPENDS_ON = ['lib:ffmpeg']
        - MANAGER or PyAVManager or PyavManager, which inherits from BaseManager

- pypi manager
    - PyPI JSON API -> https://pypi.python.org/pypi/%s/json
    - Need to either hit the PyPI on every `.installed` check, or cache versions.

- homebrew taps: homebrew+mikeboers/testbrew/foo
    - we would need a way to detect which tap it is
    - we can grep `brew info $forumla` for From: https://github.com/#{user}/#{repo}/blob/master/#{path}

- Requirement.dependencies() and Requirement.provisions()

  An AbstractRequirement is one like "lib:ffmpeg", "py:yaml", etc., that just
  know what result they want, but not where it is from. A DependencyInterface
  could be the intersection of AbstractRequirement and Requirement, such that

  Requirement.dependencies() can return real ones (e.g. from `brew deps`)
  and abstract ones. It is permitted to return different dependencies on each
  call (as they are discovered, e.g.)

  DependencyResolver can take a pool of requirements and figure out what order
  they should be installed in (via C3)

        .add(requirement)
        .rescan_dependencies()
        .linearize()