Update requirements to match kuma-lib; initial working Travis-CI tests #1945

lmorchard · 2014-01-31T23:26:22Z

This PR shouldn't really do anything to production.

I manually audited every dependency in kuma-lib, and updated our requirements files to match. In order to attempt to verify I'd gotten it right, I set up Travis-CI tests that do not check out the kuma-lib git submodule.

All of those tests, except one, passed. The tweak in the demos app is what was needed to make the test pass, because that code attempted to access constance DB tables on module load. That caused a chicken-and-egg problem because the module was loaded while creating DB tables.

This might need adjusting, but the following is what worked for a spot check on my my Mac:

git clone https://github.com/lmorchard/kuma.git kuma-pip
cd kuma-pip
git co death-to-kuma-lib
virtualenv venv
. ./venv/bin/activate
brew install libjpeg libmagic libmemcached
pip install -U pip
pip install -r requirements/compiled.txt -r requirements/prod.txt -r requirements/dev.txt
mysql -uroot -e 'create database kuma;'
mysql -uroot -e 'grant all privileges on kuma.* to ""@localhost identified by "";'
./manage.py update_product_details
FORCE_DB=1 ./manage.py test --noinput wiki

…ug 914808)

…f constance in demos forms

groovecoder · 2014-01-31T23:47:23Z

.travis.yml

+
+notifications:
+    email:
+        - [email protected]


can this be [email protected]?

Yeah, haven't gotten around to tweaking the notifications yet, still getting the travis tests working reliably and didn't want to spam mdndev or IRC

groovecoder · 2014-02-01T03:16:01Z

Looks like it also needs product_details_json before tests will run?

(reqkuma)lcrouch-14912:kuma lcrouch$ ./manage.py test actioncounters contentflagging demos devmo landing users search wiki
Traceback (most recent call last):
  File "./manage.py", line 37, in <module>
    import jingo.monkey
  File "/Users/lcrouch/python/reqkuma/src/jingo/jingo/__init__.py", line 160, in <module>
    env = get_env()
  File "/Users/lcrouch/python/reqkuma/src/jingo/jingo/__init__.py", line 48, in get_env
    loaders = [loader(p) for loader, places in x for p in places]
  File "/Users/lcrouch/python/reqkuma/lib/python2.7/site-packages/jinja2/loaders.py", line 216, in __init__
    provider = get_provider(package_name)
  File "/Users/lcrouch/python/reqkuma/lib/python2.7/site-packages/pkg_resources.py", line 247, in get_provider
    __import__(moduleOrReq)
  File "/Users/lcrouch/python/reqkuma/src/django-mozilla-product-details/product_details/__init__.py", line 104, in <module>
    product_details = ProductDetails()
  File "/Users/lcrouch/python/reqkuma/src/django-mozilla-product-details/product_details/__init__.py", line 49, in __init__
    for filename in os.listdir(json_dir):
OSError: [Errno 2] No such file or directory: '/Users/lcrouch/code/mozilla/kuma/../product_details_json'

lmorchard · 2014-02-01T06:43:41Z

Yes, you'll need to run ./manage update_product_details. That, or download the tarball used for Travis

lmorchard · 2014-02-03T16:49:33Z

I updated the PR description... going through my shell history, I think these are more exact instructions for a local check:

git clone https://github.com/lmorchard/kuma.git kuma-pip
cd kuma-pip
git co death-to-kuma-lib
virtualenv venv
. ./venv/bin/activate
brew install libjpeg libmagic libmemcached
pip install -U pip
pip install -r requirements/compiled.txt -r requirements/prod.txt -r requirements/dev.txt
mysql -uroot -e 'create database kuma;'
mysql -uroot -e 'grant all privileges on kuma.* to ""@localhost identified by "";'
./manage.py update_product_details
FORCE_DB=1 ./manage.py test --noinput wiki

The bit that grants mysql permissions to the anonymous user is a hack - the better change is to compose a settings_local.py for your local host that has proper credentials & etc. But, this seemed to work for a quick check.

groovecoder · 2014-02-03T20:59:33Z

Everything works without errors until I get to the manage.py commands:

(venv)lcrouch-14912:kuma-pip lcrouch$ ./manage.py update_product_details
Traceback (most recent call last):
  File "./manage.py", line 37, in <module>
    import jingo.monkey
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/src/jingo/jingo/__init__.py", line 160, in <module>
    env = get_env()
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/src/jingo/jingo/__init__.py", line 46, in get_env
    x = ((jinja2.FileSystemLoader, settings.TEMPLATE_DIRS),
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/lib/python2.7/site-packages/django/utils/functional.py", line 184, in inner
    self._setup()
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/lib/python2.7/site-packages/django/conf/__init__.py", line 95, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'settings.py' (Is it on sys.path?): No module named py
(venv)lcrouch-14912:kuma-pip lcrouch$ FORCE_DB=1 ./manage.py test --noinput wiki
Traceback (most recent call last):
  File "./manage.py", line 37, in <module>
    import jingo.monkey
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/src/jingo/jingo/__init__.py", line 160, in <module>
    env = get_env()
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/src/jingo/jingo/__init__.py", line 46, in get_env
    x = ((jinja2.FileSystemLoader, settings.TEMPLATE_DIRS),
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/lib/python2.7/site-packages/django/utils/functional.py", line 184, in inner
    self._setup()
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/lib/python2.7/site-packages/django/conf/__init__.py", line 42, in _setup
    self._wrapped = Settings(settings_module)
  File "/Users/lcrouch/code/mozilla/kuma-pip/venv/lib/python2.7/site-packages/django/conf/__init__.py", line 95, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'settings.py' (Is it on sys.path?): No module named py

lmorchard · 2014-02-04T17:05:39Z

Okay, that error's odd... @jezdez any ideas here, have I missed something? This doesn't happen on my box or on Travis-CI

lmorchard · 2014-02-04T17:08:42Z

@groovecoder Can you paste the result of this from your shell?

echo $DJANGO_SETTINGS_MODULE

Mine comes up blank, I wonder if something on your machine set it. Seems like it's looking for a module named "py", but that's weird

lmorchard · 2014-02-04T17:10:45Z

@groovecoder Actually, I can reproduce the error if I run this export you mentioned earlier:

export DJANGO_SETTINGS_MODULE=settings.py

a) You don't need that if you've done it, and b) I think that's meant to be a python module name and not a filename (ie. no .py extension, or else it will look for a module named py as a child to settings)

Unless you've put that export into your .profile, it's best to run the setup in a fresh shell.

jezdez · 2014-02-04T17:58:15Z

requirements/dev.txt

-e git://github.com/jbalogh/django-debug-cache-panel#egg=cache_panel
+-e git://github.com/jbalogh/check.git@1c73813fcd085020decc6f72caf232f71150db0a#egg=check
+# 2010-04-07
+-e git://github.com/robhudson/django-debug-toolbar.git@ee1811238e91ae0ad33413b0d40d2f8482101951#egg=django-debug-toolbar


This should probably point to github.com/django-debug-toolbar/django-debug-toolbar instead

Well, it's pointing to exactly what we have in kuma-lib. Didn't want to diverge much in this first round of updates, if I could help it.

Right, Rob moved the app into its own org though and I just want to make sure we don't accidentally end up with a dead end in the future. The org repo should have the same commit.

Yep, django-commons/django-debug-toolbar@ee18112

Cool, same commit. So, I'll update that, give a fresh install a shot (because I was going to play with virtualenvwrapper) and commit

jezdez · 2014-02-04T18:08:24Z

This looks good in general, I've left a few comments here and there. This should probably use peep (https://pypi.python.org/pypi/peep) for added security. Also https://github.com/ronaldevers/magnum-pi/ may be useful for generating an "index".

groovecoder · 2014-02-04T20:39:42Z

It works! The only snag I hit was that I had to create the ../product_details_json directory manually:

(pipkuma)lcrouch-14912:kuma-pip lcrouch$ ./manage.py update_product_details
Traceback (most recent call last):
  File "./manage.py", line 37, in <module>
    import jingo.monkey
  File "/Users/lcrouch/python/pipkuma/src/jingo/jingo/__init__.py", line 160, in <module>
    env = get_env()
  File "/Users/lcrouch/python/pipkuma/src/jingo/jingo/__init__.py", line 48, in get_env
    loaders = [loader(p) for loader, places in x for p in places]
  File "/Users/lcrouch/python/pipkuma/lib/python2.7/site-packages/jinja2/loaders.py", line 216, in __init__
    provider = get_provider(package_name)
  File "/Users/lcrouch/python/pipkuma/lib/python2.7/site-packages/pkg_resources.py", line 247, in get_provider
    __import__(moduleOrReq)
  File "/Users/lcrouch/python/pipkuma/src/django-mozilla-product-details/product_details/__init__.py", line 104, in <module>
    product_details = ProductDetails()
  File "/Users/lcrouch/python/pipkuma/src/django-mozilla-product-details/product_details/__init__.py", line 49, in __init__
    for filename in os.listdir(json_dir):
OSError: [Errno 2] No such file or directory: '/Users/lcrouch/code/mozilla/kuma-pip/../product_details_json'
(pipkuma)lcrouch-14912:kuma-pip lcrouch$ mkdir -p ../product_details_json
(pipkuma)lcrouch-14912:kuma-pip lcrouch$ ./manage.py update_product_details
(pipkuma)lcrouch-14912:kuma-pip lcrouch$ FORCE_DB=1 ./manage.py test --noinput wiki
nosetests wiki --verbosity=1
Creating test database for alias 'default'...
.................................................................S...S..................................................................................SSS..S.............S............SSSSSSS..................SSSSSSSSSSSSS........S..SS..........................................................................................S.........................
----------------------------------------------------------------------
Ran 351 tests in 76.392s

OK (SKIP=31)

I wasn't able to run schematic migrations/ until I added a proper username to the DATABASES in settings.py. When I did, I got:

Running migration 13:
--
-- django-constance uses manager.get(key=) but doesn't specify unique=True on the
-- key field. This migration fixes any duplicates and adds the unique constraint

CREATE TEMPORARY TABLE dup_constance_config_keys
    SELECT `key`, `value`
    FROM constance_config
    GROUP BY `key`
    HAVING count(`key`) > 1;

DELETE FROM constance_config
    WHERE `key` IN
        (SELECT `key` FROM dup_constance_config_keys);

ALTER TABLE constance_config
    ADD UNIQUE constance_config_key_unique(`key`(100));

Error: Had trouble running this: BEGIN;
--
-- django-constance uses manager.get(key=) but doesn't specify unique=True on the
-- key field. This migration fixes any duplicates and adds the unique constraint

CREATE TEMPORARY TABLE dup_constance_config_keys
    SELECT `key`, `value`
    FROM constance_config
    GROUP BY `key`
    HAVING count(`key`) > 1;

DELETE FROM constance_config
    WHERE `key` IN
        (SELECT `key` FROM dup_constance_config_keys);

ALTER TABLE constance_config
    ADD UNIQUE constance_config_key_unique(`key`(100));

UPDATE schema_version SET version = 13;
COMMIT;
stdout:
stderr: ERROR 1146 (42S02) at line 6: Table 'kuma.constance_config' doesn't exist

returncode: 1

Still, I'm okay merging this to avoid bit-rot.

groovecoder · 2014-02-04T20:59:33Z

Merging this as the "makes tests pass" milestone. Work will continue in #1934

Update requirements to match kuma-lib; initial working Travis-CI tests

lmorchard added 2 commits January 31, 2014 18:20

Update requirements files (mdn#1934) and first working .travis.yml (b…

512526f

…ug 914808)

bug 914808: Stop updating product details at test time; tweak usage o…

d2434c8

…f constance in demos forms

groovecoder reviewed Jan 31, 2014
View reviewed changes

lmorchard added 2 commits February 3, 2014 11:51

bug 914808: Notify mdndev in IRC and mailing list on travis status

cb1693e

Migration fixes for starting from a fresh DB

54fea36

lmorchard mentioned this pull request Feb 3, 2014

Speed up Travis-CI tests mozilla/django-badger#199

Open

Upgrade django-badger

67ac7bf

jezdez reviewed Feb 4, 2014
View reviewed changes

Issue mdn#1934: No longer installing PIL, so drop the insecure exception

3a8051f

Issue mdn#1934: Update repo location for django-debug-toolbar

ff7f4b0

groovecoder mentioned this pull request Feb 4, 2014

Audit and update Python dependencies #1934

Closed

groovecoder added a commit that referenced this pull request Feb 4, 2014

Merge pull request #1945 from lmorchard/death-to-kuma-lib

4ba6077

Update requirements to match kuma-lib; initial working Travis-CI tests

groovecoder merged commit 4ba6077 into mdn:master Feb 4, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update requirements to match kuma-lib; initial working Travis-CI tests #1945

Update requirements to match kuma-lib; initial working Travis-CI tests #1945

lmorchard commented Jan 31, 2014

groovecoder Jan 31, 2014

lmorchard Feb 1, 2014

groovecoder commented Feb 1, 2014

lmorchard commented Feb 1, 2014

lmorchard commented Feb 3, 2014

groovecoder commented Feb 3, 2014

lmorchard commented Feb 4, 2014

lmorchard commented Feb 4, 2014

lmorchard commented Feb 4, 2014

jezdez Feb 4, 2014

lmorchard Feb 4, 2014

jezdez Feb 4, 2014

jezdez Feb 4, 2014

lmorchard Feb 4, 2014

jezdez commented Feb 4, 2014

groovecoder commented Feb 4, 2014

groovecoder commented Feb 4, 2014

Update requirements to match kuma-lib; initial working Travis-CI tests #1945

Update requirements to match kuma-lib; initial working Travis-CI tests #1945

Conversation

lmorchard commented Jan 31, 2014

groovecoder Jan 31, 2014

Choose a reason for hiding this comment

lmorchard Feb 1, 2014

Choose a reason for hiding this comment

groovecoder commented Feb 1, 2014

lmorchard commented Feb 1, 2014

lmorchard commented Feb 3, 2014

groovecoder commented Feb 3, 2014

lmorchard commented Feb 4, 2014

lmorchard commented Feb 4, 2014

lmorchard commented Feb 4, 2014

jezdez Feb 4, 2014

Choose a reason for hiding this comment

lmorchard Feb 4, 2014

Choose a reason for hiding this comment

jezdez Feb 4, 2014

Choose a reason for hiding this comment

jezdez Feb 4, 2014

Choose a reason for hiding this comment

lmorchard Feb 4, 2014

Choose a reason for hiding this comment

jezdez commented Feb 4, 2014

groovecoder commented Feb 4, 2014

groovecoder commented Feb 4, 2014