Upgrading to 1.2beta1 causes data loss

[ingenieroariel]

Method to replicate:

1. Install GeoNode (any version from 1.1 to 1.2alpha2)
2. Upgrade to 1.2beta1
3. See your data dissapear :/

I believe this line would be the one to blame but will investigate further.

https://github.com/GeoNode/geonode/blob/master/shared/package/install.sh#L49

Ariel.

For the record, this is the kind of error the layer detail page gives:

FailedRequestError at /data/geonode:FloodProneAreasPoly1_WGS84

No store found in geonode @ http://geonode.openstreetmap.co/geoserver/rest/workspaces/geonode.xml named: FloodProneAreasPoly1_WGS84


Traceback:
File "/var/lib/geonode/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  111.                         response = callback(request, *callback_args, **callback_kwargs)
File "/var/lib/geonode/src/GeoNodePy/geonode/maps/views.py" in layer_detail
  815.         "GEOSERVER_BASE_URL": settings.GEOSERVER_BASE_URL
File "/var/lib/geonode/lib/python2.7/site-packages/django/shortcuts/__init__.py" in render_to_response
  20.     return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/loader.py" in render_to_string
  171.         return t.render(Context(dictionary))
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in render
  140.             return self._render(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in render
  823.                 bit = self.render_node(node, context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  123.         return compiled_parent._render(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in render
  823.                 bit = self.render_node(node, context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  62.             result = block.nodelist.render(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in render
  823.                 bit = self.render_node(node, context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/debug.py" in render_node
  74.             return node.render(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/debug.py" in render
  84.             output = self.filter_expression.resolve(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in resolve
  571.                 obj = self.var.resolve(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in resolve
  721.             value = self._resolve_lookup(context)
File "/var/lib/geonode/lib/python2.7/site-packages/django/template/base.py" in _resolve_lookup
  754.                         current = getattr(current, bit)
File "/var/lib/geonode/src/GeoNodePy/geonode/maps/models.py" in resource
  994.             store = cat.get_store(self.store, ws)
File "/var/lib/geonode/lib/python2.7/site-packages/geoserver/catalog.py" in get_store
  176.               raise FailedRequestError("No store found in " + str(workspace) + " named: " + name)

Exception Type: FailedRequestError at /data/geonode:colmun3corregido
Exception Value: No store found in geonode @ http://geonode.openstreetmap.co/geoserver/rest/workspaces/geonode.xml named: colmun3corregido

[ingenieroariel]

Below is a potential methodology to debug the issue:

The goal is to make sure there is enough information after the removal part to restore geoserver. The debugging will be done in two parts.

• Part one: Removing geonode and trying with a vanilla 2.1.x geoserver pointed at the data dir. All layers should be there.
• Part two: After verifying the data is there, install the same version of geonode and see if all is good

Once that is done, it makes sense to retry the upgrade.

1. Install geonode

sudo add-apt-repository ppa:geonode/snapshots
sudo apt-get install geonode

sudo pip install geonode-extract
geonode-extract http://demo.geonode.org -d demodata -u admin -p admin -i -v
geonode importlayers demodata -i

sudo apt-get remove geonode

1. Check data dir health

# Get a geoserver.war from the download page and put it in tomcat

# check layers are there

# remove geoserver

1. Install geonode again

sudo apt-get install geonode

1. Check layers are there
2. Go back to the upgrade problem, if all the above works, there is a chance upgrade works too.

[simod]

Reinstalling the same geonode is fine. My test was on ubuntu 12.04 from ppa:geonode/snapshots to ppa:geonode/snapshots.

[simod]

Did a new test on Ubuntu 10.04:

• installed geonode/release and a layer
• removed geonode
• installed geonode/snapshots

The initial error was related to a wrong version of httplib2 used by gsconfig, I think Ariel is already dealing with this.

After solving it the site is still completely broken, the error looks to be related of incompatibility between django 1.4 and python 2.6.5. In the threading module precisely.

And a deprecation warnings :

/var/lib/geonode/lib/python2.6/site-packages/django/views/generic/simple.py:8: DeprecationWarning: Function-based generic views have been deprecated; use class-based views instead.
DeprecationWarning
/var/lib/geonode/lib/python2.6/site-packages/django/views/generic/list_detail.py:10: DeprecationWarning: Function-based generic views have been deprecated; use class-based views instead.
DeprecationWarning
/var/lib/geonode/lib/python2.6/site-packages/django/template/defaulttags.py:1235: DeprecationWarning: The syntax for the url template tag is changing. Load the url tag from the future tag library to start using the new behavior.
category=DeprecationWarning)

However the geoserver data dir is still in place intact.

[simod]

New test on ubuntu 12.04:

• installed geonode/testing and uploaded a layer
• removed geonode
• installed geonode/snapshots

There is an error related to the database password and ownership, with testing the geonode db belongs to 'postgres', with snapshots the user also has to modify the password in the local_settings.py and then update the 'geonode' user in postgres.

Once solved this everything else is ok and the layer is in place correctly.

Hope this helps.

[simod]

Until now I could not reproduce the bug. The last test to do is from /release to /testing on 10.04 but since from /release to /snapshot the data was not wiped I'm wondering whether this still makes sense.

Any comments?

[ingenieroariel]

/release (1.1.1) to /testing (1.2beta1) does not really make sense. Like
you said, the important one is /release (1.1.1) to latest master (or
1.2beta2 in snapshots).

Thanks for making so much progress on this, your finding about the data dir
being present and recoverable is key.

On Thu, Sep 6, 2012 at 5:23 AM, Simone Dalmasso [email protected]:

Until now I could not reproduce the bug. The last test to do is from
/release to /testing on 10.04 but since from /release to /snapshot the data
was not wiped I'm wondering whether this still makes sense.

Any comments?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/334#issuecomment-8326994.

[simod]

Since I was not convinced by the test on ubuntu 10.04 from /release to /snapshots, I did another try(new vm). Here the full report:

• install geonode/release
• upload a layer
• remove geonode
• install geonode/snapshots
• change the geonode db owner to geonode and change the password both in postgres and local_settings
• clean up httplib2 and install the 0.7.4
• set the geoserver context params (GEONODE_BASE_URL and GEOSERVER_DATA_DIR) because they were not set
• restart tomcat and go to geoserver and enable the geonode_imports store because was disabled
• do a syncdb because was not done
• remove /var/lib/tomcat/webapps/geonetwork (because the login was broken), download a fresh war and deploy it
• do updatelayers

and voilà, GeoNode 1.2 is up with all the layers in and perfectly running.

It's a bit tricky but definetly no data have been lost.

Cheers

[ingenieroariel]

This is impressive, that list of steps is what we needed to be able to
include any workarounds in the installer.

Many of the steps you mention may by themselves be the cause of the missing
data (database ownership, postgres_imports not being enabled).

Very happy to see progress on this, the next step is to create packages
from master (after the gsconfig 0.6.1 release that deals with httplib2
versioning), set up a local apt repo (as described here [1]) and try a
direct upgrade (that should avoid the need to set the password or do a
manual syncdb). It should now be a matter of re-creating the packages and
fixing the items in your list until it is able to upgrade from 1.1.1 on
it's own. Then we can mark it as 1.2RC1 and move on with the release train.

Thanks again for taking on this challenge, I know firsthand how time
consuming it is.

[1]
http://www.hiddentao.com/archives/2011/06/27/setting-up-a-simple-ubuntu-apt-repository/

On Sat, Sep 8, 2012 at 5:15 AM, Simone Dalmasso [email protected]:

Since I was not convinced by the test on ubuntu 10.04 from /release to
/snapshots, I did another try(new vm). Here the full report:

• install geonode/release
• upload a layer
• remove geonode
• install geonode/snapshots
• change the geonode db owner to geonode and change the password both
  in postgres and local_settings
• clean up httplib2 and install the 0.7.4
• set the geoserver context params (GEONODE_BASE_URL and
  GEOSERVER_DATA_DIR) because they were not set
• restart tomcat and go to geoserver and enable the geonode_imports
  store because was disabled
• do a syncdb because was not done
• remove /var/lib/tomcat/webapps/geonetwork (because the login was
  broken), download a fresh war and deploy it
• do updatelayers

and voilà, GeoNode 1.2 is up with all the layers in and perfectly running.

It's a bit tricky but definetly no data have been lost.

Cheers

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/334#issuecomment-8385918.

[ingenieroariel]

Thank Simone for taking care of this.