Import DataPackages

[vitorbaptista]

• Add UI for importing
• Import the Data Package's metadata
• Import the resources, including their metadata
• Upload local resources
• Upload inline resources
• Make sure there's no way to upload a datapackage.json with a resource like { "path": "/etc/shadow" } or other local file. The paths MUST be only relative to its zip file.

[vitorbaptista]

What to do with homepage attribute? According to the specification, it refers to "URL string for the data packages web site". When exporting to a DataPackage, we set the homepage to be the ckan_url (i.e. the URL of the dataset in CKAN). It doesn't work on the other direction (i.e. we can't set the ckan_url).

It sounds like the source, but we also have the sources field. Maybe we need a more complicated rule, like:

1. If homepage exists, set source = homepage;
2. Else, set source = sources[0].web.

I'm not sure about that, though, because sources in datapackages seems to refer to the authors of the data (i.e. the author in CKAN).

[vitorbaptista]

What about the datapackage sources? As we've done in #27, the CKAN dataset author, author_email, and source becomes the first item in the sources array of the datapackage. What to do if there're multiple sources?

The solution seems to involve extras. We could simply add any remaining source into an extra sources key (or another one harder to clash with existing names), but then we have to make sure that they're a valid JSON with a list of objects.

[vitorbaptista]

There're a few issues with datapackage's keywords. They're mapped to CKAN's tags. My idea was to use the tag name as the slugified keyword and the display_name as the keyword itself. I planned this because a tag name can be only alphanumeric plus _.- characters, while a dataset keyword doesn't have these limitations. The problem is that I can't set the display_name through CKAN package_create. I created an issue about it at ckan/ckan#2781.

[vitorbaptista]

I've been trying to upload the datapackage's resources to CKAN but couldn't figure out yet. I've found a couple issues:

• The package_create() docs say that it accepts a resources list which attributes are as the ones defined on resource_create(). Then resource_create() says that it accepts a upload attribute to upload a file, which is true, but not when calling through package_create(). As far as I can see, package_create() doesn't use resource_create() at all, but model_save.package_dict_save() which calls model_save.resource_dict_save(), which doesn't accept the upload attribute. That being the case, we would have to create the package with package_create, then call resource_create on each resource (at least the ones we want to upload).
• Uploading a resource requires a cgi.FieldStorage, which seems to be created by Pylons when receiving a POST request with data. I was hoping to be able to manually create one of those and pass to CKAN, but haven't been able so far. I might need to write a custom class that inherits from cgi.FieldStorage and pass it instead.

@amercader do you know a better way? https://github.com/ckan/ckanapi is able to upload datasets because it uses HTTP requests, which I'm trying to avoid because I think it would open another can of worms.

[vitorbaptista]

The tests are currently failing because of an issue with CKAN solved in ckan/ckan#2801

[vitorbaptista]

Uploading inline resources work partially. The issue is described in #34.