Create system for a centralized contrib repository (Composer, Github service, project.module, or ?)

[jenlampton]

In order to get Backdrop listed as a github service (so that contrib module/theme authors can register their github projects as Backdrop extensions) we'll need to write our own code for it.

Here's the repo:
https://github.com/github/github-services

Here's the documentation:
https://github.com/github/github-services/blob/master/CONTRIBUTING.md

[sirkitree]

Just so I understand, you're talking about registering a service with GitHub which would be listed in the Service Hooks that would allow any contrib repo on GitHub to do... what exactly? I.e. what form field would we have and what would be their purpose?

I ask because the Service Hooks are mainly for detecting some sort of event on a repo and then sending that information to another server/app that would then do something with it.

If I understand you correctly, a contrib repo would go to the Service Hooks section in their repo's admin UI on GitHub, select Backdrop from the options, and perhaps just check a box that says, "Register my theme/module with Backdrop". This would in turn ping one of our servers which would validate the module or theme somehow, and add it's location as an entry into the registry.

Does that sound about right?

[quicksketch]

This would in turn ping one of our servers which would validate the module or theme somehow, and add it's location as an entry into the registry.

Yep that sounds right. Registering the module with BackdropCMS.org would allow us to pull in new versions of modules when they are released. You probably know a lot more about this than we do if that's actually beneficial. This idea came up because it's the same thing jQuery uses to manage their own repository (http://plugins.jquery.com).

If it helps, you can even swipe the code that jQuery itself uses for their plugin repository (it's MIT license): https://github.com/jquery/plugins.jquery.com

We don't absolutely need to go this way, it's just a potential pattern we could use to help centralize modules.

[sirkitree]

So I'm not totally sold on having a GitHub-only type of setup like this. What would be nice is if it really didn't matter where your code is housed, more like npm or Sublime Text, it would be nice if there was simply a repository (backdrop-contrib for example which could also be on GitHub) that anyone could send a pull-request to in order to add their module or theme to the registry.

https://github.com/wbond/package_control_channel is Sublime Text's default channel and while I'm not sold on a json file per letter I feel like this would be less overhead and could even give us a decent approval process (pull requests) for adding 'official' contribs.

Thoughts?

[quicksketch]

I like your line of thinking @sirkitree. It would definitely be nice if users could host their projects anywhere, allowing freedom to use a different repository other than Gihub (e.g. Bitbucket).

it would be nice if there was simply a repository (backdrop-contrib for example which could also be on GitHub) that anyone could send a pull-request to in order to add their module or theme to the registry.

So this would be an officially curated list of modules that pulls directly from repositories? Although approving pull requests can be straight-forward when tested with Travis CI (like the Sublime repository is), this could be tedious with several modules being released every day (in a best-case scenario).

What about packaging and versions? Would we then be responsible for checking out the associate repository, generating a tarball/zip, and then hosting it? From what I've seen, almost all the Sublime examples just do checkouts directly from master... they don't have versions per-se. Every time a module comes out with a new release, how do we detect that?

[sirkitree]

If we adopted this i'd say it's a simple JSON format and we give the option of a version which must coincide with a tag, like node's package manager.

What is a package?
A package is:
a) a folder containing a program described by a package.json file
b) a gzipped tarball containing (a)
c) a url that resolves to (b)
d) a @ that is published on the registry with (c)
e) a @ that points to (d)
f) a that has a "latest" tag satisfying (e)
g) a git url that, when cloned, results in (a).

Here's an example package.json from node-github:
Note that while this includes a lot of information, only what is listed above is required.

{
  "name": "github",
  "version": "0.1.10",
  "description": "NodeJS wrapper for the GitHub API",
  "author": {
    "name": "Mike de Boer",
    "email": "[email protected]"
  },
  "contributors": [
    {
      "name": "Mike de Boer",
      "email": "[email protected]"
    },
    {
      "name": "Fabian Jakobs",
      "email": "[email protected]"
    }
  ],
  "homepage": "http://github.com/mikedeboer/node-github",
  "repository": {
    "type": "git",
    "url": "http://github.com/mikedeboer/node-github.git"
  },
  "engine": {
    "node": ">=0.4.0"
  },
  "devDependencies": {
    "oauth": "0.9.7"
  },
  "main": ".",
  "scripts": {
    "test": "node ./test/all.js"
  },
  "licenses": [
    {
      "type": "The MIT License",
      "url": "http://www.opensource.org/licenses/mit-license.php"
    }
  ],
  "readme": "# JavaScript GitHub API for Node.JS\n\nA Node.JS module, which provides an object oriented wrapper for the GitHub v3 API.\n\n## Installation\n\n  Install with the Node.JS package manager [npm](http://npmjs.org/):\n\n      $ npm install github\n\nor\n\n  Install via git clone:\n\n      $ git clone git://github.com/mikedeboer/node-github.git\n      $ cd node-github\n      $ npm install\n\n## Documentation\n\nYou can find the docs for the API of this client at [http://mikedeboer.github.com/node-github/](http://mikedeboer.github.com/node-github/)\n\nAdditionally, the [official Github documentation](http://developer.github.com/)\nis a very useful resource.\n\n## Example\n\nPrint all followers of the user \"mikedeboer\" to the console.\n```javascript\nvar GitHubApi = require(\"github\");\n\nvar github = new GitHubApi({\n    // required\n    version: \"3.0.0\",\n    // optional\n    timeout: 5000\n});\ngithub.user.getFollowingFromUser({\n    user: \"mikedeboer\"\n}, function(err, res) {\n    console.log(JSON.stringify(res));\n});\n```\n\nFirst the _GitHubApi_ class is imported from the _node-github_ module. This class provides\naccess to all of GitHub's APIs (e.g. user, issues or repo APIs). The _getFollowingFromUser_\nmethod lists all followers of a given GitHub user. Is is part of the user API. It\ntakes the user name as first argument and a callback as last argument. Once the\nfollower list is returned from the server, the callback is called.\n\nLike in Node.JS, callbacks are always the last argument. If the functions fails an\nerror object is passed as first argument to the callback.\n\n## Authentication\n\nMost GitHub API calls don't require authentication. As a rule of thumb: If you\ncan see the information by visiting the site without being logged in, you don't\nhave to be authenticated to retrieve the same information through the API. Of\ncourse calls, which change data or read sensitive information have to be authenticated.\n\nYou need the GitHub user name and the API key for authentication. The API key can\nbe found in the user's _Account Settings_ page.\n\nThis example shows how to authenticate and then change _location_ field of the\naccount settings to _Argentina_:\n```javascript\ngithub.authenticate({\n    type: \"basic\",\n    username: username,\n    password: password\n});\ngithub.user.update({\n    location: \"Argentina\"\n}, function(err) {\n    console.log(\"done!\");\n});\n```\nNote that the _authenticate_ method is synchronous because it only stores the\ncredentials for the next request.\n\nOther examples for the various authentication methods:\n```javascript\n// OAuth2\ngithub.authenticate({\n    type: \"oauth\",\n    token: token\n});\n\n// Deprecated Gihub API token (seems not to be working with the v3 API)\ngithub.authenticate({\n    type: \"token\",\n    token: token\n});\n```\n\n## Implemented GitHub APIs\n\n* Gists: 100%\n* Git Data: 100%\n* Issues: 100%\n* Orgs: 100%\n* Pull Requests: 100%\n* Repos: 100%\n* Users: 100%\n* Events: 100%\n* Search: 100%\n* Markdown: 100%\n\n## Running the Tests\n\nThe unit tests are based on the [mocha](http://visionmedia.github.com/mocha/)\nmodule, which may be installed via npm. To run the tests make sure that the\nnpm dependencies are installed by running `npm install` from the project directory.\n\nBefore running unit tests:\n```shell\nnpm install mocha -g\n```\nAt the moment, test classes can only be run separately. This will e.g. run the Issues Api test:\n```shell\nmocha api/v3.0.0/issuesTest.js\n```\nNote that a connection to the internet is required to run the tests.\n\n## LICENSE\n\nMIT license. See the LICENSE file for details.\n",
  "readmeFilename": "README.md",
  "bugs": {
    "url": "https://github.com/mikedeboer/node-github/issues"
  },
  "_id": "[email protected]",
  "_from": "github@"
}

[sun]

As outlined in #2, the above example content and structure is essentially what composer.json files are encompassing.

There might be a GitHub service implementation for Packagist already.

[RobLoach]

1. Add a composer.json

{
      "name": "backdrop/backdrop",
      "description": "Backdrop is a fully-featured content management system that allows non-technical users to manage a wide-variety of content.",
      "type": "drupal-core",
      "license": "GPL-2.0",
      "autoload": {
         "files": [
             "core/includes/bootstrap.inc"
         ]
      },
      "require": {
      }
}

1. Submit the package on Packagist: http://github.com/backdrop/backdrop
2. Now you can install it via Composer

$ curl -sS https://getcomposer.org/installer | php
$ composer.phar create-project backdrop/backdrop

1. (optional) You can now also require it in a different composer.json file:

{
      "name": "myown/package",
      "description": "It does some awesome stuff.",
      "require": {
         "backdrop/backdrop": "1.*"
      }
}

$ composer.phar install

1. (bonus) Figure out a way to install it to the base path with Composer Installers

[quicksketch]

Thanks @sun and @RobLoach, when I saw @sirkitree's description I was thinking the exact same thing.

One downside of this is that this has an enormous amount of overlap with the module .info files. Obviously the composer.json file is standarized, but it's not comprehensive of the meta data we'd need to keep track of. Would developers find using a composer.json file a better alternative to a .info (or yml file, see backdrop/backdrop-issues#78). And although this would help downloading modules, we'd still need to do the work of building a public browsing repository on BackdropCMS.org. It'd be nice that packagist could provide a downloading repository, but we'd still need to provide a similar service for browsing exclusively Backdrop modules and themes.

[RobLoach]

With Composer Installers, modules can have their own discovery:

{
  "name": "quicksketch/awesomemodule",
  "type": "drupal-module",
  "require": {
    "backdrop/backdrop": "*",
    "composer/installers": "*"
  }
}

That will install them into modules/awesomemodule, but we could change that accordingly in DrupalInstaller.php. For themes, the type becomes drupal-theme. Might be able to do some magic with Custom Install Paths too. Introduce a type of backdrop-module, etc.

Although there might be a bit of overlap, I would not consider Composer meta-information to be the same as Drupal meta-information. Composer handles things very differently than Drupal/Backdrop. Don't try mixing the two, I've tried 😉 . But hey, since the JSON performance looks much better than the .yml use, maybe we switch to .json for the meta-information.

In regards to browsing, if you visit https://packagist.org/packages/symfony/ , you'll see this returns a search for all packages with the vendor "symfony". It's also possible to search by type, package name, etc. Browsing/searching is already done for you. Remember: http://twitter.com/justingeeslin/status/379620758890491904

[mkalkbrenner]

I created a first contrib module:
https://github.com/mkalkbrenner/xmlrpc

It's tested by TravisCI:
https://travis-ci.org/mkalkbrenner/xmlrpc/builds/11832005

The .travis.yml of this module should be turned into a template for new contrib modules.

[nedjo]

The Joomla project is working on jIssues, which uses Composer and integrates with Github (and includes stuff like Vagrant/VirtualBox). It's probably at least a good model to look at--possibly something to use and extend.

[develCuy]

I love all the ideas being discussed here! Whatever is the choice at the end, please KISS.

[kreynen]

This was discussed in the core hangout today. Another example similar to http://plugins.jquery.com/ is https://civicrm.org/extensions/. That is written in D7 and provides the structure for in app installs of CiviCRM extensions from info.xml files like https://github.com/civicrm/civivolunteer/blob/master/info.xml

I don't think CiviCRM's approach is perfect, but the code is available and worth a look...

https://github.com/civicrm/civicrm-website-org/tree/master/drupal/sites/all/modules/custom

[quicksketch]

I filed an issue similar to this one, but with a particular focus on adding version information to GitHub-hosted projects at #51.

[quicksketch]

It's been a while since we had updates here. For now we'll pursue the path of least resistance, which is simply porting Project module (with the sub-module "project_release.module"). We need the the functionality provided by project_release to power our update server. While we could rebuild an update server that pulled from a different repository, this is definitely the most direct approach to getting our update server running.

Project module includes a lot of unnecessary code for our purposes however. We don't need direct integration with VCS back-ends, issue management, maintainership, or particularly granular permissions. I expect we'll want to trim out a significant portion of the module in our port.

[sirkitree]

💃

[docwilmot]

🎶 👍

[jenlampton]

Initial designs for this are looking fabulous, we're now moving on to designs for the module node itself.

[quicksketch]

The only part left in the GitHub integration is to automatically create the project nodes themselves when a new project is created.

This is now done too. Only I found that we can only really automatically create project nodes at release time, not at the time of repository creation, as repositories usually will be empty (with maybe a README) at the time of creation on GitHub. It doesn't do us a lot of good to create the project on BackdropCMS.org with so little information.

After that, we need to get the update XML working

This is working too! We've got update XML being generated. I can even point my local Backdrop install at itself to see the "available updates" that I've created as project releases locally:

and Project Usage module updated.

I've started working on this. Project Usage is comparatively a very simple module. I hope to have it working well later this week.

We're nearly at a point where we can start implementing on backdropcms.org. It might be a good idea to get it up and running, doing all the packaging, and start seeding the site with project and project release nodes while we hammer out the designs. Then we'll have content available on which we can begin theming, and it'll flesh out the project listing for when we take it live.

[klonos]

Great job!! Thanx Nate.

Where are modules/themes going to be listed? Are we to create a new "Extend" section in the main site?

[biolithic]

Great job!! Thanx Nate. !!

On Tue, Apr 14, 2015 at 2:16 PM, Gregory Netsas [email protected]
wrote:

Great job!! Thanx Nate.

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[quicksketch]

Where are modules/themes going to be listed? Are we to create a new "Extend" section in the main site?

Initially our designs call for adding Modules, Layouts, and Themes as top-level items in the left sidebar. We'll also kill the "Home" link at the time. I think there's a good chance we'll need to put these under a single top-level page, but since the number 1 reason I visit Drupal.org (other than issues) is to find and download modules, and I expect it will be this for Backdrop as well. So I'm not opposed to making it top-level until space becomes a problem.

[develCuy]

@quicksketch, you are following Drupal standards. How about doing something new and different? Community tried to give themes and modules a better position in d.o. Also, the usage stats are obscure, just a few people have access to them (I mean the full data, not "usage stats" page).

Also, Themes are a different world than Modules, they both deserve their own identity in terms of graphic design and functionality of respective section. For example, nice and big thumbnails of each theme in the list.

Well, there is a lot of ideas that come to my mind, but the root of everything is: Let's innovate a bit here, since there is room, time and energy for it in this particular moment.

My 2c.

[quicksketch]

@develCuy: So +1 for top-level links?

Also, the usage stats are obscure, just a few people have access to them (I mean the full data, not "usage stats" page).

I think usage stats are totally public, but perhaps not well-linked on drupal.org. From the looks of project.module, the full history of stats is actually discarded after a time-period, so my guess is that any missing stats have actually been removed from the DB.

It might be neat to give usage stats a more prominent location, but initially they won't be very exciting until we've collected stats for at least a few months. But totally, I see what you're saying. We don't have to do the same thing. If anything, I'd like to see our minimalist approach to content used to our advantage to make the more important content easier to find.

[quicksketch]

Oh yeah! Our first auto-generated project nodes and releases are up on backdropcms.org!

Project module: https://backdropcms.org/node/108
Form block module: https://backdropcms.org/node/110

Apparently we don't have Pathauto on BackdropCMS.org. We can add that shortly or wait for 1.1.0.

Unfortunately, auto-generating projects requires a minor change to .info files. Because our contributed project group is a mishmash of themes, modules, and layouts, we have no way of identifying which is which. Given a module and theme .info file, there's no indication what type of project it is. Layouts can be inferred by the presence of "regions" in the .info file, but that's it.

So... in order to properly create a node on BackdropCMS.org, projects must specify a type in their .info files like such:

name = Project
description = Base module for identifying nodes as projects (or relating to projects).
type = module
package = Project
backdrop = 1.x

Then when BackdropCMS.org gets the initial ping to create a matching node, it knows which type of node to create (since themes and modules have different fields in their node types).

What I'd like to do now is get a bunch of projects creating initial releases (and including these new .info file lines) to start populating BackdropCMS.org with contrib projects. And now that the technical requirements are met, we can start implementing the designs from @dariusgarza.

[docwilmot]

👏
Great news.
Will 'release' all my ports shortly.

[sirkitree]

Release nodes should have a link back to the github project.

[docwilmot]

This also doesnt seem to have tags or author data. A "See more themes by Darius", or "List Media modules", or "Filter by fullwidth layouts" would be impossible then?

[quicksketch]

Hi guys, great feedback.

Release nodes should have a link back to the github project.

I thought this as well while doing the port. Then I realized that the breadcrumb contained a link back to the project. But we don't have breadcrumbs on BackdropCMS.org, doh! I thought the link was a good idea though, I'll put one in.

This also doesnt seem to have tags or author data. A "See more themes by Darius", or "List Media modules", or "Filter by fullwidth layouts" would be impossible then?

The only information we have right now is the data we can get from GitHub. Since GitHub doesn't have any kind of categorization, we can only work with information we can pull from the project itself. We'll start having on-site registration in the near future, allowing users to customize project nodes more extensively, but I'd love to get as much data as possible from GitHub.

This made me realize that we should automate the creation of the thumbnail image for themes. These already include a "preview.png", which we should pull in and attach to theme projects.

We could also pull in categories from the .info file (or at least the current "Group" listing) but I think we'd want to curate that list of categories on the site.

Author data can also be pulled in from GitHub username, but we don't have information about that user on BackdropCMS.org. Maybe we should bind GitHub usernames to BackdropCMS.org UIDs so we could directly make this connection?

[docwilmot]

The only information we have right now is the data we can get from GitHub

Maybe start adding more metadata in .info files and also adding tags to info files?

Author data can also be pulled in from GitHub username
Maybe we should bind GitHub usernames to BackdropCMS.org UIDs

But we dont have any way to know who's maintaining a repo, except parsing the README, which seems a bit not so nice.

Is there a plan to be able to add a project by directly creating a node on BD.org? Or will it only via this automatic release-reading process?

[quicksketch]

But we dont have any way to know who's maintaining a repo, except parsing the README, which seems a bit not so nice.

Yeah, the only thing we'll be able to do accurately is assign the owner of the node, assuming we start tracking GitHub usernames on backdropcms.org.

Is there a plan to be able to add a project by directly creating a node on BD.org?

Yes, as soon as we start allowing logins, you should be able to create/edit projects (and releases) directly on the site as well. That might be a temporary solution for distributed repositories instead of all needing to be in the backdrop-contrib group.

[klonos]

So +1 for top-level links?

Yes, 👍 form me.

Oh yeah! Our first auto-generated project nodes and releases are up on backdropcms.org!

🎉 🎈 👏

Also 👍 for linking back to the github projects and author (maintainer) data. What I would also like to see implemented in BackdropCMS.org is #59

Apparently we don't have Pathauto on BackdropCMS.org. We can add that shortly or wait for 1.1.0.

Add it now and test the upgrade path to 1.1.0 when it's out. We could use the SEO boost from having the actual project names in the URL 😉

[docwilmot]

What about adding data to .info files?

tags[] = media
tags[] = fields
tags[] = user-interface

and

maintainers[] = @quicksketch
maintainers[] = @klonos

together with from this issue:

source = url
support = url
documentation = url

Any technical reasons why not (apart from someone has to write the code to make this useful onB.org of course). I'm considering the 'inconvenience' of uploading to GitHub and still then having to go and edit a node on B.org. If it can all be done automagically, seems better no?

[kreynen]

+1 for any process that doesn't require logging into to multiple sites

[docwilmot]

How do updates work? If I edit the info node on B.org, does that update the README on GitHUb, or vice versa?

I'm asking mainly because I was holding off on creating releases until a response to the idea I proposed above, as I would want to have to release them all over again.

But if updates would be picked up, I may as well go on.

[quicksketch]

If I edit the info node on B.org, does that update the README on GitHUb, or vice versa?

Editing the description is locked on B.org if the README is being pulled from GitHub. Any updates to the README.md on GitHub are picked up immediately by B.org upon pushing them to GitHub. The option to edit the node on B.org is intended to be used only if you want an entirely different description from your README.md, in which case syncing becomes disabled. Here's a screenshot from the node edit form:

So in short, there's no reason to delay. All your changes to the README will be picked up immediately.

[docwilmot]

Thanks, but I actually was actually intending to ask about changes to the .info file, not the README, sorry. But I suspect that must be similar?

[quicksketch]

Ah! Yeah well the only thing we pull from the .info file at this point is the name of the project. All of these options for pulling categories, etc. hasn't been implemented yet.

[quicksketch]

Just a note here that I've closed #22 (provide an update server). As noted in the 1.1.0 Release mailer, our update server is now enabled!

We've now got 45 projects listed on BackdropCMS.org. We just need to get the views created for the project listing and start styling them.

[quicksketch]

Let's close this issue for now and file new issues for any changes or exploring new options.