Deduplicating deploy scripts

[domenic]

Related to #202, it's a bit annoying for every repo's commit history to contain a bunch of meta commits every time we want to update the deploy process. And, code duplication in general is bad.

Ideally we'd use the .travis.yml to set config variables:

• DEPLOY_USER
• TITLE
• SHORTNAME (=> current LS_URL, COMMIT_URL_BASE, BRANCH_URL_BASE, SERVER, WEB_ROOT)
• INPUT_FILE (or, just standardize on a single filename, e.g. index.bs or specshortname.bs... would lose a lot of Git history though, so probably not worth it)
• FILES_GLOB (to allow copying extra files, like images or encoding stuff)
• EMU_ALGIFY (boolean used by streams and console to run through emu-algify)

I guess SHORTNAME could be inferred from some git commands to get the current repository and INPUT_FILE could be inferred to be the only .bs file in the repo.

We could then I guess have a simple one- or two-liner in Travis that curls down the deploy script from whatwg/meta and runs it.

I might try prototyping this early next week; feel free to start without me though (a good first pass would probably leave out FILES_GLOB and EMU_ALGIFY)

[domenic]

We may also be able to roll the validator stuff into the deploy script

[annevk]

I guess the only thing that would go away with this is local deployment. I guess that's not a big deal?

[annevk]

I tried something at whatwg/url#205 but I'm again hitting the limitations of Travis documentation.

[annevk]

Got it to work after taking a break. Things I'd like someone else to take on:

• Making INPUT_FILE default to "$SHORTNAME.bs" if not given.
• Performing HTML checking in deploy.sh
• Deciding where to host the shared script and put it there. resources.whatwg.org?

[domenic]

Here would be my to-do for early next week, building on your work:

• Making INPUT_FILE default to "find the single .bs file in this directory"
• Making SHORTNAME default to the Git repo name
• Performing HTML checking, including downloading the checker, in deploy.sh
• Hosting the shared script on either resources.whatwg.org or whatwg/meta
• Hosting the known_hosts file in the same place and curling it down, instead of writing it out inside deploy.sh
• Making copying of .htaccess optional (i.e. the build should not fail if the repo does not have .htaccess checked in)
• Figuring out how to get some kind of local deploy working again (e.g. the Makefile copies down the deploy script)

I think whatwg/meta is better than resources.whatwg.org probably... not sure though, both are kind of overextending their boundaries. Maybe we should create a whatwg/build.

[annevk]

What is the advantage of loading known_hosts each time? As long as we have a single server, it will only need a single entry.

[domenic]

What is the advantage of loading known_hosts each time?

I don't understand this question. Why do you think my version is different from your version? Your version codes it into the build-script; mine allows it to be in a separate file. But they both load known_hosts the same number of times. It just seems cleaner to not inline files into other files with echo statements.

As long as we have a single server, it will only need a single entry.

Right, "as long as we have a single server". But we don't: we have two right now (one for streams/console/a couple others, and one for everything else). And in general IMO it will be simpler to list all the hostnames in the file, with their corresponding keys, as that will make it easier to shift between servers in the future.

[annevk]

Well, if you curl known_hosts from the build script, that's an additional fetch for Travis. If you inline it you only need to fetch the build script. And I'm pretty sure we have one server, just different users on that server.

[domenic]

Hmm, I see. I'm not really concerned about multiple fetches, but maybe given that we have a single server it's simpler to just inline it... it's more of a pain to dig through the file and update, I guess, but oh well.

[annevk]

If you really want a distinct resource, we could also use zip/unzip to avoid two fetches (until HTTP/2 is everywhere anyway).

[domenic]

whatwg/url#205 is almost ready to go as a first pass. Still to-do is figuring out where to put the deploy.sh script, and changing the Makefile to allow local deploy. (The latter depends on the former.)

After that we can start using it on the simple repos (i.e. those that only have the main spec file and maybe .htaccess). Then we can start working on the stuff mentioned in the OP to allow it to be used in more repos (e.g. those containing other files).

[domenic]

OK, this should be ready for review:

• Add common deploy script resources.whatwg.org#41
• deduplication attempt url#205

I also included editorconfig and gitattributes for URL where I was there, in the spirit of minimizing the number of meta commits.

[annevk]

I'm not sure who would be best to review since you're the person with the most knowledge of this. Maybe @sideshowbarker can look if the shell script looks reasonable?

I'll try to have a look anyway to see if I can spot anything.

[sideshowbarker]

For the record, this all looks fine to me (sorry for not having responded here earlier).