build system

[holtzermann17]

Since we are caching the XHTML generated from LaTeX in the forum postings, after making any big-scale improvements to LaTeXML enhancements (e.g. if we can deal with xypic), we may want to trigger a re-build of all XHTML from the LaTeX sources.

We also typically have to render all articles whenever we import from a legacy database.

In other words, we need something like the arXMLiv build system inside PlanetMath. It could be more lightweight though. Ideally it would be accessible through the admin dashboard for convenience.

For now, our "proto-build system" is the following Drush script:

#!/usr/local/bin/drush

set_time_limit(0);
declare(ticks = 1);

global $retry;
global $node;

$retry = 1;

$result = db_select('node', 'n')
  ->fields('n', array('nid'))
  ->orderBy('n.nid', 'ASC')
  ->condition('type','article')
  ->condition('nid',34710,'>')
  ->execute();

foreach ($result as $row) {
  $node = node_load($row->nid, NULL, TRUE);
  if(isset($node->field_latex['und'][0]['preamble']) 
     && isset($node->field_latex['und'][0]['document'])) {
    $node->field_latex['und'][0]['preamble'] =
      str_replace("\usepackage{xypic}",
      "%\\usepackage{xypic}",
          $node->field_latex['und'][0]['preamble']);
    $node->field_latex['und'][0]['document'] =
      $node->field_latex['und'][0]['document'] . "%";
    // Key line...
    $node->field_latex['und'][0]['format'] = 'tex_editor';
    // dpm($node);
    pcntl_signal(SIGALRM, "signal_handler", true);
    pcntl_alarm(300);
    node_save($node);
    pcntl_alarm(0);

    drush_print("Processed: " .
       $node->field_canonicalname['und'][0]['value'] . " (" . $node->nid .
       ")   " . date('o-m-d G:i:s'));

    $retry = 1;
    // also needed (or, maybe not anymore)
    node_view($node);
  }
}

function signal_handler($signal) {
  switch($signal) {
  case SIGALRM:
    if($retry == 1) {
      node_save($node);
      drush_print("Processed: " .
          $node->field_canonicalname['und'][0]['value'] . " (" . $node->nid .
          ")   " . date('o-m-d G:i:s') . "[1 retry]");
      $retry = 0;
    } else {
      $retry = 1;
    }
  }
}

To use: make it executable, and give it the name of the site you're working as an argument, like this:

$ ./rebuild.drush @sitename

[holtzermann17]

Note: the current script is sensitive to network failures. It should really include a, say, 5 minute timer and then retry the request up to 2 more times if the time limit is reached. Otherwise go on to the next item. (Cf. http://php.net/manual/en/function.pcntl-alarm.php)

[holtzermann17]

For the build system, it would be quite useful if drutexml_cache had an nid column, so that we could quickly and easily see which nodes have XHTML generated and which do not. (I guess the idea behind NOT having an nid column was that multiple nodes with the same content will not take up multiple storage... but that like an optimization that we're not that likely to need.)

[holtzermann17]

Aparently Deyan has made a build system, but it is not integrated with Drupal yet.

[holtzermann17]

Note, I added the script to the repository, as rebuild.drush.

[dginev]

I don't think my build system will align with a Planetary rebuild utility - the arXMLiv build system has the single purpose of stress testing LaTeXML against arbitrary TeX corpora, from the perspective of improving LaTeXML.

The workflow I always had in mind is:

• Develop LaTeXML to some dev revision/release/tag
• Stress test the release against the build system, correct all relevant issues
• Release a production release/tag of LaTeXML
• LaTeXML-based applications update their LaTeXML checkout at their own convenience.
• The LaTeXML-based application triggers a reconversion of all (relevant?) documents.

Integrating my build system with Drupal is not possible for a variety of technical reasons, but also not ideal in theory. Loose-coupling is helpful to separate concerns.

[holtzermann17]

What we could potentially do is use the build system to know which documents have changed since the previous rebuild. Then we'd know which documents needed rebuilding "downstream".

... Indeed, if we ever do get an Git/SVN "frontend" for Planetary going (#68), then I think we could potentially pull changes from a similar VCS that the Build System could write to. But I don't think the build system should write directly to Planetary storage, at least not until much testing and stabilization has taken place.

Anyway, I'm in favor of one form or another of loose coupling.

[holtzermann17]

A simpler related problem: can we rebuild articles without saving an explicit new version? Answer: yes, just update the XHTML in the drutexml_cache for the given "key". So, how about rewriting rebuild.drush to do that? This would make it so that unlord doesn't have to get editing credit for editing all articles every time we want to rebuild the content. THIS part of the problem should be pretty easy -- and currently quite useful, b/c it would allow us to update to current NNexus links.