Skip to content

Latest commit

 

History

History
2192 lines (1771 loc) · 65.5 KB

devdoc.md

File metadata and controls

2192 lines (1771 loc) · 65.5 KB

Nog Developer Documentation

By Steffen Prohaska

Package nog-error

The package nog-error helps handling errors.

Meteor provides a mechanism to prevent server-side errors from being sent to the client: It sends Meteor.Error as is. Errors of a different type, however, are only reported to the server log. Either a generic error 500 is sent to the client instead or the content of the field sanitizedError if it contains a Meteor.Error.

nog-error provides functions to maintain four information streams: {full, sanitized} x {user, developer}. reason contains information that can be displayed to users as is. details contains information for developers. The error type NogError.Error is used on the server. The sanitizedError is reported to the client.

Errors are stored upon creation to the Mongo collection nogerror.errorLog with a TTL index (see http://docs.mongodb.org/manual/core/index-ttl/) that removes error documents after some time (default: 5 days). Only server-side errors are currently stored. Errors have a short token, which can be useful to retrieve them from the log:

db.nogerror.errorLog.findOne({token: 'ma4ddq'})

Errors are created from a spec. It is an open question whether we should use many detailed specs or use fewer general purpose specs and report details in the reason. See nog-error-specs.coffee for a list of error specs.

nog-error also provides a default error handler and a template for displaying client-side errors.

NogError.createError(spec, context) (anywhere)

createError(spec, context) creates an error object. It uses the general structure from spec and the situation-specific information from context. If context has a field cause, it will be used to derive an error history, which simply is an array of the previous causes. The error object is of type NogError.Error with a sanitizedError of type Meteor.Error that would be reported to the client.

spec is usually one of the specs that are defined in nog-error-specs.coffee. The full developer documentation contains a list in a separate section below. The format of spec, which is needed to define a new error, is described below.

context is an object that may be used to provide information about the context in which the error occurred. The context information will be used when creating the error object and also be stored in the error log:

  • reason (String, optional): A message for the user that overrides the default message from the specs.
  • details (String, optional): A detailed message to developers that overrides the default message from the specs.
  • cause (Error, optional): Will be used to derive an error history.
  • Some specs require further fields that are used to construct the messages.

Usage examples:

{
  ERR_BLOB_UPLOAD
  ERR_LOGIC
  ERR_UNKNOWN_USERID
  createError
  nogthrow
} = NogError


fn = ->
  ...
  if not found
    nogthrow ERR_UNKNOWN_USERID, {uid}
  ...
  nogthrow ERR_LOGIC, {reason: 'Unknown entry type.'}


someAction (err, res) ->
  if err
    @onerror createError ERR_BLOB_UPLOAD,
      cause: err
      reason: "
          Failed to get upload URL for part #{part.partNumber}; aborting
          after #{@nTries} tries.
        "

New error codes are defined (by convention in nog-error-specs.coffee) with a spec object with the following fields:

  • errorCode (String, by convention all uppercase ERR_<topic>_<details>) identifies the type of error.
  • statusCode (Number) is a HTTP status code that is used if the error is reported via HTTP (from the REST API).
  • reason (String | (context) -> String): Text that explains the reasons (for a user); either static or generated by a function.
  • details (String | (context) -> String): Text that explains the details (for a developer); either static or generated by a function.
  • contextPattern (Match Pattern): A Meteor match pattern that is used to validate the context object. A warning is reported to the console, if the pattern does not match.
  • sanitized (null | 'full' | Object) controls how the sanitized error for the client is constructed. For null, an unspecified error with code NOGERR will be used. For full, the complete information that is available on the server will be used. sanitized can also be an Object with the following optional fields {errorCode: String, reason: String | (context) -> String, details: String | (context) -> String}) to control the details.

Example specs:

  {
    errorCode: 'ERR_APIKEY_DELETE'
    statusCode: 403
    sanitized: 'full'
    reason: 'Failed to create API key.'
  }
  {
    errorCode: 'ERR_S3_CREATE_MULTIPART'
    statusCode: 502
    sanitized: null
    reason: 'Failed to create S3 multipart upload.'
    details: (ctx) -> "
        Failed to call createMultipartUpload with S3 bucket `#{ctx.s3Bucket}`,
        and object key `#{ctx.s3ObjectKey}`.
      "
    contextPattern:
      s3Bucket: String
      s3ObjectKey: String
  }

NogError.nogthrow(spec, context) (anywhere)

nogthrow() calls createError() and throws the error.

Usage example:

{
  nogthrow
  ERR_BLOB_ABORT_PENDING
} = NogError

someFunction = (opts) ->
  try
    doSomething()
  catch cause
    nogthrow ERR_BLOB_ABORT_PENDING, {sha1: opts.sha1, cause}

NogError.defaultErrorHandler(err) (anywhere)

An error handler that stores the error for {{> errorDisplay}}.

Example usage with method call on client:

aFunction = ->
  ...
  Meteor.call 'something', (err, res) ->
    if err?
      return NogError.defaultErrorHandler(err)
    ...

Packages typically call the handler indirectly via a configurable hook:

NogBlob =
  onerror: NogError.defaultErrorHandler
  ...

aFunction = ->
  ...
  Meteor.call 'something', (err, res) ->
    if err?
      NogBlob.onerror(err)

{{> errorDisplay}} (client)

A Meteor template that displays the errors that have been reported to the error handler.

Error specs

See nog-error-specs.coffee for a list of error specs.

Package nog-settings

nog-setting helps managing Meteor settings.

defSettings({ key, val, help, match }) (server)

defSettings() helps managing Meteor settings. key is a dot path in Meteor.settings. val is the default value that will be used if none is provided by the environment. help is a short help text, starting with the last part of <key> by convention. match is a Meteor check match pattern to validate the settings value.

The recommendation is to use defSettings() in a separate file nog-<package>-settings.js to define all settings early during package initialization.

Example nog-<package>-settings.js:

import { defSettings } from 'meteor/nog-settings';

defSetting({
  key: 'public.upload.concurrentUploads',
  val: 3,
  help: `
\`concurrentUploads\` limits the number of concurrent file uploads from
a browser.
`,
  match: matchPositiveNumber,
});

Package nog-access

nog-access provides access control that is inspired by AWS:

Access is determined from a list of statements such as:

{
  principal: 'role:users'
  action: 'nog-blob/upload'
  effect: 'allow'
}
{
  principal: /// ^ username : [^:]+ $ ///
  action: 'nog-content/create-repo'
  effect: (opts) ->
    userName = opts.principal.split(':')[1]
    if userName is opts.ownerName
      'allow'
    else
      'ignore'
}
{
  principal: 'role:users'
  action: 'nog-blob/upload'
  effect: (opts) ->
    if not config.uploadSizeLimit
      'ignore'
    else if config.uploadSizeLimit is 0
      'ignore'
    else if not opts?.size?
      'ignore'
    else if opts.size <= config.uploadSizeLimit
      'allow'
    else
      {
        effect: 'deny'
        reason: "
            Upload is larger than the size limit of #{config.uploadSizeLimit}
            Bytes.
          "
      }
}

Access control uses the roles package alanning:roles. The current user is expanded to an array of principals ['username:<username>', 'userid:<userid>', 'role:<role.0>', 'ldapgroup:<group.0>', '...], and each expanded principal is tested against the access statements. Access is granted if any of the statements has the effect: 'allow' and no statement has the effect: 'deny'.

Logged-in users that have no role assigned are tested as principals ['username:<username>', 'userid:<userid>', 'guests'].

Logged-out connections are tested as principal ['anonymous'].

principal can be a string (exact match) or a regular expression.

effect can be a function (opts) -> {effect: 'access' | 'deny' | 'ignore', reason: String} that is evaluated on the opts that are passed to the access check functions. The Meteor user object is available as opts.user if a user is known. The original opts passed to checkAccess() cannot contain a field user.

The list of access control statements can be manipulated with removeStatements() and addStatement() (see below). This should only be done during startup. It is not yet clear whether we will maintain most statements centrally in the default list or use addStatement() to add them if needed.

If the user doc contains user.scopes, a special pre-check will be performed whether the action is permitted. opts.scopes is an array of objects {action: String, opts: Object}. The access check action and opts are compared against each scope. An equality match for one scope is required for the access check to proceed to the statement processing phase. Access is denied otherwise.

Usage example:

NogBlob =
  checkAccess: ->

## Use nog-access if available (weak dependency).
if Meteor.isServer
  if (p = Package['nog-access'])?
    console.log '[nog-blob] using nog-access default policy.'
    NogBlob.checkAccess = p.NogAccess.checkAccess
  else
    console.log '
        [nog-blob] default access control disabled, since nog-access is not
        available.
      '

Meteor.methods
  'startMultipartUpload': (opts) ->
    check opts, {name: String, size: Number, sha1: isSha1}
    if not Meteor.isServer then return
    NogBlob.checkAccess Meteor.user(), 'nog-blob/upload',
      _.pick(opts, 'size')
    ...

NogAccess.checkAccess(user, action, opts) (server)

checkAccess(user, action, opts) works similar to Meteor.check(). user may be an object or an user id or null. checkAccess() throws if access is denied or simply returns if access is granted. checkAccess() can be used for access control in Meteor methods, publish functions and REST request handlers.

Example use in method:

Meteor.methods
  'startMultipartUpload': (opts) ->
    ...
    NogBlob.checkAccess Meteor.user(), 'nog-blob/upload', opts
    ...

Example use in publish function:

Meteor.publish 'nog-blob/blobs', (sha1s) ->
  NogAccess.checkAccess @userId, 'nog-blob/upload', {sha1s}
  ...

Since an exception is thrown if access is denied, the subscription is terminated in this case. Access deny errors will be reported to the client's onStop() subscribe callback. Example:

Meteor.subscribe 'nog-blob/blobs', [],
  onStop: (err) ->
    console.error err

The publish function will not be re-run by the server if the logged-in user changes. The client code needs to explicitly call Meteor.subscribe() again. If this is a problem, testAccess(), which returns false if access is denied, may be the better alternative. Example:

Meteor.publish 'nog-blob/blobs', (sha1s) ->
  if not NogAccess.testAccess @userId, 'nog-blob/upload', {sha1s}
    return null
  ...

Example use in REST API handler:

NogAccess.checkAccess req.auth?.user, action, opts

req.auth.user is added by nog-auth during signature verification.

NogAccess.testAccess(user, action, opts) (server)

testAccess(user, action, opts) works similar to checkAccess() but returns true (access) or false (deny) instead of throwing and exception.

NogAccess.testAccess(action, [opts], [callback]) (client)

testAccess(action, opts) on the client works similar to testAccess() on the server but returns null if the result is not yet available, because the response from the server is pending. The result is reactively updated when the server call completes. If callback is provided, it is called with the result when it becomes available.

testAccess_ready(action, opts) can be used to test whether the result is available.

Template helper example:

Template.repoView.helpers
  mayModify: ->
    ...
    NogAccess.testAccess 'nog-content/modify', {ownerName, repoName}

Flow router middleware example:

requireUserOrGuest = (path, next) ->
  NogAccess.testAccess 'isUser', (err, isUser) ->
    NogAccess.testAccess 'isGuest', (err, isGuest) ->
      if isUser or isGuest
        next()
      else
        next('/sign-in')

{{testAccess action [kwopts]}} (client)

{{testAccess action}} can be used in a template to test access. It calls NogAccess.testAccess(action).

{{testAccess_ready action}} can be used to test whether the test results is available.

Example:

  if testAccess_ready 'nog-blob/upload'
    if testAccess 'nog-blob/upload'
      +uploadView
    else
      | You cannot upload files.
  else
    | loading...

Keyword arguments are passed as an opts object to NogAccess.testAccess(). Example:

  if testAccess 'nog-content/modify' ownerName='foo' repoName='bar'
    +repoView
  else
    | You cannot modify the repo.

It may be clearer to write a helper function that performs the toggle check if an opts object is needed.

{{testAccess_ready action [kwopts]}} (client)

{{testAccess_ready action [kwopts]}} tests whether the test result is available.

NogAccess.configure(opts) (server)

configure() updates NogAccess.config with the provided opts:

  • uploadSizeLimit (Number >= 0, default: Meteor.settings.public.upload.uploadSizeLimit or 0) limits the allowed blob upload size. Use 0 to disable the limit.

NogAccess.config (server)

The active configuration.

NogAccess.addStatement(statement) (server)

addStatement(statement) adds a statement to the access control list. See the introduction above for the format of statement.

NogAccess.removeStatements(selector) (server)

removeStatetments(selector) removes the statements that match selector from the access control list. selector can contain only a single field action. Statements whose action matches (exact string comparison) are removed.

Package nog-rest

nog-rest implements server-side routing for a REST API. It uses signature-based authentication if the package nog-auth is available.

Routes are registered with NogBlob.actions(prefix, actions) (see details below).

Access check of actions should be implemented with NogAccess.checkAccess() from package nog-access.

Links to resources should be returned as JSON objects with an href member and other alternative identifiers, such as an id or a sha1. Each resource should contain a self reference in member _id.

Usage example:

if Meteor.isServer
  actions = NogBlob.api.blobs.actions()
  NogRest.actions '/api/blobs', actions

With NogBlob.api.blobs.actions(), for example, implemented as follows:

class BlobsApi
  constructor: (opts) ->
    @blobs = opts.blobs

  actions: () ->
    [
      { method: 'GET', path: '/:blob', action: @get_blob }
      { method: 'GET', path: '/:blob/content', action: @get_blob_content }
    ]

  # Use `=>` to bind the actions to access this instance's state.
  get_blob: (req) =>
    {params, baseUrl} = req
    params = _.pick params, 'blob'
    check params, { blob: isSha1 }
    action = 'nog-blob/GET-blob'
    NogAccess.checkAccess req.auth?.user, action, req.params
    blob = @blobs.findOne params.blob
    if not blob?
      nogthrow ERR_BLOB_NOT_FOUND, {blob: params.blob}
    res = _.pick blob, 'size', 'status', 'confirmations', 'sha1'
    res._id =
      id: blob._id,
      href: Meteor.absoluteUrl(baseUrl[1..] + '/' + blob._id)
    res.content =
      href: share.getSignedDownloadUrl
        sha1: params.blob
        filename: params.blob + '.dat'
    res

  ...

Example response JSON:

{
    "data": {
        "_id": {
            "href": "http://localhost:3000/api/blobs/31968d2e8b58e29e63851cb4b340216026f11f69",
            "id": "31968d2e8b58e29e63851cb4b340216026f11f69"
        },
        "confirmations": [
            {
                "date": "2015-04-27T10:02:31.313Z",
                "message": "..."
            }
        ],
        "content": {
            "href": "https://..."
        },
        "sha1": "31968d2e8b58e29e63851cb4b340216026f11f69",
        "size": 11,
        "status": "available"
    },
    "statusCode": 200
}

NogRest.actions(prefix, actions) (server)

NogBlob.actions(prefix, actions) adds routes that start with prefix. actions is an array of {method: String, path: String, action: callback}. prefix and path use Express-style syntax as describe at https://github.com/component/path-to-regexp.

The action callback(req) receives an HTTP request object, parsed as usual: URL query in req.params, parsed JSON body in req.body. In addition to the usual fields, req.auth.user contains a Meteor user if the request signature has been verified by nog-auth. req.baseUrl contains the part of the URL that was matched by prefix. It can be used in an action callback to create URLs that use the same prefix:

href = Meteor.absoluteUrl(baseUrl[1..] + '/' + blob._id)

The action callback either returns a result object or throws an error.

A result will be send via HTTP with status code 200 and a JSON body:

{
  "statusCode": 200
  "data": result
}

If result contains a field statusCode, its value will be used instead for the HTTP code and in the JSON body:

{
  "statusCode": result.statusCode
  "data": _.omit(result, 'statusCode')
}

As a special case, the callback can return a redirect result:

{
  statusCode: 307
  location: "https://..."
}

It will be translated to the expected HTTP redirect.

An error will be translated to an HTTP error status code and a JSON body such as:

{
  "errorCode": "ERR_MATCH",
  "message": "Match error: not a sha1 in field blob",
  "statusCode": 422
}

NogRest.configure(opts) (server)

configure() updates the active configuration with the provided opts:

  • checkRequestAuth (Function, default: NogAuth.checkRequestAuth): The authentication hook (see below).

NogRest.checkRequestAuth(req) (server)

checkRequestAuth(req) is expected to add req.auth.user with the authenticated user or to throw if the authentication fails. req is a HTTP request object.

Package nog-auth

nog-auth provides signature-based authentication of HTTP requests.

See the apidoc for a details description of the signature process.

The Meteor template {{> nogApiKeys}} provides a UI to manage keys. Secret keys are encrypted before they are stored in MongoDB. The master keys must be provided in Meteor.settings.NogAuthMasterKeys as an array of key objects [{keyid: String, secretkey: String}]. The first key is the primary key. Old keys can be provided to support key rotation. nog-auth will re-encrypt all keys with the primary key when on startup. The following commands may be useful to generate random ids and secrets:

head -c 100 /dev/random | openssl dgst -sha256 | head -c 20   # id
head -c 100 /dev/random | openssl dgst -sha256 | head -c 40   # secret

NogAuth.checkRequestAuth(req) (server)

checkRequestAuth(req) authenticates an HTTP request. It throws if the authentication fails. It adds the Meteor user that owns the key as req.auth.user if the request was successfully authenticated. If the signing key has scopes (see createKey()), they will be added as req.auth.user.scopes.

checkRequestAuth() is used as the authentication hook in nog-rest.

NogAuth.signRequest(key, req) (server)

signRequest(key, req) signs an HTTP request object with the key, which is an object {keyid: String, secretkey: String}.

NogAuth.createKey(user, opts) (server)

createKey() creates a new API key for the user id opts.keyOwnerId after an access check that user has permission to create a key.

NogAuth.createKeySudo(opts) (server)

createKeySudo(opts) creates a new API key for the user id opts.keyOwnerId without access check. It returns the key object {keyid, secretkey}. The secret key is encrypted with the primary master key before it is stored in MongoDB.

opts can contain a comment and scopes. opts.comment will be displayed in {{> nogApiKeys}}. opts.scopes (an array of {action: String, opts: Object}) will be returned by checkRequestAuth() as req.auth.user.scopes. The scopes can be used by NogAccess to restrict the actions that the key can be used for.

NogAuth.deleteKey(user, opts) (server)

deleteKey() delete the access key with id opts.keyid after an access check that user has permission to delete the key.

If opts.keyOnwerId (a user id) is present, it will be used as an additional selector when finding the key. Since keyid is assumed to be unique, keyOnwerId is only a additional safety measure.

NogAuth.deleteKeySudo(opts) (server)

Same as deleteKey() but without access check.

{{> nogApiKeys}} (client)

A UI widget to create and delete API keys for the user in the data context.

Example:

with currentUser
  +nogApiKeys

NogAuth.configure(opts) (anywhere)

configure() updates the active configuration with the provided opts:

  • onerror (Function, default: NogError.defaultErrorHandler) is used to report errors.
  • checkAccess (Function, default NogAccess.checkAccess if available) is used for access control.

NogAuth.onerror(err) (client)

The hook onerror(err) is called with errors on the client.

NogAuth.checkAccess(user, action, opts) (server)

The hook NogAuth.checkAccess(user, action, opts) is called to check whether a user has permissions to manage API keys. See package nog-access, specifically NogAccess.checkAccess().

Package nog-content

nog-content implements a git-like, content-addressable data model.

The design overview from March 2015 in 2015_fuimages_meteor-spikes:design-overview_2015-03.md may contain further ideas that are not yet implemented.

The REST API is described in detail in the separate doc: apidoc.

Data model

The entry point is a repo. It contains mutable state, primarily refs. refs are like git refs. Initially, branches/* is used for branches (instead of heads/ in git). Other prefixes may be useful later (like tags).

staging will be used for preparing a commit before actually committing it. staging is not yet implemented in nog-content; it has been used in spikes and is described in the design overview document. staging works like a local temporary branch with a commit that is repeatedly amended until it is ready to be committed. It seems reasonable to use a temporary commit. It can be used to store the state of the edit form, such as the draft of the commit message. An alternative would be to use a different data structure for preparing commits that would be more similar to git's index. This could be useful if the editing operation needs more state, such as multiple versions of a file for conflict resolution. For now, a temporary commit seems fine.

The next level is a commit. Like a git commit, it contains information like authors, dates, a message, and so on. A commit points to parent commits and to a tree.

A tree contains a dictionary of metadata and a list of entries of format {type: object|tree, sha1: <id>}. It is a recursive data structure. The leaf nodes are objects. Objects also contain metadata, and can point to a blob. A blob represents a binary object that is stored in object storage (like S3).

All immutable objects have ids that are computed as sha1s over a canonical EJSON representation. The documents stored in MongoDB may contain additional non-essential fields that are not part of the canonical representation. The most obvious example is _id, which is the computed sha1. Another candidate is touchTime to store when the document was used, which might be relevant when implementing a time-based garbage collection scheme.

NogContent.repos (server, subset at client)

The Mongo collection repos contains the repositories. Repositories contain mutable state. A repo has a random _id and a unique full repo name, which is composed from the repo owner name and the repo name: <ownerName>/<repoName>.

NogContent.commits (server, subset at client)

The Mongo collection commits contains the immutable commit entries.

NogContent.trees (server, subset at client)

The Mongo collection trees contains the immutable tree entries.

NogContent.objects (server, subset at client)

The Mongo collection objects contains the immutable object entries.

NogContent.blobs (server, subset at client)

The Mongo collection blobs is a reference to NogBlob.blobs if the packages nog-blob is available (weak dependency) and null otherwise.

NogContent.contentId(content) (anywhere)

contentId(content) computes the id for content, which must contain only valid fields. The calling code, for example, must remove fields that start with underscore. One way is to use NogContent.stripped() as in contentId(stripped(content)).

NogContent.strip(content) (anywhere)

strip(content) removes special internal fields like _id and _idversion from content. content is modified in place.

NogContent.stripped(content) (anywhere)

stripped(content) returns a copy of content without internal fields.

NogContent.configure(opts) (server)

configure() updates the active configuration with the provided opts:

  • checkAccess (Function, default NogAccess.checkAccess if available) is used for access control.

  • testAccess (Function, default NogAccess.testAccess if available) is used for access control.

NogContent.checkAccess(user, action, opts) (server)

The hook NogContent.checkAccess(user, action, opts) is called to check whether a user has the necessary upload and download permissions. See package nog-access.

Meteor.settings.optStrictRepoMembership (server)

The feature toggle optStrictRepoMembership (default: true) controls whether strict repo membership checks are enabled. If active, entries can only be accessed via a repo when they are reachable via a ref or when they have been recently added to the repo. This check should be activated if some kind of strict content sharing permissions are used, such as sharing only with selected user circles. If active, nog-content will configure nog-blob to check whether blobs are reachable from a repo.

NogContent.api.repos.actions() (server)

NogContent.api.repos.actions() returns an action array that can be plucked into nog-rest to provide a REST API.

The REST API is described in detail in the separate doc: apidoc.

The Python example content-testapp/public/tools/bin/test-create-content-py demonstrates the REST API.

If nog-blob is used, NogBlob.api.blobs.actions() and NogBlob.api.upload.actions() must be mounted at corresponding paths.

Usage example:

if Meteor.isServer
  NogRest.actions '/api/repos', NogContent.api.repos.actions()
  NogRest.actions '/api/repos/:ownerName/:repoName/db/blobs',
    NogBlob.api.blobs.actions()
  NogRest.actions '/api/repos/:ownerName/:repoName/db/blobs',
    NogBlob.api.upload.actions()

NogContent.call.* (anywhere, internal use)

The object NogContent.call provides Meteor methods that are used internally, such as NogContent.createRepo().

Package nog-errata

Due to a bug in the client-side SHA1 computation in browsers, correct blob data was stored under an incorrect blob id in a few cases during early development. The blobs and objects became part of the commit history. We wanted to keep the history but somehow mark the incorrect objects.

Since entries are immutable, the inconsistent ids cannot be modified but must remain part of the immutable history. To handle such situations, content entries can have an optional field errata. Example:

{
    "errata": [{ "code": "ERA201609a" }]
}

The meaning of the errata code is deployment-specific. The recommended format is ERA<year-month-char>, for example `ERA201609a'. Admins can use this key to document deployment-specific information about the issue.

This package provides utilities to display errata in the UI, based on a description in the public settings. Example:

{
    "public": {
        "errata": [
            {
                "code": "ERA201609a",
                "description": "An incorrect data checksum has been stored for this file during upload.  You can download a copy of the correct file from repo '<a href=\"/nog/era201609a/files\">nog/era201609a</a>'.  Then upload the correct file here and remove this file in order to permanently fix the issue and get rid of this message."
            }
        ]
    }
}

Package nog-blob

nog-blob implements content-addressable file storage on AWS S3. The S3 object key is the sha1 of the file content. The sha1 is computed in the browser before uploading the content. Uploaded objects are stored in the MongoDB collection blobs, which is available as NogBlob.blobs.

The REST API is described in apidoc-blobs and apidoc-upload.

NogBlob.blobs (server, subset at client)

A Mongo.Collection with information about the uploaded blobs. _id is the content sha1.

Clients are automatically subscribed to a subset that is relevant for the current uploads from the client.

NogBlob.uploadFile(file, done | opts) (client)

uploadFile(file, callbacks) starts an upload of a web File object. It returns an id for the client-only collection NogBlob.files immediately and calls done(err, res) upon completion. res is an object {_id: String, filename: String, size: Number, sha1: String}.

The second argument can be an object opts with functions done(err, res), and onwarning(err). It so, onwarning() is called for reporting intermediate warnings instead of the default NogBlob.onerror(). Eventually, done() is called as described in the previous paragraph.

Usage example:

template(name='upload')
  form
    .form-group
      input#files(type='file' name='files[]' multiple)

Template.upload.events
  'change #files': (e) ->
    e.preventDefault()
    for f in e.target.files
      id = NogBlob.uploadFile f, (err, res) ->
        if err
          return console.log 'failed to upload file', err
        # Do something with res; for example call server to add it.
        Meteor.call 'addObject',
          name: res.filename
          blob: res.sha1
      Session.set 'currentUpload', id

NogBlob.files (client)

A client-only collection that provides a reactive data source to track upload progress.

NogBlob.fileHelpers (client)

fileHelpers is an object with template helper functions that can be used to implement a UI to display file upload progress. The helper functions expect a document from NogBlob.files in the data context.

Usage example:

template(name='currentUpload')
  .row
    if haveUpload
      with file
        .col-md-3
          span #{name}
        .col-md-9
          .progress
            div(
              class="progress-bar {{uploadCompleteClass}}",
              role="progressbar",
              style="width: {{progressWidth}}%"
            )

currentUpload = -> Session.get('currentUpload')

Template.currentUpload.helpers
  haveUpload: -> currentUpload()?
  file: -> NogBlob.files.findOne currentUpload()

Template.currentUpload.helpers _.pick(
  NogBlob.fileHelpers, 'name', 'progressWidth', 'uploadCompleteClass'
)

{{> uploadHeading}} and {{> uploadItem}} (client)

{{> uploadHeading}} and {{> uploadItem}} are templates that illustrated how to display a list of uploads. We probably will not use them as is in the production app. Either we improve them or we build a custom UI using the fileHelpers described above.

template(name='upload')
  +uploadHeading
  hr
  each uploads
    +uploadItem

Template.upload.helpers
  uploads: -> NogBlob.files.find()

{{> aBlobHref blob=<sha1> name=<filename>}} (client)

The template {{> aBlobHref blob=<sha1> name=<filename>}} inserts an <a> element that when clicked will download the blob from S3 and save it as the specified filename.

NogBlob.configure(opts) (anywhere)

configure() updates the active configuration with the provided opts:

  • onerror (Function, default: NogError.defaultErrorHandler) is used to report errors.
  • checkAccess (Function, default NogAccess.checkAccess if available) is used for access control.
  • repoSets (instance of NogContent.RepoSets or false, default: false when used without package nog-content and true when used with nog-content) is used internally by the package nog-content to inject an implementation that checks whether a blob is reachable from a repo. Checks are currently only implemented in the api.*.actions but not for method calls. See source for details.
  • See source nog-blob.coffee for further configuration options.

NogBlob.config (anywhere)

The currently active configuration.

NogBlob.onerror(err) (client)

The hook onerror(err) is called with errors on the client.

NogBlob.checkAccess(user, action, opts) (server)

The hook NogAuth.checkAccess(user, action, opts) is called to check whether a user has the necessary upload and download permissions. See package nog-access.

NogBlob.api.blobs.actions() (server)

NogBlob.api.blobs.actions() returns an action array that can be plucked into nog-rest to provide a REST API.

The mount path must contain :ownerName and :repoName when used with repoSets for repo membership checks, which is the default when package nog-content is part of the app.

The REST API is described in apidoc-blobs.

Usage examples:

if Meteor.isServer
  NogRest.actions '/api/blobs', NogBlob.api.blobs.actions()

if Meteor.isServer
  NogRest.actions '/api/repos/:ownerName/:repoName/db/blobs',
    NogBlob.api.blobs.actions()

NogBlob.api.uploads.actions() (server)

NogBlob.api.uploads.actions() returns an action array that can be plucked into nog-rest to provide a REST API for uploading blobs. The upload.actions() must be mounted at the same path as the blobs.actions().

The mount path must contain :ownerName and :repoName when used with repoSets for repo membership checks, which is the default when package nog-content is part of the app.

The REST API is described in apidoc-upload.

The Python example blob-testapp/public/tools/bin/test-upload-py demonstrates the preferred way of using the REST API for uploading data.

The Bash example blob-testapp/public/tools/bin/test-upload also demonstrates how to upload data, but the example is not ideal. It ignores the initial parts, and it constructs URLs from identifiers instead of using the provided href fields.

Usage examples:

if Meteor.isServer
  NogRest.actions '/api/blobs', NogBlob.api.blobs.actions()
  NogRest.actions '/api/blobs', NogBlob.api.upload.actions()

if Meteor.isServer
  NogRest.actions '/api/repos/:ownerName/:repoName/db/blobs',
    NogBlob.api.blobs.actions()
  NogRest.actions '/api/repos/:ownerName/:repoName/db/blobs',
    NogBlob.api.uploads.actions()

NogBlob.call.* (anywhere, internal use)

The object NogBlob.call provides Meteor methods that are used internally, such as NogBlob.call.startMultipartUpload() or NogBlob.call.getBlobDownloadURL().

Package nog-files

nog-files implements a file browser, which should be as non-technical as possible. A UI inspired by Dropbox is probably suitable.

Only a subset of functions is documented here. See the source for further details.

NogFiles.registerEntryRepr(spec) (client)

spec must provide selector functions and may provide access control functions. The selectors return either the name of a Meteor template, which will be used for rendering, or null to indicate that it should be ignored. The access controls either return a control object or null to indicate that it should be ignored.

The following selectors must be present:

  • view(treeContext): The view template. You can return nogFilesBundleView to use a generic bundle view for a tree that should not be modified.

  • icon(entryContext): The icon template for a list view. You can return nogFilesBundleIcon to use a generic bundle icon for a tree that should not be modified.

The following access controls may be present:

  • treePermissions(treeContext): Can be used to restrict the actions that may be performed in a file listing. Return {write: false} to disable all operations that would modify the tree.

The most relevant fields in treeContext are (see the code in nog-files-ui.* for details; the context is the same as for nog-tree reprs):

  • commitId (sha1) and commit (full object).
  • namePath (Array of Strings): Names of entries along the resolved path from the tree root.
  • numericPath (Array of Numbers): Indices of entries along the resolved path from the tree root.
  • contentPath (Array of Objects): Full information about entries along the resolved path from the tree root. Each object contains the index in the parent tree entries in idx; the name of the entry in name; the type of the entry in type; the entry content in content.
  • last: An alias to the last entry in contentPath.
  • tree: The root tree in the same format as the contentPath elements. The root tree itself is not part of the contentPath.
  • repo: The repo object.
  • ref, refTreePath, refType, and treePath: Information about the path (see code for details).

The relevant fields in entryContext are:

  • context.parent: The treeContext (as described above) for the parent tree.
  • context.child.content: The content of the entry for which the icon shall be returned.

Individual packages can directly register representations, which requires a package dependency on nog-files. An alternative approach is to configure the repr spec in the main app. We do not yet know, which architecture works better in practice.

NogTree.entryView(treeContext) (client, internal)

entryView() is used internally to select the template for displaying an entry.

NogTree.entryIcon(entryContext) (client, internal)

entryIcon() is used internally to select the template used for the icon in a file list view.

Package nog-flow

nog-flow implements a workspace view for workspace repositories that provides a workflow order to collect data and programs, run programs, and inspect the results.

NogFlow.call.* (server)

NogFlow.call provides methods to modify the content of repositories, for instance NogFlow.call.addPrograms(opts) or NogFlow.call.renameChildren (opts). See the source for more information.

Package nog-tree

nog-tree implements a tree browsing user interface to nog repositories for users that want to access the technical details. Some elements are inspired by GitHub's web UI to browse git repositories.

Only a subset of functions is documented here. See the source for more information.

NogTree.registerEntryRepr(spec) (client)

registerEntryRepr() registers a new representation for entries to be used for tree browsing. spec must have a function spec.selector(context), which is passed the tree data context. The selector must return either the name of the Meteor template to use for rendering, or it returns null to indicate that the representation should be ignored.

The most relevant fields in the tree data context are (see the code in nog-tree-ui.* for details):

  • commitId (sha1) and commit (full object).
  • namePath (Array of Strings): Names of entries along the resolved path from the tree root.
  • numericPath (Array of Numbers): Indices of entries along the resolved path from the tree root.
  • contentPath (Array of Objects): Full information about entries along the resolved path from the tree root. Each object contains the index in the parent tree entries in idx; the name of the entry in name; the type of the entry in type; the entry content in content.
  • last: An alias to the last entry in contentPath.
  • tree: The root tree in the same format as the contentPath elements. The root tree itself is not part of the contentPath.
  • repo: The repo object.
  • ref, refTreePath, refType, and treePath: Information about the path (see code for details).

Individual packages can directly register representations, which requires a package dependency on nog-tree. An alternative approach is to configure the repr spec in the main app. We do not yet know, which architecture works better in practice.

NogTree.selectEntryRepr(context) (client)

selectEntryRepr() is internally used to select, based on the tree data context, a template for rendering an entry . It returns null if no entry repr selector matches, which indicates that the default view should be used.

Package nog-repr-example

nog-repr-example is an example package to illustrate how to add entry representations for tree and file browsing.

Package nog-repr-image

nog-repr-image implements plugins to handle images in nog-tree and nog-files.

Package nog-repr-markdown

nog-repr-markdown implements plugins to handle markdown in nog-tree and nog-files.

Package nog-repr-flow

nog-repr-flow handles workflow-related entries in nog-tree and nog-files. It currently only configures nog-files to display certain tree kinds as bundles. More code will probably be moved from nog-tree later.

Package nog-multi-bucket

The package nog-multi-bucket provides mechanisms to manage blobs in multiple object buckets.

createBucketRouterFromSettings() creates a multi-bucket router with methods getDownloadUrl({ blob, filename }) and getImgSrc({ blob, filename }) that return blob download URLs considering a configured bucket preference order and bucket health. The mechanism is used by package nog-blob.

Multi-bucket upload routing is supported through createMultipartUpload({ key }), which uses the writePrefs settings (see below) to determine the upload bucket. getSignedUploadPartUrl() returns upload URLs for the individual parts. The upload is completed with completeMultipartUpload() or canceled with abortMultipartUpload().

Meteor Settings

The multi-bucket router is configured from package nog-blob via Meteor.settings.multiBucket.

nog-multi-bucket exports a Meteor check match pattern matchMultiBucketSettings, which can be used to validate settings as follows:

import { matchMultiBucketSettings } from 'meteor/nog-multi-bucket';
check(Meteor.settings.multiBucket, matchMultiBucketSettings);

The following are example settings for one AWS S3 bucket, which is assumed to be always healthy, and one Ceph S3 bucket with a health check that reads an object and verifies its content. The Ceph S3 bucket is preferred for download and upload:

{
    "multiBucket": {
        "readPrefs": ["nog-zib-2", "nog"],
        "writePrefs": ["nog-zib-2", "nog"],
        "fallback": "nog",
        "buckets": [
            {
                "name": "nog",
                "region": "eu-central-1",
                "accessKeyId": "AKxxxxxxxxxxxxxxxxxx",
                "secretAccessKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                "check": "healthy"
            },
            {
                "name": "nog-zib-2",
                "accessKeyId": "Cxxxxxxxxxxxxxxxxxxx",
                "secretAccessKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                "endpoint": "https://objs2.zib.nogproject.io",
                "signatureVersion": "v4",
                "check": "getObject",
                "checkKey": "_whoami",
                "checkContent": "bucket:nog-zib-2",
                "checkInterval": "15s"
            }
        ]
    },
}

The health check expects an object whose content equals checkContent. Such an object can, for example, be created with a properly configured S3cmd as follows:

echo 'bucket:nog-zib' >'_whoami'
s3cmd put '_whoami' 's3://nog-zib/_whoami'

AWS S3 Configuration

The AWS key needs s3:PutObject and s3:GetObject rights on the S3 bucket that is used by nog-blob. The recommended approach to AWS permission management is to use one AWS IAM user for the application and grant rights via groups with inline policies (use the custom policy editor). For example:

User nog-app.

Group nog-s3-get with policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1420905603000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::nog/*"
            ]
        }
    ]
}

Group nog-s3-put with policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1420905603000",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::nog/*"
            ]
        }
    ]
}

The S3 CORS configuration must allow any origin and expose the ETag header (see http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-configuring.html#Cross-Origin_Resource_Sharing__CORS_):

Use XML to configure CORS via the AWS Admin UI:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>HEAD</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
        <ExposeHeader>ETag</ExposeHeader>
    </CORSRule>
</CORSConfiguration>

Local Ceph S3 Developer Setup

A local Docker container with Ceph can be used as an alternative to AWS S3 during development. The manual steps are described below. The repo root contains a Docker Compose file that automates the steps for a basic setup.

Run the Docker image ceph/demo as follows:

docker run -it --rm \
    --name ceph \
    -p 10080:80 \
    -e NETWORK_AUTO_DETECT=4 \
    -e CEPH_DEMO_UID=nog -e CEPH_DEMO_ACCESS_KEY=Cdemo -e CEPH_DEMO_SECRET_KEY=Cdemosecret -e CEPH_DEMO_BUCKET=noglocal \
    ceph/demo

Use the following entry in Meteor.settings.multiBucket.buckets:

{
    "name": "noglocal",
    "endpoint": "http://localhost:10080",
    "accessKeyId": "Cdemo",
    "secretAccessKey": "Cdemosecret"
}

Configure an AWS credentials profile in ~/.aws/credentials:

[cephdemo]
aws_access_key_id=Cdemo
aws_secret_access_key=Cdemosecret

Configure CORS settings. With the following cors.json:

{
    "CORSRules": [
        {
            "AllowedOrigins": ["*"],
            "AllowedMethods": ["PUT", "POST", "GET", "HEAD"],
            "MaxAgeSeconds": 3000,
            "AllowedHeaders": ["*"],
            "ExposeHeaders": ["ETag"]
        }
    ]
}

Run:

aws --profile cephdemo --endpoint-url http://localhost:10080 s3api put-bucket-cors --bucket noglocal --cors-configuration file://cors.json

You can create additional buckets as follows:

aws --profile cephdemo --endpoint-url http://localhost:10080 --region localhost s3 mb s3://noglocal2

Package nog-s3 (DEPRECATED)

DEPRECATED: nog-s3 should not be used anymore. Use nog-multi-bucket instead.

nog-s3 wraps just enough of the AWS SDK to implement nog-blob. S3 exposes only a few sync functions. Errors are translated to NogError.Error. The opts are identical to params of the official AWS SDK: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html.

S3.configure(opts) (server)

configure() updates the active configuration with the provided opts:

  • accessKeyId (String, default Meteor.settings.AWSAccessKeyId).
  • secretAccessKey (String, default Meteor.settings.AWSSecretAccessKey).
  • region (String, default Meteor.settings.AWSBucketRegion).
  • signatureVersion (s3 or v4, default Meteor.settings.AWSSignatureVersion or v4): eu-central-1 requires v4.
  • s3ForcePathStyle (Boolean, default Meteor.settings.AWSS3ForcePathStyle or false): URL format for false is {bucket}.{region}...; URL format for true is {endpoint}/{bucket}. The path style may be useful with alternative S3 implementations, like Ceph RadosGW.
  • endpoint (String, default Meteor.settings.AWSEndpoint): The endpoint must accept requests from the server and from client browsers.
  • sslEnabled (Boolean, default Meteor.settings.AWSSslEnabled or true).
  • ca (String, default Meteor.settings.AWSCa): If present, must be an absolute path to a CA certificate bundle .pem file, which will be loaded and used instead of the CAs that are bundled with Node.

The key needs to have s3:PutObject and s3:GetObject rights on the S3 bucket that is used by nog-blob. The recommended approach to AWS permission management is to use one AWS IAM user for the application and grant rights via groups with inline policies (use the custom policy editor). For example:

User nog-app.

Group nog-s3-get with policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1420905603000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::nog/*"
            ]
        }
    ]
}

Group nog-s3-put with policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1420905603000",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::nog/*"
            ]
        }
    ]
}

The S3 CORS configuration must allow any origin and expose the ETag header (see http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-configuring.html#Cross-Origin_Resource_Sharing__CORS_):

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>HEAD</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
        <ExposeHeader>ETag</ExposeHeader>
    </CORSRule>
</CORSConfiguration>

AWS SDK

S3.createMultipartUpload(opts) (server)

See http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#createMultipartUpload-property.

S3.getSignedUploadPartUrl(opts) (server)

See http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#uploadPart-property.

S3.getSignedDownloadUrl(opts) (server)

See http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property.

S3.completeMultipartUpload(opts) (server)

See http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#completeMultipartUpload-property.

S3.abortMultipartUpload(opts) (server)

See http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#abortMultipartUpload-property.

Package nog-test

The package nog-test provides testing infrastructure.

Configuring the test selection

During startup, nog-test configures Mocha by default to exclude all tagged test. The tag format is _[A-Z]+_ anywhere in a test name.

To include tests, set Meteor.settings.public.tests.mocha to an object {"grep": String, "invert": Boolean} to specify a test filter. Use {"grep": ".*"} to include all tests.

NogTest.testingMethods(methods) (anywhere)

testingMethods(methods) calls Meteor.methods(methods). It ignores errors, so that methods can be defined within tests that may be called repeatedly. Example:

describe 'some test', ->
  savedAccess = null
  testingMethods
    'testing/nog-auth/disableAccessCheck': ->
      if Meteor.isServer
        savedAccess = NogAuth.access
        NogAuth.configure {access: {check: ->}}
    'testing/nog-auth/restoreAccessCheck': ->
      if Meteor.isServer
        NogAuth.configure {access: savedAccess}

  before((next) -> Meteor.call 'testing/nog-auth/disableAccessCheck', next)
  after((next) -> Meteor.call 'testing/nog-auth/restoreAccessCheck', next)

  it.client 'a test that runs without access control', (next) ->

NogTest.pause(duration_ms, fn) (anywhere)

pause() is setTimeout with a human-friendly argument order. Example:

describe 'some context', ->
  it 'a test', ->
    someWork()
    pause 1000, ->
      expect(...)

Complete list of error specs

specs = [

  {
    errorCode: 'ERR_MIGRATION'
    statusCode: 500
    sanitized: null
    reason: 'A database migration failed.'
  }

  {
    errorCode: 'ERR_UNIMPLEMENTED'
    statusCode: 500
    sanitized: null
    reason: 'The operation is not implemented.'
  }

  # Malformed as in XML structure validation: the format of a value is invalid.
  {
    errorCode: 'ERR_PARAM_MALFORMED'
    statusCode: 422
    sanitized: 'full'
    reason: 'A parameter was malformed.'
  }

  # Invalid comes after malformed: the basic structure is ok, but the value is
  # semantically invalid, such as out-of-range.
  {
    errorCode: 'ERR_PARAM_INVALID'
    statusCode: 422
    sanitized: 'full'
    reason: 'A parameter was semantically invalid.'
  }

  {
    errorCode: 'ERR_S3_CREATE_MULTIPART'
    statusCode: 502
    sanitized: null
    reason: 'Failed to create S3 multipart upload.'
    details: (ctx) -> "
        Failed to call createMultipartUpload with S3 bucket `#{ctx.s3Bucket}`,
        and object key `#{ctx.s3ObjectKey}`.
      "
    contextPattern:
      s3Bucket: String
      s3ObjectKey: String
  }
  {
    errorCode: 'ERR_S3_COMPLETE_MULTIPART'
    statusCode: 502
    sanitized: null
    reason: 'Failed to complete S3 multipart upload.'
    details: (ctx) -> "
        Failed to call completeMultipartUpload with S3 bucket `#{ctx.s3Bucket}`,
        and object key `#{ctx.s3ObjectKey}`.
      "
    contextPattern:
      s3Bucket: String
      s3ObjectKey: String
      s3UploadId: String
  }
  {
    errorCode: 'ERR_S3_ABORT_MULTIPART'
    statusCode: 502
    sanitized: null
    reason: 'Failed to abort S3 multipart upload.'
    details: (ctx) -> "
        Failed to call abortMultipartUpload with S3 bucket `#{ctx.s3Bucket}`,
        object key '#{ctx.s3ObjectKey}', and upload id `#{ctx.s3UploadId}`.
      "
    contextPattern:
      s3Bucket: String
      s3ObjectKey: String
      s3UploadId: String
  }

  {
    errorCode: 'ERR_BLOB_NOT_FOUND'
    statusCode: 404
    sanitized: 'full'
    reason: (ctx) -> "The requested blob '#{ctx.blob}' could not be found."
    details: null
    contextPattern:
      blob: String
  }
  {
    errorCode: 'ERR_BLOB_UPLOAD_START'
    statusCode: 502
    sanitized: null
    reason: 'Failed to start upload.'
    details: (ctx) -> "
        The server reported an error when calling 'startMultipartUpload' for
        file `#{ctx.fileName}`, size #{ctx.fileSize}, sha1 `#{ctx.sha1}`.
      "
    contextPattern:
      fileName: String
      fileSize: Number
      sha1: String
  }
  {
    errorCode: 'ERR_BLOB_UPLOAD'
    statusCode: 500
    sanitized: 'full'
    reason: 'Blob upload failed.'
  }
  {
    errorCode: 'ERR_BLOB_UPLOAD_EXISTS'
    statusCode: 409
    sanitized: 'full'
    reason: (ctx) -> "
        The blob '#{ctx.sha1}' already exists and cannot be uploaded again.
        You may continue assuming that the blob is available as if the
        upload succeeded.
      "
    contextPattern:
      sha1: String
  }
  {
    errorCode: 'ERR_BLOB_CONFLICT'
    statusCode: 409
    sanitized: 'full'
    reason: (ctx) -> "
        The blob '#{ctx.sha1}' already exists, but with a different size.
        You should probably contact a system administrator.
      "
    contextPattern:
      sha1: String
  }
  {
    errorCode: 'ERR_BLOB_UPLOAD_WARN'
    statusCode: 500
    sanitized: 'full'
    reason: 'Problem with blob upload that may be resolved later.'
  }
  {
    errorCode: 'ERR_BLOB_COMPUTE_SHA1'
    statusCode: 500
    sanitized: 'full'
    reason: 'Problem computing sha1.'
  }
  {
    errorCode: 'ERR_BLOB_COMPUTE_MD5'
    statusCode: 500
    sanitized: 'full'
    reason: 'Problem computing MD5.'
  }
  {
    errorCode: 'ERR_BLOB_ABORT_PENDING'
    statusCode: 502
    sanitized: null
    reason: "Failed to abort pending upload after timeout."
    contextPattern:
      sha1: String
  }
  {
    errorCode: 'ERR_DB'
    statusCode: 500
    sanitized: null
    reason: "A database operation unexpectedly failed."
  }
  {
    errorCode: 'ERR_LIMIT'
    statusCode: 413
    sanitized: 'full'
    reason: "The request is larger than a limit."
  }
  {
    errorCode: 'ERR_LIMIT_S3_OBJECT_SIZE'
    statusCode: 413
    sanitized: 'full'
    reason: (ctx) -> "
        The upload size (#{ctx.size} Bytes) is greater than the maximum
        size supported by S3 (#{ctx.maxSize} Bytes).
      "
    contextPattern:
      size: Number
      maxSize: Number
  }
  {
    errorCode: 'ERR_UPLOADID_UNKNOWN'
    statusCode: 404
    sanitized: 'full'
    reason: 'The upload id is unknown.'
  }
  {
    errorCode: 'ERR_UPLOAD_COMPLETE'
    statusCode: 502
    sanitized: 'full'
    reason: 'Failed to complete the S3 multipart upload.'
  }
  {
    errorCode: 'ERR_BLOB_DOWNLOAD'
    statusCode: 500
    sanitized: 'full'
    reason: 'Problem with blob download.'
  }

  {
    errorCode: 'ERR_UNKNOWN_MASTER_KEY'
    statusCode: 500
    reason: (ctx) -> "Unknown master key id '#{ctx.masterkeyid}'."
    contextPattern:
      masterkeyid: String
  }
  {
    errorCode: 'ERR_UNKNOWN_USERID'
    statusCode: 404
    reason: (ctx) -> "Could not find user id '#{ctx.uid}'."
    contextPattern:
      uid: String
  }
  {
    errorCode: 'ERR_UNKNOWN_USERNAME'
    statusCode: 404
    reason: (ctx) -> "Could not find user '#{ctx.username}'."
    contextPattern:
      username: String
  }
  {
    errorCode: 'ERR_UNKNOWN_KEYID'
    statusCode: 404
    reason: (ctx) -> "Could not find key id '#{ctx.keyid}'."
    contextPattern:
      keyid: String
  }

  {
    errorCode: 'ERR_AUTH_FIELD_MISSING'
    statusCode: 401
    sanitized: 'full'
    reason: (ctx) -> "Invalid signature (missing #{ctx.missing})."
    contextPattern:
      missing: String
  }
  {
    errorCode: 'ERR_AUTH_DATE_INVALID'
    statusCode: 401
    sanitized: 'full'
    reason: (ctx) -> "Invalid authdate"
  }
  {
    errorCode: 'ERR_AUTH_SIG_EXPIRED'
    statusCode: 401
    sanitized: 'full'
    reason: (ctx) -> 'Expired signature'
  }
  {
    errorCode: 'ERR_AUTH_KEY_UNKNOWN'
    statusCode: 401
    sanitized: 'full'
    reason: (ctx) -> "Unknown key."
  }
  {
    errorCode: 'ERR_AUTH_SIG_INVALID'
    statusCode: 401
    sanitized: 'full'
    reason: (ctx) -> "Invalid signature."
  }
  {
    errorCode: 'ERR_AUTH_EXPIRES_INVALID'
    statusCode: 401
    sanitized: 'full'
    reason: (ctx) -> "Invalid expires."
  }
  {
    errorCode: 'ERR_AUTH_NONCE_INVALID'
    statusCode: 401
    sanitized: 'full'
    reason: (ctx) -> "Invalid nonce."
  }

  {
    errorCode: 'ERR_ACCESS_DENY'
    statusCode: 404
    sanitized: 'full'
    reason: 'Access denied by policy.'
  }
  {
    errorCode: 'ERR_ACCESS_DEFAULT_DENY'
    statusCode: 404
    sanitized: 'full'
    reason: 'Access denied without policy.'
  }

  {
    errorCode: 'ERR_APIKEY_CREATE'
    statusCode: 403
    sanitized: 'full'
    reason: 'Failed to create API key.'
  }
  {
    errorCode: 'ERR_APIKEY_DELETE'
    statusCode: 403
    sanitized: 'full'
    reason: 'Failed to create API key.'
  }

  {
    errorCode: 'ERR_CONTENT_REPO_EXISTS'
    statusCode: 409
    sanitized: 'full'
    reason: (ctx) -> "The repo `#{ctx.repoFullName}` already exists."
    contextPattern:
      repoFullName: String
  }
  {
    errorCode: 'ERR_CONTENT_MISSING'
    statusCode: 404,
    sanitized: 'full'
    reason: (ctx) ->
      if ctx.object?
        "The object `#{ctx.object}` is missing."
      else if ctx.tree?
        "The tree `#{ctx.tree}` is missing."
      else if ctx.blob?
        "The blob `#{ctx.blob}` is missing."
      else if ctx.commit?
        "The commit `#{ctx.commit}` is missing."
      else
        "Some content is missing."
  }
  {
    errorCode: 'ERR_CONTENT_CHECKSUM'
    statusCode: 500,
    sanitized: 'full'
    reason: (ctx) ->
      "Content id checksum error for #{ctx.type} #{ctx.sha1}."
    contextPattern:
      sha1: String
      type: String
  }
  {
    errorCode: 'ERR_REPO_MISSING'
    statusCode: 404,
    sanitized: 'full'
    reason: 'The repo does not exist.'
  }
  {
    errorCode: 'ERR_REF_MISMATCH'
    statusCode: 409
    sanitized: 'full'
    reason: 'The old ref does not match.'
  }
  {
    errorCode: 'ERR_REF_NOT_FOUND'
    statusCode: 404
    sanitized: 'full'
    reason: (ctx) -> "The requested ref '#{ctx.refName}' could not be found."
    contextPattern:
      refName: String
  }

  {
    errorCode: 'ERR_CONFLICT'
    statusCode: 409
    sanitized: 'full'
    reason: 'The request conflicts with a concurrent request.'
  }

  {
    errorCode: 'ERR_LOST_LOCK'
    statusCode: 409
    sanitized: 'full'
    reason: 'The active request lost its lock to a concurrent request.'
  }

  {
    errorCode: 'ERR_LOGIC'
    statusCode: 500
    sanitized: null
    reason: 'There is a problem with the program logic.'
  }

  {
    errorCode: 'ERR_CREATE_ACCOUNT_USERNAME'
    statusCode: 401
    sanitized: 'full'
    reason: 'Cannot create account: no username.'
  }
  {
    errorCode: 'ERR_CREATE_ACCOUNT_USERNAME_TOOSHORT'
    statusCode: 401
    sanitized: 'full'
    reason: 'Username must be at least 3 characters long.'
  }
  {
    errorCode: 'ERR_CREATE_ACCOUNT_USERNAME_INVALID'
    statusCode: 401
    sanitized: 'full'
    reason: 'Username may only contain the following characters: "a-z", "0-9",  "_", and "-".'
  }
  {
    errorCode: 'ERR_CREATE_ACCOUNT_USERNAME_BLACKLISTED'
    statusCode: 401
    sanitized: 'full'
    reason: 'Username is not allowed.'
  }
  {
    errorCode: 'ERR_CREATE_ACCOUNT_USERNAME_GITHUB'
    statusCode: 401
    sanitized: 'full'
    reason: 'Username already exists as a non-github account.'
  }
  {
    errorCode: 'ERR_CREATE_ACCOUNT_USERNAME_GITIMP'
    statusCode: 401
    sanitized: 'full'
    reason: 'Username already exists as a non-gitimp account.'
  }
  {
    errorCode: 'ERR_CREATE_ACCOUNT_USERNAME_GITZIB'
    statusCode: 401
    sanitized: 'full'
    reason: 'Username already exists as a non-gitzib account.'
  }
  {
    errorCode: 'ERR_CREATE_ACCOUNT_EMAIL'
    statusCode: 401
    sanitized: 'full'
    reason: 'Cannot create account: no email address.'
  }
  {
    errorCode: 'ERR_ACCOUNT_DELETE'
    statusCode: 403
    sanitized: 'full'
    reason: 'Cannot delete account.'
  }

  {
    errorCode: 'ERR_CREATE'
    statusCode: 400
    sanitized: 'full'
    reason: 'Failed to create a resource.'
  }
  {
    errorCode: 'ERR_UNKNOWN'
    statusCode: 404
    sanitized: 'full'
    reason: 'A resource is unknown.'
  }
  {
    errorCode: 'ERR_UPDATE'
    statusCode: 400
    sanitized: 'full'
    reason: 'An update failed.'
  }

  # Example: `meta.workspace` should be an object, but it's not.
  {
    errorCode: 'ERR_NOT_OF_KIND'
    statusCode: 400
    sanitized: 'full'
    reason: 'An entry is not of the expected kind.'
  }

  {
    errorCode: 'ERR_PROC_TIMEOUT'
    statusCode: 503
    sanitized: 'full'
    reason: 'Request processing took too long.'
  }

  {
    errorCode: 'ERR_API_VERSION'
    statusCode: 409
    sanitized: 'full'
    reason: 'The API version is incompatible.'
  }

  {
    errorCode: 'ERR_UPDATE_SYNCHRO'
    statusCode: 500
    sanitized: null
    reason: 'An update on a synchro failed.'
  }

  {
    errorCode: 'ERR_SYNCHRO_MISSING'
    statusCode: 404,
    sanitized: null
    reason: 'The synchro does not exist.'
  }
  {
    errorCode: 'ERR_SYNCHRO_CONTENT_MISSING'
    statusCode: 404,
    sanitized: null
    reason: (ctx) ->
      if ctx.object?
        "The synchro object `#{ctx.object}` is missing."
      else if ctx.tree?
        "The synchro tree `#{ctx.tree}` is missing."
      else if ctx.commit?
        "The synchro commit `#{ctx.commit}` is missing."
      else
        "Some synchro content is missing."
  }
  {
    errorCode: 'ERR_SYNCHRO_STATE'
    statusCode: 400,
    sanitized: null
    reason: 'The synchro state is invalid.'
  }
  {
    errorCode: 'ERR_SYNCHRO_SNAPSHOT_INVALID'
    statusCode: 500,
    sanitized: null
    reason: 'The synchro snapshot is invalid.'
  }
  {
    errorCode: 'ERR_SYNCHRO_APPLY_FAILED'
    statusCode: 500,
    sanitized: null
    reason: 'Sync apply failed.'
  }

]

for s in specs
  NogError[s.errorCode] = s

REST API

See apidoc.