Sinopia Linked Data Editor

Technical documentation specific to the Sinopia Linked Data Editor may also be found in the wiki. The Sinopia Editor homepage is available in staging at stage.sinopia.io. The Sinopia Editor is a React application with all new user interfaces and functionality using React and the React ecosystem. Portions of the codebase originally extracted from the Library of Congress bfe project.

Installation (without docker image)

Prerequisites

• node.js JavaScript runtime https://nodejs.org/en/download/
• npm JavaScript package manager https://www.npmjs.com/

Installation Instructions

1. Install node.js
2. Install npm
3. Run npm init, and follow the instructions that appear.
4. Get latest npm: npm install -g npm@latest
5. Run npm install. This installs everything needed for the build to run successfully.

Note: Currently you need to have node version 10 (version 12 may not work). You can use the "n" node package management to manage multiple version of node: https://www.npmjs.com/package/n

Build the distribution

npm run build

Running the code

npm start

Follow installation instructions, then run npm start or node server.js to start the web server using Express. This will start up the code at http://localhost:8000.

The Sinopia Editor code is currently available via sinopia.io

Creating an account

To log into the sinopia editor in localhost, you will need a development cognito account. To do this, go to https://development.sinopia.io/ and click "Request Account". Create your account here and confirm the email address. Once created, you can go back to localhost and login with that account. You will need to be online to authenticate even when developing in localhost.

Developers

• See package.json for npm package dependencies.
• The web server used is the express web framework for node.js
• React components are located in src/components/ directory
• Characters for the diacritics components come from the Mediawiki repository's specialcharacters.json. This file has been customized for use in Sinopia by including a separate label for each character set and by removing MediaWiki-specific data structures that would have made processing by Sinopia more complicated.

Use static resource templates instead of hitting Trellis

Specify the environment variable USE_FIXTURES=true when building the application if you would like it to load resource templates from the filesystem instead of looking for Trellis.

The resource templates can be found in __tests__/__fixtures__ and are listed in __tests__/fixtureLoaderHelper.

To convert a directory of profiles (for example, ld4ptemplates to resource templates:

rm -fr templates
mkdir -p templates
rm -f rtFileNames.txt
cat *.json | jq -c '.Profile.resourceTemplates[]' | sed 's/\\"/\\\\"/g' | while read line; do echo $line > templates/$(date +%s%N).json; done

for file in templates/*.json
do
  id=$(jq -r '.id' $file)
  newid=${id//:/_}
  newfile="templates/$newid.json"
  mv $file $newfile
  echo "  '$newid.json'," >> rtFileNames.txt
done

then move templates/* to __tests__/__fixtures__ and add the contents of rtFileNames.txt to __tests__/fixtureLoaderHelper.

Run the server with webpack-dev-server

npm run dev-start

Runs the webpack-dev-server, allowing immediate loading of live code changes without having to restart the server. The webpack-dev-server is available at http://localhost:8888. Note that running the webpack server does NOT call server.js

Building with webpack

npm run dev-build (no minimization) or npm run build (with minimization)

We are using webpack as a build tool. See webpack.config.js for build dependencies and configuration.

Running the production server

npm start will spin up the production server (this depends on npm run build already having been run). The web server is available at http://localhost:8000.

Linter for JavaScript

npm run eslint

Running Tests

Tests are written with jest:

• Unit tests ("in jsdom") are written with enzyme and react-testing-library
• Integration tests ("in-browser") are written with puppeteer

We are currently in the process of switching tests to react-testing-library.

To properly run all of the tests (including integration), you'll have to provide a couple of environment variables, so that the tests have valid user info with which to login. The env vars are:

Add these to your local .env file:

COGNITO_TEST_USER_NAME='[email protected]' # a test user we have on dev and stage
COGNITO_TEST_USER_PASS='<get this from shared_configs or another developer>' # not committing the real value to a public repo

Putting it all together, to run all of the tests:

npm test
npm run integration

Note that if you have an instance of the dev server already running in a separate terminal, you may need to stop the server or you may get a port conflict when running the integration tests.

Test coverage

To get coverage data, use npm run jest-cov. Be sure to specify the ENV variables as described above:

npm run jest-cov

Once complete, you can start the dev server on your laptop as describe above and visit http://localhost:8888/coverage/lcov-report/index.html. (change the localhost port number in the URL as needed to reflect the one used by your local server)

There is a project view and also a view of each file. You can also check coveralls.

End-to-end tests

End-to-end tests are written with Cypress.

Add these to your local cypress.env.json file:

{
  "COGNITO_TEST_USER_NAME": "sinopia-devs_client-tester",
  "COGNITO_TEST_USER_PASS": "<get this from shared_configs or another developer>"
}

The end-to-end tests run against a complete environment running in docker:

docker-compose pull
docker-compose build
docker-compose up -d

To open Cypress interactively (for test development), execute npm run cypress-open and click on the test to run.

To run test non-interactively, execute npm run cypress-run.

Continuous Integration

We use circleci. The steps are in .circleci/config.yml.

In the "artifacts" tab of a particular build, you can look at code coverage (coverage/lcov-report/index.html).

Running with Docker

The Sinopia Editor supports Docker, both with images hosted on Dockerhub and with an available Dockerfile to build locally.

Running latest Dockerhub Image

To run the Docker image, first download the latest image by docker pull ld4p/sinopia_editor:latest and then to run the editor locally in the foreground, docker run -p 8000:8000 --rm --name=sinopia_editor ld4p/sinopia_editor. The running Sinopia Editor should now be available locally at http://localhost:8000.

Docker-Compose

A docker-compose configuration is also provided to allow integration of the editor with Sinopia's platform components, including Trellis, ElasticSearch, ActiveMQ, Postgres, and the Sinopia indexing pipeline. You can spin up these components via:

$ docker-compose up editor # add the '-d' flag to daemonize and run in background

Of particular interest:

• The editor is at http://localhost:8000/
• Trellis is at http://localhost:8080/

Note that this will provide you with "out-of-the-box" Trellis, with no data in it. To spin up Trellis and its dependencies with the Sinopia container structure (root, repository, and group containers) and ACLs (declared on root container) pre-created, you can do using the platformdata docker-compose service:

$ docker-compose run platformdata

NOTE: In order for the above to work, you will need to set COGNITO_ADMIN_PASSWORD, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY in a file named .env in the sinopia_editor root.

At this point, you will likely want to begin importing resource templates into the editor, after which you can begin creating linked data resources. To import a small set of interesting resource templates, consult the instructions in this README.

If you'd like to see how the indexing pipeline is indexing Trellis data into ElasticSearch, you can spy on the ElasticSearch indexes using the DejaVu app included in the docker-compose configuration:

$ docker-compose up searchui # add the '-d' flag to daemonize and run in background

To use DejaVu, browse to http://localhost:1358, and when prompted, enter http://localhost:9200 as the ElasticSearch URL and * as the index name.

Building latest Docker Image

Before building the latest Docker Image, run npm run build to update the dist folder with the current build. To build the latest version of the Sinopia Editor, you can build with the docker build -t ld4p/sinopia_editor:latest --no-cache=true . command. NOTE that images tagged with latest will not be deployed to any of our AWS environments. See below for how to build and deploy images

Important Docker Image Build Note

If you add environment variables to which the Editor needs to pay attention (e.g. for configuring connections to Cognito or other external services on a per-instance basis), you'll need to make sure they're added to lists in three places besides e.g. the Config.js function that uses the environment variable.

• the list given to new webpack.EnvironmentPlugin() in the plugins section of webpack.config.js
  • e.g. https://github.com/LD4P/sinopia_editor/commit/aadd9d6170b08ff9261392d5b2ec2c6f56470e20#diff-11e9f7f953edc64ba14b0cc350ae7b9dR58
• the build-time arguments section of Dockerfile
  • e.g. https://github.com/LD4P/sinopia_editor/commit/aadd9d6170b08ff9261392d5b2ec2c6f56470e20#diff-3254677a7917c6c01f55212f86c57fbfR10
• the env specific docker build commands in the register_image section of .circleci/config.yml
  • e.g. https://github.com/LD4P/sinopia_editor/commit/1d3e381cb0f937300242cf896f62c2508e4a57e2#diff-1d37e48f9ceff6d8030570cd36286a61R63

All three of those locations must have the environment variable name added correctly. If the webpack configuration change is omitted, the environment variable value will not be picked up by the editor, even when running locally on a dev laptop. If either of the other two changes is omitted, the environment variable value will not make it into the environment specific images that are built and deployed.

Pushing Docker Image to DockerHub

Run docker login and enter the correct credentials to your docker account (hub.docker.com). Once successfully authenticated, run

docker push ld4p/sinopia_editor:latest

Ask a member on the DevOps team to go into the AWS console to update https://sinopia.io

Updating Docker Image in AWS Dev Environment

This section assumes you've already authenticated to DockerHub via docker login in the previous section, and also assumes you've run through the AWS development environment setup documentation and configured the AWS CLI.

First, build a new sinopia_editor image tagged with dev. In order to do this, you MUST provide the dev-specific build args:

$ docker build -t ld4p/sinopia_editor:dev --build-arg TRELLIS_BASE_URL=https://trellis.development.sinopia.io --build-arg SINOPIA_URI=https://development.sinopia.io --build-arg AWS_COGNITO_DOMAIN=https://sinopia-development.auth.us-west-2.amazoncognito.com --build-arg COGNITO_CLIENT_ID=2u6s7pqkc1grq1qs464fsi82at .

Then push the dev-tagged image to DockerHub:

$ docker push ld4p/sinopia_editor:dev

Next, set an environment variable to the name of the AWS DevelopersRole profile as described in the documentation above (as stored in ~/.aws/config):

$ export AWS_PROFILE=change_to_whatever_you_named_your_dlss_development_profile

And, finally, run the following commands to refresh the dev ECS instance that runs the editor:

$ task_arn=$(aws ecs list-task-definitions --family-prefix sinopia-homepage --region us-west-2 --sort DESC --max-items 1 --profile $AWS_PROFILE | jq --raw-output --exit-status '.taskDefinitionArns[]')
$ cluster_arn=$(aws ecs list-clusters --region us-west-2  --profile $AWS_PROFILE | jq --raw-output --exit-status '.clusterArns[] | select(contains(":cluster/sinopia-dev"))')
$ aws ecs update-service --service sinopia-homepage --region us-west-2 --cluster $cluster_arn --task-definition $task_arn --force-new-deployment --profile $AWS_PROFILE

Release Management

The steps to create a tagged release are documented in the tagged-release ticket template. Please create a tagged-release ticket when performing a release.

LD4P's fork of the BIBFRAME Editor

The Sinopia Editor is forked from https://github.com/lcnetdev/bfe.

History of BFE

From lcnetdev description: bfe is a standalone Editor for the Library of Congress's Bibliographic Framework (BIBFRAME) Initiative. It can be used more generically as an editor for RDF data. bfe uses BIBFRAME Profiles to render an HTML/UI input form; it is capable of integrating 'lookup' services, which query data from external Web APIs; and implementers can define the input and extract the output. ... From a design standpoint, the objective with bfe was to create the simplest 'pluggable' form editor one can to maximize experimental implementer's abilities to create/edit BIBFRAME data. The current focus is to transform bfe into a production ready tool.

This repository includes a development example, a "production" example, and various BIBFRAME Profiles with which to begin experimenting. In order to get started with bfe quickly and easily, there are two main aspects of bfe: a javascript library and an accompanying CSS file. The packaged javascript library bundles a few additional libraries, some of which are JQuery, Lo-Dash, elements from Twitter's Bootstrap.js, and Twitter's typeahead.js. The CSS bundle includes mostly elements of Twitter's Bootstrap and a few additional custom CSS declarations.

Browser Support

• Chrome 34
• Firefox 24+
• Safari - 6+
• Internet Explorer 10+
• Opera - 12+

NOTE: bfe has also not been thoroughly tested in the browsers for which support is currently listed. It has been developed primarily using Chrome. It has been tested in both Chrome and Safari mobile versions.

Developers

From a design standpoint, the objective with bfe was to create the simplest 'pluggable' form editor one can to maximize experimental implementer's abilities to create/edit linked data. The current focus is to transform bfe into a production ready tool.

All contributions are welcome. If you do not code, surely you will discover an issue you can report.

User Authentication with AWS Cognito

We currently use AWS Cognito to manage the authentication of users. Cognito uses a client id to manage the connection to this application (see awsClientID() in src/Config.js). Whenever changes are made to the AWS configuration this client_id changes and so you must also submit a PR to change the client id in this application as well. In the production environment this will be handled via an environment variable, but for local functionality and testing, the hard-coded value must be updated, or you must set the environment variable AWS_CLIENT_ID to be the current client id.

Cross Platform Testing

We use open source BrowserStack accounts for cross-platform/browser testing. See the Sinopia Editor wiki for more details about how to get an account.

Testing Honeybadger

To trigger a test exception, doubleclick "The underdrawing for the new world of linked data in libraries" on the home page.

Acknowledgements

In addition to all the good people who have worked on JQuery, Lo-Dash, Twitter's Bootstrap, Twitter's typeahead.js, require.js, [dryice], and more, all of whom made this simpler, special recognition needs to go to the developers who have worked on Ajax.org's Ace editor and the fine individuals at Zepheira.

Using require.js, Ace's developers figured out a great way to bundle their code into a single distributable. Ace's methods were studied and emulated, and when that wasn't enough, their code was ported (with credit, of course, and those snippets were ported only in support of building the package with dryice). The Ace's devs also just have a really smart way of approaching this type of javascript project.

In late 2013, and demoed at the American Library Association's Midwinter Conference, Zepheira developed a prototype BIBFRAME Editor. Although that project never moved beyond an experimental phase, Zepheira's work was nevertheless extremely influential, especially with respect to bfe's UI design. (None of the code in bfe was ported from Zepheira's prototype.) Zepheira also developed the BIBFRAME Profile Specification.

Contributors

• Jeremy Nelson
• Kevin Ford
• Kirk Hess

Index Data:

• Charles Ledvina
• Wayne Schneider

Maintainer

• LD4P2 Sinopia Project Team
  • GitHub

License

Unless otherwise noted, code that is originally developed by Stanford University in the Sinopia Editor is licensed under the Apache 2.

Original bfe code is in the Public Domain.

http://creativecommons.org/publicdomain/zero/1.0/

NOTE: bfe includes or depends on software from other open source projects, all or most of which will carry their own license and copyright. The Public Domain mark stops at bfe original code and does not convey to these projects.