v0.14.0

• revert back to production (master) branch - without the clearmash changes
• dockerize the api web app

0.13.7

• should not raise exception in case of items from unknown collection
• #215 added support for images field - returned by image and search apis (if exists in source doc)

Deployment (done on dev)

• Need to modify elasticsearch index mapping
  • either drop your index and create new
  • or use put mapping to update an existing index
    • something like this (replace the host and index name):
    • curl -XPUT 'localhost:9200/index_name/_mapping/common?pretty' -H 'Content-Type: application/json' -d'{"properties":{"images":{"type": "object"}}}'

0.13.6

• #207 get item API should return related documents
  • added related_documents field to get item API response
  • it contains dict of related document categories (source dependant), each with the related documents data (as received for search results)
• #212 fixed linkify api for items with missing title in a lanauge
• added docker + minor improvements to elasticsearch index creation

0.13.5

• #201 preliminary support for getting item data from new ES, based on slug
• #203 add slugs attribute + general fixes for ES indexing
• #205 Fix limitation with title suggest field mapping which caused sync to fail
• #209 fix wizard search to work with the new ES data
• #208 add google embed map url to get item API (only for clearmash items which have it)

deployment (to dev)

drop index and recreate index mapping (to allow full re-sync with slugs)

gcloud compute ssh bhs-dev-3
sudo su -l bhs
# stop the pipelines
cd mojp-dbs-pipelines
make docker-stop
# delete the index
curl -X DELETE 10.240.0.7:9200/next-mojp-dev-index-1
curl -X DELETE 10.240.0.7:9200/next-mojp-dev
# create the index
cd ../api
PYTHONPATH=. env/bin/python scripts/elasticsearch_create_index.py

0.13.4

• #162 added linkify API docs
  • this API receives a block of html (or text) and returns all the places / personalities / family names which appear in that text (title and url for each item)
• #188 APIs should use data from new pipeline
  • modified APIs to handle the new data schema, mostly search API
  • pipelines content_html is converted to content_text - currently it only converts br html elements to new lines
  • /geo/places and linkify apis were modified to use elasticsearch
  • elasticsearch index creation was modified to match the new schema
  • tests were skipped, until we fix the data / expand the features of the new data schema

deployment notes

• switched elasticsearch to use next-mojp-dev-index-1

0.13.3

• #187 apply boost for search results according to collection

deployment to production

• upgrade from 0.12.6
• master...dev

dbs-back$ git checkout master && git pull bh master
dbs-back$ git pull bh 0.13.3
dbs-back$ git push bh master

0.13.2

• API response includes a URL for each image of an item
• #181 General search results include persons using all text params (including unified place params).
• #180 Persons advanced search API supports unified place field search param.

0.13.1

• #172 Fix API suggest documentation to include suggestions by collection
  • http://devapi.dbs.bh.org.il/v1/docs
• #174 make persons search case insensitive

DevOps Changes

• fabfile: keep python env directory between installations - in case OS / environment specific dependencies are installed manually, otherwise they are removed each time

0.13.0

In This Release

• #167 Added Suggestions to API documentation

  • Added documentation for suggestions includes general search box suggestions, and wizard search boxes suggestions (familyNames and places)

• #165 API returns image URLs according to source, leading to AWS s3 (for external sources) or gs for BH/BHP pictures.

  deployment notes:
  In order to implement these modifications:

  • Production server_app configuration file should be updated.
  • bagnowka items should be inserted to production databases (Mongodb and ES)

• Fix: v1_endpoints shows error log messages.

deployment notes

• Production server_app configuration file should be updated
  • @Libisch - please edit more specific instructions, which file should be editted (/etc/bhs/app_server.yaml?) what's the value that should be added?
• bagnowka items should be inserted to production databases (Mongodb and ES)
  • see these instructions for how to do it

0.12.6

0.12 final release

includes the changes from the following releases: 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.12.6

important notes

• this release requires deleting all data in Elasticsearch and then recreating it from Mongo - this process can take a few days and complicates the deployment
• however (if everything goes according to plan), no down-time is expected

summary of changes

• fixed suggest api to support the frontend suggest feature
• major refactoring and bug-fixing in the sync of items between mongo and elasticsearch
• changes to ensure proper indexing in Elasticsearch - to support search requirements
• added persons (family trees) data to elasticsearch with support for searching from the general search
• improvements and bug fixing in the API docs - http://devapi.dbs.bh.org.il/v1/docs

check the releass notes for more specific changelog

production deployment plan

pre-deploy

• ssh bhs-prod
• edit nginx conf - change the docs alias
  • sudo nano /etc/nginx/sites-available/bhs_api
  • add _build to the docs alias
• edit /etc/bhs/app_server.yaml
  • add attribute elasticsearch_data_index with value of the mongo db name
• create bhs-prod-db server
  • use instructions here - https://github.com/Beit-Hatfutsot/beit-hatfutsot-devops/blob/master/databases/ELASTICSEARCH.md#using-docker
  • same as the bhs-dev-db server
• copy latest dbs-back version on that server and configure the environment
  • wget https://github.com/Beit-Hatfutsot/dbs-back/archive/0.12.6.tar.gz
  • extract, create venv, install requirements etc..
  • create /etc/bhs/app_server.yaml - can copy from bhs-prod server
• create new elasticsearch index
  • scripts/elasticsearch_create_index.py --index mojp-prod-`date +%Y-%m-%d`
• copy all items from mongo to the new ES index - for all collections except persons
  • scripts/ensure_required_metadata.py --add --index mojp-prod-`date +%Y-%m-%d` --collection places
  • scripts/ensure_required_metadata.py --add --index mojp-prod-`date +%Y-%m-%d` --collection familyNames
  • scripts/ensure_required_metadata.py --add --index mojp-prod-`date +%Y-%m-%d` --collection personalities
  • scripts/ensure_required_metadata.py --add --index mojp-prod-`date +%Y-%m-%d` --collection movies
  • scripts/ensure_required_metadata.py --add --index mojp-prod-`date +%Y-%m-%d` --collection photoUnits
  • had to manually delete some invalid data from mongo:
    • db.persons.remove({"_id": ObjectId('581f46d776f102629df5eb8f')})
• create an alias for the index
  • curl -X PUT localhost:9200/mojp-prod-`date +%Y-%m-%d`/_alias/mojp-prod
• make sure elasticsearch instance is accessible from bhs-prod

deploy

• push the code
  • dbs-back$ git checkout master && git pull bh master
  • dbs-back$ git pull bh 0.12.6
  • dbs-back$ git push bh master
• check deployment status in codeship dbs-back
• sanity -
  • http://api.dbs.bh.org.il/v1/person?last_name=HOCH
  • http://api.dbs.bh.org.il/v1/search?from_=0&q=hoch
• edit /etc/bhs/app_server.yaml - modify elasticsearch config to point to bhs-prod-db server + mojp-prod index
• sudo supervisorctl restart uwsgi
• sanity -
  • http://api.dbs.bh.org.il/v1/person?last_name=HOCH
  • http://api.dbs.bh.org.il/v1/search?from_=0&q=hoch

post-deploy

• update the persons data - will take a long time
  • scripts/ensure_required_metadata.py --add --collection persons