diff --git a/learn/product-and-developer-guide/other/data-migration.md b/learn/product-and-developer-guide/other/data-migration.md index 1a436c9..06b9e1a 100644 --- a/learn/product-and-developer-guide/other/data-migration.md +++ b/learn/product-and-developer-guide/other/data-migration.md @@ -66,9 +66,11 @@ migration flow diagram (migration-job flow & migration-process workflow) {% hint style="info" %} Note: -1. Jenkins Job '**Neo4jElasticSearchSyncTool**' is used to insert the events into 'csp-migrator' job input topic. 'csp-migrator' job will further insert topics into 'live-node-publisher' job and 'live-video-stream-generator' jobs based on conditions. Jenkins job command: **migratecspdata** -2. '**cassandra-data-migration'** job is to be used for migration of '_dialcode\_images'_ and '_dialcode\_batch'_ cassandra tables in '_dialcodes_**'** keyspace. -3. **We suggest to run the migration flink jobs in a separate kafka setup with increased processing ability and storage for storing all kakfa events and logs.** +* Jenkins Job '**Neo4jElasticSearchSyncTool**' is used to insert the events into 'csp-migrator' job input topic. 'csp-migrator' job will further insert topics into 'live-node-publisher' job and 'live-video-stream-generator' jobs based on conditions. Jenkins job command: **migratecspdata** +* '**cassandra-data-migration'** job is to be used for migration of '_dialcode\_images'_ and '_dialcode\_batch'_ cassandra tables in '_dialcodes_**'** keyspace. +* **Run the migration flink jobs in a separate kafka setup with increased processing ability and storage for storing all kakfa events and logs.** +* **Increase the infra for neo4j. Also, increase the neo4j max heap size in neo4j conf file.** +* **Increase infra for logstash, search-indexer flink job and ElasticSearch to handle the neo4j transaction data sync.** {% endhint %} The content migration should execute in the below order only. Otherwise there is a chances of migration failure because of dependent content is not yet migrated. [more details](https://docs.google.com/spreadsheets/d/13DaXCx8uToOwinlAPxvTat8NELxiPgG4KXATcKaJm\_c/edit#gid=1675310401\&range=K3) @@ -82,18 +84,31 @@ The content migration should execute in the below order only. Otherwise there is | Sequence | Type | Sync Tool Jenkins Parameters | | -------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | Video Asset | --graphId domain --objectType Asset --mimeType video/webm,video/mp4 --limit 1000 | -| 2 | Other Asset | --graphId domain --objectType Asset --limit 1000 | -| 3 | Video Content | --graphId domain --objectType Content,ContentImage --mimeType video/mp4,video/webm -limit 1000 | -| 4 | Plugin, Youtube Content, PDF Content,EPUB Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.plugin-archive,video/x-youtube,application/pdf,application/epub --limit 1000 | -| 5 | AssessmentItem | --graphId domain --objectType AssessmentItem --limit 1000 | -| 6 | ItemSet | --graphId domain --objectType ItemSet --limit 1000 | -| 7 | Question | --graphId domain --objectType Question -limit 1000 | -| 8 | QuestionSet | --graphId domain --objectType QuestionSet,QuestionSetImage -limit 1000 | -| 9 | H5P Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.h5p-archive -limit 1000 | -| 10 | HTML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.html-archive -limit 1000 | -| 11 | ECML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.ecml-archive -limit 1000 | -| 12 | Collection | --graphId domain --objectType Collection,CollectionImage --mimeType application/vnd.ekstep.content-collection -limit 1000 | +| 1 | Video Asset | --graphId domain --objectType Asset --mimeType video/webm,video/mp4 --delay 2000 | +| 2 | Other Asset | --graphId domain --objectType Asset --delay 2000 | +| 3 | Video Content | --graphId domain --objectType Content,ContentImage --mimeType video/mp4,video/webm --delay 2000 | +| 4 | Plugin, Youtube Content, PDF Content,EPUB Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.plugin-archive,video/x-youtube,application/pdf,application/epub --delay 2000 | +| 5 | AssessmentItem | --graphId domain --objectType AssessmentItem --delay 2000 | +| 6 | ItemSet | --graphId domain --objectType ItemSet --delay 2000 | +| 7 | H5P Content | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.h5p-archive --delay 2000 | +| 8 | HTML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.html-archive --delay 2000 | +| 9 | ECML | --graphId domain --objectType Content,ContentImage --mimeType application/vnd.ekstep.ecml-archive --delay 2000 | +| 10 | Collection | --graphId domain --objectType Collection,CollectionImage --mimeType application/vnd.ekstep.content-collection --delay 2000 | + +{% hint style="info" %} +Note: + +* ECML migration can be triggered only after Asset, AssessmentItem and ItemSet migration is completed. +* For Collection migration to be triggered, pre-requisites are: + + a. Question, QuestionSet migration should be completed (as part of Inquiry BB). + + b. All assets and contents are to be migrated successfully. + + c. All migrated contents data should be synced to ElasticSearch. +{% endhint %} + + ### Migration status: migrationVersion of the node object diff --git a/use/release-notes/release-5.2.0-ongoing.md b/use/release-notes/release-5.2.0-ongoing.md index 5dd8a23..28f5573 100644 --- a/use/release-notes/release-5.2.0-ongoing.md +++ b/use/release-notes/release-5.2.0-ongoing.md @@ -64,10 +64,10 @@ valid_cloudstorage_base_urls: '["https://sunbirddevbbpublic.blob.core.windows.ne | ------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Schema upload | Deploy/Kubernetes/UploadSchema | NA | [release-5.2.0\_RC2](https://github.com/project-sunbird/knowledge-platform/tree/release-5.2.0\_RC2) | | | Kafka setup | Deploy/KnowledgePlatform/KafkaSetup | NA | [release-5.2.0\_RC3](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC3) | | -| Sync Tool | Build/KnowledgePlatform/SyncTool | [release-5.2.0\_RC3](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC3) | NA | **migratecspdata** - **** Add this command to the command list in the Deploy/KnowledgePlatform/Neo4jElasticSearchSyncTool Jenkins job. | +| Sync Tool | Build/KnowledgePlatform/SyncTool | [release-5.2.0\_RC5](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC5) | NA | **migratecspdata** - **** Add this command to the command list in the Deploy/KnowledgePlatform/Neo4jElasticSearchSyncTool Jenkins job. | | Knowledge-platform |

Build/Kubernetes/Content

Build/Kubernetes/Taxonomy

| [release-5.2.0\_RC2](https://github.com/project-sunbird/knowledge-platform/tree/release-5.2.0\_RC2) | [release-5.1.0\_RC2](https://github.com/project-sunbird/sunbird-devops/tree/release-5.1.0\_RC2) | Build and deploy the content-service and taxonomy-service. Please find the configuration changes [here](https://knowlg.sunbird.org/use/release-notes/release-5.2.0-ongoing#configurations). | -| Sunbird-learning-service | Build/KnowledgePlatform/Learning | [release-5.2.0\_RC3](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC3) | [release-5.2.0\_RC3](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC3) | | -| Knowledge-platform-jobs | Build/KnowledgePlatform/FlinkJobs | [release-5.2.0\_RC5](https://github.com/project-sunbird/knowledge-platform-jobs/tree/release-5.2.0\_RC5) | [release-5.2.0\_RC4](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC4) |

Add the below job list to the job_names_to_deploy list in the Deploy/KnowledgePlatform/FlinkJobs jenkins job

csp-migrator,

live-node-publisher,

live-video-stream-generator,

cassandra-data-migration

Deploy the

assessment-enrichment,

content-publish,

qrcode-image-generator, search-indexer,

csp-migrator,

live-node-publisher,

live-video-stream-generator,

cassandra-data-migration flink jobs.

Please find the configuration changes here.

| +| Sunbird-learning-service | Build/KnowledgePlatform/Learning | [release-5.2.0\_RC5](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC5) | [release-5.2.0\_RC5](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC5) | | +| Knowledge-platform-jobs | Build/KnowledgePlatform/FlinkJobs | [release-5.2.0\_RC5](https://github.com/project-sunbird/knowledge-platform-jobs/tree/release-5.2.0\_RC5) | [release-5.2.0\_RC5](https://github.com/project-sunbird/sunbird-learning-platform/tree/release-5.2.0\_RC5) |

Add the below job list to the job_names_to_deploy list in the Deploy/KnowledgePlatform/FlinkJobs jenkins job

csp-migrator,

live-node-publisher,

live-video-stream-generator,

cassandra-data-migration

Deploy the

assessment-enrichment,

content-publish,

qrcode-image-generator, search-indexer,

csp-migrator,

live-node-publisher,

live-video-stream-generator,

cassandra-data-migration flink jobs.

Please find the configuration changes here.

| | Sunbird-Collection-Editor | NA | [**v5.2.4**](https://www.npmjs.com/package/@project-sunbird/sunbird-collection-editor/v/5.2.4)**** | NA | NA | | Sunbird-Content-Editor | Build/Plugins/ContentEditor | [**release-5.2.0\_RC1**](https://github.com/project-sunbird/sunbird-content-editor/releases/tag/release-5.2.0\_RC1)**** | [release-5.1.0\_RC2](https://github.com/project-sunbird/sunbird-devops/tree/release-5.1.0\_RC2) | NA | | Sunbird-Generic-Editor | Build/Plugins/GenericEditor | [**release-5.2.0\_RC1**](https://github.com/project-sunbird/sunbird-generic-editor/releases/tag/release-5.2.0\_RC1) | [release-5.1.0\_RC2](https://github.com/project-sunbird/sunbird-devops/tree/release-5.1.0\_RC2) | NA |