Devrel 1205/partial sync

package.json

@@ -19,8 +19,8 @@
   "files": [
     "./build/**/*"
   ],
-  "main": "./build/src/index.js",
-  "types": "./build/src/index.d.ts",
+  "main": "./build/src/public.js",
+  "types": "./build/src/public.d.ts",
   "bin": {
     "data-ops": "./build/src/index.js"
   },
@@ -70,4 +70,4 @@
     "uuid": "^9.0.1",
     "vitest": "^2.1.1"
   }
-}
+}

src/commands/syncModel/run/README.md

@@ -4,36 +4,50 @@
 > Synchronizing content model might lead to irreversible changes to the environment such as:
 > - Deletion of content by deleting elements from a content type
 > - Deletion of used taxonomies
+> - Removing roles limitations in workflows (see [known limitations](#known-limitations))
 
-The `sync-model run` command synchronizes the **source content model** into the **target environment** via [Kontent.ai Management API](https://kontent.ai/learn/docs/apis/openapi/management-api-v2/). The source content can be obtained from an existing Kontent.ai environment (considering you have access to the required credentials) or a folder structure in a required format (see [sync-model export](../export/README.md) command for more information).
+The `sync-model run` command synchronizes the **source content model** into the **target environment** via [Kontent.ai Management API](https://kontent.ai/learn/docs/apis/openapi/management-api-v2/). The source content can be obtained from an existing Kontent.ai environment (considering you have access to the required credentials) or a folder structure in a required format (see [sync-model export](../export/README.md) command for more information). Using the CLI command you can filter data by entity type with the mandatory `--entities` parameter. For more advanced filtering, you can use the `entities` parameter through [programmatic sync](#sync-model-programmatically), where you can define custom filter predicate functions for each entity. These functions are optional and, if not provided, the entity WON'T be synced by default.
 
-In the context of this command, the content model is represented by the following entities: `Taxonomies`, `Content Types`, `Content Type Snippets`, `Web Spotlight`, and `Asset Folders`. The command begins by comparing the provided content models and generates patch operations, which are then printed as the **environment diff**.  
+> [!CAUTION]
+> Partial sync requires that all dependent entities are either already present in the environment or included in the sync process. Otherwise, the partial sync may fail, leaving your environment in an incomplete state.
 
-**We strongly encourage you to examine the changes before you proceed with the model synchronization.** Following the diff, a validation is performed to ensure the sync operation can succeed. Should the validation find any inconsistencies, the operation may be terminated. Otherwise, you will be asked to confirmation in order to begin the synchronization.
+In the context of this command, the content model is represented by the following entities: `Taxonomies`, `Content Types`, `Content Type Snippets`, `Web Spotlight`, `Asset Folders`, `Collections`, `Spaces`, `Languages` and `Workflows`. The command begins by comparing the provided content models and generates patch operations, which are then printed as the **environment diff**.  
+
+**We strongly encourage you to examine the changes before you proceed with the model synchronization.** Following the diff, a validation is performed to ensure the sync operation can succeed. Should the validation find any inconsistencies, the operation may be terminated. Otherwise, you will be asked for confirmation to begin the synchronization.
 
 ## Key principles
 - Sync matches entities between the source and the target models via a `codename`.
 - The command does not sync `external_id` properties of content model entities (existing `external_id` cannot be changed and can conflict with other entities). 
-- If the model contains `guidelines` that reference content items or assets that are not present in the target environment, they will be referenced by their `external_id`(if externalId is non-existent `id` is used as `external_id`) after the synchronization. Remember to migrate any missing content to the target environment either beforehand (preferably) or afterwards to achieve the desired results.
+- If the model contains `guidelines` that reference content items or assets that are not present in the target environment, they will be referenced by their `external_id`(if externalId is non-existent `id` is used as `external_id`) after the synchronization. Remember to migrate any missing content to the target environment either beforehand (preferably) or afterward to achieve the desired results.
 - If `Linked items` or `Rich text element` references non-existent content types, they will be referenced using the `external_id` after the synchronization (one or more entity `codenames` are used to form the `external_id`).
 
 ## Sync model conditions
 To successfully synchronize the content model, we introduced a couple of conditions your environment **must follow** before attempting the sync:
 - There mustn't be an operation that changes the content type or content type snippet's element type - checked by validation.
 - There mustn't be an operation deleting a used content type (there is at least one content item of that type) - checked by validation.
-- Source content model mustn't reference a deleted taxonomy group - not checked by validation!
-- If providing source content model via a folder, you must ensure that the content model is in a valid state - not checked by validation!
-- Both environments must have the same status of Web Spotlight (either activated, or deactivated) - not checked by validation!
+- There mustn't be an operation deleting a used collection (there is at least one content item in that collection) - checked by validation.
+- The source content model mustn't reference a deleted taxonomy group - not checked by validation!
+- If providing source content model via a folder, you must ensure that the content model is in a valid state 
+  - Files are partially checked to see whether they meet the MAPI structure using `zod` validation. Not all conditions are checked! (e.g. whether the used codename exists)
+
+## Known limitations
+Using Management API introduces some limitations:
+- Snippet element can't be referenced in the same request it's created in. Because of this, the tool can't move it to the correct place in the content type.
+- Asset folders cannot be moved so if they are in a different location in the source environment, they are removed and created in the new place.
+- Asset folders cannot be deleted (or moved, see the previous point) if they contain assets. The command is not able to check this without loading all the assets in the project so it doesn't check it. Please, make sure that all folders that will be deleted does not contain any assets. You can see what folders will be deleted in the generated diff `sync-model diff ...`.
+- Languages cannot be deleted, instead, they are deactivated. Their name and codename are replaced with the first 8 characters of a randomly generated UUID (name and codename have a limit of 25 characters).
+- Roles cannot be added or patched via MAPI and are therefore not synced.
+  - Consequently, **all role restrictions for workflow steps are lost** when adding a new workflow or adjusting an existing one during sync.
 
 ## Usage
 ```bash
-npx @kontent-ai/data-ops@latest sync-model run --targetEnvironmentId=<target-environment-id> --targetApiKey=<target-management-API-key> --sourceEnvironmentId=<source-environment-id>
+npx @kontent-ai/data-ops@latest sync-model run --targetEnvironmentId=<target-environment-id> --targetApiKey=<target-management-API-key> --sourceEnvironmentId=<source-environment-id> --entities contentTypes contentTypeSnippets taxonomies
 --sourceApiKey=<source-api-key>
 ```
 OR
 
 ```bash
-npx @kontent-ai/data-ops@latest sync-model run --targetEnvironmentId=<target-environment-id> --targetApiKey=<target-management-API-key> --folderName=<path-to-content-folder>
+npx @kontent-ai/data-ops@latest sync-model run --targetEnvironmentId=<target-environment-id> --targetApiKey=<target-management-API-key> --folderName=<path-to-content-folder> --entities contentTypes contentTypeSnippets taxonomies 
 ```
 
 > [!NOTE]  
@@ -43,7 +57,8 @@ npx @kontent-ai/data-ops@latest sync-model run --targetEnvironmentId=<target-env
 >   "sourceEnvironmentId": "<source-env-id>",
 >   "sourceApiKey": "<source-mapi-key>",
 >   "targetEnvironmentId": "<target-env-id>",
->   "targetApiKey": "<target-mapi-key>"
+>   "targetApiKey": "<target-mapi-key>",
+>    "entities": ["contentTypes", "contentTypeSnippets", "taxonomies", "collections", "assetFolders", "spaces", "languages", "webSpotlight", "workflows"]
 > }
 > ```
 
@@ -60,28 +75,20 @@ const params: SyncModelRunParams = {
   sourceEnvironmentId: "<source-env-id>",
   sourceApiKey: "<source-mapi-key>",
   targetEnvironmentId: "<target-env-id>",
-  targetApiKey: "<target-mapi-key>"
+  targetApiKey: "<target-mapi-key>",
+  entities: {
+    contentTypes: () => true, // will sync
+    taxonomies: () => false, // won't sync
+    languages: (lang) => lang.codename === "default" // will sync only codename with default codename
+  }
 };
 
 await syncModelRun(params);
 ```
 
-## Known limitations
-Using Management API introduces some limitations:
-- Snippet element can't be referenced in the same request it's created in. Because of this, the tool can't move it to the correct place in the content type.
-- Asset folders cannot be moved so if they are in a different location in the source environment, they are removed and created in the new place.
-- Asset folders cannot be deleted (or moved, see the previous point) if they contain assets. The command is not able to check this without loading all the assets in the project so it doesn't check it. Please, make sure that all folders that will be deleted does not contain any assets. You can see what assets will be deleted in the generated diff `sync-model diff ...`.
-- Languages cannot be deleted, instead, they are deactivated. Their name and codename are replaced with the first 8 characters of a randomly generated UUID (name and codename have a limit of 25 characters).
-- Roles cannot be added or patched via MAPI and are therefore not synced.
-  - Consequently, **all role restrictions for workflow steps are lost** when adding a new workflow or adjusting an existing one during sync.
-
 ## Contributing
 
-When syncing the content model, add, patch, and delete operations must come in a specific order, otherwise, MAPI won't be able to reference some entities. Check the image below for more details.
-
-![Content model operations order](./images/content_model_operations_order.png)
-
-To successfully patch a content type, its operations for content groups and elements must also be in a specific order:
+To successfully patch a content type, its operations for content groups and elements must be in a specific order:
 ![Content type operations order](./images/content_type_operations_order.png)
 
 ### Taxonomy diff handler

src/commands/syncModel/run/run.ts

@@ -2,12 +2,11 @@ import chalk from "chalk";
 import { match, P } from "ts-pattern";
 
 import { logError, logInfo, LogOptions } from "../../../log.js";
+import { syncEntityChoices, SyncEntityName } from "../../../modules/sync/constants/entities.js";
 import { printDiff } from "../../../modules/sync/printDiff.js";
-import { sync } from "../../../modules/sync/sync.js";
-import { getDiffModel, SyncModelRunParams, validateTargetEnvironment } from "../../../modules/sync/syncModelRun.js";
+import { SyncEntities, syncModelRunInternal, SyncModelRunParams } from "../../../modules/sync/syncModelRun.js";
 import { requestConfirmation } from "../../../modules/sync/utils/consoleHelpers.js";
 import { RegisterCommand } from "../../../types/yargs.js";
-import { createClient } from "../../../utils/client.js";
 import { simplifyErrors } from "../../../utils/error.js";
 
 const commandName = "run";
@@ -53,6 +52,14 @@ export const register: RegisterCommand = yargs =>
           implies: ["sourceEnvironmentId"],
           alias: "sk",
         })
+        .option("entities", {
+          alias: "e",
+          type: "array",
+          choices: syncEntityChoices,
+          describe: `Sync specified entties. Allowed entities are: ${syncEntityChoices.join(", ")}`,
+          demandOption: "You need to provide the what entities to sync.",
+          conflicts: "exclude",
+        })
         .option("skipConfirmation", {
           type: "boolean",
           describe: "Skip confirmation message.",
@@ -64,6 +71,7 @@ type SyncModelRunCliParams =
   & Readonly<{
     targetEnvironmentId: string;
     targetApiKey: string;
+    entities: ReadonlyArray<SyncEntityName>;
     folderName?: string;
     sourceEnvironmentId?: string;
     sourceApiKey?: string;
@@ -74,47 +82,33 @@ type SyncModelRunCliParams =
 const syncModelRunCli = async (params: SyncModelRunCliParams) => {
   const resolvedParams = resolveParams(params);
 
-  const targetClient = createClient({
-    apiKey: params.targetApiKey,
-    environmentId: params.targetEnvironmentId,
-    commandName,
-  });
-
-  const diffModel = await getDiffModel(resolvedParams, targetClient, commandName);
-
-  logInfo(params, "standard", "Validating patch operations...\n");
-
   try {
-    await validateTargetEnvironment(diffModel, targetClient);
-  } catch (e) {
-    logError(params, JSON.stringify(e, Object.getOwnPropertyNames(e)));
-    process.exit(1);
-  }
+    await syncModelRunInternal(resolvedParams, commandName, async (diffModel) => {
+      printDiff(diffModel, new Set(params.entities), params);
 
-  printDiff(diffModel, params);
-
-  const warningMessage = chalk.yellow(
-    `⚠ Running this operation may result in irreversible changes to the content in environment ${params.targetEnvironmentId}. Mentioned changes might include:
+      const warningMessage = chalk.yellow(
+        `⚠ Running this operation may result in irreversible changes to the content in environment ${params.targetEnvironmentId}. Mentioned changes might include:
 - Removing content due to element deletion
 OK to proceed y/n? (suppress this message with --sw parameter)\n`,
-  );
+      );
 
-  const confirmed = !params.skipConfirmation ? await requestConfirmation(warningMessage) : true;
+      const confirmed = !params.skipConfirmation ? await requestConfirmation(warningMessage) : true;
 
-  if (!confirmed) {
-    logInfo(params, "standard", chalk.red("Operation aborted."));
+      if (!confirmed) {
+        logInfo(params, "standard", chalk.red("Operation aborted."));
+        process.exit(1);
+      }
+    });
+  } catch (e) {
+    logError(params, JSON.stringify(e, Object.getOwnPropertyNames(e)));
     process.exit(1);
   }
-
-  await sync(
-    targetClient,
-    diffModel,
-    params,
-  );
 };
 
-const resolveParams = (params: SyncModelRunCliParams): SyncModelRunParams =>
-  match(params)
+const resolveParams = (params: SyncModelRunCliParams): SyncModelRunParams => {
+  const entities = createSyncEntitiesParameter(params.entities);
+
+  const x = match(params)
     .with(
       { sourceEnvironmentId: P.nonNullable, sourceApiKey: P.nonNullable },
       ({ sourceEnvironmentId, sourceApiKey }) => ({ ...params, sourceEnvironmentId, sourceApiKey }),
@@ -127,3 +121,17 @@ const resolveParams = (params: SyncModelRunCliParams): SyncModelRunParams =>
       );
       process.exit(1);
     });
+
+  return { ...x, entities };
+};
+
+const createSyncEntitiesParameter = (
+  entities: ReadonlyArray<SyncEntityName>,
+): SyncEntities => {
+  const filterEntries = [
+    ...entities.filter(a => a !== "webSpotlight").map(e => [e, () => true]),
+    ...entities.includes("webSpotlight") ? [["webSpotlight", true]] : [],
+  ] as const;
+
+  return Object.fromEntries(filterEntries);
+};

src/modules/sync/constants/entities.ts

@@ -0,0 +1,28 @@
+import { SyncEntities } from "../syncModelRun.js";
+
+export const syncEntityChoices = [
+  "contentTypes",
+  "contentTypeSnippets",
+  "taxonomies",
+  "collections",
+  "assetFolders",
+  "spaces",
+  "languages",
+  "webSpotlight",
+  "workflows",
+] as const;
+
+export type SyncEntityName = (typeof syncEntityChoices)[number];
+
+// includes transitive dependencies
+export const syncEntityDependencies: Record<keyof SyncEntities, ReadonlyArray<keyof SyncEntities>> = {
+  contentTypes: ["contentTypes", "contentTypeSnippets", "taxonomies"],
+  contentTypeSnippets: ["contentTypeSnippets", "taxonomies", "contentTypes"],
+  collections: ["collections"],
+  taxonomies: ["taxonomies"],
+  spaces: ["spaces", "collections"],
+  workflows: ["workflows", "collections", "contentTypes", "contentTypeSnippets", "taxonomies"],
+  assetFolders: ["assetFolders"],
+  languages: ["languages"],
+  webSpotlight: ["webSpotlight", "contentTypes", "contentTypeSnippets", "taxonomies"],
+};

src/modules/sync/diff.ts

@@ -156,7 +156,10 @@ const createDiffModel = <Entity extends Readonly<{ codename: string }>>(
 const getLanguageDiffModel = (
   sourceLanguages: ReadonlyArray<LanguageSyncModel>,
   targetLanguages: ReadonlyArray<LanguageSyncModel>,
-) => {
+): DiffModel["languages"] => {
+  if (sourceLanguages.length === 0 && targetLanguages.length === 0) {
+    return { added: [], updated: new Map(), deleted: new Set() };
+  }
   const sourceDefaultLanguageCodename = getDefaultLang(sourceLanguages).codename;
   const targetDefaultLanguageCodename = getDefaultLang(targetLanguages).codename;
 
@@ -207,5 +210,6 @@ const adjustSourceDefaultLanguageCodename = (
 
 const getDefaultLang = (languages: ReadonlyArray<LanguageSyncModel>) => {
   const defaultLang = languages.find(l => l.is_default);
+
   return defaultLang ?? throwError(`Language enviroment model does not contain default language`);
 };

src/modules/sync/diffEnvironments.ts

@@ -2,6 +2,7 @@ import chalk from "chalk";
 
 import { logError, logInfo, LogOptions } from "../../log.js";
 import { createClient } from "../../utils/client.js";
+import { syncEntityChoices } from "./constants/entities.js";
 import { diff } from "./diff.js";
 import { fetchModel, transformSyncModel } from "./generateSyncModel.js";
 import { createAdvancedDiffFile, printDiff } from "./printDiff.js";
@@ -86,5 +87,5 @@ export const diffEnvironmentsInternal = async (params: DiffEnvironmentsParams, c
 
   return "advanced" in params
     ? createAdvancedDiffFile({ ...diffModel, ...params })
-    : printDiff(diffModel, params);
+    : printDiff(diffModel, new Set(syncEntityChoices), params);
 };