feat: amplify export

* Refactor/packaging (aws-amplify#8547) * feat(amplify-provider-awscloudformation): refactor of the exporting resources * feat(amplify-provider-awscloudformation): refactored export to write files to external path * refactor: clean up * refactor: remove unused * fix: mispelled url * feat: fall back on push globing and template url * feat: export * feat: modify generation of lambda layer version content * fix(amplify-category-function): lambda layers filter stacks * feat: added command line for export * test(amplify-provider-awscloudformation): added tests for export resources * perf: removed unused * refactor: removed unused * fix: make the tags file pascal case * docs(amplify-provider-awscloudformation): added some documentation * feat(amplify-provider-awscloudformation): added warning and folder perms * fix: check in constants change * refactor: addressing PR feedback * refactor: es6 export * feat: minor changes for integration testing * refactor: pr comments * fix: cleared commented out code removed backup * ci: revert config file to main * Feat: export pull front end (aws-amplify#8488) * feat: export (aws-amplify#8486) * feat(amplify-provider-awscloudformation): refactor of the exporting resources * feat(amplify-provider-awscloudformation): refactored export to write files to external path * refactor: clean up * refactor: remove unused * fix: mispelled url * feat: fall back on push globing and template url * feat: export * feat: modify generation of lambda layer version content * fix(amplify-category-function): lambda layers filter stacks * feat: added command line for export * test(amplify-provider-awscloudformation): added tests for export resources * perf: removed unused * refactor: removed unused * fix: make the tags file pascal case * docs(amplify-provider-awscloudformation): added some documentation * feat(amplify-provider-awscloudformation): added warning and folder perms * fix: check in constants change * refactor: addressing PR feedback * refactor: es6 export * feat: minor changes for integration testing * refactor: pr comments * fix: cleared commented out code removed backup * ci: revert config file to main * feat: wip export pull * feat: amplify export pull to genereate front end config files * fix: merge fixes from export * refactor: removed unused * fix: some language fixes and bug fix with notification * test: e2e tests * test: codecov and test fix * Refactor/packaging (aws-amplify#8547) * feat(amplify-provider-awscloudformation): refactor of the exporting resources * feat(amplify-provider-awscloudformation): refactored export to write files to external path * refactor: clean up * refactor: remove unused * fix: mispelled url * feat: fall back on push globing and template url * feat: export * feat: modify generation of lambda layer version content * fix(amplify-category-function): lambda layers filter stacks * feat: added command line for export * test(amplify-provider-awscloudformation): added tests for export resources * perf: removed unused * refactor: removed unused * fix: make the tags file pascal case * docs(amplify-provider-awscloudformation): added some documentation * feat(amplify-provider-awscloudformation): added warning and folder perms * fix: check in constants change * refactor: addressing PR feedback * refactor: es6 export * feat: minor changes for integration testing * refactor: pr comments * fix: cleared commented out code removed backup * ci: revert config file to main * Feat: export pull front end (aws-amplify#8488) * feat: export (aws-amplify#8486) * feat(amplify-provider-awscloudformation): refactor of the exporting resources * feat(amplify-provider-awscloudformation): refactored export to write files to external path * refactor: clean up * refactor: remove unused * fix: mispelled url * feat: fall back on push globing and template url * feat: export * feat: modify generation of lambda layer version content * fix(amplify-category-function): lambda layers filter stacks * feat: added command line for export * test(amplify-provider-awscloudformation): added tests for export resources * perf: removed unused * refactor: removed unused * fix: make the tags file pascal case * docs(amplify-provider-awscloudformation): added some documentation * feat(amplify-provider-awscloudformation): added warning and folder perms * fix: check in constants change * refactor: addressing PR feedback * refactor: es6 export * feat: minor changes for integration testing * refactor: pr comments * fix: cleared commented out code removed backup * ci: revert config file to main * feat: wip export pull * feat: amplify export pull to genereate front end config files * fix: merge fixes from export * refactor: removed unused * fix: some language fixes and bug fix with notification * test: e2e tests * test: codecov and test fix * feat: export + override reconciled * test: fixed test because of bad merge * fix: added back conditional conditional rebuild * refactor(amplify-provider-awscloudformation): renamed the files to fit * test: fixed idp add storage test * test: fixes type import
akshbhu · Nov 8, 2021 · dff93a7 · dff93a7
1 parent 8240ed9
commit dff93a7
Show file tree

Hide file tree

Showing 39 changed files with 2,271 additions and 124 deletions.
diff --git a/...s/amplify-category-function/src/provider-utils/awscloudformation/types/packaging-types.ts b/...s/amplify-category-function/src/provider-utils/awscloudformation/types/packaging-types.ts
@@ -12,4 +12,5 @@ export type PackageRequestMeta = ResourceTuple & {
 export type Packager = (
   context: $TSContext,
   resource: PackageRequestMeta,
+  isExport?: boolean,
 ) => Promise<{ newPackageCreated: boolean; zipFilename: string; zipFilePath: string }>;
diff --git a/...s/amplify-category-function/src/provider-utils/awscloudformation/utils/layerCloudState.ts b/...s/amplify-category-function/src/provider-utils/awscloudformation/utils/layerCloudState.ts
@@ -28,7 +28,10 @@ export class LayerCloudState {
       const layerVersionList = await lambdaClient.listLayerVersions(isMultiEnvLayer(layerName) ? `${layerName}-${envName}` : layerName);
       const cfnClient = await providerPlugin.getCloudFormationSdk(context);
       const stackList = await cfnClient.listStackResources();
-      const layerStacks = stackList?.StackResourceSummaries?.filter(stack => stack.LogicalResourceId.includes(layerName));
+      const layerStacks = stackList?.StackResourceSummaries?.filter(
+        // do this because cdk does some rearranging of resources
+        stack => stack.LogicalResourceId.includes(layerName) && stack.ResourceType === 'AWS::CloudFormation::Stack',
+      );
       let detailedLayerStack;
 
       if (layerStacks?.length > 0) {

diff --git a/packages/amplify-category-function/src/provider-utils/awscloudformation/utils/package.ts b/packages/amplify-category-function/src/provider-utils/awscloudformation/utils/package.ts
@@ -4,7 +4,8 @@ import { ServiceName } from './constants';
 import { packageFunction } from './packageFunction';
 import { packageLayer } from './packageLayer';
 
-export const packageResource: Packager = async (context, resource) => getPackagerForService(resource.service)(context, resource);
+export const packageResource: Packager = async (context, resource, isExport) =>
+  getPackagerForService(resource.service)(context, resource, isExport);
 
 // there are some other categories (api and maybe others) that depend on the packageFunction function to create a zip file of resource
 // which is why it is the default return value here

diff --git a/...ages/amplify-category-function/src/provider-utils/awscloudformation/utils/packageLayer.ts b/...ages/amplify-category-function/src/provider-utils/awscloudformation/utils/packageLayer.ts
@@ -18,11 +18,11 @@ import { zipPackage } from './zipResource';
 /**
  * Packages lambda layer code and artifacts into a lambda-compatible .zip file
  */
-export const packageLayer: Packager = async (context, resource) => {
+export const packageLayer: Packager = async (context, resource, isExport) => {
   const previousHash = loadPreviousLayerHash(resource.resourceName);
   const currentHash = await ensureLayerVersion(context, resource.resourceName, previousHash);
 
-  if (previousHash === currentHash) {
+  if (!isExport && previousHash === currentHash) {
     // This happens when a Lambda layer's permissions have been updated, but no new layer version needs to be pushed
     return { newPackageCreated: false, zipFilename: undefined, zipFilePath: undefined };
   }
@@ -84,7 +84,10 @@ export const packageLayer: Packager = async (context, resource) => {
   }
 
   const zipFilename = createLayerZipFilename(resource.resourceName, layerCloudState.latestVersionLogicalId);
-  context.amplify.updateAmplifyMetaAfterPackage(resource, zipFilename, { resourceKey: versionHash, hashValue: currentHash });
+  if (!isExport) {
+    // don't  apply an update to Amplify meta on export
+    context.amplify.updateAmplifyMetaAfterPackage(resource, zipFilename, { resourceKey: versionHash, hashValue: currentHash });
+  }
   return { newPackageCreated: true, zipFilename, zipFilePath: destination };
 };
 

diff --git a/packages/amplify-cli-core/src/errors/index.ts b/packages/amplify-cli-core/src/errors/index.ts
@@ -17,8 +17,12 @@ export class SchemaDoesNotExistError extends Error {}
 export class AngularConfigNotFoundError extends Error {}
 export class AppIdMismatchError extends Error {}
 export class UnrecognizedFrameworkError extends Error {}
+export class UnrecognizedFrontendError extends Error {}
 export class ConfigurationError extends Error {}
 export class CustomPoliciesFormatError extends Error {}
+export class ExportPathValidationError extends Error {}
+export class ExportedStackNotFoundError extends Error {}
+export class ExportedStackNotInValidStateError extends Error {}
 
 export class NotInitializedError extends Error {
   public constructor() {

diff --git a/packages/amplify-cli-core/src/state-manager/stateManager.ts b/packages/amplify-cli-core/src/state-manager/stateManager.ts
@@ -211,11 +211,11 @@ export class StateManager {
     JSONUtilities.writeJson(filePath, localAWSInfo);
   };
 
-  getHydratedTags = (projectPath?: string | undefined): Tag[] => {
+  getHydratedTags = (projectPath?: string | undefined, skipProjEnv: boolean = false): Tag[] => {
     const tags = this.getProjectTags(projectPath);
     const { projectName } = this.getProjectConfig(projectPath);
     const { envName } = this.getLocalEnvInfo(projectPath);
-    return HydrateTags(tags, { projectName, envName });
+    return HydrateTags(tags, { projectName, envName }, skipProjEnv);
   };
 
   isTagFilePresent = (projectPath?: string | undefined): boolean => {

diff --git a/packages/amplify-cli-core/src/tags/Tags.ts b/packages/amplify-cli-core/src/tags/Tags.ts
@@ -17,7 +17,7 @@ export function ReadTags(tagsFilePath: string): Tag[] {
   return tags;
 }
 
-export function validate(tags: Tag[]): void {
+export function validate(tags: Tag[], skipProjectEnv: boolean = false): void {
   const allowedKeySet = new Set(['Key', 'Value']);
 
   //check if Tags have the right format
@@ -34,7 +34,8 @@ export function validate(tags: Tag[]): void {
   // check if the tags has valid keys and values
   _.each(tags, tag => {
     const tagValidationRegExp = /[^a-z0-9_.:/=+@\- ]/gi;
-    if (tagValidationRegExp.test(tag.Value)) {
+    const tagValue = skipProjectEnv ? tag.Value.replace('{project-env}', '') : tag.Value;
+    if (tagValidationRegExp.test(tagValue)) {
       throw new Error(
         'Invalid character found in Tag Value. Tag values may only contain unicode letters, digits, whitespace, or one of these symbols: _ . : / = + - @',
       );
@@ -56,19 +57,20 @@ export function validate(tags: Tag[]): void {
   });
 }
 
-export function HydrateTags(tags: Tag[], tagVariables: TagVariables): Tag[] {
+export function HydrateTags(tags: Tag[], tagVariables: TagVariables, skipProjectEnv: boolean = false): Tag[] {
   const { envName, projectName } = tagVariables;
   const replace: any = {
     '{project-name}': projectName,
     '{project-env}': envName,
   };
+  const regexMatcher = skipProjectEnv ? /{project-name}/g : /{project-name}|{project-env}/g;
   const hydrdatedTags = tags.map(tag => {
     return {
       ...tag,
-      Value: tag.Value.replace(/{project-name}|{project-env}/g, (matched: string) => replace[matched]),
+      Value: tag.Value.replace(regexMatcher, (matched: string) => replace[matched]),
     };
   });
-  validate(hydrdatedTags);
+  validate(hydrdatedTags, skipProjectEnv);
   return hydrdatedTags;
 }
 

diff --git a/packages/amplify-cli-core/src/utils/index.ts b/packages/amplify-cli-core/src/utils/index.ts
@@ -3,3 +3,4 @@ export * from './isResourceNameUnique';
 export * from './open';
 export * from './packageManager';
 export * from './recursiveOmit';
+export * from './validate-path';
diff --git a/packages/amplify-cli-core/src/utils/validate-path.ts b/packages/amplify-cli-core/src/utils/validate-path.ts
@@ -0,0 +1,22 @@
+import * as fs from 'fs-extra';
+import { ExportPathValidationError } from '../errors';
+
+/**
+ * Validates whether the path is a directory
+ * @throws {ExportPathValidationError} if path not valid
+ * @param directoryPath to validate
+ */
+export function validateExportDirectoryPath(directoryPath: any) {
+  if (typeof directoryPath !== 'string') {
+    throw new ExportPathValidationError(`${directoryPath} is not a valid path specified by --out`);
+  }
+
+  if (!fs.existsSync(directoryPath)) {
+    throw new ExportPathValidationError(`${directoryPath} does not exist`);
+  }
+
+  const stat = fs.lstatSync(directoryPath);
+  if (!stat.isDirectory()) {
+    throw new ExportPathValidationError(`${directoryPath} is not a valid directory`);
+  }
+}
diff --git a/packages/amplify-cli/amplify-plugin.json b/packages/amplify-cli/amplify-plugin.json
@@ -7,6 +7,7 @@
     "console",
     "delete",
     "env",
+    "export",
     "help",
     "init",
     "logout",

diff --git a/packages/amplify-cli/src/commands/export.ts b/packages/amplify-cli/src/commands/export.ts
@@ -0,0 +1,105 @@
+import { $TSContext, IAmplifyResource, stateManager, UnrecognizedFrontendError, validateExportDirectoryPath } from 'amplify-cli-core';
+import { printer } from 'amplify-prompts';
+import chalk from 'chalk';
+import { getResourceOutputs } from '../extensions/amplify-helpers/get-resource-outputs';
+import Ora from 'ora';
+import { getResources } from './build-override';
+
+export const run = async (context: $TSContext) => {
+  const options = context.input.options;
+  const subCommands = context.input.subCommands;
+  const showHelp = !options || options.help || !options.out;
+  const isPull = !!(subCommands && subCommands.includes('pull'));
+  const showPullHelp = (showHelp || !options.frontend || !options.rootStackName) && isPull;
+
+  if (showHelp && !showPullHelp) {
+    printer.blankLine();
+    printer.info("'amplify export', Allows you to integrate your backend into an external deployment tool");
+    printer.blankLine();
+    printer.info(`${chalk.yellow('--cdk')}         Export all resources with cdk comatibility`);
+    printer.info(`${chalk.yellow('--out')}         Root directory of cdk project`);
+    printer.blankLine();
+    printer.info(`Example: ${chalk.green('amplify export --cdk --out ~/myCDKApp')}`);
+    printer.blankLine();
+    printer.info("'amplify export pull' To export front-end config files'");
+    printer.info("'amplify export pull --help'  to learn");
+    printer.blankLine();
+    return;
+  }
+
+  if (showPullHelp) {
+    const frontendPlugins = context.amplify.getFrontendPlugins(context);
+    const frontends = Object.keys(frontendPlugins);
+    printer.blankLine();
+    printer.info("'amplify export pull', Allows you to genreate frontend config files at a desired location");
+    printer.blankLine();
+    printer.info(`${chalk.yellow('--rooStackName')}         Amplify CLI deployed Root Stack name`);
+    printer.info(`${chalk.yellow('--frontend')}             Front end type ex: ${frontends.join(', ')}`);
+    printer.info(`${chalk.yellow('--out')}                  Directory to write the front-end config files`);
+    printer.blankLine();
+    printer.info(
+      `Example: ${chalk.green(
+        'amplify export pull --rootStackName amplify-myapp-stack-123 --out ~/myCDKApp/src/config/ --frontend javascript',
+      )}`,
+    );
+    printer.blankLine();
+    printer.blankLine();
+    return;
+  }
+  const exportPath = context.input.options['out'];
+  if (isPull) {
+    await createFrontEndConfigFile(context, exportPath);
+  } else {
+    await exportBackend(context, exportPath);
+  }
+};
+
+async function exportBackend(context: $TSContext, exportPath: string) {
+  await buildAllResources(context);
+  const resources = await context.amplify.getResourceStatus();
+  await context.amplify.showResourceTable();
+  const providerPlugin = context.amplify.getProviderPlugins(context);
+  const providers = Object.keys(providerPlugin);
+  for await (const provider of providers) {
+    const plugin = await import(providerPlugin[provider]);
+    await plugin.exportResources(context, resources, exportPath);
+  }
+}
+
+async function buildAllResources(context: $TSContext) {
+  const resourcesToBuild: IAmplifyResource[] = await getResources(context);
+  await context.amplify.executeProviderUtils(context, 'awscloudformation', 'buildOverrides', { resourcesToBuild, forceCompile: true });
+}
+
+async function createFrontEndConfigFile(context: $TSContext, exportPath: string) {
+  const { rootStackName, frontend } = context.input.options;
+
+  const frontendSet = new Set(Object.keys(context.amplify.getFrontendPlugins(context)));
+  if (!frontendSet.has(frontend)) {
+    throw new UnrecognizedFrontendError(`${frontend} is not a supported Amplify frontend`);
+  }
+  const spinner = Ora(`Extracting outputs from ${rootStackName}`);
+
+  spinner.start();
+  const providerPlugin = context.amplify.getProviderPlugins(context);
+  const providers = Object.keys(providerPlugin);
+  try {
+    for await (const provider of providers) {
+      const plugin = await import(providerPlugin[provider]);
+      await plugin.exportedStackResourcesUpdateMeta(context, rootStackName);
+    }
+    spinner.text = `Generating files at ${exportPath}`;
+    const meta = stateManager.getMeta();
+    const cloudMeta = stateManager.getCurrentMeta();
+    const frontendPlugins = context.amplify.getFrontendPlugins(context);
+    const frontendHandlerModule = require(frontendPlugins[frontend]);
+    validateExportDirectoryPath(exportPath);
+    await frontendHandlerModule.createFrontendConfigsAtPath(context, getResourceOutputs(meta), getResourceOutputs(cloudMeta), exportPath);
+    spinner.succeed('Successfully generated frontend config files');
+  } catch (ex: any) {
+    spinner.fail('Failed to generate frontend config files ' + ex.message);
+    throw ex;
+  } finally {
+    spinner.stop();
+  }
+}
diff --git a/packages/amplify-e2e-core/src/categories/auth.ts b/packages/amplify-e2e-core/src/categories/auth.ts
@@ -1038,7 +1038,7 @@ export function addAuthWithMaxOptions(cwd: string, settings: any): Promise<void>
   } = getSocialProviders(true);
 
   return new Promise((resolve, reject) => {
-    spawn(getCLIPath(), ['add', 'auth'], { cwd, stripColors: true })
+    const chain = spawn(getCLIPath(), ['add', 'auth'], { cwd, stripColors: true })
       .wait('Do you want to use the default authentication and security configuration?')
       .send(KEY_DOWN_ARROW)
       .send(KEY_DOWN_ARROW)
@@ -1061,7 +1061,14 @@ export function addAuthWithMaxOptions(cwd: string, settings: any): Promise<void>
       .sendCarriageReturn()
       .wait('Enter your Google Web Client ID for your identity pool:')
       .send('googleIDPOOL')
-      .sendCarriageReturn()
+      .sendCarriageReturn();
+    if (settings.frontend === 'ios') {
+      chain.wait('Enter your Google iOS Client ID for your identity pool').send('googleiosclientId').sendCarriageReturn();
+    }
+    if (settings.frontend === 'android') {
+      chain.wait('Enter your Google Android Client ID for your identity pool').send('googleandroidclientid').sendCarriageReturn();
+    }
+    chain
       .wait('Enter your Amazon App ID for your identity pool')
       .send('amazonIDPOOL')
       .sendCarriageReturn()
@@ -1137,9 +1144,11 @@ export function addAuthWithMaxOptions(cwd: string, settings: any): Promise<void>
       .wait('Enter your redirect signout URI')
       .sendLine('https://signout1/')
       .wait('Do you want to add another redirect signout URI')
-      .sendConfirmNo()
-      .wait('Select the OAuth flows enabled for this project')
-      .sendCarriageReturn()
+      .sendConfirmNo();
+    if (settings.frontend !== 'ios' && settings.frontend !== 'android' && settings.frontend !== 'flutter') {
+      chain.wait('Select the OAuth flows enabled for this project').sendCarriageReturn();
+    }
+    chain
       .wait('Select the OAuth scopes enabled for this project')
       .sendCarriageReturn()
       .wait('Select the social providers you want to configure for your user pool')

diff --git a/packages/amplify-e2e-core/src/categories/storage.ts b/packages/amplify-e2e-core/src/categories/storage.ts
@@ -537,6 +537,43 @@ export function updateS3AddTrigger(cwd: string, settings: any): Promise<void> {
   });
 }
 
+export function addS3StorageWithIdpAuth(projectDir: string): Promise<void> {
+  return new Promise((resolve, reject) => {
+    let chain = spawn(getCLIPath(), ['add', 'storage'], { cwd: projectDir, stripColors: true });
+
+    singleSelect(chain.wait('Please select from one of the below mentioned services:'), 'Content (Images, audio, video, etc.)', [
+      'Content (Images, audio, video, etc.)',
+      'NoSQL Database',
+    ]);
+
+    chain
+      .wait('Provide a friendly name for your resource that will be used to label this category in the project:')
+      .sendCarriageReturn()
+      .wait('Provide bucket name:')
+      .sendCarriageReturn()
+      .wait('Restrict access by')
+      .sendCarriageReturn()
+      .wait('Who should have access:')
+      .sendCarriageReturn();
+
+    multiSelect(
+      chain.wait('What kind of access do you want for Authenticated users?'),
+      ['create/update', 'read', 'delete'],
+      ['create/update', 'read', 'delete'],
+    );
+
+    chain.wait('Do you want to add a Lambda Trigger for your S3 Bucket?').sendConfirmNo();
+
+    chain.run((err: Error) => {
+      if (!err) {
+        resolve();
+      } else {
+        reject(err);
+      }
+    });
+  });
+}
+
 export function addS3Storage(projectDir: string): Promise<void> {
   return new Promise((resolve, reject) => {
     let chain = spawn(getCLIPath(), ['add', 'storage'], { cwd: projectDir, stripColors: true });
@@ -595,7 +632,6 @@ export function overrideS3(cwd: string, settings: {}) {
   });
 }
 
-
 export function addS3StorageWithSettings(projectDir: string, settings: AddStorageSettings): Promise<void> {
   return new Promise((resolve, reject) => {
     let chain = spawn(getCLIPath(), ['add', 'storage'], { cwd: projectDir, stripColors: true });

diff --git a/packages/amplify-e2e-core/src/export/index.ts b/packages/amplify-e2e-core/src/export/index.ts
@@ -0,0 +1,36 @@
+import { nspawn as spawn, getCLIPath } from '..';
+
+export function exportBackend(cwd: string, settings: { exportPath: string }): Promise<void> {
+  return new Promise((resolve, reject) => {
+    spawn(getCLIPath(), ['export', '--out', settings.exportPath], { cwd, stripColors: true })
+      .wait('For more information: docs.amplify.aws/cli/export')
+      .sendEof()
+      .run((err: Error) => {
+        if (!err) {
+          resolve();
+        } else {
+          reject(err);
+        }
+      });
+  });
+}
+
+export function exportPullBackend(cwd: string, settings: { exportPath: string; frontend: string; rootStackName: string }): Promise<void> {
+  return new Promise((resolve, reject) => {
+    spawn(
+      getCLIPath(),
+      ['export', 'pull', '--out', settings.exportPath, '--frontend', settings.frontend, '--rootStackName', settings.rootStackName],
+      { cwd, stripColors: true },
+    )
+      .wait('Successfully generated frontend config files')
+      .sendEof()
+      .run((err: Error) => {
+        if (!err) {
+          resolve();
+        } else {
+          reject(err);
+        }
+      });
+  });
+}
+