Merge pull request #340 from 3scale/backend-from-oas

backend from OAS

README.md

@@ -2,6 +2,7 @@
 [![CircleCI](https://circleci.com/gh/3scale/3scale_toolbox.svg?style=svg)](https://circleci.com/gh/3scale/3scale_toolbox)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0)
 [![GitHub release](https://img.shields.io/github/v/release/3scale/3scale_toolbox.svg)](https://github.com/3scale/3scale_toolbox/releases/latest)
+[![codecov](https://codecov.io/gh/3scale/3scale_toolbox/branch/main/graph/badge.svg?token=ojinl2NVv5)](https://codecov.io/gh/3scale/3scale_toolbox)
 
 ## Description
 3scale toolbox is a set of tools to help you manage your 3scale product. Using the [3scale API Ruby Client](https://github.com/3scale/3scale-api-ruby).

docs/openapi.md

@@ -1,6 +1,6 @@
 ## Import API definition to 3scale from OpenAPI definition
 
-To create a new service or to update an existing service, you can import the OpenAPI definition from a local file or a URL.
+To create a new 3scale product or 3scale backend, you can import the OpenAPI definition from a local file or a URL.
 
 The `import openapi` command has the following format:
 
@@ -17,6 +17,7 @@ The `import openapi` command has the following format:
       * [URL](#url)
       * [Standard input stream stdin](#standard-input-stream-stdin)
    * [Supported OpenAPI spec version and limitations](#supported-openapi-spec-version-and-limitations)
+   * [Importing 3scale Backend from OpenAPI](#importing-3scale-backend)
    * [OpenAPI import rules](#openapi-import-rules)
       * [Idempotent](#idempotent)
       * [Product name](#product-name)
@@ -73,11 +74,46 @@ $ tool_to_read_openapi_from_source | 3scale import openapi -d <destination> -
   * Supported security schemes: apiKey, oauth2 (any flow type).
   * Multiple flows in security scheme object not supported.
 
+### Importing 3scale Backend
+
+The OpenAPI import command can be used to target a 3scale backend.
+The command line option `--backend` enables this feature.
+The OAS itself won't be stored in 3scale but a 3scale backend, private base URL,
+mapping rules and methods will be created.
+
+Some existing command options don't make sense when creating a backend.
+Valid options are listed here:
+
+```shell
+$ 3scale import openapi -d <remote> --backend <OAS>
+OPTIONS
+       --backend                                  Create backend API from OAS
+    -d --destination=<value>                      3scale target instance.
+                                                  Format:
+                                                  "http[s]://<authentication>@3scale_domain"
+    -o --output=<value>                           Output format. One of:
+                                                  json|yaml
+       --override-private-base-url=<value>        Custom private base URL
+                                                  the private URLs
+       --prefix-matching                          Use prefix matching instead
+                                                  of strict matching on
+                                                  mapping rules derived from
+                                                  openapi operations
+       --skip-openapi-validation                  Skip OpenAPI schema
+                                                  validation
+    -t --target_system_name=<value>               Target system name
+```
+
+The backend's private endpoint is read from the OpenAPI `servers[0].url` field.
+You can override this using this `--override-private-base-url=<value>` command option.
+When the OpenAPI doc does not contain `servers[0].url` and private base url is not provided,
+the command will fail.
+
 ### OpenAPI import rules
 
 #### Idempotent
 
-The command was designed to be idempotent. It can be executed multiple times without changing the result. If the command fails for some unexpected temporary issue, like a network outage, it is safe to re-run as many times as necessary. It is designed to be run from CI/CD system expecting to be run multiple times with the same parameters. 
+The command was designed to be idempotent. It can be executed multiple times without changing the result. If the command fails for some unexpected temporary issue, like a network outage, it is safe to re-run as many times as necessary. It is designed to be run from CI/CD system expecting to be run multiple times with the same parameters.
 
 #### Product name
 
@@ -265,6 +301,7 @@ DESCRIPTION
 OPTIONS
        --activedocs-hidden                        Create ActiveDocs in hidden
                                                   state
+       --backend                                  Create backend API from OAS
        --backend-api-host-header=<value>          Custom host header sent by
                                                   the API gateway to the
                                                   backend API
@@ -276,6 +313,8 @@ OPTIONS
                                                   "http[s]://<authentication>@3scale_domain"
        --default-credentials-userkey=<value>      Default credentials policy
                                                   userkey
+    -o --output=<value>                           Output format. One of:
+                                                  json|yaml
        --oidc-issuer-endpoint=<value>             OIDC Issuer Endpoint
        --oidc-issuer-type=<value>                 OIDC Issuer Type (rest,
                                                   keycloak)

lib/3scale_toolbox/cli.rb

@@ -1,4 +1,5 @@
 require '3scale_toolbox/cli/error_handler'
+require '3scale_toolbox/cli/null_printer'
 require '3scale_toolbox/cli/json_printer'
 require '3scale_toolbox/cli/yaml_printer'
 require '3scale_toolbox/cli/custom_table_printer'

lib/3scale_toolbox/cli/null_printer.rb

@@ -0,0 +1,11 @@
+module ThreeScaleToolbox
+  module CLI
+    class NullPrinter
+      def print_record(record)
+      end
+
+      def print_collection(collection)
+      end
+    end
+  end
+end

lib/3scale_toolbox/commands/import_command/openapi.rb

@@ -3,12 +3,17 @@
 require '3scale_toolbox/commands/import_command/openapi/operation'
 require '3scale_toolbox/commands/import_command/openapi/step'
 require '3scale_toolbox/commands/import_command/openapi/create_method_step'
+require '3scale_toolbox/commands/import_command/openapi/create_backend_method_step'
 require '3scale_toolbox/commands/import_command/openapi/create_mapping_rule_step'
+require '3scale_toolbox/commands/import_command/openapi/create_backend_mapping_rule_step'
+require '3scale_toolbox/commands/import_command/openapi/create_backend_step'
 require '3scale_toolbox/commands/import_command/openapi/create_service_step'
 require '3scale_toolbox/commands/import_command/openapi/create_activedocs_step'
 require '3scale_toolbox/commands/import_command/openapi/update_service_proxy_step'
 require '3scale_toolbox/commands/import_command/openapi/update_service_oidc_conf_step'
 require '3scale_toolbox/commands/import_command/openapi/update_policies_step'
+require '3scale_toolbox/commands/import_command/openapi/import_product_step'
+require '3scale_toolbox/commands/import_command/openapi/import_backend_step'
 require '3scale_toolbox/commands/import_command/issuer_type_transformer'
 
 module ThreeScaleToolbox
@@ -31,7 +36,8 @@ def self.command
               flag    nil, 'activedocs-hidden', 'Create ActiveDocs in hidden state'
               flag    nil, 'skip-openapi-validation', 'Skip OpenAPI schema validation'
               flag    nil, 'prefix-matching', 'Use prefix matching instead of strict matching on mapping rules derived from openapi operations'
-              option  nil, 'oidc-issuer-type', 'OIDC Issuer Type (rest, keycloak)', argument: :required, transform: IssuerTypeTransformer.new            
+              flag    nil, 'backend', 'Create backend API from OAS'
+              option  nil, 'oidc-issuer-type', 'OIDC Issuer Type (rest, keycloak)', argument: :required, transform: IssuerTypeTransformer.new
               option  nil, 'oidc-issuer-endpoint', 'OIDC Issuer Endpoint', argument: :required
               option  nil, 'default-credentials-userkey', 'Default credentials policy userkey', argument: :required
               option  nil, 'override-private-basepath', 'Override the basepath for the private URLs', argument: :required
@@ -41,29 +47,21 @@ def self.command
               option  nil, 'override-private-base-url', 'Custom private base URL', argument: :required
               option nil, 'backend-api-secret-token', 'Custom secret token sent by the API gateway to the backend API',argument: :required
               option nil, 'backend-api-host-header', 'Custom host header sent by the API gateway to the backend API', argument: :required
+              ThreeScaleToolbox::CLI.output_flag(self)
               param   :openapi_resource
 
               runner OpenAPISubcommand
             end
           end
 
           def run
-            tasks = []
-            tasks << CreateServiceStep.new(context)
-            # other tasks might read proxy settings (CreateActiveDocsStep does)
-            tasks << UpdateServiceProxyStep.new(context)
-            tasks << CreateMethodsStep.new(context)
-            tasks << ThreeScaleToolbox::Commands::ServiceCommand::CopyCommand::DestroyMappingRulesTask.new(context)
-            tasks << CreateMappingRulesStep.new(context)
-            tasks << CreateActiveDocsStep.new(context)
-            tasks << UpdateServiceOidcConfStep.new(context)
-            tasks << UpdatePoliciesStep.new(context)
-
-            # run tasks
-            tasks.each(&:call)
-
-            # This should be the last step
-            ThreeScaleToolbox::Commands::ServiceCommand::CopyCommand::BumpProxyVersionTask.new(service: context[:target]).call
+            if backend?
+              ImportBackendStep.new(context).call
+            else
+              ImportProductStep.new(context).call
+            end
+
+            printer.print_record context.fetch(:report)
           end
 
           private
@@ -92,6 +90,7 @@ def create_context
               backend_api_host_header: options[:'backend-api-host-header'],
               prefix_matching: options[:'prefix-matching'],
               delete_mapping_rules: true,
+              logger: logger,
             }
           end
 
@@ -103,6 +102,10 @@ def openapi_path
             arguments[:openapi_resource]
           end
 
+          def backend?
+            options[:backend]
+          end
+
           def validate
             !options[:'skip-openapi-validation']
           end
@@ -118,6 +121,29 @@ def openapi_parser
           rescue JSON::Schema::ValidationError => e
             raise ThreeScaleToolbox::Error, "OpenAPI schema validation failed: #{e.message}"
           end
+
+          def printer
+            # if product import AND output not specified -> null printer
+            # if product import AND output specified -> specified printer
+            # if backend import AND output not specified -> json printer
+            # if backend import AND output specified -> specified printer
+            default_printer = if backend?
+                                CLI::JsonPrinter.new
+                              else
+                                CLI::NullPrinter.new
+                              end
+            options.fetch(:output, default_printer)
+          end
+
+          def logger
+            if options[:output].nil?
+              Logger.new($stdout).tap do |logger|
+                logger.formatter = proc { |severity, datetime, progname, msg| "#{msg}\n" }
+              end
+            else
+              Logger.new(File::NULL)
+            end
+          end
         end
       end
     end

lib/3scale_toolbox/commands/import_command/openapi/create_activedocs_step.rb

@@ -23,7 +23,7 @@ def call
                 unless ThreeScaleToolbox::Helper.system_name_already_taken_error? errors
 
               # if activedocs system_name exists, ignore error, update activedocs
-              puts 'Activedocs exists, update!'
+              logger.info 'Activedocs exists, update!'
               update_res = threescale_client.update_activedocs(find_activedocs_id, active_doc)
               raise ThreeScaleToolbox::Error, "ActiveDocs has not been updated. #{update_res['errors']}" unless update_res['errors'].nil?
             end

lib/3scale_toolbox/commands/import_command/openapi/create_backend_mapping_rule_step.rb

@@ -0,0 +1,21 @@
+module ThreeScaleToolbox
+  module Commands
+    module ImportCommand
+      module OpenAPI
+        class CreateBackendMappingRulesStep
+          include Step
+
+          def call
+            backend.mapping_rules.each(&:delete)
+
+            report['mapping_rules'] = {}
+            operations.each do |op|
+              b_m_r = Entities::BackendMappingRule.create(backend: backend, attrs: op.mapping_rule)
+              report['mapping_rules'][op.friendly_name] = op.mapping_rule
+            end
+          end
+        end
+      end
+    end
+  end
+end

lib/3scale_toolbox/commands/import_command/openapi/create_backend_method_step.rb

@@ -0,0 +1,41 @@
+module ThreeScaleToolbox
+  module Commands
+    module ImportCommand
+      module OpenAPI
+        class CreateBackendMethodsStep
+          include Step
+
+          def call
+            missing_operations.each do |op|
+              method = Entities::BackendMethod.create(backend: backend, attrs: op.method)
+              op.set(:metric_id, method.id)
+            end
+
+            existing_operations.each do |op|
+              method_attrs = methods_index.fetch(op.method['system_name']).attrs
+              method = Entities::BackendMethod.new(id: method_attrs.fetch('id'), backend: backend)
+              method.update(op.method)
+              op.set(:metric_id, method.id)
+            end
+          end
+
+          private
+
+          def methods_index
+            @methods_index ||= backend.methods.each_with_object({}) do |method, acc|
+              acc[method.system_name] = method
+            end
+          end
+
+          def missing_operations
+            operations.reject { |op| methods_index.key? op.method['system_name'] }
+          end
+
+          def existing_operations
+            operations.select { |op| methods_index.key? op.method['system_name'] }
+          end
+        end
+      end
+    end
+  end
+end

lib/3scale_toolbox/commands/import_command/openapi/create_backend_step.rb

@@ -0,0 +1,62 @@
+module ThreeScaleToolbox
+  module Commands
+    module ImportCommand
+      module OpenAPI
+        class CreateBackendStep
+          include Step
+
+          ##
+          # Creates backend with a given system_name
+          # If the backend already exists, update basic settings like name and description
+          def call
+            # Update backend and update context
+            self.backend = Entities::Backend.find_by_system_name(remote: threescale_client,
+                                                                 system_name: system_name)
+            if backend.nil?
+              # Create service and update context
+              self.backend = Entities::Backend.create(remote: threescale_client,
+                                                      attrs: create_attrs)
+            else
+              backend.update(update_attrs)
+            end
+
+            report['id'] = backend.id
+            report['system_name'] = backend.system_name
+            report['private_endpoint'] = backend.private_endpoint
+          end
+
+          private
+
+          def create_attrs
+            {
+              'name' => title,
+              'system_name' => system_name,
+              'description' => description,
+              'private_endpoint' => private_endpoint
+            }
+          end
+
+          def update_attrs
+            {
+              'name' => title,
+              'description' => description,
+              'private_endpoint' => private_endpoint
+            }
+          end
+
+          def system_name
+            target_system_name || title.downcase.gsub(/[^\w]/, '_')
+          end
+
+          def title
+            api_spec.title
+          end
+
+          def description
+            api_spec.description
+          end
+        end
+      end
+    end
+  end
+end

lib/3scale_toolbox/commands/import_command/openapi/create_mapping_rule_step.rb

@@ -6,10 +6,12 @@ class CreateMappingRulesStep
           include Step
 
           def call
+            report['mapping_rules'] = {}
             operations.each do |op|
               Entities::MappingRule.create(service: service,
                                            attrs: op.mapping_rule)
-              puts "Created #{op.http_method} #{op.pattern} endpoint"
+              logger.info "Created #{op.http_method} #{op.pattern} endpoint"
+              report['mapping_rules'][op.friendly_name] = op.mapping_rule
             end
           end
         end

lib/3scale_toolbox/commands/import_command/openapi/create_service_step.rb

@@ -16,11 +16,16 @@ def call
               # Create service and update context
               self.service = Entities::Service.create(remote: threescale_client,
                                                       service_params: service_settings)
-              puts "Created service id: #{service.id}, name: #{service_name}"
+              logger.info "Created service id: #{service.id}, name: #{service_name}"
             else
               service.update(service_settings)
-              puts "Updated service id: #{service.id}, name: #{service_name}"
+              logger.info "Updated service id: #{service.id}, name: #{service_name}"
             end
+
+            report['id'] = service.id
+            report['system_name'] = service.system_name
+            report['name'] = service.name
+            report['backend_version'] = api_spec.service_backend_version
           end
 
           private