Prepare Design lifecycle features for develop

README.md

@@ -15,6 +15,8 @@
 
 Design Builder is a Nautobot application for easily populating data within Nautobot using standardized design files. These design files are just Jinja templates that describe the Nautobot objects to be created or updated.
 
+It also introduces the concept of a design-oriented Source of Truth with a complete lifecycle management of the design deployments (i.e., an instantiation of a design with concrete input data). With this approach, the users of the application can not only create (or populate) data within Nautobot but also update or decommission it while enforcing data protection and dependency.
+
 ## Documentation
 
 Full documentation for this App can be found over on the [Nautobot Docs](https://docs.nautobot.com) website:

development/nautobot_config.py

@@ -25,7 +25,8 @@
     if "debug_toolbar.middleware.DebugToolbarMiddleware" not in MIDDLEWARE:  # noqa: F405
         MIDDLEWARE.insert(0, "debug_toolbar.middleware.DebugToolbarMiddleware")  # noqa: F405
 
-MIDDLEWARE.insert(0, "nautobot_design_builder.middleware.GlobalRequestMiddleware")  # noqa: F405
+if "nautobot_design_builder.middleware.GlobalRequestMiddleware" not in MIDDLEWARE:  # noqa: F405
+    MIDDLEWARE.insert(0, "nautobot_design_builder.middleware.GlobalRequestMiddleware")  # noqa: F405
 
 #
 # Misc. settings

docs/user/app_getting_started.md

@@ -12,6 +12,8 @@ The easiest way to experience Design Builder is to either add the [demo-designs]
 
 ## What are the next steps?
 
+<!-- TODO: update with the new Design Navigation and new screenshoots -->
+
 The Design Builder demo designs ship with some sample designs to demonstrate capabilities. Once the application stack is ready, you should have several jobs listed under the "Jobs" -> "Jobs" menu item.
 
 ![Jobs list](../images/screenshots/sample-design-jobs-list.png)

docs/user/app_overview.md

@@ -7,12 +7,15 @@ This document provides an overview of the App including critical information and
 
 ## Description
 
-Design Builder provides a system where standardized network designs can be developed to produce collections of objects within Nautobot. These designs are text based templates that can create and update hierarchical data structures within Nautobot.
+Design Builder provides a system where standardized network designs can be developed to produce or update collections of objects within Nautobot. These designs are text based templates that can create and update hierarchical data structures within Nautobot.
+
+The deployment of a design comes with a complete lifecycle management of all the changes connected as a single entity. Thus, the design deployment can be updated or decommissioned after its creation, and the all the changes introduced can be honored when accessing the data outside of the design builder app.
 
 ## Audience (User Personas) - Who should use this App?
 
 - Network engineers who want to have reproducible sets of Nautobot objects based on some standard design.
 - Automation engineers who want to be able to automate the creation of Nautobot objects based on a set of standard designs.
+- Users who want to leverage abstracted network services defined by network engineers in a simplfied way.
 
 ## Authors and Maintainers
 

docs/user/design_development.md

@@ -114,6 +114,18 @@ The value of the `context_class` metadata attribute should be any Python class t
 
 This attribute is optional. A report is a Jinja template that is rendered once the design has been implemented. Like `design_file` the design builder will look for this template relative to the filename that defines the design job. This is helpful to generate a custom view of the data that was built during the design build.
 
+### `version`
+
+It's an optional string attribute that is used to define the versioning reference of a design job. This will enable in the future the versioning lifecycle of design deployments. For example, one a design evolves from one version to another, the design deployment will be able to accommodate the new changes.
+
+### `description`
+
+This optional attribute that is a string that provides a high-level overview of the intend of the design job. This description is displayed int the design detail view.
+
+### `docs`
+
+This attribute is also displayed on the design detail view. The `docs` attribute can utilize markdown format and should provide more detailed information than the description. This should help the users of the `Design` to understand the goal of the design and the impact of the input data.
+
 ## Design Context
 
 Primary Purpose:
@@ -339,41 +351,3 @@ class DesignJobWithExtensions(DesignJob):
         extensions = [ext.BGPPeeringExtension]
 ```
 
-## Design LifeCycle
-
-Design implementations can have a full life cycle: creation, update, and decommission.
-
-<!-- TODO: without an identifier: IDENTIFIER_KEYS = ["!create_or_update", "!create", "!update", "!get"],
-the update features are not working as expected.
-I would propose to use explicit action tags even for create: "!create:name"
- -->
-
-Once a design is "deployed" in Nautobot, a Design Instance is created with the report of the changes implemented, and with actions to decommission or update it.
-
-### Design Decommission
-
-This feature allows to rollback all the changes implemented by a design instance to the previous state. This rollback depends on the scope of the change:
-
-- If the object was created by the design implementation, this object will be removed.
-- If only some attributes were changes, the affected attributes will be rolled back to the previous state.
-
-The decommissioning feature takes into account potential dependencies between design implementations. For example, if a new l3vpn design depends on devices that were created by another design, this previous design won't be decommissioned until the l3vpn dependencies are also decommissioned to warrant consistency.
-
-Once a design instance is decommissioned, it's still visible in the API/UI to check the history of changes but without any active relationship with Nautobot objects. After decommissioning, the design instance can be deleted completely from Nautobot.
-
-### Design Updates
-
-This feature allows to re run a design instance with different input data to update the implemented design with the new changes: additions and removals.
-
-It leverages a complete tracking of previous design implementation and a reduce function for the new design to understand the changes to be implemented and the objects to be decommissioned (leveraging the previous decommissioning feature for only a specific object).
-
-The update feature comes with a few assumptions:
-
-- All the design objects that have an identifier have to use identifier keys to identify the object to make them comparable across designs.
-- Object identifiers should keep consistent in multiple design runs. For example, you can't target a device with the device name and update the name on the same design.
-- When design provides a list of objects, the objects are assumed to be in the same order. For example, if the first design creates `[deviceA1, deviceB1]`, if expanded, it should be `[deviceA1, deviceB1, deviceA2, deviceB2]`, not `[deviceA1, deviceA2, deviceB1, deviceB2]`.
-
-<!--
-TODO:
-- We could check design for update capabilities? to disable when not possible
--->

docs/user/design_lifecycle.md

@@ -0,0 +1,67 @@
+# Design LifeCycle
+
+<!-- TODO: Add the screenshoots -->
+
+According to a design-oriented approach, the Design Builder App provides not only the capacity to create and update data in Nautobot but also a complete lifecycle management of each deployment: update, versioning (in the future), and decommissioning.
+
+<!-- TODO: without an identifier: IDENTIFIER_KEYS = ["!create_or_update", "!create", "!update", "!get"],
+the update features are not working as expected.
+I would propose to use explicit action tags even for create: "!create:name"
+ -->
+
+All the Design Builder UI navigation menus are under the Design Builder tab.
+
+## `Design`
+
+A `Design` is a one to one mapping with a Nautobot `Job`, enriched with some data from the Design Builder `DesignJob` definition. In concrete, it stores:
+
+- A `Job` reference.
+- A `version` string from the `DesignJob`.
+- A `description` string from the `DesignJob`.
+- A `docs` string from the `DesignJob`.
+
+<!-- TODO: Add the screenshoot of the table of Design -->
+
+From the `Design`, the user can manage the associated `Job`, and trigger its execution, which creates a `DesignInstance` or Design Deployment
+
+## Design Deployment or `DesignInstance`
+
+Once a design is "deployed" in Nautobot, a Design Deployment (or `DesignInstance`) is created with the report of the changes implemented (i.e. `Journals`), and with actions to update or decommission it (see next subsections).
+
+The `DesignInstance` stores:
+
+- The `name` of the deployment, within the context of the `Design`.
+- The `Design` reference.
+- The `version` from the `Design` when it was deployed or updated.
+- When it was initially deployed or last updated.
+- The `status` of the design, and the `live_state` or operational status to signal its state in the actual network.
+
+<!-- TODO: Add the screenshoot of the table of DesignInstance -->
+
+### Design Deployment Update
+
+This feature provides a means to re-run a design instance with different input data. Re-running the job will update the implemented design with the new changes: additions and removals.
+
+It leverages a complete tracking of previous design implementations and a function to combine the new design and previous design, to understand the changes to be implemented and the objects to be decommissioned (leveraging the previous decommissioning feature for only a specific object).
+
+The update feature comes with a few assumptions:
+
+- All the design objects that have an identifier have to use identifier keys to identify the object to make them comparable across designs.
+- Object identifiers should keep consistent in multiple design runs. For example, you can't target a device with the device name and update the name on the same design.
+- When design provides a list of objects, the objects are assumed to be in the same order. For example, if the first design creates `[deviceA1, deviceB1]`, if expanded, it should be `[deviceA1, deviceB1, deviceA2, deviceB2]`, not `[deviceA1, deviceA2, deviceB1, deviceB2]`.
+
+<!--
+TODO:
+- We could check design for update capabilities? to disable when not possible
+-->
+
+### Design Deployment Decommission
+
+This feature allows to rollback all the changes implemented by a design instance to the previous state. This rollback depends on the scope of the change:
+
+- If the object was created by the design implementation, this object will be removed.
+- If only some attributes were changes, the affected attributes will be rolled back to the previous state.
+
+The decommissioning feature takes into account potential dependencies between design implementations. For example, if a new l3vpn design depends on devices that were created by another design, this previous design won't be decommissioned until the l3vpn dependencies are also decommissioned to warrant consistency.
+
+Once a design instance is decommissioned, it's still visible in the API/UI to check the history of changes but without any active relationship with Nautobot objects. After decommissioning, the design instance can be deleted completely from Nautobot.

docs/user/design_quickstart.md

@@ -10,6 +10,8 @@ To add a new design you will need (at a minimum) a class extending `nautobot_des
 
 For more information on creating designs see [Getting Started with Designs](design_development.md).
 
+Once the designs are loaded, you can start managing them from the "Design Builder" navigation tab.
+
 ## Sample Data
 
 Much of the time, designs will need some data to exist in Nautobot before they can be built. In a development and testing environment it is necessary to generate this data for testing purposes. The Design Builder application comes with a `load_design` management command that will read a design YAML file (not a template) and will build the design in Nautobot. This can be used to produce sample data for a development environment. Simply create a YAML file that includes all of the object definitions needed for testing and load the file with `invoke build-design <filename>`. This should read the file and build all of the objects within Nautobot.

examples/custom_design/designs/initial_data/jobs.py

@@ -19,3 +19,13 @@ class Meta:
         commit_default = False
         design_file = "designs/0001_design.yaml.j2"
         context_class = InitialDesignContext
+        version = "1.0.0"
+        description = "Establish the devices and site information for four sites: IAD5, LGA1, LAX11, SEA11."
+        docs = """This design creates the following objects in the source of truth to establish the initia network environment in  four sites: IAD5, LGA1, LAX11, SEA11.
+
+These sites belong to the America region (and different subregions), and use Juniper PTX10016 devices.
+
+The user input data is:
+    - Number of devices per site (integer)
+    - The description for one of the regions (string)
+"""

mkdocs.yml

@@ -107,6 +107,7 @@ nav:
       - Getting Started: "user/app_getting_started.md"
       - Design Quick Start: "user/design_quickstart.md"
       - Design Development: "user/design_development.md"
+      - Design LifeCycle: "user/design_lifecycle.md"
       - Frequently Asked Questions: "user/faq.md"
       - Git-based Config Context: "user/git_config_context.md"
   - Administrator Guide:

nautobot_design_builder/api/serializers.py

@@ -40,6 +40,8 @@ class DesignInstanceSerializer(NautobotModelSerializer, TaggedModelSerializerMix
     url = HyperlinkedIdentityField(view_name="plugins-api:nautobot_design_builder-api:design-detail")
     design = NestedDesignSerializer()
     live_state = NestedStatusSerializer()
+    created_by = SerializerMethodField()
+    last_updated_by = SerializerMethodField()
 
     class Meta:
         """Serializer options for the design model."""
@@ -50,13 +52,22 @@ class Meta:
             "url",
             "design",
             "name",
-            "owner",
+            "created_by",
             "first_implemented",
+            "last_updated_by",
             "last_implemented",
             "status",
             "live_state",
         ]
 
+    def get_created_by(self, instance):
+        """Get the username of the user who created the object."""
+        return instance.created_by
+
+    def get_last_updated_by(self, instance):
+        """Get the username of the user who update the object last time."""
+        return instance.last_updated_by
+
 
 class JournalSerializer(NautobotModelSerializer, TaggedModelSerializerMixin):
     """Serializer for the journal model."""

nautobot_design_builder/design.py

@@ -617,7 +617,6 @@ def get_extension(self, ext_type: str, tag: str) -> ext.Extension:
             extn["object"] = extn["class"](self)
         return extn["object"]
 
-    # TODO: this is a breaking change that needs to be revisited because it's used by Django commands directly
     @transaction.atomic
     def implement_design_changes(self, design: Dict, deprecated_design: Dict, design_file: str, commit: bool = False):
         """Iterates through items in the design and creates them.

nautobot_design_builder/design_job.py

@@ -24,7 +24,7 @@
 from nautobot_design_builder.context import Context
 from nautobot_design_builder import models
 from nautobot_design_builder import choices
-from nautobot_design_builder.recursive import reduce_design
+from nautobot_design_builder.recursive import combine_designs
 
 from .util import nautobot_version
 
@@ -38,7 +38,6 @@ class DesignJob(Job, ABC, LoggingMixin):  # pylint: disable=too-many-instance-at
     """
 
     instance_name = StringVar(label="Instance Name", max_length=models.DESIGN_NAME_MAX_LENGTH)
-    owner = StringVar(label="Implementation Owner", required=False, max_length=models.DESIGN_OWNER_MAX_LENGTH)
 
     if nautobot_version >= "2.0.0":
         from nautobot.extras.jobs import DryRunVar  # pylint: disable=no-name-in-module,import-outside-toplevel
@@ -183,14 +182,14 @@ def implement_design(self, context, design_file, commit):
             for key, new_value in design.items():
                 old_value = previous_design[key]
                 future_value = self.builder.builder_output[design_file][key]
-                reduce_design(new_value, old_value, future_value, deprecated_design, key)
+                combine_designs(new_value, old_value, future_value, deprecated_design, key)
 
             self.log_debug(f"Design to implement after reduction: {design}")
             self.log_debug(f"Design to deprecate after reduction: {deprecated_design}")
 
         self.builder.implement_design_changes(design, deprecated_design, design_file, commit)
 
-    def _setup_journal(self, instance_name: str, design_owner: str):
+    def _setup_journal(self, instance_name: str):
         try:
             instance = models.DesignInstance.objects.get(name=instance_name, design=self.design_model())
             self.log_info(message=f'Existing design instance of "{instance_name}" was found, re-running design job.')
@@ -200,13 +199,13 @@ def _setup_journal(self, instance_name: str, design_owner: str):
             content_type = ContentType.objects.get_for_model(models.DesignInstance)
             instance = models.DesignInstance(
                 name=instance_name,
-                owner=design_owner,
                 design=self.design_model(),
                 last_implemented=datetime.now(),
                 status=Status.objects.get(content_types=content_type, name=choices.DesignInstanceStatusChoices.ACTIVE),
                 live_state=Status.objects.get(
                     content_types=content_type, name=choices.DesignInstanceLiveStateChoices.PENDING
                 ),
+                version=self.design_model().version,
             )
         instance.validated_save()
 
@@ -238,7 +237,7 @@ def run(self, **kwargs):  # pylint: disable=arguments-differ,too-many-branches,t
         else:
             self.job_result.job_kwargs = self.serialize_data(data)
 
-        journal = self._setup_journal(data.pop("instance_name"), data.pop("owner"))
+        journal = self._setup_journal(data.pop("instance_name"))
         self.log_info(message=f"Building {getattr(self.Meta, 'name')}")
         extensions = getattr(self.Meta, "extensions", [])
         self.builder = Builder(

nautobot_design_builder/filters.py

@@ -40,7 +40,16 @@ class Meta:
         """Meta attributes for filter."""
 
         model = DesignInstance
-        fields = ["id", "design", "name", "owner", "first_implemented", "last_implemented", "status", "live_state"]
+        fields = [
+            "id",
+            "design",
+            "name",
+            "first_implemented",
+            "last_implemented",
+            "status",
+            "live_state",
+            "version",
+        ]
 
 
 class JournalFilterSet(NautobotFilterSet):
@@ -50,7 +59,7 @@ class JournalFilterSet(NautobotFilterSet):
 
     design_instance = NaturalKeyOrPKMultipleChoiceFilter(
         queryset=DesignInstance.objects.all(),
-        label="Design Instance (ID)",
+        label="Design Deployment (ID)",
     )
 
     job_result = NaturalKeyOrPKMultipleChoiceFilter(

nautobot_design_builder/forms.py

@@ -1,6 +1,6 @@
 """Forms for the design builder app."""
 
-from django.forms import NullBooleanField
+from django.forms import NullBooleanField, CharField
 from nautobot.extras.forms import NautobotFilterForm
 from nautobot.extras.models import Job, JobResult
 from nautobot.utilities.forms import TagFilterField, DynamicModelChoiceField, StaticSelect2, BOOLEAN_WITH_BLANK_CHOICES
@@ -13,8 +13,9 @@ class DesignFilterForm(NautobotFilterForm):
 
     model = Design
 
-    job = DynamicModelChoiceField(queryset=Job.objects.all())
+    job = DynamicModelChoiceField(queryset=Job.objects.all(), required=False)
     tag = TagFilterField(model)
+    version = CharField(max_length=20, required=False)
 
 
 class DesignInstanceFilterForm(NautobotFilterForm):
@@ -24,6 +25,7 @@ class DesignInstanceFilterForm(NautobotFilterForm):
 
     design = DynamicModelChoiceField(queryset=Design.objects.all())
     tag = TagFilterField(model)
+    version = CharField(max_length=20, required=False)
 
 
 class JournalFilterForm(NautobotFilterForm):