Added a separate UCR (edgexfoundry#3) for "Extending" Device Data, si…

…nce it may end up

related.

Removed implementation stuff from edgexfoundry#2, and added requirements,
per review comments.
Added a Use Case extension for "gateway of gateways".

Signed-off-by: Tom Brennan <[email protected]>

docs_src/design/ucr/0002-Device-Parent-Child-Relationships.md

@@ -5,7 +5,7 @@ This UCR describes Use Cases for new Device metadata for Parent to Child Relatio
 - Tom Brennan (Eaton)
 
 ## Change Log
-- [pending](https://github.com/edgexfoundry/edgex-docs/pull/800) (2022-07-18)
+- [pending](https://github.com/edgexfoundry/edgex-docs/pull/800) (2022-07-25)
 
 
 ### Market Segments
@@ -19,8 +19,9 @@ of the devices found in an EdgeX system.
 This information is generally used for either protocol data constructs or for display purposes.
 If not know or provided by the south-bound Device Service, this information might be added 
 to the Device metadata by the north-bound or analytic services, or by the user.
+
 It is desirable that the means of conveying this information become standardized for those systems
-which provide and use it, hence proposing here that there be a common definition and use of this metadata.
+which provide and use it, hence proposing here that there be a common definition and usage of this metadata.
 
 ### Target Users
 - Product Developers
@@ -33,15 +34,15 @@ which provide and use it, hence proposing here that there be a common definition
 
 ### Description
 Some north-bound protocols and some UI designs present the system devices in a hierarchial manner, 
-where it is necessary to know which devices are the parent(s) and which are their children.
+where it is necessary to know which devices are parents and which are their children.
 
 These considerations are most important for gateways that are implemented with the EdgeX framework,
 since there are potentially very many south-bound devices connected to a system.
 
 Examples are
 * BACnet - where only one "main" device is present at the point of external connection (eg, UDP port 0xBAC0) and all other devices must be presented as "virtually routed devices" connected to that main "virtual router" device.
 * Azure IoT Hub - where the normal connection for IoT Plug and Play / Digital Twin is for a single device, and any other devices need to somehow fall under that device (eg, with Device Twin "Modules")
-* UI device presentation - where devices are grouped under their parent, often rolled up until they are expanded to show their data
+* UI device presentation - where child devices are grouped under their parent, often rolled up until they are expanded to show their data
 * Multi-tenant deployments of multi-point energy meters - where a main meter has up to 80 Branch Circuit Monitoring (BCM) points connected to it, each BCM modeled as a Device consisting of the same 6 or so energy channels (Device Resources), and each BCM is assigned to a particular tenant. Tenants will be given access to the data from their BCM point(s) but not those of other tenants. A gateway may connect more than one of these multi-point energy meters.
 
 Since there are multiple similar uses for this relationship information on the north side, it is proposed to locate
@@ -50,47 +51,52 @@ locate it in each north-bound service (which would be particularly problematic f
 
 The sound-bound Device Service that creates a Device is ideally the service which establishes this relationship data, though it is possible that it is unaware of the parent-child relationship. It should be permitted, therefore, for this relationship information to also be set by north-bound services (most likely the UI) and simply ignored by the south-bound Device Service.
 
-A potential solution might add a field to the device structure such as
-```json
-"relationships": [
-        "parent": "First-Floor-Gateway"
-      ]
-```
-It is probably also necessary to indicate which device is the "main" or "publisher" device (ie, the gateway device), 
-as any devices without a configured relationship will probably be inferred to be children of that device.
+It is also necessary to indicate which device is the "main" or "publisher" device (ie, the gateway device), 
+as any devices without a configured relationship can be inferred to be children of that device.
+
+It is frequently a pattern in data servers to "walk the device tree", starting with the main device, then 
+recursively processing its direct child devices, and then the child devices (if any) of those devices, until
+all devices have been processed. This is normally part of the initialization of device data for a server,
+since the parent must be processed and initialized before its child devices. Consequently, there is a need
+for a means to answer the question "What are the child devices (if any) of device x.y.z?"; this is commonly done
+either with the device structure listing its children, or by providing a query that can answer this question.
 
 ### Extensions to the main Use Case
-1. In addition to parent-child relationships, other relationships might be indicated in this metadata, such as an
-"extends" when an analytic service extends an existing south-bound device, adding new Device Resources beyond what the south-bound service provides.
-2. Some services add "devices" which have no physical counterpart, eg, for an NTP client service where the "device" 
+1. Some services add "devices" which have no physical counterpart, eg, for an NTP client service where the "device" 
 serves simply as a container for the Resources necessary to configure and report the status of the service.
 In these cases, it would be helpful for the other services if it described itself as something like a "system" device,
 meaning one that doesn't have a physical (south-bound or hardware) counterpart.
-3. It could be possible for north-bound or analytic services to add metadata of their choice, related to an individual device, in this "relationships" area. 
-This definitely muddies the clear objective of this UCR, but consideration of it might steer the ultimate solution. 
-As application service developers, we have felt at times the desire to "annotate" a device with something more formal than just another label; 
-if other reviewers agree with this notion, it can be adopted here, since it follows the same lines 
-(metadata that other services might add to a device).
-
+2. Multi-level systems, such as a "gateway of gateways", which aggregate a large number of devices. The parent
+property will be useful for assisting to arrange the devices in a hierarchy that corresponds to their physical hierarchy. 
+In this situation, the parent property of a lower-level gateway will indicate the upper gateway it is connected to.
 
 ### Existing solutions
 The Device structure in Eaton's legacy products indicated this parent-child relationship bidirectionally: each device indicated its parent device (if any) with one field, and its child devices (if any) with a list of IDs.
 
 The Device structure in Eaton's cloud solution is a "DeviceTree", which is a recursive, hierarchial structure of the connected devices, starting with the "publisher" device and its first-level child devices.
 
-There is the BACnet "virtual routed devices" model, but I would not recommend it, as it is too artificial for a simple relationship.
+There is the BACnet "virtual routed devices" model, but I would not recommend it, as it is too artificial for this  simple relationship.
 
 The existing EdgeX UIs group devices by their Device Service, which is a good approach for simple devices without children of their own, but fails if those devices have child devices too.
 
 ### Requirements
-No additional requirements.
+1. Each device should have a property to indicate which other device is its parent device.
+2. There should be one "main" or "publisher" device, and it should indicate this via a property instead of
+indicating a parent device (since it has no parent).
+3. If a device does not indicate which is its parent device, then it shall be inferred that its parent is the main device. (This helps with backwards compatability.)
+4. Though it is preferred that the owning service set a device's parent property, this property can also be set
+by other services via the core-metadata PATCH devices API. The owning service can ignore this update if it does
+not use the parent property.
+5. Some means provided to answer the question, "What are the child devices (if any) of device x.y.z?".
+6. Some standard means to indicate which devices are "system" devices, ie, those without a physical counterpart.
 
 Not a requirement: inheritance of device status via the parent-child relationship. Apparently this was a point
 over which past consideration of parent-child relationships in EdgeX foundered, but it seems complicated
 for independent services, and can generally be inferred by other services anyway.
 
 ### Other Related Issues
-None known.
+Use Case for Application Services Extending Device Data [0003 Extending Device Data](./0003-Extending-Device-Data.md) may be related, 
+as, depending on its solution, it may have to indicate a different Device Relationship ("Extends").
 
 ### References
 - [Azure IoT Edge Gateways and Child Devices](https://docs.microsoft.com/en-us/azure/iot-edge/how-to-connect-downstream-iot-edge-device?view=iotedge-2020-11&tabs=azure-portal)

docs_src/design/ucr/0003-Extending-Device-Data.md

@@ -0,0 +1,106 @@
+## Extending Device Data
+This UCR describes the Use Case for Application Services Extending Device Data for a given south-bound Device.
+
+### Submitters
+- Tom Brennan (Eaton)
+
+## Change Log
+- [pending](https://github.com/edgexfoundry/edgex-docs/pull/800) (2022-07-25)
+
+
+### Market Segments
+Any that deploy EdgeX systems with analytic, utility, or north-bound microservices that add new Device Resources
+that are extensions of the original south-bound or service-based Device data.
+
+### Motivation
+We find a consistent need as we design microservices for our industrial products:
+The new analytic, utility, and north-bound microservices almost always need to add Device Resources to manage their configuration, transforms, and status reporting. These Resources are usually needed on a per-Device basis (rather than just overall service configuration or status), which can be seen as extending (adding to) the data of the original south-bound devices.
+
+Adding configuration and status via Devices that extend the original south-bound Device, and new Device Resources,
+make this configuration and status data easily accessible and translatable to other Application Services and to the UI via REST;
+we think that this general solution is better than disparate solutions which add custom APIs in each Application Service to Get and Set this data.
+
+What is needed is a common means of showing the relationship between these added Resources and their
+original south-bound Device; that is, to indicate that these Resources "extend" the original Device.
+
+It is desirable that the means of conveying this information become standardized for those EdgeX microservices
+which provide and use it, hence proposing here that there be a common EdgeX way defined to do this.
+
+### Target Users
+- Product Developers
+- Device Owner
+- Device User
+- Device Maintainer
+- Cloud User
+- Service Provider
+- Software Integrator
+
+### Description
+Picture the extremely simple case of a south-bound sensor device that just measures Temperature and Humidity and provides these as Device Resources. If we then add analytic and north-bound microservices:
+- A Trending service that has Device Resources to indicate that Temperature and Humidity are trended for, eg, Minimum, Average, and Maximum over a 1 hour trend interval.
+- An Alarming service that has Device Resources to describe the Alarm Rules used to monitor Temperature and Humidity, plus a device-level InAlarm status.
+- A Cloud service that reports not just the Temperature and Humidity but also their Trend configuration and Alarm Rule Resources. 
+In addition, the Cloud service adds its own Resources to direct the Cadence with which this Device's data is reported.
+
+Now scale this up to 100 such Temperature/Humidity sensors, and it grows difficult to match all of the added
+Resources to their original sensor data. And add the requirement that all these resources must be able to be seen 
+and managed locally via REST or Message Bus, and potentially from north-bound services like Modbus/TCP, and from 
+the Cloud (because everybody wants to control everything from the Cloud). 
+
+Furthermore, from the end user's point of view, the Trend configuration, Alarm Rules, and Cloud Cadence that are added for a given Device are all seen as aspects of the Temperature/Humidity Device, as is common 
+for Digital Twin representations, and not as separated, free-standing entities. 
+So there must be some means to relate the extended Device Resources to the original south-bound Device and its
+Device Resources.
+
+
+### Existing solutions
+In Eaton's legacy products, these configuration and status extensions could be added to the channels of the Device in the central data store, exist as channel metadata, or be stored in separate data objects. This was fine and 
+efficient for a self-contained, monolithic solution, but is awkward when dividing the monolith into decoupled
+microservices.
+
+In EdgeX today, Devices and Resources can be added that are not related to the south-bound Device or to each other, 
+except perhaps by well-chosen Labels or Tags.
+
+
+### Requirements
+1. Standard EdgeX means exists to relate configuration and status Resources for Application Services to the
+originating south-bound Device data, on a per-Device basis.
+2. The configuration and status Resources for Application Services can be viewed as "extending" the originating
+south-bound Device data.
+3. Application Services, including a REST-based UI, can find and use these configuration and status Resources 
+from other Application Services in addition to the orignal south-bound Device data.
+
+Not a requirement: means of using or combining Resources from multiple 
+
+### Discussion
+
+If this UCR is approved, before an ADR can be written, we must choose one of two paths forward toward a solution.
+
+One path (the ideal one) is to add the extended resources to the south-bound device and its device profile.
+1. Pros:
+    - One source of truth in core-data and core-metadata for all Resources related to each Device.
+    - No special properties have to be added to relate the extended to the original Device Resources.
+2. Cons:
+    - The south-bound Device Service could become confused by Resource entries in its Device Profile that it ought to ignore, so it would not be backwards compatible.
+    - Potential contention as multiple Application Services try to update the Device Profile, adding their Resources.
+    - Would have to add some notation so that core-command could determine which service to address to Get or Set
+    the Device Resources (original and extended), so not backwards compatible.
+
+The alternative path is to create an "Extended" Device under each Application Service, which holds its extended configuration and status Resources.
+1. Pros:
+    - Good decoupling of these extended Resources under each Application Service.
+    - Core-command can use conventional means to know which Service to reach for Get and Set of these Resources.
+    - No changes necessary to south-bound Device Services, who don't know about these extended Resources.
+2. Cons:
+    - Need to add some means to each extended Device to indicate which south-bound Device it extends.
+    - Need to add or modify some core API(s), or their query options, to allow lookup of the original south-bound Device and all of its extended Devices.
+    - Application Services need to do extra work to pull in the extended Resources in addition to the south-bound Device Resources.
+
+### Other Related Issues
+Potentially related to Use Case for Device Parent-Child Relationships [0002 Device Parent-Child Relationships](./0002-Device-Parent-Child-Relationships.md), if the "alternative path" solution is chosen here.
+
+
+### References
+
+
+