0033-new-cloud-fields.md

0033: New cloud (or related) fields

• Stage: 0 (strawperson)
• Date: 2022-07-22

This RFC is to add several new cloud or cloud related fields to ECS. As more cloud integrations/data sources are developed there are a variety of fields that don't have an ECS normalized field to put them. This would add the ability to normalize data related to cloud instance images, network/VPC information, S3/Object storage data....

Fields

Initial list of proposed fields:

• cloud.edge_location: The edge location, usually an airport IATA code (DFW), of a cloud service. Very common for DNS, CDN, or other cloud services
• cloud.instance.image.id: ID of the image used to create the virtual instance/machine
• cloud.instance.image.name: Name of the image used to create the virtual instance/machine
• cloud.vpc.name: Name of the virtual network
• cloud.vpc.id: ID of the virtual network
• cloud.subnet.name: Name of the Subnet within the VPC/Virtual network
• cloud.subnet.id: ID of the Subnet within the VPC/Virtual network
• cloud.instance.lifecycle: Type of instance normal vs spot, see #323 for more discussion
• cloud.object_store.name: Name of S3/Compatible storage
• cloud.object_store.id: Name of S3/Compatible storage

Possible alternatives proposed in #1725

• network.edge_location instead of cloud.edge_location
• cloud.image.id instead of cloud.instance.image.id
• cloud.image.name instead of cloud.instance.image.name
• network.name (already exists) instead of instead of cloud.vpc.name
• network.id instead of cloud.vpc.id
• network.subnetwork.name instead of instead of cloud.subnet.name
• network.subnetwork.id instead of cloud.subnet.id

Usage

This started with multiple new cloud integrations for the agent having edge location data fields to include Akamai, Cloudflare, AWS DNS, AWS Cloudfront... Additionally AWS Guard Duty integration has numerous data points with regards to AWS network, S3/Object storage, instance information...

Source data

AWS Cloudfront Logs

2019-12-04 21:02:31 LAX1 392 89.160.20.112 GET d111111abcdef8.cloudfront.net /index.html 200 - .......

AWS Public ZOne DNS Logs

1.0 2017-12-13T08:16:02.130Z Z123412341234 example.com A NOERROR UDP FRA6 89.160.20.112 -

Cloudflare HTTP Logs

{..."EdgeColoCode":"AMS","EdgeColoID":20,...}

GCP Flow logs:

..."src_vpc":{"project_id":"my-sample-project","subnetwork_name":"default","vpc_name":"default"}...

GCP DNS logs

..."sourceIP":"10.154.0.3","sourceNetwork":"default","vmInstanceId":8340998530665147,"vmInstanceIdString":"8340998530665147",...

AWS Guard Duty Logs

  {
    "schemaVersion": "2.0",
    "accountId": "290443255379",
    "region": "us-east-2",
    "partition": "aws",
    "id": "02bf27df0ab318783b0a8f63569dfd68",
    "arn": "arn:aws:guardduty:us-east-2:290443255379:detector/c0bf27def0a899b467ad81c4a5681b78/finding/02bf27df0ab318783b0a8f63569dfd68",
    "type": "Impact:S3/MaliciousIPCaller",
    "resource": {
      "resourceType": "S3Bucket",
      "accessKeyDetails": {
        "accessKeyId": "GeneratedFindingAccessKeyId",
        "principalId": "GeneratedFindingPrincipalId",
        "userType": "IAMUser",
        "userName": "GeneratedFindingUserName"
      },
      "s3BucketDetails": [
        {
          "owner": {
            "id": "CanonicalId of Owner"
          },
          "createdAt": 1513612691.551,
          **"name": "bucketName",**
          "defaultServerSideEncryption": {
            "kmsMasterKeyArn": "arn:aws:kms:region:123456789012:key/key-id",
            "encryptionType": "SSEAlgorithm"
          },
          "arn": "arn:aws:s3:::bucketName",
          "type": "Destination",
        }
      ],
      "instanceDetails": {
        "instanceId": "i-99999999",
        "instanceType": "m3.xlarge",
        "outpostArn": "arn:aws:outposts:us-west-2:123456789000:outpost/op-0fbc006e9abbc73c3",
        "launchTime": "2016-08-02T02:05:06.000Z",
        "platform": null,
        "productCodes": [
          {
            "productCodeId": "GeneratedFindingProductCodeId",
            "productCodeType": "GeneratedFindingProductCodeType"
          }
        ],
        "iamInstanceProfile": {
          "arn": "arn:aws:iam::290443255379:example/instance/profile",
          "id": "GeneratedFindingInstanceProfileId"
        },
        "networkInterfaces": [
          {
            "networkInterfaceId": "eni-bfcffe88",
            "privateIpAddresses": [
              {
                "privateDnsName": "GeneratedFindingPrivateName",
                "privateIpAddress": "10.0.0.1"
              }
            ],
            "subnetId": "GeneratedFindingSubnetId",
            "vpcId": "GeneratedFindingVPCId",
            "privateDnsName": "GeneratedFindingPrivateDnsName",
            "securityGroups": [
              {
                "groupName": "GeneratedFindingSecurityGroupName",
                "groupId": "GeneratedFindingSecurityId"
              }
            ],
            "publicIp": "198.51.100.0",
            "ipv6Addresses": [],
            "publicDnsName": "GeneratedFindingPublicDNSName",
            "privateIpAddress": "10.0.0.1"
          }
        ],
        "instanceState": "running",
        "availabilityZone": "GeneratedFindingInstaceAvailabilityZone",
        "imageId": "ami-99999999",
        "imageDescription": "GeneratedFindingInstaceImageDescription"
      }
    }

Scope of impact

Concerns

People

The following are the people that consulted on the contents of this RFC.

• @legoguy1000 | author

References

RFC Pull Requests

• Stage 0: #1953