Allow configuration of AppDynamics applicationName, nodeName, and tierName

[maxbrunsfeld]

Some colleagues of ours have multiple Cloud Foundry apps with the same name: one for staging, one for production etc. Currently, on their AppDynamics dashboard, their apps are listed like this:

- my-app-name
   - Cloud Foundry
        - (some instance guid)

Data from their staging and production environments is mixed together. Also, every time they re-deploy their app they see a new node name. They'd like some way to configure how their data is structured: maybe one or more environment variables they could set?

CF Community Pair
Caleb and Max (@calebamiles and @maxbrunsfeld)

[rkawala]

If the default app dynamics app name was <cf appname>_<cf space> it would be a better default.
Allowing us to pass in -Dappdynamics.agent.applicationName=CondoSafe_staging and have it take affect would be another option.

For the node name - is there a concept of instance index? That would be nicer than guid since it would be reused after redeploys.

[nebhale]

One option for separating the environments is to change the tier_name configuration value. This option is missing from the documentation, but you can see it here.

The problem of many node names is endemic to all of the monitoring systems we support. The problem is that since instance indexes (i.e. VCAP_APPLICATION['instance_index']) are reused you end up with a continuum of historical data that is inconsistently related to itself.

To be a bit more concrete. Imagine that you've got an instance #0 deployed on DEA ALPHA. It's reporting that there are no performance problems. Then you do an upgrade of Cloud Foundry. That instance is destroyed as it is migrated over to DEA BRAVO temporarily. This is still reported as instance #0 even though it is a completely new JVM on a completely different virtual (and possibly physical) host. Now, this instance #0 reports that there are massive performance problems. After the migration is complete, instance #0 is migrated back to DEA ALPHA where there are no performance problems. Looking back on the history in AppDynamics we see that for a period of time, instance #0 had performance problems so we start investigating. What DEA do we investigate? How do we know that instance #0 was temporarily migrated?

We've had discussions with both AppDynamics and New Relic about this issue and there is no panacea. The general consensus so far (again, this is just today) is that we stick with unique node names (i.e. VCAP_APPLICATION['instance_id']) and instruct users to look at "rolled up" collections of metrics. The idea being that in a cloud environment, where instances come and go regularly, the performance of an individual instance isn't particularly important. Instead, it's the performance of the application as a whole (i.e. cumulative behavior of all instances) that matters.

I'd be happy to have a call to talk this through a bit more and get an idea of how you'd like to see things behaving; while we may not make a change, it's always good to get more information about customer experiences.

/cc @rmorgan

[youngm]

FWIW, in our fork we use the instance index for app dynamics. Feedback from our application support engineers was that app dynamics rolled up statistics were not good and that they found more value looking at the cumulative long term performance of an application as a single node. We eventually relented. We decided that it was the CF team's job to find issues between DEAs and CF releases. But, our ASE's job was to look for issues in the application itself not CF or the DEAs. Use of indeance id was getting in the way of our ASEs ability to identify issues in their applications. So far our ASEs have been happy with the change.

Mike

[nebhale]

@youngm That's good input to have. Do you guys connect much with the AppDynamics engineers? Are they aware of the struggles you've had with rollup statistics and monitoring cloud applications?

[youngm]

@nebhale No, we haven't worked much with App Dynamics directly. We've just been winging it with our users.

[nebhale]

@youngm We're going to have a discussion here (and with the AppDynamics guys) about what we should do. I'll let you know when we've got an answer.

[nebhale]

The points discussed in this issue will be addressed in the following:

• https://www.pivotaltracker.com/story/show/75252406
• https://www.pivotaltracker.com/story/show/75252498

[youngm]

Will this story also support optionally pulling nodeName from credentials? Or just tierName?

[nebhale]

@youngm At the moment, I'm thinking just tierName because I believe it would address @rkawala's use-case. By having the ability to set the tier-name on a per-service basis (so per-space, and hence per-deployment type), you might end up with this:

- my-app-name
  - development
    - 0
  - qa-testing
    - 0
    - 1
  - production
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5

Does that address what you'd like to see as well, or do you think that you'd want to set nodeName as well? The problem with setting nodeName in credentials is that every instance that binds to it gets the same nodeName

You'd end up with:

- my-app-name
  - development
    - node-name (collection 1 instance's data)
  - qa-testing
    - node-name (collection 2 instances' data)
  - production
    - node-name (collection 5 instances 'data)

[youngm]

@nebhale I see. I actually got mixed up between node name and app name. That makes sense since you use the cf app name as the app dynamics "app name".

In our organization we use app dynamics for more than just CF apps. And we have so many apps that the team managing app dynamics has mandated that we organize our app dynamics app, tier, and node like so:

- Organization (Business Unit Not necessarily cf organization)
  - app - environment (Human readable tier name)
    - cf_app_name[0]
    - cf_app_name[1]

I had a small hope that this feature might allow us to uncustomize the app dynamics portion of our buildpack fork by being able to configure app name, tier name, and node name all in the service credentials. However, I now see that would be difficult seeing how we each handle cf app name differently.

Mike

[nebhale]

@youngm Am I correct then in my interpretation of the structure in the previous example:

• appdynamics.agent.applicationName is set to "Organization (Business Unit Not necessarily cf organization)"
• appdynamics.agent.tierName is set to "app - environment (Human readable tier name)"
• appdynamics.agent.nodeName is set to #{@application.details['application_name']}[<INSTANCE-INDEX>]

[youngm]

@nebhale Yes, your interpretation is accurate.

[nebhale]

@youngm What about my choices being the default and any of them being overridable via configuration or service credentials (service credentials taking priority)?

[youngm]

@nebhale That would work great. Though I'd need some way to override nodeName (at least) with a dynamic value.

For reference this is the nodeName string I use in our buildpack fork. It took some work getting all the escapes and quotes correct. :)

"#{@application.details['application_name']}[$(expr \"$VCAP_APPLICATION\" : '.*\"instance_index[\": ]*\\([0-9]\\+\\).*')]"

[nebhale]

@youngm Understood. Let me think about the best way to do that.

[nebhale]

Improvements made. After quite some time I'm going to close this out. @ALL Please feel free to open new issues if you're not satisfied with the changes here.