RFC: New environment variable POWERTOOLS_DEV to ease prototyping

[heitorlessa]

Is this related to an existing feature request or issue?

• Feature request: Simplified logger formatter when in AWS SAM Local powertools-lambda-python#1527
• Feature request: pretty printing logs in local development powertools-lambda-typescript#751

Which AWS Lambda Powertools utility does this relate to?

Logger, Event Handler, and potentially others

Summary

This RFC outlines the introduction of a new environment variable POWERTOOLS_DEV to ease local prototyping, debugging, and why not per utility env var.

Use case

Powertools optimizes certain outputs for costs (CloudWatch Logs) and to facilitate native integrations (CloudWatch Embedded Metric Format, Batch processing, etc.). It also disables certain behaviours such as tracing when running outside Lambda, so adding Powertools doesn't get in the way.

As our customer base increases, it diversifies its usage and thus lies the opportunity to adapt and improve. For example, customers are running their Lambda function locally in various forms: AWS Chalice, AWS SAM CLI, AWS Amplify CLI, Serverless framework, etc.

This increases our complexity as we try to keep track of potential sources of local emulation and their many environment variables. Notwithstanding customers who may also run Lambda Powertools in non-Lambda environments such as Glue jobs, Fargate, whether we provide support or not.

Proposal

Introduce POWERTOOLS_DEV boolean environment variable for the following effect:

Utility	Effect
Logger	Increase JSON indentation to 4. This will ease local debugging when running functions locally under emulators or direct calls while not affecting unit tests
Event Handler	Enable full traceback errors in the response, indented request/responses, and CORS in dev mode (*). This will deprecate POWERTOOLS_EVENT_HANDLER_DEBUG that achieves the same effect in Event Handler Debug Mode
Tracer	Disables tracing operations. Future-proof non-Lambda environments. As of now, we'll continue with auto-disable feature until we decide when (if) we should move beyond Lambda as a compute environment

This change future-proof additional convenience for customers and lower cognitive load on needing multiple env vars to the same end. We will create a new section in the documentation to keep track of behaviours this env var will enable.

Warning increased cost in log ingestion

We anticipate customers wanting to use POWERTOOLS_DEV in non-production AWS accounts. This might also occur unwillingly. Regardless of the intention, we should log a warning message to signal this feature is in use. This should help prevent an increase in log ingestion cost in production.

Out of scope

Change Tracer auto-disable behaviour in non-Lambda environments. This will cause a breaking change in customers unit tests, which is a great convenient Powertools feature.

We can revisit it when (if) we officially support compute beyond Lambda.

Potential challenges

An open question we have is: What if a customer intentionally deploys with POWERTOOLS_DEV=true? See Warning increased cost in log ingestion

Should we allow it or safely ignore it (e.g., POWERTOOLS_DEV && not TASK_ROOT)? Tracer only

Dependencies and Integrations

None

Alternative solutions

Introduce an environment variable per utility when the need arises, for example POWERTOOLS_LOGGER_INDENTED_LOGS, POWERTOOLS_EVENT_HANDLER_INDENTED_LOGS.

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript, and .NET

[benbridts]

Will this be expected to not be used when running (in a sandbox/dev) account on Lambda?

I might want to convenience of enabling all dev-functionality at once (logging the full event might also be one?), without it impacting how I see things in CloudWatch. POWERTOOLS_LOGGER_INDENTED_LOGS s a good example of things that I want different locally or when using CloudWatch

[heitorlessa]

Exactly, we won't impact the final runtime. I'll make that clearer in the RFC body.

I might want to convenience of enabling all dev-functionality at once (logging the full event might also be one?), without it impacting how I see things in CloudWatch

That's exactly the main argument to use POWERTOOLS_DEV locally instead of what was suggested in the original issue to change from JSON to non-JSON -- POWERTOOLS_DEV will only increase indentation so you can see it better locally but not change at Lambda runtime otherwise it'd increase costs.

[heitorlessa]

Updated the body of the RFC, and also posed a question if a customer accidentally deploys a function with POWERTOOLS_DEV=true instead of only using it locally -- lemme know if it's clearer.

---

Changes

NOTE: We should only change the experience locally, not when deployed at Runtime.

Potential challenges

A customer might accidentally deploys a Lambda function with POWERTOOLS_DEV=true, where this would change the logging output indentation from compact to pretty.

Should we have additional checks for it? e.g., POWERTOOLS_DEV && not TASK_ROOT

[benbridts]

I think POWERTOOLS_LOCAL would be a better name in that case.

I was thinking about it the other way. If this enables (in the future) things like more verbose logging, or logging of more things (inputs / outputs / ...), it might be nice to have it available in a Lambda Function too (if that function is in a developer's account and not getting a lot of traffic).

EDIT: or to put it more clearly: I assumed POWERTOOLS_DEV could also be used to toggle DEBUG-like functionality. I understand now that this is not the case

[heitorlessa]

POWERTOOLS_DEV could also be used for debugging if you intentionally deploy a function with it. We actually had issues with that in the past, where people would forget it and have a much higher bill in CloudWatch Logs, plus sensitive info being logged (or CORS in dev mode).

I do see where you're coming from tho - I made the wording crispier and brought in as a what if question before we settle on this.

In hindsight, I guess we can't protect customers 100% from forgetting that parameter, and putting a guard would prevent you from utilizing in non-prod env. I initially thought it'd be more useful in local executions, but I can see cases where you'd want that deployed when multiple functions are involved.

On the naming itself, the reason we didn't propose LOCAL was to not give the impression Powertools would provide a local execution mode of sorts (ala SAM CLI).

[benbridts]

I do see where you're coming from as you would have the intentions to have more verbose logging in non-prod environments, where this wouldn't work if we have a guard for it. I guess we can't protect customers 100% from forgetting that parameter, and putting a guard would prevent you from utilizing in non-prod env -- I'll improve the RFC.

If this will always switch indentation, I think it's fine to prevent me from using it (but maybe it should print an INFO one-liner to the logs). Setting POWERTOOLS_DEV to True and toggling indentation to be off, is the same effort as toggling individual debug settings.

On the naming itself, the reason we didn't propose LOCAL was to not give the impression Powertools would provide a local execution mode of sorts (ala SAM CLI)

Naming is always difficult :). _DEV gave me the impression that it was similar to DEBUG in tools like Django.

[heitorlessa]

Yeah, hmmm. Future wise, I can't see how we'd detect you're running on Fargate/Glue/Docker and auto-toggle that off at runtime.

What if we simply log a warning (warnings in Python) that you enabled POWERTOOLS_DEV so to instruct this might not be a good idea in prod?

That could solve our side on trying too hard to auto-toggle off this feature in scenarios we can't always predict, and keeping customers informed about the outcome when enabled.

[heitorlessa]

@dreamorosi @saragerion @rubenfonseca @leandrodamascena what do you think of POWERTOOLS_LOCAL name?

In hindsight, whatever name we settle, we could enable another one POWERTOOLS_DEBUG=true to automatically set logging as DEBUG for our internals - For Python, we ask customers to manually enable raw debugging similar to boto.

[heitorlessa]

From @jfuss

I like the idea of: “powertools_mode” and making the value an enum. Allows you to scale to other things in the future possibly.

My concern with true or false, it doesn’t leave you room if you need or want to add thing X but not y. But generally supportive of dev and believe should avoid “local” naming.

My only concern with an enum is that it opens the room for typos given this is an env var. We could technically have a default in the Enum (MODE.DEV)

[benbridts]

What if we simply log a warning (warnings in Python) that you enabled POWERTOOLS_DEV so to instruct this might not be a good idea in prod?

Would you log it everywhere, or only in lambda?

In the second case, I think the indentation would show it's not a great idea anyway.

In the first case, there is a balance between annoying and helpful, but warnings about security and/or cost seem to be on the helpful side (maybe it should log outside the handler, so that environments that keep the function warm don't log on every invocation)

[heitorlessa]

The former. Like we do in Metrics today - we will log a warning (anywhere) if you try to flush metrics without a metric per se - it might be intentional, but we won't raise an Exception unless you tell us to (raise_on_empty_metrics)

[benbridts]

The former

👍🏻

My only concern with an enum is that it opens the room for typos given this is an env var. We could technically have a default in the Enum (MODE.DEV)

If used as intended, that failure should be spotted fast ("I enabled better logging, and I'm not getting better logging"), adding a warning "DECV is not a valid value for POWERTOOLS_MODE" will also point in the right direction

However, I think as long as the intent is clear, a boolean is fine too. A ℹ️ in the docs saying that it will not be very useful when running with CloudWatch might be nice though.

[heitorlessa]

Hypothetically, say we have POWERTOOLS_MODE with three modes: DEV, PROD, DEBUG, where PROD is the default. Consider docs also outlines intent clearly.

If you use XPTO as a value, we log a warning with valid values, and fallback to PROD for safety - does that sound clear enough?

[benbridts]

If you use XPTO as a value, we log a warning with valid values, and fallback to PROD for safety - does that sound clear enough?

Yes. But having POWERTOOLS_DEV and POWERTOOLS_DEBUG as separte constants would work too. Not sure what would happen if you have POWERTOOLS_DEV=True, POWERTOOLS_DEBUG=False though. having the enum avoids confusing combinations of dev and debug

[heitorlessa]

Here's a quick POC -- from a customer point of view, they would only use an environment variable and a string value, not in their code per se.

---

>>> mode = PowertoolsMode("XPTO")

UserWarning: Invalid value: XPTO, defaulting to PROD mode. Valid options available: ['DEV', 'PROD', 'DEBUG']

>>> mode
<PowertoolsMode.PROD: 'PROD'>

import warnings
from enum import Enum


class PowertoolsMode(Enum):
    DEV = "DEV"
    PROD = "PROD"
    DEBUG = "DEBUG"

    @classmethod
    def _missing_(cls, name):
        warnings.warn(f"Invalid value: {name}, defaulting to PROD mode. Valid options available: {list(cls.__members__)}")
        return cls.PROD

[dreamorosi]

Thanks for creating the RFC.

In my opinion the goal of this environment variable should be bringing better developer experience. This should also be reflected in the docs / messaging around it, but also in the name:

• POWERTOOLS_DEBUG sounds overly generic and as already stated by others, it might cause confusion around where to use it / what it does
• POWERTOOLS_LOCAL is a good approximation. It conveys that it should not be used on a live environment and I wouldn't worry too much about confusing it with SAM Local. People who know what SAM Local is will most probably realise that this is just an environment variable.
• POWERTOOLS_LOCAL_DEV <- however, if we want to hammer the concept that this is a local & development only flag, then we could spell it out fully. This would merge both name initially proposed (POWERTOOLS_DEV) and the one that was added later.

My preference (in order):

1. POWERTOOLS_LOCAL_DEV
2. POWERTOOLS_LOCAL

For the purpose of this RFC, I would also leave out the discussion about POWERTOOLS_DEBUG. It's an interesting discussion but it also has a wider scope (many utilities potentially involved) and at the moment it's not relevant to all runtimes (i.e. Powertools for TypeScript doesn't log anything that is not an error outside of logger).

About the enum/PowertoolsMode that started out from this message, I think I'm missing some context as I'm not sure where this is coming from or why it's preferred to the simpler env var.

At this stage I'd be inclined to keep things simple and stick to POWERTOOLS_LOCAL_DEV (or similar) as an environment variable that modifies some of the behaviors in Powertools. Adding additional complexity in both the logic

About the open question(s):

What if a customer intentionally deploys with POWERTOOLS_DEV=true?
Should we allow it or safely ignore it (e.g., POWERTOOLS_DEV && not TASK_ROOT)?
They might want prettier logs and debugging data in non-prod environments;

At least at this stage, when it comes to logging, CloudWatch already indents JSON structured logs when you expand an entry in a Log group. So it's safe to say that the setting should not be used on AWS Lambda and a warning could be enough.

When it comes to extra data, while I see some value in emitting a warning, I'm also not sure how the environment variable used in local dev or unit tests could end up in the env vars configuration on Lambda. Both manual deployment (via Console) or IaC require a conscious effort to set env variables.

I'm really on the fence on this topic. Adding a warning seems sensible and relatively low effort so I'm not against it.

[rubenfonseca]

Thank you for this great discussoin.

While I understand the motivation for the change, I'm worried about POWERTOOLS_MODE as an Enum: it lowers the barrier too much for creating all kinds of strange "modes" in the futue.

I think there is room for having both environment variables: POWERTOOLS_LOCAL / POWERTOOLS_LOCAL_DEV and POWERTOOLS_DEBUG which have two separate meanings:

• LOCAL: I want to enable features that will make it easier for me to develop locally, like easier to read logs, etc.
• DEBUG: I want to enable additional logging because I'm debugging some functionality. This is true whenever I run locally or in the cloud.

So unless I'm missing something, I would love to separate the two variables that control separate behaviour.

Yes, I can have DEBUG on and LOCAL false, because I'm debugging something on the cloud. Yes, I can have LOCAL on and DEBUG off because I just want to read the logs output more clearly. And of course I can enable both to have the best of both worlds.

[heitorlessa]

Thank you @rubenfonseca @dreamorosi - that's super helpful to hear your input.

After re-reading the entire RFC, I'm inclined to move forward with POWERTOOLS_DEV and POWERTOOLS_DEBUG boolean flags. Having both helps make the intent more explicit. If a customer accidentally leaves them turned when deployed, a warning should suffice as an actionable advice on what to do.

If either @rubenfonseca @dreamorosi @leandrodamascena @saragerion @msailes disagrees, please shout out until tomorrow. I'll create a separate PR in the Python repo to document when to use which and their effects as per RFC.

Thank you also to @benbridts @jfuss

[saragerion]

@heitorlessa can we recap shortly in bullet points the difference between the 2 and the impact on each utility?

[heitorlessa]

Absolutely. I'll update the RFC body tomorrow too:

POWERTOOLS_DEV

Provides additional information when prototyping an application.

Effect on utilities:

• Logger. Increase JSON indentation to 4 to ease debugging complex operations and read logs locally.
• Tracer. Disables tracing operations. Futureproof only as we'll continue with auto-disable feature until we move beyond Lambda as a compute environment.
• Event Handler. Indent request/responses to 4, include full traceback errors in response, and CORS in dev mode (*). This will deprecate POWERTOOLS_EVENT_HANDLER_DEBUG that achieves the same effect in Event Handler Debug Mode.

POWERTOOLS_DEBUG

Configures Lambda Powertools internal logging for debug logging for all operations.

Current Python implementation following AWS Python SDK standard: https://awslabs.github.io/aws-lambda-powertools-python/latest/#debug-mode

[heitorlessa]

RFC body updated to reflect the direction we agreed. @pmarko1711 was kind enough to contribute the first implementation for Logger in Powertools for Python: aws-powertools/powertools-lambda-python#1548

Before I close as successful, does anyone know of any ready-to-use automation to convert a RFC issue into a PR as markdown?

Storing these in this repo will ease discovery of cross-languages specifics.

[dreamorosi]

@heitorlessa, I saw that Python has already merged the changes related to POWERTOOLS_DEV & Logger, however I'm not clear from the RFC body on whether we should also do these changes to Tracer.

Should POWERTOOLS_DEV=${truthy_value) disable the Tracer? I'd say yes as I consider this an explicit disable, but wanted to make sure it's in line with the RFC before moving forward.

[heitorlessa]

It would break customers test if that was the case — I documented as “future proof” and Tracer continues to work as-is (disabled when not in Lambda) https://awslabs.github.io/aws-lambda-powertools-python/latest/#optimizing-for-non-production-environments

…

On Wed, 19 Oct 2022 at 12:26, Andrea Amorosi ***@***.***> wrote: @heitorlessa <https://github.com/heitorlessa>, I saw that Python has already merged the changes related to POWERTOOLS_DEV & Logger, however I'm not clear from the RFC body on whether we should also do these changes to Tracer. Should POWERTOOLS_DEV=${truthy_value) disable the Tracer? I'd say yes as I consider this an explicit disable, but wanted to make sure it's in line with the RFC before moving forward. — Reply to this email directly, view it on GitHub <#86 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBGYTWDQIF46H2WBYD3WD7EETANCNFSM6AAAAAAQXZFZTM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dreamorosi]

It would break customers test if that was the case — I documented as
“future proof” and Tracer continues to work as-is (disabled when not in
Lambda)

https://awslabs.github.io/aws-lambda-powertools-python/latest/#optimizing-for-non-production-environments

I'm not sure I'm following. We both agree that Tracer existing behavior shouldn't change. This includes explicitly disabling it via env variable POWERTOOLS_TRACE_DISABLED or other direct settings. It also includes the current heuristics to disable automatically when outside of Lambda.

My comment was referred to: should POWERTOOLS_DEV be an additional way to disable tracing? As in, on top of the existing behavior, when a customer sets POWERTOOLS_DEV=1 Tracer is disabled.

From the doc section you linked the answer seems to be yes. But then, maybe I'm missing how it would break existing customers since it's a new setting that doesn't change/modifies other existing ones.

I'm sorry if this discussion is a bit redundant but I'm not 100% clear and I prefer to clarify

[chrisandrewcl]

Isn't this config akin to tying behavior to NODE_ENV (which I think most people agree is not a good idea)? E.g.: if I want pretty-print on my logs I will have to accept other behaviors that I may not want (it's three or four now, but if not careful it tends to grow over time).

I am not against the convenience of having an easier configuration, but please don't take away granular control.

Also, there may be a third alternative: built-in config presets (e.g.: CONFIG_PRESET=dev), which also gives space for community presets.

[leandrodamascena]

Hey @heitorlessa and @dreamorosi! Do we still need to keep this issue open or can we close it as complete?

[dreamorosi]

Hey Leo, not sure about Python but in TS we only implemented this for Logger, the proposal also mentioned Event Handler (which we don't have yet) and Tracer.

Depending on the status, and whether we still want to move forward with the proposal you can close it or leave it, I think.

[heitorlessa]

launched it