Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data-driven Terraform Configuration #6430

Closed
wants to merge 21 commits into from
Closed

Commits on May 6, 2016

  1. config: "ResourceMode" concept for resources

    Previously resources were assumed to always support the full set of
    create, read, update and delete operations, and Terraform's resource
    management lifecycle.
    
    Data sources introduce a new kind of resource that only supports the
    "read" operation. To support this, a new "Mode" field is added to
    the Resource concept within the config layer, which can be set to
    ManagedResourceMode (to indicate the only mode previously possible) or
    DataResourceMode (to indicate that only "read" is supported).
    
    To support both managed and data resources in the tests, the
    stringification of resources in config_string.go is adjusted slightly
    to use the Id() method rather than the unusual type[name] serialization
    from before, causing a simple mechanical adjustment to the loader tests'
    expected result strings.
    apparentlymart committed May 6, 2016
    Configuration menu
    Copy the full SHA
    67f4633 View commit details
    Browse the repository at this point in the history
  2. config: Data source loading

    This allows the config loader to read "data" blocks from the config and
    turn them into DataSource objects.
    
    This just reads the data from the config file. It doesn't validate the
    data nor do anything useful with it.
    apparentlymart committed May 6, 2016
    Configuration menu
    Copy the full SHA
    3275bbf View commit details
    Browse the repository at this point in the history
  3. config: Parsing of data.TYPE.NAME.FIELD variables

    This allows ${data.TYPE.NAME.FIELD} interpolation syntax at the
    configuration level, though since there is no special handling of them
    in the core package this currently just acts as an alias for
    ${TYPE.NAME.FIELD}.
    apparentlymart committed May 6, 2016
    Configuration menu
    Copy the full SHA
    225df30 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2016

  1. core: New ResourceProvider methods for data resources

    This is a breaking change to the ResourceProvider interface that adds the
    new operations relating to data sources.
    
    DataSources, ValidateDataSource, ReadDataDiff and ReadDataApply are the
    data source equivalents of Resources, Validate, Diff and Apply (respectively)
    for managed resources.
    
    The diff/apply model seems at first glance a rather strange workflow for
    read-only resources, but implementing data resources in this way allows them
    to fit cleanly into the standard plan/apply lifecycle in cases where the
    configuration contains computed arguments and thus the read must be deferred
    until apply time.
    
    Along with breaking the interface, we also fix up the plugin client/server
    and helper/schema implementations of it, which are all of the callers
    used when provider plugins use helper/schema. This would be a breaking
    change for any provider plugin that directly implements the provider
    interface, but no known plugins do this and it is not recommended.
    
    At the helper/schema layer the implementer sees ReadDataApply as a "Read",
    as opposed to "Create" or "Update" as in the managed resource Apply
    implementation. The planning mechanics are handled entirely within
    helper/schema, so that complexity is hidden from the provider implementation
    itself.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    834ba48 View commit details
    Browse the repository at this point in the history
  2. helper/schema: Resource can be writable or not

    In the "schema" layer a Resource is just any "thing" that has a schema
    and supports some or all of the CRUD operations. Data sources introduce
    a new use of Resource to represent read-only resources, which require
    some different InternalValidate logic.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    7c094c8 View commit details
    Browse the repository at this point in the history
  3. helper/schema: shim for making data sources act like resources

    Historically we've had some "read-only" and "logical" resources. With the
    addition of the data source concept these will gradually become data
    sources, but we need to retain backward compatibility with existing
    configurations that use the now-deprecated resources.
    
    This shim is intended to allow us to easily create a resource from a
    data source implementation. It adjusts the schema as needed and adds
    stub Create and Delete implementations.
    
    This would ideally also produce a deprecation warning whenever such a
    shimmed resource is used, but the schema system doesn't currently have
    a mechanism for resource-specific validation, so that remains just a TODO
    for the moment.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    9c59740 View commit details
    Browse the repository at this point in the history
  4. provider/terraform: remote state resource becomes a data source

    As a first example of a real-world data source, the pre-existing
    terraform_remote_state resource is adapted to be a data source. The
    original resource is shimmed to wrap the data source for backward
    compatibility.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    98f5880 View commit details
    Browse the repository at this point in the history
  5. helper/schema: emit warning when using data source resource shim

    For backward compatibility we will continue to support using the data
    sources that were formerly logical resources as resources for the moment,
    but we want to warn the user about it since this support is likely to
    be removed in future.
    
    This is done by adding a new "deprecation message" feature to
    schema.Resource, but for the moment this is done as an internal feature
    (not usable directly by plugins) so that we can collect additional
    use-cases and design a more general interface before creating a
    compatibility constraint.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    71b4041 View commit details
    Browse the repository at this point in the history
  6. website: Initial documentation about data sources

    This will undoubtedly evolve as implementation continues, but this is some
    initial documentation based on the design doc.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    ceef95f View commit details
    Browse the repository at this point in the history
  7. core: ResourceStateKey understands resource modes

    Once a data resource gets into the state, the state system needs to be
    able to parse its id to match it with resources in the configuration.
    Since data resources live in a separate namespace than managed resources,
    the extra "mode" discriminator is required to specify which namespace
    we're talking about, just like we do in the resource configuration.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    fb4be16 View commit details
    Browse the repository at this point in the history
  8. core: EvalValidate calls appropriate validator for resource mode

    data resources are a separate namespace of resources than managed
    resources, so we need to call a different provider method depending on
    what mode of resource we're visiting.
    
    Managed resources use ValidateResource, while data resources use
    ValidateDataSource, since at the provider level of abstraction each
    provider has separate sets of resources and data sources respectively.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    aefc4cd View commit details
    Browse the repository at this point in the history
  9. core: Separate resource lifecycle for data vs. managed resources

    The key difference between data and managed resources is in their
    respective lifecycles. Now the expanded resource EvalTree switches on
    the resource mode, generating a different lifecycle for each mode.
    
    For this initial change only managed resources are implemented, using the
    same implementation as before; data resources are no-ops. The data
    resource implementation will follow in a subsequent change.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    53d8f43 View commit details
    Browse the repository at this point in the history
  10. core: lifecycle for data resources

    This implements the main behavior of data resources, including both the
    early read in cases where the configuration is non-computed and the split
    plan/apply read for cases where full configuration can't be known until
    apply time.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    7da3f6d View commit details
    Browse the repository at this point in the history
  11. core: separate lifecycle for data resource "orphans"

    The handling of data "orphans" is simpler than for managed resources
    because the only thing we need to deal with is our own state, and the
    validation pass guarantees that by the time we get to refresh or apply
    the instance state is no longer needed by any other resources and so
    we can safely drop it with no fanfare.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    e4c6dc8 View commit details
    Browse the repository at this point in the history
  12. core: Destroy data resources with "terraform destroy"

    Previously they would get left behind in the state because we had no
    support for planning their destruction. Now we'll create a "destroy" plan
    and act on it by just producing an empty state on apply, thus ensuring
    that the data resources don't get left behind in the state after
    everything else is gone.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    d0d1502 View commit details
    Browse the repository at this point in the history
  13. core: ResourceAddress supports data resources

    The ResourceAddress struct grows a new "Mode" field to match with
    Resource, and its parser learns to recognize the "data." prefix so it
    can set that field.
    
    Allows -target to be applied to data sources, although that is arguably
    not a very useful thing to do. Other future uses of resource addressing,
    like the state plumbing commands, may be better uses of this.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    dcce293 View commit details
    Browse the repository at this point in the history
  14. command: Prevent data resources from being tainted

    Since the data resource lifecycle contains no steps to deal with tainted
    instances, we must make sure that they never get created.
    
    Doing this out in the command layer is not the best, but this is currently
    the only layer that has enough information to make this decision and so
    this simple solution was preferred over a more disruptive refactoring,
    under the assumption that this taint functionality eventually gets
    reworked in terms of StateFilter anyway.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    9bd92e7 View commit details
    Browse the repository at this point in the history
  15. command: Show id only when refreshing managed resources

    Data resources don't have ids when they refresh, so we'll skip showing the
    "(ID: ...)"  indicator for these. Showing it with no id makes it look
    like something is broken.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    87475f1 View commit details
    Browse the repository at this point in the history
  16. command: don't show old values for create diffs in plan

    New resources logically don't have "old values" for their attributes, so
    showing them as updates from the empty string is misleading and confusing.
    
    Instead, we'll skip showing the old value in a creation diff.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    6fe0062 View commit details
    Browse the repository at this point in the history
  17. command: Show data source reads differently in plans

    Internally a data source read is represented as a creation diff for the
    resource, but in the UI we'll show it as a distinct icon and color so that
    the user can more easily understand that these operations won't affect
    any real infrastructure.
    
    Unfortunately by the time we get to formatting the plan in the UI we
    only have the resource names to work with, and can't get at the original
    resource mode. Thus we're forced to infer the resource mode by exploiting
    knowledge of the naming scheme.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    4777f7f View commit details
    Browse the repository at this point in the history
  18. provider/null: null_data_source data source

    A companion to the null_resource resource, this is here primarily to
    enable manual quick testing of data sources workflows without depending
    on any external services.
    
    The "inputs" map gets copied to the computed "outputs" map on read,
    "rand" gives a random number to exercise cases with constantly-changing
    values (an anti-pattern!), and "has_computed_default" is settable in
    config but computed if not set.
    apparentlymart committed May 8, 2016
    Configuration menu
    Copy the full SHA
    1711372 View commit details
    Browse the repository at this point in the history