Improve README, add contributing guide and license

CONTRIBUTING.md

@@ -0,0 +1,50 @@
+# Contributing Guidelines
+
+Cycloid Team is glad to see you contributing to this project ! In this document, we will provide you some guidelines in order to help get your contribution accepted.
+
+## Reporting an issue
+
+### Issues
+
+When you find a bug in cost-estimation, it should be reported using [Github issues](https://github.com/cycloidio/cost-estimation/issues). Please provide key information like your Operating System (OS), Go version and finally the version of the library that you're using.
+
+## Submit a contribution
+
+### Setup your git repository
+
+If you want to contribute to an existing issue, you can start by _forking_ this repository, then clone your fork on your machine.
+
+```shell
+$ git clone https://github.com/<your-username>/cost-estimation.git
+$ cd cost-estimation
+```
+
+In order to stay updated with the upstream, it's highly recommended to add `cycloidio/cost-estimation` as a remote upstream.
+
+```shell
+$ git remote add upstream https://github.com/cycloidio/cost-estimation.git
+```
+
+Do not forget to frequently update your fork with the upstream.
+
+```shell
+$ git fetch upstream --prune
+$ git rebase upstream/master
+```
+
+## Adding a new resource
+
+### AWS
+
+1. Familiarize yourself with the official AWS pricing page for the service as well as the Terraform documentation for the resource you want to add. Note all factors that influence the cost.
+2. Download and familiarize yourself with the pricing data CSV. This can be done by first checking the [index.json](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json), finding the respective service under the `offers` key and downloading the file at the URL under the `currentVersionUrl` (replace `json` with `csv`).
+3. Find the names of all columns that contain relevant cost factors and check that the `aws/field/field.go` file contains them - add them if this is not the case. The constant name should be a correct Go identifier, while the comment should contain the name as it appears in the CSV file.
+4. Run `go generate ./...` to regenerate the field list.
+5. Create a new file with the name of the Terraform resource (without the `aws` prefix), e.g. for `aws_db_instance` it would be `db_instance.go`. It should include two new structs: `Resource` (that is an intermediate struct containing only the relevant cost factors) and `resourceValues` (that directly represents the values from the Terraform resource.) Additionally, the `Resource` struct must implement the `Components` method that returns `[]query.Component`. See the other existing resources for inspiration.
+6. Write tests for your resource. As before, check the other existing test files for inspiration.
+7. Test and make sure that estimating your resource works.
+8. Open a PR with the changes and please try to provide as much information as possible, especially: description of all the cost factors that the PR uses, links to Terraform docs and AWS pricing page, examples of a Terraform file and the resulting estimation.
+
+## Adding a new provider/backend
+
+**Please be aware that, at the moment, Cycloid only supports MySQL as a backend and AWS as cloud provider.** Based on this, please refrain from making contributions that add a new backend or cloud provider as we cannot guarantee they'd be merged and/or supported. To make improvements in this area, please instead open an appropriate issue so that we can discuss it and provide any necessary guidance.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Cycloid
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,103 @@
 # cost-estimation
 
-Go library for estimating Terraform costs using ingested cloud vendor prices.
+Go library for estimating Terraform costs using ingested cloud vendor prices. It is meant to be imported and used by programs (API's or standalone) with access to a MySQL-compatible database and the Internet.
+
+## Installation
+
+```shell
+go get github.com/cycloidio/cost-estimation
+```
+
+## Requirements
+
+- Go 1.15 or newer (older may work but are not supported)
+- MySQL database
+
+## Provider support
+
+Currently, cost-estimation supports only a few resources of the AWS provider, however we're actively working on adding more support. See the full list of supported resources [on this wiki page](https://github.com/cycloidio/cost-estimation/wiki/Supported-Resources).
+
+## Usage
+
+### Migrating the database
+
+```go
+db, err := sql.Open("mysql", "...")
+
+// Can be called on every start of your program, it does nothing if the migrations
+// have been executed already.
+err := mysql.Migrate(ctx, db, "pricing_migrations")
+```
+
+### Ingesting pricing data
+
+```go
+db, err := sql.Open("mysql", "...")
+backend := mysql.NewBackend(db)
+
+// service can be "AmazonEC2" or "AmazonRDS"
+// region is any AWS region, e.g. "us-east-1" or "eu-west-3"
+ingester := aws.NewIngester(service, region)
+err = costestimation.IngestPricing(ctx, backend, ingester)
+```
+
+### Tracking ingestion progress
+
+We're using the `github.com/machinebox/progress` library for tracking ingestion progress.
+
+1. Create a channel that will receive progress updates and set up a goroutine (it will print the bytes ingested out of bytes total and remaining time each time progress update is sent on the channel):
+
+```go
+progressCh := make(chan progress.Progress, 0)
+
+go func() {
+	for p := range progressCh {
+		// Check the docs for all available methods: https://pkg.go.dev/github.com/machinebox/progress#Progress
+		fmt.Printf("%d / %d (%s remaining)\n", p.N(), p.Size(), p.Remaining().String())
+	}
+}()
+```
+
+2. Initialize an ingester capable of tracking progress (in this example the channel will receive an update every 5 seconds):
+
+```go
+ingester := aws.NewIngester(service, region, aws.WithProgress(progressCh, 5*time.Second))
+```
+
+3. Use the ingester as in the previous section.
+
+### Estimating a Terraform plan
+
+Plan estimation is possible after all the relevant pricing data have been ingested and stored in the
+database.
+
+1. Generate a plan using `terraform` and convert it to JSON:
+
+```shell
+terraform plan -out update.tfplan
+terraform show -json update.tfplan > tfplan.json
+```
+
+2. Read the plan file, estimate it and show the resource differences:
+
+```go
+db, err := db.Open("mysql", "...")
+backend := mysql.NewBackend(db)
+
+file, err := os.Open("path/to/tfplan.json")
+plan, err := costestimation.EstimateTerraformPlan(ctx, backend, file)
+
+for _, res := range plan.ResourceDifferences() {
+    fmt.Printf("%s: %s -> %s\n", res.Address, res.PriorCost().String(), res.PlannedCost().String())
+}
+```
+
+Check the documentation for all available fields.
+
+## Contributing
+
+For Contributing Guide, please read [CONTIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+This project is licensed under the MIT license. Please see [LICENSE](LICENSE) for more details.

aws/doc.go

@@ -0,0 +1,6 @@
+// Package aws enables cost estimation of AWS resources and ingestion of pricing data from AWS.
+//
+// It should not be used directly by consumers, instead, please use the costestimation package.
+// This package contains implementations of Ingester and TerraformProviderInitializer that are
+// used internally by other packages.
+package aws

aws/field/field.go

@@ -5,6 +5,7 @@ package field
 // Field represents a single column in an AWS offer file.
 type Field uint8
 
+// List of fields used by the AWS pricing offer file (CSV).
 const (
 	SKU                Field = iota // SKU
 	OfferTermCode                   // OfferTermCode

aws/terraform/db_instance.go

@@ -12,31 +12,44 @@ import (
 	"github.com/cycloidio/cost-estimation/util"
 )
 
+// DBInstance represents an RDS database instance definition that can be cost-estimated.
 type DBInstance struct {
 	providerKey string
 
 	region       region.Code
 	instanceType string
 
-	databaseEngine   string
-	databaseEdition  string
-	licenseModel     string
+	// databaseEngine can be one of "Aurora MySQL", "MariaDB", "MySQL", "PostgreSQL", "Oracle", "SQL Server".
+	databaseEngine string
+
+	// databaseEdition is only valid for Oracle and SQL Server and denotes an edition of the database.
+	databaseEdition string
+
+	// licenseModel is only valid for Oracle and SQL Server and can be either "License included" or "Bring your own license".
+	licenseModel string
+
+	// deploymentOption can be either "Single-AZ" or "Multi-AZ".
 	deploymentOption string
 
-	storageType      string
+	// storageType can be either "standard" (magnetic), "io1" (provisioned IOPS) or "gp2" (general purpose).
+	storageType string
+
+	// allocatedStorage is how much storage should be allocated for the database, in GB.
 	allocatedStorage decimal.Decimal
-	storageIOPS      decimal.Decimal
+
+	// storageIOPS is only valid for Provisioned IOPS types of storage and denotes the amount of IOPS allocated.
+	storageIOPS decimal.Decimal
 }
 
 type dbInstanceValues struct {
-	InstanceClass string `mapstructure:"instance_class"`
-	AvailabilityZone string `mapstructure:"availability_zone"`
-	Engine string `mapstructure:"engine"`
-	LicenseModel string `mapstructure:"license_model"`
-	MultiAZ bool `mapstructure:"multi_az"`
+	InstanceClass    string  `mapstructure:"instance_class"`
+	AvailabilityZone string  `mapstructure:"availability_zone"`
+	Engine           string  `mapstructure:"engine"`
+	LicenseModel     string  `mapstructure:"license_model"`
+	MultiAZ          bool    `mapstructure:"multi_az"`
 	AllocatedStorage float64 `mapstructure:"allocated_storage"`
-	StorageType string `mapstructure:"storage_type"`
-	IOPS float64 `mapstructure:"iops"`
+	StorageType      string  `mapstructure:"storage_type"`
+	IOPS             float64 `mapstructure:"iops"`
 }
 
 type dbType struct {

cost/doc.go

@@ -0,0 +1,14 @@
+// Package cost defines structures that represent cloud resources and states in a cloud-agnostic,
+// as well as tool-agnostic way. The hierarchy of structs is as follows:
+//
+// - Component is the lowest level, it describes the cost of a single cloud entity (e.g. storage
+// space or compute time).
+//
+// - Resource is a collection of components and directly correlates to cloud resources (e.g. VM instance).
+//
+// - State is a collection of resources that exist (or are planned to exist) at any given moment
+// across one or multiple cloud providers.
+//
+// - Plan is a difference between two states. It includes the prior (current) state and a planned
+// state and it can be used to retrieve a list of ResourceDiff's.
+package cost

doc.go

@@ -0,0 +1,25 @@
+// Package costestimation provides functionality to estimate the costs of infrastructure based on Terrafom
+// plan files.
+//
+// This package depends on the pricing data located in a MySQL database to work correctly. The following
+// snippet will run all required database migrations and ingest pricing data from AmazonEC2 in eu-west-3 region:
+//
+//      db, err := sql.Open("mysql", "...")
+//      backend := mysql.NewBackend(db)
+//
+//      // Run all database migrations
+//      err = mysql.Migrate(ctx, db, "pricing_migrations")
+//
+//      // Ingest pricing data into the database
+//      ingester := aws.NewIngester("AmazonEC2", "eu-west-3")
+//      err = costestimation.IngestPricing(ctx, backend, ingester)
+//
+// With pricing data in the database, a Terraform plan can be read and estimated:
+//
+//      file, err := os.Open("path/to/tfplan.json")
+//      plan, err := costestimation.EstimateTerraformPlan(ctx, backend, file)
+//
+//      for _, res := range plan.ResourceDifferences() {
+//          fmt.Printf("%s: %s -> %s\n", res.Address, res.PriorCost().String(), res.PlannedCost().String())
+//      }
+package costestimation

mysql/backend.go

@@ -24,5 +24,8 @@ func NewBackend(querier sqlr.Querier) *Backend {
 	}
 }
 
+// Product returns the product.Repository that uses the Backend's querier.
 func (b *Backend) Product() product.Repository { return b.productRepo }
-func (b *Backend) Price() price.Repository     { return b.priceRepo }
+
+// Price returns the price.Repository that uses the Backend's querier.
+func (b *Backend) Price() price.Repository { return b.priceRepo }

mysql/doc.go

@@ -0,0 +1,2 @@
+// Package mysql implements the various domain entity repositories and includes a Backend that groups them.
+package mysql

mysql/price.go

@@ -62,6 +62,7 @@ func newPrice(pwp *price.WithProduct) (*dbPrice, error) {
 	}, nil
 }
 
+// Filter returns all the price.Price that belong to a given product with given product.ID and that matches the price.Filter.
 func (r *PriceRepository) Filter(ctx context.Context, productID product.ID, filter *price.Filter) ([]*price.Price, error) {
 	where := parsePriceFilter(filter, productID)
 	q := fmt.Sprintf(`
@@ -90,6 +91,7 @@ func (r *PriceRepository) Filter(ctx context.Context, productID product.ID, filt
 	return ps, nil
 }
 
+// Upsert updates a price.WithProduct if it exists or inserts it otherwise.
 func (r *PriceRepository) Upsert(ctx context.Context, pwp *price.WithProduct) (price.ID, error) {
 	p, err := newPrice(pwp)
 	if err != nil {
@@ -119,6 +121,7 @@ func (r *PriceRepository) Upsert(ctx context.Context, pwp *price.WithProduct) (p
 	return price.ID(id), nil
 }
 
+// DeleteByProductWithKeep deletes all the prices of the product with given product.ID except the ones in the keep slice.
 func (r *PriceRepository) DeleteByProductWithKeep(ctx context.Context, productID product.ID, keep []price.ID) error {
 	marks := make([]string, 0, len(keep))
 	values := make([]interface{}, 0, len(keep)+1)

mysql/product.go

@@ -61,6 +61,7 @@ func newProduct(p *product.Product) (*dbProduct, error) {
 	}, nil
 }
 
+// Filter returns all the product.Product that match the given product.Filter.
 func (r *ProductRepository) Filter(ctx context.Context, filter *product.Filter) ([]*product.Product, error) {
 	where := parseProductFilter(filter)
 	q := fmt.Sprintf(`
@@ -89,6 +90,7 @@ func (r *ProductRepository) Filter(ctx context.Context, filter *product.Filter)
 	return ps, nil
 }
 
+// FindByVendorAndSKU returns a single product.Product of the given vendor and sku.
 func (r *ProductRepository) FindByVendorAndSKU(ctx context.Context, vendor string, sku string) (*product.Product, error) {
 	q := `
 		SELECT id, provider, sku, service, family, location, attributes
@@ -100,6 +102,7 @@ func (r *ProductRepository) FindByVendorAndSKU(ctx context.Context, vendor strin
 	return scanProduct(row)
 }
 
+// Upsert updates a product.Product if it exists or inserts a new one otherwise.
 func (r *ProductRepository) Upsert(ctx context.Context, prod *product.Product) (product.ID, error) {
 	p, err := newProduct(prod)
 	if err != nil {

query/doc.go

@@ -0,0 +1,4 @@
+// Package query defines agnostic structures used to communicate between the tool layer (e.g. Terraform)
+// and the storage layer (e.g. MySQL). This way they can be visualized as "queries" made by the tool layer
+// to the storage layer, in order to fetch pricing data matching certain parameters.
+package query

terraform/doc.go

@@ -0,0 +1,5 @@
+// Package terraform includes functionality related to reading Terraform plan files. The plan schema is defined
+// here according to the JSON output format described at https://www.terraform.io/docs/internals/json-format.html
+//
+// The found resources are then transformed into query.Resource that can be utilized further.
+package terraform