feat: added certificate to load-balancer

[alekc]

Fix #310

@jerome-quere the code is ready for testing, however Scaleway's current implementation presents some challenges (especially from integration tests point of view).

When running tests, I was presented with some cryptic 400 bad request message. Rerunning it through the trace mode showed following:

{"name":"test-cert","letsencrypt":{"common_name":"main-domain.com","subject_alternative_name":[]}}
---------------------------------------------------------
[DEBUG] POST https://api.scaleway.com/lb/v1/regions/fr-par/lbs/76ac125b-55b9-4d88-8334-5d381d2f3758/certificates
2020/02/11 21:44:03 [DEBUG] 
--------------- Scaleway SDK RESPONSE 5 : ---------------
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 188
Content-Security-Policy: default-src 'none'; frame-ancestors 'none'
Content-Type: application/json
Date: Tue, 11 Feb 2020 21:44:02 GMT
Server: scaleway_api
Strict-Transport-Security: max-age=63072000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Request-Id: 16938c4a-0daa-4a44-b353-713e0707ebc2

{"details":[{"field":"dns_name","message":"main-domain.com not resolve to your Load Balancer IP 51.159.25.61","reason":"constraint"}],"message":"Invalid parameters","type":"invalid_field"}

Error returned in INFO mode

[DEBUG] GET https://api.scaleway.com/lb/v1/regions/fr-par/lbs/087e832f-ce1a-48a5-aac0-4bad1b2b181b
--- FAIL: TestAccScalewayLbCertificateBeta (10.09s)
    testing.go:569: Step 0 error: errors during apply:
        
        Error: scaleway-sdk-go: http error 400 Bad Request: invalid parameters
        
          on /tmp/tf-test031039100/main.tf line 6:
          (source code not available)
        
        
FAIL

There are several issues:

1. The actual error message is buried, client is presented with invalid parameter/field and the only way to understand on TF level that the issue is DNS related is to parse rawbody property (not really ideal).
2. We do not know in advance our load-balancer's ip, so we cannot have a dns pointing to it and create a certificate. In theory, if client is able to manage our dns with the same tf stack, it would be possible to create a proper chain by using depends_on (LB->domain->cert), but even then, I would be concerned about the race condition (dns is created on the nameserver, but when scw checks for it, it fails due to propagation issues).
3. I can't see a good way to create integration test since it won't accept invalid domain names, and we need valid ones in order to create a cert.
4. I've not tested it yet, but does it mean that we cannot have wildcard letsencrypt certificate?

[Sh4d1]

@alekc a few answers :)
2. You can create an IP without a LB now
4. Not possible since letsencrypt wildcard only works on DNS challenge which is not supported (yet?) on LB

[alekc]

@Sh4d1 I could but all points are still valid.

• We cannot test it through integration test
• There is a possibility of a race condition followed by a negative dns cache

Potentially integration test could be solved by providing a certain domain names for testing purposes only, not the most elegant solution but it would work.

[Sh4d1]

@alekc yep agree. I'll let @jerome-quere answer for the testing 😄

[jerome-quere]

That's a tricky problem. Could the use of xip.io service be useful?

[alekc]

Excellent idea. With xip.io the problem with testing is indeed solved (I'll write some tests hopefully tonight and send for review).

The only thing remaining for now is the obfuscation of the error. We will probably need to document in bold that a user has to have a dns pointing to certificate for it to work (some may be surprised by this). Should I try to parse raw error body and extract the error to show it to end user?

{"details":[{"field":"dns_name","message":"main-domain.com not resolve to your Load Balancer IP 51.159.25.61","reason":"constraint"}],"message":"Invalid parameters","type":"invalid_field"}

we can attempt to decode resulting json with

type AutoGenerated struct {
	Details []struct {
		Field   string `json:"field"`
		Message string `json:"message"`
		Reason  string `json:"reason"`
	} `json:"details"`
	Message string `json:"message"`
	Type    string `json:"type"`
}

and return errors.new(rawError.Details[0].Message) to enduser.

[alekc]

@jerome-quere I believe I found another bug with scaleway api

https://developers.scaleway.com/en/products/lb/api/#put-f21555 states that I shoud be able to change certificate name with update without doing the destruction first. However, when I attempted to do so, I got an unexpected result

PUT /lb/v1/regions/fr-par/certificates/ec0085c1-10c2-4870-9190-ddbb4fe3b796 HTTP/1.1
Host: api.scaleway.com
User-Agent: scaleway-sdk-go/v1.0.0-beta.5+dev (go1.13.7; linux; amd64) terraform-provider/v1.13.0-tftest terraform/0.12.7-sdk
Content-Length: 24
Content-Type: application/json
X-Auth-Token: b0da8138-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Accept-Encoding: gzip

{"name":"test-cert-new"}
---------------------------------------------------------
[DEBUG] PUT https://api.scaleway.com/lb/v1/regions/fr-par/certificates/ec0085c1-10c2-4870-9190-ddbb4fe3b796
2020/02/12 19:01:25 [DEBUG] 
--------------- Scaleway SDK RESPONSE 11 : ---------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 791
Content-Security-Policy: default-src 'none'; frame-ancestors 'none'
Content-Type: application/json
Date: Wed, 12 Feb 2020 19:01:25 GMT
Server: scaleway_api
Strict-Transport-Security: max-age=63072000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Request-Id: 99f2e87e-7afe-4bf4-a596-8b4644d5dc6e

{"id":"ec0085c1-10c2-4870-9190-ddbb4fe3b796","type":"letsencryt","status":"pending","common_name":"51.159.24.222.xip.io","subject_alternative_name":[],"fingerprint":"","not_valid_before":"0001-01-01T00:00:00Z","not_valid_after":"0001-01-01T00:00:00Z","lb":{"id":"67d5dcd8-480e-4bea-aff0-6885ba3ba7e1","name":"test-lb","description":"","status":"ready","instances":[],"organization_id":"df378afc-8280-4a58-b486-1a1247e4798a","ip":[{"id":"07674d5e-db50-48e5-ba36-c71f1240b665","ip_address":"51.159.24.222","organization_id":"df378afc-8280-4a58-b486-1a1247e4798a","lb_id":"67d5dcd8-480e-4bea-aff0-6885ba3ba7e1","reverse":"51-159-24-222.lb.fr-par.scw.cloud","region":"fr-par"}],"tags":[],"frontend_count":0,"backend_count":0,"type":"lb-s","subscriber":null,"region":"fr-par"},"name":"test-cert"}
----------------------------------------------------------
2020/02/12 19:01:25 [DEBUG] creating GET request on https://api.scaleway.com/lb/v1/regions/fr-par/certificates/ec0085c1-10c2-4870-9190-ddbb4fe3b796
2020/02/12 19:01:25 [DEBUG] 
--------------- Scaleway SDK REQUEST 12 : ---------------
GET /lb/v1/regions/fr-par/certificates/ec0085c1-10c2-4870-9190-ddbb4fe3b796 HTTP/1.1
Host: api.scaleway.com
User-Agent: scaleway-sdk-go/v1.0.0-beta.5+dev (go1.13.7; linux; amd64) terraform-provider/v1.13.0-tftest terraform/0.12.7-sdk
X-Auth-Token: b0da8138-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Accept-Encoding: gzip


---------------------------------------------------------
[DEBUG] GET https://api.scaleway.com/lb/v1/regions/fr-par/certificates/ec0085c1-10c2-4870-9190-ddbb4fe3b796
2020/02/12 19:01:25 [DEBUG] 
--------------- Scaleway SDK RESPONSE 12 : ---------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 791
Content-Security-Policy: default-src 'none'; frame-ancestors 'none'
Content-Type: application/json
Date: Wed, 12 Feb 2020 19:01:25 GMT
Server: scaleway_api
Strict-Transport-Security: max-age=63072000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Request-Id: dc500a3d-0812-4744-a1c1-ec508d8a463f

{"id":"ec0085c1-10c2-4870-9190-ddbb4fe3b796","type":"letsencryt","status":"pending","common_name":"51.159.24.222.xip.io","subject_alternative_name":[],"fingerprint":"","not_valid_before":"0001-01-01T00:00:00Z","not_valid_after":"0001-01-01T00:00:00Z","lb":{"id":"67d5dcd8-480e-4bea-aff0-6885ba3ba7e1","name":"test-lb","description":"","status":"ready","instances":[],"organization_id":"df378afc-8280-4a58-b486-1a1247e4798a","ip":[{"id":"07674d5e-db50-48e5-ba36-c71f1240b665","ip_address":"51.159.24.222","organization_id":"df378afc-8280-4a58-b486-1a1247e4798a","lb_id":"67d5dcd8-480e-4bea-aff0-6885ba3ba7e1","reverse":"51-159-24-222.lb.fr-par.scw.cloud","region":"fr-par"}],"tags":[],"frontend_count":0,"backend_count":0,"type":"lb-s","subscriber":null,"region":"fr-par"},"name":"test-cert"}

Notice that I am asking to rename test-cert to test-cert-new, however, the response returns 200 but the name in response code is the old one, and it's confirmed in the next call where I am fetching certificate details.

[alekc]

@jerome-quere could you forward the issue above to the LB team please? once thats fixed it would unlock testing and I can send this off for a review.

[jerome-quere]

The issue has been escalated to the load-balancer team.

[agirot]

fixed. Thx @alekc

[QuentinBrosse]

Hi @alekc,
The lb API seems to be fixed and we will release the v1.14.0 this week.
Can you please update your tests today or tomorrow to be in this next release?

Don't worry, if you are too busy we can include this PR in the v1.15.0. 👍

I will make a full review of the PR today. :)

[alekc]

Yep, I should be able to finish testing and send for a review today.

[QuentinBrosse]

Here is a global review. :)

You also need to add the doc please:

• Create a lb_certificate_beta.html.markdown here
• Link it here

Feel free to ask for help, thanks for your time. 🙏

[alekc]

@QuentinBrosse fixed tests and most of the changes requested, please see above for couple of those which I have replied to.

Also, found another "issue" with testing related to the custom certificate. Api doesn't play very well with self signed certs (tried to issue a 10 year one). So for now I've ended up extracting a chain from scaleway.com, just to have tests passing. I suspect that once we reach end of that certificate the test will begin to fail. Any ideas in regard? potentially we can always extract cert from somewhere (scaleway.com i.e.) when we are running the test. Not ideal but would work (since we are required to have internet access anyway for these tests).

[QuentinBrosse]

I will escalate the issue to the LB team (cc @agirot).

Can you please add the doc of the resource?

[alekc]

@QuentinBrosse docs are done, all points have been solved, branch rebased on latest master and ready to be merged.

[alekc]

Merged docs changes.

[agirot]

@alekc about your test to upload a custom certificate. You need to upload the full certif chain.

https://developers.scaleway.com/en/products/lb/api/#certificate-chain-9183b2
The full PEM-formatted include an entire certificate chain including public key, private key, and optionally certificate authorities.

[alekc]

@agirot Ah! the private key and ca. That was the missing piece of the puzzle. It's all working now, thx (pushed 10y self signed cert and a fix for failing travis build).

[QuentinBrosse]

2 details and it will be OK for me 😅

[QuentinBrosse]

LGTM

Thanks again for your great work @alekc 👏 !