New resource: azurerm_stream_analytics_reference_input_blob #3633

praneetloke · 2019-06-11T00:35:57Z

I noticed that it was not possible to create inputs for a stream analytics job of type reference. The code to represent such a resource already existed, but just wasn't exposed as a resource to be able to use it.

I also added an example, that I tested using my own subscription. Here's a screenshot of how the "reference" input looks.

praneetloke · 2019-06-11T00:37:32Z

azurerm/resource_arm_stream_analytics_reference_input_blob.go

@@ -0,0 +1,243 @@
+package azurerm


This code is pretty much an exact replica of the ..._stream_analytics_**stream**_input_blob.go.

azurerm/resource_arm_stream_analytics_reference_input_blob.go

praneetloke · 2019-06-11T00:40:05Z

azurerm/resource_arm_stream_analytics_reference_input_blob.go

+	props := streamanalytics.Input{
+		Name: utils.String(name),
+		Properties: &streamanalytics.ReferenceInputProperties{
+			Type: streamanalytics.TypeReference,


This value is what sets it distinguishes a reference input from a streaming input to the stream analytics job.

azurerm/resource_arm_stream_analytics_reference_input_blob_test.go

examples/stream-analytics/main.tf

praneetloke · 2019-06-11T00:42:09Z

website/docs/r/stream_analytics_stream_input_blob.html.markdown

@@ -37,8 +37,8 @@ resource "azurerm_storage_container" "example" {
  container_access_type = "private"
 }

-resource "azurerm_stream_analytics_stream_input_eventhub" "test" {
-  name                         = "eventhub-stream-input"
+resource "azurerm_stream_analytics_stream_input_blob" "test" {


Noticed the typos while reading the docs.

praneetloke · 2019-06-19T15:38:35Z

Ping. Anyone?

katbyte · 2019-09-14T16:43:04Z

As mentioned above, Would it just make sense to add this ability to the to the other steam input blob resource?

praneetloke · 2019-09-14T23:40:20Z

@katbyte I am going to take another look at this and get back to you. As you can see it has been a while since I opened this PR and have to refresh my memory a bit :)

praneetloke · 2019-09-15T16:59:43Z

As mentioned above, Would it just make sense to add this ability to the to the other steam input blob resource?

Ok I looked at this again. I do believe this needs to be its own resource. Unfortunately, there is some overlap in how this resource is parsed vs. how the stream input is parsed. But there are two places where we must explicitly use the reference input type in the *CreateUdpate and the *Read functions. Maybe I don't know enough about adding a new resource that I can't see how this could be a part of the existing streaming input resource.

Just to be clear, it is important to note that the difference between the two types of input is that one if a streaming input vs. what is being added with this PR is a reference input. A reference input is a static source of data for a Stream Analytics job, whereas a streaming input is something that is continuously producing data.

Please let me know if this helps. I am happy to rework this if you have other ideas. I think for a Stream Analytics job, being able to add reference data inputs is an important feature.

praneetloke · 2019-09-27T05:06:31Z

ping @katbyte

katbyte

Thanks for answering my questions, i've given this a review @praneetloke. Nothing major bt i have left some comments inline that should be addressed before merge

azurerm/resource_arm_stream_analytics_reference_input_blob.go

katbyte · 2019-09-16T16:47:34Z

website/docs/r/stream_analytics_reference_input_blob.html.markdown

+Stream Analytics Reference Input Blob's can be imported using the `resource id`, e.g.
+
+```shell
+terraform import azurerm_stream_analytics_stream_input_blob.test /subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/group1/providers/Microsoft.StreamAnalytics/streamingjobs/job1/inputs/input1


I think you'll need to update this line

katbyte · 2019-10-10T02:48:46Z

azurerm/resource_arm_stream_analytics_reference_input_blob.go

+			"path_pattern": {
+				Type:     schema.TypeString,
+				Required: true,
+			},


Could we get a validate.NoEmptyStrings here?

Just curious, though. With Required: true, isn't NoEmptyStrings enforced automatically?

Nope, you can pass an empty string, alls that is required is something is assinged to the property

Ah gotcha. Would you like me to add validate.NoEmptyStrings here then?

katbyte · 2019-10-10T02:50:25Z

azurerm/resource_arm_stream_analytics_reference_input_blob.go

+	if d.IsNewResource() {
+		if _, err := client.CreateOrReplace(ctx, props, resourceGroup, jobName, name, "", ""); err != nil {
+			return fmt.Errorf("Error Creating Stream Analytics Reference Input Blob %q (Job %q / Resource Group %q): %+v", name, jobName, resourceGroup, err)
+		}
+
+		read, err := client.Get(ctx, resourceGroup, jobName, name)
+		if err != nil {
+			return fmt.Errorf("Error retrieving Stream Analytics Reference Input Blob %q (Job %q / Resource Group %q): %+v", name, jobName, resourceGroup, err)
+		}
+		if read.ID == nil {
+			return fmt.Errorf("Cannot read ID of Stream Analytics Reference Input Blob %q (Job %q / Resource Group %q)", name, jobName, resourceGroup)
+		}
+
+		d.SetId(*read.ID)
+	} else {
+		if _, err := client.Update(ctx, props, resourceGroup, jobName, name, ""); err != nil {
+			return fmt.Errorf("Error Updating Stream Analytics Reference Input Blob %q (Job %q / Resource Group %q): %+v", name, jobName, resourceGroup, err)
+		}
+	}


Given there is vastly different logic for create & update could we maybe split this function into a create and update?

examples/stream-analytics/main.tf

examples/stream-analytics/variables.tf

website/docs/r/stream_analytics_reference_input_blob.html.markdown

katbyte · 2019-10-10T02:55:18Z

website/docs/r/stream_analytics_reference_input_blob.html.markdown

+
+* `path_pattern` - (Required) The blob path pattern. Not a regular expression. It represents a pattern against which blob names will be matched to determine whether or not they should be included as input or output to the job.
+
+* `storage_account_name` - (Required) The name of the Storage Account.


The name of the strage account to.. ?

danielrbradley · 2019-12-03T11:49:56Z

Hello, I'm personally very interested in getting Reference sources working for a project. I've read through the conversation so far and can't quite work out what the current blocker would be (beyond fixing merge conflicts). I'd be happy to try and lend a hand if I can to get this over the line and merged if you can summarise the outstanding issues that need to be resolved.

praneetloke · 2019-12-03T17:03:41Z

@danielrbradley

beyond fixing merge conflicts

I believe this and also getting the test to pass. I haven't kept my fork up-to-date with the latest from the upstream master. I'll get this restored today.

danielrbradley · 2019-12-11T13:53:04Z

@praneetloke @katbyte I've rebased and got the build working on this second PR: #5129

Quite happy @praneetloke if you'd like to pull my changes and force push up to your original branch to continue the conversation here.

…put blob datasource. Add an example for stream analytics. Add the new datasource to provider.go.

Co-Authored-By: kt <[email protected]>

praneetloke · 2019-12-12T17:49:22Z

@danielrbradley I am taking a look at restoring this PR right now. Thanks for opening yours. I'll check that too. I would prefer to fix up this PR rather than opening a new one, since the comments in this PR would be easier to follow-up on.

EDIT: Your PR was helpful in getting things all sorted out. Thank you for that!

… and other compilation errors.

praneetloke · 2019-12-12T18:48:07Z

@katbyte @tombuildsstuff I have addressed the comments and fixed-up this PR with the latest from upstream (this repo). Can you please take another look at this?

praneetloke · 2019-12-12T19:06:53Z

Almost all of the checks passed, except for the lintstatic target, which seems to have failed with a transient error:

# cd .; git clone -- https://github.com/client9/misspell /home/travis/gopath/src/github.com/client9/misspell
Cloning into '/home/travis/gopath/src/github.com/client9/misspell'...
fatal: unable to access 'https://github.com/client9/misspell/': Failed to connect to github.com port 443: Connection timed out
package github.com/client9/misspell/cmd/misspell: exit status 128
GNUmakefile:13: recipe for target 'tools' failed

I don't have access to restart the build. Could you please restart just this job?

EDIT: Never mind. I made a small update to trigger another build. I didn't want to do that initially but due to the lack of response, I wanted to make sure if someone does open this PR to merge it, that it is ready to go.

katbyte

Thanks for finishing this up @praneetloke! LGTM now

ghost · 2020-01-08T10:37:50Z

This has been released in version 1.40.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.40.0"
}
# ... other configuration ...

ghost · 2020-03-28T15:33:14Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

ghost added size/XL documentation labels Jun 11, 2019

praneetloke commented Jun 11, 2019

View reviewed changes

azurerm/resource_arm_stream_analytics_reference_input_blob.go Show resolved Hide resolved

praneetloke commented Jun 11, 2019

View reviewed changes

azurerm/resource_arm_stream_analytics_reference_input_blob_test.go Show resolved Hide resolved

praneetloke commented Jun 11, 2019

View reviewed changes

examples/stream-analytics/main.tf Show resolved Hide resolved

praneetloke commented Jun 11, 2019

View reviewed changes

praneetloke changed the title ~~Add blob reference input as a new resource for stream analytics input resources~~ New resource: Add blob reference input as a new resource for stream analytics input resources Jun 17, 2019

katbyte added service/stream-analytics waiting-response labels Sep 14, 2019

ghost removed the waiting-response label Sep 14, 2019

katbyte requested changes Oct 10, 2019

View reviewed changes

katbyte added the waiting-response label Oct 10, 2019

praneetloke requested a review from katbyte October 10, 2019 16:14

tombuildsstuff assigned katbyte Oct 31, 2019

ghost removed the waiting-response label Dec 3, 2019

danielrbradley mentioned this pull request Dec 11, 2019

New resource: Add blob reference input as a new resource for stream analytics input resources (rebased) #5129

Closed

praneetloke and others added 4 commits December 12, 2019 09:46

Add reference input blob as a resource for Stream Analytics

d2e85bb

Fix a typo in the stream input blob example. Add doc for reference in…

386b010

…put blob datasource. Add an example for stream analytics. Add the new datasource to provider.go.

Fix errors

8dbc721

Apply suggestions from code review

df25b25

Co-Authored-By: kt <[email protected]>

praneetloke and others added 4 commits December 12, 2019 09:47

Address some minor PR feedback.

3521c4f

Apply suggestions from code review

52cdd44

Co-Authored-By: kt <[email protected]>

Split the resource create and update functions.

a401cda

Add a new function to get input props.

0286c13

praneetloke added 3 commits December 12, 2019 09:54

Add a subcategory to the stream analytics reference input doc.

8fc1a2e

Add support for custom timeouts in the new reference input resource.

705bd89

Fix imports in the new stream analytics reference input blob resource…

b11fcb5

… and other compilation errors.

praneetloke and others added 3 commits December 15, 2019 17:24

Minor update to trigger a new build

effc9bc

Update stream_analytics_stream_input_blob.html.markdown

90e4ed2

Update stream_analytics_reference_input_blob.html.markdown

9d8bb04

katbyte approved these changes Dec 17, 2019

View reviewed changes

katbyte added this to the v1.40.0 milestone Dec 17, 2019

katbyte changed the title ~~New resource: Add blob reference input as a new resource for stream analytics input resources~~ New resource: azurerm_stream_analytics_reference_input_blob Dec 17, 2019

katbyte merged commit 8a5813a into hashicorp:master Dec 17, 2019

katbyte added a commit that referenced this pull request Dec 17, 2019

Update CHANGELOG.md to include #3633

acc0b60

ghost locked and limited conversation to collaborators Mar 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New resource: azurerm_stream_analytics_reference_input_blob #3633

New resource: azurerm_stream_analytics_reference_input_blob #3633

praneetloke commented Jun 11, 2019

praneetloke Jun 11, 2019

praneetloke Jun 11, 2019

praneetloke Jun 11, 2019

praneetloke commented Jun 19, 2019

katbyte commented Sep 14, 2019

praneetloke commented Sep 14, 2019

praneetloke commented Sep 15, 2019

praneetloke commented Sep 27, 2019

katbyte left a comment

katbyte Sep 16, 2019

katbyte Oct 10, 2019

praneetloke Oct 10, 2019

praneetloke Oct 10, 2019

katbyte Dec 17, 2019

praneetloke Dec 17, 2019

katbyte Oct 10, 2019

katbyte Oct 10, 2019

danielrbradley commented Dec 3, 2019

praneetloke commented Dec 3, 2019

danielrbradley commented Dec 11, 2019

praneetloke commented Dec 12, 2019 •

edited

Loading

praneetloke commented Dec 12, 2019

praneetloke commented Dec 12, 2019 •

edited

Loading

katbyte left a comment

ghost commented Jan 8, 2020

ghost commented Mar 28, 2020


		* `path_pattern` - (Required) The blob path pattern. Not a regular expression. It represents a pattern against which blob names will be matched to determine whether or not they should be included as input or output to the job.

		* `storage_account_name` - (Required) The name of the Storage Account.

New resource: azurerm_stream_analytics_reference_input_blob #3633

New resource: azurerm_stream_analytics_reference_input_blob #3633

Conversation

praneetloke commented Jun 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

praneetloke commented Jun 19, 2019

katbyte commented Sep 14, 2019

praneetloke commented Sep 14, 2019

praneetloke commented Sep 15, 2019

praneetloke commented Sep 27, 2019

katbyte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielrbradley commented Dec 3, 2019

praneetloke commented Dec 3, 2019

danielrbradley commented Dec 11, 2019

praneetloke commented Dec 12, 2019 • edited Loading

praneetloke commented Dec 12, 2019

praneetloke commented Dec 12, 2019 • edited Loading

katbyte left a comment

Choose a reason for hiding this comment

ghost commented Jan 8, 2020

ghost commented Mar 28, 2020

praneetloke commented Dec 12, 2019 •

edited

Loading

praneetloke commented Dec 12, 2019 •

edited

Loading