Datasources instantly reach status "DELETED" after creation

[kciredor]

Creating two datasources (one for observations, one for evaluations) with one CSV file in S3, in a for loop, for multiple datasets.

This results in "random" behaviour, where most of the time some or all of the newly created datasources instantly get the status "DELETED" after creation when looking up their status.

My code:

    for i, _ := range datasets {
        dataset := &datasets[i]

        // Create observations datasource.
        srcName := fmt.Sprintf("DS-OBS-%s", dataset.csvFn)
        srcId := fmt.Sprintf("%s-%s", srcName, core.RandSeq(8))

        input := machinelearning.CreateDataSourceFromS3Input{
            ComputeStatistics: aws.Boolean(true),
            DataSourceID:      aws.String(srcId),
            DataSourceName:    aws.String(srcName),
            DataSpec: &machinelearning.S3DataSpec{
                DataLocationS3:    aws.String("s3://input/" + csvFn),
                DataSchema:        aws.String(dataset.DataSchema),
                DataRearrangement: aws.String("{\"splitting\":{\"percentBegin\":0,\"percentEnd\":70}}"),
            },
        }

        res, err := awsClient.Ml.CreateDataSourceFromS3(&input)

        if err != nil {
            log.Fatalln(err)
        }

        dataset.ObsDataSourceID = *res.DataSourceID

        // Create evaluations datasource.
        srcName = fmt.Sprintf("DS-EVL-%s", csvFn)
        srcId = fmt.Sprintf("%s-%s", srcName, core.RandSeq(8))

        input = machinelearning.CreateDataSourceFromS3Input{
            ComputeStatistics: aws.Boolean(false),
            DataSourceID:      aws.String(srcId),
            DataSourceName:    aws.String(srcName),
            DataSpec: &machinelearning.S3DataSpec{
                DataLocationS3:    aws.String("s3://input/" + csvFn),
                DataSchema:        aws.String(dataset.DataSchema),
                DataRearrangement: aws.String("{\"splitting\":{\"percentBegin\":70,\"percentEnd\":100}}"),
            },
        }

        res, err = awsClient.Ml.CreateDataSourceFromS3(&input)

        if err != nil {
            log.Fatalln(err)
        }

        dataset.EvlDataSourceID = *res.DataSourceID
    }

[lsegal]

@kciredor I'm not sure this would be an issue with the AWS SDK for Go since it seems like you were successfully making these calls and the service successfully initiated the creation, which implies the SDK successfully serialized the requests. The SDK performs no other operation besides the exact operation you called.

As for why the requests are being marked as DELETED immediately after creation, my best guess would be there is some type of data validation happening on the service side causing Machine Learning to remove the data source, but I could be wrong. Also, I'm not sure of all the details of your app so I could be wrong again, but if the 2nd CreateDataSourceFromS3 call relies on the presence of the first, you should note that these calls may be "eventually consistent"; in other words, it's likely that the first call has not always completed in time for the 2nd to begin-- that would certainly explain the "randomness" portion of the report. I would suggest opening a thread on the Amazon Machine Learning forums to ask about your issue, giving the input you're providing to the calls. It's likely they would be able to provide more precise feedback as to what may be happening on the service side.

As a sidenote, and I'm not sure this is relevant but I thought it might be worth pointing out: it looks like you're setting dataset.ObsDataSourceID and dataset.EvlDataSourceID values based on the returned IDs from the successful calls, but you claim to be running this over multiple datasets. Reading through the code, it looks like it will only set the last ID, since you're only storing a single ID from each of these calls. Again, I don't know the details of your app, but that may not be what you want.

[kciredor]

Thanks very much for your swift and kind answer @lsegal - I'll proceed and close this issue.