Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: don't ignore job status update errors #22337

Conversation

nvanbenschoten
Copy link
Member

Fixes #21828.
Fixes #21846.
Fixes #21844.
Fixes #21808.

See discussion in #21802.

We were previously ignoring the errors returned by job operations in
a few places. This meant that we would fail to propagate retryable txn
errors when using the jobs API and we could think a txn succeeded when
it didn't. This change fixes that.

In doing so, it makes schema changes fail when they see errors due to
job status updates. I don't see a compelling reason why the old behavior
was better and I suspect it was allowing subtle inconsistencies.

Release note: None

@nvanbenschoten nvanbenschoten requested review from maddyblue, jordanlewis and a team February 2, 2018 21:00
@cockroach-teamcity
Copy link
Member

This change is Reviewable

pkg/sql/table.go Outdated
@@ -650,7 +651,14 @@ func (p *planner) createSchemaChangeJob(
}
job := p.ExecCfg().JobRegistry.NewJob(jobRecord)
if err := job.WithTxn(p.txn).Created(ctx); err != nil {
return sqlbase.InvalidMutationID, nil
if pgErr, ok := pgerror.GetPGCause(err); ok {
// Ignore error if system.jobs table does not exist. This will be
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure about this. @mjibson were you returning nil before specifically to avoid this issue?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooh, I have a much better solution for you, @nvanbenschoten. Gimme a sec.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember ever dealing with this. If I'm in the blame output it's probably a result of a refactoring that happened.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's now impossible for system.jobs not to exist!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @benesch!

@benesch
Copy link
Contributor

benesch commented Feb 3, 2018

Also, here's a question. Suppose I have the following code:

err := db.Txn(func (ctx context.Context, txn *client.Txn) error {
  if err := txn.Put(ctx, "key", "val"); err != nil {
    return err
  }
  // Don't really care if this succeeds.
  _ := txn.Put(...)
  return nil
})

Is it possible that db.Txn succeeds here?! Because that also seems like a bug. Or at least needs to be more loudly documented; it certainly wasn't obvious to me.

Fixes cockroachdb#21828.
Fixes cockroachdb#21846.
Fixes cockroachdb#21844.
Fixes cockroachdb#21808.

See discussion in cockroachdb#21802.

We were previously ignoring the errors returned by job operations in
a few places. This meant that we would fail to propagate retryable txn
errors when using the jobs API and we could think a txn succeeded when
it didn't. This change fixes that.

In doing so, it makes schema changes fail when they see errors due to
job status updates. I don't see a compelling reason why the old behavior
was better and I suspect it was allowing subtle inconsistencies.

Release note: None
@nvanbenschoten
Copy link
Member Author

Is it possible that db.Txn succeeds here?! Because that also seems like a bug. Or at least needs to be more loudly documented; it certainly wasn't obvious to me.

Yes, this currently is possible. See #22615.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants