Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(NODE-6391): Add timeoutMS support to explicit encryption #4269

Merged
merged 70 commits into from
Oct 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
8003163
feat(NODE-6090): Implement CSOT logic for connection checkout and ser…
W-A-James Apr 11, 2024
a216ae6
test(NODE-6120): Implement Unified test runner changes for CSOT (#4121)
W-A-James Jun 10, 2024
aca9661
refactor(NODE-6187): refactor to use TimeoutContext abstraction (#4131)
W-A-James Jun 21, 2024
3051def
refactor(NODE-6230): executeOperation to use iterative retry mechanis…
nbbeeken Jul 22, 2024
df025f4
feat(NODE-5682): set maxTimeMS on commands and preempt I/O (#4174)
nbbeeken Jul 26, 2024
83cd82b
feat(NODE-6231): Add CSOT behaviour for retryable reads and writes (#…
W-A-James Aug 1, 2024
c36dce5
feat(NODE-6312): add error transformation for server timeouts (#4192)
nbbeeken Aug 12, 2024
3fe3e01
feat(NODE-6313): add CSOT support to sessions and transactions (#4199)
nbbeeken Sep 9, 2024
7b4aa84
feat(NODE-6304): add CSOT support for non-tailable cursors (#4195)
W-A-James Sep 12, 2024
3045a34
fix(NODE-6374): MongoOperationTimeoutError inherits MongoRuntimeError…
nbbeeken Sep 12, 2024
11d059f
test: remove empty skipped context blocks (#4238)
W-A-James Sep 12, 2024
bfeeda9
feat(NODE-5844): add iscryptd to ServerDescription (#4239)
nbbeeken Sep 17, 2024
7a12914
chore: allow clientBulkWrite to use TimeoutContext (#4251)
W-A-James Sep 25, 2024
09f6d7d
feat(NODE-6274): add CSOT support to bulkWrite (#4250)
nbbeeken Oct 2, 2024
1a06868
feat(NODE-6275): Add CSOT support to GridFS (#4246)
W-A-James Oct 4, 2024
392599c
refactor(NODE-6411): AbstractCursor accepts an external timeout conte…
baileympearson Oct 4, 2024
9a1b2d0
feat(NODE-6305): Add CSOT support to tailable cursors (#4218)
W-A-James Oct 7, 2024
d26a588
feat(NODE-6389): add support for timeoutMS in StateMachine.execute() …
aditi-khare-mongoDB Oct 7, 2024
2206be1
src code change no tests
aditi-khare-mongoDB Oct 8, 2024
6330fd6
feat(NODE-6090): Implement CSOT logic for connection checkout and ser…
W-A-James Apr 11, 2024
a1206a0
test(NODE-6120): Implement Unified test runner changes for CSOT (#4121)
W-A-James Jun 10, 2024
a47e280
refactor(NODE-6187): refactor to use TimeoutContext abstraction (#4131)
W-A-James Jun 21, 2024
398066e
refactor(NODE-6230): executeOperation to use iterative retry mechanis…
nbbeeken Jul 22, 2024
c333723
feat(NODE-5682): set maxTimeMS on commands and preempt I/O (#4174)
nbbeeken Jul 26, 2024
256ca4e
feat(NODE-6231): Add CSOT behaviour for retryable reads and writes (#…
W-A-James Aug 1, 2024
8a416be
feat(NODE-6312): add error transformation for server timeouts (#4192)
nbbeeken Aug 12, 2024
52c2c9d
feat(NODE-6313): add CSOT support to sessions and transactions (#4199)
nbbeeken Sep 9, 2024
546366f
feat(NODE-6304): add CSOT support for non-tailable cursors (#4195)
W-A-James Sep 12, 2024
4f8e7c9
fix(NODE-6374): MongoOperationTimeoutError inherits MongoRuntimeError…
nbbeeken Sep 12, 2024
8b9eeef
test: remove empty skipped context blocks (#4238)
W-A-James Sep 12, 2024
1eb0b74
feat(NODE-5844): add iscryptd to ServerDescription (#4239)
nbbeeken Sep 17, 2024
580130d
chore: allow clientBulkWrite to use TimeoutContext (#4251)
W-A-James Sep 25, 2024
2e93ce7
feat(NODE-6274): add CSOT support to bulkWrite (#4250)
nbbeeken Oct 2, 2024
c637ea8
feat(NODE-6275): Add CSOT support to GridFS (#4246)
W-A-James Oct 4, 2024
c148f6b
refactor(NODE-6411): AbstractCursor accepts an external timeout conte…
baileympearson Oct 4, 2024
4488bab
feat(NODE-6305): Add CSOT support to tailable cursors (#4218)
W-A-James Oct 7, 2024
c28a365
feat(NODE-6389): add support for timeoutMS in StateMachine.execute() …
aditi-khare-mongoDB Oct 7, 2024
85d39ec
fix(NODE-6412): read stale response from previously timed out connect…
nbbeeken Oct 11, 2024
450b163
feat(NODE-6403): add CSOT support to client bulk write (#4261)
baileympearson Oct 14, 2024
35ee04c
test 1
aditi-khare-mongoDB Oct 15, 2024
7ee1fd2
tests implemented
aditi-khare-mongoDB Oct 15, 2024
dfe72c1
Merge branch 'NODE-6090' into NODE-6391/explicit-encryption
aditi-khare-mongoDB Oct 15, 2024
56c63c7
temp
aditi-khare-mongoDB Oct 15, 2024
fff7e0a
temp
aditi-khare-mongoDB Oct 15, 2024
751ecd1
temp
aditi-khare-mongoDB Oct 16, 2024
ea2089a
temp
aditi-khare-mongoDB Oct 17, 2024
fa05342
temp
aditi-khare-mongoDB Oct 17, 2024
cbb2a56
temp
aditi-khare-mongoDB Oct 17, 2024
313eaa0
feat(NODE-6403): add CSOT support to client bulk write (#4261)
baileympearson Oct 14, 2024
07cffc7
chore: fix a few flaky CSOT tests (#4278)
baileympearson Oct 17, 2024
c3f31da
feat(NODE-6421): add support for timeoutMS to explain helpers (#4268)
baileympearson Oct 21, 2024
95dd2a2
ready for review
aditi-khare-mongoDB Oct 21, 2024
cf606a0
Merge branch 'NODE-6090' into NODE-6391/explicit-encryption
aditi-khare-mongoDB Oct 21, 2024
3abd62b
remove extranous changes
aditi-khare-mongoDB Oct 21, 2024
c424c80
add back in tests from rebase
aditi-khare-mongoDB Oct 21, 2024
e78b127
Merge branch 'NODE-6090' into NODE-6391/explicit-encryption
aditi-khare-mongoDB Oct 21, 2024
19c314f
partial re-review
aditi-khare-mongoDB Oct 24, 2024
3587432
requested changes
aditi-khare-mongoDB Oct 24, 2024
abe248f
no concurrent timeoutContext
aditi-khare-mongoDB Oct 24, 2024
3fbcd2e
add in comments
aditi-khare-mongoDB Oct 24, 2024
ad8970a
typo
aditi-khare-mongoDB Oct 24, 2024
9201a08
Update src/client-side-encryption/client_encryption.ts
aditi-khare-mongoDB Oct 24, 2024
8287029
Update src/timeout.ts
aditi-khare-mongoDB Oct 24, 2024
cab26a2
bailey requested changes
aditi-khare-mongoDB Oct 28, 2024
cddad21
bailey requested changes
aditi-khare-mongoDB Oct 28, 2024
2fb01bc
fix merge conflict
aditi-khare-mongoDB Oct 28, 2024
171c766
Merge branch 'NODE-6090' into NODE-6391/explicit-encryption
aditi-khare-mongoDB Oct 28, 2024
7185be8
fixed failing tests
aditi-khare-mongoDB Oct 28, 2024
c777bb3
Merge branch 'NODE-6090' into NODE-6391/explicit-encryption
aditi-khare-mongoDB Oct 29, 2024
12333d1
lint fix
aditi-khare-mongoDB Oct 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 46 additions & 9 deletions src/client-side-encryption/client_encryption.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import { type MongoClient, type MongoClientOptions } from '../mongo_client';
import { type Filter, type WithId } from '../mongo_types';
import { type CreateCollectionOptions } from '../operations/create_collection';
import { type DeleteResult } from '../operations/delete';
import { TimeoutContext } from '../timeout';
import { type CSOTTimeoutContext, TimeoutContext } from '../timeout';
import { MongoDBCollectionNamespace, resolveTimeoutOptions } from '../utils';
import * as cryptoCallbacks from './crypto_callbacks';
import {
Expand Down Expand Up @@ -220,7 +220,13 @@ export class ClientEncryption {
socketOptions: autoSelectSocketOptions(this._client.s.options)
});

const dataKey = deserialize(await stateMachine.execute(this, context)) as DataKey;
const timeoutContext =
options?.timeoutContext ??
TimeoutContext.create(resolveTimeoutOptions(this._client, { timeoutMS: this._timeoutMS }));

const dataKey = deserialize(
await stateMachine.execute(this, context, timeoutContext)
) as DataKey;

const { db: dbName, collection: collectionName } = MongoDBCollectionNamespace.fromString(
this._keyVaultNamespace
Expand All @@ -229,7 +235,12 @@ export class ClientEncryption {
const { insertedId } = await this._keyVaultClient
.db(dbName)
.collection<DataKey>(collectionName)
.insertOne(dataKey, { writeConcern: { w: 'majority' } });
.insertOne(dataKey, {
writeConcern: { w: 'majority' },
timeoutMS: timeoutContext?.csotEnabled()
? timeoutContext?.getRemainingTimeMSOrThrow()
: undefined
});

return insertedId;
}
Expand Down Expand Up @@ -511,6 +522,7 @@ export class ClientEncryption {
}
}
];

const value = await this._keyVaultClient
.db(dbName)
.collection<DataKey>(collectionName)
Expand Down Expand Up @@ -555,16 +567,25 @@ export class ClientEncryption {
}
} = options;

const timeoutContext =
this._timeoutMS != null
? TimeoutContext.create(resolveTimeoutOptions(this._client, { timeoutMS: this._timeoutMS }))
: undefined;

if (Array.isArray(encryptedFields.fields)) {
const createDataKeyPromises = encryptedFields.fields.map(async field =>
field == null || typeof field !== 'object' || field.keyId != null
? field
: {
...field,
keyId: await this.createDataKey(provider, { masterKey })
keyId: await this.createDataKey(provider, {
masterKey,
// clone the timeoutContext
// in order to avoid sharing the same timeout for server selection and connection checkout across different concurrent operations
timeoutContext: timeoutContext?.csotEnabled() ? timeoutContext?.clone() : undefined
})
}
);

const createDataKeyResolutions = await Promise.allSettled(createDataKeyPromises);

encryptedFields.fields = createDataKeyResolutions.map((resolution, index) =>
Expand All @@ -582,7 +603,10 @@ export class ClientEncryption {
try {
const collection = await db.createCollection<TSchema>(name, {
...createCollectionOptions,
encryptedFields
encryptedFields,
timeoutMS: timeoutContext?.csotEnabled()
? timeoutContext?.getRemainingTimeMSOrThrow()
: undefined
});
return { collection, encryptedFields };
} catch (cause) {
Expand Down Expand Up @@ -667,7 +691,12 @@ export class ClientEncryption {
socketOptions: autoSelectSocketOptions(this._client.s.options)
});

const { v } = deserialize(await stateMachine.execute(this, context));
const timeoutContext =
this._timeoutMS != null
? TimeoutContext.create(resolveTimeoutOptions(this._client, { timeoutMS: this._timeoutMS }))
: undefined;

const { v } = deserialize(await stateMachine.execute(this, context, timeoutContext));

return v;
}
Expand Down Expand Up @@ -747,7 +776,11 @@ export class ClientEncryption {
});
const context = this._mongoCrypt.makeExplicitEncryptionContext(valueBuffer, contextOptions);

const { v } = deserialize(await stateMachine.execute(this, context));
const timeoutContext =
this._timeoutMS != null
? TimeoutContext.create(resolveTimeoutOptions(this._client, { timeoutMS: this._timeoutMS }))
: undefined;
const { v } = deserialize(await stateMachine.execute(this, context, timeoutContext));
return v;
}
}
Expand Down Expand Up @@ -833,7 +866,8 @@ export interface ClientEncryptionOptions {
*/
tlsOptions?: CSFLEKMSTlsOptions;

/**
/** @internal TODO(NODE-5688): make this public
*
* The timeout setting to be used for all the operations on ClientEncryption.
*/
timeoutMS?: number;
Expand Down Expand Up @@ -965,6 +999,9 @@ export interface ClientEncryptionCreateDataKeyProviderOptions {

/** @experimental */
keyMaterial?: Buffer | Binary;

/** @internal */
timeoutContext?: CSOTTimeoutContext;
}

/**
Expand Down
14 changes: 14 additions & 0 deletions src/timeout.ts
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,20 @@ export class CSOTTimeoutContext extends TimeoutContext {
return remainingTimeMS;
}

/**
* @internal
* This method is intended to be used in situations where concurrent operation are on the same deadline, but cannot share a single `TimeoutContext` instance.
* Returns a new instance of `CSOTTimeoutContext` constructed with identical options, but setting the `start` property to `this.start`.
*/
clone(): CSOTTimeoutContext {
W-A-James marked this conversation as resolved.
Show resolved Hide resolved
const timeoutContext = new CSOTTimeoutContext({
timeoutMS: this.timeoutMS,
serverSelectionTimeoutMS: this.serverSelectionTimeoutMS
});
timeoutContext.start = this.start;
return timeoutContext;
}

override refreshed(): CSOTTimeoutContext {
return new CSOTTimeoutContext(this);
}
Expand Down
163 changes: 163 additions & 0 deletions test/integration/client-side-encryption/driver.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ import {
Connection,
CSOTTimeoutContext,
type MongoClient,
MongoCryptCreateDataKeyError,
MongoCryptCreateEncryptedCollectionError,
MongoOperationTimeoutError,
StateMachine
} from '../../mongodb';
Expand Down Expand Up @@ -1050,4 +1052,165 @@ describe('CSOT', function () {
);
});
});

describe('Explicit Encryption', function () {
describe('#createEncryptedCollection', function () {
let client: MongoClient;
let clientEncryption: ClientEncryption;
let local_key;
const timeoutMS = 1000;

const encryptedCollectionMetadata: MongoDBMetadataUI = {
requires: {
clientSideEncryption: true,
mongodb: '>=7.0.0',
topology: '!single'
}
};

beforeEach(async function () {
local_key = { local: EJSON.parse(process.env.CSFLE_KMS_PROVIDERS).local };
client = this.configuration.newClient({ timeoutMS });
await client.connect();
await client.db('keyvault').createCollection('datakeys');
clientEncryption = new ClientEncryption(client, {
keyVaultNamespace: 'keyvault.datakeys',
keyVaultClient: client,
W-A-James marked this conversation as resolved.
Show resolved Hide resolved
kmsProviders: local_key
});
});

afterEach(async function () {
await client
.db()
.admin()
.command({
configureFailPoint: 'failCommand',
mode: 'off'
} as FailPoint);
await client
.db('db')
.collection('newnew')
.drop()
.catch(() => null);
await client
.db('keyvault')
.collection('datakeys')
.drop()
.catch(() => null);
await client.close();
});

async function runCreateEncryptedCollection() {
const createCollectionOptions = {
encryptedFields: { fields: [{ path: 'ssn', bsonType: 'string', keyId: null }] }
};

const db = client.db('db');

return await measureDuration(() =>
clientEncryption
.createEncryptedCollection(db, 'newnew', {
provider: 'local',
createCollectionOptions,
masterKey: null
})
.catch(err => err)
);
}

context(
'when `createDataKey` hangs longer than timeoutMS and `createCollection` does not hang',
() => {
it(
'`createEncryptedCollection throws `MongoCryptCreateDataKeyError` due to a timeout error',
encryptedCollectionMetadata,
async function () {
await client
.db()
.admin()
.command({
configureFailPoint: 'failCommand',
mode: {
times: 1
},
data: {
failCommands: ['insert'],
blockConnection: true,
blockTimeMS: timeoutMS * 1.2
}
} as FailPoint);

const { duration, result: err } = await runCreateEncryptedCollection();
expect(err).to.be.instanceOf(MongoCryptCreateDataKeyError);
expect(err.cause).to.be.instanceOf(MongoOperationTimeoutError);
expect(duration).to.be.within(timeoutMS - 100, timeoutMS + 100);
}
);
}
);

context(
'when `createDataKey` does not hang and `createCollection` hangs longer than timeoutMS',
() => {
it(
'`createEncryptedCollection throws `MongoCryptCreateEncryptedCollectionError` due to a timeout error',
encryptedCollectionMetadata,
async function () {
await client
.db()
.admin()
.command({
configureFailPoint: 'failCommand',
mode: {
times: 1
},
data: {
failCommands: ['create'],
blockConnection: true,
blockTimeMS: timeoutMS * 1.2
}
} as FailPoint);

const { duration, result: err } = await runCreateEncryptedCollection();
expect(err).to.be.instanceOf(MongoCryptCreateEncryptedCollectionError);
expect(err.cause).to.be.instanceOf(MongoOperationTimeoutError);
expect(duration).to.be.within(timeoutMS - 100, timeoutMS + 100);
}
);
}
);

context(
'when `createDataKey` and `createCollection` cumulatively hang longer than timeoutMS',
() => {
it(
'`createEncryptedCollection throws `MongoCryptCreateEncryptedCollectionError` due to a timeout error',
encryptedCollectionMetadata,
async function () {
await client
.db()
.admin()
.command({
configureFailPoint: 'failCommand',
mode: {
times: 2
},
data: {
failCommands: ['insert', 'create'],
blockConnection: true,
blockTimeMS: timeoutMS * 0.6
}
} as FailPoint);

const { duration, result: err } = await runCreateEncryptedCollection();
expect(err).to.be.instanceOf(MongoCryptCreateEncryptedCollectionError);
expect(err.cause).to.be.instanceOf(MongoOperationTimeoutError);
expect(duration).to.be.within(timeoutMS - 100, timeoutMS + 100);
}
);
}
);
});
});
});
Loading