Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy Saved Objects to Spaces API #38014

Merged
merged 84 commits into from
Aug 21, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
0051703
POC for copy-to-space
legrego Jun 4, 2019
d6d18b0
fix typings
legrego Jun 5, 2019
a3109ce
update import/export tests
legrego Jun 5, 2019
9590fd2
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Jun 5, 2019
48cbd59
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Jul 9, 2019
6054610
fix merge from master
legrego Jul 9, 2019
9278c15
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Jul 15, 2019
3135f8b
reintroduce spaces SOC namespace check
legrego Jul 15, 2019
5b041e1
export new SOC typedefs
legrego Jul 15, 2019
53ae63e
fix import/export module resolution
legrego Jul 15, 2019
445c0f3
use new excludedWrappers option
legrego Jul 15, 2019
505b1f7
rename ImportError -> SavedObjectsImportError for exportability
legrego Jul 22, 2019
05b79ba
cleanup
legrego Jul 22, 2019
deb66b5
allow resolve_import_errors to accept an optional namespace
legrego Jul 22, 2019
357b6a8
additional core changes; allowing export to accept a namespace
legrego Jul 24, 2019
6e32fc3
api testing and security fixes
legrego Jul 24, 2019
cc459f9
implementing/testing the conflict resolution api
legrego Jul 25, 2019
0726f02
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Jul 25, 2019
934cc63
doc updates following merge
legrego Jul 25, 2019
ac64e6f
jest api tests
legrego Jul 30, 2019
32944cb
API docs
legrego Jul 30, 2019
760b373
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Jul 30, 2019
4cd5e15
fix typings
legrego Jul 30, 2019
b07722a
update snapshots
legrego Jul 30, 2019
7b74d0b
extract createEmptyFailureResponse
legrego Jul 31, 2019
cfad9d6
retrieve spaces user can 'copy to space' with
legrego Jul 31, 2019
c65228f
ts fixes
legrego Jul 31, 2019
1d484dc
make conflict resolution API public, with docs
legrego Aug 1, 2019
e7675f2
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 1, 2019
d8f7633
remove unused saved object action
legrego Aug 1, 2019
6e74bb0
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 1, 2019
174e3e4
fix merge
legrego Aug 1, 2019
2eab9c3
revert unnecessary change
legrego Aug 2, 2019
5d0fbc6
using new api doc format
legrego Aug 2, 2019
4a2c343
simplify conflict api payload
legrego Aug 2, 2019
ef90d0c
fix test
legrego Aug 2, 2019
e100185
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 8, 2019
f30fa33
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 8, 2019
ca9171d
reverting checkPrivilegesDynamically
legrego Aug 8, 2019
2828bde
adjusting API documentation
legrego Aug 8, 2019
90b2dac
removing hard-coded export size
legrego Aug 8, 2019
0f52952
move to FtrProviderContext
legrego Aug 8, 2019
a7ad6f5
additional tests
legrego Aug 13, 2019
a4b43d7
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 13, 2019
5f38425
testing canManageSavedObjects
legrego Aug 13, 2019
f169117
remove unused import
legrego Aug 13, 2019
a4d72e5
Update docs/api/saved-objects/copy_to_space.asciidoc
legrego Aug 14, 2019
e8023f2
removing deepClone from rereadable_stream
legrego Aug 14, 2019
f355861
moving docs
legrego Aug 14, 2019
42d5978
Merge branch 'spaces/copy-to-space-api' of github.com:legrego/kibana …
legrego Aug 14, 2019
79cc29f
read exported SO stream to completion
legrego Aug 14, 2019
7241dd6
reenable tests
legrego Aug 14, 2019
c8cccca
make API endpoints snake_case instead of camelCase
legrego Aug 14, 2019
b4ae5d2
removing explicit auth checks prior to initiating copy operation
legrego Aug 14, 2019
6b53969
improve error handling when thrown from wrapped Saved Objects Client
legrego Aug 15, 2019
6330858
make TS happy
legrego Aug 15, 2019
22ab280
Apply suggestions from code review
legrego Aug 15, 2019
eb12dd6
rename 'get space' purpose from 'copySavedObjects' to 'copySavedObjec…
legrego Aug 16, 2019
943cf95
prefixing RPC routes with an underscore
legrego Aug 16, 2019
5f57c7b
updating functional API tests
legrego Aug 16, 2019
4eca3f9
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 16, 2019
5a749e2
Merge branch 'spaces/copy-to-space-api' of github.com:legrego/kibana …
legrego Aug 16, 2019
3219006
apply doc suggestions
legrego Aug 16, 2019
683b207
move stream creation to utility function
legrego Aug 16, 2019
e5d7682
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 16, 2019
3eb982b
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 19, 2019
857034d
Update docs/api/spaces-management/get.asciidoc
legrego Aug 19, 2019
d8a82e9
cleaning up poorly named test spaces
legrego Aug 19, 2019
be26e69
removing unused error types
legrego Aug 19, 2019
21a5bca
improving spaces_client tests
legrego Aug 19, 2019
4e07b45
Apply suggestions from code review
legrego Aug 19, 2019
3d7c8b2
Merge branch 'spaces/copy-to-space-api' of github.com:legrego/kibana …
legrego Aug 19, 2019
c5d2b9e
uncommenting test suites
legrego Aug 19, 2019
9507ac7
ensure SOC is created without spaces wrapper
legrego Aug 20, 2019
1ee8e01
testing stream read errors
legrego Aug 20, 2019
393f8e6
testing readStreamToCompletion and createReadableStreamFromArray
legrego Aug 20, 2019
23fdb80
verify the correct number of spaces were copied into
legrego Aug 20, 2019
76dbdc0
add missing tests
legrego Aug 20, 2019
61df1e2
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 20, 2019
d913b59
fix test suite name
legrego Aug 20, 2019
0acb332
fixing test reference
legrego Aug 20, 2019
302230c
fix it again
legrego Aug 20, 2019
37ac143
addressing feedback
legrego Aug 21, 2019
5c467ba
Merge branch 'master' of github.com:elastic/kibana into spaces/copy-t…
legrego Aug 21, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ interface ObjectToExport {
type: string;
}

interface ExportObjectsOptions {
export interface ExportObjectsOptions {
types?: string[];
objects?: ObjectToExport[];
savedObjectsClient: SavedObjectsClient;
Expand Down
3 changes: 2 additions & 1 deletion src/legacy/server/saved_objects/export/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,5 @@
* under the License.
*/

export { getSortedObjectsForExport } from './get_sorted_objects_for_export';
export { getSortedObjectsForExport, ExportObjectsOptions } from './get_sorted_objects_for_export';
export { objectsToNdJson } from './objects_to_ndjson';
24 changes: 24 additions & 0 deletions src/legacy/server/saved_objects/export/objects_to_ndjson.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import stringify from 'json-stable-stringify';
import { SavedObject } from '..';

export function objectsToNdJson(savedObjects: SavedObject[]): string {
return savedObjects.map(doc => stringify(doc)).join('\n');
legrego marked this conversation as resolved.
Show resolved Hide resolved
}
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,16 @@ import { extractErrors } from './extract_errors';
import { ImportError } from './types';
import { validateReferences } from './validate_references';

interface ImportSavedObjectsOptions {
export interface ImportSavedObjectsOptions {
readStream: Readable;
objectLimit: number;
overwrite: boolean;
savedObjectsClient: SavedObjectsClient;
supportedTypes: string[];
namespace?: string;
}

interface ImportResponse {
export interface ImportResponse {
success: boolean;
successCount: number;
errors?: ImportError[];
Expand All @@ -44,6 +45,7 @@ export async function importSavedObjects({
overwrite,
savedObjectsClient,
supportedTypes,
namespace,
}: ImportSavedObjectsOptions): Promise<ImportResponse> {
let errorAccumulator: ImportError[] = [];

Expand Down Expand Up @@ -73,6 +75,7 @@ export async function importSavedObjects({
// Create objects in bulk
const bulkCreateResult = await savedObjectsClient.bulkCreate(filteredObjects, {
overwrite,
namespace,
});
errorAccumulator = [
...errorAccumulator,
Expand Down
6 changes: 5 additions & 1 deletion src/legacy/server/saved_objects/import/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,9 @@
* under the License.
*/

export { importSavedObjects } from './import_saved_objects';
export {
importSavedObjects,
ImportSavedObjectsOptions,
ImportResponse,
} from './import_saved_objects';
export { resolveImportErrors } from './resolve_import_errors';
11 changes: 7 additions & 4 deletions src/legacy/server/saved_objects/import/validate_references.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ function filterReferencesToValidate({ type }: { type: string }) {

export async function getNonExistingReferenceAsKeys(
savedObjects: SavedObject[],
savedObjectsClient: SavedObjectsClient
savedObjectsClient: SavedObjectsClient,
namespace?: string
) {
const collector = new Map();
// Collect all references within objects
Expand All @@ -50,7 +51,7 @@ export async function getNonExistingReferenceAsKeys(

// Fetch references to see if they exist
const bulkGetOpts = Array.from(collector.values()).map(obj => ({ ...obj, fields: ['id'] }));
const bulkGetResponse = await savedObjectsClient.bulkGet(bulkGetOpts);
const bulkGetResponse = await savedObjectsClient.bulkGet(bulkGetOpts, { namespace });

// Error handling
const erroredObjects = bulkGetResponse.saved_objects.filter(
Expand All @@ -77,12 +78,14 @@ export async function getNonExistingReferenceAsKeys(

export async function validateReferences(
savedObjects: SavedObject[],
savedObjectsClient: SavedObjectsClient
savedObjectsClient: SavedObjectsClient,
namespace?: string
) {
const errorMap: { [key: string]: ImportError } = {};
const nonExistingReferenceKeys = await getNonExistingReferenceAsKeys(
savedObjects,
savedObjectsClient
savedObjectsClient,
namespace
);

// Filter out objects with missing references, add to error object
Expand Down
5 changes: 2 additions & 3 deletions src/legacy/server/saved_objects/routes/export.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,8 @@

import Hapi from 'hapi';
import Joi from 'joi';
import stringify from 'json-stable-stringify';
import { SavedObjectsClient } from '../';
import { getSortedObjectsForExport } from '../export';
import { getSortedObjectsForExport, objectsToNdJson } from '../export';
import { Prerequisites } from './types';

interface ExportRequest extends Hapi.Request {
Expand Down Expand Up @@ -78,7 +77,7 @@ export const createExportRoute = (
includeReferencesDeep: request.payload.includeReferencesDeep,
});
return h
.response(docsToExport.map(doc => stringify(doc)).join('\n'))
.response(objectsToNdJson(docsToExport))
.header('Content-Disposition', `attachment; filename="export.ndjson"`)
.header('Content-Type', 'application/ndjson');
},
Expand Down
9 changes: 9 additions & 0 deletions src/legacy/server/saved_objects/saved_objects_mixin.js
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ import {
createLogLegacyImportRoute,
} from './routes';

import { getSortedObjectsForExport, objectsToNdJson } from './export';
import { importSavedObjects } from './import';

function getImportableAndExportableTypes({ kbnServer, visibleTypes }) {
const { savedObjectsManagement = {} } = kbnServer.uiExports;
return visibleTypes.filter(
Expand Down Expand Up @@ -141,6 +144,12 @@ export function savedObjectsMixin(kbnServer, server) {
setScopedSavedObjectsClientFactory: (...args) => provider.setClientFactory(...args),
addScopedSavedObjectsClientWrapperFactory: (...args) =>
provider.addClientWrapperFactory(...args),
importExport: {
importSavedObjects,
getSortedObjectsForExport,
objectsToNdJson,
legrego marked this conversation as resolved.
Show resolved Hide resolved
},
schema,
};
server.decorate('server', 'savedObjects', service);

Expand Down
11 changes: 10 additions & 1 deletion src/legacy/server/saved_objects/service/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@
*/

import { ScopedSavedObjectsClientProvider } from './lib';
import { SavedObjectsClient } from './saved_objects_client';
import { SavedObjectsClient, SavedObject } from './saved_objects_client';
import { ExportObjectsOptions } from '../export';
import { ImportSavedObjectsOptions, ImportResponse } from '../import';
import { SavedObjectsSchema } from '../schema';

export interface SavedObjectsService<Request = any> {
// ATTENTION: these types are incomplete
Expand All @@ -28,7 +31,13 @@ export interface SavedObjectsService<Request = any> {
getScopedSavedObjectsClient: ScopedSavedObjectsClientProvider<Request>['getClient'];
SavedObjectsClient: typeof SavedObjectsClient;
types: string[];
schema: SavedObjectsSchema;
getSavedObjectsRepository(...rest: any[]): any;
importExport: {
importSavedObjects(options: ImportSavedObjectsOptions): Promise<ImportResponse>;
getSortedObjectsForExport(options: ExportObjectsOptions): Promise<SavedObject[]>;
objectsToNdJson(objects: SavedObject[]): string;
};
}

export { SavedObjectsClientWrapperFactory } from './lib';
Expand Down
2 changes: 1 addition & 1 deletion x-pack/plugins/security/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ export const security = (kibana) => new kibana.Plugin({
return new savedObjects.SavedObjectsClient(callWithRequestRepository);
});

savedObjects.addScopedSavedObjectsClientWrapperFactory(Number.MIN_SAFE_INTEGER, ({ client, request }) => {
savedObjects.addScopedSavedObjectsClientWrapperFactory(Number.MAX_SAFE_INTEGER - 1, ({ client, request }) => {
legrego marked this conversation as resolved.
Show resolved Hide resolved
if (authorization.mode.useRbacForRequest(request)) {
return new SecureSavedObjectsClientWrapper({
actions: authorization.actions,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ describe('#all', () => {
});
});

describe('#manage', () => {
test(`returns saved_object:manage`, () => {
const savedObjectActions = new SavedObjectActions(version);
expect(savedObjectActions.manage).toBe('saved_object:1.0.0-zeta1:manage');
});
});

describe('#get', () => {
[null, undefined, '', 1, true, {}].forEach((type: any) => {
test(`type of ${JSON.stringify(type)} throws error`, () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ export class SavedObjectActions {
return `${this.prefix}*`;
}

public get manage(): string {
return `${this.prefix}manage`;
}

public get(type: string, operation: string): string {
if (!type || !isString(type)) {
throw new Error('type is required and must be a string');
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,15 @@ export function checkPrivilegesDynamicallyWithRequestFactory(
): CheckPrivilegesDynamicallyWithRequest {
return function checkPrivilegesDynamicallyWithRequest(request: Record<string, any>) {
const checkPrivileges = checkPrivilegesWithRequest(request);
return async function checkPrivilegesDynamically(privilegeOrPrivileges: string | string[]) {
return async function checkPrivilegesDynamically(
legrego marked this conversation as resolved.
Show resolved Hide resolved
privilegeOrPrivileges: string | string[],
spaceId?: string
) {
if (spaces.isEnabled) {
const spaceId = spaces.getSpaceId(request);
return await checkPrivileges.atSpace(spaceId, privilegeOrPrivileges);
return await checkPrivileges.atSpace(
spaceId || spaces.getSpaceId(request),
privilegeOrPrivileges
);
} else {
return await checkPrivileges.globally(privilegeOrPrivileges);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ export class FeaturePrivilegeSavedObjectsManagementBuilder extends BaseFeaturePr
if (feature.id !== 'savedObjectsManagement') {
return [];
}
const canManage = privilegeDefinition.savedObject.all.length > 0;
legrego marked this conversation as resolved.
Show resolved Hide resolved
return uniq([
...flatten(
privilegeDefinition.savedObject.all.map(type => [
Expand All @@ -28,6 +29,7 @@ export class FeaturePrivilegeSavedObjectsManagementBuilder extends BaseFeaturePr
...privilegeDefinition.savedObject.read.map(type =>
this.actions.ui.get('savedObjectsManagement', type, 'read')
),
...(canManage ? [this.actions.savedObject.manage] : []),
]);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -679,6 +679,8 @@ describe('savedObjectsManagement feature', () => {

actions.ui.get('savedObjectsManagement', 'all-savedObject-read-1', 'read'),

actions.savedObject.manage,

actions.allHack,
]);

Expand All @@ -702,6 +704,8 @@ describe('savedObjectsManagement feature', () => {
actions.ui.get('savedObjectsManagement', 'read-savedObject-all-1', 'read'),

actions.ui.get('savedObjectsManagement', 'read-savedObject-read-1', 'read'),

actions.savedObject.manage,
]);
});
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ export class SecureSavedObjectsClientWrapper {
await this._ensureAuthorized(
type,
'create',
options.namespace,
{ type, attributes, options },
);

Expand All @@ -42,16 +43,18 @@ export class SecureSavedObjectsClientWrapper {
await this._ensureAuthorized(
types,
'bulk_create',
options.namespace,
{ objects, options },
);

return await this._baseClient.bulkCreate(objects, options);
}

async delete(type, id, options) {
async delete(type, id, options = {}) {
await this._ensureAuthorized(
type,
'delete',
options.namespace,
{ type, id, options },
);

Expand All @@ -62,6 +65,7 @@ export class SecureSavedObjectsClientWrapper {
await this._ensureAuthorized(
options.type,
'find',
options.namespace,
{ options }
);

Expand All @@ -73,6 +77,7 @@ export class SecureSavedObjectsClientWrapper {
await this._ensureAuthorized(
types,
'bulk_get',
options.namespace,
{ objects, options },
);

Expand All @@ -83,6 +88,7 @@ export class SecureSavedObjectsClientWrapper {
await this._ensureAuthorized(
type,
'get',
options.namespace,
{ type, id, options },
);

Expand All @@ -93,26 +99,27 @@ export class SecureSavedObjectsClientWrapper {
await this._ensureAuthorized(
type,
'update',
options.namespace,
{ type, id, attributes, options },
);

return await this._baseClient.update(type, id, attributes, options);
}

async _checkSavedObjectPrivileges(actions) {
async _checkSavedObjectPrivileges(actions, namespace) {
try {
return await this._checkPrivileges(actions);
return await this._checkPrivileges(actions, namespace);
} catch(error) {
const { reason } = get(error, 'body.error', {});
throw this.errors.decorateGeneralError(error, reason);
}
}

async _ensureAuthorized(typeOrTypes, action, args) {
async _ensureAuthorized(typeOrTypes, action, namespace, args) {
const types = Array.isArray(typeOrTypes) ? typeOrTypes : [typeOrTypes];
const actionsToTypesMap = new Map(types.map(type => [this._actions.savedObject.get(type, action), type]));
const actions = Array.from(actionsToTypesMap.keys());
const { hasAllRequested, username, privileges } = await this._checkSavedObjectPrivileges(actions);
const { hasAllRequested, username, privileges } = await this._checkSavedObjectPrivileges(actions, namespace);

if (hasAllRequested) {
this._auditLogger.savedObjectsAuthorizationSuccess(username, action, types, args);
Expand Down
Loading