Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

merge code form metadata management to master #1420

Merged
merged 22 commits into from
Jun 12, 2023
Merged

merge code form metadata management to master #1420

merged 22 commits into from
Jun 12, 2023

Conversation

PravinRanjan10
Copy link
Collaborator

This Pr contains, all the file of metadata management branch to master.
The overall feature contains:

  • Metadata collect
  • metdata store
  • metadata search/query based on parameters
  • all api of metadata management

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:

/kind new feature
/kind bug fix
/kind cleanup
/kind revert change
/kind design
/kind documentation
/kind enhancement

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Test Report Added?:

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind TESTED
/kind NOT-TESTED

Test Report:

Special notes for your reviewer:

PravinRanjan10 and others added 22 commits December 22, 2022 07:39
@PravinRanjan10
Metadata service (#1371)
f2f1c84 
* metadata-framework

* some update
@josephjacobmorris
Metadata Manager : List Metadata API enhancement as per latest design…
b38dddd 
… docs (#1374)

* updating the listmetadata api request as per design doc for grpc

* updating the listmetadata api response as per design doc for grpc

* generate code using protoc based on new metadata.proto file

* fetch request params for ListMetaData api , assign default values when required and send to metadata manager micro-service using grpc

* reuse existing function to get pagination params

* handling err in constructing ListMetaDataRequest

* change message name to avoid conflicts with existing names

* change SizeOfObject, SizeOfBucket  representation to make it more understandable

* update error message

* renaming usage of Bucket message

* changing type of size to int64 and remove id field for bucket and object metadata

* rearrange to avoid repeated hard-coded value

* remove commented code

* update field type to Timestamp for date fields

* Revert "update field type to Timestamp for date fields" till query runner is implemented

* log msg update

* err msg update
@PravinRanjan10
adding api and response updation
302a258
@PravinRanjan10
end to end sync framework
c91b9c4
@josephjacobmorris
[Metadata management] Query Translation & Query Runner (#1386)
6a01cf2 
* [metadata manager m8s] [Validator] implementation for validator

* [metadata manager m8s] API enhancement -corrections

* [metadata manager m8s] [Validator] log err msg update

* [metadata manager m8s] mr conflict resolution

* [metadata manager m8s] [Validator] comment resolution

* [metadata manager m8s] [query translator]

* [metadata manager m8s] [query runner]

* [metadata manager m8s] [query translator] uncomment findMatching documents

* [metadata manager m8s] refactor code to utils

* [metadata manager m8s] [query translator] self-review comment resolution

* [metadata manager m8s] [query translator] remove extra spaces

* [metadata manager m8s] comment resolution : remove dots in info msg, formatting

* [metadata manager m8s] [query translator] refactoring query into multiple functions to increase readability

* [metadata manager m8s] [query runner] remove unused file

* [metadata manager m8s] comment resolution : giving example, renaming functions, and o/ping variable value in info log
@subi9
[Backend manager] Crud API for Backends (#1388)
95cc3e8 
* Removing tiers from scripts

* Proper message to be shown when backend id does not exists

* Removed tiers functions from the script

* Removed tiers from testutils backend service

* Removed tiers from testutils-data and repository script

* Removed tiers from backend service test

* Removed tiers from api-pkg-s3 scripts
@PravinRanjan10
Revert "[Backend manager] Crud API for Backends (#1388)"
1c13ab0 
This reverts commit 95cc3e8.
@josephjacobmorris
[metadata manager m8s] [Result Paginator] impl for result pagination (#…
5eb3321 
…1390)

* [metadata manager m8s] [Result Paginator] impl for result pagination

* Delete QueryRunner.go

* remove space

* [metadata manager m8s] [Result Paginator] fetching and using sort order

* [metadata manager m8s] [Result Paginator] make empty sort param as valid and refactoring out constants

* [metadata manager m8s] comment resolution : add licenses, move to single folder...

* [metadata manager m8s] formatting

* [metadata manager m8s] using constant

* [metadata manager m8s] info -> infoln
@PravinRanjan10
Adding logic for all sycn and gcp driver framework
311174b
@devanshjain7
[Metadata Management] added aws storage driver for metadata sync serv…
a436d38 
…ice (#1387)

* added aws storage driver for metadata sync service

* some refactoring in aws driver and db metadata update

* added error handling in aws driver and refactored go routine functions

* removed BucketList and ObjectList methods from driver interface

* changed object metadata model field type for ServerSideEncryption

* removed duplicate bson library import

* updating messages

* refactored metadata storage, error handling and db write operation

* updated metadata.pb.go

* added new context passing for replaceOne operation in mongodb
@devanshjain7
[Metadata Management] added aws storage driver for metadata sync serv…
73c990a 
…ice (#1387)

* added aws storage driver for metadata sync service

* some refactoring in aws driver and db metadata update

* added error handling in aws driver and refactored go routine functions

* removed BucketList and ObjectList methods from driver interface

* changed object metadata model field type for ServerSideEncryption

* removed duplicate bson library import

* updating messages

* refactored metadata storage, error handling and db write operation

* updated metadata.pb.go

* added new context passing for replaceOne operation in mongodb
@josephjacobmorris
[metadata manager m8s][query translator] calculate total size of obje…
3dc3541 
…ct and number of objects and number of buckets (#1391)
@josephjacobmorris
Issuefix for metadata manager fixes (#1393)
4cfc695 
* [metadata manager m8s] region name coming as bucket name fix

* [metadata manager m8s] removing query result from log

* [metadata manager m8s] fixing msg when type is valid and region not specified

* [metadata manager m8s] converting bucketSizeOperator  and objectSizeOperator to camelCase

* [metadata management] store objects even though some api failed

* [metadata manager m8s] removing query result from log

* [metadata manager m8s] supporting caseinsensitive cloud type and coverting to lowercase internally
@devanshjain7
aws driver updated to collect bucket metadata associated with backend…
15dad6f 
… region only (#1395)

* aws driver updated to collect bucket metadata associated with backend region only

* resolve race condition and remove unnecessary error returns

* refactor an if else statement while storing bucket tags
@josephjacobmorris
Issuefix for empty buckets missing without any filter & buckets with …
12db78e 
…no objects showing with object level filter (#1396)

* [metadata manager m8s] fixing expires date coming as zero value if no value is provided

* [metadata manager m8s] empty buckets are missing in o/p for without any filter case fix

* [metadata manager m8s] adding document Id to o/p

* [metadata manager m8s] [query translator] adding filtered aggregation fields

* [metadata manager m8s] [query translator] fixing  buckets  with no objects after filtering also getting displayed when querying object level

* [metadata manager m8s] renaming fields as objectSizeInBytes,bucketSizeInBytes

* [metadata manager m8s] remove unnecessary initialization

* [metadata manager m8s] mr conflict resolution correction
@josephjacobmorris
Issuefix : Storage Class not displayed, mgo driver removal (#1398)
4b427c5 
* [metadata manager m8s] fixing expires date coming as zero value if no value is provided

* [metadata manager m8s] empty buckets are missing in o/p for without any filter case fix

* [metadata manager m8s] adding document Id to o/p

* [metadata manager m8s] [query translator] adding filtered aggregation fields

* [metadata manager m8s] [query translator] fixing  buckets  with no objects after filtering also getting displayed when querying object level

* [metadata manager m8s] renaming fields as objectSizeInBytes,bucketSizeInBytes

* [metadata manager m8s] remove unnecessary initialization

* [metadata manager m8s] mr conflict resolution correction

* [metadata manager m8s] adding storage class to output

* [metadata manager m8s] remove log msg

* [metadata manager m8s] remove mgo driver usages from metadata management ,#1389
@devanshjain7
collect additional metadata for buckets and objects in aws driver (#1399
07cac9f 
)

* collect additional metadata for buckets and objects in aws driver

* change names of some metadata fields

* move head object to a separate function, change map/array initialisation method
@josephjacobmorris
vendor files updated using "go mod vendor" (#1409)
c372b6d
@josephjacobmorris
GCP Driver for metadata manager (#1411)
5be67bb 
* vendor files updated using "go mod vendor"

* [metadata-manager] gcp object store driver impl

[metadata-manager] gcp object store driver impl part2

* [metadata-manager] gcp driver store unit cases

* Update model.go

revert changes

* revert go.sum changes
@josephjacobmorris
[metadata manager m8s] Cache Impl for List Metadata (#1405)
7e016cc 
* [metadata manager m8s] fixing expires date coming as zero value if no value is provided

* [metadata manager m8s] empty buckets are missing in o/p for without any filter case fix

* [metadata manager m8s] adding document Id to o/p

* [metadata manager m8s] [query translator] adding filtered aggregation fields

* [metadata manager m8s] [query translator] fixing  buckets  with no objects after filtering also getting displayed when querying object level

* [metadata manager m8s] renaming fields as objectSizeInBytes,bucketSizeInBytes

* [metadata manager m8s] remove unnecessary initialization

* [metadata manager m8s] mr conflict resolution correction

* [metadata manager m8s] adding storage class to output

* [metadata manager m8s] remove log msg

* [metadata manager m8s] remove mgo driver usages from metadata management ,#1389

* [metadata manager m8s] Cache Impl for List Metadata

* [metadata manager m8s] log info msg update

* [metadata manager m8s] remove panic from code on failing to close cache

* [metadata manager m8s] remove unneccesary file

* [metadata manager m8s] easyjson dependency addition and getting generated code

* [metadata manager m8s] conflict resolution

* [metadata manager m8s]  cache eviction and max-memory setting

* [metadata manager m8s]  cache err renaming
@josephjacobmorris
Minorfixes - Cleanup log directives and make Validation err errMsg ac…
fb1c43f 
…cessible outside pkgs (#1402)

* [metadata manager m8s] fixing expires date coming as zero value if no value is provided

* [metadata manager m8s] empty buckets are missing in o/p for without any filter case fix

* [metadata manager m8s] adding document Id to o/p

* [metadata manager m8s] [query translator] adding filtered aggregation fields

* [metadata manager m8s] [query translator] fixing  buckets  with no objects after filtering also getting displayed when querying object level

* [metadata manager m8s] correct directives, make errMsg accessible to other pkgs
@devanshjain7 @PravinRanjan10
upgrade azure-storage-blob-go library to v0.15.0 (#1413)
fcd0668 
Co-authored-by: Pravin Ranjan <[email protected]>
sushanthakumar
sushanthakumar approved these changes Jun 12, 2023
View reviewed changes
Copy link
Collaborator

@sushanthakumar sushanthakumar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

skdwriting
skdwriting approved these changes Jun 12, 2023
View reviewed changes
Copy link
Collaborator

@skdwriting skdwriting left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
(dev branch tested to master)

@skdwriting skdwriting merged commit 2b363b5 into master Jun 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sushanthakumar sushanthakumar sushanthakumar approved these changes

@skdwriting skdwriting skdwriting approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@PravinRanjan10 @skdwriting @sushanthakumar @josephjacobmorris @subi9 @devanshjain7