New Module: 51Degrees #3650

jwrosewell · 2024-04-30T07:50:52Z

Type of change

Feature
New module

Description of change

51Degrees module enriches an incoming OpenRTB request with 51Degrees Device Data.

51Degrees module sets the following fields of the device object: make, model, os, osv, h, w, ppi, pxratio - interested bidder adapters may use these fields as needed. In addition the module sets device.ext.fiftyonedegrees_deviceId to a permanent device ID which can be rapidly looked up in on premise data exposing over 250 properties including the device age, chip set, codec support, and price, operating system and app/browser versions, age, and embedded features.

The module is based on the open source 51Degrees device detection library: https://github.com/51Degrees/device-detection-go.

Integration details are specified in the modules/fiftyone_degrees/device_detection/README.md.

Maintainer contacts:

[email protected]

Co-authored-by: Sarana-Anna <[email protected]> Co-authored-by: James Rosewell <[email protected]>

Indentation fix.

* expose watch_file_system option * expose make_temp_copy option * improved performance

# Conflicts: # go.mod # go.sum

after mergin master

bsardo · 2024-06-15T21:16:52Z

Hello. Thanks for your patience. I just want to give you a quick status update. For various reasons, we are going to need a couple more weeks before we're able to tackle this. There's a lot here so it requires a significant time investment to review, and on top of that, we may also need to perform a security review of your library and quite possibly lock the version in PBS. Stay tuned.

jwrosewell · 2024-06-20T08:38:09Z

@bsardo - Thank you. To help make the review process more efficient for everyone I'd be happy to setup a virtual meeting with the reviewers to answer questions and clarify feedback in real time. Please let me know if this is something you or other reviewers would like to arrange.

remove fields that could be overwritten

SyntaxNode · 2024-07-23T19:52:01Z

modules/fiftyone_degrees/device_detection/account_validator.go

+	}
+}
+
+func (x *AccountValidator) IsWhiteListed(cfg Config, req []byte) bool {


Nitpick: Consider renaming to "IsAllowed" to align with the "AccountFilter.AllowList" terminology and to use use more inclusive language.

SyntaxNode · 2024-07-23T19:57:57Z

modules/fiftyone_degrees/device_detection/request_headers_extractor.go

+func (x EvidenceFromRequestHeadersExtractor) extractEvidenceStrings(r *http.Request, keys []dd.EvidenceKey) []StringEvidence {
+	evidence := make([]StringEvidence, 0)
+	for _, e := range keys {
+		lowerKey := strings.ToLower(e.Key)


Nitpick / Performance: There is no need to convert the key to lower case r.Header.Get is case insensitive.

SyntaxNode · 2024-07-23T19:59:37Z

modules/fiftyone_degrees/device_detection/utils.go

+package device_detection
+
+// Contains checks if a string is in a slice of strings
+func Contains(source []string, item string) bool {


Consider using the built-in slices.Contains added in Go 1.21.

bsardo

Looks good. I left a lot of comments but I imagine most warrant easy changes or no action.

I still need to finish going through the tests, which I will do this week, but I wanted to give you feedback on the code first.

bsardo · 2024-07-18T18:06:42Z

go.mod

+	github.com/tidwall/gjson v1.17.1 // indirect
+	github.com/tidwall/match v1.1.1 // indirect
+	github.com/tidwall/pretty v1.2.0 // indirect
+	github.com/tidwall/sjson v1.2.5 // indirect


As discussed, I ran some gjson vs jsonparser benchmark tests and while jsonparser is slightly faster for more complex cases with less bytes per operation, we consider the difference small enough that we are good with using gjson in this module. We do also like that gjson has a sister library sjson that you can use. The jsonparser library has a set feature but it is experimental and we have found that it is buggy.

Here are the benchmark test results:

goos: darwin goarch: amd64 pkg: github.com/prebid/prebid-server/v2/modules/fiftyone_degrees/device_detection cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz BenchmarkSingleTopLevelGJSON-12 3016309 398.3 ns/op 80 B/op 1 allocs/op BenchmarkSingleTopLevelJSONParser-12 2739925 426.3 ns/op 0 B/op 0 allocs/op BenchmarkSingleNestedGJSON-12 2841622 423.3 ns/op 24 B/op 1 allocs/op BenchmarkSingleNestedJSONParser-12 2601579 406.7 ns/op 0 B/op 0 allocs/op BenchmarkManyNestedGJSON-12 622218 1928 ns/op 624 B/op 6 allocs/op BenchmarkManyNestedJSONParser-12 769302 1388 ns/op 304 B/op 8 allocs/op

See the attached file containing the benchmark tests.
json_benchmark_test.txt

bsardo · 2024-07-18T18:22:42Z

modules/fiftyone_degrees/device_detection/README.md

+
+### Account-Level Config
+
+To start using current module in PBS-Java you have to enable module and add `fiftyone-devicedetection-entrypoint-hook` and `fiftyone-devicedetection-raw-auction-request-hook` into hooks execution plan inside your config file:


Typo: PBS-Java --> PBS-Go

bsardo · 2024-07-18T18:27:02Z

modules/fiftyone_degrees/device_detection/README.md

+    fiftyone_degrees:
+      device_detection: 
+        enabled: true
+        data-file:


Typo: data-file --> data_file

bsardo · 2024-07-23T14:18:54Z

modules/fiftyone_degrees/device_detection/module.go

+	var cfg Config
+	ncfg, err := ParseConfig(rawConfig)


Nitpick: I think you can get rid of cfg and just use ncfg and get rid of the cfg = ncfg assignment on line 46.

bsardo · 2024-07-23T16:24:33Z

modules/fiftyone_degrees/device_detection/hook_auction_entrypoint.go

+// handleAuctionEntryPointRequestHook is a hookstage.HookFunc that is used to handle the auction entrypoint request hook.
+func handleAuctionEntryPointRequestHook(cfg Config, payload hookstage.EntrypointPayload, deviceDetector deviceDetector, evidenceExtractor evidenceExtractor, accountValidator accountValidator) (result hookstage.HookResult[hookstage.EntrypointPayload], err error) {
+	// if account/domain is not whitelisted, return failure
+	if accountValidator.IsWhiteListed(cfg, payload.Body) != true {


Nitpick: go best practice would be to use the not operator instead of != true:
if !accountValidator.IsWhiteListed(cfg, payload.Body) {

bsardo · 2024-07-24T20:10:39Z

modules/fiftyone_degrees/device_detection/fiftyone_device_types.go

+// fiftyOneDtToRTB converts a 51Degrees device type to an OpenRTB device type.
+// If the device type is not recognized, it defaults to PC.


Can you move this comment down so it is right above the fiftyOneDtToRTB function?

bsardo · 2024-07-24T20:22:05Z

modules/fiftyone_degrees/device_detection/device_info_extractor.go

+func (x DeviceInfoExtractor) getValue(
+	results Results,
+	propertyName DeviceInfoProperty) string {


Nitpick: please put the whole function signature on the same line.

bsardo · 2024-07-24T20:28:06Z

modules/fiftyone_degrees/device_detection/device_info_extractor.go

+		",",
+	)
+	if err != nil {
+		log.Printf("ERROR: Failed to get results values string.")


I'm curious why you're using log here and in a few other places instead of glog. I think we should use glog throughout, in which case this would probably be a glog.Errorf.

bsardo · 2024-07-24T20:30:07Z

modules/fiftyone_degrees/device_detection/device_info_extractor.go

+		",",
+	)
+	if err != nil {
+		log.Printf("ERROR: Failed to get results values string.")


Also if you do get this error, it looks like the error would get returned via lines 120-122. Would it be better to just early return here so it is more obvious?

bsardo · 2024-07-24T20:36:15Z

modules/fiftyone_degrees/device_detection/account_info_extractor.go

+
+// Extract extracts the account information from the payload
+// The account information is extracted from the publisher id or site publisher id
+func (x AccountInfoExtractor) Extract(payload []byte) *AccountInfo {


As a general comment applicable to all public functions and variables, please assess whether these need to be exported. I don't think most of these are meant to be accessed outside of the device_detection package, in which case they should begin with a lower case letter.

totally agree, thanks

justadreamer · 2024-07-24T20:44:52Z

Thanks for this review @SyntaxNode and @bsardo! The recent push addresses Scott's comments, we'll work on addressing Brian's comments tomorrow.

justadreamer · 2024-07-30T09:50:36Z

Hi @bsardo, thanks for the super-thorough code review! The above comments have been addressed with the latest pushes from @legendko and myself - we are ready for the next review pass :).

bsardo

I've finished going through the updates made in response to my last review as well as all of the tests. This is very close. Most of my comments are nitpicks and quick tweaks. The more time consuming ones are recommendations to move some of the tests to a table-based format with additional test cases. I've provided some of those to expedite things. Hopefully we can get this into this week's release.

@SyntaxNode do you have time this week to review?

bsardo · 2024-07-29T18:53:53Z

modules/fiftyonedegrees/devicedetection/device_info_extractor.go

+		",",
+	)
+	if err != nil {
+		glog.Errorf("ERROR: Failed to get results values string.")


Nitpick: I think you can remove ERROR: from the message since glog will handle indicating that fact.

bsardo · 2024-07-29T19:26:27Z

modules/fiftyonedegrees/devicedetection/device_detector.go

+		onpremise.WithProperties([]string{
+			"HardwareVendor",
+			"HardwareName",
+			"deviceType",


Is this supposed to be "DeviceType",?

Yes, thanks for noticing!

bsardo · 2024-07-29T19:27:43Z

modules/fiftyonedegrees/devicedetection/device_detector.go

+func (x defaultDeviceDetector) getDeviceInfo(evidence []onpremise.Evidence, ua string) (*deviceInfo, error) {
+	results, err := x.engine.Process(evidence)
+	if err != nil {
+		return nil, errors.Wrap(err, "Failed to Process evidence")


Should Process be lowercase?

bsardo · 2024-07-29T19:38:30Z

modules/fiftyonedegrees/devicedetection/device_info_extractor.go

+const (
+	deviceInfoHardwareVendor        deviceInfoProperty = "HardwareVendor"
+	deviceInfoHardwareName          deviceInfoProperty = "HardwareName"
+	deviceInfoDeviceType            deviceInfoProperty = "deviceType"


"deviceType" should be "DeviceType"?

Yes, thanks again! It seems like some renaming went wrong

bsardo · 2024-07-29T20:06:42Z

modules/fiftyonedegrees/devicedetection/hook_raw_auction_request.go

+				er := fmt.Sprintf("error hydrating fields %s", err)
+				glog.Error(er)
+				return rawPayload, hookexecution.NewFailure(er)


I'm curious if we really want to log an error here. There are plenty of other hook execution failures caught both in this file and in the module where we don't do that. Thoughts?

that's a leftover from the previous version of code, before your first review.
it had only the log part: glog.Errorf("error hydrating fields %s", err) but you suggested returning a hookexecution Failure error here instead of just logging. I'll remove the log here as it is redundant now.

bsardo · 2024-08-05T02:31:29Z