[BREAKING] Support for fetching facets from value edge list

[ashish-goswami]

Fixes #4081

Covers following things:

• Support for fetching facets from value edge list
• Response format change in case of facets to have uniform response format across, uid/value/list edges
• Fix broken tests

---

This change is 

[golangcibot]

line is 108 characters (from lll)

[pawanrawal]

Please also check with @campoy, but I think the idea was to put the change in response format behind a feature flag so as not to break backward compatibility for existing users.

Reviewed 1 of 5 files at r2, 3 of 4 files at r4, 4 of 4 files at r5.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @animesh2049, @ashish-goswami, @mangalaman93, @manishrjain, and @martinmr)

---

posting/list.go, line 972 at r5 (raw file):

	var facets []*api.Facet
	err := l.iterate(readTs, 0, func(p *pb.Posting) error {
		if len(p.LangTag) == 0 {

Why do we skip these? The posting which belongs to a language can also have facets. Can you please check what happens in the case of expand(_all_) @facets where one of the expanded predicate is of scalar list type and has facets on language posting.

---

posting/list.go, line 978 at r5 (raw file):

	})

	return facets, err

I have a feeling that this might not work well in all scenarios. So we are collecting facets across different postings, each posting can have multiple facets. Like below.

"ashish" : []*api.Facet{}
"pawan": []*api.Facet{}

The number of facets belonging to each value(posting) can be different. If we put all of them in the same list, then how do we later know which posting do certain facets belong to?

---

posting/list.go, line 1164 at r5 (raw file):

	if listType {
		fs, err := l.AllUntaggedFacets(readTs)

Facets method already has a read lock and AllUntaggedFacets is also acquiring it again and it doesn't need to. AllUntaggedFacets should instead assert that a read lock is held and should instead be an internal function.

---

query/outputnode.go, line 222 at r5 (raw file):

	}

	for fName, facetList := range facetMap {

I'd need you to walk me through the changes in this file as I don't have a very good mental map of what is happening here.

---

query/query_facets_test.go, line 1631 at r5 (raw file):

	query := `{
		q(func: uid(0x1)) {
			name@en

Can we fetch the facets for this as well?

---

query/query_facets_test.go, line 1632 at r5 (raw file):

		q(func: uid(0x1)) {
			name@en
			alt_name @facets

Can you also add a case where alt_name has multiple values but not all the values have the same number of facets.

Also, please add a case where we only fetch a subset of values, like

alt_name @facets(origin)

[martinmr]

Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @animesh2049, @ashish-goswami, @mangalaman93, @manishrjain, and @pawanrawal)

---

posting/list.go, line 972 at r5 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Why do we skip these? The posting which belongs to a language can also have facets. Can you please check what happens in the case of expand(_all_) @facets where one of the expanded predicate is of scalar list type and has facets on language posting.

tag and language is the same term. So if the method is meant for retrieving only facets from untagged values, then it should skip values that have a language tag.

[ashish-goswami]

Reviewable status: 4 of 8 files reviewed, 7 unresolved discussions (waiting on @animesh2049, @ashish-goswami, @golangcibot, @mangalaman93, @manishrjain, and @pawanrawal)

---

posting/list.go, line 978 at r5 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

I have a feeling that this might not work well in all scenarios. So we are collecting facets across different postings, each posting can have multiple facets. Like below.

"ashish" : []*api.Facet{}
"pawan": []*api.Facet{}

The number of facets belonging to each value(posting) can be different. If we put all of them in the same list, then how do we later know which posting do certain facets belong to?

Thanks for pointing it out. We fixed it.

---

posting/list.go, line 1164 at r5 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Facets method already has a read lock and AllUntaggedFacets is also acquiring it again and it doesn't need to. AllUntaggedFacets should instead assert that a read lock is held and should instead be an internal function.

Done.

---

query/outputnode.go, line 213 at r4 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

line is 108 characters (from lll)

Done.

---

query/query_facets_test.go, line 1631 at r5 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Can we fetch the facets for this as well?

Done.

---

query/query_facets_test.go, line 1632 at r5 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Can you also add a case where alt_name has multiple values but not all the values have the same number of facets.

Also, please add a case where we only fetch a subset of values, like

alt_name @facets(origin)

Done.

[pawanrawal]

Reviewed 4 of 4 files at r6.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @animesh2049, @ashish-goswami, @golangcibot, @mangalaman93, and @manishrjain)

---

posting/list.go, line 978 at r5 (raw file):

Previously, ashish-goswami (Ashish Goswami) wrote…

Thanks for pointing it out. We fixed it.

This should still be an internal method allUntaggedFacets and should assert read lock is held.

---

posting/list.go, line 965 at r6 (raw file):

// AllUntaggedFacets returns all facets of all untagged values.
func (l *List) AllUntaggedFacets(readTs uint64) ([]*pb.Facets, error) {

Please add a comment that this is only called for predicates of type list.

[ashish-goswami]

Reviewable status: 4 of 8 files reviewed, 4 unresolved discussions (waiting on @animesh2049, @golangcibot, @mangalaman93, @manishrjain, and @pawanrawal)

---

posting/list.go, line 978 at r5 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

This should still be an internal method allUntaggedFacets and should assert read lock is held.

Done.

---

posting/list.go, line 965 at r6 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Please add a comment that this is only called for predicates of type list.

Done.

---

query/outputnode.go, line 222 at r5 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

I'd need you to walk me through the changes in this file as I don't have a very good mental map of what is happening here.

Discussed this offline.

[manishrjain]

Got a couple of comments. Prefix the title with a big [BREAKING], so we all know. Let's keep it in a branch and then merge in master for v1.2.

Reviewed 1 of 4 files at r4, 2 of 4 files at r5, 1 of 4 files at r6, 4 of 4 files at r7.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @animesh2049, @ashish-goswami, @golangcibot, @mangalaman93, and @pawanrawal)

---

posting/list.go, line 971 at r7 (raw file):

	err := l.iterate(readTs, 0, func(p *pb.Posting) error {
		if len(p.LangTag) == 0 {
			facets = append(facets, &pb.Facets{Facets: p.Facets})

facets = append(facets, p.Facets...)

Also, do whatever copying you need here.

---

posting/list.go, line 1169 at r7 (raw file):

		for _, fcts := range fs {
			fcs = append(fcs, &pb.Facets{Facets: facets.CopyFacets(fcts.Facets, param)})

Avoid copying here.

---

query/outputnode.go, line 213 at r4 (raw file):

Previously, ashish-goswami (Ashish Goswami) wrote…

Done.

(fj *fastJsonNode) attachFacets(...)

[robsws]

Please let me know if I should post this elsewhere, but I've built this branch from source and I have an issue with the indexes in the facet data object. Here's an example - I'm running the following query:

{
  countries(func: type(Country)) {
    Name @facets(start_time, end_time) 
  }
}

That gives me the following results:

{
  "data": {
    "countries": [
      {
        "Name|end_time": {
          "0": "9999-12-31T00:00:00Z"
        },
        "Name|start_time": {
          "0": "1753-01-01T00:00:00Z"
        },
        "Name": [
          "Isle of Man"
        ]
      },
      {
        "Name|end_time": {
          "1": "9999-12-31T00:00:00Z"
        },
        "Name|start_time": {
          "1": "1753-01-01T00:00:00Z"
        },
        "Name": [
          "Chile"
        ]
      },
      {
        "Name|end_time": {
          "2": "9999-12-31T00:00:00Z"
        },
        "Name|start_time": {
          "2": "1753-01-01T00:00:00Z"
        },
        "Name": [
          "Moldova"
        ]
      },

      ...

      {
        "Name|end_time": {
          "250": "9999-12-31T00:00:00Z"
        },
        "Name|start_time": {
          "250": "1753-01-01T00:00:00Z"
        },
        "Name": [
          "Sahrawi"
        ]
      },
      {
        "Name|end_time": {
          "251": "9999-12-31T00:00:00Z"
        },
        "Name|start_time": {
          "251": "1753-01-01T00:00:00Z"
        },
        "Name": [
          "Serbia"
        ]
      }
    ]
  },

I was expecting that the keys for the "Name|end_time" and "Name|start_time" objects would refer back into the "Name" array, but it seems that they increment with each successive node in the result set. Was my assumption wrong here or is this not working as intended?

[pawanrawal]

@robsws that seems like a bug with the current implementation. Thanks for sharing this. @ashish-goswami could you please see that this is addressed before the PR is merged? We need some test cases which start with multiple uids at the root.

[ashish-goswami]

Hey @robsws, Thanks for pointing this. We have fixed it now. Please verify.

[zhaolin-st]

will this feature merge in the future version?

[ashish-goswami]

@zhaolin-st Yes this will be merged in future, most likely in v1.2.

[martinmr]

Hey, #4497 is fixed by this PR. But could you add an additional test to ensure facets in preds of list type work correctly with the expand function? The example in the issue should be helpful for that.

Reviewable status: 6 of 8 files reviewed, 6 unresolved discussions (waiting on @animesh2049, @ashish-goswami, @golangcibot, @mangalaman93, @manishrjain, and @pawanrawal)

[zhaolin-st]

@zhaolin-st Yes this will be merged in future, most likely in v1.2.

@ashish-goswami
Thanks! This would solve my problems.
https://discuss.dgraph.io/t/how-can-i-query-facets-with-multi-property-better-facets-dgraph-in-v1-1-1/5724
and this one
https://discuss.dgraph.io/t/how-to-use-expand-sometype-if-some-predicate-of-the-type-are-list/5764/4

[ashish-goswami]

Reviewable status: 1 of 9 files reviewed, 6 unresolved discussions (waiting on @animesh2049, @ashish-goswami, @golangcibot, @mangalaman93, @manishrjain, and @pawanrawal)

---

query/outputnode.go, line 213 at r4 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

(fj *fastJsonNode) attachFacets(...)

done.