Add relation extraction

[pklpriv]

Adding support for relation-extraction task.

[yoavkatz]

This is not needed. It's not called by CustomF1. It's a global scope method used only by TokenOverlap.

[yoavkatz]

This is not needed, we always that the first reference in NER (we assume there is only one reference).
In unitxt, FM eval a reference is a full "correct answer". Note in this case an answer is a List[Tuple[str,str,str]].

[yoavkatz]

Suggested change

	prediction_type = "List[Tuple[str,str]]"
	prediction_type = "List[Tuple[str,str,str]]"

[pklpriv]

done

[yoavkatz]

Hi. I added my comments. I think you should create a card that uses the tasks, and loads the raw data from the file, and converts it to the format required by the task.

[elronbandel]

I think we need a better name, it took my time to understand what it is maybe "relation_entity_subjects"
"relation_entity_objects" or something more explicit

[pklpriv]

done

[piotrhm]

I think we do not want this line?

[yoavkatz]

Right. Why was this added?

[elronbandel]

Since this is an important NLP task i suggest we try to get it merged asap:

My suggestion is to follow the conventions and naming in the TACRED dataset as a representivte of the jargon concensus:

The simple naming for the task output fields is:

subjects: List[str]
relations: List[str]
objects: List[str]

The more verbosed version:

subjects_mentions: List[str]
relations_types: List[str]
objects_mentions: List[str]

I personally prefer the simple one.

Now to the second observation: There are two types of tasks really (1) To produce only mentions and relations (2) To produce mentions with their exact location in the text. Both have different metrics and different use cases IMO.

So my practical suggestion here is to actually have two different tasks:

1. tasks.relation_extraction
2. tasks.relation_extraction.with_positions

The second ofcourse should be with :

subjects_starts: List[int]
subjects_ends: List[int]
objects_starts: List[int]
objects_ends: List[int]

What do you think @pklpriv and @yoavkatz ?

[yoavkatz]

Since this is an important NLP task i suggest we try to get it merged asap:

My suggestion is to follow the conventions and naming in the TACRED dataset as a representivte of the jargon concensus:

The simple naming for the task output fields is:

subjects: List[str]
relations: List[str]
objects: List[str]

The more verbosed version:

subjects_mentions: List[str]
relations_types: List[str]
objects_mentions: List[str]

I personally prefer the simple one.

Now to the second observation: There are two types of tasks really (1) To produce only mentions and relations (2) To produce mentions with their exact location in the text. Both have different metrics and different use cases IMO.

So my practical suggestion here is to actually have two different tasks:

1. tasks.relation_extraction
2. tasks.relation_extraction.with_positions

The second ofcourse should be with :

subjects_starts: List[int]
subjects_ends: List[int]
objects_starts: List[int]
objects_ends: List[int]

What do you think @pklpriv and @yoavkatz ?

I agree that the short names are better, and that there are two tasks (with position and without position). They not only differ in the input, but also in the prediction type (e.g. the position need to be checked). Initially we can support only tasks.relation_extraction. Even if the data is with positions, it should be converted to lists of strings in the pre-processing phase of the card (and not in the template, as done today in NER)

[pklpriv]

@elronbandel @yoavkatz

I agree with You, I think this is good way to implement that. Additionally I am currently implementing task/metric for datasets that do not have specified and named relation like (obj1, relation, obj2). Instead, for tuple (ob1,obj2,...,obj_n) we consider being in the same tuple as relation itself.

[elronbandel]

@pklpriv Can you explain more about the n-ary tuple with unnamed relation? Can you give one input and required output example?

[pklpriv]

@elronbandel

I will allow myself to cite Yoav:

“As of December 31, 2011, we had 4,260 employees of whom 2,155, or 51%, are employed in the U.S.; 1,165, or 27%, are employed in Europe; 615, or 14%, are employed in Asia and 325, or 8%, are employed in the Middle East.“,

This is the expected output:

[ {\“DATE\“:\“December 31, 2011\“,\“EMPLOYEE_NUMBER\“:\“4,260\“},
{\“DATE\“:\“December 31, 2011\“,\“EMPLOYEE_NUMBER\“:\“2155\“,\“EMPLOYEE_PERCENT\“:\“51%\“,\“GEOGRAPHY\“:\“U.S.\“}, {\“DATE\“:\“December 31, 2011\“,\“EMPLOYEE_NUMBER\“:\“1165\“,\“EMPLOYEE_PERCENT\“:\“27%\“,\“GEOGRAPHY\“:\“Europe\“}, {\“DATE\“:\“December 31, 2011\“,\“EMPLOYEE_NUMBER\“:\“615\“,\“EMPLOYEE_PERCENT\“:\“14%\“,\“GEOGRAPHY\“:\“Asia\“}, {\“DATE\“:\“December 31, 2011\“,\“EMPLOYEE_NUMBER\“:\“325\“,\“EMPLOYEE_PERCENT\“:\“8%\“,\“GEOGRAPHY\“:\“Middle East\“}]

As You see, there is no object that represents relation itself (in opposite to f.e. (John,employedBy,Hannah) where 'employedBy' is relation stated as an object). Instead, it is defined by sentence itself.