Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add record checking for allennlp and spacy #5

Closed
wants to merge 9 commits into from
13 changes: 12 additions & 1 deletion forte_wrapper/allennlp/allennlp_processors.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# limitations under the License.

import logging
from typing import List, Dict
from typing import List, Dict, Set

from allennlp.predictors import Predictor
from forte.common import ProcessorConfigError
Expand Down Expand Up @@ -186,3 +186,14 @@ def _create_srl(input_pack: DataPack, tokens: List[Token],
tokens[arg_span.end].end)
link = PredicateLink(input_pack, pred, arg)
link.arg_type = label

@classmethod
def expected_types_and_attributes(cls) -> Dict[str, Set[str]]:
r"""Method to add expected type for current processor input which
would be checked before running the processor if
:meth:`~forte.pipeline.Pipeline.enforce_consistency` was enabled for
the pipeline.
"""
expectation_dict: Dict[str, Set[str]] = dict()
expectation_dict["ft.onto.base_ontology.Sentence"] = set()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a demonstration, probably we can make this function simpler by using the literal implementation:

Suggested change
expectation_dict["ft.onto.base_ontology.Sentence"] = set()
expectation_dict: Dict[str, Set[str]] = {
"ft.onto.base_ontology.Sentence": set()
}

return expectation_dict
12 changes: 11 additions & 1 deletion forte_wrapper/spacy/spacy_processors.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from typing import Optional
from typing import Optional, Dict, Set

import spacy
from spacy.language import Language
Expand Down Expand Up @@ -125,3 +125,13 @@ def _process(self, input_pack: DataPack):

# Process sentence parses.
self._process_parser(result.sents, input_pack)

def record(self, record_meta: Dict[str, Set[str]]):
r"""Method to add output type record of current processor
to :attr:`forte.data.data_pack.Meta.record`.

Args:
record_meta: the field in the datapack for type record that need to
fill in for consistency checking.
"""
record_meta["ft.onto.base_ontology.Sentence"] = set()