[LLM pipeline] Add normalize text component #246
Conversation
| @@ -0,0 +1,20 @@ | |||
| name: Normalize text. | |||
| description: A component that normalizes text. | |||
| image: ghcr.io/ml6team/text_normalization:latest | |||
There was a problem hiding this comment.
| image: ghcr.io/ml6team/text_normalization:latest | |
| image: ghcr.io/ml6team/normalize_text:latest |
There was a problem hiding this comment.
As mentioned in chat, agree with consistent naming, but would actually prefer text_normalization here as it will group all text components alphabetically.
There was a problem hiding this comment.
Hmm not necessarily agree on that one, we can group them ourselves in the docs?
Would prefer to have verb + noun for all our components
There was a problem hiding this comment.
Thanks @mrchtr! One small change is needed to the logging, otherwise LGTM.
| @@ -0,0 +1,20 @@ | |||
| name: Normalize text. | |||
| description: A component that normalizes text. | |||
| image: ghcr.io/ml6team/text_normalization:latest | |||
There was a problem hiding this comment.
As mentioned in chat, agree with consistent naming, but would actually prefer text_normalization here as it will group all text components alphabetically.
| from fondant.component import PandasTransformComponent | ||
| from fondant.logger import configure_logging | ||
|
|
||
| configure_logging() |
There was a problem hiding this comment.
This is outdated, please rebase on / merge with main.
Component which applies different text normalization (nfc, lowercasing and regex pattern replacements) This component is needed for the LLM dataset creation pipeline.
Component which applies different text normalization (nfc, lowercasing and regex pattern replacements)
This component is needed for the LLM dataset creation pipeline.