Ensure tokenizer sizers with truncation parameters count their overflow encodings #646
Job | Run time |
---|---|
1m 35s | |
1m 52s | |
2m 3s | |
1m 52s | |
18s | |
1m 12s | |
1m 43s | |
4m 33s | |
1m 54s | |
2m 9s | |
1m 50s | |
1m 39s | |
0s | |
22m 40s |
Job | Run time |
---|---|
1m 35s | |
1m 52s | |
2m 3s | |
1m 52s | |
18s | |
1m 12s | |
1m 43s | |
4m 33s | |
1m 54s | |
2m 9s | |
1m 50s | |
1m 39s | |
0s | |
22m 40s |