[Bug] BLEUScorer uses wrong default tokenizer.

[landert]

🐛 Bug

vizseq.scorers.bleu.BLEUScorer does not use Tokenizer13a by default. When I look at the code, it looks like it should be used by default. sacrebleu library uses the Tokenizer13a by default as well.

To reproduce

Minimal Code/Config snippet to reproduce

import vizseq

scorer = vizseq.scorers.bleu.BLEUScorer()
print(scorer.score(["This is really nice."], [["That's really nice."]]))
# corpus_score = 31.947

scorer = vizseq.scorers.bleu.BLEUScorer(extra_args={'tokenizer': '13a'})
print(scorer.score(["This is really nice."], [["That's really nice."]]))
# corpus_score = 39.764

Stack trace/error message

The problem is here. Variable tokenizer is set to string none. When calling method get_default_args (here), default value 13a for parameter tokenize is not used, because the string none is passed.

Expected Behavior

vizseq.scorers.bleu.BLEUScorer should use Tokenizer13a by default.

System information

• vizseq==0.1.15
• python==3.7.3
• macOS