WordTokenize

6 references to WordTokenize

Microsoft.Extensions.AI.Evaluation.NLP (6)

BLEUEvaluator.cs (2)

86reference => SimpleWordTokenizer.WordTokenize(reference).ToArray()).ToArray(); 88string[] hypothesis = SimpleWordTokenizer.WordTokenize(modelResponse.Text).ToArray();

F1Evaluator.cs (2)

75string[] reference = SimpleWordTokenizer.WordTokenize(context.GroundTruth).ToArray(); 76string[] hypothesis = SimpleWordTokenizer.WordTokenize(modelResponse.Text).ToArray();

GLEUEvaluator.cs (2)

85reference => SimpleWordTokenizer.WordTokenize(reference).ToArray()).ToArray(); 87string[] hypothesis = SimpleWordTokenizer.WordTokenize(modelResponse.Text).ToArray();