01-02-2016, 08:54 AM
'word' and 'token' have both been used in the past to designate a group of characters delimited by (apparent) spaces, but unfortunately also with their meanings swapped.
The two questions:
- how many words are there in the Voynich MS?
- how many different words are there in the Voynich MS?
could be answered (*) with:
N words and M tokens. But which is which? I don't see an obvious preference -
or do we indeed need a new term for one or both?
I decided recently to follow the terminology of Reddy and Knight, who use 'word tokens'
and 'word types'. Here it is more naturally understood that several 'word tokens' can be the same (*)
but all 'word types' are different, and there are fewer word types than word tokens.
'word length distribution' is ambiguous. The graph looks different for word tokens and word types.
I would be interested to know (and happy if someone here knows) whether this is an accepted, standard
usage.
- - -
Note (*):
Of course we don't know, but can only estimate based on some assumptions.
The two questions:
- how many words are there in the Voynich MS?
- how many different words are there in the Voynich MS?
could be answered (*) with:
N words and M tokens. But which is which? I don't see an obvious preference -
or do we indeed need a new term for one or both?
I decided recently to follow the terminology of Reddy and Knight, who use 'word tokens'
and 'word types'. Here it is more naturally understood that several 'word tokens' can be the same (*)
but all 'word types' are different, and there are fewer word types than word tokens.
'word length distribution' is ambiguous. The graph looks different for word tokens and word types.
I would be interested to know (and happy if someone here knows) whether this is an accepted, standard
usage.
- - -
Note (*):
Of course we don't know, but can only estimate based on some assumptions.