I’ve been thinking about the Types/Hapax problem. Ultimately, I’ve come to the conclusion that approximately 11 words per page are likely to be filler words or erroneous words. These cannot be replicated within the model but can only be simulated using a fixed list (which, in principle, could be any list). I’m not really satisfied with this “insight,” since it ultimately amounts to an unverifiable assumption.
generated.zip (Size: 21.76 KB / Downloads: 16)
Short words : 1438 (15.1%)
Common words: 3933 (41.4%)
Common combinations: 2017 (21.2%)
Hapax (from Pool): 1103 (11.6%)
Ledger-words: 1009 (10.6%)
Total: 9500
Hapax total: 1844
Types: 2813
generated.zip (Size: 21.76 KB / Downloads: 16)
Short words : 1438 (15.1%)
Common words: 3933 (41.4%)
Common combinations: 2017 (21.2%)
Hapax (from Pool): 1103 (11.6%)
Ledger-words: 1009 (10.6%)
Total: 9500
Hapax total: 1844
Types: 2813

