Vord length distribution
#28
(28-06-2020, 07:08 PM)Anton wrote: You are not allowed to view links. Register or Login to view.I think a random text (where each word is a random string constrained by length) will have normal distribution of word type frequency.

Does anybody have a sample?

I saw now that there are already some done experiments posted in this thread. But true, this will not give a normal distribution. If you treat space as just another character with a constant probability of occurrence s, this will give a probability p(n) where n is the length of a word, equal to
p(n) = s*(1-s)^{n-1}
This is a power-law distribution which is asymmetrical (it has a long tail which is always decreasing). The average word-length will be 1/s -1. If you constrain the length you will only cut off the tail.

Edit: This will be a distribution with respect to all words, not word-types. I.e. shorter words will more likely be identical in a random string, therefore making the distribution word-type will decrease frequency of shorter words in the distribution.
[+] 1 user says Thank You to Alin_J for this post
Reply


Messages In This Thread
Vord length distribution - by Anton - 27-06-2020, 11:54 PM
RE: Vord length distribution - by Anton - 28-06-2020, 01:43 AM
RE: Vord length distribution - by Alin_J - 28-06-2020, 07:21 AM
RE: Vord length distribution - by -JKP- - 28-06-2020, 07:40 AM
RE: Vord length distribution - by Koen G - 28-06-2020, 09:55 AM
RE: Vord length distribution - by Alin_J - 28-06-2020, 10:45 AM
RE: Vord length distribution - by -JKP- - 28-06-2020, 10:16 AM
RE: Vord length distribution - by MarcoP - 28-06-2020, 10:18 AM
RE: Vord length distribution - by -JKP- - 28-06-2020, 10:33 AM
RE: Vord length distribution - by MarcoP - 28-06-2020, 10:42 AM
RE: Vord length distribution - by -JKP- - 28-06-2020, 11:01 AM
RE: Vord length distribution - by -JKP- - 28-06-2020, 11:32 AM
RE: Vord length distribution - by Anton - 28-06-2020, 01:55 PM
RE: Vord length distribution - by ReneZ - 28-06-2020, 02:11 PM
RE: Vord length distribution - by Anton - 28-06-2020, 02:55 PM
RE: Vord length distribution - by Alin_J - 28-06-2020, 02:15 PM
RE: Vord length distribution - by Anton - 28-06-2020, 03:00 PM
RE: Vord length distribution - by Anton - 28-06-2020, 03:45 PM
RE: Vord length distribution - by Anton - 28-06-2020, 04:17 PM
RE: Vord length distribution - by Anton - 28-06-2020, 04:21 PM
RE: Vord length distribution - by Koen G - 28-06-2020, 05:16 PM
RE: Vord length distribution - by MarcoP - 28-06-2020, 05:27 PM
RE: Vord length distribution - by Anton - 28-06-2020, 05:20 PM
RE: Vord length distribution - by Koen G - 28-06-2020, 05:30 PM
RE: Vord length distribution - by Anton - 28-06-2020, 07:08 PM
RE: Vord length distribution - by Alin_J - 29-06-2020, 05:55 AM
RE: Vord length distribution - by RobGea - 28-06-2020, 08:50 PM
RE: Vord length distribution - by Anton - 28-06-2020, 10:10 PM
RE: Vord length distribution - by ReneZ - 29-06-2020, 06:40 AM
RE: Vord length distribution - by Anton - 29-06-2020, 01:27 PM
RE: Vord length distribution - by ReneZ - 29-06-2020, 01:53 PM
RE: Vord length distribution - by Anton - 29-06-2020, 02:19 PM
RE: Vord length distribution - by MarcoP - 29-06-2020, 03:33 PM
RE: Vord length distribution - by ReneZ - 29-06-2020, 04:07 PM
RE: Vord length distribution - by -JKP- - 29-06-2020, 04:11 PM
RE: Vord length distribution - by Anton - 29-06-2020, 06:24 PM
RE: Vord length distribution - by MarcoP - 29-06-2020, 07:45 PM
RE: Vord length distribution - by -JKP- - 29-06-2020, 07:07 PM
RE: Vord length distribution - by Alin_J - 29-06-2020, 07:18 PM
RE: Vord length distribution - by -JKP- - 29-06-2020, 11:27 PM
RE: Vord length distribution - by Anton - 29-06-2020, 08:31 PM
RE: Vord length distribution - by Koen G - 29-06-2020, 09:08 PM
RE: Vord length distribution - by Anton - 29-06-2020, 10:30 PM
RE: Vord length distribution - by aStobbart - 29-06-2020, 10:50 PM
RE: Vord length distribution - by Anton - 29-06-2020, 11:36 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  About word length distribution tikonen 32 18,914 09-05-2025, 05:04 PM
Last Post: Bluetoes101

Forum Jump:


Users browsing this thread: 1 Guest(s)