An Artificial Construction
#37
(14-05-2026, 03:03 PM)oshfdk wrote: You are not allowed to view links. Register or Login to view.One thing to note, your suggestion of splitting pinyin by any main vowel is at odds with what I intended and mixing many different B together dilutes the signal.

Agreed.

However, this comment and the comparison of the top left and bottom right tables shows a problem with this approach.  Namely, in order to show that LanguageX "does not behave like Voynichese", you would have to compute the table for every possible spelling system for LanguageX, every possible set of prefix-core-suffix splitting rules, and any choice of the core B that gives sufficient data.

To make the table for Mandarin/pinyin slightly less incomparable to that of Voynichese, you could consider mapping the tone diacritic to a digit 1-4 immediately preceding the vowel, define that digit as the core, and use any of the four digits as the target core "B".  For example, "xiáng" would become "xi2ang" and would be parsed as «xi-2-ang».  Then, if you pick "4" as B, you will still have ~190 word types in that sample file.

Another problem is that the color scheme maps to red every cell whose combination is absent from the file, even if its expected count is well below 1.   The English and Latin tables have lots of red because they have lots of prefixes and suffixes, therefore most prefix-suffix combinations simply have no chance of showing up in the sample text.  The Voynichese tables have little red because there are very few prefixes and suffixes, and thus most combinations can show up.

This problem could be alleviated  by using a color formula like

   (yellow)*alpha*gamma + (blue)*(1-alpha)*gamma + (gray)* (1-gamma)

where alpha is a number in [0,1] that is a function of log(actualCount/expectedCount), as you have now, and gamma is a number that is near 1 when the expected count is large, and near 0 when the expected count is 1 or less.  (But be sure to compute the expectedCount as float, not int.) 

Quote:If we add the pinyin split that you proposed at the lower right corner, it still looks closer to other languages to me

The visual similarity is mostly due to the fact that, with those splitting rules, there are basically only six suffixes that occur in significant numbers: empty, "n", "ng", "o", "i", "u".  For this reason, that table has four columns of almost solid red; and that is  is the main similarity between that table and those of English and Latin.  If we exclude those four columns, the last table looks a lot more similar to Voynichese and less similar to the other languages.

Quote:the distinction between a natural language and Voynichese is very clear.

True if the "a" is understood as "one".  But there are still some 50 monosyllabic languages to try, each with a zillion possible spelling systems and p-c-s parsing rules...

All the best, --stolfi
Reply


Messages In This Thread
An Artificial Construction - by dashstofsk - 25-06-2025, 10:33 AM
RE: An Artificial Construction - by oshfdk - 25-06-2025, 11:21 AM
RE: An Artificial Construction - by RobGea - 25-06-2025, 12:21 PM
RE: An Artificial Construction - by dashstofsk - 25-06-2025, 12:37 PM
RE: An Artificial Construction - by oshfdk - 25-06-2025, 03:04 PM
RE: An Artificial Construction - by dashstofsk - 25-06-2025, 12:58 PM
RE: An Artificial Construction - by RobGea - 25-06-2025, 02:15 PM
RE: An Artificial Construction - by RobGea - 25-06-2025, 02:53 PM
RE: An Artificial Construction - by dashstofsk - 25-06-2025, 03:19 PM
RE: An Artificial Construction - by dashstofsk - 25-06-2025, 03:39 PM
RE: An Artificial Construction - by dashstofsk - 25-06-2025, 04:46 PM
RE: An Artificial Construction - by davidma - 25-06-2025, 05:32 PM
RE: An Artificial Construction - by dashstofsk - 25-06-2025, 08:25 PM
RE: An Artificial Construction - by Jorge_Stolfi - 17-11-2025, 12:39 AM
RE: An Artificial Construction - by ReneZ - 17-11-2025, 09:24 AM
RE: An Artificial Construction - by JoJo_Jost - 13-05-2026, 06:52 AM
RE: An Artificial Construction - by oshfdk - 13-05-2026, 09:54 AM
RE: An Artificial Construction - by dashstofsk - 13-05-2026, 07:56 AM
RE: An Artificial Construction - by JoJo_Jost - 13-05-2026, 08:09 AM
RE: An Artificial Construction - by oshfdk - 13-05-2026, 04:29 PM
RE: An Artificial Construction - by oshfdk - 13-05-2026, 07:57 PM
RE: An Artificial Construction - by Jorge_Stolfi - 13-05-2026, 10:29 PM
RE: An Artificial Construction - by oshfdk - 13-05-2026, 10:59 PM
RE: An Artificial Construction - by Jorge_Stolfi - 13-05-2026, 11:33 PM
RE: An Artificial Construction - by oshfdk - 14-05-2026, 10:06 AM
RE: An Artificial Construction - by nablator - 14-05-2026, 10:59 AM
RE: An Artificial Construction - by oshfdk - 14-05-2026, 11:10 AM
RE: An Artificial Construction - by Jorge_Stolfi - 14-05-2026, 02:17 PM
RE: An Artificial Construction - by oshfdk - 14-05-2026, 03:03 PM
RE: An Artificial Construction - by Jorge_Stolfi - 14-05-2026, 10:24 PM
RE: An Artificial Construction - by oshfdk - 15-05-2026, 12:14 AM
RE: An Artificial Construction - by nintus - 13-05-2026, 07:41 PM
RE: An Artificial Construction - by dashstofsk - 14-05-2026, 03:19 PM
RE: An Artificial Construction - by oshfdk - 14-05-2026, 04:37 PM
RE: An Artificial Construction - by rikforto - 14-05-2026, 08:36 PM
RE: An Artificial Construction - by oshfdk - 14-05-2026, 10:16 PM
RE: An Artificial Construction - by ReneZ - 14-05-2026, 11:42 PM
RE: An Artificial Construction - by oshfdk - 15-05-2026, 12:04 AM
RE: An Artificial Construction - by Jorge_Stolfi - 15-05-2026, 11:33 AM
RE: An Artificial Construction - by Jorge_Stolfi - 15-05-2026, 12:24 AM
RE: An Artificial Construction - by ReneZ - 15-05-2026, 12:27 AM
RE: An Artificial Construction - by oshfdk - 15-05-2026, 08:58 AM
RE: An Artificial Construction - by rikforto - 15-05-2026, 01:41 PM
RE: An Artificial Construction - by oshfdk - 15-05-2026, 11:40 PM
RE: An Artificial Construction - by JoJo_Jost - 16-05-2026, 05:21 AM
RE: An Artificial Construction - by oshfdk - 16-05-2026, 05:22 PM
RE: An Artificial Construction - by Jorge_Stolfi - 16-05-2026, 10:12 PM
RE: An Artificial Construction - by oshfdk - 16-05-2026, 10:46 PM
RE: An Artificial Construction - by ReneZ - 16-05-2026, 05:39 AM
RE: An Artificial Construction - by JoJo_Jost - 16-05-2026, 07:21 AM
RE: An Artificial Construction - by ReneZ - 16-05-2026, 05:45 AM
RE: An Artificial Construction - by ReneZ - 16-05-2026, 08:15 AM
RE: An Artificial Construction - by JoJo_Jost - 16-05-2026, 09:07 AM
RE: An Artificial Construction - by oshfdk - 16-05-2026, 05:32 PM
RE: An Artificial Construction - by RadioFM - 16-05-2026, 06:48 PM
RE: An Artificial Construction - by nablator - 16-05-2026, 07:19 PM
RE: An Artificial Construction - by oshfdk - 16-05-2026, 08:50 PM
RE: An Artificial Construction - by oshfdk - 16-05-2026, 05:56 PM
RE: An Artificial Construction - by nablator - 16-05-2026, 06:32 PM
RE: An Artificial Construction - by nablator - 16-05-2026, 06:59 PM

Forum Jump:


Users browsing this thread: 1 Guest(s)