A One-Page Ledger Method for Generating Voynich-Like Text

[Dunsel]

In fact, this way of creating a fake language is much simpler and produces a much more plausible result:
1) create a list of common words, about 20-30, write them down
2) create 10-20 common combinations of these common words, write them down
3) start writing mixing common words and common word combinations with random longer words, which would occasionally repeat
4) done

There is nothing wrong for a text in a natural language to have some repetitions, it's expected. In some styles of texts (recipes, log entries) it's even mandatory. There is nothing wrong with that, there is no need to fight against this and the result looks very much like an unknown language. The statistics would differ significantly from Voynichese, but maybe they would be closer to real languages, and what is more, no-one in the XV century would go over the text with a scientific calculator and measure token type ratios.

Your python generator per your exact specification:

And I hope you don't mind but I did have to toss a few rules in there like... the alphabet to make your random words with. I used the Voynich alphabet.  A max word length so it just didn't keep randomly jamming letters together.  A minimum word length for your 'longer words' and some weighting so it didn't just keep repeating your longer random words.  Number of pages, words per line, words per page, things python can't just guess at very well.  I even tossed in a seed so you can reproduce it every time the same way

Code:

# oshfdk_literal_generator.py # No external files required. # # Literal version of: # 1) create 20-30 common words # 2) create 10-20 common combinations # 3) mix common words + common combinations + random longer words # 4) random longer words occasionally repeat # # No mutation. # No ledger. # No syllables. # No hidden morphology. import random from collections import Counter SEED = 42 PAGES = 100 WORDS_PER_PAGE = 95 WORDS_PER_LINE = 8 # Standard EVA-ish alphabet, sorted alphabetically. VOYNICH_ALPHABET = "acdefhiklnoqrstxy" LONG_WORD_MIN_LEN = 5 LONG_WORD_MAX_LEN = 12 LONG_WORD_REPEAT_CHANCE = 0.25 RECENT_LONG_WORD_WINDOW = 10 COMMON_WORDS = [     "daiin", "dain", "ol", "or", "chol",     "chedy", "qokain", "qokeedy", "qotedy", "otedy",     "shey", "cthey", "cthol", "shol", "chor",     "dair", "qokair", "saiin", "aiin", "okain",     "sary", "okol", "qol", "qokal", "chdy", ] COMMON_COMBINATIONS = [     ["qokain", "daiin"],     ["chol", "chedy"],     ["qokeedy", "qokedy"],     ["ol", "chedy"],     ["dain", "chol"],     ["shey", "qokain"],     ["cthey", "daiin"],     ["shol", "chor"],     ["qotedy", "qokain"],     ["otedy", "ol"],     ["saiin", "daiin"],     ["qokair", "dair"],     ["chol", "daiin"],     ["cthol", "chedy"],     ["or", "chol"], ] def make_random_long_word():     length = random.randint(LONG_WORD_MIN_LEN, LONG_WORD_MAX_LEN)     return "".join(random.choice(VOYNICH_ALPHABET) for _ in range(length)) def generate_page():     page = []     recent_long_words = []     while len(page) < WORDS_PER_PAGE:         mode = random.choice(["common_word", "combination", "long_word"])         if mode == "common_word":             page.append(random.choice(COMMON_WORDS))         elif mode == "combination":             combo = random.choice(COMMON_COMBINATIONS)             page.extend(combo)         else:             if recent_long_words and random.random() < LONG_WORD_REPEAT_CHANCE:                 word = random.choice(recent_long_words)             else:                 word = make_random_long_word()                 recent_long_words.append(word)                 if len(recent_long_words) > RECENT_LONG_WORD_WINDOW:                     recent_long_words.pop(0)             page.append(word)     return page[:WORDS_PER_PAGE] def char_ngrams(words, n):     counts = Counter()     for word in words:         if len(word) < n:             continue         for i in range(len(word) - n + 1):             counts[word[i:i + n]] += 1     return counts def analyze(all_words):     token_count = len(all_words)     type_count = len(set(all_words))     word_freq = Counter(all_words)     char_bigram_counts = char_ngrams(all_words, 2)     char_trigram_counts = char_ngrams(all_words, 3)     print()     print("=" * 60)     print("GLOBAL WORD STATISTICS")     print("=" * 60)     print()     print(f"Tokens : {token_count}")     print(f"Types  : {type_count}")     print(f"TTR    : {type_count / token_count:.4f}")     print(f"Hapax  : {sum(1 for c in word_freq.values() if c == 1)}")     print()     print("Top 25 word tokens")     print()     for word, count in word_freq.most_common(25):         print(f"{word:15} {count}")     print()     print("=" * 60)     print("CHARACTER BIGRAMS")     print("=" * 60)     print()     for bg, count in char_bigram_counts.most_common(30):         print(f"{bg:5} {count}")     print()     print("=" * 60)     print("CHARACTER TRIGRAMS")     print("=" * 60)     print()     for tg, count in char_trigram_counts.most_common(30):         print(f"{tg:5} {count}") def print_page(page_num, page):     print()     print("=" * 60)     print(f"PAGE {page_num}")     print("=" * 60)     print()     for i in range(0, len(page), WORDS_PER_LINE):         print(" ".join(page[i:i + WORDS_PER_LINE])) def main():     random.seed(SEED)     all_words = []     for page_num in range(1, PAGES + 1):         page = generate_page()         all_words.extend(page)         print_page(page_num, page)     analyze(all_words) if __name__ == "__main__":     main()

Example output:

============================================================

PAGE 1
============================================================

alkkfe scadik sktlahso qokeedy qokedy chdy chol chedy
cthol cthey otedy ol chol daiin qokal qotedy
qokain cthol saiin cthol chedy idckndkerl saiin daiin
qokeedy qokedy shey qokain okol qokair dair khtrlko
qokeedy shey dain chol qokain ixrtflfkls fyxdcefh rrtyla
oenshtaly qokair sary cthol chedy fqhyaoxa cthey ol
chedy qokeedy xdffxh qotedy qokain inrqtyte qokeedy shey
aiin ixrtflfkls ol dcodyklx saiin qokal xsieesqs shol
chor ixrtflfkls cthol kiitfs qotedy ol chedy chor
xsieesqs ol chedy shol chor ol chedy or
chol chedy qokain daiin dain chol dain chol
qokair dair finiccoccxyy dain oenshtaly okain okol

============================================================
PAGE 50
============================================================

chol daiin aynxasdcici aiin qokain daiin qokair dair
aiin qotedy qokain cthol chol aynxasdcici qotedy qokain
qotedy qokain qokeedy qokedy chdy olrentat ihecreiinin shol
chor sary olrentat qokair otedy qokain shey dair
trhxxnkdea okain daiin or chol otedy cthey daiin
ecqiisq qotedy saiin daiin otedy or qokeedy qokedy
shol chor or xhaed ecqiisq axdfidrra dair cthol
chedy cthey daiin daiin lnxila ktacsxkc shey qokain
okain shol chor qotedy qokain cthey oqshrqt chol
daiin qokeedy qokedy sqsdlfoqtthc qotedy qokain axdfidrra firdsey
trhxxnkdea sqsdlfoqtthc oxeoqryr aiin qokair dair qokair dair
qokeedy iayexfd rnieiaaa chol chedy chedy oqshrqt

============================================================

PAGE 100
============================================================

qsqfrayh qokeedy qokedy ldfyyrsedffi ol saiin daiin qsqfrayh
chol chedy aiin chol daiin onrotll or chol
qokair dair chedy saiin daiin dain chol or
chol or onrotll dirtiyaofc saiin itsyne xdlrfhadl kolceeht
saiin cthol qokain qokair or chol or chol
shol chor qokain daiin xsektrfqaxs shexfc qotedy qokain
cneyrkckhrn lstnedrcoka chol daiin chol daiin lstnedrcoka aknfoetcedqe
cthol shexfc qokain daiin dain chol or or
chol ol shexfc xsektrfqaxs cthol chol or otedy
ol ytqly chol shexfc aknfoetcedqe ol chedy asqtko
dair shey qokain cthey daiin cneyrkckhrn daiin qokal
otedy chdy chol xdlrfhadl akotexxdxaic cthey daiin


Report:

Code:

============================================================ GLOBAL WORD STATISTICS ============================================================ Tokens : 9500 Types  : 1800 TTR    : 0.1895 Hapax  : 1330 Top 25 word tokens chol            748 daiin          736 chedy          589 qokain          530 ol              427 qokeedy        289 or              277 saiin          276 otedy          275 cthol          268 dair            268 shey            247 cthey          246 qokair          246 shol            242 qotedy          231 chor            230 dain            227 qokedy          185 aiin            107 qokal          98 okol            93 sary            89 chdy            84 okain          80 ============================================================ CHARACTER BIGRAMS ============================================================ ai    2538 in    2011 ol    1901 dy    1729 ch    1711 qo    1704 ed    1632 ok    1615 ho    1546 da    1291 ii    1179 he    1136 ka    1029 ir    580 sh    573 te    571 ot    569 th    561 or    561 ey    557 ct    550 ke    538 sa    433 ee    346 ar    179 al    159 ko    154 ry    147 hd    139 fa    85 ============================================================ CHARACTER TRIGRAMS ============================================================ edy  1571 qok  1348 hol  1260 dai  1237 iin  1126 aii  1122 cho  986 oka  961 kai  863 ain  839 hed  594 che  591 air  522 cth  516 ote  515 ted  510 hey  493 oke  485 eed  293 kee  290 sai  277 tho  268 she  256 the  248 sho  244 qot  232 hor  230 ked  190 kal  102 kol  96

Your zipf curve:

   

Your word length distribution chart.

   


Oh, and for that run, some more stats I had codex retrieve.

Fixed starting vocabulary: 26 words
Random-long word occurrences: 2,342
Distinct random-long words: 1,774
Random-long words that repeated: 444
Single-use random-long words: 1,330
Extra repeat events beyond first use: 568
Highest repeat count for one random word: 5