(26-05-2026, 08:42 PM)Dunsel wrote: You are not allowed to view links. Register or Login to view.Your python generator per your exact specification:
I don't think it's a good idea to code something according to some interpretation of loose specs without clarifying the details.
I never specified how exactly words should be mixed, how random longer words should be generated, and the exact meaning of "occasionally repeat".
Let me make it more specific and yet still manageable to create manually:
1) Given three groups of: common words (A), common word combinations (B) and random long words ( C ) I think it makes sense to never put A and B next to one another and C can go between 1 and 3 times in a row. When selecting from A or B select top of the list more often. Reality check: very easy and simple to follow rule, the mixing order is straightforward: get either one of A or one of B then between 1 and 3 of C, then repeat. Prioritizing top of the list is the same as going down the list and randomly selecting the first word that seems fine, this will naturally give higher probability to higher A/B words.
2) Having just a list of common words the easiest way of generating long words is by trigrams taken from the common words list. Take any word starting bigram ab from existing preset A words and then continue appending one character at a time using another existing trigram, like abc, bcd, then cde, etc, and always finish with a word ending trigram. If at any moment the process is stuck (no non word ending trigram is available), it's ok to continue using a bigram. So, given "qokedy chol kotaiin chedy olkedy daraly" we can generate words like cholkedy. Reality check: given a paper with the list of common words this is a simple task, even though it may require a bit of thinking to avoid failing step 3. Edit: This step can be sped up significantly if a list of trigrams is prepared separately from A as a table. This won't even be anachronistic, as far as I understand, the idea of creating tables to enumerate some elements was a very common practice in computus manuscripts.
3) You idea of repeating one of last 10 words 25% of the time is a good one and would work nicely when writing, but I would adjust this a bit. Remembering my own list of restrictions for what a scribe could comfortably do, I'd expand the list of last words to 20. Additionally, if we want it to look like a natural language, we should never generate new words that either belong to A or belong to the last 20 of C. And we better avoid repeating the same word twice, so we'll never copy a word that will repeat the last word we added. Reality check: sometimes can be tedious, but generally not very hard. Given there are only 20 preset common words it's easy to remember them and then to tell apart generated and preset words, and when creating a new word it's not very hard to guide the process to avoid generating a word from A or a recently used word.
Code:
# mixwords_literal_generator.py
# No external files required.
#
# Literal version of:
# 1) write down 20-30 common words
# 2) create 10-20 random common pairings from these words
# 3) choose either a common word or a common combination
# 4) after that, add 1-3 bigram-seeded, trigram-built longer words
# 5) once 20 long words exist in the pipeline, repeat one of them 25% of the time
#
# No mutation.
# No ledger.
# No syllables.
# No hidden morphology.
import argparse
import random
from collections import Counter
SEED = 42
PAGES = 100
WORDS_PER_PAGE = 95
WORDS_PER_LINE = 8
COMMON_COMBINATION_MIN_COUNT = 10
COMMON_COMBINATION_MAX_COUNT = 20
COMMON_COMBINATION_WORDS = 2
LONG_WORD_MIN_LEN = 5
LONG_WORD_MAX_LEN = 11
LONG_WORD_REPEAT_CHANCE = 0.25
RECENT_LONG_WORD_WINDOW = 20
LONG_WORDS_PER_MIX_MIN = 1
LONG_WORDS_PER_MIX_MAX = 3
COMMON_WORD_CHANCE = 0.5
COMMON_RANK_TOP_WEIGHT = 3.0
COMMON_RANK_BOTTOM_WEIGHT = 1.0
MAX_TRIGRAM_WORD_ATTEMPTS = 64
MAX_MIXED_LONG_WORD_ATTEMPTS = MAX_TRIGRAM_WORD_ATTEMPTS * 8
COMMON_WORDS = [
"daiin", "ol", "or", "qoky", "dary",
"chol", "qokair", "sair", "opdor", "ockhol",
"chedy", "qokchy", "ofain", "ykal", "oteedy",
"sheey", "otey", "cphol", "sholdaly", "cheol",
"saral", "doroldal", "qolky", "qokeedy", "chdy",
]
COMMON_WORDS_ENGLISH = [
"the", "and", "are", "a", "this",
"is", "unless", "more", "each",
"together", "well", "anything", "right", "wrong",
"stay", "move", "wish", "however", "two",
"one", "four", "because", "always", "trouble",
]
COMMON_COMBINATIONS = []
COMMON_SOURCE_WORD_TOKENS = []
COMMON_SOURCE_WORD_SET = set()
COMMON_WORD_SET = set()
COMMON_WORD_WEIGHTS = []
COMMON_COMBINATION_WEIGHTS = []
WORD_START_BIGRAMS = []
BIGRAM_OPTIONS = {}
TRIGRAMS = []
TRIGRAM_OPTIONS = {}
NONFINAL_TRIGRAMS = set()
FINAL_BIGRAMS = set()
FINAL_TRIGRAMS = set()
WORD_FINAL_SUFFIXES = {}
LONG_WORD_FALLBACKS = []
def make_common_combinations():
target_count = min(
random.randint(
COMMON_COMBINATION_MIN_COUNT,
COMMON_COMBINATION_MAX_COUNT,
),
len(COMMON_WORDS) * (len(COMMON_WORDS) - 1),
)
combinations = []
seen = set()
while len(combinations) < target_count:
# Avoid A A pairs like "daiin daiin".
pair = tuple(random.sample(COMMON_WORDS, COMMON_COMBINATION_WORDS))
if pair in seen:
continue
seen.add(pair)
combinations.append(list(pair))
return combinations
def make_rank_weights(items):
count = len(items)
if count <= 1:
return [COMMON_RANK_BOTTOM_WEIGHT] * count
step = (
COMMON_RANK_BOTTOM_WEIGHT - COMMON_RANK_TOP_WEIGHT
) / (count - 1)
return [
COMMON_RANK_TOP_WEIGHT + (step * index)
for index in range(count)
]
def weighted_choice(items, weights):
return random.choices(items, weights=weights, k=1)[0]
def initialize_common_sources():
global COMMON_COMBINATIONS
global COMMON_SOURCE_WORD_TOKENS
global COMMON_SOURCE_WORD_SET
global COMMON_WORD_SET
global COMMON_WORD_WEIGHTS
global COMMON_COMBINATION_WEIGHTS
global WORD_START_BIGRAMS
global BIGRAM_OPTIONS
global TRIGRAMS
global TRIGRAM_OPTIONS
global NONFINAL_TRIGRAMS
global FINAL_BIGRAMS
global FINAL_TRIGRAMS
global WORD_FINAL_SUFFIXES
global LONG_WORD_FALLBACKS
COMMON_COMBINATIONS = make_common_combinations()
COMMON_WORD_SET = set(COMMON_WORDS)
COMMON_WORD_WEIGHTS = make_rank_weights(COMMON_WORDS)
COMMON_COMBINATION_WEIGHTS = make_rank_weights(COMMON_COMBINATIONS)
COMMON_SOURCE_WORD_TOKENS = COMMON_WORDS + [
word for combo in COMMON_COMBINATIONS for word in combo
]
COMMON_SOURCE_WORD_SET = set(COMMON_SOURCE_WORD_TOKENS)
WORD_START_BIGRAMS = [
word[:2] for word in COMMON_SOURCE_WORD_TOKENS if len(word) >= 2
]
TRIGRAMS = [
word[i:i + 3]
for word in COMMON_SOURCE_WORD_TOKENS
for i in range(len(word) - 2)
]
BIGRAM_OPTIONS = {}
TRIGRAM_OPTIONS = {}
NONFINAL_TRIGRAMS = set()
FINAL_BIGRAMS = set()
FINAL_TRIGRAMS = set()
WORD_FINAL_SUFFIXES = {}
for word in COMMON_SOURCE_WORD_TOKENS:
if len(word) >= 2:
FINAL_BIGRAMS.add(word[-2:])
if len(word) >= 3:
FINAL_TRIGRAMS.add(word[-3:])
for i in range(len(word) - 1):
bigram = word[i:i + 2]
BIGRAM_OPTIONS.setdefault(bigram[0], []).append(bigram)
for i in range(len(word) - 2):
trigram = word[i:i + 3]
TRIGRAM_OPTIONS.setdefault(trigram[:2], []).append(trigram)
if i < len(word) - 3:
NONFINAL_TRIGRAMS.add(trigram)
suffix = word[i:]
WORD_FINAL_SUFFIXES.setdefault(suffix[:2], []).append(suffix)
for prefix, suffixes in WORD_FINAL_SUFFIXES.items():
WORD_FINAL_SUFFIXES[prefix] = sorted(suffixes, key=len)
LONG_WORD_FALLBACKS = [
word for word in COMMON_SOURCE_WORD_SET if len(word) >= LONG_WORD_MIN_LEN
]
def continue_with_bigram(word, target_length, disallowed_words):
next_bigrams = BIGRAM_OPTIONS.get(word[-1], [])
allowed_bigrams = [
bigram
for bigram in next_bigrams
if (
len(word) + 1 < target_length
or (
word + bigram[1] not in disallowed_words
and (word + bigram[1])[-2:] in FINAL_BIGRAMS
and (word + bigram[1])[-3:] in FINAL_TRIGRAMS
)
)
]
if not allowed_bigrams:
return None
return word + random.choice(allowed_bigrams)[1]
def make_trigram_long_word(disallowed_words=None):
target_length = random.randint(LONG_WORD_MIN_LEN, LONG_WORD_MAX_LEN)
fallback = None
disallowed_words = set(disallowed_words or ())
for _ in range(MAX_TRIGRAM_WORD_ATTEMPTS):
word = random.choice(WORD_START_BIGRAMS)
completed = False
while len(word) < target_length:
suffixes = WORD_FINAL_SUFFIXES.get(word[-2:], [])
exact_suffixes = [
suffix
for suffix in suffixes
if len(word) + len(suffix) - 2 == target_length
and word + suffix[2:] not in disallowed_words
]
if exact_suffixes:
word += random.choice(exact_suffixes)[2:]
completed = True
break
next_trigrams = TRIGRAM_OPTIONS.get(word[-2:])
allowed_trigrams = [
trigram for trigram in next_trigrams if trigram in NONFINAL_TRIGRAMS
] if next_trigrams else []
if allowed_trigrams:
word += random.choice(allowed_trigrams)[2]
continue
word = continue_with_bigram(word, target_length, disallowed_words)
if word is None:
break
if len(word) == target_length:
completed = True
break
if not completed or len(word) < LONG_WORD_MIN_LEN:
continue
if fallback is None:
fallback = word
if word not in COMMON_SOURCE_WORD_SET:
return word
if fallback is not None:
return fallback
allowed_fallbacks = [
word for word in LONG_WORD_FALLBACKS if word not in disallowed_words
]
if allowed_fallbacks:
return random.choice(allowed_fallbacks)
return random.choice(LONG_WORD_FALLBACKS)
def append_recent_long_word(recent_long_words, word):
recent_long_words.append(word)
if len(recent_long_words) > RECENT_LONG_WORD_WINDOW:
recent_long_words.pop(0)
def make_mixed_long_word(recent_long_words, previous_word=None):
if (
len(recent_long_words) >= RECENT_LONG_WORD_WINDOW
and random.random() < LONG_WORD_REPEAT_CHANCE
):
repeat_candidates = [
word for word in recent_long_words if word != previous_word
]
if repeat_candidates:
return random.choice(repeat_candidates)
fallback = None
recent_long_word_set = set(recent_long_words)
disallowed_words = COMMON_WORD_SET | recent_long_word_set
for _ in range(MAX_MIXED_LONG_WORD_ATTEMPTS):
word = make_trigram_long_word(disallowed_words)
if word in COMMON_WORD_SET or word in recent_long_word_set:
continue
if fallback is None:
fallback = word
if word != previous_word:
return word
if fallback is not None and fallback != previous_word:
return fallback
distinct_fallbacks = [
word
for word in LONG_WORD_FALLBACKS
if (
word != previous_word
and word not in COMMON_WORD_SET
and word not in recent_long_word_set
)
]
if distinct_fallbacks:
return random.choice(distinct_fallbacks)
if fallback is not None:
return fallback
raise RuntimeError(
"Unable to generate a valid long word that is not a common word "
"or one of the recent long words."
)
def append_marked_word(words, marker, word):
words.append((marker, word))
def generate_words(total_words, recent_long_words):
words = []
while len(words) < total_words:
if random.random() < COMMON_WORD_CHANCE:
append_marked_word(
words,
"A",
weighted_choice(COMMON_WORDS, COMMON_WORD_WEIGHTS),
)
else:
for word in weighted_choice(
COMMON_COMBINATIONS,
COMMON_COMBINATION_WEIGHTS,
):
if len(words) >= total_words:
break
append_marked_word(words, "B", word)
long_word_count = random.randint(
LONG_WORDS_PER_MIX_MIN,
LONG_WORDS_PER_MIX_MAX,
)
for _ in range(long_word_count):
if len(words) >= total_words:
break
previous_word = words[-1][1] if words else None
word = make_mixed_long_word(recent_long_words, previous_word)
append_marked_word(words, "C", word)
append_recent_long_word(recent_long_words, word)
return words[:total_words]
def char_ngrams(words, n):
counts = Counter()
for word in words:
if len(word) < n:
continue
for i in range(len(word) - n + 1):
counts[word[i:i + n]] += 1
return counts
def analyze(all_words):
token_count = len(all_words)
type_count = len(set(all_words))
word_freq = Counter(all_words)
char_bigram_counts = char_ngrams(all_words, 2)
char_trigram_counts = char_ngrams(all_words, 3)
print()
print("=" * 60)
print("GLOBAL WORD STATISTICS")
print("=" * 60)
print()
print(f"Tokens : {token_count}")
print(f"Types : {type_count}")
print(f"TTR : {type_count / token_count:.4f}")
print(f"Hapax : {sum(1 for c in word_freq.values() if c == 1)}")
print()
print("Top 25 word tokens")
print()
for word, count in word_freq.most_common(25):
print(f"{word:15} {count}")
print()
print("=" * 60)
print("CHARACTER BIGRAMS")
print("=" * 60)
print()
for bg, count in char_bigram_counts.most_common(30):
print(f"{bg:5} {count}")
print()
print("=" * 60)
print("CHARACTER TRIGRAMS")
print("=" * 60)
print()
for tg, count in char_trigram_counts.most_common(30):
print(f"{tg:5} {count}")
def format_token(entry, debug=False):
marker, word = entry
if debug:
return f"{marker} {word}"
return word
def print_page(page_num, page, debug=False):
print()
print("=" * 60)
print(f"PAGE {page_num}")
print("=" * 60)
print()
for i in range(0, len(page), WORDS_PER_LINE):
print(" ".join(format_token(entry, debug) for entry in page[i:i + WORDS_PER_LINE]))
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--debug", action="store_true")
parser.add_argument("--english", action="store_true")
args = parser.parse_args()
if args.english:
global COMMON_WORDS
COMMON_WORDS = COMMON_WORDS_ENGLISH
random.seed(SEED)
initialize_common_sources()
recent_long_words = []
total_words = PAGES * WORDS_PER_PAGE
all_entries = generate_words(total_words, recent_long_words)
all_words = [word for _, word in all_entries]
for page_num in range(1, PAGES + 1):
start = (page_num - 1) * WORDS_PER_PAGE
end = start + WORDS_PER_PAGE
page = all_entries[start:end]
print_page(page_num, page, debug=args.debug)
analyze(all_words)
if __name__ == "__main__":
main()The TTR stats:
Tokens : 9500
Types : 727
TTR : 0.0765
Hapax : 223
What it looks like:
oteedy chedary qoldaly cholky qoky saiiral chedy cpholky
chdykaiin chedary ol daiin chdoroldal otey opdorolky ockhol
ykair otey shedal chedaiin doroldal dorolkchol sheey qokain
orolky opdor daiin qokal oteedor oteedykaiin qoky daiin
chdaiiiral dorolkchol qokal dary dorolkchol sair otey daraly
cholky ykaiiral chdy chol dalykal qoldarary cholkykal oteedy
sholdaly choldaldaly oroldaiin daraly chdy chol oldalykal oroldalky
qokeedaly qolky sholdal oldaiiral doroldal doroldaly ockhol dorolkchol
qokeedy opdor oroldaraiin ockholdaly qokal dary oldaldaly qokeey
daiin qokal sair daldalykair oteey sholkeey chdy choldaiin
sholdaly ykal cpholkykal dary oteedykaiin dary cholky cpholky
qokain qokeedy ockholkair cpholkykal ockholky cheol choldal
However, I also wouldn't choose the strange fixed order sequences of the Voynichese if I was to create a fake language. For comparison, the same algorithm when seeded with some English words:
The TTR stats:
Tokens : 9500
Types : 1296
TTR : 0.1364
Hapax : 595
What it looks like:
one the troneach anythe and thingethe one moreach
twover wroneach a onlessshing wrong more alwayshess foublestach
righell one the howell tronghthe alwaysshis and the
becaushis eacaustay twaything right thinythis always anytheach more
wish ongethe are unless anythevever howevething two wrong
anytheach wroneserong more this wever wrong more aronghis
onleseress are therongethe morouble moreless each mover howeless
are unless thing alwaysever stay tronless howevething always
four arong two wrong howevever right anytheach twong
anythe is tronless wrong more wrouble because trouble
tronlesever is eachething thether thecause and the unlestay
because trouble morouble and issereless wrouble thecausstay
Add a simple substitution on top of this (replace each English letter with an invented alphabet) and you have what looks like a plausible text in some exotic language.