18-01-2021, 09:33 AM
I have checked the cleaning script. The unusual high number of word repeats is obviously the result of an incomplete cleaning process.
For instance the Wikipedia website You are not allowed to view links. Register or Login to view. contains the following code fragment:
The raw files are cleaned by using the following script You are not allowed to view links. Register or Login to view.
The result is
see You are not allowed to view links. Register or Login to view.
For instance the Wikipedia website You are not allowed to view links. Register or Login to view. contains the following code fragment:
Code:
<table border="1" align="right" cellpadding="4" cellspacing="0" width="300" style="margin: 0 0 1em 1em; background: #f9f9f9; border: 1px #aaaaaa solid; border-collapse: collapse; font-size: 95%;">The raw files are cleaned by using the following script You are not allowed to view links. Register or Login to view.
Code:
delete_uncommon_chars(doc) # Delete characters with freq < .0001
replace (border))? align ((left)|(right)|(center) with ''
replace (cellpadding) with ''
replace (cellspacing) with ''
replace (width style margin em em background) with ''
replace (border collapse collapse) with ''The result is
Code:
f f f border aaaaaa solid font sizesee You are not allowed to view links. Register or Login to view.


![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)