(17-10-2025, 07:32 AM)quimqu wrote: You are not allowed to view links. Register or Login to view.the punctuation for language A is 3 times the punctuation for language B
Punctuation? Do you mean cumulative weight or number of words or some other metric?
Since the detected language at the page and paragraph level doesn't always match Currier's assessment, would it be useful to re-train the models using the detected languages as input until (at the first iteration hopefully) the result is stable, i.e. the detection matches the training? And maybe drop the page model, if the paragraph model is more detailed and accurate.
It would be interesting to know how reliable is the classification by a "stable" model as either A or B, and how many "in-between" (undecidable) paragraphs there are. If each paragraph is strongly A or B, we have a discontinuous process (like a change in plaintext language), if not there could be a continuous drift between A and B and possibly more "languages" or "dialects". Which hypothesis is more likely?