28-01-2025, 12:10 PM
(This post was last modified: 28-01-2025, 12:14 PM by 008348dc760f858fd668476b75fb6f.
Edit Reason: update
)
Yep, that's my main concern, the scoring function and if I'm inadvertently introducing a pattern where one doesn't exist. I've tried to mitigate this as explained in the post, using the set of words in the first folio and dividing the score by the total words in the 2nd folio.
I think I'll run this again with a modified scoring function that doesn't count how many times a word appears in the 2nd folio just whether it appears or not. That way I don't need to remove the over-represented words (my opinion is that they represent *something* even if it's just a null word).
I'm not super convinced that the differing output based on the "hand" writing would contradict the idea, I would expect that the scribes are using different settings on the encoding mechanism, but, I think it would be good for me to add some indication of the likely scribe to the sheet to see what, if anything, it brings up.
==update==
It looks like I was using the updated scoring idea already:
I think I'll run this again with a modified scoring function that doesn't count how many times a word appears in the 2nd folio just whether it appears or not. That way I don't need to remove the over-represented words (my opinion is that they represent *something* even if it's just a null word).
I'm not super convinced that the differing output based on the "hand" writing would contradict the idea, I would expect that the scribes are using different settings on the encoding mechanism, but, I think it would be good for me to add some indication of the likely scribe to the sheet to see what, if anything, it brings up.
==update==
It looks like I was using the updated scoring idea already:
Code:
folioScore = sum([list(set(compFolio)).count(w) for w in set(baseFolioWords)])
ret.append(str( folioScore / (len(set(baseFolioWords)) + len(set(compFolio))) ))