I thought about how to predict the start and end chunks. The result is as follows (if there is a flaw in my reasoning, please let me know):
A script reads a file in which each word is divided into start, middle, and end chunks. These parts are separated from each other. The middle chunks of each line are then converted into binary vectors to calculate the similarity between these lines. For each line, only those lines that achieve a certain minimum similarity (threshold value 0.6) are taken into account. This forms a group of similar lines. From this group, the most probable start and end chunks are predicted using weighted voting. Result:
Median group size: 25.0
Average similarity in groups: 0.636
Min similarity in groups: 0.600 (Threshold)
Max similarity in groups: 0.741
Exact matches (Actual == Predicted): 81 / 3885
Here is the output file: You are not allowed to view links. Register or Login to view.
The corpus used: You are not allowed to view links. Register or Login to view.
A script reads a file in which each word is divided into start, middle, and end chunks. These parts are separated from each other. The middle chunks of each line are then converted into binary vectors to calculate the similarity between these lines. For each line, only those lines that achieve a certain minimum similarity (threshold value 0.6) are taken into account. This forms a group of similar lines. From this group, the most probable start and end chunks are predicted using weighted voting. Result:
Median group size: 25.0
Average similarity in groups: 0.636
Min similarity in groups: 0.600 (Threshold)
Max similarity in groups: 0.741
Exact matches (Actual == Predicted): 81 / 3885
Here is the output file: You are not allowed to view links. Register or Login to view.
The corpus used: You are not allowed to view links. Register or Login to view.
