07-05-2026, 10:36 AM
(This post was last modified: 07-05-2026, 10:40 AM by Labyrinthinesecurity.)
Using switchable templates, it looks like we can improve Zattera's slot machine performance:
Metric Before After Change
Generated 3,110 5,524 +2,414
TP types 1,226 1,680 +454
TP tokens 20,454 24,662 +4,208
Precision 0.3942 0.3041 -0.0901
Recall 0.1468 0.2011 +0.0543
F1 0.2139 0.2421 +0.0282
TokCov 0.5491 0.6620 +0.1129
Note that we worked on the IVTFF corpus, not Zattera's homebrew Slot Alphabet.
The improved grammar (F1=0.242) outperforms Zattera's grammar (F1=0.214) when both are scored on our corpus. However, Zattera reports F1=0.270 on his own filtered corpus (~5,105 types), and our grammar was F1-trained on the test corpus while his was not.
Metric Before After Change
Generated 3,110 5,524 +2,414
TP types 1,226 1,680 +454
TP tokens 20,454 24,662 +4,208
Precision 0.3942 0.3041 -0.0901
Recall 0.1468 0.2011 +0.0543
F1 0.2139 0.2421 +0.0282
TokCov 0.5491 0.6620 +0.1129
Note that we worked on the IVTFF corpus, not Zattera's homebrew Slot Alphabet.
The improved grammar (F1=0.242) outperforms Zattera's grammar (F1=0.214) when both are scored on our corpus. However, Zattera reports F1=0.270 on his own filtered corpus (~5,105 types), and our grammar was F1-trained on the test corpus while his was not.
