Could it be that this represents a system for generating music?

[Jorge_Stolfi]

Sorry if it was explained somewhere but I can't find it: what is a "ur-book"?

That "ur-" is an old-fashioned academic prefix borrowed from German to mean "original", "primeval", "archetypal" etc.  Like "urtext" for the hypothetical master text from which several surviving copies derived, or "ur-language" ("ursprache") for a hypothetical proto-language, etc.  

I have used "ur-book' to mean the pre-existing book from which each section of the VMS was copied/translated/transcribed by the VMS Author (the person who decided to create Beinecke MS 408, devised the script, defined the text and illustrations, etc).  Because "original book" could be interpreted as the draft written by the VMS Author, before it was copied to vellum by the Scribe; or to the copy as just produced, before all the subsequent binding, painting, fading, spilling, retouching, "marginalizing", etc.

All the best, --stolfi