25-01-2018, 09:09 PM
(25-01-2018, 12:41 PM)DonaldFisk wrote: You are not allowed to view links. Register or Login to view....
Despite citing Gordon Rugg's paper, it didn't seem to occur to the authors to test their method on text known to be meaningless.
This is a very good point.
Part of the method should be testing the algorithms on meaningless text to see how the software evaluates something that is not in any specific language. Even meaningless text can be generated according to a wide variety of algorithms... and still be meaningless.
It would not be easy to do this. How does one algorithmically distinguish hundreds of different languages AND, at the same time, filter out those with no useful content? This will probably keep the developers busy for some time.
----------
Even so, if it is known that a sample of text is meaningful, but the underlying language is not known, having a computer algorithm presort it and give some rational guesses as to what languages it might be would be a great tool for historical research.
