Need advice for testing of hypotheses related to the self-citation method

[Mauro]

I'm never sure how to determine these priors.

Indeed, that is the weak spot of Bayes's formula.  The result is highly dependent on the prior probabilties, and they are inherently subjective.

To apply Bayes one does not need a prior probability at all. Indeed, discriminating between prior and evidence is ihmo unnecessary and misleading. Start from a condition of absolutely zero knowledge, that is to say P(H) = 0.5 and P(notH) = 0.5 This is the only 'prior' one needs, and it's trivial. Then start factoring in every single piece of knowledge you have: they are all evidences now.

But it's true that, many times, also the probabilities of the evidence vs. H and vs. notH are hard to pin down, even broadly. There are however some methods which can be used (ie. Laplace's rule of succession, the reference class method) which minimize subjectivity, but it's not the case to discuss them (nor Bayesian logic in general) here.