Stochastic Models in Population Genetics: Genealogy and Genetic Differentiation in Structured Populations

Abstract

The theory of probability and stochastic processes is applied to a current issue in population genetics, namely that of genealogy and genetic differentiation in subdivided populations. It is proved that under a reasonable model for reproduction and migration, the ancestral process of a sample from a subdivided population converges weakly, as the subpopulation sizes tend to infinity, to a continuous-time Markov chain called the "structured coalescent". The moment-generating function, the mean and the cond moment of the time since the most recent common ancestor (called the "coalescence time") of a pair of genes are calculated explicitly for a range of models of population structure. The value of Wright's coefficient FST, which serves as a measure of the subpopulation differentiation and which can be related to the coalescence times of pairs of genes sampled within or among subpopulations, is calculated explicitly for various models of population structure. It is shown that the dependence of FST on the mutation rate may be more marked than is generally believed, particularly when gene flow is restricted to an essentially one-dimensional habitat with a large number of subpopulations. Several more general results about genealogy and subpopulation differentiation are proved. Simple relationships are found between moments of within and between population coalescence times. Weighting each subpopulation by its relative size, the asymptotic behaviour of FST at large mutation rates is independent of the details of population structure. Two sets of symmetry conditions on the population structure are found for which the mean coalescence time of a pair of genes from a single subpopulation is independent of the migration rate and equal to that of two individuals from a panmictic population of the same total size. Under graph-theoretic conditions on the population structure, there is a uniform relationship between the FST value of a pair of neighbouring subpopulations, in the limit of zero mutation rate, and the migration rate