Inferring demography with LD
As described in the linkage disequilibrium and LD Parsing sections, we use a family of normalized LD and heterozygosity statistics to compare between model expectations and data. We optimize demographic model parameters to find the expected binned LD and heterozygosity statistics that maximize a composite likelihood over all pairs of SNPs and recombination bins.
In this section, we’ll describe the likelihood framework, how to define
demographic models that can be used in inference, how to run optimization using
moments’ built-in inference functions, and how to compute confidence
intervals. We include a short example, following the parsing of data simulated under an isolation-with-migration
model, to illustrate the main features and options.
Likelihood framework
For a given recombination distance bin indexed by \(i\), we have a set of
computed LD statistic means \(D_i\) from data along with the
variance-covariance matrix \(\Sigma_i\) as returned by
moments.LD.Parsing.bootstrap_data. We assume a multivariate Guassian
likelihood function, so that a model parameterized by \(\Theta\) that has
expected statistics \(\mu_i(\Theta)\) has likelihood
The likelihood is computed similarly for heterozygosity statistics, given their variance-covariance matrix. Then the composite likelihood of two-locus data across recombination bins and single-locus heterozygosity (indexed by \(i=n+1\) where \(n\) is the total number of recombination bins), is
In practice, we work with the log of the likelihood, so that products turn to sums and we can drop constant factors:
As the data \(\{D_i, \Sigma_i\}\) is fixed, we search for the model parameters \(\Theta\) that provide \(\{\mu_i\}\) that maximizes \(\log\mathcal{L}\).
Defining demographic models
There are a handful of built-in demographic models for one-, two-, and
three-population scenarios that can be used in inference (see here). However, these are far from comprehensive and it is
likely that custom demographic models will need to be written for a given
inference problem. For inspiration, moments.LD.Demographics1D,
Demographics2D, and Demographics3D can be used as starting points and
as illustrations of how to structure model functions.
Demographic models all require a params positional argument and rho and
(optionally) theta keyword arguments. theta, the population-size scaled
mutation rate, does not play a role in inference using relative statistics, as
the mutation rate cancels in \(\sigma_d^2\)-type statistics.
For example, the IM model we simulated data under in the LD Parsing section could be parameterized as
def model_func(params, rho=None, theta=0.001):
nu0, nu1, T, M = params
y = moments.LD.Numerics.steady_state([1], rho=rho, theta=theta)
y = moments.LD.LDstats(y, num_pops=1)
y = y.split(0)
y.integrate([nu0, nu1], T, m=[[0, M], [M, 0]], rho=rho, theta=theta)
return y
In the input demographic model to the simulations, we had the ancestral
effective population size as 10,000, the size of deme0 was 2,000, and the size
of deme1 was 20,000. The populations split 1,500 generations ago, and exchanged
migrants symmetrically at a rate of 0.0001 per-generation. Converted into
genetic units, nu0 = 0.2, nu1 = 2, T=1500 / 2 / 10000 = 0.075, and
M = 2 * 10000 * 0.0001 = 2.0.
Running optimization
Optimization with moments.LD, much like moments optimization with the
SFS, includes a handful functions that serve as wrappers for scipy
optimizers with options specific to working with LD statistics. The two primary
functions in moments.LD.Inference are
optimize_log_fmin: Uses the downhill simplex algorithm on the log of the parameters.optimize_log_powell: Uses the modified Powell’s method, which optimizes slices of parameter space sequentially.
Each optimization method accepts the same arguments. Required positional arguments are
p0: The initial guess for the parameters inmodel_func.data: Structured as a list of lists of data means and data var-cov matrices. I.e.,data = [[means[0], means[1], ...], [varcovs[0], varcovs[1], ...]], with the final entry of the lists the means and varcovs of the heterozygosity statistics.model_func: The demographic model to be fit (see above section). Importantly, this is a list, where the first entry is the LD model, which is always used, and the optional second entry is a demographic model for the SFS (which is a rarely used option and can be ignored). So usually, we would setmodel_funcas[model_func_ld].
Additionally, we will almost always pass the list of unscaled recombination
bin edges as rs = [r0, r1, ..., rn], which defines n recombination bins.
The effective population size plays a different role in LD inference than it does in SFS inference. For the site frequency spectrum, \(N_e\) merely acts as a linear scaling factor and is absorbed by the scaled mutation rate \(\theta\), which is treated as a free parameter. Here, \(N_e\) instead rescales recombination rates, and because we use a recombination map to determine the binning of data by recombination distances separating loci, \(N_e\) is a parameter that must be either passed as a fixed value or simultaneously fit in the optimization.
If Ne is a fixed value, we specify the population size using that keyword
argument. Otherwise, if Ne is to be fit, our list of parameters to fit by
convention includes Ne in the final position in the list. Typically, Ne
is not a parameter of the demographic model, as we work in rescaled genetic
units, so the parameters that get passed to model_func are params[:-1].
However, it is also possible to write a demographic model that also uses Ne
as a parameter. In this case we set pass_Ne to True, so that Ne
both rescales recombination rates and is a model parameter, and all params
are passed to model_func.
Ne: The effective population size, used to rescalersto getrhos = 4 * Ne * rs.pass_Ne: Defaults toFalse. IfTrue, the demographic model includesNeas a parameter (in the final position of input parameters).
Other commonly used options include
fixed_params: Defaults toNone. To fix some parameters, this should be a list of equal length asp0, withNonefor parameters to be fit and fixed values at corresponding indexes.lower_bound: Defaults toNone. Constraints on the lower bounds during optimization. These are given as lists of the same length of the parameters.upper_bound: Defaults toNone. Constraints on the upper bounds during optimization. These are given as lists of the same length of the parameters.statistics: Defaults toNone, which assumes that the data contain the full set of statistics for the number of populations in the model, as returned by the parsing functions. If the data contain a subset of statistics, we list them here. See below.normalization: Defaults to0. The index of the population to normalize by, which should match the population index that we normalized by when parsing the data.verbose: If an integer greater than 0, prints updates of the optimization procedure at intervals given by that spacing.
Note
verbose=1 prints a line for every function evaluation, which can be many
lines per second. Displaying them in a visible terminal window can slow down
the optimization, because the program waits for the terminal to draw each
line (it may run much faster when the window is minimized). Use a larger
value, such as 10 to 100, to avoid this.
Normalizing statistics and the statistics option
The data are normalized by \(\pi_2\) and \(H\) in the normalizing
population k (see Parsing), so the
normalizing statistics pi2_k_k_k_k and H_k_k equal 1 in both the data and
the model, and have zero variance. They carry no information, so they are left out
when computing likelihoods. This is handled the same way by all LD inference,
uncertainty, and plotting functions, in both moments.LD and moments.Demes:
The data passed to these functions should include the normalizing statistics, as returned by the parsing functions. They are removed internally, and the data passed in are not modified.
If
statisticsis not given (the default), the data must contain the full set of statistics for the number of populations, in the order given bymoments.LD.Util.moment_names(num_pops). Data with any other number of statistics raise an error.If the data contain a subset of statistics,
statisticsmust list exactly those statistics, as[ld_stats, h_stats], in the same order as the data. This must be the standard order ofmoments.LD.Util.moment_names(num_pops), which is the order returned by the parsing functions. For example,moments.LD.Parsing.subset_datareturns the list of statistics along with the subset data. If the normalizing statistics are in the list, they are removed along with the data.The normalizing statistics in the data must equal 1. If they don’t, the data were normalized by a different population than
normalization(for example, if the normalizing population was left out withsubset_data), and an error is raised.In
moments.LDfunctions,normalizationis the index of the normalizing population. Inmoments.Demesfunctions, it is the population’s name.
For example, with the data mv returned by bootstrap_data:
# all statistics, as returned by bootstrap_data, so statistics isn't needed
opt_params, neg_LL = moments.LD.Inference.optimize_log_fmin(
p_guess, [mv["means"], mv["varcovs"]], [model_func], rs=r_bins
)
# a subset of the statistics, without Dz, so we pass their names
r_edges, means, varcovs, stats = moments.LD.Parsing.subset_data(
mv, ["deme0", "deme1"], remove_Dz=True
)
opt_params, neg_LL = moments.LD.Inference.optimize_log_fmin(
p_guess, [means, varcovs], [model_func], rs=r_edges, statistics=stats
)
Note
Before version 1.7.0, get_bootstrap_sets and subset_data removed the
normalizing statistics from their outputs. Data saved from those versions need
to be recomputed, or have the normalizing statistics added back in their
positions (they equal 1, with zero variance).
Example
Using the data simulated in the Parsing section, we can
refit the demographic model under a parameterized IM model. For this, we could
use the moments.LD.Demographics2D.split_mig model as our model_func,
which is equivalent to the function we defined above (which we use in this
example). After loading the data and setting up the inference options, we’ll
use optimize_log_fmin to fit the model.
import moments.LD
import pickle
with open("data/means.varcovs.split_mig.100_reps.bp", "rb") as fin:
data = pickle.load(fin)
rs = [0, 1e-6, 2e-6, 5e-6, 1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4, 1e-3]
# an initial guess, perturbed to get the starting parameters
p_guess = [0.1, 2.0, 0.075, 2.0, 10000]
p0 = moments.LD.Util.perturb_params(p_guess, fold=0.2)
# run optimization, which returns the negative log-likelihood
opt_params, neg_LL = moments.LD.Inference.optimize_log_fmin(
p0,
[data["means"], data["varcovs"]],
[model_func],
rs=rs,
verbose=40,
)
# get physical units, rescaling by Ne
physical_units = moments.LD.Util.rescale_params(
opt_params, ["nu", "nu", "T", "m", "Ne"]
)
print("best fit parameters:")
print(f" N(deme0) : {physical_units[0]:.1f}")
print(f" N(deme1) : {physical_units[1]:.1f}")
print(f" Div. time (gen) : {physical_units[2]:.1f}")
print(f" Migration rate : {physical_units[3]:.6f}")
print(f" N(ancestral) : {physical_units[4]:.1f}")
40 , -332.63 , array([ 0.162578 , 2.31173 , 0.0574049 , 2.17875 , 9691.2 ])
80 , -188.495 , array([ 0.162033 , 2.30575 , 0.0579513 , 2.13596 , 11034.6 ])
120 , -154.358 , array([ 0.143319 , 2.0923 , 0.0479332 , 1.97624 , 11567.5 ])
160 , -99.0697 , array([ 0.155263 , 1.92572 , 0.0517665 , 1.77756 , 11432.9 ])
200 , -74.5396 , array([ 0.188409 , 1.96867 , 0.0661713 , 1.75043 , 10720.2 ])
240 , -73.1196 , array([ 0.190973 , 1.98928 , 0.0681279 , 1.75797 , 10686.9 ])
280 , -73.1031 , array([ 0.191864 , 1.99411 , 0.0685795 , 1.76098 , 10663.1 ])
320 , -73.1028 , array([ 0.191921 , 1.99414 , 0.0686025 , 1.76086 , 10661.7 ])
360 , -73.1004 , array([ 0.192222 , 1.99525 , 0.0687359 , 1.76013 , 10652.5 ])
400 , -73.0723 , array([ 0.192072 , 1.99856 , 0.0684619 , 1.74819 , 10655.4 ])
440 , -73.0122 , array([ 0.193818 , 2.01921 , 0.0690004 , 1.7165 , 10609.2 ])
480 , -72.8726 , array([ 0.193059 , 2.05711 , 0.0683662 , 1.6739 , 10578.6 ])
520 , -72.8704 , array([ 0.193055 , 2.06084 , 0.0682751 , 1.66898 , 10575.5 ])
560 , -72.87 , array([ 0.193124 , 2.06049 , 0.0683267 , 1.67003 , 10573.2 ])
best fit parameters:
N(deme0) : 2041.9
N(deme1) : 21786.9
Div. time (gen) : 1444.8
Migration rate : 0.000079
N(ancestral) : 10573.0
These should be pretty close to the input demographic parameters from the simulations. They won’t be spot on, as this was only using 100Mb of simulated data, but we should be in the ballpark.
Computing confidence intervals
When running demographic inference, we get a point estimate for the best fit demographic parameters. However, for an unknown underlying true value, it’s important to also estimate what’s called a confidence interval. The CI tells us the probability that the true value lies within some range, and provides some information about which parameters in our demographic model are tightly constrained and which parameters we have little power to pin down.
moments.LD can estimate confidence intervals using either the Fisher
Information Matrix (FIM) or the Godambe Information Matrix (GIM). In almost all
cases when using real data (or even most simulated data), the FIM will estimate
a much smaller CI than the GIM. This occurs because the FIM assumes all data
points that we’ve used are independent, when in reality there is linkage that
causes data points to be sometimes highly correlated between pairs of loci and
between recombination bins. The Godambe method uses bootstrap-resampled
replicates of the data to account for this correlation and does a much better
job at estimating the true underlying CIs [Coffman2016].
Note
If you use the Godambe approach to estimate confidence intervals, please
cite [Coffman2016]. Alec originally implemented this approach in dadi,
and moments has more-or-less used this same implementation here.
To create bootstrap replicates from the dictionary of data sums computed over
regions, where region_stats = {0: ld_stats_0, 1: ld_stats_1, ...}, e.g., we use
num_boots = 100
norm_idx = 0
bootstrap_sets = moments.LD.Parsing.get_bootstrap_sets(
region_stats, num_bootstraps=num_boots, normalization=norm_idx)
The bootstrap sets have the same statistics as the means and varcovs from
bootstrap_data, including the normalizing statistics, which are removed when
computing uncertainties (see above).
These bootstrap sets can then be used as the inputs to the moments.LD.Godambe
methods. The two CI estimation methods are
FIM_uncert: Uses the Fisher Information Matrix. Usage isFIM_uncert(model_func, opt_params, means, varcovs, r_edges=rs).GIM_uncert: Uses the Godambe Information Matrix. Usage isGIM_uncert(model_func, bootstrap_sets, opt_params, means, varcovs, r_edges=rs).
In each case, the model function is the same as used in inference (some
manipulation may be needed if we had any fixed parameters), means and varcovs
are the same data as input to the inference function, and r_edges are the
bin edges used in the inference. Additional options for some corner cases are
described in the API reference for LD methods.
Example
We’ll use both the FIM and GIM to compute uncertainties from the above example inference.
Using the FIM approach:
# using FIM
uncerts_FIM = moments.LD.Godambe.FIM_uncert(
model_func,
opt_params,
data["means"],
data["varcovs"],
r_edges=rs,
)
# lower and upper CIs, in genetic units
lower = opt_params - 1.96 * uncerts_FIM
upper = opt_params + 1.96 * uncerts_FIM
# convert to physical units
lower_pu = moments.LD.Util.rescale_params(lower, ["nu", "nu", "T", "m", "Ne"])
upper_pu = moments.LD.Util.rescale_params(upper, ["nu", "nu", "T", "m", "Ne"])
print("95% CIs:")
print(f" N(deme0) : {lower_pu[0]:.1f} - {upper_pu[0]:.1f}")
print(f" N(deme1) : {lower_pu[1]:.1f} - {upper_pu[1]:.1f}")
print(f" Div. time (gen) : {lower_pu[2]:.1f} - {upper_pu[2]:.1f}")
print(f" Migration rate : {lower_pu[3]:.6f} - {upper_pu[3]:.6f}")
print(f" N(ancestral) : {lower_pu[4]:.1f} - {upper_pu[4]:.1f}")
95% CIs:
N(deme0) : 1825.7 - 2269.1
N(deme1) : 18860.8 - 24880.0
Div. time (gen) : 1266.8 - 1632.7
Migration rate : 0.000069 - 0.000088
N(ancestral) : 10166.3 - 10979.6
And using the GIM approach:
with open("data/bootstrap_sets.split_mig.100_reps.bp", "rb") as fin:
bootstrap_sets = pickle.load(fin)
# using GIM
uncerts_GIM = moments.LD.Godambe.GIM_uncert(
model_func,
bootstrap_sets,
opt_params,
data["means"],
data["varcovs"],
r_edges=rs,
)
# lower and upper CIs, in genetic units
lower = opt_params - 1.96 * uncerts_GIM
upper = opt_params + 1.96 * uncerts_GIM
# convert to physical units
lower_pu = moments.LD.Util.rescale_params(lower, ["nu", "nu", "T", "m", "Ne"])
upper_pu = moments.LD.Util.rescale_params(upper, ["nu", "nu", "T", "m", "Ne"])
print("95% CIs:")
print(f" N(deme0) : {lower_pu[0]:.1f} - {upper_pu[0]:.1f}")
print(f" N(deme1) : {lower_pu[1]:.1f} - {upper_pu[1]:.1f}")
print(f" Div. time (gen) : {lower_pu[2]:.1f} - {upper_pu[2]:.1f}")
print(f" Migration rate : {lower_pu[3]:.6f} - {upper_pu[3]:.6f}")
print(f" N(ancestral) : {lower_pu[4]:.1f} - {upper_pu[4]:.1f}")
95% CIs:
N(deme0) : 1570.6 - 2562.0
N(deme1) : 17084.3 - 26962.7
Div. time (gen) : 1020.1 - 1917.6
Migration rate : 0.000059 - 0.000096
N(ancestral) : 9845.8 - 11300.2
We can see above that the FIM uncertainties are considerably smaller (i.e. more constrained) than the GIM uncertainties. However, the GIM uncertainties are to be preferred here, as they more accurately estimate the underlying true uncertainty in the demographic inference.
Likelihood ratio tests
To test whether a more complex model fits the data significantly better than a simpler, nested model (one that is a special case of the complex model with some parameters fixed), we can use a likelihood ratio test. The test statistic is twice the difference in log-likelihoods of the best fits, \(D = 2 (\ell_{complex} - \ell_{simple})\). If the data points were independent, \(D\) would follow a \(\chi^2\) distribution under the simple model, with degrees of freedom equal to the number of nested parameters.
The LD likelihood is a composite likelihood, which treats the recombination
bins as independent, so this \(\chi^2\) test is badly overconfident.
Following [Coffman2016], moments.LD.Godambe.LRT_adjust uses the bootstrap
replicates to compute an adjustment factor adj, and the adjusted statistic
\(D_{adj} = adj \times D\) is then compared to the \(\chi^2\)
distribution. For LD statistics, adj is usually much smaller than one,
because the statistics in different recombination bins are computed from the
same pairs of SNPs and so are strongly correlated.
LRT_adjust takes the complex model function and the parameters of the best
fit of the simple model, given as parameters of the complex model with the nested
parameters set to their fixed values (and Ne as the last entry), along with the
indices of the nested parameters:
LRT_adjust(model_func, bootstrap_sets, p_nested, nested_indices, means,
varcovs, r_edges=rs). The p-value is given by moments.Godambe.sum_chi2_ppf.
Example
We’ll test whether migration is needed to explain the data from the example
above, by fitting a model without migration, in which the migration rate is
fixed at zero. Because zero is on the boundary of the allowed range of the
migration rate, the test statistic follows an equal mixture of \(\chi^2\)
distributions with zero and one degrees of freedom, which we specify with
weights=(0.5, 0.5).
def model_nomig(params, rho=None, theta=0.001):
nu0, nu1, T = params
return model_func([nu0, nu1, T, 0], rho=rho, theta=theta)
opt_params_nomig, neg_LL_nomig = moments.LD.Inference.optimize_log_fmin(
[0.1, 2.0, 0.075, 10000],
[data["means"], data["varcovs"]],
[model_nomig],
rs=rs,
)
# the best fit without migration, as parameters of the model with migration
p_nested = [*opt_params_nomig[:3], 0, opt_params_nomig[3]]
adj = moments.LD.Godambe.LRT_adjust(
model_func,
bootstrap_sets,
p_nested,
[3],
data["means"],
data["varcovs"],
r_edges=rs,
)
D = 2 * (neg_LL_nomig - neg_LL)
D_adj = adj * D
print(f"Test statistic D : {D:.2f}")
print(f"Adjustment factor : {adj:.4f}")
print(f"Adjusted D : {D_adj:.2f}")
p_unadj = moments.Godambe.sum_chi2_ppf(D, weights=(0.5, 0.5))
p_adj = moments.Godambe.sum_chi2_ppf(D_adj, weights=(0.5, 0.5))
print(f"p-value, unadjusted: {p_unadj:.3g}")
print(f"p-value, adjusted : {p_adj:.3g}")
Test statistic D : 114.55
Adjustment factor : 0.0632
Adjusted D : 7.23
p-value, unadjusted: 0
p-value, adjusted : 0.00358
Without the adjustment, the p-value is effectively zero. After adjusting for the correlations between recombination bins, the test still rejects the model without migration, which we know to be the right answer for these simulated data, but with a much less extreme p-value.
When the adjustment is too conservative
LRT_adjust (for both the SFS and LD) can be very conservative. Its
adjustment factor shrinks roughly as \(1/D\), so the adjusted statistic
stays small even when \(D\) is very large. In simulations testing for weak
migration, the adjusted SFS test rarely rejected the model without migration at
a 1% level, and the adjusted LD test did so in fewer than half of the
replicates.
A large \(D\) (hundreds or more) with an adjusted statistic of only a few
units suggests the adjustment is too conservative, and a non-significant result
should then be treated as inconclusive. LRT_adjust also warns when the
Hessian of the complex model at the simple model’s best fit is indefinite, in
which case the adjustment factor is unreliable. In either
case, the alternatives are, from cheapest to most expensive:
A Wald test from the GIM uncertainties of the complex model. This needs no further fits: compute \(SE_{GIM}\) at the complex model’s best fit with
moments.LD.Godambe.GIM_uncert(ormoments.Godambe.GIM_uncertfor the SFS). For a parameter fixed at a boundary, such as a migration rate of zero, use the one-sided \(z = (\hat{\theta} - \theta_0) / SE_{GIM}\) with p-valuescipy.stats.norm.sf(z). Caveats: the GIM uncertainty is least reliable near the boundary, the false-positive rate of this test has not been checked, and the result depends on the parameterization (e.g., a rate or its log).Refitting the complex model to the bootstrap replicates. Reject the simple model if the bootstrap interval of the nested parameter excludes its fixed value. This needs one converged optimization per replicate, which is the cost the Godambe methods avoid. It is an interval test, not a null distribution for \(D\), and bootstrap intervals are imperfect near a boundary.
A parametric bootstrap under the simple model. Simulate genomes (e.g., with msprime) under the simple model’s best fit, matching the data’s sample sizes, sequence lengths, and recombination and mutation rates, and fit both models to each replicate. The p-value is the fraction of simulated \(D\) at least as large as the observed one. This is the most reliable option but the most expensive. Sampling from the composite likelihood instead (Poisson SFS, or independent Gaussian bins for LD) ignores linkage and is far too liberal.
For the data above, the Wald test gives much stronger evidence for migration than the adjusted likelihood ratio test:
import scipy.stats
z = opt_params[3] / uncerts_GIM[3]
print(f"Wald z : {z:.2f}")
print(f"p-value, one-sided: {scipy.stats.norm.sf(z):.3g}")
Wald z : 6.47
p-value, one-sided: 4.95e-11