Pith. sign in

REVIEW 4 major objections 5 minor 3 references

In the Shadow of Silence: Modelling Missing Data in the Dark Networks of Crime and Terrorists

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The pattern of missing ties in covert network data is itself a network, and an ERGM fitted to the missingness indicators turns MCAR, MAR, and MNAR into parameter restrictions.

desk verdict Modeling missingness indicators as an ERGM is a real step forward, but Table 1's classification is false and needs repair before the paper can be accepted. read the letter →

arxiv 2501.15825 v1 pith:JY2ECDWR submitted 2025-01-27 stat.ME

Jonathan Januar , H Colin Gallagher , Johan Koskinen This is my paper · ORCID
classification stat.ME
keywords covertnetworksmissingnetworkdatamissingnessmodelexponentialrandomgraphnotatsimulationsocialanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Covert network data are missing in patterns shaped by secrecy, surveillance priorities, and investigative habits, not by chance. This paper proposes modelling the missingness itself: since the matrix of missingness indicators has the same binary, symmetric form as the network's adjacency matrix, an exponential random graph model (ERGM) can be fitted to the pattern of missing ties, with parameters for missing ties clustering together, dependence on observed attributes, and dependence on the true network. Inside this model, missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR) become special cases obtained by setting certain parameters to zero. Simulations that degrade six empirical covert networks show that edge-targeted missingness can severely bias estimated triadic closure, that standard errors inflate under MNAR, and that zero-imputing missing ties artificially shrinks uncertainty. The payoff for a reader is a way to make the missingness assumption explicit, to simulate realistic data-collection biases, and to move beyond the MAR assumption built into current estimation software.

What carries the argument

The carrying object is the missingness indicator matrix $D$, a binary symmetric matrix with $d_{ij}=1$ when the tie between $i$ and $j$ is unobserved; it occupies exactly the same sample space as the adjacency matrix $X$. The exponential random graph model (ERGM) is the machinery applied to this matrix: it is a probability distribution over binary networks in which the log-probability of a network is a linear combination of network statistics, $\log \Pr(Y) \propto \eta^\top z(Y)$, with a normalising constant that makes the distribution proper. Chosen statistics $z(D)$ encode the assumed observational biases: edges for baseline missingness, geometrically weighted degree and clustering terms for missingness that propagates through shared nodes or triads, entrainment terms $d_{ij}x_{ij}$ for missingness that targets ties rather than nulls, and node or dyad covariates for observable drivers of invisibility. Because the same family is used for the data model and the missingness model, the paper can compare MCAR, MAR, and MNAR as parameter restrictions and simulate their consequences by sampling from the fitted missingness ERGM.

What would settle it

Construct a fully observed network with known ground truth (for example, a complete communication log or a simulated network with a known generative process), generate missingness by a mechanism that depends only on observed information, such as sequential snowball sampling with all sampled nodes' ties observed, and fit an ERGM to the resulting missingness indicators. If an ERGM with endogenous dependence terms, like geometrically weighted degree or clustering, fits the missingness pattern substantially better than an independent model, the paper's claim that MAR cannot be motivated through endogenous missingness dependence would be falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that an exponential random graph model applied to the missingness indicator matrix $D$ gives a generative account of why ties in covert networks go missing. With $d_{ij}=1$ marking an unobserved tie, the model takes the form $\Pr(D\mid X,\beta,\psi,\theta)=\exp(\psi^\top f(D)+\beta^\top f(D,C)+\theta^\top f(D,X)-\kappa(\beta,\psi,\theta))$, where the $\psi$ terms make missing ties depend on other missing ties, the $\beta$ terms make them depend on measured attributes, and the $\theta$ terms make them depend on the true network $X$. Setting $\theta=0$ yields MCAR, $\theta=0,\beta\neq0$ yields MAR, and $\theta\neq0$ yields MNAR, so the usual missingness taxonomy is a set of parameter restrictions on one model. The paper then shows, across 7,200 degraded versions of six covert networks, that assuming MAR when the true mechanism is MNAR attenuates estimated triadic clustering, inflates standard errors, and that treating missing ties as zeroes narrows uncertainty that should remain wide.

Load-bearing premise

The paper's load-bearing premise is that the six empirical covert networks are the true networks and that the ERGM fitted to each complete network gives the true generative parameters; if those public networks are themselves partial or biased constructions, the simulated biases and failure rates describe the models' behaviour relative to an assumed ground truth rather than to real covert networks.

Editorial extensions

If this is right

  • When missing ties are set to zero before estimation, the analysis implicitly asserts that every missing tie is a null tie; the paper shows this narrows uncertainty and drives coefficients toward zero as missingness grows.
  • Under edge-targeted missingness (MNAR with positive entrainment), estimated triadic closure is biased downward even when estimation follows the standard MAR-based routine, so substantive claims about clustering in covert networks should be treated as sensitive to the missingness assumption.
  • A richer model specification with attribute covariates greatly lowers estimation failure rates for a given network, but it does not remove the bias introduced by the missingness mechanism; the structure of the true network is what drives the bias.
  • Because the missingness model is generative, it can be used before data collection to simulate the consequences of surveillance prioritisation, informant coverage, or record availability, and after data collection as a sensitivity-analysis tool.
  • The framework is designed to extend Bayesian data augmentation for partially observed networks beyond the MAR/ignorability assumption, which would allow MNAR mechanisms to be incorporated directly into estimation of the network-generating model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same missingness-model machinery applies outside covert settings; survey non-response, organisational rosters, and interaction logs all produce node-clustered missingness, and fitting an ERGM to $D$ would let researchers test that clustering directly rather than assume independence.
  • Editorial inference: the observed sharp transition in inferred centralisation as the degree-based missingness parameter crosses zero suggests there may be observation-effort thresholds that regime-shift the apparent network structure; a testable consequence is that moderate increases in surveillance could produce qualitative rather than gradual changes in estimated network metrics.
  • Editorial inference: because the simulations assume the six published covert networks are the true networks, the numerical bias magnitudes should be read as relative illustrations; converting them into operational error bounds would require ground-truth networks, for instance complete communication logs from a closed criminal case, degraded and re-estimated in the same way.
  • Editorial inference: one could use the missingness ERGM as a prior or penalty in estimation, treating the missingness parameters as sensitivity hyperparameters; this would give a practical route to estimating network models that are robust across a range of MNAR mechanisms rather than only under MAR.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes to model missing tie-variable indicators D in covert networks with an exponential random graph model, P(D|X) = exp(ψ'f(D) + β'f(D,C) + θ'f(D,X) - κ), arguing that this framework nests MCAR, MAR, and MNAR through parameter restrictions (Table 1). It then reports a Monte Carlo study that degrades six empirical covert networks under four missingness models (independent, latent-space, endogenous MCAR ERGM, and MNAR ERGM), re-estimates ERGMs under a MAR assumption ('Miss') or zero imputation ('Zero'), and compares failure rates and relative bias. A follow-up inspection varies the entrainment parameter θ1 and degree-covariate parameter θ2 to show their effects on density, centralization, and triadic closure estimates.

Significance. If the nesting claim were correct, this would be a useful framework for formalizing and simulating missingness mechanisms in network analysis, especially for covert networks whose observations are layered and biased. The paper's strengths include a clear mathematical formulation, the explicit use of an ERGM for the missingness indicator matrix, a simulation across six real covert networks, and practically relevant findings about the behavior of the 'ergm' package under missing data. However, the central classification result in Table 1 is not generally true: a nonzero θ can index a mechanism that depends only on the observed part of X and is therefore MAR, not MNAR. The simulation also treats ERGM fits to the (possibly missing) empirical networks as ground truth, and it excludes the most degenerate network from the bias plots. These issues make the paper's theoretical claims and some of its empirical conclusions (as stated) unsupported, though the framework appears repairable.

major comments (4)
  1. [§4.2–5.2, Eq. (4), Table 1] The classification in Table 1 (θ ≠ 0 ⇒ MNAR) is not implied by Eq. (4). For a two-dyad ERGM on D with statistic s(D,X) = d_ij(1−d_ik)x_ik + d_ik(1−d_ij)x_ij, the statistic is nonzero only on dyads whose X entries are observed (d = 0), so P(D|X) = P(D|X_obs) for every realization of D. This is MAR by the paper's own definition in Eq. (5), yet θ ≠ 0. This is not a remote construction: Section 5.2's bivariate MAR example g10(x_ik, ψ) has exactly this pattern, and that mechanism is never represented in Table 1 or in the simulation's 'MNAR' conditions (Table 3), which use self-censoring terms d_ij x_ij. Consequently, a user following Table 1 would label a genuinely MAR mechanism as MNAR and would incorrectly conclude that the face-value likelihood of Eqs. (6)–(7) is unavailable. The Discussion's claim (Section 8) that MAR cannot arise from dependence within the missingness mechanism is also false under this construction. The framework is repairable by restricting f(D,X) to statistics that select observed entries (or by redefining the Table 1 conditions), but as stated the central nesting claim is unsupported.
  2. [§6, §7.2, Fig. 3] The simulation treats the six empirical covert networks as the true networks (Section 6: 'We assume that the empirical covert networks are the true network') and the complete-data ERGM estimates as the true generative parameters. Since the paper's own introduction argues that covert network data are partial, censored, and multiply constructed, these reference networks are themselves likely to contain missingness. The relative-bias and failure-rate results therefore characterize the behavior of estimators relative to an assumed ground truth, not relative to the actual covert networks. The paper explicitly acknowledges the assumption, but the substantive conclusions in Section 8 ('the true network structures ... have the greatest effects on the relative biases') should be phrased as conditional on that assumption, with a discussion of how violations of the assumption would affect the reported magnitudes.
  3. [§7.2, Fig. 3] Network 3 is excluded from all relative-bias plots (Figure 3 note: 'Network 3 is absent in these plots due to model estimation problems'). This is the network with the most severe estimation failure rates. The cross-network comparison of biases therefore omits the most extreme case, which is directly relevant to the paper's claim that true network structure drives bias patterns. The authors should either present the available partial information for Network 3 (e.g., estimates from conditions that did converge) or explicitly discuss how its exclusion affects each cross-network conclusion.
  4. [§7, §9] The paper does not ship code or data, despite the simulation being the paper's central evidence: Section 7 says code is 'available on request from the authors,' and Section 9 repeats 'available at request.' For a statistical methods paper, this is insufficient for reproducibility. Moreover, the simulation details are incomplete: the independent and latent-space missingness models listed in Section 6 are not fully specified (their parameter values and how the 10/35/60% missingness proportions are targeted are not given), and Table 3 provides parameter values only for the two ERGM missingness models. Without these details, the failure-rate and relative-bias results cannot be independently verified.
minor comments (5)
  1. [§5.2] The notation X_obs and X_mis is used before being formally defined; Section 5.2 defines X_obs = (X_ij : d_ij = 0) and X_mis = (X_ij : d_ij = 1), but this should appear earlier, at the first use in Section 3.
  2. [§7.2, Eq. (rBias)] The relative bias rBias = (η̃ − η)/η can be unstable when the baseline parameter η is close to zero (e.g., Table 2, Network 6, absdiff(Arrests) = −0.01). The paper should state how near-zero η values were handled or use an alternative metric.
  3. [§6, §7.3] The independent and latent-space missingness models are mentioned as part of the four models in Section 6, but Figure 3's caption and text do not clearly identify which models are included in each panel. The caption should name all four missingness models and the subset of parameters shown.
  4. [§7.3, Table 4] The term 'entrainment' is used to describe the parameter θ1 but is never explicitly defined in the text; the paper should define it at first use, e.g., as the effect of the true tie variable x_ij on the missingness indicator d_ij.
  5. [§2.1] The review of cognitive biases in crime investigation is lengthy and not directly used in the model or simulation; condensing it would sharpen the connection between the motivating examples and the formal framework.

Circularity Check

1 steps flagged · score 4.0 of 10

Table 1's θ≠0 ⇒ MNAR mapping is stipulated rather than derived, giving the nesting claim a definitional character; the simulation study itself is self-contained.

  1. self definitional [Section 5.1-5.2, Equation (4) and Table 1]
    "Furthermore, the model can also be used to parameterise the extent to which the missingness model violates conventional missingness assumptions of M(C)AR. How Equation (4) generalises the conventional missingness assumptions is summarised in Table 1. ... The parameters θ refer to interactions between the missingness mechanism and the true network. ... θ ≠ 0 MNAR"

    Equation (4) defines θ as the weight on f(D,X), a joint function of D and X, with no restriction to the missing part Xmis. MAR is defined in Equation (5) as P(D|X)=P(D|Xobs), and Xobs is a subvector of X. A statistic such as (1-dij)xik is a function of the observed part yet carries a nonzero θ; the paper's own bivariate MAR example g10(xik,ψ) has exactly this form. Therefore θ≠0 does not imply MNAR, and the row 'θ≠0 ⇒ MNAR' in Table 1 is not a consequence of Equation (4). It is an additional stipulation that re-labels any X-dependence as MNAR. The claimed nesting of MNAR by parameter restrictions is thus the stipulation itself, not a derivation, and it excludes genuine MAR mechanisms (and the associated availability of the face-value likelihood in Equations (6)-(7)) from the taxonomy.

full rationale

Most of the paper is a self-contained simulation study rather than a derivation from first principles. Missingness-model parameters in Tables 3 and 4 are chosen by the authors, not fitted to the data and then renamed as predictions, so the 'fitted input called prediction' pattern does not occur. The six empirical networks are used under an explicit ground-truth assumption ('We assume that the empirical covert networks are the true network'), which the paper labels as a reference point; the relative-bias results describe how estimators behave under the specified mechanisms, so this is not a hidden circularity. Self-citations to Koskinen et al. (2013, 2019) and to Handcock and Gile (2010) are standard methodological background, not load-bearing uniqueness claims. The only step with a definitional-circular character is the taxonomy in Table 1: the correspondence between θ≠0 and MNAR is stipulated rather than derived from Equation (4), since f(D,X) may depend on observed entries only (as in the paper's own g10(xik,ψ) MAR example), which would make θ≠0 compatible with MAR as defined in Equation (5). This makes the claimed nesting of missingness assumptions partly a matter of definition, but it does not undermine the independent simulation demonstrations of bias, variance inflation, and estimation failure under the specific mechanisms studied.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central simulations depend on the chosen missingness-model coefficients in Table 3 and on the fixed-vertex-set assumption. No new physical entities are introduced.

free parameters (4)
  • psi2_GWDegree = 2 and 0.4 (MCAR and MNAR)
    Chosen in Table 3 to induce endogenous degree-based missingness dependence.
  • psi3_GWESP = 2 and 0.5
    Chosen to induce triadic clustering in missingness.
  • theta1_entrainment = 0.8 (main), varied from -1 to 1 in inspection
    Controls whether missing ties are more likely to be edges or null ties.
  • theta2_degree_covariate = 0.2 (main), varied in inspection
    Controls whether high-degree nodes have more missing ties.
assumptions (5)
  • domain assumption The vertex set V is fixed and complete; missing actors are not considered.
    Stated in Section 3: 'We will not be exploring missing actors...' This is a strong assumption for covert networks where boundary specification is a known problem.
  • domain assumption Covert network ties are undirected.
    Section 2 argues directed ties are avoided because intentionality is hard to observe.
  • standard math Rubin's missingness taxonomy (MCAR/MAR/MNAR) applies to network tie variables.
    Section 5.2 adopts Rubin (1976) definitions.
  • domain assumption The empirical covert networks from UCINet are treated as the true networks, and the ERGM fits on them are the true generating parameters.
    Section 6: 'We assume that the empirical covert networks are the true network.' This is load-bearing for interpreting all bias results.
  • ad hoc to paper Markov dependence can be extended from tie variables to missingness indicators.
    Section 4.2 proposes this extension as the basis for using ERGMs on D.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In the Shadow of Silence: Modelling Missing Data in the Dark Networks of Crime and Terrorists." pith.science (2026). https://pith.science/paper/JY2ECDWR

@misc{pith2026250115825,
  author       = {Pith},
  title        = {Pith review of: In the Shadow of Silence: Modelling Missing Data in the Dark Networks of Crime and Terrorists},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JY2ECDWR}},
  note         = {Machine review of arXiv:2501.15825}
}
read the original abstract

The clandestine nature of covert networks makes reliable data difficult to obtain and leads to concerns with missing data. We explore the use of network models to represent missingness mechanisms. Exponential random graph models provide a flexible way of parameterising departures from conventional missingness assumptions and data management practices. We demonstrate the effects of model specification, true network structure, and different not-at-random missingness mechanisms across six empirical covert networks. Our framework for modelling realistic missingness mechanisms investigates potential inferential pitfalls, evaluates decisions in collecting data, and offers the opportunity to incorporate non-random missingness into the estimation of network generating mechanisms.

Figures

Figures reproduced from arXiv: 2501.15825 by the authors.

Figure 1
Figure 1. The six empirical covert networks used in the simulation studies with node sizes and [PITH_FULL_IMAGE:figures/full_fig_p026_1.png] view at source ↗
Figure 2
Figure 2. Six bar plots describing the failure rate for each empirical network. The x-axis rep [PITH_FULL_IMAGE:figures/full_fig_p030_2.png] view at source ↗
Figure 3
Figure 3. Each row refers to a specific estimation parameter, [PITH_FULL_IMAGE:figures/full_fig_p032_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The lines refer to the mean of the re-estimated parameters, ˆη [PITH_FULL_IMAGE:figures/full_fig_p035_4.png]
Figure 5
Figure 5. Figure 5: Identical plotting specifications to Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p035_5.png]
Figure 6
Figure 6. Figure 6: Identical plotting specifications to Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p036_6.png]
Figure 7
Figure 7. Figure 7: The effect on centralisation by θ2. The left plot refers to the simulated alternating star mean value parameter estimates and the right plot refers to the global centralisation metric. The shaded area refers to the 95% confidence interval. A dark blue reference line wa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages

  1. [159]

    https://doi.org/10.1016/j.scijus.2014.12.007 Divi´ ak, T. (2020). Criminal networks: Actors, mechanisms, and structures, 202. Eerland, A., Post, L. S., Rassin, E., Bouwmeester, S., & Zwaan, R. A. (2012). Out of sight, out of mind: The presence of forensic evidence counts more than its absence.Acta Psychologica, 140 (1), 96–100. https://doi.org/10.1016/j.a...

    arXiv 2020

  2. [354]

    https://doi.org/10.1111/j.1467-985X.2007.00471.x Henrico, S., & Putter, D. (2024). Intelligence Collection Disciplines—A Systematic Review [Pub- lisher: Routledge eprint: https://doi.org/10.1080/19361610.2023.2296765]. Journal of Applied Security Research, 1–25. https://doi.org/10.1080/19361610.2023.2296765 Hoff, P. D., Raftery, A. E., & Handcock, M. S. (...

    arXiv 2024

  3. [527]

    https://doi.org/10.1016/j.socnet.2013.07.003 Kossinets, G. (2006). Effects of missing data in social networks. Social Networks, 28 (3), 247–268. https://doi.org/10.1016/j.socnet.2005.07.002 Krause, R. W., Huisman, M., Steglich, C., & Snijders, T. A. (2018). Missing Network Data A Comparison of Different Imputation Methods. 2018 IEEE/ACM International Con-...

    work page arXiv 2006

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.