The reproducible parts of the AudioFacialMatrix experiments (Section IV of R. Singh and R. Singh, "AudioFacialMatrix: Dataset for Voice and Face AI"): cross-modal clustering of facial landmark ratios (IV-A), the liquid time-constant network (LTNN) run on the audio front end (IV-B), and BYO-media preprocessing utilities for the cleaning steps described in Section III-B.
It does not implement the data collection in Section III. See Scope for why, and for exactly where the line is drawn.
Landmark ratio features. The paper notes that 68 landmarks give 2278 pairwise distances and about 2.6 million ratios of those distances, then fixes the nose length as a common numerator. Fixing the denominator is what makes the space tractable: 2.6 million candidates become 2277, each scale-invariant against a single reference length that is roughly stable under head pose.
from afm import ratio_features
from afm.landmarks import select_by_variance
features = ratio_features(landmarks) # (n_faces, 2278)
features = features[:, select_by_variance(features, 26)]Ratios are invariant to scale and translation and not to rotation, which is a frontal-crop assumption worth stating out loud. There is a test for each of those three properties.
Faces with a degenerate nose length raise rather than divide. A near-zero denominator turns a bad crop into an outlier large enough to drag a k-means centroid across the space unaided, and the paper reports exactly this class of noise surviving the automated filters.
Choosing k when the criteria disagree. The paper reports that silhouette peaked
at k = 5 and fell almost monotonically after, but that the k = 5 clusters were
unevenly distributed and the best trade-off was at k = 8. That reasoning is sound
and it is usually done by eye. sweep_k scores each k on separation and on balance
and combines them with a weight the caller sets:
from afm import sweep_k, select_k
rows = sweep_k(features, k_values=range(2, 13), balance_weight=0.5)
select_k(rows, by="silhouette") # what separation alone prefers
select_k(rows, by="combined") # what separation and balance prefer togetherBalance is the normalised entropy of the cluster size distribution, in [0, 1], rather than a min/max size ratio, so it responds to the whole distribution and not only its extremes.
The two criteria disagreeing is a finding, not a bug, and select_k is built to let
you report both rather than resolve it silently.
One thing the sweep cannot do. Balance rises with k almost monotonically, so no
choice of balance_weight pins down a particular k on its own. Push the weight up
and the winner walks off the end of the range. The paper's k = 8 comes from knowing
there were eight source countries, which is external structure the sweep did not
recover. The sweep's job is to show what that prior costs in separation, not to
replace it.
Cross-modal agreement. The second AFM hypothesis is that individuals grouped into a voice subset belong to the corresponding face subset. Two independent clusterings of the same people, with arbitrary integer labels and no correspondence between them, so the measure is partition agreement and not accuracy.
from afm import cross_modal_agreement
print(cross_modal_agreement(face_clusters, voice_clusters, n_permutations=1000))n=720 ARI=+0.0680 NMI=0.1945 AMI=+0.1814
permutation null: mean=+0.0001 sd=0.0035 p=0.0020
agreement exceeds the independent null at the 5% level
The permutation test is not optional here. Adjusted Rand Index has an expected value near zero under independence, but its sampling variance at realistic cluster counts is wide enough that a positive ARI on its own says very little. The null shuffles which face cluster belongs to which individual while leaving both marginal size distributions intact, so it isolates the pairing from the shapes of the two partitions. The p-value carries an add-one correction and never reports zero off a finite number of draws.
ltnn is a liquid time-constant (LTC) recurrent network in PyTorch, plus the MFCC
front end it was used with in the AFM audio experiment. An LTC unit is a
continuous-time state whose time constant is a function of its input: a gated RNN
decides whether to update its state, an LTC decides how fast the state evolves.
That buys two concrete things: sequences sampled at uneven intervals can be
consumed directly without resampling, and different units can settle on different
timescales in the same layer.
import torch
from ltnn import LTCClassifier
model = LTCClassifier(input_size=39, num_classes=8, hidden_size=128, tau_init=8.0)
x = torch.randn(16, 300, 39) # (batch, frames, mfcc features)
logits = model(x) # (16, 8)Defaults follow the paper's Section IV-B configuration: a 128-unit recurrent layer into a linear head, trained with Adam (lr=0.001) on categorical cross-entropy against the face-side cluster assignments, batch size 32, early stopping patience 10.
ltnn.features implements the AFM preprocessing with numpy and scipy only:
resample to 16 kHz, pre-emphasis, 25 ms windows at a 10 ms hop, a 40-band mel
filterbank, 13 MFCCs, first/second-order deltas, for 39 features per frame. The log
floor is relative to each clip's own peak rather than absolute, since an absolute
floor clips more mel bins in a quiet recording than a loud one and lets recording
level leak into the shape coefficients (test_mfcc_is_invariant_to_recording_level
covers this across a 400x gain range).
Two implementation details matter and are not obvious from the Hasani et al.
paper: the dynamics need a signed, input-dependent target (not a per-unit constant
reversal potential, which collapses every sequence to the same attractor), and the
conductance bias should start negative (gate_bias_init = -2.0) so units begin in
the long-memory regime. Both are guarded by tests
(test_final_state_actually_depends_on_the_sequence,
test_solver_is_stable_on_large_inputs). examples/ltnn_irregular_sampling_benchmark.py
demonstrates the mechanism against a GRU baseline on a synthetic uneven-sampling
task (LTC 0.743 accuracy at 2340 params vs. GRU 0.728 at 3588 params, one seed,
illustrative not a benchmark result).
afm.preprocessing implements the cleaning steps Section III-B describes, as
generic functions over media you already have -- no search, no scraping, no
download, no identity or nationality targeting anywhere in this module. See
Scope.
from afm.preprocessing import (
deduplicate_frames, select_single_subject_frames,
trim_silence, has_sufficient_speech_energy, fixed_duration_clip,
)
frames = deduplicate_frames(frames) # drop near-duplicate frames (dHash)
frames = select_single_subject_frames(frames, my_detector) # keep only single-detection frames
clip = trim_silence(audio, sample_rate) # drop leading/trailing near-silence
ok = has_sufficient_speech_energy(clip, sample_rate) # coarse "is this mostly dead air" gate
clip = fixed_duration_clip(clip, sample_rate, start_s=0.0, duration_s=10.0)select_single_subject_frames and has_sufficient_speech_energy are deliberately
thin: no face detector or speech/music classifier is bundled. Bring your own via
the detect_fn callback (for frames) or substitute a real VAD (for audio) if your
use case needs to tell speech apart from music or count speakers -- the bundled
energy heuristic only flags near-silent or empty clips.
git clone https://github.com/rahulrikisinghcode/AudioFacialMatrix.git
cd AudioFacialMatrix
pip install -e ".[dev]"
pytestltnn pulls in torch, so this install is the slow one; afm (clustering and
preprocessing) is numpy, scipy and scikit-learn.
examples/k_selection_and_agreement.py builds synthetic landmark data with the
structure the paper describes: eight groups, four close together and four far
apart, so that separation and balance pull in different directions. It runs the
sweep, shows the two criteria disagreeing, and runs the agreement test against both
a voice partition that tracks the groups and one that does not.
examples/ltnn_irregular_sampling_benchmark.py builds a task where the elapsed
time between observations carries the signal (sinusoids at four frequencies,
sampled at random uneven intervals) and compares the LTC against a GRU baseline
given the same delta_t information.
The data in both scripts is synthetic. Nothing in their output is a result about faces, voices, or nationality; they exist so the procedures can be run and argued with without the dataset.
All three panels come from a live run of examples/k_selection_and_agreement.py;
see scripts/render_visuals.py, which reuses that script's own module-level
state rather than recomputing anything, so the numbers here match its printed
output exactly. On this run, silhouette alone peaks at k=2 while the balance-aware
combined score peaks at k=5, the same disagreement the paper reports at k=5 versus
k=8 (left panel). The middle panel shows why it matters: even the balance-aware
pick still leaves 600 of 720 faces in one dominant cluster, because 5 clusters
isn't enough to both separate the four close-together core groups and keep the
four outlier groups apart. Right: the observed cross-modal ARI (+0.068) sits well
outside its 800-draw permutation null, which is what "exceeds the independent null
at the 5% level" looks like as a picture instead of a p-value.
Regenerate with:
pip install -e ".[viz]"
PYTHONPATH=src:examples python scripts/render_visuals.pyThe LTNN visuals (accuracy curves, learned timescales, MFCC gain invariance) live in the standalone liquid-time-constant-networks repo linked in the Citation section below, rather than duplicated here.
This repository implements the analysis (Section IV) and the generic media cleaning a pipeline like this needs (Section III-B), not the collection (Section III-A). There is no web crawler, no video downloader, no search-by-region or search-by-language logic, and no pipeline that assembles face/voice pairs from public video on its own.
That is a deliberate omission, not an oversight. The collection method gathers biometric identifiers, faces and voiceprints, from identifiable people who did not consent, labelled by nationality. The paper is candid that named public figures ended up in the set. Describing that methodology in a paper and shipping a one-command tool that reproduces it are different acts with different consequences, and only the first is what a reader of the paper needs. The preprocessing utilities above stop at "clean media I already have" and do not cross into "find more of it."
Two further limits are worth stating plainly, because code in this area gets reused past its warrant.
The labels in AFM are source language and search region, not nationality and certainly not ancestry. The paper itself flags the confound: the Swedish subset contains Swedish-born Asian reporters. Cluster structure recovered from these features is structure in a corpus of news broadcasts, and a corpus of news broadcasts is not a population.
The paper's suggested applications include narrowing suspects in criminal investigation. Nothing here is fit for that. Cluster agreement above an independent null is a statement about a partition, not a statement about any individual, and it carries no per-person error rate that anyone could be held to.
@misc{singh_afm,
author = {Singh, Rahul and Singh, Rita},
title = {AudioFacialMatrix: Dataset for Voice and Face AI},
note = {Carnegie Mellon University, Language Technologies Institute}
}
@inproceedings{hasani2021liquid,
author = {Hasani, Ramin and Lechner, Mathias and Amini, Alexander
and Rus, Daniela and Grosu, Radu},
title = {Liquid Time-constant Networks},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = {35},
number = {9},
pages = {7657--7666},
year = {2021}
}A standalone, general-purpose version of the LTC implementation (not tied to AFM) also lives at liquid-time-constant-networks.
MIT. See LICENSE.