Corpus Shift Auditor

Objective and purpose

A classifier trained on refund and shipping text can later receive technical incident reports. Assessing the need for retraining requires distinguishing changes in the input distribution from changes in predictive performance. This project characterizes differences between a reference corpus and incoming text; it does not measure downstream predictive degradation.

Distribution-shift measurements

The implementation provides a command-line workflow, reusable JSON profiles, a smoothed n-gram model, and an interpretable comparison report. The implementation uses Python’s standard library, runs offline, and requires no external service or downloaded model.

Each input file counts as one document. Sentence and token summaries are computed separately. The output is a JSON report; the current tool does not provide a streaming monitor, historical dashboard, or neural drift detector.

Sequence likelihood and perplexity

The default trigram model estimates a token from the two tokens before it. Tokenization lowercases text and retains punctuation, Unicode word characters, and some internal apostrophes and hyphens. Additive smoothing with alpha 0.1 assigns nonzero probability to unseen combinations. Unfamiliar vocabulary maps to an unknown-token marker.

Perplexity summarizes how surprising a sequence is under that model. The report divides incoming perplexity by the reference corpus’s own perplexity. The reference value comes from the same text used to fit the model, not an independent holdout, so the ratio should not be read as a calibrated generalization measure.

Vocabulary coverage

The out-of-vocabulary rate counts incoming token occurrences absent from the reference token set. It includes punctuation and distinguishes surface forms such as “order” and “orders.” A high value can reflect a new subject, but also a small vocabulary or harmless wording changes.

Lexical distribution

Jensen–Shannon divergence compares token-frequency distributions. The base-two implementation ranges from zero to one; zero means the distributions match. The metric captures changes in the mix of tokens, without determining whether their meaning or task relevance changed.

Structural summaries

The report measures absolute relative changes in average sentence length, average word-token length, and punctuation per token. These summaries can reveal different writing patterns. They do not parse syntax or identify grammatical relationships.

Heuristic risk score definition

The perplexity term ignores decreases and saturates at an eightfold increase. Both example ratios exceed that point, so each receives the maximum perplexity contribution. The structural term averages the changes in sentence length, word-token length, and punctuation rate.

The weighted sum maps to four labels: low below 0.25, moderate below 0.50, high below 0.75, and critical otherwise. These weights and thresholds are fixed implementation choices. A score of 0.77 is not a 77% probability that a model fails. The checked-in configuration documents the defaults, but the program does not currently load configurable risk weights from that file.

Synthetic example and threshold limitations

The demonstration reproduces the repository’s synthetic examples with the default trigram configuration. The results characterize baseline behavior without evaluating downstream model quality.

The GPU example has more unfamiliar vocabulary and greater lexical divergence, and it receives the higher combined score. Perplexity alone does not rank the examples in that order: the GPU text has slightly lower perplexity than the related support text.

Both examples receive high shift scores relative to the three-sentence reference. The related text concerns the same subject but uses different surface forms. This sensitivity to reference size and vocabulary limits the use of the example for selecting operational thresholds. The demonstration does not evaluate a downstream classifier.

Validation scope and downstream evaluation

A fresh verification passes all 10 software tests. A new command-line fit and audit reproduce both synthetic scores reported here. The completed artifact is a statistical baseline; comparative research and downstream-performance evaluation remain incomplete.

The software tests cover tokenization, probability normalization, serialization, example rankings, identical-corpus comparisons, JSON structure, and the CLI round trip. These tests verify implementation behavior but do not establish a relationship between the score and downstream accuracy loss.

Lexical change can occur without performance loss. Conversely, familiar words can express new meanings or label relationships that these statistics miss. Small corpora produce unstable estimates, and scores built from different reference profiles should not be compared as if they share one calibrated scale.

A downstream evaluation requires held-out domains or chronological splits, model predictions, and labels for measuring performance degradation. The evaluation can then compare syntax-based and embedding-based detectors with this statistical baseline. Threshold selection requires uncertainty estimates and explicit false-alarm costs. The current report supports distribution-shift analysis without establishing a need for retraining.