Objective and purpose
A conventional caption does not necessarily identify the observations that support its statements. When a caption contains an error, that omission can prevent a reader from tracing the statement to its source. The objective is to preserve the relationship between caption statements and their supplied evidence so readers can inspect and correct the output.
The longer-term motivation includes assistive descriptions, where unsupported details can mislead the person relying on the caption. The project has not yet evaluated accessibility outcomes or tested the system with intended users.
Observation validation and caption assembly
The implementation includes input and output schemas, a deterministic caption baseline, a command-line interface, synthetic examples, and a separate beam-search utility. The baseline establishes a contract that a future media model and verifier could share. The implemented contribution is a traceable evidence representation with inspectable baseline behavior.
- 01Observation validationCheck identifiers, text, confidence ranges, and the geometry or timing of evidence references.
- 02Observation selectionApply confidence, evidence, observation-count, and word-budget rules in the supplied order.
- 03Caption and evidence exportAssemble the text and retain claim records, evidence identifiers, scores, and warnings.
An input JSON object identifies the media and supplies observations. Each observation contains text, confidence, and optional evidence references. Evidence can point to an image rectangle, a time interval, a transcript span, or metadata. Image rectangles use coordinates from zero to one, so their meaning is independent of the image’s pixel dimensions.
Validation checks the schema version, unique observation identifiers, nonempty text, valid confidence values, and evidence-specific fields. Evidence identifiers must be unique within an observation. When retained observations share an evidence identifier, the associated evidence records must match. An output claim cannot point to missing evidence.
By default, the baseline requires evidence and an observation confidence of at least 0.5, retains at most four observations, and targets a 60-word caption. It processes observations in their supplied order, normalizes whitespace, and adds terminal punctuation when needed.
The first eligible observation is retained even if it exceeds the word target, with an explicit warning. Later observations that would exceed the remaining budget are skipped. This preserves complete supplied units, although a unit may itself contain more than one assertion.
Synthetic caption and evidence records
Repository example · Synthetic observations
Supplied observation 1: “A brown dog rests on a blue blanket.”
Supplied observation 2: “A red ball lies beside the dog.”
Saved caption: “A brown dog rests on a blue blanket. A red ball lies beside the dog.”
| Claim | Evidence reference | Confidence |
|---|---|---|
| Dog on the blanket | ev-dog-region · x: 0.12–0.78, y: 0.22–0.83 | 0.96 |
| Ball beside the dog | ev-ball-region · x: 0.72–0.89, y: 0.63–0.84 | 0.91 |
The repository contains these invented observations and their saved output, with no corresponding photograph. The output warns that no model inspected the source media and that the description requires human review. The example demonstrates how evidence references survive caption assembly.
Operational definition of grounding
A claim is marked grounded when it has an evidence pointer. The baseline does not verify that a rectangle contains a dog or that the dog is on a blanket. Structural validation can accept an incorrect upstream observation.
Claim confidence is the lower of the observation’s confidence and the highest confidence among its attached evidence items. This combines supplied values; it does not estimate calibrated uncertainty. Each observation becomes one claim record, without splitting the text into independently checkable assertions.
The configuration can allow observations without evidence, in which case the output explicitly marks unsupported claims. The request also accepts audience and language fields. Audience does not currently alter the wording, and language is copied into metadata without language-specific generation.
Independent beam-search utility
The repository includes a generic beam-search utility for a future learned caption decoder. A caller supplies token scores, and the utility expands and retains candidate sequences according to their scores. The implementation sums log probabilities for numerical stability and applies length normalization to account for sequence length.
The utility supports minimum length, forbidden tokens, and repetition controls. A synthetic example demonstrates decoding behavior. The observation-copy baseline does not call this utility, and the project does not yet contain a trained media-captioning model. Sequence-search performance alone does not establish factual support.
Validation scope and unresolved limitations
A fresh verification passes all 17 software tests and reproduces the saved two-claim output exactly. The independent beam-search demonstration returns three candidates. These checks confirm the implemented milestone; they do not establish empirical captioning performance.
The authored tests cover schema failures, evidence references, confidence handling, configurable unsupported output, decoding behavior, and CLI paths. These software tests do not measure caption accuracy, hallucination reduction, or usefulness to intended users.
The current baseline can preserve incorrect observations, conflicting statements, and awkward ordering. The word budget permits an explicit exception for the first eligible observation. Evidence references support inspection of these decisions, but the implementation lacks independent factual verification.
The proposed empirical evaluation requires a trained image-captioning baseline, claim segmentation, and a verifier that checks each assertion against the media. Quality evaluation should distinguish missing details from invented details, assess confidence calibration, and include the intended users. Those steps are necessary before treating this evidence contract as an evaluated assistive captioning system.
Evidence schema, examples, and source code
- Synthetic observation input and saved caption output — the complete example with evidence and warnings.
- Baseline, schemas, and beam-search implementation — implemented caption, validation, and decoding behavior.
- Milestone report — project scope and next steps.
Evidence: implemented baseline, schemas, decoding utility, synthetic examples, and authored software tests. The CLI currently consumes the baseline configuration; planned decoding and grounding settings are not active caption controls.