Semantic Action Extractor

Objective and purpose

The objective is to extract structured actions from English text while preserving exact source spans, then test whether an explicit predicate signal improves BERT’s semantic-role predictions.

An action record represents an actor, predicate, patient, and associated qualifiers. Extraction from raw prose requires predicate detection and assignment of source spans to the corresponding participants.

The research experiment evaluates a narrower task: prediction of semantic roles given a sentence and a known predicate. The distinction separates predicate-detection errors from role-labeling errors. Performance with a supplied predicate does not establish performance in a raw-text pipeline that must also identify predicates.

Rule-based extraction baseline

The source-linked extraction baseline and neural experimental infrastructure are separate processing paths. The baseline is usable independently while the neural study remains incomplete.

The baseline splits English text into sentences and tokens, recognizes a bounded vocabulary of verbs and inflections, and looks for an actor before the predicate. It collects a patient after the predicate until a preposition introduces a qualifier. Additional rules filter noun phrases and let coordinated actions inherit an actor.

The returned spans use character offsets into the original input. Validation checks that each extracted value matches its exact source substring. Substring validation detects invalid offsets and text absent from the input; it does not establish correct semantic-role assignment.

The example’s confidence value is 0.85. The value combines rules and bonuses; it is not an 85% estimated probability of correctness. The CLI accepts text from standard input or a file, and a TOML configuration can adjust domain verbs and the threshold.

The implementation avoids inventing an actor for imperatives. It can warn about negation, but the output schema does not represent polarity as a complete semantic field. Passive voice, long-distance references, and complex clauses remain difficult for the local rules.

Predicate-signal ablation study

The experiment compares two versions of the same fully trainable BERT encoder and linear token-classification head. In the predicate-signal version, the first WordPiece of the supplied predicate receives a distinct token-type identifier. In the ablation, all token-type identifiers are zero. The intended comparison holds the rest of the recipe fixed.

Words can split into multiple BERT subwords. The alignment code propagates BIO labels across those pieces, excludes special and padding tokens from the loss, and decodes each word from its first piece. BIO labels mark the beginning and continuation of a role span, such as a multiword participant.

The proposed recipe uses batch size 32, maximum sequence length 128, a learning rate of 0.00001, and two epochs. Development-set argument F1 selects the checkpoint. The label vocabulary comes from training data only.

Dataset preparation and alignment checks

The preparation pipeline joins a pinned PropBank English Web Treebank annotation skeleton with Universal Dependencies English Web Treebank words using normalized document identifiers and sentence positions. It checks structure, contradictions, duplicates across splits, and repeated evaluation semantics. This is an inferred join between releases, not an official reconstruction of the licensed corpus.

The prepared training vocabulary contains 111 labels. The primary metric requires both a correct span boundary and the correct argument label. It excludes the supplied predicate. Token accuracy and BIO-repair diagnostics serve as secondary checks. PropBank labels such as ARG0 and ARG1 depend on the verb’s frame; they do not universally mean “actor” and “patient.”

Completed execution checks

A fresh verification passes 14 focused tests covering the rule-based extractor, schema validation, and command-line interface. The supplied example reproduces the saved two-action JSON exactly. These checks establish baseline execution, without evaluating neural model quality.

A six-run rehearsal on 14 invented examples exercises training, checkpoint reload, aggregation, and benchmarking. A separate preflight performs one optimizer step on actual prepared data using Apple’s MPS backend. The recorded batch contains 32 examples, sequence length 118, and 111 labels; the step takes about 3.01 seconds with roughly 3.21 GB currently allocated.

These records verify selected execution paths. The synthetic rehearsal does not establish model quality, and the one-step preflight measures neither sustained throughput nor peak memory. A partial research directory exists, but none of the six required research runs is complete.

The project also implements checkpoint integrity, interrupted-run recovery, dataset validation, and exact-span evaluation. These controls support auditable execution; empirical conclusions still require completed research runs.

Unresolved research questions and next experiments

The effect of explicit predicate marking on exact argument-span F1 remains unmeasured under the shared data and optimization recipe. The next empirical step is to complete the paired runs and report the results for each seed.

A raw-text neural system would additionally need predicate discovery and a deliberate mapping from PropBank roles to the product’s action schema. Source spans and semantic roles alone do not establish intent, confirm that an action occurred, or fully resolve domain-specific meaning. Coreference, passive constructions, and negation need separate evaluation.