The Filing Date Is Part of the Number

Jun 27, 2019

Published June 27, 2019

The Filing Date Is Part of the Number

A revenue figure dated December 31 can be announced on January 29, completed in a 10-Q on February 1, and replaced by an amendment on March 14. A database that stores only December 31 has deleted the sequence that made the figure usable.

A fundamental stock screener needs two clocks for every observation:

1. The period the fact describes.
2. The time the fact became available to the system.

The first clock supports accounting comparisons. The second clock supports honest decisions.

Most financial databases preserve the first and approximate the second. That is enough for a current company profile. It is insufficient for quantitative backtesting, historical screening, factor research, and any fair valuation engine that claims to reproduce an earlier state of the market.

One fact has several dates

Consider a quarterly revenue observation.

• The fiscal period ends on December 31.
• Management publishes an earnings release on January 29.
• EDGAR accepts the 10-Q on February 1.
• A data pipeline parses the filing at 21:14 UTC.
• An amended filing changes the value on March 14.
• A vendor may publish the revision several hours later.

Every timestamp answers a different question.

period_end answers which operations produced the revenue.

public_at answers when an external observer could first obtain the source.

loaded_at answers when a particular system obtained and validated it.

known_from answers when the value became eligible for a model run.

known_to answers when a later version superseded it.

A row with period_end and value cannot answer any historical question involving information availability. Adding filing_date helps only when the filing is the sole source and the system assumes zero ingestion latency. Earnings releases, delayed parsers, amendments, and manual corrections break that assumption.

The minimum useful model is bitemporal. Financial valid time describes the economic period. Transaction time describes the database’s knowledge.

The ordinary query leaks future information

A typical historical screen asks for the latest quarter available before a strategy date.

The common implementation filters on fiscal dates:

def latest_quarter(rows, strategy_date):
    eligible = [
        row for row in rows
        if row.period_end <= strategy_date
    ]
    return max(eligible, key=lambda row: row.period_end)

This code can select a December quarter during the first week of January. The quarter had ended. Its figures had not been published.

Changing the predicate to filing_date <= strategy_date still loses earlier versions. A March amendment can overwrite the February filing and then appear inside a January backtest after a routine database refresh.

Point-in-time financial data requires interval containment:

def visible(version, as_of):
    return (
        version.known_from <= as_of
        and (
            version.known_to is None
            or as_of < version.known_to
        )
    )

The interval is half-open: [known_from, known_to). Half-open intervals compose cleanly. A replacement can begin at the exact timestamp the prior version ends. No timestamp belongs to two versions.

Use an accounting key, then add a knowledge interval

A robust fact key starts with the identity of the accounting observation.

from __future__ import annotations

from dataclasses import dataclass
from datetime import date, datetime
from decimal import Decimal
from typing import Optional, Tuple


Dimensions = Tuple[Tuple[str, str], ...]


@dataclass(frozen=True)
class FactKey:
    cik: int
    concept: str
    period_start: Optional[date]
    period_end: date
    unit: str
    dimensions: Dimensions


@dataclass(frozen=True)
class FactVersion:
    key: FactKey
    value: Decimal

    public_at: datetime
    loaded_at: datetime
    known_from: datetime
    known_to: Optional[datetime]

    source_type: str
    accession: str
    sequence: int

    decimals: Optional[int]
    scale: int

The key includes XBRL dimensions. A consolidated revenue fact and a geographic segment revenue fact can share the same concept, period, and unit. Their dimensional contexts make them different observations.

The XBRL context identifier is unsuitable as a permanent key. It is document-local. Two filings can use different context IDs for the same semantic context. Canonicalize the dimension members, sort them, and hash the canonical representation for storage.

Preserve decimals and scale as provenance. Inline XBRL may display a rounded number and apply a scale before yielding the machine value. Converting everything to binary floating point during ingestion loses both exact decimal semantics and the evidence needed to resolve conflicts.

known_from should be materialized. For an internal research system, a conservative value is:

def knowledge_time(public_at, loaded_at):
    return max(public_at, loaded_at)

This rule prevents a backtest from using a filing before the pipeline had actually processed it. A vendor-neutral historical product may instead model public availability and ingestion availability as separate clocks. The choice belongs in the product contract, not inside an ad hoc query.

Never update a fact in place

A correction is a new event.

The append-only ingestion table should retain every accepted observation, including duplicates that later lose a conflict-resolution rule. A derived version table can compress those events into non-overlapping intervals.

from dataclasses import replace
from itertools import groupby
from typing import Iterable, Iterator, List


SOURCE_PRIORITY = {
    "earnings-release": 10,
    "10-Q": 20,
    "10-K": 20,
    "10-Q/A": 30,
    "10-K/A": 30,
}


def choose_same_time(events: Iterable[FactVersion]) -> FactVersion:
    return max(
        events,
        key=lambda event: (
            SOURCE_PRIORITY[event.source_type],
            event.sequence,
        ),
    )


def build_intervals(
    events: Iterable[FactVersion],
) -> Iterator[FactVersion]:
    ordered = sorted(
        events,
        key=lambda event: (
            event.known_from,
            event.sequence,
        ),
    )

    collapsed: List[FactVersion] = []
    for _, group in groupby(
        ordered,
        key=lambda event: event.known_from,
    ):
        collapsed.append(choose_same_time(group))

    for index, current in enumerate(collapsed):
        next_start = (
            collapsed[index + 1].known_from
            if index + 1 < len(collapsed)
            else None
        )
        yield replace(current, known_to=next_start)

This function operates on events for one FactKey. The production pipeline should partition by the complete key before interval construction.

The source priority is explicit because “latest row wins” is not a financial-data policy. An earnings release and a 10-Q can disagree for legitimate reasons. Adjusted measures, reclassifications, dimensional detail, and presentation precision can all differ. A GAAP concept from a filed statement should not silently inherit the semantics of a similarly named press-release field.

Keep raw observations. Build curated facts from declared rules. Rebuild the intervals whenever the rules change.

An as-of join is a search over ordered boundaries

Once versions are closed into intervals, point-in-time lookup becomes a predecessor search.

from bisect import bisect_right
from typing import Optional, Sequence, Tuple


class AsOfIndex:
    def __init__(self, versions: Sequence[FactVersion]):
        ordered = tuple(
            sorted(
                versions,
                key=lambda version: version.known_from,
            )
        )

        for left, right in zip(ordered, ordered[1:]):
            if left.key != right.key:
                raise ValueError("index contains multiple fact keys")
            if left.known_to != right.known_from:
                raise ValueError("version intervals are not contiguous")

        self._versions: Tuple[FactVersion, ...] = ordered
        self._starts: Tuple[datetime, ...] = tuple(
            version.known_from for version in ordered
        )

    def at(self, as_of: datetime) -> Optional[FactVersion]:
        index = bisect_right(self._starts, as_of) - 1
        if index < 0:
            return None

        candidate = self._versions[index]
        if (
            candidate.known_to is not None
            and as_of >= candidate.known_to
        ):
            return None

        return candidate

The lookup cost is logarithmic in the number of versions for a fact. The larger systems problem is grouping and locality.

A useful physical order is:

(entity, concept, period_end, unit, dimensions_hash, known_from)

That order keeps all versions of one observation adjacent. It also supports a range scan for a company, concept, and reporting period.

For a broad historical screen, the query shape reverses. The engine needs one visible version for many keys at one as_of timestamp. Sort each partition by known_from, advance a cursor through strategy dates, and emit only boundary changes. Repeating an independent binary search for every company, metric, and date wastes the monotonicity of time.

Daily snapshots are easy to query and expensive to correct. Version intervals are compact and preserve the source chronology. Materialize snapshots only at expensive product boundaries, such as an index rebalance universe or a published factor dataset.

Public does not mean tradable

An EDGAR acceptance timestamp can occur after the closing auction. An earnings release can arrive during a halt. A parser can complete after the next bar has already opened.

The model needs a decision clock and an execution clock.

from bisect import bisect_left
from datetime import timedelta
from typing import Sequence


def first_executable_bar(
    public_at: datetime,
    loaded_at: datetime,
    bars: Sequence[datetime],
    minimum_latency: timedelta = timedelta(seconds=2),
) -> datetime:
    decision_ready = max(public_at, loaded_at) + minimum_latency
    index = bisect_left(bars, decision_ready)

    if index == len(bars):
        raise LookupError("no executable bar after decision time")

    return bars[index]

The bars sequence must already reflect the venue calendar, session boundaries, early closes, and the strategy’s permitted execution points. A daily strategy that trades at the close needs the information before its order cutoff, not merely before midnight.

This distinction removes a common source of phantom alpha. A value can be public before the backtest’s date label and still be unavailable to the simulated trade.

Trailing twelve months is a point-in-time computation

TTM revenue looks like a sum of four quarters. The arithmetic is trivial. The selection is where errors enter.

At an as-of timestamp:

1. Resolve the visible version of each candidate quarter.
2. Select four non-overlapping fiscal intervals.
3. Verify continuity.
4. Reject duplicate transition quarters.
5. Preserve the source and knowledge timestamp of every component.
6. Set the TTM observation’s known_from to the maximum knowledge time of its components.

A derived metric inherits the latest availability of its inputs.

If three quarters were known in February and the fourth arrived in March, the TTM value became knowable in March. Assigning it to the last fiscal period end creates the same leak as assigning a 10-Q to its quarter end.

Restatements require dependency invalidation. When one quarter changes, every TTM value whose window includes that quarter receives a new version. Destructive recomputation erases the research state that existed before the amendment.

The same rule applies to:

• return on invested capital
• free cash flow yield
• net debt to EBITDA
• Piotroski-style scores
• accruals
• multi-period growth rates
• fair value estimates

A valuation engine is a directed acyclic graph of versioned facts. Each output needs a reproducible dependency set and a knowledge interval.

The database must distinguish absence from zero

Before a cash-flow statement is filed, cash from operations is unknown. It is not zero.

This matters in ranking and filtering. A screen such as free_cash_flow_yield > 8% should exclude an unavailable numerator. Coercing missing facts to zero can move a company into or out of a screen depending on which side of the ratio is absent.

Use three states:

• observed value
• explicitly reported zero
• unavailable at the as-of time

A fourth state is often useful: structurally inapplicable. Banks and insurers do not fit every industrial-company metric. Structural absence should not share a sentinel with filing latency.

Null handling belongs in metric definitions. It should never emerge accidentally from a dataframe fill operation.

Restatements need regression tests

The core invariant is simple: a later correction must never alter an earlier answer.

from datetime import datetime, timezone
from decimal import Decimal


UTC = timezone.utc


def dt(year, month, day):
    return datetime(year, month, day, tzinfo=UTC)


def test_amendment_does_not_leak_backward():
    key = FactKey(
        cik=1,
        concept="Revenue",
        period_start=date(2018, 10, 1),
        period_end=date(2018, 12, 31),
        unit="USD",
        dimensions=(),
    )

    original = FactVersion(
        key=key,
        value=Decimal("10210000000"),
        public_at=dt(2019, 1, 29),
        loaded_at=dt(2019, 1, 29),
        known_from=dt(2019, 1, 29),
        known_to=None,
        source_type="earnings-release",
        accession="release-2019-01-29",
        sequence=1,
        decimals=-6,
        scale=0,
    )

    amended = FactVersion(
        key=key,
        value=Decimal("10180000000"),
        public_at=dt(2019, 3, 14),
        loaded_at=dt(2019, 3, 14),
        known_from=dt(2019, 3, 14),
        known_to=None,
        source_type="10-Q/A",
        accession="amendment-2019-03-14",
        sequence=2,
        decimals=-6,
        scale=0,
    )

    index = AsOfIndex(
        tuple(build_intervals([original, amended]))
    )

    assert index.at(dt(2019, 2, 20)).value == Decimal(
        "10210000000"
    )
    assert index.at(dt(2019, 3, 20)).value == Decimal(
        "10180000000"
    )

Add this test pattern for every derived metric. Freeze a small set of historical as-of dates and compare the complete screener output after each ingestion change. A parser improvement may change current values. It should not rewrite prior system states unless the change is an explicit historical correction with a recorded migration.

Point-in-time data changes the product

A current-value database can answer:

What is the company’s latest revenue?

A point-in-time database can also answer:

What revenue value was available at 15:45 UTC on February 20?

Which source supplied it?

Which later filing replaced it?

When could a strategy first trade on it?

Which valuation outputs depended on it?

These questions are operational requirements for a serious fundamental stock screener. They also produce a useful user interface. A screen result can expose the age, source, and revision history of every input instead of presenting a ratio as an unexplained scalar.

The filing date is part of the number. The acceptance timestamp, ingestion timestamp, and supersession timestamp are part of it as well. Remove those clocks and the database retains accounting values while discarding financial history.