The Ghost Stories Had the Best Data

A Song-dynasty ghost story will tell you where the haunted bridge is. Not vaguely: the exact location, the officials posted nearby, what the local economy ran on, the social rank of everyone who walks onto the page. All of that arrives first. Only once the ledger is complete does the ghost show up.

I spent a semester building a searchable database of 2,700-odd literary works from the tenth through thirteenth centuries — poems, essays, fiction, administrative records — most of which had never been cataloged in a way that let anyone search across them. I went in expecting the interesting problem to be the extraction. What turned out to be interesting was which texts the extraction kept finding things in.

It was the ghost stories. Consistently, and by a wide margin.

The supernatural needs an address

At first this looks like an accident of taste. It follows instead from what the genre has to do.

A supernatural tale only works if the reader accepts the world it interrupts. The haunting is the anomaly, so everything around it has to be ordinary enough to make the anomaly legible, and the author supplies that ordinariness in the same four slots every time.

That realism is load-bearing. Without it the ghost has nothing to be strange against.

Compare the literature that got taken seriously. Poetry and philosophical essays are allusive by design; their skill lies in compression and reference, in saying one thing by gesturing at another, which makes them wonderful and nearly useless if what you want to know is who administered a county. Canonical texts are abstract because abstraction is the achievement. In the entertainment, concreteness is just how the trick works.

What you get is an inversion of the usual hierarchy. Literary scholars filed these tales for centuries under minor work. For a historian reconstructing where people lived and how they were ranked, they are among the densest sources in the collection, largely because nobody was performing significance in them.

What the pipeline was actually for

I would not have noticed this by reading. I noticed because entity extraction turns a corpus into something you can count.

Classical Chinese does not cooperate with modern tooling. There is no punctuation in the original and no spaces between words, and the grammar left spoken Chinese centuries ago. A single character can be a noun, a verb, or a particle depending on context that takes real historical knowledge to resolve. I ran a few off-the-shelf Chinese NER models over sample passages and the results were bad enough to be funny: a model trained on People's Daily has no idea what to do with a sentence where one character means "to govern" in one clause and is somebody's surname in the next, and where the place name refers to a region that stopped existing during the Southern Song.

So I went hybrid. Rule-based heuristics took the predictable cases, since reign-era dates follow fixed formats, official titles come from a known set, and certain character sequences reliably mark a geographic reference. For ambiguous names, literary allusions, and places referenced obliquely, I fine-tuned on a hand-annotated subset. That training data was expensive, because your annotators have to actually read classical Chinese, and it generalized across genres and reigns better than I had expected.

The piece that paid off most was cross-referencing every extracted entity against existing historical databases, so that a person who also appears in the official dynastic histories scores high and a reference that could be three different people gets flagged for review. That is where the roughly 70% cut in manual cataloging time came from.

Once every text had an entity count attached, the distribution was visible in an afternoon. Nobody had bothered to index the tales that were carrying the addresses.

I suspect that generalizes past this corpus. Which texts a tradition treats as serious tells you what the tradition valued, and almost nothing about which texts know the most. Here those were different piles.