The sheer volume of untapped knowledge sitting in public archives is staggering
For my MA in Modern History back in 2002-2003 I spent a long time in the Tyne & Wear archives working towards my dissertation: 'The Origin and Application of Educational Ideas in Mid to Late Victorian England'. None of what I was looking at was digitised, but if it were, imagine how quickly I could have found things using AI.
Jesse Waites is a software architect, not an historian, but what he describes here is the promise of digital humanities: being able to discover things that would previously have taken (literally) a lifetime. Going forward, historians need computational thinking, I reckon.
Also, a note on the composition of the page, which is very well done – especially the meteorite that explodes on the page!
If I sat down to read just the Dutch East India Company pages myself, at two minutes a page, eight hours a day, five days a week, it would take me about 70 years. And that’s before the newspapers. My homebrew AI lab got through the entire archive in a single twelve-hour overnight run.
The trouble with old documents is that nobody spelled anything the same way twice. Handwriting and old print come out of text recognition full of errors, and 17th-century Dutch spells “rhinoceros” about fifteen different ways. A plain keyword search would miss most of what I was looking for. That overnight run was my graphics card turning every passage into a mathematical fingerprint of its meaning, 5.7 million passages from the Dutch East India Company archive alone. That let me search for what a passage was about, not just which words it happened to use.
[...]
The sheer volume of untapped knowledge sitting in public archives is staggering. Over just a few days, a single GPU and a tailored AI pipeline surfaced a forgotten meteorite, three lost rhinos, and several unrecorded eruptions. To make this kind of research accessible, I’m open-sourcing the workflow I created for this investigation as a small toolkit, Antiquity, enabling anyone with a question and a coding agent to conduct similar historical archival investigations.
Breen and I searched the same archive. He went looking for dodos and found a dodo. I went looking for meteorites and lost eruptions, and found those. The pages had been sitting there the whole time, for anyone to read. An archive doesn’t give anything up on its own; it only answers the questions someone thinks to ask.
Source: Original article · Are.na block · Finds channel