May 14, 2024
Last time, I talked about a small class project where I counted entries in a book’s index to explore where sixteenth-century mapmakers were most active in Western Europe. I noticed that the types of questions I could ask were limited because counting and entering around two thousand maps into a spreadsheet was too time consuming (and boring!).
I proposed an independent study to my professor, Dr. Victoria Morse. I initially had two goals:
- Transform the book indexes from paper reference to digital spreadsheet, and
- Build or find a tool to make it easy for history students to explore the data in a meaningful way (ie build data visualizations without coding knowledge).
I went into this independent study thinking that the primary goal would be the second one, about data visualization. After all, I thought that the map I created was the cool part of my class project from Post 1. But by going through the process of that first goal, getting the data out of the book, I realized that this was an interesting challenge in and of itself.
Additionally, through conversations with Victoria, I realized that transforming a book reference to a digital reference is perhaps the more unique digital scholarship question. After all, data visualization has plenty of great existing resources. But turning a print catalog-like thing into data? I couldn’t find other people talking about it, and it seemed like it could generalize to other applications in history and other fields with catalog-like resources.
In the rest of this blog post, I will show some of the challenges that I encountered when transforming a paper index to a spreadsheet and what choices I needed to make along the way.
It is not immediately obvious how to transform a book index into a dataset. Robert W. Karrow’s indexes1 in Sixteenth Century Mapmakers and their Maps are highly organized, but not very compatible with how computers store, sort and filter data.
I decided to create a spreadsheet that would store some of the metadata about these maps and books. But what should go into the spreadsheet? What should a row look like?
In the previous post, I created a spreadsheet where each row was a city, and then the data points in each row were the number of first edition maps published in that city and then the latitude and longitude of the city for easy plotting. This is convenient when asking questions about where the maps included in Karrow’s book were published, but it limits the type of questions that can be explored. By grouping the data by publication location, I lost all the information about the date of publication, authorship, and what location was displayed in the map.
To be able to ask a variety of questions, I wanted to create a more fine-grained dataset. I decided that one row per map or book would be the most useful while still being feasible.
Choosing what to store: uncertain publication dates
What information should I store about each map? Some pieces of information are clear-cut, like the author. But others, like the date of publication, do not lend themselves well to the precise form of the spreadsheet. For some maps, Karrow records the exact year that they were published. Others, he only narrowed it down to a range like 1540s or 1524-26. Some are marked with uncertainty in other ways, like a map published “ca. 1559-60” or one published “by 1561”.
This was one of the trickiest decisions that I had to make for this project. In humanities classes I have taken that included digital projects, I had heard professors talk about the struggles of trying to use tools that simply were not designed for the type of data that humanists have. Here, I encountered a perfect example. The spreadsheet, a seemingly benign tool, is not naturally suited to recording uncertainty.
I couldn’t simply erase the uncertainty; these dates with their irregular fuzziness are part of the information contained in Karrow’s book. After conversations with Victoria and with Lin Winton, the director of Carleton’s Quantitative Resource Center, I honed in on a solution. I could break up the information into multiple columns: the earliest possible publication date (or 10 years before a date like “ca. 1559-60”), the latest possible publication date, the exact text from the index, and a column that either has the word “certain” or “uncertain”. The idea here is that when someone is filtering the spreadsheet for, say, maps published between 1565 and 1570, they can use the earliest and latest dates to get both maps that were certainly published in that time frame, as well as ones that potentially were. It’s then up to the person using the data to decide what to do with the uncertain maps.
This solution does have some downsides. By neglecting to provide one exact date, I made it harder to ask certain questions about trends over time, like “how did the number of maps produced in Europe change over the decades of the sixteenth century?”. Someone studying this question would have to take care to avoid having one map double counted in two decades. However, this dilemma is a feature of the data itself, not my spreadsheet. Asking this question requires making choices no matter what the format.
Now that I had a roadmap for what my spreadsheet should look like (one map or book per row, additional columns to handle uncertainty in dates), I was ready to start physically getting it out of Karrow’s book and into a spreadsheet. This is broken down into three phases:
1. Scanning the index and using OCR
When it came to scanning, I was stubborn. I was convinced that the best way to scan the index was via a scanning app on my phone. After all, if I used my phone, I could reliably hold Karrow’s 800 page doorstopper of a book flat. However, the fancy flatbed scanners at Carleton’s Gould Library do in fact produce the most consistently sharp scans, with minimal effort. I finished up this step by using Adobe Acrobat to apply Optical Character Recognition (OCR) to the scans, and then copying out the text into a separate text file. Conveniently, Acrobat was able to recognize the columns of text. Less conveniently, it grouped headers and footers, like page numbers, in with the columns of text. It also ignored bold and italics.
2. Doing a little manual cleaning up
I copied and pasted the text from the index out of the PDF and into a text file. Right away, it was clear that the OCR struggled in a few areas. In the font Karrow used, capital J was tricky to recognize, often showing up as a square bracket. I noticed when I opened up the text file using a text editor designed for coding that highlights unmatched brackets in red.
A few of the maps looked like they were published in the year 154057, which should have been 1540-57. Mysteriously, the city Rome showed up as Коте, where that first letter is a Greek kappa and the third letter is a Greek tau. When I noticed these mistakes, I fixed them in my text file, and I then searched the document for other similar errors. For example, I searched for all the square brackets, and manually checked if they should be J’s or brackets. This is feasible since the index is only three pages long.
I almost certainly missed some errors. A particularly hard to catch category is where the OCR mistook one character for another in a way that looked like a valid reference number or date, like if 36/J appeared as 36/1, or if a period and dash get swapped in just the wrong location, like 30/2-4 (which is three different maps, listed together) becoming 30/2.4 (a single map, in its fourth edition).
But all in all, the OCR did an acceptable job at turning a scan into a text file, and a little manual spot checking fixed a good chunk of the remaining inaccuracies. A few mistakes do not foil our goal of looking for patterns in the maps categorized by Karrow.
3. Using some Python code to transform the format
The final piece of the puzzle is to transform the text file into a spreadsheet. Here, I put on my computer science hat. Karrow organized his index very meticulously; as a human, I can see that the map with reference number 49/14 was published in 1558 in Vienna, Austria.
But to get the computer to do this, I needed to find some sort of pattern that would let me identify countries vs cities vs dates vs reference numbers. The cities are perhaps the easiest. They come at the start of a new line, preceded by a dash, and followed by a colon. Likewise, once I have identified cities, identifying dates and map reference IDs isn’t too bad, even though the OCR did not recognize the bold. There is always a date immediately after a city, then there is a comma, and then there are map IDs separated by commas until the last one, which I can identify because it is followed by a semi-colon. The punctuation here is crucial. It is very easy to split a long string of text based on a character.
I struggled the most with splitting based on country. When I copied the text out of the OCR’d PDF, each country was still on a new line after the previous country’s data, but there wasn’t an extra line of blank space like in the original print index. I couldn’t think of any simple patterns that would reliably pick up all the countries and only the countries since there was no punctuation. I was in the process of dreaming up a convoluted solution when Victoria pointed out that there’s only about 20 countries. I could very easily go into my text file and hit “Enter” before each country to make the coding easier. This had the added advantage of providing a focused way to lightly skim the entire file for OCR errors. I noticed that one page accidentally had OCR text for a part of the document that I had cropped out (the facing page partially scanned), which I was able to delete as I went through looking for country names.
The moral here is that this data is on a very specific scale. On one hand, the index contains a lot of data; my spreadsheet has around 1400 rows. Entering it all by hand would be mind-numbingly tedious. But on the other hand, the index is relatively short: less than three pages. Visually scanning for errors and manually adding a little bit of formatting are feasible, even quick tasks. Additionally, there are a small enough number of data points that we can keep the ones with uncertain dates rather than filtering them out or coercing them into a more precise, but wrong, bucket.
If you’re curious about the code to do this type of parsing, it is all publicly available on Github: https://github.com/DigitalCarleton/map-metadata-extraction. It also includes a more detailed description of the technical aspects involved.
It was at this point in my independent study that I fully abandoned the idea of creating tools for data visualization. The whole process of making this spreadsheet was more interesting than I had originally anticipated, and I felt in my bones that there must be other use cases where people might find these techniques helpful. I had also come to the realization that I would be reinventing the wheel in a sense if I worked on tools for future students to use this data. Carleton already has resources for professors looking to integrate data into their assignments, like the Quantitative Resource Center. I decided that this was something that other people could likely do better. Luckily, the beauty of an independent study is that I can make the choice to pivot!
In the next blog post, I will show off some examples where I take different books and transform references like tables of contents and indexes into spreadsheets.
- In the Acknowledgements section of Karrow’s Sixteenth Century Mapmakers and Their Maps, he thanks Patrick Morris and Charles Cullen for their help in creating the indexes of the book. ↩︎