Reflecting on My work, Big Data and the History Manifesto

My great grandfather wrote this postcard from the western front to my  grandfather who was seven years old. Two weeks after he wrote this my great grandfather became yet another soldier killed in World War I.

My great grandfather wrote this postcard from the front to my grandfather who was seven years old. Two weeks after he wrote this my great grandfather was killed.

Everything about World War I was massive. It was industrial-scale warfare fought along frontlines that stretched for hundreds of kilometres, manned and supplied by millions of people. This too was a war which produced an unprecedented stream of words. It was not just the politicians and officers who sat down to pen their thoughts. Ordinary soldiers near the front and their families from around the world, recorded their experiences and comforted each other through diaries and letters.

In just one month in 1916, the Australian army post headquarters in London successfully sorted nearly three million letters but there were another four hundred thousand letters which could not be delivered as they were inadequately addressed (‘The Soldiers’ Mails’, The Age, 10/2/1917, p. 4). Australia’s five million people were prodigious writers during the war.

Any historian who seeks to understand World War I needs to come to grips with the enormity of it. I am studying just a tiny fraction of the archives produced by that war, yet I am grappling the problems and possibilities of dealing with a huge number of words. The other day I worked out that the collection of soldier diaries I’m working with contains over seven million words. To put it in more comprehensible terms, my corpus is currently the equivalent of over thirteen volumes of Leo Tolstoy’s War and Peace. This collection will grow as I add more diaries to my research corpus. I also have to read other primary sources such as court martials and Australian Imperial Force (AIF) unit diaries.

My interest is the personal beliefs of Australian soldiers as they served on the frontline in World War I. I’m interested in the wide spectrum of belief, from atheism to traditional faiths to the unconventional. In his iconic book, The Broken Years, Bill Gammage observed that few soldiers wrote about their religious beliefs so I’m searching for the proverbial needle in the haystack. While the soldiers did not write much about their beliefs on the front, the comments that they do make are loaded with meaning and can reveal some events that say much about the life of faith on the front.

I’m tackling this issue in a very different way to Gammage. He sat in libraries and research rooms for years reading soldiers’ letters and diaries. Close reading is an essential research technique of historians, but I have chosen a hybrid method. I have been extracting information from the digitised diaries and manipulating this data using a combination of Open Refine (formerly Google Refine) and programming in Python. For the type of data manipulation I’m doing Open Refine is a much better tool than Excel spreadsheets. I am also in the process of setting up a database to assist my research and will soon be learning to use a Named Entity Recognition tool to extract names out of my data. These methods help me to find the needle and to examine the context surrounding it through strategic close reading.

Historians, Jo Guldi and David Armitage recognise the exciting possibilities offered by Big Data and history and devote an entire chapter to it in their thought-provoking book, The History Manifesto. I reviewed their book and in my next post, ‘The History Manifesto and Big Data’, I took a closer look at their Big Data chapter.

This book has reminded me why I delved into World War I in the first place. I am one of a number of researchers interested in questioning the secularisation thesis, a theory that has gripped the humanities and social sciences for nearly one hundred years.  For most of the twentieth century researchers believed that as societies modernised the influence of religion would fade away.

Religion has been severely challenged in the west but has not disappeared and retains some influence. Think of the rise of the Pentecostal churches and other new age religions. Newspapers throughout the western world still give prominence to statements by the Pope and his travels. There are still many institutions in our society which are clearly religious – schools, chaplains in schools and the armed forces, prayers before the start of parliamentary sessions, Christian holy days which are observed as public holidays.

Yet it has been a rocky road for Australian religions. Australian researchers have quite rightly highlighted the disillusion many Australians have had with religious institutions for a long time. The twentieth century in the west was an era where people questioned their inherited beliefs and rejected hypocrisy, corruption and irrelevance.

Over a number of research projects covering more than a century of history, I am developing a more complex picture of the challenge to traditional western religious beliefs than the secularisation thesis accommodates. My research has also uncovered the persistence of religious beliefs in the Australian population over time. World War I would surely be a significant event that would cause the Australian soldiers to question the beliefs they were stoked with during their childhood… or was it?

One hundred years is not the long-term or longue-durée history advocated by Guldi and Armitage. In the area of secularisation the longue-durée history of note is A Secular Age by philosopher, Charles Taylor. The Immanent Frame website is a hub of further work in this area. While I think that there is a place for longue-durée history, I have not been convinced by Guldi and Armitage’s arguments to change my research focus. Rather than trying to cover it all myself, I am making a small contribution to the body of research on this issue.

As I noted in my discussion about the ‘Big questions, big data’ chapter of The History Manifesto, historians need to be trained to use the tools necessary to mine messy, historic data. I am fortunate that I learned programming at school and university in the 1980s. In the late 1980s accountants spent hours each day in front of pre-Windows era PCs using the DOS operating system, spreadsheeting packages like Lotus 1-2-3 and word processing packages such as Word Perfect..

Hence I developed a good basic set of technical skills. But technology is always changing and I’m also doing different things with technology. At the moment I’m on a steep learning curve trying to understand and use the principles of object oriented programming, setting up my database in a system I’ve never used before and learning how to use ‘Named Entity Recognition’. I’m very fortunate that Hubble is there to patiently advise me when I get stuck but the process is intense… and at times it is intensely frustrating!

While The History Manifesto did not lead me to change my approach it made me examine what I was doing and how I was doing it. This is a good thing. It is too easy to get caught up with some exciting bit of technology and lose sight of the purpose of the research. At the start of this project I vowed to periodically stop and question my methodology. The History Manifesto helped me to do that.

I have not been able to learn new technical skills and keep blogging at the same time which is one reason why I didn’t blog during December. I’m also trying to explore Singapore and make the most of time here. Life is certainly not dull but I wish there was more time to fit in all I want to do!

6 thoughts on “Reflecting on My work, Big Data and the History Manifesto

  1. Great! Your overall ideas are fascinating. Certainly the “rejection of religion” needs to be re-examined, in the US as well as Australia. I look forward to learning more about exactly what the new technology makes it possible for you to do.


    • Thanks Marilyn. I will be writing more about my project on this blog but will limit the technical aspect of my work as this blog is for the general reader. I will update my digital humanities blog, Stumbling Through the Future, with more details about the technology. I maintain that blog to share my technical learning, as I figure other people want to use technology for historical research but need a basic ‘how to’ to get them started.


  2. I am not convinced by Taylor’s own religious secular thesis. I have written on this in 2007 and I agree with Steve Bruce’s arguments that the anti-secularisation crew tend to set up straw-man versions of the thesis while ignoring clear evidence from the changing nature of western religion itself. As A.C. Grayling has also pointed out, the retreat is into ineffability. Hence Pentecostalism and NRMs are very good at the theatre of religion but it becomes nicely compartmentalised in the modern worldview and the modern ways of life.

    Lost is the expectation in an explanation from the theatre which can add new insight in the way we strive for understanding across many different fields of knowledge. Alfred North Whitehead, Henri-Louis Bergson, and William James, and Carl Jung, were probably the last great thinkers to take any possible insight from a strictly religious perspective, whether one agrees with any of their arguments or not. As Walter Kaufmann pointed out, theology has always followed and imitated philosophy, which had been formulated outside of organised religion. To say, that the best of these philosophies were “spiritual” is hedging the truth we find in a wide survey of the discipline. The term today is an example of the ineffability that plagues any sensible debate of the subject. In the tradition of Nietzsche, certainly one who saw himself as a spiritual philosopher, we need a genealogy of spirituality. My own thought is that René Descartes took an understanding of the term as far as it could reasonable go, but in a post-Cartesian perspective of consciousness there is little that the term can offer.

    On the question of the numbers, the statistics, for Australia at least, are still in the declining direction for adherence to religious identification. Ritualism and cultural practices will no doubt continue far into the future and idea of religion will probably continue to be associated with these practices, but the concept of religion will increasingly be seen as problematic for those who see no reason for a sense of wonder to be made sacred.


  3. Now for the history and historiography, two thoughts; firstly, do you think that any of those digital dairies cover soldiers from Queensland? I am, of course, looking for databases to mine for my Q ANZAC 100 project at the SLQ. I am rather envious of your adventure, Yvonne. I am faced with 250 SLQ records, mostly hard-copies with some PDF documents, but no actual content databases I can think of.

    Secondly, I am very pleased that you have considered widening your search to pick up expressions of doubt and skepticism, along with religious expressions. For my exploration of Queensland community thinkers and their social-political formations during World War I, I am attempting the widest view on the spectrum of beliefs. I agree it is the proverbial needle in the haystack, but strangely it will make it easier to gather. We both are clear what we are looking for, and notable beliefs, competitively fewer to pick up in the mass, will be easier to identify. They will stand out from the bulk of idle commentary which is common in correspondence; so the needles are luminous needles.


    • Regarding the question about soldiers from Queensland, I suggest that you search the and see if any of the diarists are from Queensland. I’ve created my own database of the diaries I have, something that I imagine you’ll have to do for your project.

      It is impossible to understand the rise of secular society without including atheist, agnostic and non-traditional beliefs. That is why each project I have embarked on has explored the wide gamut of beliefs. I generally focus on a public debate about the role of religion in some form or other as it is in those debates that the underlying beliefs and understanding of religion by various groups is exposed.


      • Many thanks, I will have a look at the soldier diaries database and see how it might be able to help me. I currently have 33 Queensland ‘combatants’ listed and identified, mostly soldiers in the European and Middle East theatres of war. When I find something on these gentlemen regarding belief, I let you know.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.