{"id":5181,"date":"2015-01-19T20:03:43","date_gmt":"2015-01-19T09:03:43","guid":{"rendered":"https:\/\/stumblingpast.wordpress.com\/?p=5181"},"modified":"2022-08-04T00:25:11","modified_gmt":"2022-08-04T00:25:11","slug":"reflecting-on-my-work-big-data-and-the-history-manifesto","status":"publish","type":"post","link":"https:\/\/stumblingpast.com\/2015\/01\/19\/reflecting-on-my-work-big-data-and-the-history-manifesto\/","title":{"rendered":"Reflecting on My work, Big Data and the History Manifesto"},"content":{"rendered":"
\"My<\/a>
My great grandfather wrote this postcard from the front to my grandfather who was seven years old. Two weeks after he wrote this my great grandfather was killed.<\/figcaption><\/figure>\n

Everything about World War I was massive. It was industrial-scale warfare fought along frontlines that stretched for hundreds of kilometres, manned and supplied by millions of people. This too was a war which produced an unprecedented stream of words. It was not just the politicians and officers who sat down to pen their thoughts. Ordinary soldiers near the front and their families from around the world, recorded their experiences and comforted each other through diaries and letters.<\/p>\n

In just one month in 1916, the Australian army post headquarters in London successfully sorted nearly three million letters but there were another four hundred thousand letters which could not be delivered as they were inadequately addressed (‘The Soldiers’ Mail<\/a>s’, The Age<\/em>, 10\/2\/1917, p. 4). Australia\u2019s five million people were prodigious\u00a0writers during the war.<\/p>\n

Any historian who seeks to understand World War I needs to come to grips with the enormity of it. I am studying just a tiny fraction of the archives produced by that war, yet I am grappling the problems and possibilities of dealing with a huge number of words. The other day I worked out that the collection of soldier diaries I\u2019m working with contains over seven million words. To put it in more comprehensible terms, my corpus is currently the equivalent of over thirteen volumes of Leo Tolstoy’s\u00a0War and Peace<\/em>. This collection will grow as I add more diaries to my research corpus. I also have to read other primary sources such as court martials and Australian Imperial Force (AIF) unit diaries.<\/p>\n

My interest is the personal beliefs of Australian soldiers as they served on the frontline in World War I. I’m interested in the\u00a0wide spectrum of belief, from atheism to traditional faiths to the unconventional. In his iconic book, The Broken Years, <\/em>Bill Gammage observed that few soldiers wrote about their religious beliefs so I\u2019m searching for the proverbial needle in the haystack. While the soldiers did not write much about their beliefs on the front, the comments that they do make are loaded with meaning and can reveal some events that say much about the life of faith on the front.<\/p>\n

I\u2019m tackling this issue in a very different way to Gammage. He sat in libraries and research rooms for years reading soldiers\u2019 letters and diaries. Close reading is an essential research technique of historians, but I have chosen a hybrid method. I have been extracting information from the digitised diaries and manipulating this data using a combination of Open Refine<\/a> (formerly Google Refine) and programming in Python. For the type of data manipulation I’m doing Open Refine is a much better tool than Excel spreadsheets. I am also in the process of setting up a database to assist my research and will soon be learning to use a Named Entity Recognition<\/a> tool to extract names out of my data. These methods help me to find the needle and to examine the context surrounding it through strategic close reading.<\/p>\n

Historians, Jo Guldi and David Armitage recognise the exciting possibilities offered by Big Data and history and devote an entire chapter to it in their thought-provoking book, The History Manifesto<\/em><\/a>. I reviewed their book<\/a>\u00a0and in my next post, ‘The History Manifesto and Big Dat<\/a>a’, I took a closer look at\u00a0their Big Data chapter.<\/p>\n

This book has reminded me why I delved into World War I in the first place. I am one of a number of researchers interested in questioning\u00a0the secularisation thesis, a theory that has gripped the humanities and social sciences for nearly one hundred years. \u00a0For most of the twentieth century researchers believed that as societies modernised\u00a0the influence of religion would fade away.<\/p>\n

Religion has been severely challenged in the west but has not disappeared and retains some influence. Think of the rise of the Pentecostal churches and other new age religions. Newspapers throughout the western world still give prominence to statements by the Pope and his travels. There are still many institutions in our society which are clearly religious – schools, chaplains in schools and the armed forces, prayers before the start of parliamentary sessions, Christian holy days which are observed as public holidays.<\/p>\n

Yet it has been a rocky road for Australian religions. Australian researchers have quite rightly highlighted the disillusion many Australians have had with religious institutions for a long time. The twentieth century in the west was an era where people questioned their inherited beliefs and rejected hypocrisy, corruption and irrelevance.<\/p>\n

Over a number of research projects covering more than a century of history, I am developing a more complex picture of the challenge to traditional western religious beliefs than the secularisation thesis accommodates. My research has also uncovered the persistence of religious beliefs in the Australian population over time. World War I would surely be a significant event that would cause the Australian soldiers to question the beliefs they were stoked with during their childhood… or was it?<\/p>\n

One hundred years is not the\u00a0long-term or longue-dur\u00e9e <\/em>history advocated by Guldi and Armitage. In the area of secularisation the longue-dur\u00e9e <\/em>history of note is A\u00a0Secular Age<\/em><\/a> by philosopher, Charles Taylor. The Immanent Frame<\/a> website is a hub of further work in this area. While I think that there is a place for longue-dur\u00e9e <\/em>history, I have not been convinced by Guldi and Armitage’s\u00a0arguments to change my research focus. Rather than trying to cover it all myself, I am making a small contribution to the body of research on this issue.<\/p>\n

As I noted in my discussion about the \u2018Big questions, big data\u2019 chapter of The History Manifesto<\/em><\/a>, historians need to be trained to use the tools necessary to mine messy, historic data. I am fortunate that I learned programming at school and university in the 1980s. In the late 1980s\u00a0accountants spent hours each day in front of pre-Windows era PCs using the DOS operating system, spreadsheeting packages like Lotus 1-2-3 and word processing packages such as Word Perfect..<\/p>\n

Hence I developed\u00a0a good basic set of technical skills. But technology is always changing and I\u2019m also doing different things with technology. At the moment I\u2019m on a steep learning curve trying to understand and use the principles of object oriented programming, setting up my database in a system I\u2019ve never used before and learning how to use \u2018Named Entity Recognition\u2019. I\u2019m very fortunate that Hubble is there to patiently advise me when I get stuck but the process is intense\u2026 and at times it is intensely frustrating!<\/p>\n

While\u00a0The History Manifesto<\/em> did not lead me to change my approach it made me examine what I was doing and how I was doing it. This is a good thing. It is too easy to get caught up with some exciting bit of technology and lose sight of the purpose of the research. At the start of this project I vowed to periodically stop and question my methodology.\u00a0The History Manifesto<\/em> helped me to do that.<\/p>\n

I have not been able to learn new technical\u00a0skills and keep blogging at the same time which is one reason why I didn\u2019t blog during December. I\u2019m also trying to explore Singapore and make the most of time here. Life is certainly not dull but I wish there was more time to fit in all I want to do!<\/p>\n","protected":false},"excerpt":{"rendered":"

Everything about World War I was on an industrial scale including writing. Soldiers and their families wrote billions of words during the war. The diaries that I am working with are a small fraction of what was written – only seven million words. To assist me in dealing with such a large volume of words I’ve turned to programming and other technology.<\/p>\n

While reading The History Manifesto<\/i> I reflected on the reason why I am researching the beliefs of Australian soldiers during World War I. In these reflections I share a glimpse of the big picture that my work is part of and what I’ve been working on over the last couple of months. <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[4,13,14],"tags":[43,81,106,159,168,192,199],"featured_image_src":null,"featured_image_src_square":null,"author_info":{"display_name":"Yvonne Perkins","author_link":"https:\/\/stumblingpast.com\/author\/yvonne\/"},"_links":{"self":[{"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/posts\/5181"}],"collection":[{"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/comments?post=5181"}],"version-history":[{"count":1,"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/posts\/5181\/revisions"}],"predecessor-version":[{"id":7064,"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/posts\/5181\/revisions\/7064"}],"wp:attachment":[{"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/media?parent=5181"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/categories?post=5181"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stumblingpast.com\/wp-json\/wp\/v2\/tags?post=5181"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}