Bringing Light to the Black Hole: Creating Web- Based Born Digital Collections in Art Libraries

Transcription

1 Bringing Light to the Black Hole: Creating Web- Based Born Digital Collections in Art Libraries Deborah Kempe Chief, Collections Management and Access, Frick Art Reference Library, The Frick Collection, New York, USA. Abstract: One of the fundamental challenges facing art libraries today is how to capture, deliver, and preserve art history resources that are increasingly web-based or born digital. The concern is that if these web resources which by their nature are ephemeral are not captured and saved for posterity, they will be lost forever, thus leaving a digital black hole in the art historical record. Web archiving as a library service has been slowly emerging over the past few years, with national libraries and large universities leading the way. Efforts by art libraries to archive websites have been even slower to develop, due to various obstacles, including complex copyright issues associated with images of artwork and small reserves of funding, research, and staffing. The New York Art Resources Consortium (NYARC) seeks to harness and give access to the increasing number of born-digital resources just as it always has for print materials related to art and art history. Since 2010, with funding from the Andrew W. Mellon Foundation, the NYARC libraries have investigated best practices for incorporating born digital resources into their collective collection and are now developing an active program that may lead the way for other art libraries. Keywords: art libraries; born digital documents; digital preservation. It is a great thrill and an honor to participate in this interesting conference in one of the world s great cities. Although I am a long-time fan of Brazilian music, literature, and food, it is only now that I finally realize a goal to actually visit this country. I deeply thank the sponsors of this conference for making it possible for me to participate, especially Isabel Ayres Maringelli, who is a wonderful professional colleague and a highly respected leader of forward-looking international initiatives by art libraries and archives. I will begin with a challenge, as the conference theme invites. A couple of years ago James Cuno, President and Chief Executive Officer of the J. Paul Getty Trust in Los Angeles, wrote a blog post 203

2 that received wide attention. Its title was How Art History is Failing at the Internet, in which he stated: One of the biggest challenges scholars and curators of contemporary art and architecture face currently, and will increasingly face, is how to store, retrieve, and investigate born-digital materials. Two years later, that statement remains true. Furthermore, the challenge is not limited to scholars and curators of contemporary art; rather it is true for all periods of art research. It is a challenge that must be met by archivists and librarians as part of our ongoing mission as stewards of the collective cultural record. I d like to ask you to keep this quote in mind as we continue through my presentation, while I back up a bit to outline what I intend to cover today. The focus of my presentation is a program in development to archive and create access to web- -based art resources at the New York Art Resources Consortium, also known as NYARC. To begin, I would like to put this ambitious program in context by providing some background on my own institution, The Frick Collection. From there I will continue with the formation of NYARC, a brief history of web archiving, and what drove NYARC to make capture, access, and preservation of born-digital resources on the web an early priority of our strategic plan for the future. You will hear about the five-year investigation into web archiving that led to NYARC s current program of creating publicly accessible web archives, made possible by a series of grants that provide the essential funding for planning and execution of our ambitions. I will cover the staff we have working on this initiative and on various new collaborations with outside organizations. I will go into some of the basics of web archiving and the tools and products we are implementing in our program. The subject of this conference drives me to address the challenges of creating born-digital collections and making them accessible through documentation. I will close with some thoughts addressing sustainability of these new collections. By means of introduction, I feature The Frick Art Reference Library on the Upper East Side of New York City, where I have worked for more than a few years and have seen many changes, even while its tranquil historic setting remains constant. The Library is located in a landmark library adjacent to the renowned Frick Collection, former home of the Gilded Age industrialist Henry Clay Frick and now a much-loved art museum housing a collection primarily of Old Master paintings and wonderful examples of European decorative arts. If you haven t been to The Frick, I hope you will make a trip soon including, of course, a visit to the Library. We have some rather nice paintings of our own there, not to mention a very deep and complete collection of books, photographs, auction catalogs, and e-resources. 204

3 The Frick Art Reference Library is a leading site for research on the history of collecting and for doing provenance research. In 2007, the Center for the History of Collecting was established, and its programs offer symposia, publications, fellowships, research grants, an extensive database of collectors archives, and an oral history program. Please visit our website at to learn more about the Library s many programs and services. In 2006, a pioneering partnership of three independent libraries took place, with the establishment of the New York Art Resources Consortium, or NYARC. The Frick joined forces with the libraries and archives of two other world-renowned museums the Brooklyn Museum and the Museum of Modern Art in order to create cost-effective solutions and to build a collective library with global geographic coverage that encompasses all of art history from prehistory to today. With the creation of a shared online catalog called Arcade, we virtually merged the excellent special holdings of each library to create an unsurpassed collection of more than one million items. Registered users of any NYARC library can request books to be delivered to their preferred reading room or remotely access shared e-resources. We were already a consortium when the 2008 Great Recession arrived and were thus better prepared than some to weather the budget cuts that resulted. In fact, NYARC has been the key to our success in securing grants because collaboration has become the name of the game in U.S. funding circles. Our commitment to working together in search of new solutions to key challenges is a core component of NYARC s success and has been crucial to the web archiving program we ve undertaken. NYARC has its own website at I urge you to have a look at it to learn more about the various programs we have undertaken. The program I ve been asked to address today is our most ambitious. Its goal is to avoid a situation that is often termed the Digital Black Hole. So what do we mean when we say Digital Black Hole? I think everyone is familiar with those frustrating 404 Not Found messages, alas. Links that once led somewhere are now simply gone. Good luck finding them. Here is another way of expressing it. There is a veritable ocean of data gushing into the Internet, but only a tiny bit is being archived in any organized way. The possibility of a Digital Black Hole of missing data is real. Because of the dilemma of modern media, more than one person has noted that in the future, without preservation, it s entirely possible that more may be known about the history of the 20 th century than of our current century. As Robert Wolven, an esteemed colleague based at Columbia University puts it, preservation is the global warming issue for digital content. 205

4 Web archiving is emerging as one way to harness the world as most of us experience it on the anarchic, lush, and vibrant landscape of the Internet. Indeed, so many of us live in this ever-present place of the Web that it is increasingly recognized as a common utility, at least for people living in the parts of our planet with easy access to the Internet network.. The Web is increasingly a source of what could be described as traditional academic publications as well as a forum for new models of scholarly exchange. Web archiving is still very much a nascent technology and is not widely adopted in libraries. Therefore, it is useful to consider the nature of a web archive. Although nothing is simple in this realm, a web archive may be defined as a collection of archived URLs grouped by theme, events, subject, or web address. A web archive contains as much as possible of the original resources of websites and documents the changes of websites over time. It is a priority to recreate the same experience a user would have if they had visited the live site on the day it was archived. Globally, there are substantial national initiatives in place to collect web materials emanating from within national borders, especially in Europe. The Bibliotheque nationale de France and the British Library are pioneers in collecting the output of national domains and their web archives are vast in scope and size. Switzerland, Austria, Portugal, Finland, Denmark, Japan, and Australia also have notable programs to capture national output. Wikipedia has a rudimentary list of international web archiving initiatives. Contrary to these global examples, there are no similar federal initiatives for whole-scale capture and preservation of websites in North America. The United States Library of Congress, alongside leading universities, have blazed the paths in the United States, but the efforts so far are rather modestly scaled projects to archive selective subjects of interest. But the longest running and biggest web archive to date is one based in the U.S. It is accessible through the Wayback Machine, part of the Internet Archive, thanks to one American individual by the name of Brewster Kahle. The Wayback Machine is a vast trove of web archives at the network level. Although imperfect and uneven in coverage, it is nevertheless vast and often offers the only hope of tracking down older versions of websites. Archived information found only in the Wayback Machine is heavily used by journalists and is increasingly cited by scholars. The world should be grateful the Wayback Machine exists, but one has to be worried about the permanence of this sole operation that is based largely on one person s vision and support. To quote from the Library of Congress Web Archiving Frequently Asked Questions: The Internet Archive is often thought to be archiving the entire web but in reality it is just a slice of what s available. 206

5 Therefore, it is important for libraries and archives to also select and create collections of archived web content. From this vantage point, it is easy to comprehend the need for additional efforts to create repositories that archive the web. Collaboration will be required to build lasting valuable collections. NYARC s interest lies in what has been termed the specialist web. In NYARC s case, this could be broadly defined as a subset of the web containing resources useful for research about art and its related disciplines. Even thus reduced, it encompasses a vast potential trove. Think of the presence of thousands of art gallery websites there are probably hundreds, if not thousands, of such sites for Brazilian galleries alone. Fig. 1 Baró Galeria, São Paulo, captured by Wayback Machine, Nov. 14, If the mission of libraries that support the study of art continues to be to collect, document, and provide access to art and its history for the long term, then obviously we must collect the web and its embedded documents. Several years ago, we at NYARC began to feel the subtle nature of a digital shift. Many resources in our collecting scope began a slow but steady migration from analog print to a born-digital format. Galleries, art dealers, and auction houses were turning away from printed catalogs and moving to the more dynamic format of the web to showcase their artworks. Although academic libraries have largely ignored the so-called ephemeral documentation of the art market, small publications from galleries, dealers, and artists have always been an important part of the NYARC collections. The appearance on the Web of traditional scholarly resources another core category of NYARC collections as born-digital was at that 207

6 time negligible, due to the conservative nature of the academy and a tenure-based publications track that refused to validate web-based publications. Nevertheless, even that category of materials had a detectable shift to documents that were accessible only via the Web. Despite the considerable naysaying of many in the art historical community, an inexorable shift was afoot. NYARC had no inclination to change our collecting mission merely because the nature of formats was changing, but the challenge seemed huge. This fundamental shift towards the digital was alarming because we had no ready means or solutions for making born-digital resources accessible. Even so, when we sought to highlight this critical challenge at meetings and conferences, the response was often blank stares. Scholars and some librarians in the field of art history seemed not yet able to comprehend the problem or why preservation of the Web would be necessary. Won t the publishers do this? was often the reply. Many just shrugged: Way too complicated, or Art books will always be printed, or People will always be able to find what they need using Google were some of the off-the-mark responses. As the trend from print to digital continues, the fundamental question remains: Who is going to make sure this information is around in the future? NYARC felt strongly that its future hinged on being part of a solution. We made access to born- -digital information a strategic priority in our plans and decided that inaction was not an option. We knew we d have to begin creating web archives and began tackling some basic questions to get started. For us, the question was not Why Archive the Web? but rather: How to Archive the Web? Who Should Archive the Web? Who Pays for Archiving the Web? How do People Navigate Web Archives? Five years later, we are still grappling with the challenges. Dealing with the web is complex and ever-changing. With the first question, How to archive the web? we ve made significant progress. For the remaining questions, the answers remain far from clear. The Web is always ahead of the technology to capture it. As the Web becomes ever more dynamic, wonderfully interactive websites like that of São Paulo-based Zipper Gallery push the boundaries of capture. Furthermore, what even constitutes capture or an archive when no two people have the same engagement with a website? It s complicated! or It s a Mess! are phrases we often find ourselves using. Patience and a sense of humor are required in this endeavor, or it would be all too easy to just give up. 208

7 But as we all know, you have to start somewhere, and where we got started was at a presentation in New York in 2009 by Kristine Hanna of the Internet Archive. We were encouraged to hear about the efforts by the Internet Archive and the new tools they were developing. The good folks there were kind enough to set us up with a pilot project to test Archive-It, the web capture tool they released in As we had no extra staffing or funding to devote to this project, we recruited graduate student interns to help. Thankfully there are several Library and Information graduate degree programs in the New York Metropolitan area which provided brilliant students. Our first pilot projects concentrated on capturing auction house websites, and we quickly learned just how challenging the process of web archiving is. These studies gave us a working knowledge of both the challenges and possibilities of web archiving sites that are specific to art history and art collecting. On the basis of this field experience and a proven reputation as effective collaborators, NYARC was subsequently awarded a planning grant by the Andrew W. Mellon Foundation to determine what would be required to proceed with building born digital collections of specialist art resources. The first project of its kind to address web archiving in this discipline, we named it Reframing Collections for a Digital Age and shared the results of our findings with the art library community. One outcome of the project was to examine when the tipping point that would change the domination of print to one of digital might occur. Fortunately we concluded there was some breathing room. A consultant engaged for the grant to investigate this occurrence estimated that the tipping point was 3 years or more beyond. Despite the slow rate, there remained a growing mass of Web-based art resources, with very little response by libraries or archives. We needed a road map. At the end of our grant in 2013 we had some recommendations that gave us a compass for planning PLANNING GRANT RECOMMENDATIONS Use Archive-It as the web archiving tool Plan incremental growth of collection Develop an open nominations tool Establish a permissions framework Join the National Digital Stewardship Alliance (NDSA) Look for ways to further automate metadata creation Enlist students into the program, especially for quality assurance Collaborate, Collaborate, Collaborate 209

8 We incorporated these recommendations into a proposal to build a carefully selected collection of websites for art research that would be considered part of the curated NYARC collection and be made available to the global community over the web, of course. The Andrew W. Mellon Foundation once again supported NYARC, and later that year, awarded us a two-year implementation grant. We knew we were not going to fully close the black hole, so we called our program Making the Black Hole Gray: Implementing the Web Archiving of Specialist Art Resources. I am the Principal Investigator for the grant, and we are now entering the second and final year. The grant provides funding to hire staff and consultants to harvest and catalog approximately 2 TB of WARC (Web ARChive file format) files from websites, to install a software solution to facilitate discovery of our web archive, and to plan and implement new workflows that incorporate best practices that can benefit the library and archives community. The website collections we are currently building focus on subjects and formats of prime importance to our staff and library audience: our own institutions (i.e. the websites of The Frick Collection, the Museum of Modern Art, The Brooklyn Museum, and NYARC); auction houses; catalogues raisonnés; artists websites; galleries based in New York City; and resources that document research on provenance and restitution of looted art. Fig. 2 Isamu Noguchi Catalogue Raisonné, captured with Archive-It in

9 Our current workflow can be described as consisting of five basic elements, beginning with nomination and selections of websites, moving to a permissions process (asking for and hopefully receiving permission); proceeding to harvesting and quality control; providing description and means of access; and finally, long-term preservation. Fig. 3 Web Archiving Worklflow To carry out this process requires a fair amount of technology tools, and they all have a cost. We are currently using: Archive-It to harvest WARC files; Wayback Machine to display our collected websites; Hanzo Archives to capture websites not successfully captured by Archive-It; and Filemaker Pro to build a trackable permissions database. In the next year, we will utilize Ex Libris Primo as our primary discovery tool and Duracloud for backup copies of WARC files. We ll also look into BitCurator and other sources of digital forensic tools. Archive-It is our main tool for web archiving. An annual subscription service based on open source tools and international standards, Archive-It is continually developed and maintained by the Internet Archive. We have been pleased with how its staff responds to the increasingly sophisticated demands of a growing user community. There are other web archiving tools available, and it is also possible to create in-house systems that require substantial technical support. More than 330 partners throughout the world use Archive-It and the numbers continue to expand. However, the number of museums and art libraries using it is very small. Currently, only twelve such partners are listed and some of them are no longer active. 211

10 Archive-It allows you to build specific collections by crawling a curated set of URLs, or seeds, the term used by Archive-It. A seed is basically any part of a website, from the entire domain, to a group of pages, to a specific document. Archive-It also allows each partner to determine the scope of the crawl how far into the site it will go, and the frequency of the captures. These are important but difficult to determine factors in building web archives. Backend features of Archive-It track captures and facilitate entry of Dublin Core metadata. Subscription to Archive-It provides ongoing training, support, and distributed backups of data. They will also supply copies of WARC files. Together with the broader web archiving community, we are grappling with how to insure long- -term preservation of the WARC files we re creating. For the near future, the Archive-It service and additional redundant backups through Duracloud will be a prudent safeguard. In a welcome development, Duracloud has partnered with Archive-It to provide automated backups. To NYARC s good fortune, we have a specially-funded National Digital Stewardship Resident who is currently investigating best solutions for secure and long-term preservation of our web archives. When Archive-It cannot successfully capture content we have selected, we turn to a commercial vendor called Hanzo Archives. Hanzo uses proprietary and considerably more costly methods for capture and has so far been able to capture the particularly challenging sites we have requested. Whether this component will be feasible post-grant is not certain, but it demonstrates the need for further software developments to capture the especially dynamic nature of cutting-edge web design. Thanks to our grant, we worked together with Archive-It to successfully integrate WARC files from Hanzo into the Wayback Machine display, demonstrating that this enhancement could be a solution for Archive-It partners with WARC files obtained from third-party sources. Using Archive-It alone, however, is only one piece of the web archiving pie, and this fact is fundamental to the theme of our meeting. Documentation by experts will be required to adequately connect researchers to the collections. Metadata, discovery, and access are areas in development at this stage of our program. From the onset, we realized that neither Google searches nor the MARC records in our online catalogue would be sufficient for discovery of our new Archive-It collections. We intended to implement a new software platform to facilitate discovery of integrated content, but the problem was none of the available products seemed able to handle web archives. As it turned out, only two providers, Ex Libris Primo and Deep Web Technologies Explorit Everywhere! products were able to provide a Proof of Concept demonstrating the ability to include results of 212

11 Archive-It searches alongside other resources. With input from a NYARC committee tasked to evaluate the two proposals, Primo was selected as our discovery platform. When we implement the Primo product next year, it will sit on top of numerous NYARC resources, including Archive-It. Searching in Primo will bring back integrated results from our Archive- -It collections using an API. We are confident this enhancement will direct users wherever they are to the resources they need, regardless of the format, be it print, online databases, image repositories, or web archives. We ve been describing websites in our online catalog for several years and have more than 500 records describing and taking people to important sites for research. These sites are not products we buy, but places on the Web where valuable information lives. As such, they are free to access via search engines, but we enhance discovery by adding descriptive metadata in our NYARC catalog. It s not always easy to discover these sites in Google, and we have vetted them as having research value. Not only do we include the current url in our bibliographic records, we add a link to any existing occurrences in the Wayback Machine. Directing researchers to historical instances of websites harnesses the power of the Wayback Machine. As further advice to web archiving without an investment in Archive-It, you can perform on the fly screen captures in the Wayback Machine using a feature called Save Page Now. There are additional useful tools and tips on the Wayback Machine website. Perma.cc is another valuable and free solution that helps scholars create permanent links to the online sources cited in their work. To help the cause, advise web content creators to use methods friendly to web archiving. There are a number of resources available that can provide this advice. I encourage you to become familiar with them and to tell your colleagues, patrons, and web managers about them. There remain many challenges in instituting metadata and workflow practices. Our staffs are busy enough with cataloging incoming print and e-resources. How can they possibly take on the multitude of digital documents embedded in websites? Item-level documentation has been our standard practice for many years. We have to rethink this model. The discovery layer, Primo, will help, but we must analyze how it works when implemented. There is a need to further automate metadata generation. Systems need to talk to each other better and APIs are generally the solution. We would like for the Wayback Machine API to interface with OCLC for metadata exchange. 213

12 With this new field of web archiving, the boundaries between archives and libraries become blurred, and it is sometimes difficult to decide on appropriate levels of description. On the other hand, bringing more staff together to solve common problems is overall a positive thing. Best of all, the NYARC group has two renowned consultants on board to help us plan for a future that addresses the importance of digital records. This is an area where you ll have to stay tuned. All of these situations I have described become even more daunting as we try to build sustainable programs. Changes in workflow will be required, transfers of funding will be required, and changes in attitudes will be required. None of this is easy. It requires institutional commitment and individual determination that web archiving services are a priority for libraries and archives. The most important conclusion I can make is to return to the word COLLABORATION. A multi-pronged approach will be required to secure the digital record of art history. Simply put, libraries alone will not be able to secure the digital record of the Web, though it must remain a priority. A broader form of commitment from governments, national libraries, academic and special libraries, as well as content providers and publishers must be in place to create a sustainable archive. It must involve the entire ecosystem of scholarly production. The practical pathway for NYARC is to continue building partnerships and collections with other libraries, and reaching out to educate content producers on the value of preserving their own born digital content. Certainly we look to the larger universities that have set precedents in building web archives. Columbia University, for example, has initiated shared collections of contemporary music websites and of geographically based architecture, and we are closely following these efforts as a model for our own goals to extend shared collections related to art history. The library and archives community has collective expertise in information organization, taxonomy, metadata, and web architecture we must effectively leverage these skills to encourage development of tools to standardize and improve workflow, including the development of APIs. In the end, it all comes down to articulating our core mission. As the nature of scholarship, enabled by the Web, becomes ever less respectful of boundaries, our practices to create collective digital collections must follow suit. The very future of art libraries depends on it, and I think all of us gathered today should be a part of it. 214

13 Links to resources cited, and other useful information on born digital content Collecting and preserving the World Wide Web: a feasibility study undertaken for the JISC and Wellcome Trust (Febr. 2003) A Survey on Web Archiving Initiatives (Foundation for National Scientific Computing, Lisbon, 2011) Archiving the Web: A Case Study from the University of Victoria, Code(4)Lib Journal, Issue 26, Web Archiving in the United States: A 2013 Survey: An NDSA Report blog posting by Stephen Bury, June 18, 2012 New York Art Resources Consortium (NYARC) FAQs on Web Archiving org/content/faq-web-archiving Columbia University Web Archiving Summit Archiving the Web for Scholars, by Steve Kolowich news/2011/05/06/libraries_try_to_preserve_and_archive_websites_for_academic_study Overview of Web Archiving, by Jinfang Niu Web Archives for Researchers: Representations, Expectations and Potential Uses, by Peter Stirling, Philippe Chevallier and Gildas Illien. D-Lib Magazine Mar./Apr. 2012, Vol. 18, No. 3-4, A Memory of Webs Past, Ariel Bleicher, 28 Febr., internet/a-memory-of-webs-past/0 Digital Scholarship s Digital Curation Resource Guide 215

14 Further Resources Library of Congress, The Signal: Digital Preservation blog: Library of Congress Web Archiving website of the International Internet Preservation Consortium (IIPC) SAA Web Archiving Roundtable Archive-It Knowledge Center Guidelines for Preservable Websites / Columbia University Libraries edu/bts/web_resources_collection/guidelines_for_preservable_websites.html Supporting the Changing Research Practices of Art Historians / Matthew P. Long, Roger C. Schonfeld, Apr. 30, Changing-Research-ArtHist_ pdf 216