The advantage of working with highly customizable, open-source tools is the ability to tailor your workflow to the specific needs of your institution or collection. The disadvantage is that you have to customize your workflow, and by that you may find yourself testing configurations in seemingly endless fashion. With so many manually entered steps along the way, a single mismatched configuration can throw pieces of your processing workflow into a tailspin and leave you, hoping, in bouts of desperation, for a one-size-fits-all magical turnkey solution to all digital archival objects for their forever-and-ever-nothing-will-ever-be-lost deep storage.
Currently at the WCS Library and Archives, we are relying on BitCurator and Archivematica to process our digital collections. We’ve also been working with ArchivesSpace to describe our digital collections and DuraCloud as our cloud-based storage. A part of our Legacy Digital Project has been to test and implement procedures for DuraCloud managed archival storage. We’ve also been busy testing and producing workflow policies for disk imaging legacy media in a variety of formats (3.5” floppy disks, 5.25” floppy disks, zip disks, CD’s and DVD’s) and getting files off their physical carriers (USB flash drives and external hard drives) in a forensically sound manner, using BitCurator’s suite of forensic tools.
The third phase of our project has been to test, configure, and deploy our workflows using Archivematica to process our digital objects. Archivematica is open-source and standards based (on the OAIS functional model), and it is managed through a web-based dashboard. A nice feature in Archivematica is that it allows you to configure a space, which could be your preferred cloud-based storage option with a specific location, a subdivision of a space which can be assigned for a specific purpose or usage, such as AIP (Archival Information Package) storage, DIP (Dissemination Information Package) storage, or even as a Transfer Source on your Archivematica pipeline (i.e. a local installation of an Archivematica dashboard).
Our first task was to set up DuraCloud as our archival storage on Archivematica. We did so, closely–almost obsessively, following the Administrator Manual’s Quick Start Guide. With that worked out, we were able to test our ingest capabilities with the contents of a collection of 69 3.5” floppy disks. The floppies are from 2001 or earlier and are associated with the WCS’s Asia Program. The entire collection amounted to approximately 2,000 files which we were able to successfully process, transfer, and ingest with Archivematica. I felt like Brian Eno in Before and After Science, a cosmonaut, guiding thousands of files through pathways, towards their archival safe haven.
Look at all those successful microservices!
But, I fell into despair as I started to process another collection, from the WHHP Field Vets records (an amazing treasure trove of documentation from WCS veterinarians working in the field across the globe), and the ingest of DIPs (Dissemination Information Packages) started to fail. Like that, a domino effect of failures started to unfold upon me. After cruising Archivematica’s User Forum, and some tinkering with Archivematica’s processing configuration settings, the “Generate AIP METS microservice” started to fail as well. When switching my transfer type from a Standard one to one attuned for disk images (for which I had to create rules and commands for it to “accept” ISO and E01 disk images), the “Generate AIP METS” microservice would not fail, but the “Verify AIP” microservice would consistently fail, throwing an additional obstacle my way.
A bouquet of FAIL!
At the time of writing, we have not yet figured out a solution to our microservice errors, but we are persisting. Though not all is lost, I find it useful to read other digital archivists’ journey towards “figuring it out”. The Society of American Archivists’ Electronic Records Section blog series on #digitalarchivesfail has been an enlightening, empathic exercise, and a reminder to hang in there while figuring it out. A lot of insight and learning can come of it!
“Hang in there, baby” poster from The Simpsons
This post highlights work that is being completed as part of our Legacy Digital Removable Media Project, which has been generously funded by the Leon Levy Foundation. For more about the project, please see this October 2016 post.