http://www.theatlantic.com/technology/archive/2013/04/google-death-a-tool-to-take-care-of-your-gmail-when-youre-gone/274934/

http://googlepublicpolicy.blogspot.com/2013/04/plan-your-digital-afterlife-with.html

Looks like a great outreach tool for use with potential donors who are not quite ready to share their digital stuff with an archives.  All you need to do is talk people into listing your archives email account as a contact!

Many reasons exist, however, for why files may wend their way into repositories without file extensions. Considering divergent file-naming practices over time and changes in the ways software renders extensions, as well as the multitude of events that corrupt or irrevocably alter files during the course of their lives, it is no surprise that file extensions are either missing or a vestige of a file-naming convention that has long since been superseded. Without file extensions, the file has lost a portion of what the OAIS reference model calls its “representation information” (the data the computer needs to render the file by opening it in the proper piece of software). Being able to recover such information with a file identifying utility is vital to providing access to electronic content.

As Chris has previously mentioned, unless file formats are correctly identified, it will be difficult to preserve them for posterity and research use. Likewise, he notes that many applications which are used in the management of electronic records require that records be in ‘archive ready’ form – i.e. that they have the correct file extensions and be situated in directories without other formats, etc. Applications such as JHOVE and DROID extract technical metadata and can thus be exceedingly useful tools to use in identifying file formats. As an appraisal tool, DROID’s analysis of file formats can help us better understand the components of a particular set of electronic records, though it does not append file extensions or assist in the processing of an access copy of records.

It may be helpful to use a file identifier like TrID with the capability to append file extensions in conjunction with an application like DROID during the process of arrangement to create an access copy of electronic content.

As part of a project to process the born-digital component of a hybrid collection, I noticed that file extensions were missing from a large number of files and decided to try a file identifying utility called TrID in order to process an access copy of the records. Created by Italian software developer, Marco Pontello, TrID is a CLI executable file identification utility that analyzes a file’s internal header information and compares it to a comprehensive list of file signatures, similarly to DROID. While DROID uses PRONOM’s database of file signatures, TrID relies upon an extensible list of file definitions. TrID’s file signature definitions currently consist of 5037 file types, which are continuously increasing. Users can actively add to the definitions by utilizing TrIDScan, which analyzes the binary format of a set of files of a known file type in order to build a recognizable signature. This is exceedingly useful, especially if one has a large number of file formats not recognized by TrID. Unless a file format is not within TrID’s extensive list of file signatures, virtually any kind of file can be identified and the correct file extension appended. For those who are not comfortable with the command line, it is worth noting that TrID has a GUI called TrIDNet, as well as an online version.

Installing the CLI version of TrID proved to be fairly simple — I downloaded the zip file along with its file-type definitions from the website. I saved the file definitions in the same folder as the application, since the program looks there for the file by default. TrID supports various arguments, or command options, that can be used to modify its behavior (Figure 1).

Figure 1 – TrID command options.

For instance, it can be told to append file extensions automatically (the “-ae” flag), replace existing file extensions with the guessed extensions (the “-ce” flag), or to leave the file unmodified (default behavior). In addition to analyzing files and appending extensions, it also provides a percentage of confidence as a gauge of accuracy for its identificatory capability (Figure 2). While having this information is certainly important, it is hard to evaluate the significance of ambiguity represented by statistics where the file has not been identified with a high percent of confidence. Nevertheless, I noticed that TrID proved to be an effective processing tool for an access copy of electronic content and was able to identify the vast majority of the file types that had previously been elusive.

Figure 2- TrID identifies different file types with varying degrees of confidence.

Despite TrID’s effectiveness, it is worth noting that it does have some shortcomings. Although it is very good at identifying ASCII files, it is not especially adept at distinguishing between different flavors of ASCII. Because ASCII files are somewhat of a “digital free for all,” this limitation is understandable. TrID provides this warning when analyzing ASCII files: “TrID is best suited to analyze binary files!” (Figure 3). The vast majority of the time this will not be problem, as the mere identification of the digital object as an ASCII file is sufficient for most processing work that is aimed at bit-level preservation. An instance may arise, however, where this could be problematic. For example, after running TrID against an ASCII PKCS7 digital certificate file (file extension .p7b), TrID correctly identified the file as an ASCII digital certificate, but thought it was merely formatted as a CER file. That is not incorrect, but it is also not as correct as it could be. It is apparent that TrID loses some granularity in its identificatory acumen when analyzing ASCII files.

Figure 3 – TrID output after analyzing an ASCII file.

It is also important to note that while TrID supports wildcard characters, and can evaluate many files with single command iterations, it is unable to simultaneously inspect the files in a directory and a subdirectory. The practical import of this is that each directory and its subdirectories need to be evaluated separately. Furthermore, TrID currently lacks the capability to generate reports, which would be important to include in an Archival Information Packet. These two criticisms, however, must be tempered with the knowledge that TrID is a very lightweight utility that can be easily called from, or integrated into other programs. A relatively simple script can be written to find all the files in a given set of directories and subdirectories, and those files paths can be passed to TrID as variables. For instance, a Windows PowerShell script can be used to recursively analyze a directory structure to append file extensions in subdirectories. Similarly, if one has a report writing or data analysis program, TrID can be easily called from that program to give it enhanced functionality.

TrID is a very effective and powerful utility that is straightforward to use and can be an exceedingly practical tool to incorporate into an electronic records processing workflow. It already has a vast collection of file definitions, the list of which will grow as users continue to engage with the program. As a processing tool, TrID complements such applications as FITS, JHOVE, and DROID, though its important to note that it should be only be utilized for the creation of an access copy of records, not a preservation copy. In order to truly test its robustness, as well as its compatibility with DROID, it would be an interesting exercise to compare the TrID’s results with those of DROID in the analysis of the same set of files. Such a comparison would yield a more telling picture of the accuracy of our digital tools and lead to new ways they can be implemented and integrated into an e-records processing workflow.

Evaluation Criteria

• Installation/Configuration/Supported Platforms: For a novice like me, installation of TrID was easy, as it only required the downloading the program and its definitions in a zip file. It can be run in a Windows or Linux/Unix environment as well. 20/20.
• Functionality/Reliability: Despite the ease with which TrID can be integrated into existing or new workflows as a processing tool, its GUI option, while user-friendly, can only analyze one file at a time; it does not provide an automated way to analyze multiple files in a directory at once as one can on the command line. Additionally, since it does not distinguish among ASCII files, this can be problematic if one is processing records that contain a large number of ASCII files.  18/20
• Usability: For those familiar with the command line, TrID is easy to use. As mentioned above, a GUI option is available as well.  10/10
• Scalability: Since the GUI option can only be run against one file at a time and the command line option can only be run against single directories, the utility has limitations dealing with large datasets. 7/10
• Documentation: The utility is so user-friendly that it require little documentation; however, it would be nice if the website would provide a means to evaluate the analysis of file types where it does not have 100% confidence.  7/10
• Interoperability/Metadata Support: Since anyone can add to the TrID file definition library, it’s very interoperable because it has a built-in learning capability. 10/10
• Flexibility/Customizability: While TrID is not itself customizable, it is a utility that can be integrated into other programs. 9/10
• License/Support/Sustainability/Community: TrID is freeware that can be used for personal, non-commercial, educational, and research use. It also has an informal online support forum. Because the TrID definitions can added to by anyone, the value and capabilities of the utility will grow over time as it gains more users. 10/10

Overall Ranking: 91/100

Bethany has a MLIS from the University of Texas at Austin, with a concentration in Archives and Records Management; she also holds and MA in Near Eastern Langauges and Civilizations from the University of Chicago and a BA in Anthropology from the University of Michigan.  She most recently worked at the Dolph Briscoe Center for American History at the University of Texas.

As part of her initial training, I asked Bethany to process to the paper and electronic files of William Greenough, a former professor of Molecular and Integrative Physiology, Psychology, and Psychiatry.  As part of this, she discovered a very useful tool, which she’ll be reviewing in the next day or two.

In one sense, the content of the course summarizes much of what I set out to learn and communicate during the the sabbatical project that gave rise to this blog.

I really enjoy teaching the course, and I think that now that it has been expanded into two days, the course is an ideal format to maximize the learning and implementation potential concerning the critical topics it covers.  One the first day,  we go over the the fundamental concepts (such as functional requirements, selecting tools and services that meet repository needs, forming an archival information packet, the value of DACS). and undertake some exercises aimed at helping people develop a workflow for their repository.  On the second day, we introduce advanced topics and have lots of time for hands on work, practice, and individual attention/question answering/discussion.

I think the course is particularly helpful since it touches on nearly every aspect of digital archives work, in hopes of helping people develop an understanding of the myriad factors–from appraisal to access–that must be considered to make an archival program effective at providing access to born digital materials.

My goal is to ensure that every single attendee leaves the course with a solid, practical plan to make immediate progress in arranging, describing, and providing access to born digital materials–whatever his/her background and institutional context.

If this is a topic that interests you, there are still spots available for the next time I teach the course:

• April 8-9: Eastern Washington University, Cheney, WA.

Register while you can (and note the early bird deadline is nearly here!)

New University of Illinois Archives Website, as of July 24, 2012

The site includes site level navigation along the top, as well as a context sensitive navigation system on the left.  A blog is integrated under the ‘what’s new’ tab.  In addition, I added a custom graphics and adjusted the size of the header and other menus.  A few small changes need to be made before the site can be launched to the public, but it is basically functional and ready for usability testing.

If found the following resource very helpful in setting up WordPress as a CMS using a Thematic Child Theme.  I’ve put these in a logical order, and if you also want to implement Thematic as a CMS, I’d suggest working through them in the order listed.

1. Catswhocode Thematic Tutorial.  This beginner’s level tutorial will help you get started on theme customization, the right way.
2. Simple Section Navigation Widget makes it a snap to have a cont context-sensitive “local” menu on the left or right side of your site, allowing people to browse through your static pages.
3. Themeshaper Theme Development Tutorial. This very extensive series provide a great deal of background information you will need to be successful in creating a child theme.  It lists tools, provides background on the structural HTML elements typically used in the <div> elements of a WordPress site, recommends a suggested template and directory structure, tells you how to fill in the templates (such as header, footer, and index.php), and helps you understand how to call and use functions to fill in the templates.  The information is useful in a general sense, but does assume a certain amount of background knowledge, so I suggest reading item number two and three (shortcodes and functions) while going through this tutorial.  One should also be aware that the tutorial is talking about ‘regular’ (not parent/child) themes, so a lot of the specific techniques discussed in this tutorial apply to all wordpress development.  But, it is good to understand what is happening across wordpress before delving into a child theme creation.
4. WordPress Child Theme Basics.  This very helpful tutorial describes how to link your child theme to a parent theme and how to begin customizing your site.  It outlines the structure of the theme, noting how you import the parent theme styles and functions, then use style.css to override the parent theme styles.  It also delves a bit into using the file functions.php to override some of the parent themes functions.  (This is important if you want to change the output of a particular element in the HTML output.)
5. Thesis Folder Structure.  Eric Hansel recommends a logical and easy to understand structure for your theme files.  Using this will help you keep everything organized and understandable to others (and yourself.)
6. Themeshaper action hooks tutorial. WordPress itself, as well as most themes and plug ins make use of a ‘hooks’ system.  Action hooks are typically used to put some content into an ‘empty’ placeholder area on the page.  (Theme developers provide these areas so widgets or other content content can be added.  For example, you may want to add a copyright noticed after the footer; in thematic you would do this by ‘hooking’ the action to the thematic_below_footer() function.
7. ThemeLab Thematic Tutorial.  This is an intermediate level tutorial, which goes into quite a bit more detail about action hooks and how to modifying some of thematic’s built in functions by using filter hooks.
8. Themeshaper filter hooks tutorial.  Like many themes, and WordPress itself, Thematic includes action hooks.  Action hooks allow you to ‘hook into’ and modify or filter the content that will be placed on the page.  For example, if you want to add an image to a header, you might use a filter hook to to replace part of the existing header content and to put the image into a particular location in the file.  See also this page explaining the differences between action and filter hooks.
9. This layout page shows the locations of many of the widget areas, divs, and action/filter hooks.  It does not include all of the hooks or filters in the latest versions of Thematic.
10. Net magazine has a tutorial regarding shortcodes, which are are macros or shortcuts you can use to put a piece of dynamic content onto a page.  It explains how to create, register and reference your own shortcodes.  You might need to define some shortcodes if you plan to create a custom widget or code that might be reused on several pages.
11. WordPress Hooks Snags and Barbs. This is a very advanced tutorial, which goes into the details regarding how action and filter hooks, as well as the wordpress loop really work; it might be helpful to refer to this only if you need to diagnose a ‘misbehaving’ hook, and understanding all of it will make you a much better wordpress developer!

At this point, I have a functional website in place, and by early August, we will be enabling the site.  At the same time Jameatis Johnson, our new Archivist for Outreach and Engagement, will begin a round of iterative usability testing on the site.    Before we begin that project, I would like to describe the rationale for the decisions taken to date, as well as to outline some of the basic work that has been completed to set up the site.

Firs, a bit of review:  In my first post, I described the rationale for attempting to run an archives website as Content Mangement System powered by WordPress. My second post covered the basics of WordPress installation.  After I had installed the site, I decided to blow it away, and to reinstall with WordPress enabled for a multisite installation; I covered this topic in my third post. Finally, I gave some background on the WordPress Hooks  system, since understanding hooks is a necessity if you want to do serious customization work with WordPress.

Now it is time to discuss theme selection.

It is easy to pick an off the shelf theme, install it, and to never look back.  However, the real power of wordpress emerges if you are able to customize the theme to your own liking.  Furthermore, you can get a much slicker and user friendly web presence if you implement a good theme.  This is particularly true for if you plan to run WordPress as a CMS.

Several themes support CMS like features out of the box.  On the paid side, the Thesis and, especially, the Genesis Frameworks are particularly good.  I investigated both of them, but ultimately decided not to buy either one of them.   Why?  First, I am cheap, and don’t like paying for something or going through the hassle of getting approval, processing the invoice, etc.  Second, both of them use quite a few theme-specific hooks or functions.  Any functionality I build on top it might run a big risk of theme lock-in.  But most importantly, I ran a usability analysis using the University of Illinois Functional Accessiblity Evaluator on the output of some Thesis and Genesis implementations.  I was very concerned that neither theme included good web design practices to make the pages easily useful for people with disability.  (Not only is it Illinois Law that University sites provide good usability, it is in our interest and is just plain the right thing to do.)

Therefore, I started looking at some free/open source themes, browsing the free themes repository for a theme that was a) CMS-friendly, b) reasonably friendly to people with disabilities, c) easily customized, and d) flexible.   Suffusion seemed to be a good option, since much customization could be done via a graphical interface, and one librarian had a lot of success with it. (Here is a link to a book about using WordPress as a CMS.)  I did install suffusion, and had some luck customizing it.  However, I found that it was very difficult to predict how the changes made in the graphical theme editor would affect the output.  I also had trouble making some changes without editing core source files in the theme–a bad idea since any changes would need to be updated next time.  Finally, I was again concerned that if I chose the theme, I’d be locked in to it for life, since it makes extensive use of theme hooks and extensions.  While many of these extend core functions in wordpress, some of them seemed likely to be theme specific, so that features integrated on the site would not necessarily translate to a new version of  wordpress.  Therefore, I uninstalled Suffion.

About this time, I began looking at themes that would allow me to develop my own child theme.  Child themes work by referencing a ‘parent’ theme.  The parent theme does the heavy lifting of laying out the page and plopping content into it.  However, any elements of the parent theme can be modified in the custom theme, while running minimal risk that updates to the parent theme will break the child theme or the site as a whole.

After investigating several options, I settled up on the Thematic Framework. There are several reasons:

• Out of the box, the theme is very plain, in the best sense of the word.  The pages are laid out in well-labeled, logical <div> elements, and no colors are applied, allowing me as the developer to apply styles very easily.
• The theme is very friendly to people with disabilities.  Its output passes the functional accessibly evaluator tests.  For example, the menus it produce include a hidden link to allow people using screen readers to skip to the main portion of the site.
• The themes ‘back end’ files are laid out in a very logical, easy to understand way.  (A screenshot of the overall organization is below this bullet.)  If I need to understand where a certain style is applied or function is located, I can find it easily, study its content, and then replace or extend the code in the child theme.  As you can see, there is a simple list of php pages that produce the output of a post/page, as well as a master css file, then libraries which include all of the functions, page layouts, substyles, etc.

• Quite a few people are using Thematic, and there are a wide range of resources for understanding the theme and for implementing it.  I’ll have a list in my next post.
• It is developed by Automattic, the developers of WordPress itself.
• It is very rich on features and lean on bloat; very unlikely that any functions introduced in it will not mesh well with WordPress as a whole.
• In my test of the theme, I found it very, very easy to make simple modifications to the output.  Every change I made (whether it was to page layout, colors, etc) required only a few changes in the sample child theme supplied with the theme.  I was even able to tap into the theme functions and to modify them via the child theme, without any real trouble, after reading a bit of the documentation. The changes had the expected results, and I was able to roll them back without any problems.
• The fact I would be developing a child theme allowed for the possibility to share my work with the WordPress free themes project, as well as to use it as the basis for sub-sites within our WordPress multisite installation.

Next post, I’ll walk through my initial customization of the site by explaining how I developed the basic child theme.

Now that I am an archivist, I retain an interest in these topics, but with the twist that I also like to keep tabs on how records about financial history are generated, used, saved, uncovered, preserved, and used.  I typically spend  a portion of my lunch hour reading about economic topics.  For the most part, these topics are documented via the ephemeral communication formats used in modern society: blog posts, PDF reports, tweets, instant messages and—especially—email messages.  The news regarding our economic problems provides a strong rationale for society’s need to identify and preserve records that have permanent archival value.

To date, I have seen no greater evidence of this fact than the LIBOR scandal and the alleged crimes that allowed that scandal to develop.   I would even go so far as to say unless sufficient evidence regarding LIBOR manipulation is preserved, we will never understand the true causes of our current economic problems, much less be able to fix them in a way that will prevent them from happening again.

If you haven’t heard of the LIBOR scandal, it involves the falsification of information concerning the interest rates at which 18 ‘megabanks’ borrow money from each other.  Here are some sites I’ve found particularly useful in wrapping my head around what this scandal is all about:

• Business week article about fines given to Barclays Bank, and resignation of CEO Bob Diamond.
• A simple explanation of how LIBOR frauds operated, and their its effects on ordinary people.
• NPR piece on the scandal
  
• Former bank regulator William K Black on the role of US Federal Reserve
• Blog post by the London Banker blog, providing an insider perspective
• Economist magazine editorial
• Blog post by Simon Johnson

The fact that an interest rate was manipulated sounds like an arcane piece of financial news.  (The scandal has not been heavily reported in the US media, but it caused a firestorm in Britain.) But given what I have read, it is possible that the scandal will continue to grow, and become bigger and bigger news over the summer.

If you read each of the above pieces carefully, you will find that a small number of people engaged in self-dealing at a systematic, global scale, without any effective check on their power–exactly the type of situation our founding fathers warned against.  The fraudulent behavior of financiers and banks potentially harmed billions of people: anyone who has borrowed or lent money over the past ten years.  LIBOR is one example, but how many others lie waiting to be uncovered?

Many Americans (Tea Party Members and 99%’ers alike) feel that the financial system and economy are rigged against them. LIBOR shows one specific way that they are correct, in the most fundamental way.  It also shows that our governments have no way to prevent future scandals of a similar nature and that that nothing of substance has changed since the 2008 financial crisis.

The scam worked like this: banks falsely reported information used to calculate a benchmark interest rate, a rate against which against which over 500 trillion dollars of other securities are priced. They conspired to report either higher or lower interest rates than they were actually paying.  This allowed them to falsely price securities to their own advantage, instead of allowing rates to be calculated on the open market.  They may even have shaved fractions of a percentage of interest from those other securities, to benefit the bank as a whole or an individual trader, to the detriment of everyone else.

Anyone who holds a mortgage, a municipal bond, a derivatives contract on a future bushel of corn, or any other security pegged to LIBOR was potentially harmed.   As a result of this activity, Barclay’s bank has already paid a fine of $455 million dollars, and the other banks are being investigated.

As several of the authors above argue, the LIBOR scandal lays bare the corruption of the the global system on which modern finance and financial contracts are based.  The lack of mainstream media attention to these events does not obviate the fact that they took place, and that similar events will continue to take place unless those responsible are brought to justice and the financial system is repaired.

For that reason, it is interesting to note that much of the evidence concerning these incidents is drawn from email.  The legal pleading submitted by the Commodities Future Trading Commission and leading the the Barclays fine makes this clear.  A portion of the evidence cited in the report, particularly as it relates to trades directed by senior managers is drawn from recorded phone messages, but most evidence regarding daily manipulation, to benefit Barclay’s trading position, comes from email records of individual traders, who became so blaze about their behavior that that even set calendar reminders to themselves so that they would not forget to submit fraudulent reports.

Kind of make you wonder how other banks, the Federal Reserve, and other organizations are managing their email and instant messages, doesn’t it? Wouldn’t that be the kind of thing that might be of interest to a congressional committee—if our political system were ever able to muster up the courage to investigate the banking industry like Ferdinand Pecora did in 1932?

Since these remarks will be revised for publication, please see the important notices at the bottom.  Also please note that no citations are included here; if you would like a copy of the paper with citation, please contact me.

Facilitating the Generation of Archives in the Facebook Era

Keynote Address: 17^th Brazilian Congress on Archival Science

Rio de Janeiro, June 21, 2012

Last November, I boarded a train at Union Station in Chicago, Illinois.  I had just a left a meeting of the Society of American Archivists’ Fundamental Change Working Group.  This group was charged with revising the Fundamentals Series, which comprises the heart of our society’s publishing program. Everyone at the meeting was acutely aware of two facts: 1) that newly trained archivists need a sophisticated set of digital skills, and 2) that our new instructional manuals must facilitate these skills.

Moving quickly to find a seat on the train, I spotted a person from my University.  I’ll call this person “Dr. Important.”  It was someone holding a senior position.  After the requisite chic-chat, Dr. Important asked me what I had been working on lately.

“Well, I’ve been writing a guide to email preservation.”

“Oh, that’s interesting.  Maybe you can help me.”

Who doesn’t like to be asked for help?  Maybe I could tell Dr. Important how to organize email and export it to a preservation-ready format.  If lucky, I might even convince Dr. Important to transfer email to the University Archives, where it would become a public research resource.  In this way, it would be accessible much like the print correspondence from many earlier “Dr. Importants.”

“You see, I went to look for something I sent back in 2009.  I’ve been keeping a copy of all of my important emails, one folder for each month.  But when I went back to find the message I needed, all the folders were gone.”  Dr. Important told me that technical staff could not restore the emails, which likely went missing during a system migration that had taken place several months prior.

As an archivist, I mourned the death of the evidence Dr. Important had created and cared for over many years.  But I felt helpless and let the conversation float toward another topic.

This story, and many others that I could tell from my time at the University of Illinois, illustrate one of the biggest challenges that we as a profession face: ensuring the preservation of evidence when people’s communication tools have, in effect, become the organization’s official recordkeeping mechanism.

Given the breakdown of centralized recordkeeping systems, what can we (as an archival profession) do to make sure that usable records are fixed into a medium that will facilitate their perseveration and use?  In order to answer that question, we must understand the ways in which information is dispersed within modern organizations and external social networks.  More to the point, we must understand the way in which technology affects the ‘recordness’ of information with human social networks.  With a better understanding of how records are formed and used within human social networks, we will be better positioned to capture and preserve not only information, but also contextual data about how that information was dispersed, used, and reused.

‘Recordness’ and the Need for a Personal Archives Service

In the Facebook era, record capture and preservation systems must take three factors into account: 1) the perceived lack of value accorded to preserving digital communications; 2) the communication and information management practices used by individuals and; 3) the specific ways in which contextual data transforms information into evidence, within human social networks.

The perceived lack of value accorded to preserving digital communications

In 1899, the American sociologist Thorstein Veblen wrote that “the cheap, and therefore indecorous, articles of daily consumption in modern industrial communities are commonly machine products.”   Such articles are much used but little valued, at least in a monetary sense.  For that reason, they are easily lost or discarded. Any American who has eaten at a Fourth of July picnic knows how easy it is to throw plastic utensils and plates into the trash, in spite of their utility.

In post-industrial societies, digital communications comprise the cheap and indecorous articles of daily consumption.  We are familiar with the forms that these materials take: Email messages, blog posts, Facebook updates, tweets, online videos.  Each can be inexpensively produced with the help of an electronic device.  Each is certainly less decorous than the format for communication that it replaced, such as handwritten letters, finely detailed diaries, or professionally produced films.

Given this fact, one may expect that the biggest challenge in digital preservation might lie in convincing people that their personal digital communications are important enough to preserve.  But this strikes me as patently wrong.  In the abstract, many people value digital materials highly.  Instead, the broader information ecology makes it difficult for both organizations and individuals to identify, capture, and preserve the records that have the most long-term archival value, unless extraordinary actions are taken.

For example college and departmental subject files, which in past years documented faculty teaching, research, and service at the University of Illinois, have largely disappeared.  Replicating them in a digital environment would require extensive changes to work habits and procedures, as well as the implementation of an expensive software product.  Similarly, staff in the office of our chief administrative officer (the Chancellor) are worried that email messages documenting critical policy decisions never make their way into an expensive electronic records management application, since administrators don’t like to change their work habits.  Few, if any, of our most distinguished faculty members keep systematic correspondence files, outside of their email accounts, probably for the same reason.

On the other hand, our campus has put many resources into building systems that preserve other types of information.  For example, we maintain expensive business systems that store transactional data relating to financial affairs, personnel management, and student academic records.  Most, if not all of the data managed by these systems lack long-term administrative and archival value.   The maintenance of these systems is certainly necessary for the daily operation of the University, and the data within them is used to produce aggregated management information, which certainly of long-term archival value.  Similarly, the University of Illinois and many other libraries operate well-designed and successful applications that preserve formal research outputs, such as published scholarly papers.  A rich literature has arisen around the development and implementation of these repositories.

By comparison, relatively little has been written concerning the theory and practice of preserving personal or professional correspondence or informal communications.  Email preservation is not included in most large-scale digital preservation projects.  As a result, we, as archivists, can easily be left in the position of sweeping up the few crumbs of information from a faculty member or administrator’s computer or closets, much like I did when cleaning out the office of a distinguished chemist.

If we do want a better future for historical research, we must develop a method for archivists and records creators to work together, so that communications in email systems and social networking technologies can be kept alive long enough to be accessioned to an archives.

Personal information communication and management practices

An email I received while on sabbatical illustrates how recordkeeping functions have devolved to individual initiative.  The author, who wished to remain anonymous, nearly lost his entire email record when his former employer abruptly terminated access.  The prominent American journalist James Fallows relates a story concerning his wife’s Gmail account, which was hacked, and its entire contents deleted.  Ten years of messages were salvaged only through Fallows’ personal connections.  Stories like these illustrate that efforts to preserve personal records in the Facebook era must focus on helping people generate archives that live outside of their communication utilities.

In two articles entitled “Rethinking Personal Digital Archiving,” Microsoft Principal Researcher Cathy Marshall notes that most people exhibit an information archiving instinct.  At an extreme, this tendency can exhibit itself as compulsive information hoarding either of hard copy or digital materials.  Many people exhibit much more modest self-archiving behaviors, such as the six methods that Marshall describes in the first part of her article.  Technologies facilitate each of these strategies, as well as others that people have developed since Marshall conducted her research.

However, the technologies are often used ineffectively, if the goal of ‘archiving’ is the long-term preservation of evidence of human activity.  For example, one faculty member whom I know contracts with Dropbox to backup her computer desktop, which includes photographs, reports, and working documents.  The information on this computer may have some long-term value, particularly if the photographs or documents contain embedded metadata (which is doubtful).  The information on her computer is lifeless until it is communicated or shared with someone else.  Communication and sharing take place via her three email accounts, her blog, and her Facebook page.  Unless the email, blog posts, and status updates from these sources are saved in a fixed format, along with their metadata, the full impact of her work will be lost.  Her career and influence will be less well understood than it might be, because the evidential value of these records will have been lost.

Why is saving these records important?  Academic libraries in the United States are currently placing a great deal of emphasis on the development of data curation and data management services, often with the aim of encouraging faculty to preserve their research data in an institutional or disciplinary repository.  Several national programs have been launched, aiming to help librarians become more data centric.  My own institution launched a “Year of Data Stewardship.”  Internationally, a rich literature is developing around this issue.  While these efforts are certainly important, it is easy for archivists to see them as a threat.  Some archivists privately argue that librarians are trespassing on the archival domain.  Instead, those practicing data curation should be regarded as valued partners, to whom we can to articulate the archival emphasis on evidence.  The field is open to us.  We can develop complementary tools and services, which will preserve the evidence that makes research data useful and interpretable.

Recent archival literature articulates a conceptual framework for personal digital archives.  This framework complements the library community’s emphasis on data curation.  For example, the work of Richard Cox, Kathy Marshall, Cal Lee, and others offers a theoretical basis for preserving the evidence found in dispersed personal digital archives.  Similarly, some scholars and members of the popular media have argued that individuals should establish digital legacy plans or even provide instructions regarding the disposition of their digital assets within their estate plans.

However useful this literature may be, what people need most are practical tools and services.  By designing and implementing such tools, archivists can work to build trusted relationships with people.  If we help people generate true archives from their currently dispersed digital communications, they (or their heirs) will have the ability and motivation to deposit those archives in one of our repositories.

How contextual data transforms information into evidence

The tools and services that I have in mind must capture and preserve not merely information, but evidence.  John Seely Brown and Paul Duguid open their book The Social Life of Information with a point that may seem trite: ““[I]nformation and individuals are always part of rich social networks.” However, the implications of this thought are profound for archival practice.  Elaborating their conception of information’s social life, they argue that “documents . . . help structure society, enabling social groups to form, develop, and maintain a shared sense of identity.” Brown and Duguid note that information can easily clot within formal organizations or records systems, so that critical information is not communicated to or known by those who most need it. (I recently experienced this phenomenon when a key decision-maker never heard about a report the University Archives developed.  As a result, he spent over three months essentially duplicating previously completed work.)

On the other hand, in the Facebook era information quickly leaks from within organizations to cross-cutting networks of affinity or practice.  This is particularly true within an academic setting, but is also true in business and government, since the influential members of an organization typically work within a complex, interwoven professional community/social network.  For example, a professor with whom I am currently working has exercised a leadership role on numerous faculty committees that exist within our organization, but she also contributes to professional associations, maintains a blog, and helps mentor current and former students via Facebook.   For each of the social networks within which she is engaged she uses particular tools and services, dispersing evidence across multiple systems

As a profession, we must develop tools and services to capture and preserve something more than information or data. We must find a way to aggregate information held in dispersed systems, in a way that enhances its value as evidence of people’s activities.

The preservation of the evidential value has been one of the archival profession’s defining values and it bears the need for constant reemphasis and renewal.  At this time, it must be refreshed to address the rise of social networking technologies.

As Brown and Duguid note, the media that hold information do not merely contain, carry, or convey that information, they also structure its use within communities.  For example, while paper documents are relatively mobile they are also largely immutable.  The 95 Theses of the Augustinian Monk, Martin Luther, were dispersed through Europe on foot, without much modification, in a fixed format and at a relatively slow pace.  On the other hand, electronic communications are both instantly mobile and highly mutable, particularly when they are shared via email, websites, or, especially, social networking technologies.  A blog post can be changed at a moment’s notice, and many versions of it may be distributed or replicated with a single keystroke or computer command.

In order to provide useful research materials to future generations, we must find a way to fix these mobile and mutable records into place and in a controlled fashion.  This should be done in a manner that preserves enough metadata to make the communications understandable within their original social context. Furthermore, the systems should also include communications sent via other systems and networks, so that no one form of communication is a privileged status.

I propose that this objective can be accomplished if we develop an extensible self-archiving application.  This tool will allow people to aggregate their personal digital communications into one replicated storage location in a fixed, preservation-ready format, that they can control.  I have given this service the provisional name of myKive (“My Archive”).

The Proposed myKive Service

Traditionally the ‘fixing’ function has taken place when the papers of individuals are deposited in an archives, likely after sitting in storage for a long period of time.  It is a method with which archivists are deeply familiar. Whereas every archivist can easily remove records from a closet, relatively few can easily remove them from an email account or a blog.

The proposed myKive service will be an open-source software application that makes it easy for people to work with an archivist, so that they can save their digital communications and other records in a trusted location.  The service will provide individuals or their heirs the ability to deed their myKive to a public archives or manuscript repository later in life.

Why a New Service?

It is tempting to think that people already have access to good tools that allow them to backup and preserve their files.  This is simply not the case.  Many users write files to hard drives or other locations, but Cathy Marshall’s research shows that they rarely use the software consistently or correctly. One faculty member with whom I am working complains that Apple’s ‘Time Machine’ overflowed the backup disk.  Others who use them do not know if they are capturing email, or only desktop documents.  In any case, backup programs do not create archives, because materials deleted from the source disk may also be deleted from the backup device, unless the user is technically savvy enough to prevent that misfortune.

Commercial vendors also offer backup services.  Carbonite, Mozy, and Crashplan incrementally mirror defined files to an external disk or off-site server.  Dropbox can also function as a backup service.  While superficially attractive, these services may leave the user extremely vulnerable to data loss or corruption.  The end user license agreements (EULAs) for such services typically provide no protection to the user, and the services do not provide the types of digital preservation or migration services that the academic community is ideally placed to provide.  Similar problems afflict emergent services, such Backupify and Nuffly, that individuals can use to backup data in cloud-based email or social media.

On the other hand, one might argue that preservation of social media or email is not a problem since the services themselves function like an archives.  For example, Facebook’s new Timeline feature provides users the ability to present a time-ordered stream of documents, photographs, comments and other materials, what Eric Freeman and David Gelertner called a Lifestream. Superficially, the features provided by Facebook and Google make their services look like an archives, but as Jason Scott of Archiveteam notes “Google is a library or archive like a supermarket is a food museum.” While the services they provide are valuable, many of the current social media application, such as Facebook, are predicated upon a business model that grants the service provider the ability to exploit personal data as an economic commodity. In its EULA, Facebook has no positive obligation to preserve data.  In spite of its size, the entire service is as susceptible to business failure as any other company.  Facebook does, thankfully, provide a method for users to download all of their data, and a current version of the EULA gives people ownership of their records, but few users are likely to know what to do with the data once it has been downloaded.

Finally, most people use multiple communication or social media services.  By bringing personally created content together in one location and fixing it into defined formats, we can preserve a fuller, more contextually rich picture of a person’s life and influence.

 Functional, Technical, and Development Model

In its initial stages, the myKive project seeks to develop an open source software package that University of Illinois faculty members and students can use to collect email messages, social media, blog postings, reports, desktop files, and other fugitive materials. These records will be saved to a redundant server in an encrypted, standardized, and preservation-ready format.  Regular integrity checks will be run to ensure materials are maintained in a trustworthy manner.  Content will be stored with sufficient technical and structural metadata to permit its long-term preservation.

In other words, the myKive project does not seek to replace a person’s existing applications or to affect their daily behavior, but simply to provide a method by which they can aggregate and control content that is currently dispersed across multiple services.  Materials in a person’s myKive will remain wholly under their control, subject to a strict privacy policy, but stored on a central server, which the contents replicated to an offsite location.  Users will be provided visualization or other tools to make their records useful.  People will be provided the opportunity to donate materials to an established archives or manuscript repository, based on mutual agreement and negotiation, at any time they or their heirs wish.  Once materials are donated to a public institution, they will be managed under an access agreement outlining the terms under which archival users can access the files.

MyKive will initially be developed as a pilot service at the University of Illinois.  As part of the myKive pilot, the University of Illinois will develop three pieces of software and will wrap them into a web-based dashboard application. The service will be built upon an extensible core and application programming interface (API), so that people can gather information from other services, as new modules are added.  Initially, the dashboard will provide access to three tools:

• A social media archiving tool customized for use by libraries and archives to allow the preservation of account data for multiple users.  I propose that this work be undertaken by incorporating the core code from the ThinkUp project into the project.  It will include ways to harvest and aggregate tweets and Facebook content.
• An email archiving tool , which will use the SMTP protocol to transfer all sent and/or received mails (or, optionally, a filtered set) to a designated archival store.  The open-source ‘Muse’ software is suggested as a candidate technology for incorporation into myKive, since it uses an open storage format that facilitates data reuse and transformation , including visualizing, graphing, searching, and browsing.
• A desktop archiving tool, which will mirror files from a local computer, using standard encryption techniques and secure socket layer technologies.

By focusing on these three critical areas (social media, email, and desktop files), the project seeks to target format types that are most widely used.  Since the project will use or extend existing open-source projects and tools, it will itself be made freely available via open-source code repositories.  This will encourage a collaborative development model.  The community can construct extensions and plug-ins so that people can harvest records from other services.  For example, one or more of the existing technologies might be added to the myKive software core, so that additional record types can be preserved:

• Blogs, using backup tools such as ArchivePress and WordPressDatabase Backup;
• Photographs, using capture tools such as parallel-flickr;
• Web pages, using harvesting tools such as wget, warc, and NutchWax; and
• Personal reference/citation libraries, using tools such as Zotero’s application programming interface.

The proposed technical and governance model will be described in more detail at the upcoming UNESCO Memory of the World Conference.  For now, it is important to note several factors regarding the technical platform:

1. The base technology will be built using on the very common LAMP (Linux, Apache, MySQL, PHP) platform, which will ensure the availability of a large community of open-source developers and code contributors.
2. It will include a service-oriented architecture and the use of microservices, which will allow the processing load to be spread among several servers, if necessary.
3. It will include internationalization features (for example, the capability for multi-language support), allowing for world-wide adoption
4. The tool will include visualization tools and other methods that people can use to search, mine, copy, and repurpose records, without affecting the authenticity of the original files.  By including such tools, the service will provide an immediately tangible benefit.
5. If the pilot is successful, we propose to contract for storage with the non-profit DuraCloud organization, which has expressed provisional interest in serving as a project advisor and partner.  This will provide a trusted level of digital preservation.

Conclusion

In 1965, the founding archivist of the University of Illinois Archives, Maynard Brichford, wrote that: “Wherever the archivist may be located organizationally, he should be out of his office two-thirds of the time.  While processing must be done in the Archives, the archivist should define and standardize processing procedures so that he may spend his time in locating the historical documentation relating to the activities of the university’s staff and students. Effective appraisal must be done in offices, storerooms, stockrooms, and basements.

This advice is no less valuable today than it was in 1965.

The myKive service seeks, ultimately, to provide the archival profession the ability to spend a good portion of our time ‘outside the office’ and to help people in the process.  We will still rescue materials from storerooms, closets, and basements, but we will also rescue materials from Facebook profiles, email listservs, and blog commenting systems. We can help people save these materials in a preservation-ready and accessible format, where they will be more useful to them today.  And in the future, we can help them donate those materials to an archives or manuscript repository—at a time of their choosing and our choosing.

In its initial stages, the University of Illinois Archives seeks simply to demonstrate that the myKive application can be built and that it has utility to us and to our project partners.  But if fully realized, the project I have outlined could open new possibilities to many other archives.

If this vision excites you, I invite you to support the project and to lend your own perspective and expertise as it develops over the next year, as an initial project partner.

Important Notice: Because this post has been revised and  submitted for publication in the Proceedings of the 17th Brazilian Congress on Archival Science and also in Archivo & Administração, the journal of the Assocaição dos Arquivistas Brasileiros (to be published in Portuguese), it is not provided under the creative commons license that applies to the majority of items on this site.

This post is © Copyright 2012, Christopher J. Prom.  All rights reserved.  This draft is made available for private study and comment in anticipation of possible future publication. You may print one copy for personal use, but do not redistribute or quote extensively without written permission of the author.

Today that service was announced, and it is a great day for practical approaches to electronic records because all kinds of useful tools and services can and will be developed from the UDFR knowledge base.

Author: Carol Kussmann, May 30, 2012

Checksums are one method available to assist with preservation of digital files.  Calculating and re-calculating checksum values on files over time allows archivists to monitor the health of their digital files.  Checksum values should remain the same over time, if they do not; the file has been modified or corrupted in some way on the bit level (the ones and zeros a computer uses to read a file).

Checksums are often calculated on files before they are transferred to another location.  This is because after the files are transferred hash values can be recalculated and compared to the original values to determine if the files that were transferred are the same as what was received.  This is a fundamental requirement of authentic documents – that the files have not been changed.

Working with government records, I was looking for a program that would suit my needs for creating checksums on documents before transferring to an offsite repository.  I was looking for a program that could:

• Create reports of the calculated checksum values that could be sent along with the files to the offsite repository and verified after the file transfer.
• Have the flexibility of both calculating and verifying hashes.
• If possible have the option of selecting hash methods or using multiple hash methods.

In the end I reviewed five programs: FastSum, ExactFile, HashMyFiles, the File Checksum Integrity Verifier Utility (for Windows), and Checksums for Windows.  General information about each program can be found below, for more detailed information on each program review their individual reports found here.

FastSum

FastSum can both calculate and verify checksum values using a wizard that walks you through the process or by using the application itself.  Checksums can be run on individual files, the contents of single folders, and the content of nested folders.  Checksums use the MD5 algorithm only.

The resulting checksum values can be saved in a Checksum List.  The Checksum List includes at a minimum the checksum value and the file name; other properties can be added to the report by editing the settings.  Checksum lists can be created and saved as one per file, one per folder, or one per root (top level) folder.  This adds flexibility to the reuse of the checksum values.

ExactFile

ExactFile will let you calculate checksums on individual files as well as on a group of files.  ExactFile can calculate checksums with many different algorithms including those in the MD, SHA, RIPEMD and TIGER families.   Checksum values can be calculated with multiple methods at one time on single files, while one method must be selected when creating a digest (checksums on multiple files).

If you calculate checksums on a set of nested folders, the resulting checksums will be saved in a file in the top level folder, rather than with the individual files.  This file is saved with a file extension for the method being used.

This program can also be used to verify checksums.  Any file with a hash method extension (.md5, .sha1) can be opened with this program and can be used to verify the checksums.  (This program opened the Checksum Lists created by FastSum and simply verified the checksums.)  To verify checksums, open the checksum file, point to the location of the associated files if necessary, and click run.  Reports will identify any problems.

Another feature of this program is that it allows you to create a TestFile Applet.  This Applet was created for use when burning files to CD, but also works well on folders on servers.  A digest is created that lists the checksums for multiple files, but it also creates at TestFile.exe file that can be run from the CD or folder in which it is in.  This allows users without the ExactFile program to make sure the files have not changed.

There is also a command line tool available for this program.

HashMyFiles

HashMyFiles is a program that supports multiple checksum methods including MD5, CRC32, SHA-1, SHA-256, SHA-512, and SHA-384.  Hash values are immediately calculated upon file selection.

Files on which checksums are to be run can be added in a variety of ways, however there is only one way to add content that includes nested folders.

Reports include the file name, hash values of the chosen methods, the full path of the file, modified time of the file, created time of the file, file size, file version, product version, identical, extension, and file attribute.  Reports however are not automatically saved.  To save the information, files must be selected.  This allows the flexibility of saving all, none, or part of the information in the report.

Reports can be saved as a text file (.txt), a HTML file (.htm, .html), an XML file (.xml), or a comma delimited file (.csv).  Unlike with some other programs that automatically save reports in the same location as the hashed files, you need to choose where to store the reports.

This program has additional features that color codes identical hash values in a report, allows you to customize the report columns, and displays individual file properties in a new window if desired.

This program can also be run from the command line on files and folder as well as from within a Windows Explorer menu on individual files.

File Checksum Integrity Verifier Utility

This is a command line utility that supports MD5 and SHA1 checksum algorithms.  MD5 is the default method.

Using the command line in conjunction with a file structure that is part of a larger organization is very difficult.  The command line does not allow spaces in file names.  This is very difficult in a shared Windows environment where the folders are automatically created and named for you such as My Documents and My Pictures.  To use this application, everything had to be moved to my local computer which was not very practical.  Because of this, I did not explore this program any further; however, I was able to create MD5 hash values on files stored on my hard drive.

Checksums for Windows

If you are looking for a program to assist you with verifying checksum values on an individual file basis, CheckSums for Windows provides this simple service for MD5, SHA-1, SHA-256, SHA-384, and SHA-512.  You can calculate and verify checksums for each of these algorithms one file at a time.  This program does not do anything in batch format and does not save the values for you.

These five programs are just a sampling of easily accessible tools available for creating and verifying checksums.   To find other available tools it is best to perform Internet searches to narrow down your results based on your needs and requirements.