SAA 2015: Session 204, Measure Up: Assessment Tools and Techniques from the Field

In advance of the 2015 Annual Meeting, we invited SNAP members to contribute summaries of panels, roundtable and section meetings, forums, and pop-up sessions. Summaries represent the opinions of their individual authors; they are not necessarily endorsed by SNAP, members of the SNAP Steering Committee, or SAA.

Guest Author: Michael Barera, Archivist at Texas A&M University-Commerce

This session consisted of a number of lightning sessions on assessing and auditing in an archival context.

The first lightning talk was “SAA-ACRL/RBMS Joint Task Force on the Development of Standard Holdings Counts and Measures”, by Katy Rawdon of Temple University. Rawdon began by noting that the task force is assembled by SAA and ACRL’s RBMS, and that there is also a partner task force that exists, as well as a third joint task force. She then briefly showed a slide demonstrating the task force’s extensive and diverse membership. According to her, the task force is focused on assessment and standardization, developing guidelines such as metrics, definitions, and best practices. Work so far has included 17 meetings (mostly by phone), completing a self-assessment of counting practices and reasons for counting, and a posted call for survey instruments. The final product will consist of guidelines, definitions, a tiered approach (minimum, optimum, and added value), and possibly tools for assisting with counting. Further resources include an open-to-the-public meeting held the next day during the Annual Meeting, as well as the RBMS journal and the task force’s microsite.

The second talk was “Tourism Dollars: Evaluating Your Archives’ Impact” by David Carmicheal, formerly of the Georgia State Archives (GSA) and currently the Archivist of Pennsylvania. His presentation outlined two surveys of the GSA’s users while he was employed there. The first attempted to determine the question “who visits?”; it found that 71% of GSA users were genealogists, and 31% came from out of state to visit the archive. A second survey was given to the out-of-state “tourist” users to determine more about them and their behaviors; it found that they mostly visited the GSA in the spring/summer, most came with one other person, most stayed within 5 miles of the archives, 91% [of the out-of-state users] were genealogists, many continued on to other parts of the state during their visit to Georgia, most spent $100-500 per visit, most stayed in the state for 3-5 days/nights. The survey instrument also asked these out-of-state users for more information, including their ZIP code and how they learned about the GSA. Carmicheal concluded by discussing the various benefits of conducting the surveys, which included simply giving the GSA more data, starting to uncover the economic impact of visitors, the fact that they “encouraged tourism officials to take us seriously”, and that they provided for feedback.

The third lightning talk was “Metrics and Change Management”, by Fynnette Eaton. Eaton began by outlining the goals of change management, which she defined as ensuring that normal work continues, building/maintaining momentum, and dealing with the “human side”. From here, she came to her conclusions about how to measure success by asking key questions; “is the staff producing enough descriptions?”, “are staff speaking more positively about what is happening?”, and “is staff as productive as you thought they’d be after the changes have occurred?”

The fourth talk was “Baseline Audit from a System Perspective using MIT’s Drupal TRAC tool”, by Courtney C. Mumma. Mumma began by defining TRAC as an “auditing tool for assessing repositories”. She then outlined internal TRAC review in Drupal, which she described as a frame for iterative review and internal and external audit, which is available as a free download from the Artefactual website. This tool affords a number of exciting new possibilities, including incremental and cumulative ratings and results, revision tracking, a natural language question field, staff-only action items and notes, and an audit login capability with a notes field for peer/external audit support. She then displayed screenshots of the tool, before concluding by outlining the next steps for the project: accumulating results, defining roles for various stakeholders, and assigning responsibilities.

The fifth lightning talk was “Information Governance Assessment Tools”, by Elizabeth W. Adkins. To begin, Adkins began by listing the “Generally Accepted Recordkeeping Principles” issued by ARMA International: accountability, integrity, protection, compliance, availability, retention, disposition, and transparency. From here, she introduced the concept of “Information Economics Process Assessment”, which is designed to assess 19 information lifecycle processes for legal, RIM, business, privacy, and IT purposes. She concluded by comparing two popular and competing assessment tools: ARMA vs. CGOG:

Issued by professional association Issued by IBM-sponsored forum
Driven primarily from RIM perspective Reflects input from IT, RIM, and legal professionals
$5,000-6,000 for 12-18 month license Free to registered site users
66 questions covering the 8 principles 18 processes to be rated; scale of 1-4
Lots of professional jargon in questions Simple, easy to use
Hosted tool permits benchmarking, raises security concerns Manually generates a risk heat map and process scorecard

The sixth talk was “How to Assess Your Archives Using Digital Preservation Capability Metrics”, by Lori Ashley of Tournesol Consulting. Ashley began by presenting a few of the many digital preservation assessment options: DPCSA and DPCMM, European Ladder, and ISO standards. Shen then homed in on the DPCMM, first by displaying a model of it and then by describing its functions; “15 key process areas (KPAs) from Digital Preservation Policies to Access, and 5 capacity levels from Normal to Optimal”. She then discussed version 2.7 of DPCMM, noting its framework, the fact that it draws on the two ISO standards, it helps to establish priorities and explain to stakeholders, and it is road tested and proved (at US State and Territorial Archives as well as the ICA). At this point, she noted five features common to maturity models: commitment to perform, ability to perform, activities performed, measurement and analysis, and finally verifying. She concluded by discussing building and sustaining capabilities, and the related capacity maturity model, while noting that the whole process is more complicated than it first appears by stating that it is “more like a treadmill with an incline with 12 screens”.

The seventh and final lightning talk was “Information Governance at the IMF: possible solution”, by Salvador Barragan. Barragan began with some background, on the legal framework of the IMF and its legal immunity. From here, he noted that they are looking at evidence and content, and how to govern information. A few years ago, an auditor found a lot of problems at the IMF. This precipitated a study looking at solutions regarding the issue of governance; they decided to look at all content, not just “records” they developed a proof of concept, for this process. They then defined “information governance” very broadly, and noted that for them the main keys would be director information, controlling classification and declassification, andapplying restoration and disposition. The IMF launched a proof of concept in 2015, using a pilot deployment. There has also been data mapping of relative and absolute information, contract, applied policies, and other elements. The result so far has been a success, and the next steps will stem from a green light to more formal and full implementation, and later content.

Early questions from the audience were for Carmicheal, starting with whether he considered looking at archive-influenced tourism, in terms of items leading people to specific places (which he did not) and if he was planning on a similar study in Pennsylvania (“maybe”). Other questions pertained to the acceptance of the digital maturity model, the relationship between information governance and records management (answers ventured by the speakers included “we’re in the embryonic stage”, maybe the “records management” term is being phased out, “whoever strikes first probably strikes best”, and the possibility that maybe records management is one part of the broader “information governance” discipline), and further elaboration on the human element of change management as its impact on people (which Eaton admitted was “very simplistic”).


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s