Evaluating the I-WIRE project

Overview of the project

The I-WIRE Project was a two-year JISC funded project to establish a quick deposit tool that would enable academic and administrative staff to deposit publication data and full text items easily into Cardiff University's institutional repository, Online Research @ Cardiff (ORCA), with the ultimate aim to make ORCA a central function in the day to day research management processes at Cardiff. The project took a user centric approach to gathering requirements, with individual interviews with stakeholders and group discussions with academic authors. 

The project's output, Manage My Publications, is integrated into Cardiff University's intranet with three choices of deposit route: quick deposit, which asks for the minimum amount of publication data to be entered and is auto-populated as much as possible for ease of deposit; DOI deposit, which used CrossRef to automatically populate the publication data; and Web of Science deposit, which similarly searches the Web of science database for an author's publications. All three were developed from user suggestions, as was the Selected Publications feature.

Alpha testing began in December 2010 with members of Information Services staff, and continued in January 2011 with volunteer testers from academic schools. Beta testing with selected schools began in March, and a gradual rollout over the academic year.

Challenges Faced

The project was late starting due to a lengthy recruitment process which resulted in the scope of the requirements gathering phase being reduced, but with effective management this was overcome.

The project was required initially to work with a pre-existing IBM system. As the project progressed it became apparent that components of this system would not be fully available within the project's timescales, so an alternative technical solution was proposed. This caused some delay to the technical development on the project.

As usual, issues surrounding copyright and publisher's open access policies threatened to prove a barrier to engagement with the project and the repository. Strictly speaking, such concerns, while valid, are outside the scope of the project and user education will be part of the planned advocacy project.

Lessons Learned

The successful delivery of the quick deposit tool, Manage My Publications, can be attributed in part to the fact that the project was engaged with users from the start; having user requirements drive projects like these is essential for success.

A dedicated project team, plus representatives from different academic and administrative communities on the Project Management Group, was essential in the success of the project, ensuring that it did not 'drift' and that issues were kept firmly in scope.

Strong communications, both internally and externally, were also a key part of the project. The team raised awareness of the project and its goals and developments by regular agenda items at high-profile committees and working groups, presentations at library service briefings and at national and international conferences, and by maintaining a website, blog and Twitter account.

Looking forward

The project has fulfilled its objective of delivering a quick deposit tool and received very good feedback from users. There are a number of issues however that need to be considered and addressed in the next few months in order to successfully embed the tool in the University's research management processes:

While deposit has been made easy for users, the mediated approach means that there needs to be a suitable level of staff resources available to ensure the quality of the data entered into ORCA.

An Advocacy and Embedding Project has been approved by the University Library Service and our next steps are to discuss with subject librarians how they can help to publicise ORCA and Manage My Publications within their individual academic schools.

The integration of Manage My Publications with existing and varied school workflows and systems needs to be carefully considered and managed.

The ongoing maintenance and support of the complex systems used in Manage My Publications is essential.

For further detailed information on the outcomes of the I-WIRE project, please follow the link to the external evaluation report: http://www.cardiff.ac.uk/insrv/resources/iwire/Evibase%20Summative%20evaluation%20of%20the%20IWIRE%20project.pdf

This is the final blog entry for the official I-WIRE project but we will be keeping you up to date with the rollout of Manage My Publications and further developments with ORCA. Watch this space!
 

Workflow and Portlet Design

The I-WIRE High Level Design and Technical Specification has just gone to our Project Management Group and IT team leads for review and approval.  Here are the key design decisions and overview of the solution that the project will be developing and testing over the Summer.

Portlet and Integration

The simplified deposit workflow will be surfaced in the MWE (Modern Working Environment) Portal via a portlet developed and hosted using IBM WebSphere.  The portlet will be connected to ORCA using SWORD for deposit, and using the EPrints API and EPrints 3.2 REST interface for other functions.

Portlet Layout

The portlet will contain the following tabs, which are explained further below:

• Quick Deposit
• DOI Deposit
• Web of Science Import
• Search ORCA
• Browse ORCA
• My Publications

The landing-tab (default) will be My Publications, and the portlet will initially carry the title ‘ORCA Lite’ in order to differentiate it from the full ORCA service which provides a different user experience.

All of the above is subject to end user feedback during the development phase, and further concept proofing in one of the MWE environments during the early part of the development phase.

Mediated Deposit

All items deposited using the portlet will be reviewed by the University Library Service Cataloguing Team for completeness, and the bibliographic data supplemented where necessary, before being made available in live ORCA.  The publisher’s copyright policy will also be checked to ensure compliance.  This mediated approach to deposit is essential to ensure consistency and accuracy of data, particularly important as future projects are likely to integrate ORCA with other research related processes and systems, in particular for the REF.

Alternative approaches to the review process (e.g., review by exception) can be explored as a separate project if the volume of new deposits becomes too big to manage on a one-by-one basis.

Quick Deposit

The Quick Deposit tab will capture from the user the minimum bibliographic data required for the Cataloguing Team to supplement the data, whilst keeping effort to a minimum for the user.  If a user decides that the minimal bibliographic data is not enough for a particular deposit and they wish to populate the full bibliographic data themselves (abstract, keywords, etc.), they will be able to select an Advanced Deposit link to ‘full’ ORCA that will take them to the extended deposit screens that are currently in existence and will continue to be available in parallel to the MWE Portal-based deposit route.  Single Sign On will be implemented to avoid the user having to log-in to ORCA if they chose the Advanced Deposit from the portlet.

The IdMan unique person identifier will be added to each Cardiff author automatically during the deposit process.

DOI Deposit

The DOI Deposit tab will allow the user to enter a DOI and retrieve the associated bibliographic data from the CrossRef service.  The search result will be displayed to the user for checking before submitting to ORCA, and a full text file can be added by the user at this point if desired.

The Cardiff authors associated with items returned by the search will have to be identified and their email addresses added by the Cataloguing Team during the review process. The addition of email addresses will also trigger the addition of IdMan identifiers via a planned overnight automated process.

The project team will explore if duplicate items can be identified at this point by a lookup of the DOI in ORCA at the same time as retrieving the data from CrossRef, in which case the user would be prevented from depositing the duplicate item.

Web of Science Import

The Web of Science Import tab will allow the user to search Web of Science using an author name and optional date range.  The user can identify and select individual publications from the list presented, and a full text file can be added for each individual publication.  This service could help a user to retrospectively populate ORCA with their entire publication lists, where their discipline is covered by Web of Science.

As with the DOI Deposit, the Cardiff authors associated with items returned by the search will have to be identified and their email addresses added during the review process. The addition of email addresses will also trigger the addition of IdMan identifiers via the planned overnight automated process. 

Organisational Data

The School(s) associated with a particular item will be selected by the user during deposit.  It is not desirable to automatically populate this as the current affiliation for an author may not have been valid at the time the publication was written.

Research Centres and Research Groups will not be included in ORCA by the I-WIRE project due to the huge variations between them and the potential for inconsistent data.  This will be kept as a candidate for a potential future project.

Embargos

Due to the additional fields that would be needed to capture embargo type and date, a user that wishes to specify an embargo for a publication’s full text item, and optionally its bibliographic data, will need to select the Advanced Deposit link to ‘full’ ORCA.  Embargo functionality will function as it does for the current ORCA, i.e., the Cataloguing Team will decide during the review process what steps are to be taken.

My Publications

The My Publications tab will present the logged in user with a list of their publications.  Long lists will be navigated by either a scroll-bar within the portlet, or via multiple navigable pages containing a sub-sets of the full list.  The preferred approach will be explored during user feedback in the development phase and could be steered by any constraints with the Portal technology.

The user will have the option of exporting their publication list to a file.  Within the I-WIRE project, this will export the entire list.  The ability to select a sub-set of publications for export is seen as complex due to the required caching of a user’s selection across multiple pages (if pages are used), and therefore is being kept as a candidate for a potential future project.

The user will also have the option of selecting their top publications on an individual publication basis.  A marker will be available in the standard data feed and may be used by Schools to identify selected publications for profile web pages.  Each selection - either checking or unchecking an item - will update the ORCA database in real-time.  The portlet will not implement any constraints on the number of items selected as each School is likely to have a different requirement, therefore, it is up to the School to decide how to deal with any constraint violations, e.g., ignoring anything after the first 6 selected publications.  This function will require an extension to the ORCA database to introduce an attribute that allows marking of a publication on an author basis to cover scenarios where there is more than one Cardiff author associated with one publication.

Browse

The Browse tab will offer browse by author name, item type, school and year.  Item type and school will be further grouped by year due to the volume of items associated with them.  A scroll-bar or hyperlinked grouping will be implemented to aid navigation.  An icon will be included in the publication lists to easily indicate the presence of an associated full text file.

Search

The Search tab will offer a keyword or phrase search.  The tab will include a link to full ORCA for the Advanced Search, offering many more fields that help limit and focus the search results.  However, the experience of by the team has shown that the Simple Search will usually return the desired results.

Reuse of Bibliographic Data

The primary method of retrieving bibliographic data from ORCA for reuse will be calling the data feed by constructing an HTTP GET URL and calling it from either a web browser (manually), or from an application on a system (programmatically).  The result of the call will be bibliographic data presented in the JSON (JavaScript Object Notation) format, and a data dictionary document will be provided to help the Schools understand and use the data feed in their systems.  It will be possible for the Schools to convert the JSON feed to a data format of their choice for processing in their own systems and databases.

The HTTP GET URL can include filters to limit the results by item type, school, author, email address and year.

The IdMan unique person identifier will be included in the data feed.  The data feed will also include the ‘selected’ publications marker introduced by the project to help Schools indicate top publications on author’s profiles in their web pages.

A JavaScript version of the feed that retrieves a publications list for a specified individual author’s email address will be made available to authors for inclusion in their own personal web pages.

Importing Data

Importing publication lists will continue to be supported by the current mediated process that takes publications lists from the Schools and validates the data before loading it into ORCA.  Apart from the Web of Science tab, bulk import will not be surfaced in the portlet.

Deposit to Multiple Repositories

The ability to submit an item to other repositories at the same time as ORCA is being explored by the project team.  While PubMed Central would be the most obvious choice, it does not offer a SWORD client.  The benefit of a SWORD client has been notified to PubMed Central, but in the meantime a deposit to the arXiv subject repository will be explored by way of a drop down list in the Quick Deposit tab that allows the optional selection of arXiv.

Integration with Research Publication Management Systems

The User Needs Analysis phase of the project identified a number of Schools that have implemented - or are in the process of implementing - Publication Management systems.  This provides opportunities to integrate ORCA with the Publication Management systems and associated processes that are in use in those Schools.  However, the variety of systems and their different levels of maturity would be too wide for the project to tackle on an individual basis without putting it at severe risk of not delivering a usable solution within the project’s timescales.

Consideration has been given to the opportunity for integration, particularly as this could help the project with one of its key objectives of delivering an enhanced deposit workflow that is firmly embedded in the research management process.  This will be balanced with delivering an enhanced workflow and toolset that meets the needs as far as possible of Schools that do not have any Publication Management system in place.

The I-WIRE Project Management Group agreed that the project team should explore integration with MEDIC’s implementation of the Symplectic Elements Publication Management system, as MEDIC are one of the project’s partners and a member of the Project Management Group.  This is being conducted in parallel to the delivery of the I-WIRE solution.

Managing Duplicates

ORCA management information reports will be used by the Cataloguing Team to identify duplicate items.  At first a manual process, this will - over time - provide a set of rules that could be automated to run directly against the database in the future, and reduce the effort associated with this activity.

Citation Count Data

Access to citation count data is increasingly important for authors, and the portlet would be a convenient place to show this data alongside an author’s publications, without having to log-in to separate databases such as Web of Science.  However, until there is more clarity on the citation data that the REF will be utilising, and the method of retrieving that data, this development will be kept out of scope of the I-WIRE project and kept on the list of potential projects that specifically support the REF.  This will avoid the possibility of any re-work when this area is better understood.

ResearcherID

A number of Thomson Reuters services are available to institutions to assist in the management of publication data and help improve and maintain the quality of the data.  The ResearcherID service is one that is particularly high profile and relevant as it provides a globally unique researcher or author identifier which in turn can be used to retrieve publication lists and citation data from Web of Science, and a host of other Web of Science bibliometric data.

Initial discussions with Thomson Reuters have identified ResearcherID as an opportunity to improve data quality in ORCA as a separate activity to the deposit workflow, and it will therefore be explored outside the I-WIRE project.  There may be an opportunity in the future to include this data in the deposit workflow but it is not yet understood enough to be able to be committed to within the I-WIRE project timescales.

Automated Email Reminders

One of the suggestions made during the latter part of the User Needs Analysis phase was for email reminders to authors that hadn’t updated their publication lists for some time.  More analysis with authors would be needed to understand the timing and frequency of these reminders, as the time span between publications can vary immensely between disciplines and authors, and the function would therefore need to be configurable to meet these different user needs.  Such user configuration would add a level of complexity to the portlet that should be addressed by a future project in order to keep the I-WIRE delivery as risk-free as possible, and to gain feedback from users on what the valuable future features would be.

The following diagram illustrates the scope of the solution being delivered by the I-WIRE project in terms of functions, and the systems and primary data flows supporting those functions.

Design Decisions for Deposit User Story

As the I-WIRE Project is now half way through the Workflow and Toolset Design work package, it's worth taking some time out to summarise some of the design decisions that have been made by the team to date.  These decisions relate to the Deposit User Stories, as they have been the primary focus of our initial design iterations.

Mediated Deposit Process

We will continue to review all deposits before moving them to the live repository.  In the short term, this will require the same level of staff in the University Library Cataloguing team as we currently have.  In the longer term, as volumes grow due to the launch of the enhanced deposit process, and a planned University mandate, the team will need to explore other review methods using management information reports and a 'review by exception' approach.

Due to the importance of the repository in supporting REF, and the potential future integration with other research systems and processes, the continuation of this approach is seen as key in maintaining the quality of the publication data.

Minimal Data Entry

Minimal data entry is a top priority requirement from Academics.  We have agreed a minimal data set that will provide enough information for the Cataloguing team to check and enhance the data at the review stage:

Article Title, Author(s), Journal/Conference/Book Title, Year, School, URL

Plus the Publisher and ISBN for books and book sections, and the Patent Applicant for patents.

Publisher and ISSN will be automatically retrieved from the RoMEO database for journal articles.

Authors email addresses will be auto-completed from a lookup table, and their associated Unique Identifier will be populated in the background.

The portal will include a link to the full ORCA service for Academics that want to populate more than the minimum data themselves.

Searches

To make things even easier for Academics, were are aiming to provide two additional services in the portal:

• DOI Deposit will retrieve the full set of data from the CrossRef database using the DOI supplied by the Academic.
• Web of Science Import will enable an author to search for their publications, check those that are returned and import the basic metadata into the repository.

Both of these methods will also allow a full text file to be added to the deposit.

Research Centres and Groups

We have explored but decided not to pursue the inclusion of Research Centres or Groups in the I-WIRE Project:

"Any ORCA outputs designation other than School (which is the University's constitutional unit of currency) is problematic.  There are huge variations in Research Centres and if we ask people to 'self-declare' attribution to a Research Centre we will get very very messy and inconsistent data.  Alternatively, if we produce a drop down list of research centres for people to choose from it will not be complete and we have no way of validating that they are in fact members of the Centre.  There is a level of complication here that would be hard to unpick if we get it wrong from the start, so we have agreed to have this as a developmental option after the implementation (of I-WIRE)."