Measuring ROI: Challenges and Issues




The Data Collection Challenge

This column begins a series describing the challenges, issues and concerns that may arise when measuring the ROI of your programs and projects. This first article focuses on data collection, specifically the use of questionnaires. Data collection has always been a problem, which is sometimes exacerbated when the ROI Methodology is applied because gathering credible data for ROI analysis is imperative. While there are a variety of data collection instruments that can be used, more often than not a low-cost option is pursued. For this reason, questionnaires are the more popular data collection instrument because they can be used to collect a variety of data for minimum cost.

We’ll start the column with a brief review of the ROI Methodology, then move right into the issues involved in collecting data with questionnaires.

ROI Methodology

The ROI Methodology captures six types of data categorized into five levels:

  • Reaction and Planned Action (Level 1)
  • Learning (Level 2)
  • Application and Implementation (Level 3)
  • Business Impact (Level 4)
  • ROI (Level 5)
  • Intangible Benefits

These datasets represent a balanced viewpoint of success of a particular program or project. The data represents qualitative and quantitative data, as well as financial and non-financial data. Data is captured through a series of systematic steps as shown in Figure 1.

Figure 1: ROI Process Model

During the evaluation planning phase, program objectives are set and the evaluation plans are completed. Upon rolling out the program or project, the data collection process begins.

Data Collection Methods

Four types of data are collected during the evaluation process. Two types, reaction (Level 1) and learning (Level 2) data, are collected during project implementation. Two other types of data, application (Level 3) and business impact (Level 4), are collected sometime after the program when new knowledge, skills, behaviors and/or attitudes are applied on a routine basis. Figure 2 shows the different methods of data collection typically used to capture these first four levels of data.

Figure 2: Data Collection During and After Program

Method Level 1 2 3 4
Surveys x x x
Questionnaires x x x x
Observation x x x
Interviews x x x
Focus Groups x x x
Tests/Quizzes x
Demonstrations x
Simulations x
Action planning/Improvement plans x
Performance contracting x x
Performance monitoring x x

Capturing reaction and learning data during project implementation is usually easy because the participants are often captive. They are involved in a learning session, a formal meeting or working sessions around a particular human resources program or project. The challenge comes when collecting application data (Level 3) and business impact data (Level 4). This is often done weeks or months after a project has been implemented.

A fundamental principal in selecting a data collection method is the balance between cost of data collection and accuracy of the results. The more accurate the data collection, the more expensive it will be. Concerns about resource availability often cause individuals conducting studies to default to low-cost methods, such as a questionnaire. In our most recent benchmarking, the questionnaire is the number one method used to capture Levels 3 and 4 data, which are then used in the ROI analysis. This is a dilemma. One of the weakest methods, the questionnaire, is the most used method to evaluate at the ROI level.

Use of Questionnaires

The questionnaire is used because of the resources, cost and convenience. But it has it challenges, including what to ask. After all, if the questionnaire is the principal source for data collection for ROI analysis; it must capture data that represent business impact, as well as set the stage for the ROI calculation. Figure 3 shows a checklist of the topics covered in questionnaires designed to collect Level 3 and Level 4 data.

Figure 3: Follow-Up Questionnaire Content Issues Checklist

Content Issues Optional Content Issues
Progress with objectives Improvements and accomplishments
Action plan implementation Improvement linked with program
Relevance of Program Monetary impact
Perceived value Confiednce level
Use of materials
Knowledge/skill enhancement
Skills Used
Changes with work
Linkage with output measures
Other benefits
Barriers
Enablers
Management support
Other solutions
Recommendations for target audience
Suggestions for Improvement
Other comments


These are topics that reflect on reaction of participants and learning, but more importantly, they ask respondents to reflect on how knowledge, skills and information have been applied and how that application has influenced business measures. Questions are often asked to help in the step to isolate the effects of a program, as well as to convert data to monetary value. These optional questions link the program to results and convert business measures to money, allowing the respondent to do much of the work for the evaluator. Figure 4 shows a series of specific questions targeted toward impact (Level 4) and ROI (Level 5) data.

Figure 4: Key Questions Important to ROI Analysis

Key Questions
1. How did you use the material from this project or program?
2. What influence did it have in your work? Team?
3. What specific measure was influenced? Define it.
4. What is the unit value of the measure? (Profit or Cost)
5. What is the basis of this value?
6. How much did the measure change since the project was implemented?
7. What is the frequency of the measure? Daily, weekly, monthly, etc.
8. What is the total annual value of the improvement?
9. List the other factors that could have caused this total improvement?
10. What percent of the total improvement can be attributed to this project?
11. What is your confidence estimate, expressed as a percent, for the above data? 0 percent = no confidence, 100 percent = certainty


The Challenge with Questionnaires

While questionnaires allow us to ask a variety of questions that can lead us to ROI, the challenge is how to ensure the sources of data respond to the questionnaire. Between the types of questions being asked and the potential length of the questionnaire, given the number of questions asked, success with response is at risk. To ensure credible analysis, a response rate of 60-90 percent is the typical target for most users of the ROI Methodology. So, how do we overcome this challenge?

It’s all in the planning.

Planning a strategy for successful response to questionnaire is critical. This requires more than just offering an incentive for response (although, that is one of the strategies). Multiple techniques are used to ensure a high response rate, as well as good responses. Figure 5 lists a variety of techniques used by organizations to generate an appropriate response rate to questionnaires.

Figure 5: Techniques to Increase Questionnaire Response Rates

Techniques
  • Provide advance communication about the questionnaire.
  • Clearly communicate the reason for the questionnaire.
  • Indicate who will see the results of the questionnaire.
  • Show how the data will be integrated with other data.
  • Keep the questionnaire simple and as brief as possible.
  • Keep questionnaire responses anonymous–or at least confidential.
  • Make it easy to respond; include a self-addressed, stamped envelope/e-mail.
  • Use the local manager to distribute the questionnaires, show support, and encourage response.
  • If appropriate, let the target audience know that they are part of a carefully selected sample.
  • Use one or two follow-up reminders.
  • Have the introduction letter signed by a top executive.
  • Enclose a giveaway item with the questionnaire (pen, money, etc.).
  • Provide an incentive (or chance of incentive) for quick response.
  • Send a summary of results to target audience.
  • Distribute questionnaire to a captive audience.
  • Consider an alternative distribution channel, such as e-mail.
  • Have a third party gather and analyze data.
  • Communicate the time limit for submitting responses.
  • Consider paying for the time it takes to complete the questionnaire.
  • Review the questionnaire at the end of the formal session.
  • Carefully select the survey sample.
  • Allow completion of the survey during work hours.
  • Add emotional appeal.
  • Design questionnaire to attract attention with a professional format.
  • Let participants know what actions will be taken with the data.
  • Provide options to respond (paper, e-mail, Web site).
  • Use a local coordinator to help distribute and collect questionnaires.
  • Frame questions so participants can respond appropriately and make the questions relevant.

Although most items on the list are self-explanatory, the following four are the most powerful techniques:

  1. Provide advance communication about the data collection. Advanced communication lets the participants involved know from the beginning that they must provide data in a questionnaire as part of the project. This shifts responsibility to the participants not only to achieve success and drive business impact, but report the results of their efforts.
  2. Review the questionnaire at the end of the formal session. Although data will not be collected for weeks or months after the particular project is implemented, the questionnaire is reviewed explaining question by question to ensure participants understand the questions and the importance of each one.
  3. Have the introduction letter signed by a top executive. Engaging a top executive in administering the questionnaire often puts a new perspective on the activity of responding.
  4. Provide the summary of results to the participants and show them what improvements are made. This is a two-part process. Provide a summary of what all the participants have said, often the participants want to see what others have provided. Then, a few weeks later, show what changed as a result of their input. This indicates to participants that their input is driving change.

While these four are the most effective, all of the techniques listed above should be considered. A simple planning process is to work with the evaluation team to decide upfront what you will do before the questionnaire is administered, during the evaluation period and after the evaluation period to ensure a positive response. Figure 6 presents an example of a simple plan to ensure positive, credible responses.

Figure 6: Plan for Ensuring Positive Response Rates

Program: Leadership Challenge
Duration: Five Days
Number of Participants: 150
Response Rate Target: 75 percent
Questionnaire to be Administered: Six months after program (give participant three weeks to respond)
Prior to Administering Questionnaire
  • Explain the importance of the evaluation process in the pre-work
  • Provide sample questionnaire in pre-work
  • Review questionnaire on Day 5
  • Tell participants when the questionnaire will be administered
  • Test questionnaire with sample group
During Evaluation Period
  • CEO sends initial letter to participants to remind them of importance of the evaluation and the need for them to provide data
  • Administer on date discussed during the course
  • Provide incentive for immediate response—book on leadership relevant to course content
  • Send reminders at the end of weeks one and two during the three week evaluation period
After Evaluation Period
  • Send thank you for responding, including the final response rate (from CEO)
  • Within two weeks, send summary of results to respondents and other stakeholders
  • Within five weeks, send notice describing how input is being used to improve the program (or other actions taken)


A fundamental standard supporting the ROI Methodology is involved here—no data, no improvement. Essentially, for each non-response, it is assumed no value has been gained by investing in that person’s participation in the program. This conservative rule ensures that we don’t assume benefits even though there may have some. This is one of several standards that help us remove subjectivity from the less than robust methods of data collection. This is also your incentive to go after the data!

For more detail on how to obtain a high level of response, visit us at www.roiinstitute.net and download the article, "Return to Sender" published in Performance Improvement Journal. Also, for an example of using a questionnaire for an ROI study on a leadership development program, contact us at info@roiinstitute.net and ask for the study "International Car Rental (IRC)."

Our next column will address additional challenges of applying the ROI Methodology.