Summative Evaluation Methods 

Because summative evaluation typically focuses on participant outcomes, it is common to assess changes over time (pre-post tests) or differences between participants and nonparticipants. Surveys, test or assessments, financial indicators, and focus groups or interviews are common data sources for summative evaluations. Information about several of the most common evaluation methodologies is provided below. This is not an exhaustive list. Although these methods are commonly used to examine outcomes, it should be noted that these methods described below also could be used in formative evaluations with different subject matter and questions.

Surveys/Assessments

Surveys and assessments are the most common evaluation method used by financial education educators. In formative evaluations, surveys focus on participant needs, program resources, and their perceptions of program quality. Similar to the questions used for focus groups and interviews, these items can be designed to explore participants’ reactions to financial education programs. In summative evaluations, surveys and assessments can be used to explore participant outcomes by assessing knowledge, skills, attitudes, behavioral intentions, etc. Designing high-quality evaluation survey tools is a professional task and requires experience. The NEFE Financial Education Evaluation Toolkit makes it easy for financial educators to design professional-caliber questions and measurement instruments based on their local program needs, particularly focused on outcomes assessment. Educators are urged to visit toolkit.nefe.org to access the online survey database.

Focus Groups/Interviews

Speaking with program stakeholders/participants in a more formalized manner via focus groups and interviews can provide highly detailed verbal descriptions and insights to answer the evaluation questions. Focus groups typically involve 6-8 people, and are useful for obtaining answers to evaluation questions from many participants at once, which is more time efficient than interviews. However, interviews are more appropriate in situations where sensitive questions may be asked or when participants might be influenced by the presence of other people during the process. It is useful to create a protocol or script to use for each focus group or interview to standardize the process and ensure that you are asking the same questions in the same manner across each instance. The textbox displayed here presents several example questions that could be asked of participants during focus groups or interviews. Typically, these discussions are recorded and transcribed to obtain written records that can be used to identify and organize themes.

Document review

In some circumstances, there are documents or other written data sources that can be used to track how the program is being implemented or to track participant outcomes. These documents may be specific to the evaluation context. For example, written lesson plans could be used to examine how many lessons were covered from a particular financial education curriculum. Document reviews can be particularly useful when there are few resources for data collection, other more intensive data collection strategies are not possible, or the evaluation is taking place after the program already has been implemented. These documents would be examined for themes or coded for numerical analysis.

Writing or Selecting Survey Items

As financial educators commonly use surveys to assess evaluation outcomes, a special section covering the development of survey items and the selection of survey concepts is included.

The accuracy and reliability of data are essential qualities for implementing evaluation recommendations with confidence. In the context of evaluation, validity (or accuracy) and reliability both relate to errors in the data. Some errors are caused by the lack of accuracy in the data collection method or instrument (validity), while other errors are caused by problems with the consistency and stability of the measurement instrument (reliability). Therefore, special attention should be paid to ensure the accuracy and reliability of evaluation data and information. The following factors help ensure the accuracy and reliability of evaluation data, particularly for surveys, focus groups or interviews.

Clarity of Questions

It is common to use self-responding surveys to gather data. Therefore, evaluation questions should be written clearly and concisely to avoid confusion and help the participant answer accurately. Instructions should be clearly stated to help participants complete the survey easily. Generally, closed-ended and open-ended questions are used in evaluation instruments. If the possible response choices are provided (as in a multiple choice question), this is a closed-ended question. If the question is asked so that the respondent must use his or her own words to answer it, this is an open-ended question. When closed-ended questions are used, the educator should be sure that the answer key contains all the possible responses to prevent response errors and/or that an “other” option is included with space for the respondent to fill in his or her own response.

Reading Level of Target Audience

The reading level of questions is an important determinant of the accuracy and reliability of evaluation data. The reading level of the written language used to design the evaluation tool should not exceed the reading level of the target participant group to avoid potential errors in data collection. This is particularly important when working with children and youth, or those who may be learning English (ESL learners).

Sensitive Data and Information

Collecting sensitive information such as age, income and other financial information can be somewhat challenging because generally participants do not like to reveal this information. It is important to get this data in a way that the participant is comfortable in providing it. One approach is to present question-and-answer choices that have ranges instead of exact values of sensitive data. For example, instead of asking, "What is your annual household income?" the question might be "In what range is your annual household income?" and list the possible income categories from which the participant can choose their response.

Selecting Survey Constructs

The survey items written or selected for the evaluation survey tool should align strongly with the outcomes displayed in the logic model. Consistent with the outcomes identified in the logic model section above, below are example survey items to explore these outcomes.

Reactions/Experiences

These survey items focus on gaining participant perspectives about their experiences in the program, including satisfaction, perceived impact, perceived relevance and more. These types of survey items are typically measured on an agreement scale (“Strongly Disagree” to “Strongly Agree”).

Example Items:

1. I would recommend this program to others.
2. The financial education lessons are interesting to me.
3. The skills I am learning in this program are useful in my life.

Knowledge

Knowledge change is the most common impact indicator in any financial education program, and it can be recorded by asking questions related to the learning content of the program. The educator either may use multiple choice questions or true/false questions. The true/false question format (examples below) is suitable for low-literacy audiences or younger populations.

Example Items:

1. It is a good idea to make only the minimum payments on credit cards.
2. When you must pay a bill late, it’s important to call the company before the bill is due.
3. Interest rates and fees are about the same on all credit cards.

Confidence

The confidence to carry out a financial management tasks is a reflection of one’s financial management skills. These items can be measured on an agreement scale (“Strongly Disagree” to “Strongly Agree”) or on a confidence scale (“Not Confident” to “Very Confident”).

Example Items:

1. I am confident I can save money regularly.
2. I am confident I can balance a checkbook every month.
3. I am confident I can reduce my personal debt.

Attitudes

To assess participants’ attitudes, a scale can be developed using value statements related to the financial practices that the program is planning to teach. These types of survey items typically are measured on an agreement scale (“Strongly Disagree” to “Strongly Agree”).

Example Items:

1. Saving money regularly is important to me.
2. Planning my personal budget is a priority.
3. Starting an emergency savings fund is important to me.

Intention to Engage in Positive Financial Behaviors

Survey items in this category ask respondents to report the likelihood of engaging the financial behaviors in the future, if they are not already engaging in these behaviors. The following format is appropriate to record participants' aspirations to adopt desired financial practices. These items can be assessed on a scale from “No” to “Already doing this.”

Example Items:

1. As a result of this program, I plan to set a goal to get out of debt.
2. As a result of this program, I plan to keep track of my spending debt.
3. As a result of this program, I plan to pay bills on time every month debt.

Financial Behavior Changes

Recording participants' actual behavior change is possible only if the financial education program takes place over time. For comparison, data must be collected at least two different times to be able assess participants' actual behavior changes. In a multi-session financial education program, the educator meets the same group of participants more than once. As a result, the educator has an opportunity to record the participants' financial behavior related to the content of the program before and at the end of the program series to document changes in behavior. These items can be assessed on a scale from “I am not doing this” to “I am doing this all the time.”

Example Items:

1. I keep track of my spending.
2. I pay bills on time each month.
3. I find ways to decrease my expenses

Financial and Economic Indicators.

In financial education evaluations, there also are financial indicators that can be used to examine the impact of program participation. These numerical indicators may include accounting balances, ratio of income to expenses, debt, number of credit cards, payment late frees, credit card interest rates, etc.

Example Indicators:

1. How much is your credit card debt?
2. How many credit cards do you have?
3. What is the balance of your total savings?