Formative Evaluation Methods 

As a reminder, formative evaluations commonly seek to clarify the presence and quality of resources, needs of the target population, and fidelity and quality of intervention activities. In addition, formative evaluations typically center on determining and assessing the critical features of the implementation of an intervention. The question is then how to identify the most critical components of implementation to be evaluated. Evaluation scholars have identified several critical elements for how an intervention or program is delivered (fidelity of implementation) that are relevant to many unique evaluations (Century, Rudnick & Freeman, 2010; Dane & Schneider, 1998). These components of high-quality interventions can be explored during evaluation data collection for formative evaluations. In general, these features are the critical ingredients for delivering an effective educational intervention. To select the critical components of the implementation to be evaluated, it is useful to consider what you believe to be the features of ideal implementation. The topics provided above are relevant to most educational evaluations in a broad sense, but these can be prioritized or selected based on the specific evaluation context. These topics might be included on observation protocols, and/or be asked of participants or educators on surveys or during focus groups/interviews, etc.

Exposure and Dosage	Attendance and participation levels over time
Delivery Quality and Pedagogy	Method and practice of teaching and instruction
Instructor Knowledge	Knowledge possessed by the educator in the subject matter
Participant Responsiveness	The level of engagement in the services provided
Curriculum Procedures and Adherence	The degree to which the educator follows the curriculum as written

 

Specific to financial education, NEFE has developed some guidelines for what constitutes ideal implementation for financial education. NEFE created a concise list of five key factors for effective financial education that can be used to outline the critical features of program implementation for formative evaluations.

 

1. Well-Trained Educator: Presenter is confident, competent and knowledgeable.
2. Vetted/Evaluated Program Materials: Materials are appropriate for the audience, developed by experts, accurate and up-to-date.
3. Timely Instruction: Instruction is linked to information learners are able to readily utilize.
4. Relevant Subject Matter: Learners can relate to the topics, examples and content.
5. Evidence of Impact (Evaluation): Well-designed evaluations examine program impact.

These best practices could be indicators or components of fidelity of implementation when conducting a formative evaluation. For example, the formative evaluation of a financial education program might assess how relevant the subject matter and materials are to the audience, the level of knowledge and competence possessed by the educator, the timing of instruction compared to real-life decision-making, etc.

Different methods may be employed to evaluate these features of formative evaluations, depending on which components are most relevant to the evaluation questions. For example, if the educator is interested in dosage (or attendance), attendance records can be used to examine trends in participation. If the quality of the delivery of the intervention is of interest, observations might be used to capture trends in service delivery. Depending on the questions of interest, any of the following methodologies may be employed to answer formative evaluation questions. Two common methods for examining formative evaluation priorities are observations and dosage/attendance records.

Observations

Program observations provide an objective assessment of the quality and fidelity of program services. Observations can explore the content of the intervention or service models, including whether services are practical, suitable and relevant to the target audience, as well as the process of service delivery, including coverage of intervention models, adherence to intervention models, quality of services delivered, and responsiveness to or engagement in the services delivered. Typically, observations are conducted by an external person to ensure that no bias is introduced during the observation process. Similar to focus groups, an observation protocol should be designed to allow the observer to record and rate the critical evaluation indicators of interest. Observers can use the protocol to record both numerical ratings and/or written observation notes.

Dosage/attendance records

Program attendance, or dosage, is an important determinant of intended outcomes. For most programs or interventions, it is useful to record and analyze trends in participation to understand whether participants are exposed to enough of the financial education services or lessons to foster the intended outcomes. Attendance records should be gathered throughout the duration of the program and are typically analyzed to explore means or patterns.