Evaluation of the B.C. Family Maintenance Enforcement Program's Pilot Outreach Project

2004-FCY-4E

• [ Previous Page |
• Table of Contents |
• Next Page ]

3.0 DESCRIPTION OF THE EVALUATION FRAMEWORK AND METHODS

3.1 Description of the Evaluation Framework

An Evaluation Framework, developed in the design phase, provided the plan for assessing Pilot Outreach Project outcomes, effectiveness, impact and client and service provider satisfaction.

Table 3 summarizes all aspects of the Evaluation Framework including Project and Evaluation Objectives, Evaluation Questions, Measurement Indicators and Methodologies. The evaluation objectives were developed from existing program documents, interviews with key respondents and input from the Evaluation Subcommittee advising on the project.

Each component of the Evaluation Framework is described below:

• Project Objectives:
  • Define the measurable objectives for all components of the Pilot Outreach Project;
  • The objectives were identified from planning documents and from consultation with key FMEP staff and members of the Evaluation Subcommittee.
• Evaluation Objectives:
  • Define the objectives of the evaluation research arising out of the project objectives.
• Evaluation Questions:
  • Identify the major questions to be addressed in the evaluation research.
• Measurement Indicators:
  • Define the data that indicate whether an evaluation objective has been achieved;
  • Ensure measurement indicators are specific and measurable.
• Methods:
  • Define the specific evaluation strategies that will be used to meet the data requirements of the Measurement Indicators;
  • Strategies are varied and consist of surveys, telephone interviews and file and document reviews.

3.2 Evaluation Framework

Table 3. FMEP Pilot Outreach Project: Evaluation Framework

3.3 Data Collection Methods

Seven data collection methods are referred to in the Evaluation Framework and were used in the implementation of the evaluation. These are described below.

3.3.1 Document and Statistical Review

A number of documents, data collection forms and statistics were reviewed for the evaluation. These included statistics related to the Client Meetings and Payment Conferences, data gathered from data collection forms (Family Justice Centre Referral Forms, Payment Conference Summary and Client Referral Forms) and the FMEP Progress Report. Information from broader program documents (Pilot Outreach Project Charter, program planning documents) was reviewed in the design phase and was also incorporated in this report.

3.3.2 Telephone Survey of Family Justice Counsellors

A telephone survey was conducted with Family Justice Counsellors (FJCs) who make referrals to Client Meetings. Twenty-four FJCs (65 percent) were randomly selected from a list of the 34 FJCs currently working at 10 Family Justice Centres. Two Intake Workers were included in the list of respondents. The highest proportion of survey participants were from Vancouver (Robson) and West Fraser (Surrey).

The survey was administered by telephone from September to October 2002 and collected information on:

• The approximate number of payors and recipients referred to Client Meetings by the Family Justice Counsellors;
• Criteria used by the FJCs for screening and referral of payors and recipients;
• Specific client needs that led to a referral;
• The helpfulness and benefits of the Payor Outreach Project to recipients, payors and Family Justice Counsellors;
• Barriers to service experienced by payors;
• Limitations of the Client Meetings and ways these could be addressed;
• Whether, and the degree to which, the Client Meetings and Pilot Outreach Project enhance the comprehensiveness of services delivered by the Family Justice Centres.

3.3.3 Client Meeting Survey

A survey was conducted with clients attending Client Meetings who had been referred by Family Justice Counsellors between September 9 and December 6, 2002. The survey consisted of a brief written questionnaire that was distributed to and completed by clients. To preserve confidentiality, after completing the questionnaire clients placed the form in an envelope which was then sealed before being given to the Outreach Officer. The survey was completed by both payors and recipients and collected information on:

• Case characteristics (e.g. payment regularity);
• Reasons for referrals;
• Information and/or referrals provided by the Pilot Outreach Project;
• Perceived usefulness and applicability of information;
• Value of face to face meeting in comparison with telephone contact;
• Client assessment of meeting quality;
• Client satisfaction.

Fifty-three clients (75 percent) out of a total of 71 clients who attended the meetings during the study completed the survey form.

3.3.4 Payment Conference Outcome Assessment Form

The Payment Conference Outcome Assessment Form was used to collect information on the outcomes of the Payment Conferences. The data was collected from FMEP case management records (primarily the running record), the Payment Conference Summary Form and FMEP Referral Sheets.

The Payment Conference Outcome Assessment Form collected information on:

• Payor demographic and case characteristics (including status of arrears);
• Payment agreements arising from the Payment Conference;
• Other outcomes arising from the Payment Conference including:
  • The provision of new asset or financial information;
    
  • The provision of new employment or self-employment information;
    
  • Information on the payor’s ability/inability to pay;
    
  • Information on the payor’s need to vary the order;
    
  • Status of the Default Hearing;
    
  • Other outcomes (e.g. increased communication and/or cooperation between the FMEP and the payor);
    
  • Type and level of referrals to other resources.

Outcome assessments were completed on 25 cases, representing 23 payors (two payors were involved in two cases). These constituted all the payors who had attended a Payment Conference (in person or by telephone) between August 2001and August 2002.

3.3.5 Payment Conference Telephone Survey

3.3.5.1 Description

A telephone survey of payors who were notified to attend Payment Conferences was carried out in order to assess the impact and effectiveness of the Payment Conference in removing barriers to the payment of arrears. The Payment Conference Telephone Survey addressed the following:

• Payment characteristics and history;
• History of contacts with the FMEP;
• The payor’s assessment of face to face as opposed to telephone contact;
• The payor’s assessment of the utility of the Payment Conference in addressing barriers to payment;
• Barriers to payment;
• Outcomes of Payment Conference.

3.3.5.2 Contact Process for Payor Telephone Survey

All payors whose files were reviewed for the Payment Conference were contacted for the telephone survey. In order to preserve confidentiality the Director, Family Justice Programs Division, sent a contact letter describing the purpose and nature of the research along with more specific information about the research provided by the consultant. The letters were sent by regular mail.

3.3.5.3 Payor Response Rates

Payor response rates for the Telephone Survey were low. Only 35 percent of the payors (8/23) participated in the research. This represented 40 percent (10/25) of the cases. Low response numbers were due to “not in service” numbers (4), payors refusing to participate (4), payors not returning calls in response to a telephone message (3), or payors saying they had not attended the Payment Conference (2). Two respondents denied being or knowing the payor.

The following steps were taken to address the issue of low response:

• FMEP case data was reviewed in order to ascertain whether updated contact information was available;
• A minimum of five calls to each payor contact number was made at different times of the day;
• Varying messages were left on voice mail describing the research and encouraging payor participation.

Despite these attempts, response rates were not improved.

3.3.6 Key Respondent Interviews

Six key respondent interviews were held with key respondents in the research implementation phase to clarify issues related to policies or processes related to the Pilot Outreach Project. Key respondents were from the Pilot Outreach Project, the FMEP and the Family Justice Centres. Information from these interviews built upon more extensive key respondent interviews conducted with 14 key respondents in the design phase.

3.3.7 Other Methodologies

A seventh evaluation method (MHR Recipient Feedback Form) was designed but not implemented due to changes in the structure and delivery of this component of the program (Section 2.6.2).

• [ Previous Page |
• Table of Contents |
• Next Page ]