Evaluation Report 2002-2011


Download
DELTA Evaluation Report
Document summarizing Discovery Dating Implementation from 2002-2011. May also be read online below!
Evaluation Report FINAL.docx
Microsoft Word Document 1.8 MB

Evaluation findings show that Discovery Dating has promise in positively impacting personal agency and improving healthy relationship norms.

Introduction

This document summarizes evaluation of Discovery Dating, a curriculum utilized for prevention of teen pregnancy, domestic violence, and sexual assault, funded through a collaboration of the Wisconsin Coalition Against Domestic Violence (WCADV) and the Center for Disease Control and Prevention (CDC) Domestic Violence Prevention Enhancement and Leadership Through Alliances (DELTA) grant. Discovery Dating has been utilized in many settings, however this particular evaluation focuses on implementation with middle school students at a Tribal elementary school in Wisconsin. Our information was gathered through pre- and post- surveys about Discovery Dating students’ knowledge, attitudes and behaviors related to personal agency and healthy relationship norms.

 

This data comes from various implementations of the curriculum from 2002-2010 in order to gauge the impact of the class on students and to determine where changes might be needed. This report includes background to the evaluation, a description of evaluation methodology and administration, and a review of the results from the evaluation, including commentary on the data. The reader may find our Logic Model and most recent Survey Tools in the Appendices.

 

Evaluation findings show that Discovery Dating has promise in positively impacting personal agency and improving healthy relationship norms. In one implementation, students who received Discovery Dating experienced drastically fewer pregnancies than their peers who did not receive the curriculum. In later years, survey results indicate that Discovery Dating has decreased students’ acceptance of unequal power and control in relationships. Finally, in a most recent implementation, Students significantly increased their personal agency by 12% (p = .006) as evidenced by increases in results on the survey tool.

Background

Implementation

Discovery Dating is a healthy relationship development tool created by Wise Women Gathering Place (WWGP) Director, Alice Skenandore. It provides a process for exploration of personal values, discernment of character traits of others, and practice of informed decision making. Discovery Dating details steps which enable the student to predict repercussions of certain choices and the benefits of true partnership based upon discernment of facts and information.

 

Discovery Dating has been used with full or partial implementation in myriad settings, including elementary school classrooms, educational sessions for mothers receiving public benefits, youth clubs, high school enrichment offerings, and women’s support groups. This evaluation report focuses on Discovery Dating implementation with middle school students at Oneida Elementary School in Oneida, WI. The elementary school offers a full curriculum combining regular school programs with Oneida culture. Students need not be Oneida to enroll, but must demonstrate one quarter Native American ancestry. Most students are from Oneida, Green Bay, and surrounding communities.

 

At present, students attend a 45-minute Discovery Dating class weekly for one semester. The class is co-taught by members of WWGP staff. Class sizes may vary dependent upon scheduling by the school, but have ranged from 10-25 students.

Evolution

Discovery Dating has been utilized with youth in the Oneida community for nearly 10 years. From 2002-2007, it was integrated as part of a Community Based Abstinence Culture (C-BAC) project offered by Wise Women Gathering Place. During the C-BAC years, 8th grade students participated two times per week for an academic year in programming that enmeshed Discovery Dating components with other curriculums designed to prevent teen pregnancy. They were further engaged in year round events including drama, summer camps, and producing public service announcements. C-BAC was offered simultaneously in a nearby Tribal community, in identical approach.

 

In the 2007-08 school year, Discovery Dating was offered to Oneida 8th grade students on a smaller scale, typically one 45-minute session per week over the course of one semester. From 2008-present, 6th grade students have additionally received the curriculum in this fashion.

 

In 2010, Wise Women Gathering Place received further grant funding from the Native American Center for Excellence (NACE) to improve evaluation capacity. New survey instruments were piloted, and in addition to 6th & 8th grade students, 7th grade Oneida students received the implementation.

 

Wise Women Gathering Place looks forward to implementing and researching results with students and a control group at a Tribal school in Montana in the 2011-12 school year. These students will receive the first-year curriculum at the 7th and 8th grade level.

Program Intended Outcomes

Desired outcomes for Discovery Dating are twofold: 1) increased personal agency, and 2) improved healthy relationship norms. Personal agency is defined by Zimmerman and Cleary (2006) as “one’s capability to originate and direct actions for given purposes.”

 

The ultimate goal of Discovery Dating is a community of individuals and families who possess motivation, knowledge, resources, and skills to peacefully co-inhabit their environment, each having the individual sense of safety, opportunity, fulfillment and belonging, and each fully committed to sustaining these values for every person for all time.

Evaluation Methods

Evaluation and Survey Design: 2002-07, C-BAC

Data for evaluation was collected several ways during the C-BAC years. Pre-, post-, and 5-year follow-up surveys were conducted, and data was gathered from the local health center.

 

Throughout C-BAC, students took a pre- and post- test to measure the impact of Discovery Dating in regard to teen pregnancy prevention. The questionnaire included knowledge and behavior questions that were administered with a confidential code, so that pre- and post- surveys could be linked to each other. At the 5-year follow-up survey, we included an anonymous behavior survey. Evaluators only knew whether or not the student had taken Discovery Dating before, but no other identifying information was collected.

 

Lastly, the county in which one of the C-BAC programs took place is unique as its borders are identical to those of the reservation. Therefore, Tribal Health Clinic data is also data for the county. At the end of the C-BAC implementation, the 157 names of all students who participated in the 8th grade C-BAC program (aged 13-19 at the time) were submitted the community clinic and asked for a birth rate. The clinic additionally offered the birth rate of the remaining 136 youth within the same age range 13-19 in the community who did not receive the C-BAC intervention.

Administration: 2002-07, C-BAC

Surveys administered during the C-BAC implementation were taken using paper and pencil. The five-year follow-up survey was given to 9th-12th grade students who were still attending the same Tribal schools, at a response rate of about 56%, totaling 196 responses. The survey was taken on computers and handheld devices.

Survey Design:2008-10

A pre- and post- test to measure the impact of Discovery Dating on students was developed by identifying an increase in personal agency and improved relationship norms as desired outcomes of the project. Then, quantitative questions were developed in concordance with these desired outcomes.

 

The survey was a one page paper document with 22 questions total. Participants were asked 15 attitudinal questions that are on a scale from strongly agree to strongly disagree, asked 2 behavior questions, and 5 true/false questions. Survey questions measured self-efficacy through students’ own self-assessment and measured improved relationship norms through students’ approval or disapproval of behaviors in relationships.

 

The survey has undergone numerous revisions, but has retained certain items throughout various editions. These similar items will be commented on in this report. Please see Appendix B for the final revision of this tool.

Administration: 2008-10

Pre-tests were administered in the first class session, and as new students were added to the class throughout the semester, they completed a pre-test upon introduction to the class. Post-tests were administered during the last class session. All surveys were administered by Wise Women Gathering Place staff, and were taken with paper and pencil. There was very little attrition; class attendance and size did not fluctuate greatly during the semesters. Most classrooms contained 12-25 students as determined by school administrators.

Survey Design:2010-11

With the NACE funding in 2010, Wise Women Gathering Place was able to receive extensive technical assistance to make the research design more reliable and valid. With the help of a professional evaluator, four tools that have been tested for reliability and validity were piloted as pre- and post- tests during the 2010-11 school year. These tools were: the Behavior Identification form (Vallacher & Wegner) to measure Personal Agency, an Attitudes Towards Violence scale (Anderson, Benjamin, Wood & Bonnacci) to measure attitudes toward corporal punishment of children and intimate partner violence, and the Rosenberg Self-Esteem scale to measure self esteem. The students also completed a Developmental Assets Profile (Search Institute).

 

The Behavior Identification Form was the most unique survey administered during this time period. Its 25 items provide participants an everyday scenario and then ask them to select the response they believe is more like the scenario. For example: Making a list: a) Writing things down, or b) Getting organized. In each item, one response reflects higher personal agency: in the previous example, b) Getting organized, would be the desired response. Students received an overall score reflecting their percentage of desired responses.

 

The Attitudes Towards Violence Scale and Self Esteem scale ask students to rate the level they agree or disagree with twenty statements regarding interpersonal violence and ten statements regarding personal self-esteem, respectively. Lastly, the Developmental Assets Profile’s 58 items are a mixture of behavioral, environmental, and opinion questions. Each student receives a score based on the internal assets of Commitment to learning, Positive values, Social competencies, and Positive identity, the external assets scores of Support, Empowerment, Boundaries & expectations, and Constructive use of time, and an overall asset score. Please see Appendix C for copies of all four 2010-11 questionnaires.

Administration: 2010-11

Surveys were administered via two different modes in the 2010-11 school year. Pre-tests in Fall 2010 were administered with paper and pencil answer sheets. Items were displayed on a screen at the front of the classroom and read aloud by Discovery Dating Facilitators. Developmental asset profiles were administered entirely with paper and pencil.

 

Post-tests for Fall 2010 and pre- and post- tests for the Spring 2011 semester were administered using handheld electronic devices. Students were able to individually record responses to each item. Developmental asset profiles were administered in the computer lab, with each student at a personal computer station. Benefits and drawbacks of these administration methods will be noted later in this report.

Results

Table: Results, 2002-07, C-BAC

 

C-BAC Students

Non-C-BAC Students

Number of Pregnancies

(All Respondents)

5 (N=157)

32 (N=132)

During initial sexual experience, students reported they “did not want to do it, but he or she convinced me”

(Sexually active respondents only)

9.5%

27.3%

Used a condom the last time they had sex

(Sexually active respondents only)

72%

60%

Commentary: 2002-07, C-BAC

During the C-BAC years, we hoped to increase students’ personal agency through their ability to make a commitment to themselves to avoid teen pregnancy, and then carry out this commitment through behaviors that would prevent pregnancy. In a follow up survey, we saw an incredible difference between students who received Discovery Dating as C-BAC students and those who did not receive Discovery Dating.  157 C-BAC students experienced 5 pregnancies in the 5 years following implementation of the curriculum. Of 136 students who did NOT receive C-BAC, there were 32 pregnancies reported in the same timeframe. Here we see through behavior that personal agency was increased. 

 

157 C-BAC students experienced 5 pregnancies in the 5 years following implementation of the curriculum. Of 136 students who did NOT receive C-BAC, there were 32 pregnancies reported in the same timeframe.

 

Refusal skills have been noted as increased due to Discovery Dating. During the C-BAC years, 27.3% of students who did NOT receive Discovery Dating reported that during their initial sexual experience, they ‘did not want to do it, but he or she convinced me.’ Only 9.5% of Discovery Dating C-BAC students reported the same. This data could perhaps support the Discovery Dating outcome of Improved Relationship norms as the Discovery Dating student is better able to listen and/or speak up for themselves in their relationships.

 

Finally, although Discovery Dating curriculum does not include comprehensive sexual education, we noted an increase in the use of condoms among C-BAC students. 72% of sexually active C-BAC students reported using a condom the last time they had sex, while only 60% of non-C-BAC students reported using a condom. These results seem to oppose the common assertion that youth who are taught ‘Abstinence-only” classes will not know how to take precautions against pregnancy and STD’s if and when they do decide to become sexually active. This data parallels the measurements of the Behavior Identification form utilized in 2010-11. The Behavior Identification form is rooted in “action identification theory” which posits that any action can be identified in many ways, ranging from low-level identities that specify how the action is performed to high-level identities that signify why or with what effect the action is performed. High-level agents think about their acts in encompassing terms that incorporate the motives and larger meanings of the action and in turn have higher levels of personal agency. In 2002-07, it appears that C-BAC students were perhaps “high-level agents” that were more able to make wise choices about condom use, even though they were not specifically taught about condoms in the Discovery Dating classroom. Therefore, we believe this data additionally demonstrates an increase in personal agency.

Results 2008-2010

The data found in the charts below originates from school years 2008-09 and 2009-10, when similar, although not identical, survey tools were administered. Slight changes in verbiage from year to year are noted in parentheses. The percentage and number of students marking the desired response are displayed at pre- and post- test. 72 students took the pre-test and 67 students took the post-test.

Table 1: Increased Personal Agency, 2008-10

Question

Pre (N=72)                           Desired Responses

Post (N=67)                                                                   Desired Responses

It is easy for me to stick to my aims and accomplish my goals.*

70%  (51)

67% (45)

If I am in trouble, I can usually think of a satisfactory solution.

78% (56)

73% (49)

Lots of times, I feel afraid for myself at school.

89% (64)

88% (59)

(09-10) Deciding to use alcohol or drugs can cause problems in my life. /(08-09) I think it is a poor choice to use alcohol or drugs.

89% (64)

88% (59)

 

I am sure that I have the skills to say “no” if I don’t want to do something.

94% (68)

90% (60)

Commentary: Increased Personal Agency, 2008-10

The use of personal agency as a measurement criteria is to determine to what extent adolescent students are able to design and execute actions for given purposes (Zimmerman & Cleary, 2006). Therefore, a large focus during the 2009-10 school year was an activity called “Dreams and Goals,” in which students identified a new personal goal to accomplish in the time between Discovery Dating classes. Above, we note a very slight decrease in student’s self-assessment of whether “it is easy to stick to aims and accomplish goals” and also whether they can “think of a satisfactory solution” when they are in trouble. We believe this may be due to a heightened awareness of the difficulty of accomplishing goals through this focus on goal setting each week, and also an awareness of troublesome situations. Prior to Discovery Dating, they may not have recognized unachieved goals or the depth of troubling situations and their responses to both without a process to identify them.

 

It appears that Discovery Dating has decreased students’ acceptance of unequal power and control in relationships.

 

The majority of our students do not report being afraid for themselves at school, and we notice little movement in this fact from the beginning to end of the semester. We also noticed little movement regarding attitudes about alcohol and drugs. Discovery Dating classes may not directly impact safety. This question could perhaps be rephrased to elicit responses regarding students’ ability to seek safe outlets at school or choose positive alternatives during unsafe situations. Regarding alcohol and drugs, Discovery Dating teachers may be more explicit about having class discussions regarding this topic specifically to increase positive responses.

 

Unlike the C-BAC results noted above, in 2008-10, it appears that a returned focus to these refusal skills would benefit the students, as we note a slight decline in their ability to say “no” if they don’t want to do something.

Table 2: Relationship Norms 2008-10

Question

Pre (N=72)

Desired Responses

Post (N=67)

Desired Responses

(09-10) It is natural for one spouse to be in control of the other/(08-09) Controlling another person in a relationship is sometimes okay.*

49% (35)

53% (36)

Just “talking with a person” is all you need to do to learn about him/her**

56% (40)

37% (25)

People need to know how to say “no” to sexual advances.*

73% (53)

71% (48)

Hurtful shoving, pinching or hitting is sometimes okay in a relationship*

73% (53)

80% (54)

Having a child outside of marriage can cause many problems for the child, the child’s parents, or society.

True: 71% (51)

False: 29% (21)

True: 71% (48)

False: 29% (29)

Commentary: Improved Relationship Norms 2008-10

It appears that Discovery Dating has decreased students’ acceptance of unequal power and control in relationships in the 2008-10 data. Although the item was worded differently each year, we note a drop in responses that are accepting of one individual controlling the other in both the marital and non-marital phrasings. We also see a slight change in student’s disapproval of hurtful shoving, pinching or hitting in relationships. This may be enhanced if lessons explicitly defining physical abuse were included to supplement lessons on healthful self-recognition and ability to discern others’ characters.

 

Discovery Dating purports that “just talking with a person” is not a sufficient way to get to know them, but rather that careful consideration of facts and information gathered over a long period of time is the preferred method. Therefore, we were happy to note a decline in student’s accepting attitudes of this item.

 

Finally, we note no change in the final item displayed in Table 2, “Having a child outside of marriage can cause many problems for the child, the child’s parents, or society.” During the C-BAC years, comparison between C-BAC and non-C-BAC students reveals they had relatively similar opinions on this item as well. This question will be reconsidered in the next draft of the survey tool.

Results: 2010-11

The data found in the final charts is from the 2010-11 school year. These brand new, pilot surveys (Behavior Identification, Attitudes Towards Violence, Self Esteem scale, and Developmental Asset Profile) were scored and analyzed based on students’ total scores. 77 students took the pretest and 75 students completed the post test. To date, students’ responses to individual items have not been extensively analyzed. The summary of results can be seen below.

Table 3: Piloted Survey Tools, 2010-11

Survey

Pre (N=77)

Post (N=75)

Behavior Identification (measures Personal Agency)

Average % of Desired Responses: 57.68%

 

Average % of Desired Responses: 65.21%

 

Attitudes Towards Violence

Average % of Desired Responses: 76.11%

Average % of Desired Responses: 73.75%

Self Esteem

Average % of Desired Responses: 71.64%

Average % of Desired Responses: 67.46%

Developmental Asset Profile

Average Asset Score:

37.93

Average Asset Score:

37.03

Commentary: Piloted Survey Tools, 2010-11

Students significantly increased their personal agency by 12%

 

(p = .006) as evidenced by increases in their Behavior Identification Form scores.

 

We were very curious at the outset what kind of results the Behavior Identification tool would give us. We were very excited to notice the greatest change in personal agency over the other three tools! The Independent-Samples T Test procedure was used to compare the means of the pre-test and post-test scores. Independent-Samples T Tests were conducted instead of Paired-Samples T Tests as there were challenges matching the pre- and post-test scores to each individual based on ID numbers. Findings suggest that students significantly increased their personal agency by 12% (p = .006) evidence by increases in their BIF scores. Looking towards future implementations, this tool is one we would like to continue using.

 

We were left unsatisfied with the Attitudes towards Violence scale because of the strong wording of many items. (Refer to Appendix C for a copy of this tool.) Students vocalized strong emotional reactions to the questions while completing pre- and post- tests that may have impacted the honesty of their responses. Additionally, we saw little to no change in students’ responses. We will continue seeking a better tool to measure attitudes about healthy relationship norms in the future.

 

Since it is such a short tool, the Self Esteem scale was integrated into the pre- and post- tests as a measure to explore whether or not self-efficacy is the best theory to examine Discovery Dating. The Self Esteem scale showed little change as well, and upon further analysis of the Self-Efficacy theory, WWGP staff believes that Self-Esteem would not be impacted by Discovery Dating. This tool will not be used in the coming school year.

 

Lastly, the Developmental Asset Profile data was also inconclusive. Many items on the questionnaire are behavior related and deal with issues outside of the Discovery Dating classroom, such as neighborhood safety, performance in school, etc. Therefore, these areas of students’ lives may be outside the scope of the Discovery Dating curriculum. Additionally, hand scoring these surveys was burdensome in staff time, and administering via the Search Institute website proved challenging as well. It is unlikely that this tool will be used in the future.

Conclusions

The C-BAC implementation of Discovery Dating gave us great hope around the curriculum’s effectiveness in preventing teen pregnancy and also increasing personal agency and improved relationship norms. As the curriculum is now implemented in a different fashion than in the C-BAC years, we are eager to discover just what tool best measures the results in a strictly classroom setting.

 

Reviewing the 2008-10 data would suggest that either we were not having as great an impact on personal agency as we had hoped, or, perhaps more likely, that the questions were not effectively measuring the outcomes we hoped they would. We are very excited about the results of the new Behavior Identification Form to measure personal agency. We will test two new tools in the coming school year to further explore personal agency. They are the General Self Efficacy Scale and the Youth Resiliency Scale.

 

Other questions that perhaps were not measuring precisely what they had hoped they would include items on safety outside of school, and items on alcohol and drug use. Discovery Dating may not directly impact safety outside of school, and the curriculum is not explicit on issues of alcohol and drug use without supplements from the instructor. However, survey questions to learn more about how students are able to choose healthy alternatives could shed light on whether or not Discovery Dating indirectly improves students’ safety and lowers their substance use.

 

Data on improved relationship norms seems slightly more promising in 2008-10 than in 2010-11 and after gathering a few more years of data, perhaps from similar survey questions, we will be able to assess our impact with further depth. Regarding refusal skills, we noted a decrease in desired responses 2008-09 compared to C-BAC years, so perhaps a returned focus to these skills would serve students well. We are inspired by an apparent drop in students’ acceptance of power and control in relationships, and are curious if more explicit lessons on physical abuse would enhance these results.

 

It is challenging to assess the last few years holistically as different survey tools were used, so many questions are altered from year to year. Additionally, the intensity of situations depicted in items on the Attitudes Towards Violence Scale elicited strong verbal comments from students as they took the test, which may have skewed the results. It may be necessary to begin anew with a fresh survey tool that measures outcomes as planned, rather than by recycling a more ineffective tool. The General Self Efficacy Scale may afford us an opportunity to do this as it requires the addition of a few survey items context-specific to the area of intervention. Our intent is to pull improved relationship norm questions from previous years’ survey tools. In this way, WWGP plans to include questions on healthy relationship norms in the coming school year’s survey.

 

Survey administration has evolved throughout the years as well. Due to the staggering data from the C-BAC follow-up survey, WWGP would like to conduct another follow-up survey with students from the more recent years in Oneida. Funding and logistics on this project are being researched. Pre- and post- surveys have been administered with paper and pencil, orally, and with handheld devices. In 2010-11, when surveys were administered orally, Discovery Dating facilitators noted an increase in restlessness among participating students, which was very disruptive and may have impacted survey results. Later in the year, surveys were administered with handheld devices, and while students took the surveys in a more subdued manner, many questions were left blank due to the handheld interface. Solutions to these challenges are being investigated, and surveys may be administered on desktop computers in coming years.

 

Regarding overall program changes, many things have been learned in the last few years! In the 2009-10 school year, Wise Women Gathering Place staff noted just how important it is for the same staff member to report to class each time, instead of sending a mixture of staff. This creates consistency necessary for the relationship-building that best facilitates learning the Discovery Dating lessons for students. We are also currently assessing our teacher’s manual, so that we are able to be more faithful from year to year with the material. Lastly, as our initial 6th grade students get older, we will have access to them a second time as 7th and 8th grade students. We look forward to examining data from students who have received material from the course two or three times at different maturity levels.