Assignment Paper: Instructional Design Solution and Evaluation Plan: Career and Program Plan Instructional Workshop at Netsilik School, Taloyoak The Instructional Design Solution and Evaluation Plan: Career and Program Plan Instructional Workshop
Instructional Designer: Deborah Maguire
Client: Netsilik School, Taloyoak
Instructional program to be evaluated: Teacher Instruction workshop on implementation of the Career and Program Plan (CPP).
Information is directed to the stakeholders: principal, Junior/Senior High teacher team, District Education Council, and Kitikmeot Board of Education
One of the many challenges that face schools in the north is student dropout. Despite the creation of a community high school, many Inuit students in Taloyoak still choose to leave school early. Consequentially, the staff at Netsilik School wishes to find ways to improve this situation. Before contacting an instructional designer, a critical look was taken at the Junior/Senior High problem. It became evident to the staff that improvements to the Career and Program Plan implementation could be made which may help to reduce the number of students who choose to dropout.
Research in The Review of Secondary Education in the NWT 1998 (Department of Education, Culture and Employment, 1998), identifies an essential component of senior secondary programs in NWT to be Career Development and Planning (CPP). As well, a research study entitled Northern Lights: A Research Study of Successful High School Students Across Nunavut by DaSilva & Hallett (unpublished) found that successful students are those who "have goals for their lives after graduation and they tend to see graduation from high school as an essential step towards attaining those goals" (p.5). Thus, improvement to the CPP implementation was the focus of consideration in reducing student dropout at Netsilik School.
Earlier, a Needs Assessment Plan and Constraint Analysis confirmed these initial perceptions. The Constraint Analysis determined that the two major negative impacts on the goal of implementing the CPP were teacher competency and time. The strategies considered to most effectively reduce these constraints are retraining teachers in career counselling skills and reorganising schedules to increase the time available for implementation. It is the CPP instructional workshop that is the subject of this evaluation plan.
To review, the Career and Program Plan (CPP) is a continually revised plan mandated for use in Alberta and NWT schools (this includes Nunavut since April 1999). It takes into account requirements for graduation, career in the work world and admission to post secondary institutions (Storey, 1997). It is a plan that is created by the student and parents "with the assistance of the school, career development officers, partners in the community and mentors for the student prior to entrance into Grade 10" (Storey, 1997, p. 3). Teachers play a key role in the success of the program because they facilitate the learning and guide the process. It is because of these responsibilities that to improve the CPP involves improving the abilities of teachers to implement the CPP at Netsilik School. Participating teachers will be involved in a three day workshop in order fulfil goal of improved CPP implementation. Due to the high turnover of teachers at Netsilik School, CPP instructional workshops are purposed for every other year with a short update of resources and information on the off years. The following document explores two areas:
1) an overview of the three day CPP instructional workshop, materials and methods; 2) the plan for evaluating this CPP instructional workshop to determine success.
This discussion requires clarification of some key terms and ideas. The participating teachers in the workshop and evaluation are the JH/SH teacher team of four teachers and one principal. The facilitator is an external person who is skilled in career counselling and knowledgeable about CPP implementation. The CPP instructional workshop is a three-day session for teachers, which is outlined in this document.
To avoid conflict of interest allegations, a statement about the instructional designer should be made. The instructional designer and the author of this document, is also a JH/SH teacher at Netsilik School and will be participating in the CPP instructional workshop. To create a non-bias evaluation plan, I must step away from my role as teacher and take an objective look at CPP implementation. It will be in this objective tone that I will explore the issue of improving the quality of the CPP process for my client, the Kitikmeot Board of Education. One positive aspect of my being a teacher at in the same school that is requesting assistance is that I will feel a clear sense of ownership in the success of the problem solving. Also, I will have the ability to perceive the impact of the workshop and subsequent evaluation on teachers.
When designing an instructional workshop, it is important to seriously consider the objectives of the program and the format of the instruction. Teachers at Netsilik School were involved in thirty-minute problem solving session based on two questions regarding the instructional design of the CPP Teacher Workshop:
What type of instruction program do you think would be most valuable? (i.e. program delivery, materials, topics to be covered, frequency etc)
What do you think is most important in such a program? or: What criteria would you use to measure success of the instructional program?
Teachers were involved in this brainstorming session to foster ownership of the solution. Also, because the school uses a site-based decision making model, it is natural to include teachers in decisions made about their work. This gives opportunity for teachers to be involved in their learning and professional development. The instructional design which follows incorporates the ideas discussed during this thirty-minute problem solving session.
The objectives of the CPP instructional workshop are as follows:
Teachers will support students in their career choices.
Teachers will appreciate their own and others knowledge with career program planning.
Teachers will find value in learning skills in CPP to help students.
Teachers will be willing to work with students to achieve awareness about career choices.
Teachers will recognise the importance of their role in the CPP implementation.
Teachers will appreciate the importance of the CPP for students.
Teachers will access career resource information using the Internet.
Teachers will complete application forms for educational institutions, financial institutions and alternative programs.
Teachers will write a CPP year plan or action plan outlining activities and procedures for the implementation of the CPP in the classroom.
Teachers will gain communication skills to deliver CPP in a variety of settings (one-on-one, small groups, whole classes, with parents, etc)
Teachers will use a variety of career activities (role-play, games, modelling etc)
Teachers will review the necessary qualifications for graduation of high school.
Teachers will investigate the various options available for students (trade school, college, university, alternative programs) and recognise differences in qualifications for programs.
Teachers will analyse possible CPP activities.
Teachers will learn where and how to access information.
Given these objectives, activities for the CPP instructional workshop will be planned. The tentative agenda for the three days is shown if Figure 1.
It was strongly recommended in the thirty-minute problem solving session that a career counsellor specialist from Yellowknife, NWT facilitate the workshop. In this way, teachers will be best served in fulfilling the required objectives directly from a person who specialises in the field of career counselling. At this point, the instruction is required on a local, one-school basis so it is possible to have a specialist flown in to provide appropriate instruction. If this instruction is seen as valuable for other schools in the region, a train-the-trainer model may be implemented. The teacher group also felt that it was important to have one person on the Junior/Senior High teacher team with a special interest in CPP to act as a contact person and be responsible for keeping a CPP Resource Manual updated. A CPP Resource Manual will be continuously updated with new CPP information. This manual will be accessible for teachers and will contain all instructional information used in the CPP instructional workshop. It is hoped that new teachers who have not yet participated in a three-day CPP instructional workshop use this CPP Resource Manual. The contact may also act as the facilitator for CPP instruction sessions held for teachers on the off years.
Now that the objectives, activities and desired format for the teaching instructional program are outlined, it is possible to consider ways to evaluate success of the program. The next section of this paper deals with the second question posed to the teachers at the thirty-minute problem solving session: What do you think is most important in such a program? or: What criteria would you use to measure success of the instructional program? The evaluation plan for the CPP instructional workshop is explained in the next section of this document.
Why do an Evaluation Plan?
According to Kilpatrick (1996), training professionals should consider evaluation the final step in implementing an instructional development program. Thus, instructional designers must consider in advance how they will go about the evaluation of the programs success. Kazanas & Rothwell (1998) go on to state that: "Most instructional designers believe . . . that instruction is not finished until it is apparent that the targeted learners can learn from the material" (p.264). Kazanas & Rothwell (1998) encourage instructional designers to "develop a formative evaluation plan that focuses attention on the instruction materials" (p. 267). Beyond instructional materials, Kilpatrick (1996) promotes evaluation on four levels: reaction, learning, behaviour, and results. He writes, "I considered measuring participants reaction to the program, the amount of learning that took place, the extend to their change in behaviour after they returned to their jobs, and any final results that were achieved by participants after they returned to work" (Kilpatrick, 1996, p.55).
This formative evaluation plan will outline how to measure the success of the Career and Program Plan (CPP) teacher instructional program implemented at Netsilik School in Taloyoak. This evaluation plan will consider the appraisal of instructional materials and the appraisal of instructional methods (Kazanas & Rothwell, 1998) and the extent of learning by teachers. As well, the plan will reveal the success of the program on the four levels of Kilpatrick (1996). The evaluation is the final step in the process to improve the implementation of CPP as described by Storey (1997) in the document entitled, The Nuts n Bolts of the Career and Program Plan.
It is important to remember that the focus of the evaluation is the effectiveness of the instructional workshop. Although, the evaluation uses information gathered from the participating teachers, they are not the focus of the evaluation. Thus, evaluation tools will be completed confidentially and the results are to reflect on the effectiveness of instructional workshop rather than on the professional abilities of the teachers. It is assumed that the quality of their evaluations reflects the quality of the instructional workshop and that all participating teachers will be professional in their study of CPP implementation at the instructional workshop.
Statement of Purpose, Objectives, Audience, and Subjects
Evaluation of the CPP instructional workshop intends to determine the programs level of success and to gain information to be used in program revision. Success is defined in a number of ways depending on perspective. In the thirty minute problem solving session, the group felt that the ultimate show of success of the CPP instructional workshop is the increase of students who are able to articulate a career interest that is both realistic and achievable. However, evaluating the program solely using this reference requires the added evaluation of student learning on a long-term basis. Although, the CPP program is intended to benefit students, it is beyond the scope of this evaluation plan; thus, the evaluation of the program will be based primarily on the three-day CPP instructional workshop.
For school administration, success is determined by the ability of participating teachers to implement the CPP program effectively and efficiently with students. For participating teachers, success of the workshop can be linked to their feelings of competence with the CPP implementation and their skills in the career counsellor role. Students might view success as the increase of support and expertise exuded by their teachers in the CPP classes and in one-on-one interview sessions. Parents may relate success of the program by judging the skills their children gained in creating their long-term plans. For the Kitikmeot Board of Education and District Education Authority (DEA), success is dependent on the amount of positive feedback given by the community and the number of students who are able to articulate their career and program goals. The Federation of Nunavut Teachers (FNT) may interpret success as being the level of competency and feelings of adequacy that teachers display by participating in the workshop. Finally, the workshop facilitator and instructional designer will interpret success by the amount of positive feedback provided by the evaluation. All of these stakeholders will be interested in the outcome of the evaluation of this CPP workshop. This evaluation plan considers these stakeholders as the audience of the evaluation report and will conduct data collection accordingly.
The subject of study will be teachers who are involved in the workshop: four Junior/Senior High homeroom teachers and the principal of the school who also teaches two courses in this division. Because the CPP workshop is being presented to such a small number of participants, it is unrealistic to limit the evaluation to a sample. Thus, all participants in the CPP instructional workshop will also be involved in the evaluation of the workshop. As previously mentioned, evaluation will consider the workshop materials, the workshop methods as well as the reaction, learning, behaviour, and results from the perspective of the teachers involved (Kilpatrick, 1986). People involved in the evaluation include all participating teachers, the principal, and workshop leader. The principal will have two roles in the evaluation process. The principal will be evaluated as a teacher as much as possible. They will also become the evaluators for the classroom supervision portion. The workshop facilitator and instructional designer will be involved in the evaluation of the program, as they will guide the principal in the type of observations that should be made. The facilitator and instructional designer are responsible for the interpretation of the results of the evaluation. Together, the facilitator and instructional designer will make suggestions as the improvement of the instructional process.
As stated earlier, there are a number of stakeholders who have differing information needs. The instructional designers will be interested in how the program (materials and delivery methods) can be revised to make them more effective for learners. The Kitikmeot Board of Education, DEA, and school administration will want to know how well the materials meet the instructional needs and increase teacher competency. They may also want to evaluate the financial support that is necessary to ensure that the CPP instructional workshop is properly performed in the school. They will want to know how well the materials and methods presented the pertinent information; it is likely that these groups will provide funding for the purchase of the materials and service of the facilitator. These groups will measure success using primarily numerical information as statistics and graphs.
The principal will be interested in knowing how well the instruction has achieved teacher learning so that they can supervise the application of program and give support where necessary. The principal will need to know the level of accountability that can be placed upon the teachers as a result of the workshop instruction. They will want to ensure that the information presented in the workshop is applicable to the school and can be used easily by staff in the work place. They will want to know if the materials are such that they can be used independent of the workshop so that new staff members can work through the information on their own during the off years of instruction. The principal will measure success through teacher reaction and comments about the course which will be collected anecdotal recording methods. Also, success will be evident in classroom supervisions.
The FNT will wish to know the benefits of the program so that they can promote more learning in this area, thereby increasing the value of teachers and justify demands for increased salaries. They will be interested in effectiveness of the materials and delivery methods in presenting information; the FNT will be involved in the financial support of the program. They will be interested primarily in numerical data but may find value in the anecdotal records of teacher reactions.
The participating teachers will be interested in the level of competency that the instructional workshop promotes. They may be concerned with the use of evaluation results by supervisors with regard to the future of the CPP instructional workshops. They will be interested in the thoroughness of the materials, presentation of relevant information, and the level of difficulty of the instructional workshop. It is hoped that the teachers will not find the evaluation threatening as it is intended to obtain information to determine the programs success and pending revision. Teachers will be interested in numerical and anecdotal information.
The facilitator and instructional designer will be interested in the feedback provided by the evaluation and the impact of the results in the revision of the instructional workshop. As they will be involved in the evaluation, they will have immediate results and will be able to draw conclusions from the raw data. At the conclusion of the evaluation plan, it is requested that they work together to prepare a report of recommendations for the program. This report will be presented to Netsilik School to aid in their future decisions about CPP instructional workshops.
The word evaluation is often misunderstood and at times it connotes negative impressions. For the sake of impartiality in communication, it is valuable to change the wording of public documents, letters, requests of permission and published results of the evaluation. Instead of using the term evaluation, it is more appropriate to label the exercise as a review of the program. It is hoped that this change in label will create a less threatening climate surrounding the events. In this document, the word evaluation will remain because the audience, Netsilik School, is knowledgeable about the goals of the project and have been involved in the planning from the beginning.
When evaluating any program, it is important to consider proper protocol and to open lines of communication. In this case the participating teachers and school administration (the principal and vice-principal) have been involved in the CPP discussion at the onset. They participated in two, thirty-minute problem solving sessions. The first meetings objective was to brainstorm the various enabling and constraining forces which impact the implementation of the CPP at Netsilik School. The second meeting was held to discuss the possible topics and workshop format of a CPP workshop and to brainstorm criteria to measure success of the workshop instruction. Their participation at the beginning stages of this project is important because it provides an opportunity for those involved to voice their concerns and give input on their learning needs and styles. As well, the participating teachers and school administration will be consulted about the expectations of the evaluation throughout the process. They will each receive the final evaluation report in paper form.
The DEA will also be consulted about the evaluation through a formal letter and a presentation of the program evaluation at a DEA monthly meeting. Myself, Deborah Maguire, who is the vice-principal and instructional designer on the CPP implementation project, will deliver the presentation. The presentation will stress the importance of having an effective CPP program offered at the school and the hope that such a program will reduce the number of students who leave school without plans for their futures. The presentation will also explain how the CPP instructional workshop is important for teachers who act as key players in the CPP program. Rational for the CPP implementation workshop and information obtained from an earlier needs assessment and constraint analysis will be referred to in the meeting. As well, the purposed agenda and topics for the CPP instructional workshop will be discussed and input from the DEA will be considered. The focus of the presentation is the evaluation plan and how the results of the evaluation will be used. The DEA will be able to peruse the evaluation plan and criteria before the evaluation is to take place. It is hoped that the DEA will approve the evaluation of the CPP instructional workshop. They will be notified of the evaluation results in paper form, one per DEA member.
The Kitikmeot Board of Education and FNT will receive a package of information detailing the CPP program, Needs Assessment, Constraint Analysis, CPP instructional workshop information and Evaluation Plan. Telephone contact will be made with Jean Phelps, director of the Kitikmeot Board of Education and with Donna Stephania, president of the FNT. The instructional designer will answer all outstanding questions.
Students and parents will not be involved in the evaluation of the program or the teachers until after one year of implementation has been accomplished. At that point, a sample of students and parents will be involved in a short interview of that is further described in this document. Communication with parents and students is important so any pertinent comments or concerns about the program before the first year is completed will be recorded by the facilitator or instructional designer and reviewed when the surveys are underway. Later, students may become objects of study when their progress in the CPP project is evaluated with respect to the results of the CPP workshop. Parents will learn of the results of the program evaluate in a more general and public way. They will not receive formal evaluation results but will have access to information that summarises the conclusions of the evaluation. Summary information will be published in the Netsilik News, the school newspaper. Parents may become aware of the results if the FNT decides to use the information to increase public relations.
As with any type of study, there are formal permissions required before the evaluation of the CPP workshop can take place. Formal permission letters will indicate specifically how the evaluation will take place, what the evaluation will measure, who will be involved, the amount of time required and how the information will be used. A management plan describing all evaluative methods will be included in the letters of permission. Letters of permission will be sent out early in the process to allow for those affected to consider the impact of their participation. Any outstanding questions not answered in the letter will be directed to the instructional designer who will take time to meet with those people needing further clarification. It is hoped that all communication with stakeholders will be clear and concise. All documents including letters, evaluation forms, surveys and other forms of communication, will be translated into Inuktitut so that Inuktitut speakers will able to understand precisely what is being expressed. The DEA secretary who works at the school will do translation; the services will be allotted in the budget of the CPP instructional workshop.
The participating teachers must be willing to partake in the evaluation and they need to understand how the results will be used. They need to be given time to complete any evaluation that takes place outside of the instructional workshop days. They require assurance that the evaluation will not directly impact their job security or be reflected in their personal teaching evaluations. The evaluation will be completed at the convenience of the teachers so that external stress does not impact the results. The teachers will be requested to complete permissions for the release of their evaluation for the purpose of improving the quality and effectiveness of the CPP instructional workshop.
The Federation of Nunavut Teachers (FNT) and the Kitikmeot Board of Education need to be approached for formal permission. A formal letter of permission will requested adequate time for the evaluation of the CPP instructional workshop. The FNT and Kitikmeot Board of Education will be approached to permit adequate time for the evaluation to take place; the management plan to guide the study indicates exactly quantity of time and when the time will be required. It is hoped that the cost involved in obtaining time for the evaluation (i.e. substitute teachers) will be absorbed by the Kitikmeot Board of Education. Again, communication with these two larger parties will occur in written form and will be clear and precise. Any queries will be fielded by the instructional designer and may be referred to the facilitator for issues related to the content of the program.
The three-day CPP instructional workshop is a complex and multifaceted endeavour. Thus, it is valuable to use many different techniques in the evaluation of such an instructional workshop. Included below are a variety of different evaluative methods intended to capture the quality of the workshop from a variety of perspectives: multiple choice tests, product checklist, live performance checklist, rating scales, interviews, and surveys. The information gained from the evaluation process will be used determining the success of the project and will guide decisions in the future. It will determine whether the instructional workshops should continue and what improvements should be made for future workshops. Participating teachers will remain anonymous throughout the evaluation process as it is wished that the evaluation be completed in a non-threatening climate.
There are six main types of evaluation that will occur. Figures 2 - 4 outline the categories of consideration: the six types, issues (What is begin evaluated?), description, evaluator, setting, time, positive and negative anticipated outcomes. The six types of evaluations to be used are as follows:
Pretest - (multiple choice, written survey answers)
Constructing a CPP Year Plan - (product checklist, performance checklist)
Post test - (multiple choice, written survey answers)
Classroom Supervision - (live performance checklist)
Survey - three month point - (written survey answers)
Interview stakeholders -(oral PMI interview sessions -plus/minus/I wonder...)
The next section of this document outlines the intent and the specifics of each evaluation tool. An explanation of how each evaluation tool is interpreted is also included in this portion of the document. The objectives of the program drive the evaluation format so these are incorporated in the discussion of each evaluation tool. The list above is written in chronological order to reflect the progress of evaluation through the instructional workshop and subsequential implementation of the CPP project.
1. Pretest - (multiple choice, survey written answers)
Many overlook the importance of using a before and after instructional test. In this case the pretest is a way to evaluate the participating teachers knowledge, attitudes and concern for non-instructional issues. It consists of items that measure entry behaviours (prerequisite skills) and items that test selected instructional objectives. This pretest is to occur almost immediately after the initial introduction to the course in the designated workshop room; it is planned to take about ten minutes at the onset of Day 1. Test papers will be numerically coded to ensure anonymity. Throughout this evaluative process, this coding will be used to identity individual growth from the pretest and the post-test.
The evaluation uses multiple-choice questions to measure factual knowledge about the CPP format, graduation requirements, student options, financial assistance issues and resource availability. The written survey answers allow participating teachers to voice concerns about non-instructional issues like constraints (time, resources, instructional needs of students, motivation etc). As well, the written answers give an opportunity for the evaluator to judge attitudes of the behaviour.
Multiple choice questions are easily marked either correct or incorrect to give a quantitative score per participant. The facilitator and instructional designer will evaluate the survey sections by looking for common themes that arise from the participants responses. The facilitator will use a rating scale to evaluate the content of the writing; each quality level will use verbal descriptors that represent specific criteria. The rating scale allows a quantitative mark to be placed on the written survey answers and show the pre-instructional strengths and weakness.
It is expected that this pretest will bring a focus to the workshop. It will give an opportunity for participants to voice concerns they may have about the CPP program so that they can be addressed later in the workshop. Hopefully, the pretest will not result in flaring up excessive negative feelings about CPP and turn the morning into a venting session which will lead to the loss of focus and concentration on the goal of the instructional workshop. The cost of such an evaluative tool is minimal with the majority of cost in the preparation of the tool itself and the evaluation of the results by the facilitator and instructional designer.
Participating teachers are expected to create a year plan which outlines the objectives, content of study, activities, and evaluation of the CPP process. The participating teacher with their homeroom class will use this year plan. The teachers will be given approximately three hours in the afternoon of Day 3 to construct their plans. They will have the opportunity to brainstorm with others in the group. Both the workshop room and computer lab will be available for use during this time. Internet access and other resources will be made available for this process. The school principal will evaluate the plan as to the appropriateness of the activities with the abilities of the students in the homeroom; they will consider the practicality of the year plan with respect to school and community culture.
Two checklists will be used in this second step: a product checklist and a performance checklist. The product checklist portion considers the actual product of a CPP year plan for use by participating teachers in the upcoming year with students. The performance checklist portion considers that behaviours and attitudes about CPP that are demonstrated by participating teachers. Teachers will be given prior knowledge as to what is expected by this activity so that they will be better able to reflect on their own product and performance.
The facilitator who will calculate the merit of the activities and content included in the year plan will evaluate data from the two checklists. The product checklist evaluates the content of the program offered, the appropriateness of activities, and the procedures or methods of CPP delivery as described in the year plan. The results show the level of skill and knowledge that participating teachers have about CPP implementation. The performance checklist evaluates the behaviour of the teacher in this planning process. For example, the selection of activities reveals the participating teachers knowledge of the various options available for students and the level of willingness in working with students and parents to raise their awareness of career choices. Both checklists allow for a quantitative value to be placed on the work. The benefits include the number of various elements that can be observed in a given amount of time with relative speed and consistency. An overall performance score can be easily obtained as a result of using checklists. There is relatively no cost to this evaluation other than the creation of the tools.
It is anticipated that the teachers will become more aware of the importance of CPP as a subject of study. Having time allotted specifically for this plan of action allows teachers to work effectively with colleagues. Participating teachers will be able to implement what they learn in the workshop immediately in their classrooms. It is hoped that the process will create a sense of ownership in the plan, which will increase the likelihood that it will be followed.
It is anticipated that some problems may arise from this evaluative tool. Creating a year plan is demanding and it is possible that some teachers will resist participating in the process because CPP is not usually considered a subject area, which requires a year plan. Also, a good plan does not necessarily mean that it will be properly enacted in the classroom. For example, teacher may be able to put a good plan down on paper but they may not have the required classroom management skills or motivational techniques to provide an adequate program for students. Others may criticise product and performance checklists as being too subjective in nature and interpretations can vary.
A post-test is used in conjunction with the pretest in order to measure the learning that is related to the program. The two tests can be compared to show development in the participating teachers knowledge and attitudes regarding CPP. As before, participating teachers will remain anonymous through the use of coded papers. Pretests and post-tests will have identical codes for the set. The post-test will be administered in workshop room by the facilitator; it will occur in the last ten minutes at the end of Day 3 just prior to the celebratory announcements of completion.
The post-test is similar to the pretest in structure. It uses multiple choice questions to evaluate factual knowledge. The survey, using written answers, measures attitudes about career planning, personal capabilities, and student development. It also allows participating teachers to express any outstanding issues that were not addressed during the workshop. The multiple-choice questions result in a quantitative score per participant. Again, the workshop facilitator will evaluate the survey section by using a rating scale. From this, common themes relating to the strengths and weaknesses can be identified and summarised in the final evaluation report. It is desired that this before-and-after approach will indicate learning that is related to the program.
It is expected that this post-test will neatly conclude the workshop. It will allow participating teaching to reflect on their learning and evaluate their personal / professional growth. It is hoped that the post-test does not end the session on a negative note nor minimise the learning to a single test. As with the pretest, the major cost to the post-test is in its creation and the analysis of data.
A very common evaluation tool used by school administrators is classroom supervision. Already, principals are required to do classroom supervisions on new staff for the first two year of probation and then every other year following. Teachers can request a classroom supervision at any point for assistance, mentoring or to update their current professional record. Therefore, participating teachers are knowledgeable of the expectations of classroom supervision as an evaluation tool and are familiar with the approach.
In this case, the school principal will act as an evaluator because of their skill and experience in this role. The supervisions of CPP classes will occur three times over the first year of implementation. The supervisions will occur in the participating teachers choice of room (homeroom or computer laboratory). Participating teachers will be given the opportunity to schedule the day and time of the supervision as well as to schedule a pre-visit and post visit. A pre-visit will occur at least a day before; in this pre-visit the participating teacher will discuss their lesson plan with the principal. In the classroom supervision visit the principal will observe the lesson and use a performance checklist to evaluate objectives. The post visit will occur in the principals office and will review the lessons and discuss the elements observed by the performance checklist. The participating teacher will be encouraged to self-reflect on their lesson. Mentoring is encouraged during this time and the principal should be able to provide suggestions on the improvement of the participating teachers program.
Using classroom supervision is a good way to see the CPP program in action. It allows the principal to observe the participating teachers willingness to work with students and their level of support for students. The classroom activities demonstrate the participating teachers knowledge of appropriate CPP projects and the ability to investigate the various options available for students. Skills in communication and in the participating teachers ability to use resources effectively can be observed in most classroom supervisions.
The performance checklist will be provided to the participating teacher at the beginning of the school year so that they are able to work towards the appropriate expectations. The checklist acknowledges quality differences using the distinctions: exceeds expectations, meets expectations, and below expectations. The performance checklist is a good tool for evaluating a variety of elements in a short amount of time. Once the principal is completed the classroom supervisions, visits and the performance checklists, the instructional designer will transfer the three distinctions into numerical values and produce a final grade.
Again, it is important to remember that classroom supervision is a tool for evaluating the CPP instructional workshop not to evaluate the teacher directly. Indirectly, teachers will benefit from this experience because they are encouraged to work towards demonstrating their learning from the CPP instructional workshop. There is also a level of accountability that can be observed from this technique. It is expected that participating teachers will value the regular feedback provided in this model and will appreciate the chance to demonstrate their learning in an authentic format.
Criticisms of this evaluation tool are derived from the subjective nature of the classroom supervisions. Participating teachers need to be assured that although this supervision is in the same format as those that evaluates their personal performance, the goal of this supervision is to evaluate the effectiveness of the CPP instructional workshop. All documentation will use coded identification and participating teachers will remain anonymous throughout the evaluative process. Poor quality supervisions will reflect on the CPP instructional workshop because the expectations are related to the objectives on the CPP instructional workshop. The cost for producing a performance checklist is relatively low and requires clear understanding of the objectives of the CPP instructional workshop.
Approximately three months into the CPP implementation, a survey of participating teachers will be executed. Participating teachers will be given six main questions to respond to in written form. The open-ended questions will reflect the main objectives of the CPP instructional workshop: knowledge of options available for students; knowledge of appropriate CPP activities; skills in accessing resource information; skills in communication with students about CPP; and, attitudes towards their role in the CPP project. Careful wording of the questions will encourage participating teachers to reflect on their professional growth throughout the CPP instructional workshop process and onto their classroom teaching experiences.
Participating teachers will be able to negotiate when the survey is to be completed. They will be given one and half-hours of school time. Substitute teachers will cover classes to facilitate this completion and the board will absorb the cost of the substitutes. The facilitator and instructional designer using rating scales to achieve a numerical value per participant will evaluate these surveys. Placing a numerical value on open-ended questions is quite a subjective form of evaluation. To balance this subjectivity, written comments will be thematically categorised to illustrate overall areas of strengths and weakness of the CPP instructional workshop. These themes will be published as part of the evaluation report and excerpts of the written portions will be printed as anecdotal qualitative data.
The positive outcomes anticipated by this survey procedure are many. It is expected that participating teachers will value the opportunity to self-reflect on their learning and provide input about their concerns about the CPP instructional workshop. The three-month point allows enough time for the implementation of the CPP to be underway and determines the longer-term development of the learning from the instructional workshop. Despite the amount of evaluation that has occurred to this point, it is hoped that the participating teachers do not feel overly scrutinised.
The last phase of CPP instructional workshop evaluation is the interviewing of a representative number of stakeholders directly impacted by the efforts to implement CPP. Approximately fifteen people will be interviewed: three teachers, one principal, six students, four parents, and one DEA member. The interview will occur after the first year of CPP implementation and will be held in a non-threatening environment either in the school building or at an external location where confidentiality is ensured. The instructional designer, one-on-one will conduct all ten-minute interviews with the interviewees. It will use numerical codes to identify respondents; it is valuable to know from which perspective the comment derives so the codes will reflect the stakeholder group designation (ex. S1 = student one).
The interview structure examines the positive impressions, negative impressions and areas of confusion of the stakeholders. This format is called a PMI where P represents plus, M represents minus, and I represents the sentence stem "I wonder..." The interview sessions will evaluate the level awareness of CPP given the emphasis on the program by the CPP instructional workshop. Questions will identify the strengths and weakness of the CPP program as implemented by teachers who participated in the CPP instructional workshop.
The instructional designer will record the comments as stated by the respondent. These comments will be considered according to the generative themes that develop. Various issues can be identified using this process of evaluation and the results will contribute the qualitative aspect of this evaluation study. Careful coding of generative themes is required to make accurate conclusions about the data gathered in the interview process.
The positive anticipated outcomes of this interview structure are numerous. This type of evaluative tool allows the instructional designer to see the larger picture or overall progress for the CPP program. It gives voice to stakeholders like students and parents who have not had the chance to provide input to this point. The interview structure allows any outstanding needs to be expressed. It is hoped that the interviewees are truthful in their responses. Some negative anticipated outcomes include the time required to complete the evaluation of fifteen, ten-minute interviews and the time to evaluate thematic categories. Teachers may also feel threatened by non-educators evaluating a school program.
The last step of the evaluation plan is to outline how the final report will be published and who will read it. As mentioned earlier, any public documentation regarding the evaluation of the CPP instructional workshop will use the word review instead of evaluation. It is hoped that the word review is less critical in connotation. Thus the title of the document will be The Review of the CPP Instructional Workshop". It should be limited in the number of pages so that those who wish to read it do not feel overwhelmed by data. As much as possible, information should be jargon-free, clear and concise. Summary information will be available in languages, English and Inuktitut.
The report will include direct quotations from interviews and written survey questions. It will include quantitative information displayed using a variety of visual graphing techniques. The instructional designer will include explanatory information regarding the structure of the CPP three-day instructional workshop as well as the evaluation progress. Summary information regarding the interpretation of the results will be included. Recommendations will be made concerning how to better the effectiveness of the CPP instructional workshop. Other issues may be included; for example, a need for resources or more time for implementation may be uncovered through the evaluation process despite the evaluation focus on instructional needs. It is hoped that any recommendations to improve the program will be followed through. If it so happens that the evaluation of the program deems the CPP instructional workshop inappropriate in fulfilling the needs of Netsilik School, recommendations will be made as to other options available.
Evaluation of the CPP instructional workshop is very important in guaranteeing that appropriate decisions can be made about CPP implementation. The constraint analysis previously identified teacher competency to be a major negative impact on the implementation of the CPP at Netsilik School. Retraining is a strategy to reduce this impact and the development of a CPP instructional workshop is intended to fulfil this need. The final, but often overlooked, step in designing instructional solutions is to plan the evaluation of the instruction. This document attempts to outline these two main components: 1) the CPP instructional workshop, and 2) the evaluation of this instructional solution.
Based on research by Kilpatrick (1996) and Kazana & Rothwell (1998) instructional designers must consider an evaluation plan as the final step in instruction. The evaluation plan outlined about incorporates the four levels of evaluation as recommended by Kilpatrick (1996): reaction, learning, behaviour and results. The CPP instructional workshop is evaluated in ways that uncover the level of success regarding each of these areas. Participating teachers are used in the data collection to measure reaction, learning and behaviour. Pretest and post-test tools show learning growth. The CPP year plan determines the level of efficiency of skills and knowledge relative to the content of the instructional workshop. The live performance checklists recorded during the planning the CPP year plan and the classroom supervision measure behaviour. The results of the program are measure both quantitatively and qualitatively through rating scales, surveys and interviews of stakeholders.
Kanazas & Rothwell (1998) outline specific information that must be considered in an evaluation plan: statement of purpose, data collection plans, communication plans, data analysis considerations, anticipated results, and time estimates. The CPP instructional workshop evaluation plan attempts to satisfy these needs. It is hoped that evaluation will encourage decision-makers to make wise choices when judging the effectiveness of the CPP instructional workshop at Netsilik School.