So you came back?

1
7311

• So you came back?

2
What will we do today?

• How did we do on assignment 1?
• What did we learn from assignment 1?
• What was up with the readings?
• What do we do with this next set of data again?

3
Assignment 1

• You tell me first
• Ill give you some back story
• Well learn some things together

4
What did we learn?

• Groups
• Report out

5
readings

• Your questions please
• My ideas

6
Chapter 5

• Expert Oriented
• Expert panels
• Accreditation
• Site visits
• Peer review
• Keeps you out of trouble

7
What trouble is that?

• Contrary to 21st century belief
• Not everybody knows what is going on
• Surveys for example
• Just because you have survey monkey doesnt mean
  it is a good survey
• Other areas?
• Accountability
• Statistics is not ed measurement
• Data analysis is not ed leadership

8
2 models (examples)

• Peer reviewed journals
• Others who know the details dont get fooled
• Others who know the details may have a religion
• USED pilots etc.
• Others who do it and know it know what is wrong
• Some areas are full of opinionated experts

9
Chapter 5

• Consumer oriented
• Scriven (learn the name)
• Checklistsconsumer reports

10
Chapter 6

• Objectives oriented
• Tyler
• Set out your broad goals
• Classify goals
• Define objectives (behavioral)
• Find situations to observe
• Select measurement techniques
• Collect data
• Compare with objectives

11
Some goals

• Intellectual
• Emotional
• Physical and recreational
• Aesthetic and cultural
• Moral
• Vocational
• Social

12
More 6

• Logic model
• Theory based very theory driven
• Goal free
• Avoid finding out what program is supposed to do
• See what it does do

13
Chapter 7

• Decision oriented
• Help make decisions
• CIPP
• Context
• Input
• Process
• Product

14
The digression

• Ed leadership requires lots of LEADERSHIP
• Often you are explaining the Mona Lisa to a group
  of blind faculty (or so it feels)
• If you are starting someplace your staff has
  never been, some times it feels like a strange
  place to them

15
Names to use in conversation

• Popham objective plus
• Stufflebeam management
• Scriven consumer
• Eisner expertise
• Stake qualitative research and participant
  evaluation

16
These are your questions

•  How does the validity or reliability of the
  customer oriented evaluation approach stand once
  a customer sues for falsifying information?  For
  example, there were commercials ran advertising
  that the Sketcher sneaker will help the consumer
  to lose weight or build muscles in their legs.
  Now, there is a law suit due to the false
  information and all customers are to receive
  their money back.  Was the Scriven checklist
  implemented and why would this be allowable?
• How would one determine to implement the best
  practice of using the logic model or theory based
  evaluation to review a program design?
• What company or organization has noted success
  with implementing the CIPP model, UCLA or the
  Utilization focused evaluation?

17
More

• How might the expertise oriented evaluation
  approach to teacher education systems change in
  order to bring about improved quality of teacher
  education programs?
• Can an evaluation approach be a hybrid of both
  the consumer and objectives oriented
  approaches?  
• Step 5 (cost-benefit analysis) will not be
  optional and actually is the reason why a
  mediation is necessary. Again, my frustration
  level rises as I wonder why the process was
  implemented along the way so that data could be
  presented without mediation???
• With the implementation of digital common
  assessments, do you think that the
  objectives-oriented approach will be the best way
  to measure student achievement? It is still not
  clear how a student's content mastery will be
  determined.
• What is your knowledge of Reading 3D and MAP
  assessments? How have these programs been
  evaluated for district selection?

18
More 2

• My aha was how I compared this theory to the
  new accountability model, which seems to have a
  more objective-oriented approach in grading
  schools. Am I thinking through this accurately
• What are the criteria for Wingate Faculty to
  serve as advisor for Capstone Project?
• I am wondering how many good educational programs
  have been discarded due to an emphasis on program
  outcomes rather than on program theory? Also,
  how often do we implement or adopt a new
  educational program because it was sold as an
  effective model to improve test scores but we do
  not understand the rationale or understanding
  behind the program?
• In your opinion, do you feel site base team
  members are often given too much power for
  decisions to be made at the school without
  consulting every member of the staff?

19
Tests and items

• Standardized tests
• Not what you think
• Multiple choice as a life style
• Classical
• IRT

20
Classical

• Across the whole test
• Perfect score means?
• correct
• The more you get right the better you did

21
Item Response Theory

• Each item has meaning
• Look at what you got right
• Harder questions mean you know more
• Discrimination a
• Difficulty b
• Guessing c
• Theta!!!

22
The one FLAW!

• How do we know Theta?
• Usually by looking at how the subjects did on the
  test
• Prefer outside validation
• Teacher identification
• Demonstrated factors
• Good luck finding it!

23
More 3

• I recently read the report which evaluated
  Teacher Education Programs in the United States.
  As I read chapter 5, I was reminded of many key
  elements and evaluative approaches that go in to
  such a report. Some thoughts that emerged are
  was this report/evaluation meant to be formative
  or summative? What will the response of each
  university/college be? What changes will result
  from these findings from the evaluator(s) as well
  as the evaluatees? With this being the first
  evaluation of its kind, what will we likely see
  and how often will this evaluations take place?
• I struggled to fully grasp the Logic Models and
  Theory Based Evaluation Approaches application.
  It seems, but I may be mistaken, that we use
  these models/approaches in evaluating educational
  programs and projects. Am I incorrect in this
  assumption? It seems that if one of its major
  criticisms is that it tells us little about how
  the program achieves its objectives, then why
  would we use it as seemingly often as we do?

24
Nice article

• http//www.eric.ed.gov/ERICDocs/data/ericdocs2sql/
  content_storage_01/0000019b/80/19/db/68.pdf

25
Tests and items

• IRT
• Difficulty
• Discrimination
• Guessing
• Todays guest is difficulty

26
So what?

• When you dont know the parameters of what is
  being asked, how do you know the rigor?
• How else do you know there is an issue?
• How about behavior?
• Maybe parent signatures?

27
Data set 2 is fun

• This time ABCs
• Looking at the files
• Using Excel to crunch the numbers

28
More 4

• I never knew the history of the accreditation of
  our schools. It traces back to the GI Bill passed
  by Congress after World War II to make sure that
  returning soldiers were attending quality
  institutions of higher education. Is this really
  how this process came about?
• When the state takes over control of accrediting
  its own schools, does the public even notice? Is
  accreditation as big of a deal as it used to be?
•  So are the evaluators ever a part of the
  decision making?
• Could Consumer-Oriented Evaluation like What
  Works Clearinghouse be applied to schools and
  districts for growth and not proficiency?
• Aren't using logic models as a formative measure
  more practical than a formal evaluation?
• Progress Monitoring is primarily about
  accountability than improvement.  How can we
  focus it more on improvement?

29
More 5

• NCATE accreditation-how rigorous is this really?
   Teachers in our schools who are so
  ill-prepared...major difference among regions and
  universities/colleges.  How powerful would it be
  to look at data from students of teachers who
  graduate from certain universities??
• Let's say you were evaluating a reading
  program...what would theory-based evaluation tell
  you?  I can visualize the purpose of
  objective-oriented, logic, and goal-free but I'm
  not quite understanding theory-based....use/purpos
  e.  Is it just saying what you think the purpose
  of the program is?
• So, is the idea for our Capstone that we will
  evaluate a program?  Would that look like one of
  these evaluation approaches or multiple
  approaches?

30
Welcome back to Excel!
31
Context to the assignment

• What are the ABCs
• Why do we have to calculate districts?
• What is growth vs. proficiency
• Other fun

32
How is it going?

• Clinical
• Portfolio

33
For next time

• Read and write questions
• Chapters 8-10
• Data assignment 2 (not 2 and 3 just 2)