Week 4 – Discussion: Non-experimental Design
School Academic Counseling course
Respond to the following prompts in the Threats to Validity discussion
forum by Wednesday November 8 2023:
- Select a non-experimenal design
- What type of questions would you ask for your selected non-experimenal design?
- Regarding this weeks article, what are the general ed teachers level of understanding about evidenced based practice in teaching students with Autism?
Educational Measurement and Quantitative Designs
PPS 6025: Methodology of Educational Research
Created by Helen Y. Sung, Ph.D.
Educational Measures
Cognitive: measure learning and reasoning
Achievement
Aptitude
Intelligence
Learning style
Critical thinking
Creativity
Noncognitive: measure social and emotional characteristics
Personality
Attitude
Values
Interest inventory
Emotional Intelligence
Consider the following:
Which assessments do school psychologist use most and for what purpose?
What assessment tools to school counselors mostly use?
Tests
Norm-referenced:
The scores are interpreted compared to the well-defined norms.
The scores are compared with the scores of others who had taken the test.
Criterion-referenced:
Shows how individuals compare with established level of performance or skills.
Large scale standardized tests:
There is a uniform procedure for administration and scoring.
SAT, ACT, State Achievement Test, GRE
Standard based tests:
Standardized achievement tests with criterion based interpretation
Criteria for each grade level skills
Standardized Aptitude Tests:
Measures knowledge or skills that predict future performance.
Aptitude means to have predictive nature of the instrument.
These are the various types of tests used to measure progress.
You will want to know the difference between each of these test designs.
Norm Referenced
Standard Scores:
Raw scores are transformed into z scores by raw score –mean/standard deviation.
Standard scores are derived from the z scores.
Grade Equivalent:
Individual compared with the normative group in terms of grade level.
This can be confusing to a parent while it looks familiar.
Standard scores and grade equivalent scores compare the student with normative population. However, standard scores are typically use to report the level of functioning.
Types of Data
Questionnaire: True/False
Rating Scale: Likert Scale or variations of
Checklist: Provide number of options to choose from
Rank order: Put number of categories in sequential order
Observations: Descriptions of observed behaviors
Things to be cautious of:
Response set: refers to the tendency to respond the same way.
Faking: deliberately giving inaccurate response
This happens when the researcher indicates certain positive outcome.
The subjects response is to please the researcher.
Observer Effects
Observer bias:
Error can happen because of the background, expectations, or frame of reference.
Example – researcher may have had a bad experience with a person of different race.
Contamination:
Observer has knowledge of one or more aspects of the study and this knowledge affect subsequent observations
Example – male/female students on communication.
Halo Effect:
Observer’s initial impression influence subsequent observations
Example: poor or unkempt person
Interviews
Structured questions:
Subjects are given choice of answers.
Census
Semi-structured questions:
Questions are open-ended with specific intent, allowing individual responses.
Job interview
Unstructured questions:
Open-ended and broad.
The interview has general goal in mind and asks questions relevant to the goal.
Oprah’s interview
Nonexperimental Design
Descriptive:
It uses statistics such as frequency, percentages, averages and variability.
Often use graphs and visual images to describe the results.
Comparative:
There is a relationship between two variables but does not necessarily reveal the cause.
For example: increase in homework completion lead to higher test scores.
Other variables unaccounted for might be parent support, motivation, students ability, opportunities and exposer to other source of knowledge.
Nonexperimental Design continued
Correlational:
Pairs of scores are used to produce a scatter gram to calculate a correlation coefficient.
In prediction study, correlation coefficient shows how one variable can predict another.
Ex Post Facto:
There is no active manipulation of independent variables because it has already occurred.
Investigators decide on one or more preexisting condition have caused difference in subjects who experienced one type compared to a different condition.
Example:
Cooperative learning in relation to achievement.
Students were already engaged in a paired computer-learning situation.
The research question asks, “What type of interaction promoted increased achievement?”