EDUR 9131: Course Index

EDUR 9131
Doctoral Research Methods

Spring 2015

Instructor/Facilitator: Bryan W. Griffin
My personal web pages can be found at http://coe.georgiasouthern.edu/foundations/bwgriffin/

Updates for future sections of EDUR 9131

a. Review sample project 1 manuscript and revise to include more attention to instrumentation (validity reliability present sample items etc), add regression component, maybe use sample proposal written for 7130 as background for new example manuscript report. Also, make cristal clear that regression analysis is required to be included. Also make clear that results must include APA styled tables for correlations and regression, and table of qualitative results. Make clear summary section must be present with research questions or hypotheses.

b. revise current activities not updated (e.g. sampling, effect size calcualtion, etc)

c. add instructions for composite scores calculation in exel and spss

Announcements:

Added as needed.

Instructor note - add chat notes prior to meeting but without answers

Course Index (check frequently for alterations)

A. Course Administration

Syllabus
Course Calendar and Reading Assignments -- see D below.

B. Activities that count toward course grade

Test 1: Due date to be determined; expect Test 1 after about coverage of Coding Open-ended Responses (after about 7th class session)
Test 2: Made available after the final class session, responses are due May 4 April 30 (so as not to conflict with comprehensive tests).
Activity 1: Research Project: This is a group activity and will be due as a PDF attachment by 9:00pm on May 4; see timeline and instructions on the linked page.
- Topic Assignments: Group Topics -- See this link to learn which constructs were assigned to your group.
- An example of a completed research project can be viewed here: sample research project manuscript.
- A sample questionnaire that can be used as a guide can be found here: dissertation_process_questionnaire_example.
- Andy Field (U. of Sussex) requires a similar project. He provides useful insight into the steps one must take to develop a questionnaire so I recommend you review his project instructions: Design a Questionnaire

C. Activities that do not count toward course grade

Activity 2 (Individual, non-graded) Instrument Construction Steps (complete and bring responses to second class session)
Activity 3 (Individual, non-graded) Internal Consistency (Answers to this activity can be found here: Internal Consistency Answers; Note to Instructor -- update data with reversed items)
Activity 4 (Individual, non-graded) Coding Open-ended Responses (due date to be determined).
- Data for coding: open_ended_coding_data_doctoral_program_assessment.pdf
Activity 5 (Individual, non-graded) Sample Size Calculations (Do not attempt this activity yet, it may be revised Spring 2015; due date to be determined)
Activity 6 (Individual, non-graded) Composite Scores (to be developed; due date to be determined)
Activity 7 (Individual, non-graded) Factor Analysis (to be developed; due date to be determined)
Activity 8 (Individual, non-graded) Effect Sizes (to be developed; due date to be determined)

D. Course Calendar and Content

Sessions

Date

Topic

1/19

Covered via Discussion Forum posts

Reading assignments are noted in the grey box below.

1. Syllabus Review

2. Big Picture -- what is goal of this course?

3. Activity 1: Research Project -- Form Groups for Activity 1

4. Activity 2: Instrument Construction Steps

5. Refresher review of educational research basics (variables, hypotheses, validity, reliability, etc.)

6. Collect data for later in-class analysis: http://tinyurl.com/5wcuv3n (private link to raw data here: http://tinyurl.com/4wb6unu Per Control, Per Competence, Goal Internal Menon 2001)

Read (or skim as a refresher):

Review material from EDUR 7130, Introduction to Educational Research

Variables and Scales of Measurement (practice exercise)
Hypotheses (practice exercise) (Notes on Writing Hypotheses )
Videos explaining how to write hypotheses and avoid common errors (videos from EDUR 8131 course):
- (a) Writing Hypotheses
- (b) Practice Writing Hypotheses
- (c) Common Errors When Writing Hypotheses
- Notes presented in videos are linked above as "Writing Hypotheses"
Reliability - brief review of concepts
- Measurement: Instrument/Test Reliability
- William Gabrenya's Reliability
Measurement: Instrument/Test Validity (practice exercise)

1/24

On-campus Class 9am Room 3165 College of Education

In-class Presentation Outline

Reading assignments are noted in the grey box below.

(Note: May need to briefly review previous week exercises on Variables and Hypotheses)

1. Questionnaire Item Development - introductory review

Instructor note - see Questionnaire Development Steps and Questionnaire Develop Steps in /content folder

Review Activity 2: Instrument Construction Steps
- Holmbeck & Devine (2009) Questionnaire Item Development Model
- Checklist only from Holmbeck and Devine
Content Validity --
- Review Holmbeck and Devince checklist, see also Table 1 in Floyd, Phaneuf, & Wilczynski 2005
- Step-by-step walkthrough with anxiety example; see four dimensions here: http://en.wikipedia.org/wiki/Test_anxiety
Incorporate validity check or a global or "overall" item when create scales of constructs (e.g., overall item, see Activity 1 example)

2. Literature Searches

Brief review using Google and Google Scholar
Literature Review Presentation Google Doc
(Instructor note: add literature review writing, list vs. integration, early vs. late citations, transitions)

3. SPSS Introduction and Review of Descriptive Statistics and Inferential Statistics (entering data, frequencies, M, Md, Mo, R, SD, Variance, Alpha, Beta, Power, Type 1 & 2 errors, p-value, r)

Review of Basic Descriptive Concepts (Google Doc presentation in class)
- Frequencies, Central Tendency, Variability
- SPSS Data Entry
- Hypothesis Testing: Alpha, Beta, Power, Type 1 & 2 Errors, p-values (Instructor Note: draw from ANOVA workshop material)
Logic of Hypothesis Testing; Coin flop comparing obtained p vs. alpha
Hypothesis Testing with p-values: Decision rule with alpha (Decision Rule: If p <= alpha reject Ho; if p > alpha fail to reject Ho )

4. Pearson Correlation, r

Pearson r review (reporting t-test results in APA Style: Reporting Statistics )
Example Correlation Results:
- Menon 2001, see Table 1, p. 1
- Hallam, Price, & Katsarou 2006, see Tables 1 and II, p. 115 (lacks *p<.05, t value)
- Scahill et al. 2001, see Table 1, p. 653
- Danielidou & Horvath 2006, see Table 2 p. 416
- Pintrchi & De Groot 1990 (motivated learning strategies)
- Bryson 1974 (student ratings correlated with achievement)

5. Group mean comparison with t-test

Independent samples t-test review (reporting independent samples t-test results in APA style: Reporting Statistics )
Example t-test results:
- Hallam, Price, & Katsarou 2006, see Tables 1 and II, p. 115 (lacks *p<.05, t value)
- Goodman, Hans, & Bernstein 2005, see Table 1, p. 555
Correlated samples t-test review (add material to chat; APA style: Paired-samples (correlated-samples) t-test )

Read (or skim as a refresher as needed):

de Vaus, Chapters 13, 14, and 15: Univariate Analysis, Bivariate Analysis
Descriptive Statistics and Graphical Displays
Inferential Statistics
Logic of Hypothesis Testing
Pearson Correlation, r
- Course notes from EDUR 8131 "Introductory Statistics": Notes 6 Correlation
- SPSS Example (see linked tutorial in Course Index; also William Gabrenya's correlation in SPSS)
- Practice Exercise: Correlation (answers)
Two Independent Samples t-test
- Course notes from EDUR 8131 "Introductory Statistics": Notes 5 t-tests
- SPSS Example (see linked tutorial in Course Index; also William Gabrenya's t-test in SPSS)
- Practice Exercise: One and Two Samples t-test (answers) -- Note, address only exercise items 1 and 4; items 2 and 3 are one-sample t-test.

Read if additional review is needed:

1. Review of Educational Research (linked readings below were taken from EDUR 7130, Introduction to Educational Research)

Read (if review is needed): de Vaus text chapters 1, 2, 3, 4, 5, 6, 7, 13 (section on levels of measurement), 14 (section on graphs), and 15 (sections on t-test and correlation)

Overview of Research/Scientific Method (general overview of research can be found here: Reader's Guide )
Format of a Research Paper
Variables and Scales of Measurement (practice exercise)
Purpose Statements (sometimes call Problem Statements)
Literature Review (on-line searches; common writing errors -- lists vs. integration, citations early vs. late, use of transitions)
Hypotheses (practice exercise) (Notes on Writing Hypotheses )
Research Questions
Sampling (practice exercises)
Descriptive Statistics and Graphical Displays (correlation coefficients and scatterplots) (practice exercise for displaying data)
Overview of Inferential Statistics (practice exercise) (integrate "Reading of Statistical Tables" from PDF links below
Measurement: Instrument/Test Reliability
Measurement: Instrument/Test Validity (practice exercise)
Quantitative Research Matrix (practice exercise)

2. Review of Descriptive Statistics and Inferential Statistics

Descriptive Statistics in SPSS (brief review of concepts: William Gabrenya's Descriptive Statistics)
p-values and hypothesis testing (logic, reading SPSS results, reading statistical tables)
- Logic of Hypothesis testing, probabilistic nature of hypothesis testing -- p-values
  - Supplemental review: http://www.sjsu.edu/faculty/gerstman/StatPrimer/Hyp-test.PDF
  - Supplemental review: William Gabrenya's Hypotheses and Inferential Statistics Overview)
- Types of Errors (see table of errors here: logic of hypothesis testing)

1/26

Live Chat Monday 6:30pm with Adobe Connect

Resume coverage of Saturday material.

2/2

Live Chat Monday 6:30pm with Adobe Connect

Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Questionnaire Development

General tips for format and layout, instructions, appreciation statements
Item pool development and item selection
Reading Factor Analysis results for purpose of selecting items
- Menon 2001
- Ragheb & Beard 1982
- Cheng 2004 (messy results)
Sample Likert Scale Response Options
- List of Step Descriptions for Five-point Likert Responses
- Dr. Siegle (U. Conn.) offers these: Likert Response Options
- Dr. Vagias (Clemson U.) provides these: Likert Options

2. Creating Electronic Questionnaires in Google Docs, Qualtrics

Google Support page: https://support.google.com/docs/answer/87809?hl=en
How to Build an Online Survey with Google Docs
Creating a Survey in Qualtrics

3. Brief review of t-tests if not already covered

4. Effect Size d - standardized mean difference between groups.

Effect size d formula: Thalheimer & Cook 2002 Becker 2000 (see also Notes 5 t-tests pp. 12-13)
- Interpretation of d (mean differences in standard deviation differences)
- Visualization of Cohen's d: Interpreting Cohen's d effect size an interactive visualization
- Cohen's recommended effect sizes (Cohen 1992 Table 2 (n per group))
- Calculation of d
  - d with control group SD
  - d with pooled SD (Thalheimer & Cook 2002 page 4) (Note, Thalmeimer and Cook list pooled SD lists denominator as n+n whereas it should be n+n-2)
  - d approximated from t-ratio result (Thalheimer & Cook 2002 bottom of page 5)
  - Use Table 1 Goodman, Hans, & Bernstein 2005, p. 555
- Effect size conversions: see Excel spreadsheet to calculate sample size or power linked above in (b), or see content\sample_size_effect_size_conversion.xls

Read for Questionnaire Development:

de Vaus, Chapter 4 Developing Indicators for Concepts
de Vaus, Chapter 7 Constructing Questionnaires (note section on Wording Questions for useful questions to consider when developing/selecting items)
de Vaus, Chapter 11 Building Scales

Supplemental Reading on Questionnaire Development

Brief Presentations on Questionnaire Development:
- Questionnaire Design
- Hints for Designing Effective Questionnaires (Frary 1996) Note Frary's recommendation against open-ended items; consider one's targeted population for a questionnaire when deciding whether open-ended items will elicit valuable information. The more educated, the better open-ended items may function. For less educated populations, interviews may work well when structured items are inappropriate.
- A Brief Guide to Questionnaire Development (Frary 2002) See comment above about Frary's recommendation against open-ended items.
- Guidelines for Constructing a Survey (Passmore et al. 2002)
- Hale 2010 (appendix on constructing measures; note list of things to consider; factor analysis; item analysis)

Review for Creating Questionnaires in Google Docs and Qualtrics

Google Support page: https://support.google.com/docs/answer/87809?hl=en
How to Build an Online Survey with Google Docs
Creating a Survey in Qualtrics

Read for Effect Size d

Daniel J Denis (2012). Understanding Cohen�s d
Thalheimer & Cook 2002, read through page 5
Visualization of Cohen's d: Interpreting Cohen's d effect size an interactive visualization

2/9

Live Chat Monday 6:30pm with Adobe Connect

Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Reliability: Stability of Scores with Test-Retest (scores must be correlated, and means should be similar; correlation and correlated samples t-test)

Instructor Note: Prepare data early for merging data file on respondent id. Can be lengthy process, so check Excel merging options.

Example data from class responses to this questionnaire: http://goo.gl/forms/wvccscLa7B (EDUR 9131 Employment Thoughts 2nd)
Test-retest Reliability (note include ICC, mean comparisons with r) with SPSS (warning -- rough draft): appropriate items, how performed; single item score [e.g. income] vs. composite scores [construct]; scores must be linked (Note add kappa, etc. other measures to assess test-retest reliability)
- Correlation may not be best since ignores mean differences
- Intra-class correlation (ICC) is better since it takes into account mean differences
  - Using page 1 data in my notes above, r = .958 and ICC = .961 (absolute agreement) and ICC = .958 (consistency)
  - if +3 is added to time 2 for the first data set, r remains .958, but ICC = .435 (absolute agreement) and ICC = .958 (consistency)
  - Use absolute agreement to test whether means differ, consistency ignores item variance and mean differences; absolute tests variability due to raters while consistency does not; absolute tests whether same scores are provided by each rater, consistency test whether relative scores show similar patterns
  - Single rater vs. multiple rater ICC results -- use single if assessing possible reliability of a single rater from several, use multiple if one plans to average ratings across several judges; multiple ICC tests whether multiple raters would provide similar mean ratings across several items
- Krippendorff's alpha
  - Ordinal results from page 1 data K alpha = .938 (t1 and t2)
  - Ordinal results from page 1 data with +3 added to t2 K alpha = -.17 (shows no agreement), note Krippendorff measures agreement, not consistency
Check Mean Equivalence: Correlated samples t-test
Published Examples:
- Kush & Watkins 1996, see Table 2 for test-retest correlations, repeated measures ANOVA to test mean differences p. 317
- Sapountzi-Krepia 2005, see Table 3 for use of Kappa to assess stability of categorical responses; Table 4 for intra-class correlation and rho (Pearson r) for stability and mean differences (also, nice discussion of correlation limitation, alternatives)
Single Item Measure Assessment for Reliability and Validity
- K Milton, F C Bull, A Bauman 2010 Reliability and validity testing of a single-item physical activity measure, Br J Sports Med; see Tables 3 and 4, Table 1 presents similar studies.
- Moss 2008 Single Item Measures, Psychlopedia entry

2. Equivalent forms (parallel forms) -- statistical assessment executed in same way as test-retest reliability

No additional readings

3. Reliability: Internal Consistency with Cronbach's Alpha

See material for 4 below

4. Item Analysis, Composite Score, and Reverse Scoring

Logic of Internal Consistency (use Table 3 and 4 examples: http://www.bwgriffin.com/gsu/courses/edur7130/content/reliability.htm )
Cronbach's alpha conceptualized: "the percent of variance the observed scale would explain in the hypothetical true scale composed of all possible items in the universe. Alternatively, it can be interpreted as the correlation of the observed scale with all possible other scales measuring the same thing and using the same number of items." Source: G. David Garson
Cronbach's alpha with SPSS
- SPSS reports two Cronbach's alpha values if correlations are requested
  - Cronbach's alpha = calculated on raw data items that usually have different variances, unequal variances
  - Cronbach's alpha based on standardized items = estimate of reliability if all items have equal variances; called Spearman-Brown stepped-up reliability coefficient; value of alpha obtained if all variables are standardized to have equal variances.
Example 1: Employment Thoughts Questionnaire http://goo.gl/forms/qIbDF5S7PE (use 1st administration data)
- Compare reliability estimates -- both alpha and test-retest to those reported by Menon 2001 (p. 171 alpha and test-retest reliabilities reported on 9 item scale)
SPSS analysis of Example 1 data
Dimensionality
- Cronbach's alpha is not a measure of unidimensionality (does not assess internal structure, use factor analysis instead)
- should not be used as overall internal consistency measure on instruments with diverse constructs
- Use Employment Thoughts data
Reverse Scoring Items and Forming Composite Scores
- Formula: Reversed Score = (minimum score) + (maximum score) � actual score
- Source: www.theanalysisfactor.com/easy-reverse-code , see first comment
- Calculation Check: Correlate original and reversed item, r = -1.00
- Composite: Sum vs. Mean
- Item Weighting Assumption: Equal contribution (simple sums/means) vs. Unequal Contribution (factor scores)
Second Example: Leisure Activities (raw data, private access)
- Spring 2011 results (SPSS file)
- Add 2015 data to 2011 data
Published Examples
- Fassinger 1994: Development and Testing of the Attitudes Toward Feminism and The Women's Movement (FWM) Scale. Item-total correlation, Table 1 p. 395
- Menon 2001: Reported in text format; p. 164 Cronbach's alpha and test-retest reliabilities reported on 15 item scale; p. 171 Cronbach's alpha and test-retest reliabilities reported on 9 item scale
- Kanning, B�ttcher & Herrmann 2012: Reported in table format; see Table 2 p. 145 (alpha and test-retest)
- Frey & Bos 2012: Reported in table format; see Table 4 p. 34 (alpha, item-total correlations minimum and maximum)

(e) Scorer-rater; inter-judge agreement -- Discussed below with coding open-ended responses

Read for Reliability:

1. William Gabrenya's Reliability Overview

2. de Vaus text

5th Edition

Chapter 11 pages 180 to 186
Reliability pages 52 to 53
Scaling Checklist p. 1959

6th Edition

Chapter 11 pages 179 to 185
Reliability pages 48 to 51
Scaling Checklist p. 194+

3. Joseph A. Gliem and Rosemary R. Gliem (2003) Calculating, Interpreting, and Reporting Cronbach�s Alpha Reliability Coefficient for Likert-Type Scales.

4. Tavakol and Dennick (2011). Making Sense of Cronbach's Alpha, International Journal of Medical Education.

5. SPSS: Cronbach's Alpha

6. Test-retest Reliability: Hopkins 2011 New View of Statistics: Measures of Reliability - simple yet detailed introduction to test-retest reliability

Supplemental Reliability Readings

Hopkins, 2000. Measures of Reliability in Sports Medicine and Science. Excellent sources for details of reliability estimates, how to design studies to assess reliability, uses of reliability, etc.

Klaas Sijtsma (2009) On the Use, the Misuse, and the Very Limited Usefulness of Cronbach�s Alpha. Psychometrika. 74(1): 107�120.

2/16

Live Chat Monday 6:30pm with Adobe Connect

Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Resume coverage of Reliability Material

2. Validity

Content Validity (Logical Validity) - brief review
Structural Validity
Construct Validity Overview (seeking anticipated/hypothesized patterns among structured response scores from instruments)
- McKenna & Kear 1990 p. 15 of PDF (predicted behavior with other variables, factor analysis?)
- Fassinger 1994 Convergent and divergent/discriminate validity examples , Table 2 p. 397
- Ragheb & Beard 1982 (correlation with similar measures; factor analysis)
- Menon 2001 (various)
Examples of how to report questionnaire development results in journal publications:
- Publication focus is primarily on questionnaire development:
  - Menon 2001
  - Ragheb & Beard 1982
- Publication focus is primarily on other research but questionnaire development briefly discussed:
  - McKenna, Kear, & Ellsworth 1995 p. 943+
  - Pintrchi & De Groot 1990 p. 34+
  - Kush & Watkins 1996 p. 316 (last paragraph in Instrument section)

Readings for Validity:

1. Validity Evidence

http://www.bwgriffin.com/gsu/courses/edur9131/readings/Validity_Evidence_Standards.pdf

Source: US Department of Education Office of Special Education Programs

Summary notes: http://www.bwgriffin.com/gsu/courses/edur9131/content/Validity_According_to_Standards.pdf

2. Table Outlines of Validity Evidence

Floyd, Phaneuf, & Wilczynski 2005; scan article and note particularly Table 1

Goodwin & Leech 2003; scan article and study Table 1 (2 MB file, may be slow download)

Review again the checklist from Holmbeck and Devine

Possible future readings to incorporate:

Haynes, S.N., Richard, D.C.S., & Kubany, E.S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7, 238-247.
Clark, L.A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309-319.

Quantitative approach to content validity: Lawshe (1975) A quantitative approach to content validity. Personnel Psychology, 563-575

2/23

Live Chat Monday 6:30pm with Adobe Connect

Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Coding open-ended questionnaire responses (conceptual analysis discussed here; see link in readings section to Palmquist 2008 for relational analysis example).

(a) Qualitative Data Analysis

Brief review of process

Study, read data carefully

Begin coding data

Inductive: codes, categories, themes are derived from raw data, emergent

Deductive: codes, categories, theme developed prior to analysis of data

Mixed: initial codes developed or expected prior to analysis, but revisions, deletions, additions occur during analysis

Identify categories or groupings of codes, develop codebook and codesheet

Identify patterns and connections among and within categories

Repeat process until one believes little more can be gained from repeating steps above

Independent coders should attempt to code data or samples of data and reliability of coding assessed

Develop report of results/findings - tables of codes/categories, quotations, descriptions

(b) Codebooks/Code sheets

Development: See readings below.

Sample codebooks and code forms (many examples here): http://academic.csuohio.edu/kneuendorf/content/hcoding/hcindex.htm

Sample Questionnaire and Coding Form:

Sample questionnaire: sample_survey_open_ended_authorship.pdf

Sample code form: sample_code_sheet_open_ended_authorship.pdf

Coded response #420: open_ended_coding_example_420.pdf

Coded response #1140: open_ended_coding_example_1140.pdf

Published version of questionnaire and results presented above: Moore & Griffin 2006

(c) Reliability -- inter-rater and intra-rater agreement

Rating Scales are Categorical/Nominal (angry, fearful, contempt, disgust) or Few Ranked Categories (poor, good, very good)

Presentation notes: Inter-rater Agreement with Nominal/Categorical Ratings with SPSS commands

Graham, Milanowski, & Miller (2012). Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings.

See their Table 1 for a nice illustration of the difference between reliability and agreement. Pearson r vs. percent agreement: r may be 1.00 despite no agreement.

Percentage Agreement

Simple count of "agreement codes" / "total codes" forms percentage agreement

Percentage agreement can be calculated for more than two raters

Generally percentage agreement is not recommended since it can capitalize on chance agreements, thus it may overstate the level of agreement, but do report percent agreement.

Cohen's Kappa

Useful for two raters with unordered rating scale or few ordered options

Discussion: Viera & Garrett 2005; present table for interpreting kappa; also note limitations of kappa

Example of kappa use: Uiters et al 2006, see page 4 and table

SPSS

Requires symmetrical scores: all codes must be present for each rater (Rater A uses 1 to 5, Rater B uses 1 to 4, no kappa) - update, new versions of SPSS correct this

Likely not useful if number of categories is large (many themes to coding responses, good for overall judgments with limited codes [excellent, pass, fail])

SPSS notes

http://www.stattutorials.com/SPSS/TUTORIAL-SPSS-Interrater-Reliability-Kappa.htm

http://www.sma.org.sg/smj/4412/4412bs1.pdf -- page 617

(problem, all categories must be present for both raters): http://www.ats.ucla.edu/stat/spss/faq/kappa.htm (can 3+ raters be used?)

Scott's pi (or Fleiss Kappa for two raters)

Same formula as Cohen's Kappa but calculates expected disagreement differently; often results are similar

Excel file to calculate Fleiss Generalized Kappa: http://www.bwgriffin.com/gsu/courses/edur9131/content/fleiss_kappa2.xls (Note, spreadsheet not working)

Original link: http://www.ccitonline.org/jking/homepage/interrater.html (Excel files for Fleiss kappa; large excel may be problematic)

How to calculate Fleiss kappa in Excel: http://www.real-statistics.com/reliability/fleiss-kappa/

Krippendorff's alpha

Krippendorff agues

alpha superior to Kappa and pi

serious work should see alpha > .80

can handle missing data when raters are 3+ unlike kappa and pi

SPSS Syntax for running Krippendorff alpha

Hayes' website: http://www.afhayes.com/public/kalpha.sps

Copied syntax here: SPSS syntax

Spring 2015 students noted errors with SPSS version 21 and 22; check syntax

Knut De Swert (2012). Calculating inter-coder reliability in media content analysis using Krippendorff�s Alpha. University of Amsterdam. Explains how to use Hayes' SPSS syntax to run K alpha, and meaning of values of K alpha. Provides examples.

Three or More Raters

All of the agreement indices noted above can be extended to more than two raters

See Inter-rater Agreement with Nominal/Categorical Ratings for examples and instructions

On-line Reliability Calculators:

Geertzen, J. (2012). Inter-Rater Agreement with multiple raters and variables. Retrieved February 20, 2015, from https://mlnl.net/jg/software/ira/

Deen Freelon (2015) ReCal: reliability calculation for the masses

Both report (a) mean percentage agreement, (b) mean kappa, (c) Scott pi or Fleiss kappa, (d) Krippendorff alpha

Material to Add

Index of concordance = A / (A+D) where A=agreement and D=disagreement

Replicate Table 3 to show problem with Cohen/Fleiss Kappa: Joyce (2013) Blog Entry: Picking the Best Intercoder Reliability Statistic for Your Digital Activism Content Analysis (PDF version of page)

Rating Scales are Ordinal with Several Steps, or Interval/Ratio Scale

Presentation notes: To be Revised Inter-rater Agreement with Ranked/Interval Data (revisions needed, ICC absolute agreement vs. consistency -- AA is same score, consistency is similar scores)

Two Raters

Agreement vs. Consistency

Are scores similar between raters: Question of Agreement

Is the pattern of scores similar between raters: Question of Consistency

See Inter-rater Agreement with Ranked/Interval Data for illustration of this difference.

Liao, Hunt, & Chen (2010). Comparison between Inter-rater Reliability and Inter-rater Agreement in Performance Assessment.

p 613 "Inter-rater agreement and inter-rater reliability are both important for PA. The former shows stability of scores a student receives from different raters, while the latter shows the consistence of scores across different students from different raters."

Not sure I agree with this statement. Consistency does not show consistency of scores across different cases. Scores can be widely different but perfectly consistent.

Update: They claim data in Table 2 show high agreement but low consistency. The are incorrect, agreement indices are very low. K alpha = -.29 (ordinal, interval, and ratio), and ICC for agreement = -.38 single and -5.00 multiple raters. Moreover subject means in Table 2 show almost 2 SD difference, large ES difference.

Is it possible to have high agreement but low consistency?

Pearson r and Correlated t-test

Can be used, but Pearson r is not a measure of agreement

Cronbach's Weighted Kappa for ordinal data

gamma? others?

Krippendorff's Alpha for ordinal, interval, and ratio data

Hayes' website: http://www.afhayes.com/public/kalpha.sps

Copied syntax here: SPSS syntax

Spring 2015 students noted errors with SPSS version 21 and 22; check syntax

Intra-class Correlation Coefficient

Absolute Agreement (scores match) vs. Consistency (patterns match)

Multiple raters vs. single rater: multiple for studies that use scores combined across raters, use single for studies using scores from a single rater

Sources:

Hallgren (2012). Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol.

David P. Nichols (1998) Choosing an intraclass correlation coefficient (in SPSS). Explains different model options, difference between consistency and absolute agreement, and between single measure vs. average measure.

Yaffee (1998) Enhancement of Reliability Analysis: Application of Intraclass Correlations with SPSS/Windows v.8. Explains differences in ICC provided by SPSS and gives examples.

Notes:

Wuensch (2014). Inter-rater Agreement. SPSS instructions with discussion.

Krippendorff Alpha for ordinal data provides much better assessment than nominal measures (kappa, etc.) whenever there is any order to data. Source: Antoine, Villaneau, Lefeuvre. (2014) Weighted Krippendorff 's alpha is a more reliable metrics for multi- coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation.

Three or More Raters

Krippendorff's alpha

Key benefit is ability to handle missing data from multiple raters

Handles ordinal, interval, and ratio data

Intra-class Correlation

SPSS instructions and discussion: Wuensch (2013) The Intraclass Correlation Coefficient.

Cronbach's alpha

Not a measure of agreement, instead measure of pattern consistency

Cronbach's alpha = ICC for multiple raters using consistency, but not for absolute agreement

Can be used as aggregate measure of consistency across raters if mean score is to be used

Source: Hayes & Krippendorff (2007; Answering the Call for a Standard Reliability Measure for Coding Data) explain this p. 81

Other Measures of Agreement/Consistency

Many measures exist. Some discussion can be found here:

Barnhart et a. (2014). Choice of agreement indices for assessing and improving measurement reproducibility in a core laboratory setting

Banerjee, Capozzoli, McSweeney, and Sinha (1999). Beyond kappa: A review of interrater agreement measures. Reviews a number of agreement measures.

Euclidean coefficients -- https://conservancy.umn.edu/bitstream/handle/11299/114459/v15n4p321.pdf?sequence=1

Loglinear models to examine patterns of agreement.

Latent Class models

Bennet's sigma

Gwet's gamma

Aickin alpha

Concordance correlation coefficient (CCC)

Readings for Open-ended Responses:

1. Qualitative Data Analysis (QDA)

Many of you probably have taken a Qualitative Research course so the readings below are brief and highlight the basic steps in QDA./p>

de Vaus Chapter 9

Taylor-Powell and Renner (2003) Analyzing Qualitative Data An easy to read, brief (10 pages) summary of QDA

Kent Crippen (no date) Analyzing Qualitative Data Outlined summary of QDA (3 pages plus examples)

Tobacco Control Evaluation (no date) Content Analysis: A Method for Analyzing Qualitative Data Another brief introduction to QDA (4 pages)

Video presentation on QDA, watch if interested: http://tobaccoeval.ucdavis.edu/files/ReadyTalk/Analyzing_Qualitative_Data/lib/playback.html

Supplemental Readings on QDA -- Explore these if interested

Le Compte 2000 Analyzing Qualitative Data. Theory into Practice, 39, 146-154

Palmquist 2008 Content Analysis

Glenn 2007 Analyzing Open-ended Questions (blog or discussion board post of simple steps to follow for QDA)

Guide to Content Analysis in Chemistry

2. QDA Codebooks

de Vaus Chapter 9

Stephen Kodish & Joel Gittelsohn (2011). Systematic Data Analysis in Qualitative Health Research: Building Credible and Clear Findings. Sight and Life.

Supplemental Readings on Codebook Development -- Explore these if interested

MacQueen, McLellan, Kay & Milstein, CDC (no date). Codebook Development for Team-Based Qualitative Analysis.

LaPelle (2004). Simplifying Qualitative Data Analysis Using General Purpose Software Tools.

Blog post: Manage evolving coding schemes in a codebook: 3 simple strategies

3. Reliability of Coders

Review of inter-rater agreement: http://www.bwgriffin.com/gsu/courses/edur9131/content/Inter_Rater_agreement.pdf (revisions needed)

Joyce (2013) Blog Entry: Picking the Best Intercoder Reliability Statistic for Your Digital Activism Content Analysis (PDF version of page)

Viera & Garrett (2005) Understanding Interobserver Agreement: The Kappa Statistic.

Hayes & Krippendorff (2007); Answering the Call for a Standard Reliability Measure for Coding Data; make the argument that Krippendorff alpha should be used for all inter-rater reliability/agreement situations no matter what scale of measurement used or number of raters.

Hrushchka et al (2004) Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research; read this because they make cogent arguments for assessing reliability of coders in QDA

Supplemental Reading on Reliability of Coders -- Explore these if interested

Graham, Milanowski, & Miller (2012). Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings. More detailed presentation of inter-rater reliability assessment. See Table 1 for a nice illustration of the difference between reliability and agreement.

Kilem L. Gwet (2011) On The Krippendorff�s Alpha Coefficient. Provides simplified method for calculating Krippendorff alpha.

Krippendorff (2013). Computing Krippendorff�s Alpha-Reliability. Walk-through for computing K alpha.

Stemler 2001 An Overview of Content Analysis (brief discussion of rater reliability)

Instructor Note: Other material to consider

Email explaining percentage agreement, Fleiss kappa, Krippendorff alpha

http://www.kenbenoit.net/courses/tcd2014qta/readings/Banerjee%20et%20al%201999_Beyond%20kappa.pdf

Krippendorff alpha details http://repository.upenn.edu/cgi/viewcontent.cgi?article=1286&context=asc_papers

http://www.agreestat.com/research_papers/wiley_encyclopedia2008_eoct631.pdf

Fleiss kappa http://en.wikipedia.org/wiki/Fleiss'_kappa

Fleiss kappa 1971 introduction: http://www.bwgriffin.com/gsu/courses/edur9131/content/Fleiss_kappa_1971.pdf (shows formula for 2+ rater agreement as noted in my email link

Liao, Hunt, & Chen (2010). Comparison between Inter-rater Reliability and Inter-rater Agreement in Performance Assessment. Illustrates how high reliability does not equal to high agreement, and how high agreement does not equal to high reliability.

They claim data in Table 2 show high agreement but low consistency. The are incorrect, agreement indices are very low. K alpha = -.29 (ordinal, interval, and ratio), and ICC for agreement = -.38 single and -5.00 multiple raters. Moreover subject means in Table 2 show almost 2 SD difference, large ES difference.

Artstein and Poesio (2008) Inter-Coder Agreement for Computational Linguistics. Excellent detailed source for explaining inter-coder agreement measures including K alpha -- good review source.

2/28

On-campus Class 9am Room 3165 College of Education

In-class Presentation Outline

Reading assignments are noted in the grey box below.

Interesting Validity/Reliability Assessment Dissertation: Kroupin (2011) VALIDITY AND RELIABILITY OF THE POWER/CONTROL SCALES. U. of Minn. 58 pages.

Shows that dissertations need not be long and can focus on assessing validity of scores obtained from a scale.

1. Class Example of Activity 1 Data Collection and Analysis

To be revised

Social Attitudes Scale: http://goo.gl/forms/C2Iusf6pQv

Social Attitudes Data: https://docs.google.com/a/georgiasouthern.edu/spreadsheets/d/1byIW3fYi4a7KZBeybdnseBxNuNwqFmk8THciH63tADI/edit?usp=sharing

Social Attitudes Items: https://docs.google.com/a/georgiasouthern.edu/document/d/1TLiYEkncOdtzE_O1zY2EcZZvD4JAfEqNZpiy4yk6cPs/edit?usp=sharing

R 11, 12, 15; 1-8 FA, 9-12 EPI, 13-16 SPI

Note to instructor:

Revise in Google to provide numeric responses -- Illustrative Questionnaire: EDUR 9131 Research Methods Review or http://tinyurl.com/yhfntjx (note, items worked poorly fall 09)
Data file to above questionnaire: http://tinyurl.com/4mzdj3l
(Note: 1-6 instructional rating; 7-10 value; 11-13 methods efficacy/confidence [r 12]; 14-17 effort regulation [r 14 15])
Poor results for last two constructs
Maybe use Leisure example, but no reversed items
Develop new example

2. Sample Size Determination for Categorical Variables (Chi-square tests)

To find sample sizes for studies involving simple proportions, such as responses to Yes/No type questions (assumes binomial distribution)

sample_size_proportion.xls (Excel [or OpenOffice] file download)
D. A. de Vaus (2004). Surveys in social research (5th ed.). Routledge press. See Tables and pages 80 through 83.
http://www.raosoft.com/samplesize.html

3. Sample Size Determination and Effect Sizes for Linear Models (t-test, Pearson r, Regression, ANOVA, ANCOVA):

Workshop Presentation: We will follow this during class
Excel spreadsheet for calculating sample size is located here: http://www.bwgriffin.com/samplesize/samplesize.htm

Group Comparisons: t-tests, ANOVA

To find sample sizes for group comparisons (e.g., t-tests, ANOVA type analyses)

sample_size_in_ANOVA_Maxwell_Delaney.pdf (per group)

Cohen 1992 Table 2 (per group)

compare results above to Excel calculator

Correlation

Sample sizes for Pearson's correlations

Arcsine transformation -- material to be added

Cohen 1992 Table 2 (recall slight difference with Pearson r sample sizes)

sample_size_correlation.xls (Excel [or OpenOffice] file download)

(Source for formula used in above Excel file: Dell, Holleran, & Ramakrishnan 2002)

An on-line calculator for sample size for correlation can be found here: http://www.biomath.info/power/corr.htm

Regression

Cohen 1992 Table 2

ANCOVA

Calculating adjusted effect sizes for ANCOVA - material to be added for

Sample Size Calculators

G Power -- free software for sample size determination

Free sample size software: http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/

Also downloadable from course web site (9.7MB): content\GPower3Windows.zip

On-line sample size calculators:

http://stat.ubc.ca/~rollin/stats/ssize/index.html

http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/SampleSize.htm

http://www.biomath.info/power/index.htm

Readings for Sampling and Sample Size:

Read: de Vaus chapter 6;
Jacob Cohen (1992) A Power Primer.

3/2

Live chat if needed

Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Finish Sample Size Determination

3/9

Live Chat Monday 6:30pm with Adobe Connect

No Chat Held

3/16

No Chat -- Spring Break

3/23

Live Chat Monday 6:30pm with Adobe Connect

Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Inter-coder Agreement for Ordinal, Interval, and Ratio Data (to be covered 4/13 or 4/20)

2. Regression Analysis

Examples data with unexpected results for multiple regression: Partial Correlation/Partialed Data
Reporting regression results in APA style: Reporting Statistics

3. Brief review of Factor Analysis

4. Study Guide for Factor Analysis (to be posted later this week)

Readings for Regression:

de Vaus

4th edition -- Chapter 11: Bivariate Analysis: Alternative Methods, p. 173-184

5th edition -- Chapter 15: Bivariate Analysis for Interval-level Variables, p. 279-289

6th edition -- Chapter 15: Bivariate Analysis for Interval-level Variables, p. 278-285

Online Notes from EDUR 8131 (Introduction to Statistics)

Notes 8a Simple Regression

Notes 8b Multiple Regression

Regression Coefficients Interpretation

Burns, W. (1997). Spurious Correlations. Explains necessity of "multiple" or partialed regression by illustrating mis-interpretation of correlational evidence.

Supplemental Material -- Review if Needed

Notes 8a Simple Regression (linked above) Videos

Instructional Videos and Related Materials

Supplemental notes shown in video: Supplemental Notes 8a

Sections 1 and 2: Regression Model and Figure 1

Section 2: Figure 2

Section 2: Figure 3

Section 3: Least Squares and Residuals

Section 4: Literal Interpretation of Coefficients

Section 4: Example 1 Barometric Pressure and Water Boiling Point

Data File in SPSS Format: Barometric Pressure and Boiling Point of Water

Section 4: Example 2 Irrigation and Cotton Yield

Data File in SPSS Format: Irrigation and Cotton Yield

Section 4: Example 3 Car Weight and Mile Per Gallon (MPG)

Data File in SPSS Format: Car Weight and MPG

Section 5: Model Fit

Section 6: Inference in Regression - Overall Model Fit (Part 1, Overall Model Fit on page 10 through Calculated and Critical F Ratio on page 11)

Section 6: Inference in Regression - Overall Model Fit (Part 2, starts with Exercises on page 11 and ends at Coefficient Inference on page 12)

Section 6: Inference in Regression - Coefficient Inference

Section 6: Inference in Regression - Confidence Intervals for Coefficients

Section 7: APA Style for Student Grades and Instructional Ratings

Notes 8b Multiple Regression (linked above) Videos

Instructional Videos and Related Materials

01 Purpose, Equation, and Partial Effects

02 Partial Effects

03 Partial Effects Part 2

04 Residuals and Literal Interpretation

05 Ice Cream Example

06 House Price Example

07 Model Fit

08 Inference

09 Confidence Intervals (Short video showing Confidence Intervals in SPSS with different levels of confidence)
10 APA Styled Results (Word document of House Price Example in APA Style )

3/28

No On-campus Class for EDUR 9131 -- B Griffin unavailable

3/30

No Chat -- B Griffin unavailable

4/6

No Chat -- B Griffin unavailable

4/13

Live Chat Monday 6:30pm with Adobe Connect

Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Finish Regression Presentation

2. Inter-coder Agreement for Ordinal, Interval, and Ratio Data (to be covered 4/13 or 4/20)

4/20

Live Chat Monday 6:30pm with Adobe Connect

Chat Presentation Outline

Reading assignments are noted in the grey box below.

1. Inter-coder Agreement for Ordinal, Interval, and Ratio Data (to be covered 4/13 or 4/20)

2. Factor Analysis

Readings for Factor Analysis:

de Vaus

4th edition -- Chapter 15: Factor Analysis, p. 257-275

5th edition -- Chapter 11: Factor Analysis, p. 186-192

6th edition -- Chapter 11: Factor Analysis, p. 185-191

Field (2005). Factor Analysis Using SPSS

Author unknown What is Factor Analysis? A Simple Explanation

Kootstra (2004) Exploratory Factor Analysis

Neill (2008) Writing Up A Factor Analysis -- provides example of how to reported factor analysis details.

Jackman (2005) An Introduction to Factor Analysis   A more challenging introduction

4/25

On-campus Class 9am Room 3165 College of Education

In-class Presentation Outline

Reading assignments are noted in the grey box below.

1. Factor analysis
2. Meta-analysis

4/27

Chat if needed

1. Factor analysis
Data 1: http://www.bwgriffin.com/gsu/courses/edur9131/temp/studentratingsdata.sav
Data 2: http://www.bwgriffin.com/gsu/courses/edur9131/temp/alphadata.sav
Alpha Data Questions: http://www.bwgriffin.com/gsu/courses/edur9131/activities/Assignment_6_internal_consistency_data.pdf
2. Meta-analysis

5/4

Finals week (Test 2 due April 30 so not to conflict with EdD comprehensive tests)

E. SPSS Tutorials (obtained from various on-line sources: Dr. McKnight at U. Oklahoma, Dr. Elvers at U. of Dayton, Guang-Hwa Change at Youngstown State U.)

Entering data into SPSS (variable labels, value labels, setting variable as numeric, etc.)
Data Entry A, Data Entry B
Frequencies/Explore Commands (including descriptive statistics, box plots, histograms, stem-and-leaf, etc.)
Frequencies A, Frequencies B, Stem and Leaf, Various Graphical Displays
t-tests (one sample, independent sample, and correlated samples)
t-tests, independent samples t-tests
Correlation
Correlation A, Correlation B, For scatter plots see Various Graphical Displays above
Regression
Regression A, Regression B
ANOVA
ANOVA A (using ONEWAY command), ANOVA B (using ONEWAY command), ANOVA C (using UNIVARIATE command)
ANCOVA
ANCOVA A

F. PDF File Creation

Since groups will be required to submit this project, I will only accept PDF attachments from one individual in each group. See detailed instructions on this in the Research Project link above. If you do not know how to create a PDF file, use one of the following sources:

Here are a few free web page that convert files to PDF over the internet:

http://convert.neevia.com/

http://www.gohtm.com/

http://www.pdfonline.com/

I use the following free software to create my PDF files (it leaves no watermark): http://www.primopdf.com/

I also use OpenOffice to create some free PDF files. Open Office is a free Office Suite similar to Microsoft Office (Open Office leaves no watermark): http://www.openoffice.org/

If you want further tips and links for converting to free PDFs, read this site http://www.pruittfamily.com/paul/freepdf.htm or this http://en.wikipedia.org/wiki/List_of_PDF_software .

Note: The material indented with grey background are comments for instructor about content to review for possible incorporation into course. Students should ignore this information.

Add links to on-line statistical programs, e.g.,

http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/SampleSize.htm#rmenu
http://www.statlets.com/
http://www.statlets.com/_vti_bin/shtml.dll/log_in.htm
http://gsociology.icaap.org/methods/resrch.htm (On-line Research Publications)

Reading Research Reports --- t-tests and ANOVA

Note -- Links to possible Reading Research report tables