Posted by Matthew Koehler on July 1, 2010 · Announcements, Featured · Leave a Comment 

Overall, the analyses presented are clear, in that they are widely accepted procedures (Chi-squared, t-tests, ANOVA, and regression), with dependent and independent variables clearly defined.

  • Family wise, and Experimentwise error rate become quickly large in this study.  By my count there were 12 t-tests and anovas that needed to be conducted for Table 1, each at alpha = .05. Similarly, I count 44 different tests to compute table 2, each at alpha .05. That is, each test is comparing two groups (in Table 1) to see if the difference in numbers is less than 5% likely due to chance.  That does mean, however, there is a 5% chance of saying there was a significant difference, even if there is none in reality (i.e., there was still a 5% chance the difference could be attributed to random variation). With 56 tests done overall, each at a 5% chance of making a mistake, we would expect somewhere in the neighborhood of 56*.05 = 2.56 mistakes to be made. Meaning, on average, we would conclude 2-3 significant differences that were in fact not differences at all.  Given the few significant differences found in the study, one could begin to question which are in fact real differences, and which in fact may be the result of testing so many thing in so many ways.
  • The authors stated they used t-tests, anova, and chi-squared in table 1. The use of anova doesn’t exactly make sense. There were two types of analyses — (1) categorical data independent variables (e.g., male or female) with categorial dependent variables (i.e, friendship possible or not) (2) And categorial independent variables (i.e., friendship possible or not) with a continuous independent variable (e.g., age).  Because there were only 2 groups in each (1) and (2) as the independent variable, its not clear why ANOVA is needed. (1) should be performed as Chi-Squared, and (2) should be done as a t-test. [NOTE: Learning issue ... t-test is for comparing differences between 2 groups, ANOVA is for comparing differences beyond simply 2 groups, or using multiple independent variables at once]
  • The authors fail to present sufficient statistics, an APA requirement. Even though they certainly were not writing in APA style, most disciplines will require sufficient statistics.  READ http://my.ilstu.edu/~mshesso/apa_stats.htm for more information (you will have to scroll down or search for the term “sufficient”). The basic idea is that they should present enough descriptive information for readers to run some basic “fact checking”. For a largely correlational study such as this, that means minimally a correlation table between all the variables measured.   For table one, there needs to be means and standard deviations, perhaps even at more than one cut of the data. That is, for the finding that there were differences for African Americans, it would be helpful to see the descriptive data of African American participants by every other factor (e.g., broken down by gender, by internet experience, etc). This would allow readers to explore hypothesis, including that the difference is not one of race or ethnicity, but rather of an interaction with another variable (for example, what if on average the african american participants in this study also spent a lot less time on the internet than the other participants … what’s the real cause for the difference?)
  • “However, in spite of being mostly matched demographically and otherwise, about half the sample population does not think it is plausible to gain new friends through social media.” The term “matched” has a few specific methodological definitions, none of which I believe the authors are referring to (a way of sampling, or assigning to groups, or conducting analysis).
  • Statistical assumptions — You could easily look up the statistical assumptions of each procedure using google, e.g., “statistical assumptions of ANOVA”. I would list them, and say something like, “The authors do not provide enough information to know whether or not the data meet this statistical assumptions”.  NOTE, the assumptions of “regression” are very different than “logistical regression”, so be careful. Also note, you would have little chance at this point to understand the assumptions of logistical regression, don’t feel bad. There are many people who complete their degrees who still wouldn’t understand them :)
  • Number of models presented in the regression (Table 2). The authors use four different models, which is not uncommon. What does require more clarification, however, is the lack of explanation for the purpose of the 4 different models, and how the experimenters decided which variables to use in the first one (what was the rationale?), and what order they were added for models 2, 3, 4.
  • Use of qualitative data – It was not clear why the qualitative data was collected, or why it was used the way in which it was (e.g., used to answer a completely different question — what are the differences between face to face and online interaction). Instead, a better use of the data might be to look for differences in the types of explanations given between groups that were observed to be quantitatively different (i.e., african americans).  That is, are participants from that sub-population giving different types of reasoning than the groups they were compared to? Using data this way may shed light on the reason, and perhaps causes, for the observed differences.

Leave your response!

You must be logged in to post a comment.