Some of the Most Common Questions Asked of Statistical Consultants: Our Favorite Responses and Recommended Readings
Keywords:
ReadingsAbstract
We addressed some of the most common questions or concerns encountered when consulting with applied researchers: detecting and managing outliers; handling missing data; multiple comparisons and familywise alpha rate protection; the disadvantages of dichotomization; the nature of the general linear model; writing the results section; determining the number of factors to retain in
factor analysis; power analysis; parametric versus nonparametric statistics; appropriate numbers of predictor variables for use in multiple regression; variable selection procedures in multiple regression; interpreting interaction effects; and alternative analyses for pretest-posttest control group designs. We offer brief responses, not exhaustive theoretical expositions, but we believe they will help fellow consultants, teachers, and researchers to answer their own questions and those of their consultees, students, and associates.
References
Abrahams, N.M., & Alf, E.F. (1978). Relative costs and statistical power in the extreme groups approach. Psychometrika, 43, 11-17.
Alf, E.F., & Abrahams, N.M. (1975). The use of extreme groups in assessing relationships. Psychometrika, 40, 563-572.
Allison, D.B., & Gorman, B.S. (in press). Power analysis for testing differences between dependent and independent correlations: Equations and software. Educational & Psychological Measurement.
American Psychological Association. (1983). Publication manual of the American Psychological Association (3rd ed.). Washington, DC: Author.
Baggaley, A.R. (1981). Multivariate analysis: An introduction for consumers of behavioral research. Evaluation Review, 5, 123-131.
Bailar, J.C., & Mosteller, F. (1988). Guidelines for statistical reporting in articles for medical journals. Annals of Internal Medicine, 108, 266-273.
Barnette, V. (1978). The study of outliers: Purpose and model. Applied Statistics, 27, 242-250.
Bartko, J.J., Pulver, A.E., & Carpenter, W.T. (1988). The power of analysis: Statistical perspectives. Part 2. Psychiatry Research, 23, 301-309.
Bird, K.D., & Hall, W. (1986). Statistical power in psychiatric research. Australian and New Zealand Journal of Psychiatry, 20, 189-200.
Borenstein, M., Cohen, J., Rothstein, H.R., Pollack, S., & Kane, J.M. (1990). Statistical power analysis for one-way analysis of variance: A computer program. Behavior Research Methods, Instruments, and Computers, 22, 271-282.
Cliff, N. (1988). The eigenvalue-greater-than-one rule and the reliability of components. Psychological Bulletin, 103, 276-279.
Cohen, J. (1965). Some statistical issues in psychological research. In B.B. Wolman (Ed.). Handbook of clinical psychology (pp. 120-142). New York: McGraw-Hill.
Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312. •
Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
Cole, D.A. (1988). Statistics for small groups: The power of the pretest. Journal of the Association for Persons with Severe Handicaps, 13, 142-146.
Comrey, A.L. (1985). A method for removing outliers to improve factor analytic results. Multivariate Behavioral Research, 20, 273-281.
Cook, R.D., & Weisberg, S. (1982). Residuals and influence in regression. New York: Chapman Hall.
Darlington, R.B. (1990). Regression and linear models. New York: McGraw-Hill.
Davis, C., & Gaito, J. (1984). Multiple comparison procedures within experimental research. Canadian Psychology, 25, 1-13.
Dupont, W.D., & Plummer, W.D. (1990). Power and sample size calculations: A review and computer program. Controlled Clinical Trials, 11, 116-128.
Edgington, E.S. (1969). Statistical inference: The distribution-free approach. New York: McGraw-Hill. ,
Edgington, E.S. (1980). Randomization tests. New York: Marcel Dekker.
Edgington, E.S. (1987). Randomized single-subject experiments and statistical tests. Journal of Counseling Psychology, 34, 437-442.
Eysenck, H.J. (1970). The classification of depressive illnesses. British Journal of Psychiatry, 117, 241-250.
Falzer, P.R. (1974). Representative design and the general linear model. Speech Monographs, 41, 127-138.
Feldt, L.S. (1961). The use of extreme groups to test for the presence of a relationship. Psychometrika, 26, 307--316.
Friedman, H. (1982). Simplified determinations of statistical power, magnitude of effect and research sample sizes. Educational and Psychological Measurement, 42, 521-526.
Gaito, J. (1970). Nonparametric methods in psychological research. In E.F. Heerman & L.A. Breskamp (Eds.), Readings in statistics for the behavioral sciences (pp. 38-49). Englewood Cliffs, NJ: Prentice-Hall.
Gibbons, J.D. (1971). Nonparametric statistical inference, New York: McGraw- Hill.
Goldstein, R. (1989). Power and sample size via MS/PC-DOS computers. American Statistician, 43, 253-260.
Gorsuch, R.L. (1983). Factor analysis. Hillsdale, NJ: Erlbaum.
Gorsuch, R.L. (1990). UniMult guide. Altadena, CA: Fuller Theological Seminary.
Hartigan, J.A., & Hartigan P.M. (1985). The Dip Test of Unimodality. Annals of Statistics, 13, 70-84.
Harwell, M.R. (1988). Choosing between parametric and nonparametric tests. Journal of Counseling and Development, 67, 35-38.
Henderson, D.A., & Denison, D.R. (1989). Stepwise regression in social and psychological research. Psychological Reports, 64, 251-257.
Hertel, B.R. (1976). Minimizing error variance introduced by missing data routines in survey analysis. Sociological Methods and Research, 4, 459-474.
Herzberg, P.A. (1969). The parameters of cross validation. Psychometrika, Monograph supplement, No. 16.
Hocking, R.R. (1976). The analysis and selection of variables in linear regression. Biometrics, 32, 1-49.
Huck, S.W., & McLean, R.A. (1975). Using a repeated measures ANOVA to analyze a pretest-posttest design: A potentially confusing task. Psychological Bulletin, 82, 511-518.
Humphreys, L.G. (1978a). Doing research the hard way: Substituting analysis of variance for a problem in correlational analysis. Journal of Educational Psychology, 70, 873-876.
Humphreys, L.G. (1978b). Research on individual differences requires correlational analysis, not ANOVA. Intelligence, 2, 1-4.
Humphreys, L.G., & Fleishman, A.I. (1974). Pseudo-orthogonal and other analysis of variance designs involving individual difference variables. Journal of Educational Psychology, 66, 464-472.
Jaccard, J., Becker, M.A., & Wood, G. (1984). Pairwise multiple comparison procedures: A review. Psychological Bulletin, 96, 589-596.
Johnson, A.F. (1985). Beneath the technological fix: Outliers and probability statements. Journal of Chronic Diseases, 38, 957-961.
Kendall, M.G. (1962). Rank correlation methods (3rd ed.). London: Griffin.
Kidder, L.H., & Judd, C.M. (1986). Research methods in social relations. New York: Holt, Rinehart, and Winston.
Klockars, A.J., & Sax, G. (1986). Multiple comparisons. Beverly Hills, CA: Sage.
Knapp, T.R. (1978). Graphics and human information processing. Journal of the American Statistical Association, 80, 499-512.
Kraemer, H.C. (1985). A strategy to teach the concept and application of power of statistical tests. Journal of Educational Statistics, JO, 173-195.
Kraemer, H. C., & Thieman, S. (1987). How many subjects? Statistical power analysis in research. Newbury Park, CA: Sage.
Lipsey, M.W. (1990). Design sensitivity. Statistical power for experimental research. Newbury Park, CA: Sage.
Little, R.J.A., & Rubin, D.B. (1987). Statistical analysis with missing data. New York: John Wiley.
Maher, B.A. (1978). A reader's, writer's, and reviewer's guide to assessing research reports in clinical psychology. Journal of Consulting and Clinical Psychology, 46, 835-838.
Marascuilo, L.A., & McSweeny, M. (1977). Non-parametric and distributionfree methods for the social sciences. Monterey, CA: Brooks/Cole.
Maxwell, S., Delaney, H., & Dill, C. (1984). Another look at ANCOVA versus blocking. Psychological Bulletin, 95, 136-147.
Mcsweeny, M., & Katz, B.M. (1978). Nonparametric statistics: Use and misuse. Perceptual and Motor Skills, 46, 1023-1032.
Meyer, D.L. (1991). Misinterpretation of interaction effects: A reply to Rosnow and Rosenthal. Psychological Bulletin, 110, 571-573.
Mood, A.M. (1950). Introduction to the theory of statistics. New York: McGraw-Hill.
Muenz, L.R. (1989). Power calculations for statistical design. In N. Schneiderman, S.M. Weiss, & P. Kaufmann (Eds.), Handbook of research methods in cardiovascular behavioral medicine, (pp. 615-633). New York: Plenum Press.
Murphy, E.A. (1982). Biostatistics in medicine. Baltimore, MD: The Johns Hopkins University Press.
NCSS. (1991). Power analysis and sample size. Kaysville, UT: Author.
Park, C., & Dudycha, A. (1974). A cross validation approach to sample size determination for regression models. Journal of the American Statistical Association, 69, 214-218.
Pirot, M., & Lustig, S.D. (1984). The conceptual and mathematical unity of analysis of variance, simple correlation, and multiple regression. Perceptual and Motor Skills, 59, 751-756.
Puri, M.L., & Sen, P.K. (1969). A class of rank order tests for a general linear hypothesis. Annals of Mathematical Statistics, 40, 1325-1343.
Puri, M.L., & Sen, P.K. (1971). Nonparametric statistics in multivariate analysis. New York: Wiley.
Rasmussen, J.L., & Dunlap, W.P. (1991). Dealing with non-normal data: Parametric analysis of transformed data vs. nonparametric analysis. Educational & Psychological Measurement, 51, 809-820.
Rawlings, J.O. (1988). Applied regression analysis: A researcher's tool (Chapter 7, Model development: Selection of variables, pp. 168-191). Belmont, CA: Wadsworth & Brooks/Cole.
Raymond, M.R. (1986). Missing data in evaluation research. Evaluation and the Health Professions, 9, 395-420.
Rosenthal, R., & Rosnow, R.L. (1991). Essentials of behavioral research: Methods and data analysis. New York: McGraw-Hill.
Rosnow, R. L., & Rosenthal, R. (1989). Definition and interpretation of interaction effects. Psychological Bulletin, 105, 143-146.
Rossi, J.S. (1990). Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical Psychology, 58, 646-656.
Rothstein, H.R., Borenstein, M., Cohen, J., & Pollack, • (1990). Statistical power analysis for multiple regression/correlation: A computer program. Educational and Psychological Measurement, 50, 819-830.
Saville, D.J. (1990). Multiple comparison procedures: The practical solution. The American Statistician, 44, 174-180.
Saville, D.J. (1991). Reply to Holland and Lea. American Statistician, 45, 166-167.
Share, D.L. (1984). Interpreting the output of multivariate analyses: A discussion of current approaches. British Journal of Psychology, 75, 349-362.
Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill. •
Southwood, K.E. (1978). Substantive theory and statistical interaction: Five models. American Journal of Sociology, 83, 1154-1203.
Stevens, J.P. (1984). Outliers and influential data points in regression analysis. Psychological Bulletin, 95, 334-344.
Stevens, J.P. (1986). Applied multivariate statistics for the social sciences. Hillsdale, NJ: Erlbaum.
Tabachnick, B.G., & Fidell, L.S. (1989). Using multivariate statistics. New York: Harper & Row.
Timm, N.H. (1975). Multivariate analysis: With applications in education and psychology. Monterey, CA: Brooks-Cole.
Tukey, J.W. (1977a). Some thoughts on clinical trials, especially problems of multiplicity. Science, 198, 679-684.
Tukey, J.W. (1977b). Exploratory data analysis. Reading, MA: Addison & Wesley.
Veiel, H.O.F. (1988). Base-rates, cut-points and interaction effects: The problem with dichotomized continuous variables. Psychological Medicine, 18, 703-710.
Velicer, W.F., Fava, J.L., Zwick, W.R., & Harrop, J.W. (1988). Program CAX (Component Analysis extended). Unpublished computer program, University of Rhode Island.
Wampold, B.E., & Freund, R.D. (1987). Use of multiple regression in counseling psychology research: A flexible data-analytic strategy. Journal of Counseling Psychology, 34, 372-382.
Weiss, B., Williams, J.H., Margen, S., Abrams, B., Caan, B., Citron, L.J., Cox, C., McKibben, J., Ogar, D., & Schultz, S. (1980). Behavioral responses to artificial food colors. Science, 207, 1487-1489.
Welch, W.P., Frank, R.G., & Costell, A.J. (1983). Missing data in psychiatric research: A solution. Psychological Bulletin, 94, 177-180.
Winer, B. J., Brown, D.R., & Michels, K. M. (1991). Statistical principles in experimental design (3rd ed., pp. 333-342, 442-445). New York: McGraw-Hill.
Zwick, W.R., & Velicer, W.F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442.
Published
Issue
Section
License
Copyright (c) 2025 Journal of Psychodrama, Sociometry, and Group Psychotherapy
This work is licensed under a Creative Commons Attribution 4.0 International License.