Research thoughts of the day: What makes bad research?

I’ve been wanting to share some thoughts on this for a while. What makes bad research? I’ve read some dogmatic and untrained views and opinions being expressed strongly on Twitter and so just wanted to share a more balanced view from an academic standpoint.

The following are some reasons why an MSc research dissertation may get fewer marks, or none at all. I think this is a good starting point for a discussion.

A piece of work would be awarded a zero if it was copied or plagiarised and showed no intellectual input from the student.

If there was obvious low input of effort and superficial write-up conveying little of the context or value of the research the work would get a mark but still fail. It would be judged inadequate.

If the research was obviously deficient in effort and/or arguments and discussions were poorly resourced, if there was an uncritical use of literature, if there was an uncritical use of literature, if there was little sign of analytical techniques used or depth, if there was no clear programme of work and in sufficiently clear objectives… The work would still fail… But you’d be allowed to re-submit at the discretion of the exam board.

If your research is pedestrian or lacking in imagination and critical insight, or you failed to achieve your objectives fully, or your programme of work was not particularly ambitious or innovative… You might just scrape through with a pass. But that’s it. Thanks for turning up.

If you did well overall but had some unfulfilled potential… You might get a merit.

But if you don’t have an identifiable professional research approach, if you don’t fully complete your task, if you don’t achieve your stated objectives and include a good philosophical review of shortcomings, if you don’t demonstrate clear critical appreciation of your subject area, study methods and findings… You won’t be on for a distinction, and your work isn’t considered anywhere near potentially publishable as a journal paper.

The pinnacle of success for a good piece of research would be if it met the above criteria and was deemed potentially publishable as a journal paper, only with editing and minor revision needed.

Is that all? Any more thoughts?

If this is the case, why are so many teachers up in arms when papers get published about theories and ideas that are later reviewed, critiqued and “debunked”? Isn’t all part of the learning and exploring and the bigger journey of research contributing to knowledge? The fact that someone out there has explored and presented an idea is part of that journey. The fact that someone else has then come along and experimented with it and critiqued it and maybe even trashed it is part of that journey too. It’s all valid research.

3 thoughts on “Research thoughts of the day: What makes bad research?

  1. The most godawful dreck routinely gets published in fairly eminent journals; for heaven’s sake that is not an indicator of quality. I think the publication and attendant publicity is what irks people about bad research, not that it gets done – even fairly low-quality student stuff is fine as an unpublished pilot study, I’m sure no one care about that.

    This is not unique to educational journals, though they are very bad. This week a paper claiming that the sound of a ticking clock affected low SES women’s reproductive decisions got published in Human Nature. Yes, it was every bit as silly as it sounds. In comes from a long tradition of laughably atheoretical 20-per-cell social psych papers with eye-catching titles that are about publicity, not science. Diederik Stapel was famous for these – it turned out he actually did fabricate his data, but to be honest the rest of the field is only one step up.

    “Effort” is not an indicator of quality either. I’m sure Fred Lord’s papers explaining his paradoxes took all of an afternoon to write, given how short they are. And yet they are classics that anyone conducting regression analysis should read.

    I can add some useful indicators of poor quantitative research; small n, no power analysis, marginal p-values, no confidence intervals presented, no measure of effect size, strange erratic number of participants per cell, obviously exploratory research dressed up as confirmatory, outliers included or not seemingly at random, restricted-range samples used to make sweeping claims about population parameters…and on and on.

    Like

  2. Hey, wow, you sound like an erudite professor of very high acclaim, and have obviously won a Nobel prize or two to levy those kinds of criticisms my friend. So tell me, what amazing research have you done lately that is so groundbreaking?
    Good grief. You make me laugh.
    Are you are a teacher? At what level? Are you a researcher? In what field? Are you working in academia? In what role? Are you a nurturer or a destroyer? Are you engaging in the research conversation and journey with everyone else who is trying their best or are you just sitting there in your armchair dissing what other people have done because it’s fun for you, makes you feel clever and you can?
    I really don’t care about your opinion. I’m not seeking your approval for my own views. So basically my own response is that educating people in research is not all about tearing down. It’s also about working with what you have. Looking at what people have done and engaging in the process of improvement based on some basic criteria.
    There are many people involved in research, just trying their best. With many different limitations and constraints.
    I for one am quite pleased that there was the ticking clock research published in Nature this week. I’m happy because it means there are people out there asking what if. There are people out there exploring the boundaries of what we do and do not know. The more of that the better.
    I’m pro innovation and open minded enquiry. If people can get funding for that then great. I wish there were more open minded liberated thinkers out there exploring more whacky ideas. If you’re not one of them then I feel sorry for you. I feel sorry for you because you must be crippled in your thinking and it must be quite a small range of possibility you see before you in your mind’s eye.
    Have you ever had any fun with ideas and tried to postulate a new thought? Have you ever identified a gap in knowledge and tried to pose a question to try to contribute to knowledge? Have you ever been given a challenge to see if it is possible to create new tools and methods out of a completely unlikely theoretical domain just to see if there is any value in that because we don’t know yet and there might be?
    I like what researchers in HE have the freedom to get up to and explore. Research is great. May there be more wild and whacky research papers published. I beg for a proliferation of them. And if that’s what you call bad research, then the worse the better!

    Like

  3. I won’t comment on the original article because it’s about what makes good student research and not about the research field generally.

    I’m not an academic either and also cynical about research but I think you are wrong to refer to content. As Carol says (on Twitter) bad research really relates to methodology, epistemology or ways of gathering data that are limited, or geared to, achieving certain results. Finally there are research fields that I think make epistemological errors.

    For examples: learning styles developed because positivists in Psychology leveraged very limited data and people got excited by it. In the end it turned to be somewhat limited but it did make the profession think about how it delivers learning materials and differentiated learning.

    You could easily make the same comments about Ed Hirsch and DT Willingham who, in my opinion, leverage quite specific studies beyond their actual meaning in research, in order to, to communicate with teachers.

    I would also suggest that Daisy Christodoulou has a whole ideology based upon the work of Hirsch and Willingham placing memory and knowledge as the substantive functions of cognition based, which I personally think is discursive and based on quoting a number if friendly studies.

    There are fields in educational computing that are happy to quote learning styles literature without context, which I think is poor.

    In my opinion, there are fields where ontological and epistemological errors seem to be endemic particularly , again in educational computing, Adaptive systems etc, related to theory. However I can’t see research in adaptive systems going away any time soon.

    Some papers seem to fail to understand the work of those they are critiquing; Kirschner, sweller and Clarke springs to mind.

    You have alluded to what you consider is good Quantitative research but that only represents a fraction of what is possible in education where causal relationships are complicated by many variables and access issues.

    What happens, if you only have limited data? Or there are risks to participants.

    None of that has anything to do whatsoever with researchers conducting blue sky research in young fields looking for interesting results. The problem of research is nearly always where researchers start to build a career representing a certain political and ideological point of view that are adopted by politicians and practitioners as fact.

    Dunn and Dunn, Willingham, Hirsch could all be accused of that but who knows they may in the end be vindicated as may Daisy Christodoulou.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s