55 Quality (Information Research)
Description
This article is from the Information Research FAQ, by David Novak david@spireproject.com with numerous contributions by others.
55 Quality (Information Research)
Information has value. It also has other qualities that will assist you
to judge information you may consider buying.
Accuracy: the factual nature of the information presented. If the
statistics purport to show a particular trend - how large is the margin
of error? How large is the sample size? How likely are there to have
been factual errors in their development? The measurement of
statistical error is now a refined science in some fields. A
statistical result can be inaccurate when the sample size is too small,
if the margin of error is too large, the sample collection procedure
incorrect, or a number of other situations.
Reliability: the support for trusting the solutions, both from
additional resources and from being able to duplicate the conclusions.
This includes the reputation of the researchers. No matter how
inaccurate and biased you may believe certain facts to be, successful
independent support of a suggested fact does improve its value.
Bias: conscious or subconscious influences that affect information.
Bias can occur in collection, preparation and presentation of
information. Most information you find will be tainted. Secondary
information is deeply affected. Statistics are not necessarily less
biased.
We counter bias in several ways. Firstly, we try to be aware of bias.
Where is bias likely? Which direction would the bias affect the
information? Secondly, we try to collect information with different
bias. This is why research based solely on government research, no
matter how accurate and reliable, is less valuable. Often information
from different countries can counter bias. Thirdly, we need to accept
bias is likely to exist. This is why primary sources are often more
valuable than secondary sources. This is why tertiary sources, like
experts, can rarely stand alone.
Age: The date information was created or compiled will feature
prominently in the value of information. Dates given sometimes mean the
date information was created, or the date information was compiled. How
old is a book compiled in 1995, which took the author 10 years to
finish? I find statistics often forecast information, prominently
displaying recent compilation dates but still use old census data or
the like to draw their conclusions. Information on the internet
typically has no date, and can be severely challenged because of this.
Purpose: purpose merits further discussion. When you are uncertain
about potential bias, you can look for reasons to distrust the
information instead. Suspicion is not equivalent to bias, but it can be
thought provoking. Privately, I have heard repeated rumours important
national statistics have been fudged in different countries. A
government research report investigating the price of books in
Australia would have a political purpose, a purpose that provides the
climate for some potentially significant bias. A tell-all book by
industry experts often includes a tremendous quality of insider
experience difficult to find elsewhere. While there may be a purpose of
self-aggrandizement, the purpose is less a climate for significant
bias. Medical research has perhaps the greatest climate for significant
bias, and this suggests the greatest standard of proof and external,
reliable support.
Accuracy, reliability, bias, age and purpose are very important in
research. This is what leads us to an appraisal of value. For years,
the tobacco industry funded 'independent' research finding smoking
minimally harmful to health. It is now likely there may have been
errors brought on by accuracy, and bias. Certainly, purpose was in
doubt. As new studies show smoking is harmful, we can also say the
original research lacked reliability. In some topics, like the
internet, research is perpetually suspect because it also ages so
quickly.
I have seen further discussions that add 'Coverage' and 'Authority' to
this checklist. Both have bearing on the value of the information
contained. By coverage, we mean how much detail is invested in covering
a specific topic. Sparse or shallow coverage is closely tied to missing
critical aspects of information. News stories frequently have limited
coverage.
Once you are acclimatized to these elements, you begin to see potential
for error in a whole range of information. Real-estate association
figures, expert opinions, Toothpaste advertisements and National GDP
figures all occasionally display some degree of warping and
manipulation, clouding the truth. The solution is awareness, comparison
and careful analysis. As a personal aside, this is part of the reason
for my personal dislike for market research: it is often taken far more
seriously than warranted and mean far less than suggested.
Continue to:
My Books
