58 The Database Industry - Searching as Industry (Information Research)
Description
This article is from the Information Research FAQ, by David Novak david@spireproject.com with numerous contributions by others.
58 The Database Industry - Searching as Industry (Information Research)
The commercial information sphere existed in the 1970's and earlier. It
is far more developed, far better organized, far better funded, almost
always far more valuable and expensive than every other research
resource.
For the most part, commercial information is arranged reasonably
uniformly in large databases of full-text or bibliographic information.
Some databases are small, single source documents, while others are
vast unfocused collections of, for example, all the news from the last
15 years.
Most directories and journals can be made into a database, but
single-source databases do not enjoy much financial success. The market
is too limited and the cost of promotion too high (except in a local
market with newspapers). To overcome this difficulty, single sources
are grouped together into larger collections of databases on a
particular topic. These large database groups have become primary tools
in commercial research.
Developing these databases requires considerable expertise and expense.
Sometimes data requires abstracting, interpreting, and as with some
Lexis-Nexis and WestLaw databases, even expert legal interpretation.
Sometimes firms develop a portfolio of databases. Sometimes firms build
just one.
The marketing and consumer billing of such databases is then provided
by a relatively small collection of large database retailers. A list
can be found in our "Commercial Databases" article. As an indication of
the size of this market, Knight-Ridder sold Dialog & Datastar for a
figure approaching half a billion dollars.
This industry consisting of a wide collection of players, each
improving and developing the information from individual periodicals,
journals, news items - all very confusing for the end user. This is
elegantly illustrated by the database descriptions for Lexis-Nexis
databases (their preferred term is libraries). See
http://www.lexis-nexis.com/lncc/sources/ as an example of specific
databases. In particular, see their library on patents.
Many single-sources appear in different commercial databases. Further,
different databases sometimes include different information from the
same single-source. One database may include just abstracts, another
may include fulltext, chemical indexing and more.
As a result, most researchers are unfamiliar with what exactly is being
searched.
This state of affairs is not unproductive. Searching a 'Database about
Patents', is uncomplicated. You receive information on patents. It is
simple, informative and incomplete. Of course, researchers are busy
people. Time is critical. Results matter. We are familiar with this
system from searching the web too. Just what are the differences
between All-the-Web, Lycos and Altavista? If we fully understood the
complexities of each available database, yet still have a few databases
to consider - would our search be better? Often not. This system of
incomplete information also leads to great customer loyalty to database
retailers. Comparative information is dropped in favour of simplicity.
Ultimately, I am hard pressed to compare prices let alone describe the
differences between information products.
Prices actually model many a developed industry, remarkably similar to
the telephone or banking industry. As one friend commented, "bullshit
baffles the brains". The prices are complex on purpose. It becomes very
unrewarding to compare prices, and any conclusions are only valid in
specific circumstances - and will not hold in others. This trend,
familiar to us as a multitude of banking changes and telephone pricing
schedules, reinforces our need to stop price hunting and trust our
favoured information retailers.
This is not to say we should not compare prices, just that you will
find comparing prices a most unrewarding experience. It really requires
you to search and retrieve the same information on different systems -
and this does not even begin to touch different databases, or database
groupings, or variables that change over time like download speeds.
Optimistically, there are actually very few important databases in each
field. It may be simple to browse each of the databases in your field
and compare directly. You may never need to know more than a few
databases intimately.
Realistically, you will yearn for a simpler solution.
The commercial information industry has distributed information this
way for several decades. It is both sophisticated and quite difficult.
You will need to become experienced with inverted indexes, search
techniques (Boolean, truncation, proximity, field limits ...) and
properly phrasing the question in a way that will be answered by a
database search. I have always found the value of a database search
directly proportional to the length of the search query.
If you are incompletely skilled at database research, you will take
longer, pay more and locate far more information (or unwisely discard
more) than desired.
This is very different from searching Altavista and Webcrawler.
Doing your own research offers an opportunity to more closely influence
the research process. Sometimes only you understand the topic and
sometimes you can more quickly discard unimportant details. Certainly
it is becoming simpler to undertake some work yourself.
Many of the commercial databases are also available in a CD format.
Substantial subscription costs limit their availability to large
research institutions and libraries, but exceptions exist. I believe
world books in print costs AU$5000+. Provided you can find casual
access, it will cost you far less. Keep an eye on the age, though.
Sometimes (and only sometimes) online information is more recent.
The decision between undertaking research on your own or seeking
external help is really a decision based on your research expertise,
your budget, your access to information, your time, and the importance
of finding all the information available. It also depends on your
access to some decent research assistance. I will soon be able to help
with this.
What I do know is a newcomer to the commercial information sphere will
seriously underestimate the difficulty involved in searching, and
underestimate both the cost of research and the cost of research
assistance. Keep in mind this same system serves the needs of large
commercial conglomerates, professional legal research, and well
financed government studies. The commercial information sphere contains
far more valuable information than you need. Sometimes the internet is
just an interesting sneeze in comparison.
ยค Article: The State of Databases Today:2000 by Martha E Williams,
tracts the development of this industry with survey results. Found as
the foreword of the Gale Directory of Databases.
Continue to:
My Books
