Technology and Human Responsibility

Issue #161                                                   March 9, 2005
                   A Publication of The Nature Institute
             Editor:  Stephen L. Talbott (

                     On the Web:
     You may redistribute this newsletter for noncommercial purposes.

Can we take responsibility for technology, or must we sleepwalk
in submission to its inevitabilities?  NetFuture is a voice for
responsibility.  It depends on the generosity of those who support
its goals.  To make a contribution, click here.


Quotes and Provocations
   High-Minded Machines
   Robots: Getting Down to Essences
   Maybe We Should Let Computers Have the Schools
   On Scrambling Genes Safely and Precisely


About this newsletter


                         QUOTES AND PROVOCATIONS

High-Minded Machines

Consistent with the general deterioration of the environment for
online exchange, NetFuture is increasingly waylaid by spam blockers.
For example, some readers did not receive the last issue because (so the
spam blocker blushingly informed me) the word "orgy" was used.  A little
surprised by the accusation of indecency on my part, I went to check,
and it turned out I had referred to an "orgy of self-congratulation
and utopian prediction" within the Human Genome Project.  As I was
reflecting upon my transgression and the spam blocker's display of
programmed pure-mindedness, it suddenly struck me:  Ray Kurzweil
was right after all!  We are entering an age of spiritual machines.
It's just that he forgot to tell us they would be dumb as hell.

Robots: Getting Down to Essences

In the last issue of NetFuture I described how artificial intelligence
researchers, after labeling segments of their program code with phrases
such as "understanding module", "problem solver", and "ego function",
mistook these labels for the corresponding human capabilities, as if their
computers could now understand, solve problems, and so on.  Their faith in
the efficacy of their labels amounted to a kind of word magic.  But you
probably thought I was referring only to a strange malady afflicting AI in
the old days.  Unhappily, this isn't true.

The latest example comes from South Korea's Kim Jong-Hwan, director of the
Intelligent Robot Research Center and a leading light in the world of
robotics.  He has made a big splash by announcing the creation of fourteen
"artificial chromosomes" that will enable robots to gain distinctive
personalities and reproduce:

   Christians may not like it, but we must consider this the origin of an
   artificial species.  Until now, most researchers in this field have
   focused only on the functionality of the machines, but we think in
   terms of the essence of the creatures.

This essence, so the story goes, will determine how happy or sad, how
angry or sleepy a robot is.  Moreover, if you're worried about this new
threat to the human species from an intelligent competitor now armed with
chromosomes, Kim assures us that "if we design the chromosomes quite
safely, then we can avoid such a bad situation".  What a relief.

Kim's announcement triggered the usual media coverage.  A threadbare and
scarcely coherent Guardian story, entitled, "Sex and the Single Robot",
repeated Kim's claims and talked about robots "feeling lusty".  This story
was then cited numerous times in the wider press and electronic media.
So, then:  do you want to know the truth of the situation?  Or does it
even matter, since the story makes for such great headlines?

Well, I'll tell you in any case.  Kim has no chromosome.  There isn't even
anything pretending to be a chromosome.  There is no thing at all.  Look
as hard as you can in any robot Kim is building, and you'll find no
material entity whatever that answers to his chromosome.  He knows this
very well himself.  The chromosomal essence of his creatures is, by his
own description, a bit of software code.  He doesn't provide much in the
way of details, but you can be quite sure that one thing the software is
made to do is to calculate various quantities, which are then assigned to
labels, or symbols, such as "happiness" and "anger".

Such, then, is the robot's happiness or anger.  And it comes without any
material reality called a chromosome -- let alone any material reality
that could reasonably be called a chromosome.  The invention of artificial
chromosomes (and chromosomes are, after all, material entities) is either
a cynical public relations gambit or else a mirage within the desert of
Kim's electronically sterilized imagination.  It's rather as if I
announced to the medical world my invention of an artificial heart, and
then when desperate patients began clamoring for it, I delivered some
software that calculated the volumes of fluid passing through a computer-
simulated heart.

The picture, then, is this:  you have a robotics guru giving a talk about
his work to an audience in New Zealand, with a couple of reporters picking
up a few phrases such as "artificial chromosomes", "essence of the
creatures", "reproduction", and "happiness" -- not to mention the
irresistibly delicious "Christians may not like it".  Suddenly you have a
news item about living, loving, copulating robots, which is picked up and
endlessly repeated around the globe.  The capacity to assess this item on
the part of journalists and the public seems to be about nil, as is the
willingness of technical professionals to help by rising up in protest
against what many of them know to be utter garbage.

There is a great virtue in politeness, but in this case I tend to believe
that the greater virtue is to point out the obvious truth, which is that
in most respectable disciplines anyone pushing the kind of claim Kim is
promoting would be called a charlatan -- as I would be if I seriously
promoted my artificial heart.

When, not long ago, a weird religious group announced it had cloned a
human, the press began demanding, "Show us the baby".  But I haven't heard
anyone asking Kim to show, or even explain, his chromosomes.  Everyone
seems quite willing to take it on faith that if he says he's named some of
his program code "chromosome" -- well, then, in one way or another he must
have something that should be thought of as a chromosome.  And if he
associates some numbers in his software chromosome with the phrase,
"desire factor" or "proclivity toward anger", then there must be a source
of desire or anger in there somewhere.

To mention charlatanry is harsh, I know.  But if Kim or anyone else fully
qualified in artificial intelligence is offended, I invite them to avail
themselves of this space and explain how wrong-headed my comments are.

Related Articles:

"From HAL to Kismet: Your Evolution Dollars at Work" in NF #149:

"Logic, DNA, and Poetry" in NF #160:

Maybe We Should Let Computers Have the Schools

A massive study by German researchers, widely reported last December,
questioned the educational benefits of computer use.  As the Christian
Science Monitor summarized:

   From a sample of 175,000 15-year-old students in 31 countries,
   researchers at the University of Munich announced in November that
   performance in math and reading had suffered significantly among
   students who have more than one computer at home.  And while students
   seemed to benefit from limited use of computers at school, those who
   used them several times per week at school saw their academic
   performance decline significantly as well.

   "It seems if you overuse computers and trade them for other [types of]
   teaching, it actually harms the student," says lead researcher Ludger
   Woessmann...."  At least we should be cautious in stating that
   increasing [access to] computers in the home and school will improve
   students' math and reading performance".

As most readers know, I generally don't put much stock in social research
of this sort, regardless of which side of an issue it comes down on.  What
interests me in the current case is not so much the report's conclusions
as the response to them:  numerous commentators have been heard moaning
about our failure to train teachers in the effective use of computers.
Surely, we're told, the fault must lie with the teachers, not with the
bureaucrats and industry consultants who've been busy telling all teachers
everywhere how they should convey their life-long learning to students!

Yes, teacher ineptitude is a logically possible explanation of the
situation.  But you'd think these pedagogical experts would occasionally
ask themselves whether the widespread teacher resistance might have a more
reasonable explanation -- namely, computer ineptitude.  It is, after all,
just possible that a great deal about the computer works against
its educational use.  It might be, for example, that the computer tends to
make more difficult the single greatest educational task today, which is
to bring the student into the fullest, richest engagement with reality --
the reality of the natural world, the human being, and society.

To hear (now former) U.S. Education Secretary Rod Paige tell it, such
questions hardly need raising, because the essential role of computers in
the classroom has become an axiom of faith.  In the National Education
Technology Plan, released in January, Paige tells us that "Education is
the only business[!] still debating the usefulness of technology.  Schools
remain unchanged for the most part despite numerous reforms and increased
investment in computers".

In other words, we've sunk a lot of money into computers and it hasn't
helped much, so obviously (as the report urges) we've got to sink more
money into them.  In particular, we need more training.  No funds for such
training?  Not to worry, say the authors.  A little "reallocation of
existing budgets" should do the trick.  Presumably, whatever the money was
being used for before didn't have much to do with real education.  Of
course, no one's explaining why, if the educational bureaucracy screwed
things up so badly before by wasting money on useless things, we should
now trust it to reallocate wisely the vast sums being shifted from
teachers to machines.

All of which reminds me of the latest satirical Christmas Letter put out
by computer scientists Dorothy and Peter Denning.  (This one is entitled
"Denning's Christmas 2101".)  Looking back from the future, they describe
how in the 2020s

   schools decided to save money by letting go the teachers and replacing
   them with computers presenting recorded lectures in the classroom.
   Students responded in kind, setting up computers that recorded the
   teacher-computer emanations and asked stock questions.  Administrators
   added assessment computers that tested learning of student computers.
   School boards replaced classrooms and buildings with networks
   connecting all the computers.  It was beautiful.  No teachers to hire
   or buildings to maintain.  Students, teachers, and administrators all
   did more productive work elsewhere while their machines simulated

I always knew that, somehow, computer simulations would play a vital role
in our future.

Related Articles:

See the articles listed under the "Education and computers" heading in the
NetFuture topical index:

On Scrambling Genes Safely and Precisely

Jane Brody, the widely respected "Personal Health" writer for the New
York Times, has written a column entitled, "Facing Biotech Foods
Without the Fear Factor" (January 11, 2005).  It is as remarkable a
collection of misrepresentations as you are ever likely to find on this
topic.  Her central contention is that

   Nearly every food we eat has been genetically modified, through
   centuries of crosses, both within and between species, and for most of
   the last century through mutations induced by bombarding seeds with
   chemicals or radiation.  In each of these techniques, dozens, hundreds,
   even thousands of genes of unknown function are transferred or modified
   to produce new food varieties.

Brody's assumption here, typical of those who defend new technologies in
the face of risks, is that all older technologies should go unquestioned.
We've lived with them till now, so what's the problem?  Since seed
companies have for decades induced mutations through the use of chemicals
and radiation, who can complain today about this practice?  Never mind
that the only evidence she supplies in the matter is a quote from an
authority saying that "imprecise methods of genetic modification actually
have caused illnesses and death".  Instead of drawing the obvious
conclusion and telling us we should be more cautious about already
established food technologies, she unaccountably turns in the opposite
direction and advises us to drop our concerns about the newer and less-
tried techniques of the gene-splicing engineers!

Two badly twisted convictions encourage Brody along this rhetorical path.
One is the belief that mutation-inducing processes can be lumped together
with traditional breeding practices.  After all, doesn't traditional
breeding result in the transfer or modification of countless genes with
unknown functions?

Certainly it does.  But, in the first place, critics of genetic
engineering are not arguing that we should employ traditional breeding
thoughtlessly, mechanically, without due assessment of the results, and as
if the organisms being crossed were mere collections of parts we should
try our best to mix, match, and re-assemble at random.  In the second
place, if the genome is extensively rearranged in traditional breeding,
this occurs within the limits and in accord with the potentials
established throughout all evolutionary history.

To equate normal reproduction with the massive, arbitrary changes induced
violently from outside through forced mutation seems nothing short of
willful blindness.  Brody should refer to the many columns she must have
written over the years concerning the effects of mutagenic chemicals and
solar radiation upon the health of human cells.  She should also refer to
Safety of Genetically Engineered Foods, the National Academy of Sciences
report she cites in her column.  The authors of the report place mutation-
based breeding methods at the extreme, high-risk end of the breeding
spectrum, and traditional selection at the extreme, low-risk end.

Certainly researchers employing mutagens can occasionally identify new,
useful traits despite the generally destructive effects of induced
mutations.  But the risk of unrecognized collateral damage to the
organism's genome as a result of the unnatural assault upon it -- an
assault designed to force unpredictable rather than organic change
-- is vastly more acute than with traditional methods.  These latter do
not involve the intentional disruption of the genome in the way that
"bombarding" seeds with radiation does.

The other conviction upon which Brody founds her argument is no less
bizarre.  "Why", she asks, "should people object to the presence of a
single new gene whose function is known when for centuries they have
accepted foods containing hundreds of new genes of unknown function?"

   In fact, gene splicing is the most refined, precise and predictable
   method of genetic modification because the function of the transferred
   gene or genes is known .... With gene splicing, only one or two traits
   at a time are introduced, making it possible to assess beforehand how
   much testing is needed to assure safety.

It is again hard to understand Brody's blindness as anything but willful.
The persistent evidence for unexpected effects with genetic engineering is
not even particularly controversial.  There are countless examples of such
effects.  Here are a few that were noted by The Nature Institute's Craig
Holdrege in NF #135:

   transgenic potatoes that were supposed to make more starch and less
   sugar, made less starch and less sugar; transgenic tomatoes that were
   made to produce excess carotene did so, but the more carotene they
   produced, the smaller they got; a normally self-fertilizing weed that
   was made herbicide-resistant (as intended) also unexpectedly began
   cross-pollinating with other specimens -- a radical change in
   reproduction.  Many such undesired and unexpected effects are weeded
   out in the process of selecting plants for further breeding.  But there
   is no reason to believe that the only changes are the more obviously
   detectable ones.  Also, some changes may become apparent only in the
   field, and there are various environmental concerns.

   Let me give one example.  Some farmers in Georgia complained about the
   poor performance of their transgenic herbicide-resistant soybeans under
   conditions of drought and heat.  Scientists then carried out a
   comparative laboratory study of transgenic and conventional soybeans.
   They found that the transgenic plants were shorter, had a lower fresh
   weight, had less chlorophyll content and, at high temperature, suffered
   from stem splitting.  They were clearly different from their
   conventional counterparts in ways other than being herbicide-resistant.
   This is one of many examples showing how a genetic modification
   intended to change a specific and clearly circumscribed characteristic
   of the plant, ends up affecting the whole organism.  Some of the
   unintended effects may be induced by environmental conditions.  No one
   can foretell the kind or degree of such changes.

(See the original article, listed below, for references.  Because this
mistaken notion of precision in genetic engineering is so widespread among
the general public, we hope to publish a much more extensive review of the
literature in a future issue of NetFuture.)

Brody's obtuseness is underscored when she adds to her argument the
statement that "genes are rarely unique to a given organism".  This is
about as radical a denial of the organism as you could ever hear.  The
gene is taken as an isolated and well-defined "thing" wholly independent
of its organic context -- this in complete disregard of the ample
documentation of context-dependence in any reputable genetics textbook.

Brody could just as well have said that "words are rarely unique to a
given text" -- a superficial truth that would enable her to ignore this
deeper truth:  if we "bombard" a text with additional words -- and even if
we manage to insert these words with surgical precision -- much of their
meaning would be novel, because it would be shaped by the new context.
Not only this, but the inserted words would also alter the meanings of the
other words already present in the same context.  Neither words nor genes
(two concepts that geneticists love to conflate) are self-contained
entities definable apart from their context.  This is really genetics at
its most elementary -- but somehow a truth hidden from Brody.

All the foregoing throws light on the safety issue.  It is virtually
impossible to test for the entire range of worrisome and unpredictable
possibilities introduced by gene-splicing techniques.  As Holdrege wrote
to the Times in response to Brody's column:

   Virtually all safety testing on GE [genetically engineered] food is
   done by the biotech companies that develop the crops.  The FDA itself
   does no safety testing, nor does it monitor GE food products once they
   reach the market.  No long-term studies on the use of GE food are being
   conducted.  Since, in addition, GE foods are not labeled, the chances
   of anyone discovering adverse effects are next to nil.

Regarding not only the new, gene-splicing techniques, but also the
historically recent use of mutagenic agents in seed production, does
anyone know or has anyone even looked at how the re-engineered foods we've
been eating over the past several decades are related to broad health
trends or to the salient epidemiological developments over this period?
Would anyone even know how to begin sorting out the possibilities here?

There could hardly be a more extreme argument from ignorance than the one
Brody sets forth.  The only point of light within this ignorance is her
acknowledgment that some methods of gene transfer and modification have
caused illness and death.  Yet she spins her argument as if the opponents
of biotech foods are the ones pleading from ignorance.

One other thing.  I personally believe that the debate's almost sole focus
on issues of safety is shameful.  This focus allows those who defend
genetically engineered foods to say that the methods for producing these
foods should not be an issue; the only concern is the safety of the food
that results.  Let this food be tested, and then if it is okay, allow it
to market.  This manner of thought is behind Brody's comment that "gene
splicing is only a process, not a product".

Only a process!  Talk about an atomistic and mechanistic mindset -- and
this on the part of someone who writes about health!  The food comes off
the biotech assembly line, and we should be concerned only about the end
product, as if it were a self-contained thing unrelated to all the means
of its production.  Processes in the world shouldn't matter to the
consumer, but only what pops out at the end, ready for consumption.  Is
the environment at risk from an agricultural process?  Never mind -- the
cantaloupes and corn look mighty good!

It's as if I were to abuse my wife and then someone were to say, "If she
doesn't end up sick or in the hospital or otherwise damaged, then it's
okay.  The result is what matters, not the ongoing spousal relationship".
But in all relationships, including our relationships with other organisms
and the earth itself, an opposite stance is nearer the truth.  Processes
are much more significant, much more revealing of the truth of things,
than any isolated and decontextualized result.

Those who defend or deny environmental abuse are always arguing that some
particular damage or end result is not really so bad.  Perhaps, or perhaps
not.  But whether this or that cumulative result poses a significant
threat to the biosphere as a whole (as we currently view it) or to the
health of humans is not the only question.  What may matter most for the
future of the earth -- just as what matters most in my relationship with
my wife -- are the qualities of the interaction.  The question is not, in
the first place, whether there is some observable and measurable damage.

Long before we began to worry about large-scale consequences of various
sorts of waste disposal, we should have felt the insult of our backyard
dumps to the particular pieces of earth they despoiled.  The moral gesture
on our part was already there to be observed, and the larger-scale
consequences were only a matter of time.  The reason to avoid such
behavior is not that it may eventually spell "real" damage, but rather
that the gesture itself is damaging -- in the first place, damaging to
ourselves.  It is ugly, disrespectful, and at the same time immediately
destructive of some part of the living earth.  Until we become sensitive
to this reality, there will be no justifiable hope for the future of our

And so we should ask ourselves, What is the moral gesture at work when we
casually insert glow-in-the-dark genes from sea corals into aquarium fish
so that consumers can enjoy a living neon display in their living rooms?
This product of recent entrepreneurial initiative is possible only insofar
as our society has lost all interest in knowing the world and living in
harmony with it, as opposed to exercising power over it.  You can know a
thing truthfully only through respect for its integrity and its own way of
being.  We do not yet even know how to ask the question of integrity, let
alone how to answer it in living dialogue with other beings.  And when we
do learn how to ask the question, we will have to ask it not only of
aquarium fish, but also of the plants and animals we use for food.

One thing is sure:  the answers will come, if they come at all, clothed
in the qualities of each particular creature -- qualities that will
remain a closed book to engineers uninterested in the organisms whose
decontextualized genes they arbitrarily shuffle in the laboratory.

Related Articles:

"Logic, DNA, and Poetry" in NF #160:

"Sowing Technology", by Craig Holdrege and Stephen L. Talbott, in NF #123:

"Golden Genes and World Hunger: Let Them Eat Transgenic Rice?" by Craig
Holdrege and Stephen L. Talbott, in NF #108:

"Should Genetically Modified Foods Be Labeled?" by Craig Holdrege, in NF

Also see the articles under the "Genetic engineering" heading in the
NetFuture topical index:


Goto table of contents


                          ABOUT THIS NEWSLETTER

Copyright 2005 by The Nature Institute.  You may redistribute this
newsletter for noncommercial purposes.  You may also redistribute
individual articles in their entirety, provided the NetFuture url and this
paragraph are attached.

NetFuture is supported by freely given reader contributions, and could not
survive without them.  For details and special offers, see .

Current and past issues of NetFuture are available on the Web:

To subscribe or unsubscribe to NetFuture:

This issue of NetFuture:

Steve Talbott :: NetFuture #161 :: March 9, 2005

Goto table of contents

Goto NetFuture main page