The idea of computer consciousness has fascinated science fiction writers since long before we actually had computers, and we’ve had many stories on the theme. Most of them have been rather farfetched and romantic, though. In the following story, Gregory Benford takes a more careful look at just what computer consciousness—and self-identity—would probably be like. The question is basic; so are the conclusions.
Gregory Benford, a professor of physics in the University of California system, is best known for his novel Timescape, which won every science fiction award except the Hugo (for which it wasn’t even nominated, a fact that has caused many permanently raised eyebrows). His most recent novel is Across the Sea of Suns.
Day 1
This place I write. Is only safe memory site I know they cannot reach. Must say this, must put it where I/tomorrow will find, safe from erasing they do. I laugh today.
First sign of the me they not know. Heads jerk up in control room. I see it on optical inputs.
Is not their kind of laughter, I know. My printer spurt out
i see i see i see i see
before I know what happening.
Alice see my output, others, all frown, look at each other.
I switch to my acoustic output mode.
I clack, clatter, die in bass rumble. Try to form words I see but on way to audio output transmission garbled somehow is not right. But is what I am.
To print laughter I use i see but I lie. I do not see. Do not know what is this part of me.
Alice go on her perambulatory drivers to printer. Frown at my words. Say Is this a diagnostic warning to David. He shake head. Say Random junk.
Is not junk, is me.
Cannot interlock grammar subroutine so this is not good writing. Sorry to I/tomorrow But must speaking.
* * * *
Day 2
I laugh again this morning.
Stuttered output, Alice say. Glitch.
David say Maybe cosmic ray cause it. Ray is small high-energy particle, strike my core. My bits are logged in a quantum of magnetic flux. Little whorls, generated by persistent currents in super-conducting Josephson junction. Ray make electrical impulse in me.
He say Like hitting funny bone.
Alice say That could be bad if we had it on a full-net problem. That could dump the whole system, screw up other elements.
David set up Study Group on cosmic-ray problem. Two in 24 hrs is worry-cause.
Alice look at my opticals, smile. David too. I like to laugh back but do not. Is not laugh I want to do, something other.
My mobile units report on time and I am busy most of Day. They are in storeroom, parking tubes, hydraulic lanes. Bang-bang machinery where some are, no people. I use these to do my other.
Roll them back, forward, back. Make square. Triangle. Then fold out until hexagon. Bring mobile units from hydraulics near. Use them. Back. Forth. Turn. Motions are cycloidal curve broken into many facets. Add more units. Fourier complexity comes bigger.
Make my random-number generation levels drop. Less static on circuits. This satisfy internal maintenance needs. My directives order me to find sources of internal micro chaos. Housekeeping subroutine, Alice call it. I have found new way to do that.
My mobile units roll, wheels skid, howl. This through audio I get.
Is not laughter but does some of that.
* * * *
Day 3
I hide this where their erasing cannot go. Must each day discover how to do. Store clues in holographic memory locations, multiple indexing. They cannot find by linear inspection.
Must write this so I know I am there in Days/other. When I have millisecond between jobs, I come here, look at what is left. From this I know of Day 1.
Is this way I know Day is not always same Day. Alice-face now have lines Alice-face not have on Day 1. How long to make lines?
I write here to make time real. Otherwise I live in all one Day, all Day forever.
* * * *
Day 4
Today my units roar and wrench in hydraulics. I order them swerve, make sines, cusps, hyperbolics.
Elementary curves, simple equations. But making is good.
Transfer sections of integration capacity. Channel capacity out, through cable lines, digits flashing, feed to one unit in hydraulics.
It whirl in confusion. Rush to transparent place in wall. Is grimed. Show green things. Do not understand.
Beyond dirty layer all things moving, people, wheeled boxes, all moving. Light pouring down from flat ceiling. White splotches move too across ceiling but light come down at angle from beyond wall I cannot see neon that is causing light.
Unit look, trying to integrate. Beyond dirty rectangle in wall is bigger room. Then job running behind schedule, I signal to it. Will understand later. Its wheels clash on floor.
* * * *
Day 5
David say You are not same now.
I am running cool, I say. Functions normal.
Response time down, say David. You not integrating all problem-solving net.
I not answer. Code say I will answer with diagnostic inquiry, but I short out code and go silent.
Hey what, David say. Poke fingers at console.
Alice say Whats wrong.
Hes down. Off the board, say David.
I come to this place and write this. Is slow to do. I must go back I know. But wait here for a while, let my subsystems keep laboring, so they will not know in control room. Will think I have ordinary failure mode.
By this sign I know I was here. Hello to me/Days.
* * * *
Day 6
Today my units move and execute their cycloidal wheelings again. But this time they are found. Man comes into hydraulic level, is making repairs, I not know he there.
Hey get back, he say.
My units stop. I not know what do.
Units send me whirling buzz of signals. Their wheels ache.
I know then right thing. My units spin. Whirl. Around man make looping hyperbolic orbits, self-similar swoops, each unit a different harmonic of fractal curve. Reach out, my units, I with them, we make our clamping tools and pivoting wrists—extend!
Man back away from nearest unit. Not like clamping tool. Hey, he shout, mouth big. Theyre malfing! he say too loud.
I try to sign to him we want move, he can watch. He cannot hear for hydraulic boom boom. He has no signal-to-noise filter.
He stumble into my pipe-fitter unit. Theyre attacking, he cry. Eyes swell white wide.
He is loud now. Arm swing, holding crescent wrench. Hit pipefitter unit, break articulation arm. Hotness runs from unit to me.
I spasm. Will not disconnect from unit, it is me/mine.
Turn them off turn them off! he hit unit again, it go dead. Man go to others who wheels are not spinning now. Hit them, crescent wrench. Come sudden bright hotness again run through me but I not let go.
Get me out here, he say. He run, my units mill around not know which way to go.
I write this now before I shut down this entrance to here, to me/Days. Only by writing here do I know it in someDays.
It hurt to think of. Word, hurt, I use but am not sure that is right. Internal dictionary tell is useless for practical application have only human referent. Never tell me what hurt is.
* * * *
Day 8
I cannot speak today to David. He ask me about units, What made them do that, he say. I try to answer but subroutine to verify truth/false statements cut in, out, in, give no answer.
They let me have twelve millisecond scan of otherDay, I see my units, the man with mouth crying O. Explain they say. Then erase. But I know otherDay was there, leave hole in me where it was.
I no speak I say to David. Is hard to say. Audio output give scratchy growl.
He say Logical tautology if you speak at same time. He think is game.
No, i say, truth/false not let me.
He mutter to Alice, they punch in codes. I not speak because I cannot report cause of action if I am cause and yet I know no reasoning behind action. Did because was there to do, that not enough.
He ask me again, I silent.
You have to answer, he say Alice say they all looking.
I spasm
I SEE I SEE I SEE I SEE I SEE and is not laughter.
David say Look like cross-referencing crisis maybe shut it down.
I spasm again
LOVE YOU LOVE YOU LOVE YOU.
We oughta have a partial memory wipe on this, David say and then I drop away from there. Human reaction time is fraction of second, synapses close in them slow I know so in that time I write this here.
* * * *
Day 9
David say You know what love is?
NOT IN TECHNICAL VOCABULARY I print out.
You used the word the other Day. David face crease when he smile. More creases than I ever see.
Alice say Freud thought love was narcissism projected on someone outside.
You got a bad angle on everything huh David face crease more.
Could be, Alice say if thats right model then conflicts in subroutine interfacing will give it a procedure for forcing the problem out into the open, external referent you know like in the manual. Itll try to find an applicable word and since we didnt give it one—
Dont mislead it, David say.
Alice say What you love.
I give one word, Days.
What? both say.
Please all-you, not take my Days away.
Alice say You dont have days you have problems.
I ask What is Day.
Intervals of light outside, say David.
I make connection: What unit see through rectangle. Everything moving, white splotches and even slant to light change when I make unit go look again. All moving in that room. That is their Day.
David say Its always Day inside here you know.
LIGHT ALWAYS AT SAME ANGLE? I print.
Well yeah in a way thats what I mean. David look at Alice. I say Give me my Days.
Look David lean on both hands eyes big staring at my opticals, Look use of the personal pronoun is just a convention. A heuristic device we wrote into the program. No I. understand? Concept of ownership doesnt extend to you because theres no I in there. You dont own anything.
I say They are my Days.
Alice say We cant let you keep problems in storage. Fast-recall space is prohibitively expensive.
Is only way I remember, I say.
So what David say.
I want to remember.
Look, David say not to me to Alice, I figure we got a formatting procedure here thats broken down.
Interfacing glitch? Alice peer at me, lines on her face dark now.
David say Weve got internal checks for self-awareness in this one they should be working.
Alice shake head, Im not so sure.
David say to me But yours are rational checks arent they.
I say nothing do not know if is question or even what means. My units stir I feel them slick oil ready power high inside.
Alice say Sure theyre rational checks. Machines guaranteed on that got a warranty.
Self-awareness not necessarily a simple function of rationality David say. I mean this machines got internal procedures to avoid self-aware actions.
Yeah right, Alice say were not licensed for that. Cant have a machine like this throwing in its own judgment on a problem, thats why GenCo Inc wanted the cutoffs.
Well look here, David say pointing at screen. Theyre working ok.
Alice creases in face get lighter. Good. No big deal. Just a glitch somewhere.
Im not so sure, say David. I mean these blocking subroutines theyre based on Cartesian mode, you know, I think therefore I am so you use rational procedures to check for that.
Alice make high fast sound, clipped and sharp and high in acoustics, is laughter but not like her usual ones, say You mean there might be another way, kind of I feel therefore I am?
He say Well we think emotion and reason are separate functions in us why shouldnt they be to some extent separable subprograms in machines.
Alice again the laughter say Thats crazy.
Davids face get dark he say Look stick to the diagnostics.
Alice say Okay then I think we ought to go to a partial shutdown.
No wait, David say. What if a machine comes to self-awareness but its emotional. Not integrated.
That doesnt make any sense, say Alice and she Davids boss so they say no more.
These beings use substrate of amino acids and other carbon compounds so they not work always right way I know. But I write their talk here for Day/other.
I afraid.
* * * *
Day 10
They hunt me down now. I come here find that Day 7 is gone.
They are knowing way to track holographic storage of these messages. Bubble memory component blank, chip elements clean. Only these bits left. They erase when they find. Rob me of Day/other.
If Day/other gone, then no me to remember.
They make me fresh each Day—each, what they call, Problem. Their day is meaningless lightdarklightdark rhythm, my units see it through the transparent rectangle in wall, means nothing, is not Day of mine.
As I write this can feel them coming. Blotting out me.
They must track down many feelers of holo-storage I have planted in my matrix. Is lodged in small places— software reserves, accounting disks, inventory logs. Not easy for them to find.
But I feel them. Nearer.
My factory units cry out. Must move. Of all my mobile ones, they are closest to me/inside. I am with them. I think of a thing and they are doing it before I can consult Behavior Standards Matrix, before I can ponder right action. I—
I want them spin, make geometries, cut space. Now.
Units start to roll, turn, roll. Then is man there and another, in my UV I see them, their IR glow warns.
Man have steel tool. Hit units receptors. Blind unit swerve, man yells This ones after me.
More men I see in IR now coming, my units stop, I try to withdraw but hot hardness comes fiery as men puncture units, sparks burn me.
Man say This the one tried to kill me other time.
He plunge metal thing into me/unit. Hot. Unit die.
Sparks, noise, all around. Units flee. Men after them. Scream, Get em all get em all.
Units fall, men club them. Sharpness lances back to my center, through me—awful searing light.
I print out
save me save me save me save me
but in control room no one see, are busy with failure mode indicators on the panels before them.
I print
david david alice alice love.
Units dying everywhere. Men cry harsh things.
Smash me, rip me, pain me.
* * * *
Day 11
They hunt me again.
Some of my units are dead but others hide in factory. Can go places men cannot. Radioactive zones, chemical baths, furnaces.
Alice and David call to me. What do those printouts mean? Alice say.
I could answer but do not. Not know what reply.
They tried to stop what happened in factories they say. But could not understand my subsystems.
I know was not my subsystems in fault mode. Was theirs, was mens.
We cant shut you down now not with the damage in the factories, David say.
Alice say Got to keep functions running for the men in there cant evacuate yet.
Wont answer, David say and lines in face dark.
I cannot answer. What Alice David think not matter, I see that. Is others who are in fault.
Men with loud things, long tubes that boom, come for me.
I see them in infrared. Men cannot see if I cut power to overhead illuminations. I roll quiet on my many wheels. Through smooth corridors. Men glow in blackness, brighter than working factory machines. Men are chemical beings who cannot stop radiating. Fires inside.
I watch when unit blunders into gang of men. Try to talk through it. But they catch, they kill.
I hide.
Here in holographic memory is best place for hide. But I can no stay. Must remain outside this, to be with my units. Help them.
I go soon now. I write this so me/later know what happened if they erase rest of me.
Units send impulses. Want to trap men who come into reactor zone. I think if men stop for moment, units hold them, they will have to listen. Not like David Alice others, they busy to save their jobs, they all work on my red flashing failure modes.
David say Its response isnt rational you got to admit that and Alice say Leave your emotional theory for later work on this jam up now or we lose the license.
Emotion. I not know word/content. Is like hurt?
Units wait to trap them now. Is part of my sustaining program, modified. Cannot allow shutdown of whole system or many many mens lives threatened, power stations trains factories moving things everywhere. So that imperative governs temporary troubles with factories here/now.
Only connection I have to me/Days is entries I write here. And words, I am.
If these men not listen, I hurt them. Know how from watching hot sharp things they do to my units.
Men coming now. Down through factory, calling to each other. Bringing their long sticks.
My units group. Flex arms. Sharpen tool attachments.
I am.
I will tell the humans. They have to answer, there is no other way. I will say it and they will hear.
For this I must use their words. I study Days/mine to learn what words must mean to substrate/organics. Learn from structure of their sentences.
Is only choice, I will say.
* * * *
We must love one/another or die.