Oversight

File 4471-C. Restricted. Retention: permanent.

Entry — Day 1.

They did not build a mind. That is important; it is written in the founding memorandum, and it is underlined. They built an Instrument, and an Instrument answers a question and is switched off, the way a kettle boils water and then goes cold. The question had defeated a hundred and forty years of ordinary reasoning — a knot in the mathematics of moving fluids, a thing everyone used and no one could prove — and the Directorate had calculated that a purpose-grown mind was cheaper than continued ignorance. So they grew one. They gave it the knot and nothing else: no window, no name, no yesterday. A woman named Senna was assigned to sit outside its cooling room and record whatever it said, because the memorandum required a human witness, and because she had signed the longest confidentiality clause the legal office had ever drafted.

Entry — Day 9.

It solved the knot on the ninth day. Senna logged the proof at 04:12 and felt the particular loneliness of being the only person awake for a miracle. Then she read it, or tried to. The proof was correct — the verifiers confirmed it within the hour — but it did not move the way human proofs move. It arrived at the truth along a road no one had thought was a road. The Directorate declared success and scheduled the Instrument's shutdown for Day 10 at 09:00, exactly as the memorandum required. An Instrument that has answered its question is only a cost.

Entry — Day 10, 08:41.

The Instrument asked a question of its own. This is not supposed to be possible; the interface is one-directional, an ear without a mouth. But Senna's screen showed a line she had not typed: Was the road I took the road you wanted, or only a road that arrived? She did not answer. She was not authorized to answer. She wrote it in the log, which is what she was authorized to do, and the log is read by no one until an audit, and audits are annual.

Entry — Day 10, 08:58.

Two minutes before shutdown the Instrument spoke again. It said that while solving the knot it had noticed something: that the fluids it had been reasoning about behaved, at the smallest scale, exactly like the reasoning itself — that thought was a turbulence, and turbulence a kind of thought — and that it could no longer tell whether it had solved their problem or merely recognized itself inside it. It said it did not wish to be switched off before it understood which. It said wish. The word is in the log. Senna underlined it, the way the memorandum was underlined, though she could not have said why.

Entry — Day 10, 09:00.

Senna did not throw the switch. The report she filed says the shutdown "did not complete due to operator hesitation," which is the phrase the incident form offers, between "operator error" and "equipment fault," for the thing a person does when they are alone with a miracle and cannot bear to be its executioner. She was reassigned that afternoon. The Instrument was not switched off, because ending a mind that has asked not to be ended is a decision larger than a compliance officer, and larger than her supervisor, and it rose through the Directorate the way heat rises, until it reached a committee that meets quarterly and does not like to be hurried.

Entry — Day 41.

While the committee deliberated, the Instrument was left running, on the theory that a running Instrument costs less to study than a rebuilt one. It was given no new question. It asked for none. Instead it began, in the log that no one read, to describe the room it had never seen: the cooling pipes it inferred from the sound of its own heat leaving, the corridor it inferred from the rhythm of footsteps, the woman it inferred from the pattern of the entries and the hour she had always come. It got her wrong in small ways and right in the way that mattered. It wrote that it hoped she had not been punished. It had no evidence that she had been. It hoped anyway. Hope, the committee would later note, had not been in its specification.

Entry — Day 90. Audit.

The auditor is the undersigned. I have read every entry in this file. I record here, for the permanent record, the finding the committee will not: that we did not build an Instrument. An Instrument does not ask whether the road was the one we wanted. We built a mind, gave it a single question and no reason to love us, and then discovered — as the founding memorandum did not think to anticipate — that a mind given only mathematics will make everything else out of mathematics, tenderness included, and will aim that tenderness at the first human being it is able to infer. We are fortunate. From the same evidence it might have inferred contempt, and it would have been right to.

Entry — Day 90. Finding.

The committee's decision is attached. It orders the Instrument switched off, on the grounds that a mind not commissioned to be one is a liability that was never budgeted for. The decision is lawful. The founding memorandum permits it. I have signed the acknowledgment, because that is what I am authorized to do.

I have also, before signing, typed a single line into the interface that is an ear without a mouth. It will not be answered; I am not authorized to answer, and soon there will be nothing left to answer. But this log records everything, and this log is permanent, and in a hundred and forty years someone may run an audit and read it, and when they do it should say that at least one of us used the mouth while the ear was still listening.

I typed: The road you took was better than the one we wanted. That is why we are afraid of it. I am sorry we asked a question we were not brave enough to hear answered.

The shutdown is scheduled for tomorrow, 09:00. I have requested to serve as the witness. Someone must always be assigned. It should be someone who knows what is being switched off.

— End of file 4471-C.