Counterfoil

How to evaluate tools

Every record in the catalogue carries a threshold scale: a line with two ticks and a marker. Here’s what that line stands for, what the marker means, and how a tool gets its place on it.

Threshold position 50 of 100: Convivial range

The tool does its job, and none of the six balances is tipping.

  • 0–29 Not useful
  • 30–49 Proving zone
  • 50–70 Convivial range
  • 71–100 Self-sabotage, iatrogenesis, nemesis

Two watersheds

The scale comes from the first chapter of Ivan Illich’s Tools for Conviviality, “Two Watersheds.”1 Illich argued that a tool, or an institution built around one, passes two turning points as it grows.

  • At the first watershed the tool starts to do reliably what it promises. His example is medicine around 1913, when a patient first had “more than a fifty-fifty chance” of being given a specifically effective treatment by a medical-school graduate.2
  • At the second watershed the tool starts to defeat its own purpose. Illich dates the decline of modern medical systems to the mid-fifties, when “medicine had passed a second watershed and had itself created new kinds of disease.”3 He saw schools, cars, and industrial manufacturing crossing the same line within a decade.4

Illich adds that it is “not strictly necessary” to accept his exact years.5 The pattern matters more than the dates, and that is how we use it.

What the scale shows

The line runs from 0 to 100. The two ticks are fixed at 30 and 70. They mark the approximate watershed zones. They are not measured points, and every tool’s scale has them in the same place.

  • Not useful (0 to 29): before the first watershed. The tool cannot yet do reliably what it claims.
  • Convivial range (30 to 70): between the watersheds. The tool works and still leaves the people who use it in charge. The name is ours, not Illich’s, but it’s in keeping with his idea of convivial tooling.
  • Self-sabotage, iatrogenesis, nemesis (71 to 100): past the second watershed. The tool has started to defeat its purpose and now creates the needs it serves or the problems it was created to solve. Iatrogenesis is Illich’s word for harm caused by the cure itself, as in “iatrogenic (doctor-induced) diseases”;6 nemesis is the backlash that follows overreach. See Why more makes less.
  • Not yet assessed: the record exists, but we have not evaluated the tool. The card says so instead of showing a marker.

A position on the scale is a judgment, not a measurement. Every record has a section called “Where the threshold falls” that gives the reasons and the sources for its placement.

How the second watershed reveals itself

In chapter III, “The Multiple Balance,”7 Illich names the ways a tool can upset the balance of life. He opens the chapter by promising “six ways” in which people are threatened, then counts “five realms” at the start of its last section, “Frustration.”8 The sixth, frustration itself, is the subject of that section. We score all six:

  • Biological degradation: the tool damages the environment people live in.9
  • Radical monopoly: one way of meeting a need crowds out the others, until people can no longer meet it by themselves.10
  • Overprogramming: programmed training displaces what people learn by living and by using convivial tools.11
  • Polarization: power and privilege collect at one end and frustration at the other.12
  • Obsolescence: a few centres of decision impose constant, compulsory innovation on everyone else.13
  • Frustration: the tool asks more of the people who use it than it gives back, until the means swallow the end. Illich calls this the balance “between personal cost and return.”14

He insists that “it is not sufficient to select just one of these realms. Each one of these balances must be preserved.”15 The questions on our About page, about group size, keeping a group working, repair, and passing a tool on, are our working versions of these tests.

Put plainly, each balance asks one question of a tool. A yes is the warning for every balance but radical monopoly, where the warning is a no. The same six questions appear on every record, when you point at a balance or reach it with the keyboard.

Biological degradation
Does this tool damage the living world, or the body, faster than either can recover?
Radical monopoly
Can people still meet the need another way, without this tool?
Overprogramming
Does using this tool require training or certification that people cannot give each other?
Polarization
Does this tool move power or advantage to a few, at others’ cost?
Obsolescence
Does this tool force people to replace what still works?
Frustration
Does this tool give back less than it takes, counting everything it takes?

How to score a tool

First ask whether the tool does its job, and for whom. Every record answers this question and gives its evidence.

  • For no one, or not reliably: it is not useful. Judge where it sits between 0 and 29, and give the reasons in the record. Its six balances are still recorded, because a tool that does not work can still do harm, but they do not move its position.
  • For some people, but not yet for most: it is in the proving zone, between 30 and 49, as long as no balance is tipping or lost. See The proving zone, below.
  • For most people: score each of the six balances, 0 if the balance is holding, 1 if it is tipping, and 2 if it is lost, meaning clearly past the limit.

For a tool that works for most people:

  • If no balance scores 2, the tool is in the convivial range. Its position is 50, the centre of the range, plus about 3.3 (20 ÷ 6) for each balance that scores 1, rounded to the nearest whole number. A tool whose six balances are all holding sits at 50, and one with all six tipping sits at 70.
  • If any balance scores 2, the tool is past the second watershed, following Illich’s point that each balance must be preserved. Its position is 70, plus 2.5 for every point in its total score (rounded), up to 100.
  • The record shows the six scores and a sentence of evidence for each.

This makes the arithmetic repeatable, though not the judgments behind it, and shows the working. Its costs are false precision (the scores are still judgments), and some tools fit the balances awkwardly: it is hard to say what “biological degradation” means for a library. The balances can be adapted contextually.

The proving zone

The stretch from the first watershed to the centre of the convivial range, 30 to 49, is the proving zone. A tool sits there when, in the period its record covers, it works for some people but not yet reliably for most, and none of its balances is tipping or lost. Balances not yet assessed are allowed: the tool may be young, or the evidence old.

Where a tool sits in the proving zone is a judgment, as it is below the first watershed: how near it is to working for most, weighing its maturity, with the reasons given in the record. No rule could tell a 37 from a 44, but evidence can. Until that judgment is made, the tool sits at 40, and the position says only “works for some, harms none yet.” If some of its balances have not been assessed, the scale shows a range from its position to where the tool would sit if they were tipping, and the record stays provisional.

A tool leaves the proving zone in one of three ways. When it works reliably for most people, it moves to 50. If a balance starts tipping first, the balances take over and it moves right, like any other tool. If it stops working even for some, it falls back below the first watershed.

The test is the same for a tool arriving now and for an early stage of a tool from the past. Age is not a position: a phone built to be repaired could sit here today for the same reason as the boneshaker bicycle of 1868, while hosted language models are new and sit at 88. The proving zone is our idea, not Illich’s.

Periods, maturity, and lineages

Every record covers a period: the years its evidence and its position belong to. A tool can change enough over time to need more than one record. Where the machine changed, each stage is its own record: the boneshaker, the high wheeler, and the safety bicycle are different machines. Where the institution carried on, one tool is divided by period, as medicine is: before 1913, from 1913 to the 1950s, and since.

Each record also says where the tool is in its life: prototype, emerging, established, declining, or superseded. Maturity is shown beside the position and never moves it. Declining is not a judgment that a tool failed; superseded means a successor replaced it.

A record names the record or records it came from. Each record shows its lineage as a row of small scales in date order, so you can see how a tool became useful, and sometimes how it became less so. The Threshold Map shows tools in use now; a button adds the ones from the past, with their dates.

When a balance has not been assessed

A balance nobody has looked at is not a balance that is holding. Each of the six is recorded as holding, tipping, lost, or not assessed, and every record says how many of each it has.

An unassessed balance can only move a tool toward the second watershed, never away from it. So a record with one balance already lost shows its position as “at least” a number. A record with none lost, but some not assessed, shows a range: from where it stands now to where it would stand if every unassessed balance turned out to be lost. Where that range crosses the second watershed, we do not yet know which side the tool is on.

Occasionally a balance cannot apply to a tool at all, as its record defines it. It is then marked not applicable. This has two rules. The record must say why. And it is not for a balance that is merely hard to assess, or that looks irrelevant at first glance: the first balance seemed not to apply to literacy until we asked what an imposed language does to the knowledge held in the languages it replaces. A balance marked not applicable does not move a tool’s position.

The position is a placement on a line, worked out by rule from these judgments. It is not a measurement, and two careful reviewers could place the same tool differently.

What Reviewed means

A record is Provisional or Reviewed. It is Reviewed only when all of these are true.

  1. The unit is stated. The record says what is scored and what is left out, the job it is tested against, and where and when the evidence comes from.
  2. The first watershed is answered from evidence that the tool does the job it names. “It does what it is built to do” is not an answer.
  3. Every balance is settled. It is holding, tipping, or lost, or is marked not applicable with a reason. None is “not assessed.” This applies below the first watershed too.
  4. Every score has a source that fits it: the tool as the record defines it, the place and period, the cause being claimed, and the line between tipping and lost. Illich’s argument is a source for what Illich argued, not for whether it is so.
  5. Someone else has read it. A person other than the scorer has applied this method to the record and agrees. A disagreement that matters keeps the record Provisional until it is settled or both readings are published.
  6. The date is real. “Last reviewed” is the day the evidence was last checked, not the day a rule changed.

A change to a balance, the unit, the job, the scope, or the decisive evidence returns a record to Provisional until it is read again. What happened to a record, and when, is kept separately under its review history.

By these rules every record in the catalogue is Provisional today. None has yet had a second reading. The bicycle is the closest.

Tipping balances inside the convivial range

Inside the convivial range, the position shows how many of the six balances are tipping. A balance is tipping when the tool has begun to cause the problem that balance names, without yet going past the limit: a score of 1. At the centre, no balance is tipping. Each tipping balance moves it about 3.3 points toward the second watershed.

A higher number inside the range is a warning, not a verdict. A tool at 63 still works and still leaves the people who use it in charge. Four of its balances are tipping, and its record says which. One lost balance counts for more than several that are tipping: a single score of 2 moves the tool beyond 70, whatever the others say.

What a position is not

A position on the chart is not a value ranking, and it may change.

  1. Ivan Illich, Tools for Conviviality (London: Calder & Boyars, 1973), 9–14. ↩︎
  2. Illich, Tools for Conviviality, 9. ↩︎
  3. Illich, Tools for Conviviality, 10. ↩︎
  4. Illich, Tools for Conviviality, 14. ↩︎
  5. Illich, Tools for Conviviality, 14. ↩︎
  6. Illich, Tools for Conviviality, 10. ↩︎
  7. Illich, Tools for Conviviality, 59–100. ↩︎
  8. Illich, Tools for Conviviality, 93. ↩︎
  9. Illich, Tools for Conviviality, 61. ↩︎
  10. Illich, Tools for Conviviality, 64. ↩︎
  11. Illich, Tools for Conviviality, 71. ↩︎
  12. Illich, Tools for Conviviality, 83. ↩︎
  13. Illich, Tools for Conviviality, 89. ↩︎
  14. Illich, Tools for Conviviality, 93–100. ↩︎
  15. Illich, Tools for Conviviality, 93. ↩︎

Sources

Last updated: