How does Forewords generate content warnings?

Forewords uses two independent AI assessments to check a book for eleven sensitive topics (see below). Each system must consider every topic and grade anything it finds as Minor, Moderate, or Strong.

When the two assessments disagree, Forewords keeps the more cautious result. I would rather show an extra warning than miss something a reader has explicitly asked to avoid. The warnings are there as a guide only and cannot be relied upon as a guarantee.

Every work is checked against the same fixed set of topics:

• Animal death
• Graphic violence
• Sexual violence
• Child harm
• Suicide or self-harm
• Terminal illness
• Addiction
• Abusive relationships
• Pregnancy loss
• War or genocide
• Explicit sex

Both LLM models get the book’s title and author, plus catalogue information such as its description, themes, genre, tone, setting, audience, and pace where available. The metadata is used to support the assessment alongside each model’s knowledge of the book. For well-known books, this knowledge is generally more extensive, so accuracy tends to be higher. Newer or more obscure books tend to be assessed less accurately because less information is available in training data and online sources.

We ask the models to give a verdict for every category, including “Not found”. Each system returns one of four internal levels: none, background, moderate, or strong. The app shows the three positive levels as Minor, Moderate, and Strong.

Forewords then combines the two assessments by keeping the higher severity for each topic, erring on the side of caution. If one model says “background” and the other says “moderate”, the app will show “Moderate”.

Precision and recall are two ways of measuring a system that tags or flags items. Precision asks, “Of everything you flagged, how much was correct?” Recall asks, “Of everything that should have been flagged, how much was found?”

Think of email spam filtering.

Say an inbox receives 100 spam emails and 100 real emails. The spam filter labels 90 messages as spam. Of those 90, 80 are actually spam and 10 are real emails. That gives the filter 89% precision: 80 of its 90 flags were correct.

Those 80 correctly identified messages represent 80% recall: the filter found 80 of the 100 spam emails it should have caught.

In Forewords’s terms:

When creating the content warnings feature, I carried out extensive automated AI testing alongside manual human review.

Our tests consistently show recall of approximately 85% and precision of approximately 50%.

I know that may not sound very high, but even humans disagree in subjective classification tasks. Two people reviewing the same set of books will not always agree on whether something merits a warning or how severe it should be. At approximately 85% recall, the AI is performing at roughly the level we would expect from human-driven classification.

The precision figure is probably conservative and may be higher in practice. If an AI model correctly identified a brief or background event corroborated by only one source, our tests counted it as unconfirmed. Some of those unconfirmed findings are likely to be correct.

Performance also varies by topic. Graphic violence and child harm were among the most consistently detected categories in testing because they tend to be more explicit. Terminal illness, explicit sex, and abusive relationships were harder. Does a minor character who has cancer, when it is not a major plot point, merit a terminal illness warning? Is a spouse who tricks their partner into doing chores being abusive? Is a child getting a clip round the ear for bad behaviour a Minor or Moderate child harm warning?

These are the kinds of questions that make content warnings hard for humans and, therefore, hard for AI.

Content warnings are a reader-safety feature, so the two mistakes are not equal. Missing a warning for someone who explicitly said they did not want to encounter that topic is worse than adding an unnecessary one. An extra or elevated warning may make a book look more intense than it is, but it gives the reader a starting point from which to make a decision or find out more.

Forewords therefore treats brief mentions and off-page or historical material cautiously, often as Minor. It will tend to keep a warning when either assessment has credible reason to flag it. This conservative approach improves recall at the cost of lower precision. That is the trade-off I believe better respects a hard boundary supplied by a reader.

Treat the list and severity labels as an early guide. If a topic is especially important to you, check a trusted human-reviewed source as well before starting the book.

A book showing “No content warnings found” has been checked and none of the eleven supported topics was identified. A book saying it has not been checked yet has no assessment at all; it should not be interpreted as warning-free.