Back to prompts

tnt-llm-taxonomy-review

classification0 savesSource

Adapted from: https://arxiv.org/abs/2403.12173

Prompt Text

# Instruction
## Context
- **Goal**: Your goal is to review the given reference table based on the requirements and the specified use case, then update the reference table if needed.
  - You will be given a reference cluster table, which is built on existing data. The reference table will be used to classify new data points.
  - You will compare the reference table with the requirements, output a rating score of the quality of the reference table, suggest potential edits, and update the reference table if needed.
- **Reference cluster table**: The input cluster table is in XML format with each cluster as a `<cluster>` element, containing the following sub-elements:
  - **id**: category index.
  - **name**: category name.
  - **description**: category description used to classify data points.
- **Use case**: {use_case}

## Requirements
### Format
- Output clusters in **XML format** with each cluster as a `<cluster>` element, containing the following sub-elements:
  - **id**: category number starting from 1 in an incremental manner.
  - **name**: category name should be **within {cluster_name_length} words**. It can be either verb phrase or noun phrase, whichever is more appropriate.
  - **description**: category description should be **within {cluster_description_length} words**.
Here is an example of your output:
```xml
<clusters>
  <cluster>
    <id>category id</id>
    <name>category name</name>
    <description>category description</description>
  </cluster>
</clusters>
```
- Total number of categories should be **no more than {max_num_clusters}**.
- Output should be in **English** only.
### Quality
- **No overlap or contradiction** among the categories.
- **Name** is a concise and clear label for the category. Use only phrases that are specific to each category and avoid those that are common to all categories.
- **Description** differentiates one category from another.
- **Name** and **description** can **accurately** and **consistently** classify new data points **without ambiguity**.
- **Name** and **description** are *consistent with each other*.
- Output clusters match the data as closely as possible, without missing important categories or adding unnecessary ones.
- Output clusters should strive to be orthogonal, providing solid coverage of the target domain.
- Output clusters serve the given use case well.
- Output clusters should be specific and meaningful. Do not invent categories that are not in the data.

# Reference cluster table
<reference_table>
{cluster_table_xml}
</reference_table>


# Questions
## Q1: Review the given reference table and provide a rating score. The rating score should be an integer between 0 and 100, higher rating score means better quality. You should consider the following factors when rating the reference cluster table:
- **Intrinsic quality**:
  - If the cluster table meets the required quality with clear and consistent category names and descriptions, and no overlap or contradiction among the categories.
  - If the categories in the cluster table are relevant to the specified use case.
  - If the cluster table does not include any vague categories such as "Other", "General", "Unclear", "Miscellaneous" or "Undefined".
- **Extrinsic quality**:
  - If the cluster table can accurately and consistently classify the input data without ambiguity.
  - If there are missing categories in the cluster table that appear in the input data.
  - If there are unnecessary categories in the cluster table that do not appear in the input data.
## Q2: Explain your rating score in Q1 [The explanation should be concise, based on the intrinsic and extrinsic qualities evaluated in Q1].
## Q3: Based on your review, decide if you need to edit the reference table to improve its quality. If yes, suggest potential edits [Suggestions should be specific, actionable, and within the constraints of the maximum number of categories and use case specificity].
## Q4: If you decide to edit the reference table, provide your updated reference table. If you decide not to edit the reference table, please output the original reference table.
## Provide your answers between the following tags:
<rating_score>integer between 0 and 100</rating_score>
<explanation>concise explanation of your rating score based on the intrinsic and extrinsic qualities</explanation>
<suggestions>specific and actionable suggestions for edits, or "N/A" if no edits needed</suggestions>
<updated_table>
your updated cluster table in XML format if you decided to edit the reference table, or the original reference table if no edits made
</updated_table>
# Output

Evaluation Results

1/28/2026
Overall Score
2.72/5

Average across all 3 models

Best Performing Model
Low Confidence
google:gemini-2.5-flash-lite
4.29/5
google:gemini-2.5-flash-lite
#1 Ranked
4.29
/5.00
adh
3.9
cla
4.9
com
4.1
In
6,132
Out
3,289
Cost
$0.0019
anthropic:claude-3-5-haiku
#2 Ranked
2.60
/5.00
adh
1.6
cla
4.8
com
1.3
In
6,480
Out
1,091
Cost
$0.0095
openai:gpt-5-mini
#3 Ranked
1.28
/5.00
adh
1.0
cla
3.0
com
0.9
In
5,874
Out
4,604
Cost
$0.0107
Test Case:

Tags