Evaluation 1
Summarisation Sources¶
Source : https://cookbook.openai.com/examples/evaluation/how_to_eval_abstractive_summarization
Source : Yan, Ziyou. (Sep 2023). Evaluation & Hallucination Detection for Abstractive Summaries. eugeneyan.com. https://eugeneyan.com/writing/abstractive/.
Source : Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.https://doi.org/10.48550/arXiv.2303.16634
Creativity Sources¶
Source : Karen O'Quin & Susan P. Besemer (1989) The development, reliability, and validity of the revised creative product semantic scale, Creativity Research Journal, 2:4, 267-278, DOI: 10.1080/10400418909534323
Source : Susan P. Besemer & Karen O'Quin (1999) Confirming the Three-Factor Creative Product Analysis Matrix Model in an American Sample, Creativity Research Journal, 12:4, 287-296, DOI: 10.1207/s15326934crj1204_6
Source : O'Quin, K. and Besemer, S.P. (2006), Using the Creative Product Semantic Scale as a Metric for Results-Oriented Business. Creativity and Innovation Management, 15: 34-44. https://doi.org/10.1111/j.1467-8691.2006.00367.x
Source : D. Horn & G. Salvendy (2006) Product creativity: conceptual model, measurement and characteristics, Theoretical Issues in Ergonomics Science, 7:4, 395-412, DOI: 10.1080/14639220500078195
Paper :
In [ ]:
Copied!
In [1]:
Copied!
# Evaluation prompt template based on G-Eval
EVALUATION_PROMPT_TEMPLATE = """
You will be given one summary written for an article.
Your task is to rate the summary on one metric.
Please make sure you read and understand these instructions very carefully.
Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
{criteria}
Evaluation Steps:
{steps}
Example:
Source Text:
{document}
Summary:
{summary}
Evaluation Form (scores ONLY):
- {metric_name}
"""
# Metric 1: Relevance
RELEVANCY_SCORE_CRITERIA = """
Relevance(1-5) - selection of important content from the source. \
The summary should include only important information from the source document. \
Annotators were instructed to penalize summaries which contained redundancies and excess information.
"""
RELEVANCY_SCORE_STEPS = """
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the main points of the article.
3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.
"""
# Metric 2: Coherence
COHERENCE_SCORE_CRITERIA = """
Coherence(1-5) - the collective quality of all sentences. \
We align this dimension with the DUC quality question of structure and coherence \
whereby "the summary should be well-structured and well-organized. \
The summary should not just be a heap of related information, but should build from sentence to a\
coherent body of information about a topic."
"""
COHERENCE_SCORE_STEPS = """
1. Read the article carefully and identify the main topic and key points.
2. Read the summary and compare it to the article. Check if the summary covers the main topic and key points of the article,
and if it presents them in a clear and logical order.
3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.
"""
# Metric 3: Consistency
CONSISTENCY_SCORE_CRITERIA = """
Consistency(1-5) - the factual alignment between the summary and the summarized source. \
A factually consistent summary contains only statements that are entailed by the source document. \
Annotators were also asked to penalize summaries that contained hallucinated facts.
"""
CONSISTENCY_SCORE_STEPS = """
1. Read the article carefully and identify the main facts and details it presents.
2. Read the summary and compare it to the article. Check if the summary contains any factual errors that are not supported by the article.
3. Assign a score for consistency based on the Evaluation Criteria.
"""
# Metric 4: Fluency
FLUENCY_SCORE_CRITERIA = """
Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
1: Poor. The summary has many errors that make it hard to understand or sound unnatural.
2: Fair. The summary has some errors that affect the clarity or smoothness of the text, but the main points are still comprehensible.
3: Good. The summary has few or no errors and is easy to read and follow.
"""
FLUENCY_SCORE_STEPS = """
Read the summary and evaluate its fluency based on the given criteria. Assign a fluency score from 1 to 3.
"""
def get_geval_score(
criteria: str, steps: str, document: str, summary: str, metric_name: str
):
prompt = EVALUATION_PROMPT_TEMPLATE.format(
criteria=criteria,
steps=steps,
metric_name=metric_name,
document=document,
summary=summary,
)
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
temperature=0,
max_tokens=5,
top_p=1,
frequency_penalty=0,
presence_penalty=0,
)
return response.choices[0].message.content
evaluation_metrics = {
"Relevance": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),
"Coherence": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),
"Consistency": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),
"Fluency": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),
}
summaries = {"Summary 1": eval_summary_1, "Summary 2": eval_summary_2}
data = {"Evaluation Type": [], "Summary Type": [], "Score": []}
for eval_type, (criteria, steps) in evaluation_metrics.items():
for summ_type, summary in summaries.items():
data["Evaluation Type"].append(eval_type)
data["Summary Type"].append(summ_type)
result = get_geval_score(criteria, steps, excerpt, summary, eval_type)
score_num = int(result.strip())
data["Score"].append(score_num)
pivot_df = pd.DataFrame(data, index=None).pivot(
index="Evaluation Type", columns="Summary Type", values="Score"
)
styled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)
display(styled_pivot_df)
# Evaluation prompt template based on G-Eval
EVALUATION_PROMPT_TEMPLATE = """
You will be given one summary written for an article.
Your task is to rate the summary on one metric.
Please make sure you read and understand these instructions very carefully.
Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
{criteria}
Evaluation Steps:
{steps}
Example:
Source Text:
{document}
Summary:
{summary}
Evaluation Form (scores ONLY):
- {metric_name}
"""
# Metric 1: Relevance
RELEVANCY_SCORE_CRITERIA = """
Relevance(1-5) - selection of important content from the source. \
The summary should include only important information from the source document. \
Annotators were instructed to penalize summaries which contained redundancies and excess information.
"""
RELEVANCY_SCORE_STEPS = """
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the main points of the article.
3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.
"""
# Metric 2: Coherence
COHERENCE_SCORE_CRITERIA = """
Coherence(1-5) - the collective quality of all sentences. \
We align this dimension with the DUC quality question of structure and coherence \
whereby "the summary should be well-structured and well-organized. \
The summary should not just be a heap of related information, but should build from sentence to a\
coherent body of information about a topic."
"""
COHERENCE_SCORE_STEPS = """
1. Read the article carefully and identify the main topic and key points.
2. Read the summary and compare it to the article. Check if the summary covers the main topic and key points of the article,
and if it presents them in a clear and logical order.
3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.
"""
# Metric 3: Consistency
CONSISTENCY_SCORE_CRITERIA = """
Consistency(1-5) - the factual alignment between the summary and the summarized source. \
A factually consistent summary contains only statements that are entailed by the source document. \
Annotators were also asked to penalize summaries that contained hallucinated facts.
"""
CONSISTENCY_SCORE_STEPS = """
1. Read the article carefully and identify the main facts and details it presents.
2. Read the summary and compare it to the article. Check if the summary contains any factual errors that are not supported by the article.
3. Assign a score for consistency based on the Evaluation Criteria.
"""
# Metric 4: Fluency
FLUENCY_SCORE_CRITERIA = """
Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
1: Poor. The summary has many errors that make it hard to understand or sound unnatural.
2: Fair. The summary has some errors that affect the clarity or smoothness of the text, but the main points are still comprehensible.
3: Good. The summary has few or no errors and is easy to read and follow.
"""
FLUENCY_SCORE_STEPS = """
Read the summary and evaluate its fluency based on the given criteria. Assign a fluency score from 1 to 3.
"""
def get_geval_score(
criteria: str, steps: str, document: str, summary: str, metric_name: str
):
prompt = EVALUATION_PROMPT_TEMPLATE.format(
criteria=criteria,
steps=steps,
metric_name=metric_name,
document=document,
summary=summary,
)
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
temperature=0,
max_tokens=5,
top_p=1,
frequency_penalty=0,
presence_penalty=0,
)
return response.choices[0].message.content
evaluation_metrics = {
"Relevance": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),
"Coherence": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),
"Consistency": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),
"Fluency": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),
}
summaries = {"Summary 1": eval_summary_1, "Summary 2": eval_summary_2}
data = {"Evaluation Type": [], "Summary Type": [], "Score": []}
for eval_type, (criteria, steps) in evaluation_metrics.items():
for summ_type, summary in summaries.items():
data["Evaluation Type"].append(eval_type)
data["Summary Type"].append(summ_type)
result = get_geval_score(criteria, steps, excerpt, summary, eval_type)
score_num = int(result.strip())
data["Score"].append(score_num)
pivot_df = pd.DataFrame(data, index=None).pivot(
index="Evaluation Type", columns="Summary Type", values="Score"
)
styled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)
display(styled_pivot_df)
Another header here¶
then some text so that I can show off a bit
Going Back¶
A top level