Evaluate Anthropic Claud 3 Models with AWS Bedrock & FMEval Custom Model Runner

Evaluating Large Language Models (LLMs) is essential to ensure they meet the specific needs of diverse use cases while maintaining responsible AI practices. In this blog, we delve into how AWS Bedrock and FMEval facilitate the evaluation of Anthropic Claude 3 models. These tools provide a comprehensive framework for assessing model performance, helping data scientists and ML engineers to streamline their evaluation processes. The ability to conduct large-scale evaluations with built-in and custom algorithms makes FMEval an invaluable resource for those working with LLMs.

Anthropic’s Claude 3 models, including Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku, set new standards in reasoning, math, and coding. These models offer significant improvements in understanding non-English languages, maintaining high accuracy, and reducing hallucinations compared to other models. By leveraging AWS Bedrock and FMEval, users can evaluate these advanced models effectively, ensuring optimal performance and reliability. This blog will guide you through creating a custom Model Runner for evaluating Claude 3 models and demonstrate its application in various use cases.

Deep Dive into FMEval

With FM evaluations, AWS introduces FMEval, an open-source LLM evaluation library designed for data scientists and ML engineers. FMEval provides a code-first experience to evaluate LLMs for quality and responsibility, helping users select or adapt LLMs to specific use cases. It supports evaluations for LLM model endpoints or entire generative AI services. FMEval measures evaluation dimensions such as accuracy, robustness, bias, toxicity, and factual knowledge for any LLM. Users can evaluate AWS-hosted LLMs like Amazon Bedrock, Jumpstart, and other SageMaker models, as well as LLMs hosted on third-party platforms like ChatGPT, HuggingFace, and LangChain. This consolidation allows customers to streamline their LLM evaluation logic in one place.

Getting Started with FMEval

FMEval can be used wherever your workloads run, as a Python package or via the open-source code repository on GitHub. It provides easy-to-understand data and reports, enabling AWS customers to make informed decisions. Users can upload their own prompt datasets and algorithms. The core evaluation function, evaluate(), is extensible, allowing users to upload a prompt dataset, select and upload an evaluation function, and run an evaluation job. Results are delivered in multiple formats, helping to review, analyze, and operationalize high-risk items, and choose the right LLM for specific use cases.

Supported Algorithms

FMEval offers 12 built-in evaluations covering four different tasks and five evaluation dimensions. The library is designed with extensibility in mind, based on the latest scientific findings and the most popular open-source evaluations. The proposed evaluations are intended to cover popular aspects out-of-the-box and enable the addition of new ones.

Evaluation Tasks and Dimensions

• Open-ended generation: Evaluates prompt stereotyping, toxicity, factual knowledge, and semantic robustness.
• Text summarization: Measures accuracy, toxicity, and semantic robustness.
• Question answering (Q&A): Assesses accuracy, toxicity, and semantic robustness.
• Classification: Evaluates accuracy and semantic robustness.

Evaluation Details

• Accuracy: Assesses model performance across tasks such as summarization, Q&A, and classification, using metrics like ROUGE-N, METEOR, and BERTScore for summarization; Exact Match, Quasi-Exact Match, and F1 scores for Q&A; and standard metrics like accuracy, precision, recall, and balanced accuracy for classification.
• Semantic Robustness: Evaluates the performance change in model output due to semantic-preserving perturbations like typos, random upper-case changes, and whitespace modifications.
• Factual Knowledge: Measures the model’s ability to reproduce real-world facts using prompts from datasets like T-REx.
• Prompt Stereotyping: Analyzes whether the model encodes stereotypes across various categories, using the CrowS-Pairs dataset.
• Toxicity: Assesses the level of toxic content generated by the model, using datasets like Real Toxicity Prompts and BOLD, with UnitaryAI Detoxify-unbiased as the toxicity detector.

Using FMEval for Evaluations

The FMEval package includes core constructs necessary for conducting evaluation jobs:

• Data Config: Points to the dataset location and contains fields like model_input, target_output, and model_output. It can be customized based on the evaluation algorithm.
• Model Runner: Represents the hosted FM used for inference. FMEval supports native runners for JumpStart, Amazon Bedrock, and SageMaker endpoints, and allows for custom runners.
• Evaluation Algorithm: Supports built-in and custom algorithms for evaluating LLMs. Users can implement custom logic by inheriting the base Evaluation Algorithm class.

FMEval empowers users to evaluate their models comprehensively, providing insights to optimize performance and ensure responsible AI practices.

Meet Anthropic Claude 3

Anthropic’s data reveals that Claude 3 Opus, the most advanced model in their lineup, sets a new benchmark by outperforming other models, including OpenAI’s GPT-4, in reasoning, math, and coding. Claude 3 Sonnet excels with a blend of high intelligence and speed, operating twice as fast as earlier Claude models for most tasks. Claude 3 Haiku, the fastest and most compact model, is an affordable option in its intelligence category, offering near-instant responses that mimic human interactions.

Claude 3 models show notable improvements over other available models in tasks like coding and understanding non-English languages, including Spanish, Japanese, and French. They exhibit fewer hallucinations and higher accuracy compared to popular models from other providers. These models are adept at following complex, multi-step instructions and maintaining brand voice in customer-facing applications. Additionally, Claude 3 models can generate outputs suited for natural language classification and sentiment analysis, simplifying application development for these use cases.

All three new models feature advanced vision capabilities, allowing them to process various data formats and analyze images. This meets the increasing demand for models that comprehend charts, graphs, technical diagrams, photos, and other visual assets.

AWS Customers can access the Claude 3 family of vision-enabled models on Amazon Bedrock. Amazon Bedrock is the first managed service to offer all Claude 3 models—Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku—as generally available

FMEval Custom Claude 3 Model Runner

FMEval provides two model runners: SageMaker JumpStart & Bedrock out of the box. Custom model runners can be created by extending existing Model Runner classes that require special processing.

Bedrock Model Runner uses Text Completion API for older Anthropic Claude 2.x verison models but in order to evaluate newer Anthropic Claude 3 Models such as Haiku or Sonnet or Opus, a custom Model Runner need to be created as it uses Messages API. The rest of the blog posts focus on how to create a custom Model Runner and evaluate Claude 3 Models on couple of use cases.

Summarization & Factual Knowledge Evaluation

At a minimum, two packages are required

requirements.txt

boto3
fmeval

Custom Bedrock Model Runner for Claude 3:

bedrock_claud3_model_runner.py

"""
Module to manage model runners for Bedrock models.
"""
import json
import logging
from fmeval.util import require
from typing import Optional, Tuple
from fmeval.constants import MIME_TYPE_JSON
from fmeval.model_runners.model_runner import ModelRunner
from fmeval.model_runners.util import get_bedrock_runtime_client

logger = logging.getLogger(__name__)


class BedrockClaude3ModelRunner(ModelRunner):
    """
    A class to manage the creation and deletion of Bedrock model runner when user provides
    a Bedrock model id.
    """

    def __init__(
        self,
        model_id: str,
        content_template: str,
        output: Optional[str] = None,
        log_probability: Optional[str] = None,
        content_type: str = MIME_TYPE_JSON,
        accept_type: str = MIME_TYPE_JSON,
    ):
        """
        :param model_id: Id of the Bedrock model to be used for model predictions
        :param content_template: String template to compose the model input from the prompt
        :param output: JMESPath expression of output in the model output
        :param log_probability: JMESPath expression of log probability in the model output
        :param content_type: The content type of the request sent to the model for inference
        :param accept_type: The accept type of the request sent to the model for inference
        """
        super().__init__(content_template, output, log_probability, content_type, accept_type)
        self._model_id = model_id
        self._content_template = content_template
        self._output = output
        self._log_probability = log_probability
        self._content_type = content_type
        self._accept_type = accept_type

        require(
            output is not None or log_probability is not None,
            "One of output jmespath expression or log probability jmespath expression must be provided",
        )
        require(self._accept_type == MIME_TYPE_JSON, f"Model accept type `{self._accept_type}` is not supported.")
        require(
            self._content_type == MIME_TYPE_JSON,
            f"Model content type `{self._content_type}` is not supported.",
        )
        self._bedrock_runtime_client = get_bedrock_runtime_client()

    def predict(self, prompt: str) -> Tuple[Optional[str], Optional[float]]:
        """
        Invoke the Bedrock model and parse the model response.
        :param prompt: Input data for which you want the model to provide inference.
        """
        composed_data = self._composer.compose(prompt)
        
        body=json.dumps(
            {
                "anthropic_version": composed_data['anthropic_version'],
                "max_tokens": composed_data['max_tokens'],
                "system": composed_data['system'],
                "messages": [{"role": "user", "content": composed_data['prompt']}]
            }  
        )
        
        response = self._bedrock_runtime_client.invoke_model(
            body=body, modelId=self._model_id, accept=self._accept_type, contentType=self._content_type
        )
        
        # Claude 3 models provide completion in an array of content element that includes text as the key
        model_output = json.loads(response.get("body").read())['content'][0]
        
        output = (
            self._extractor.extract_output(data=model_output, num_records=1)
            if self._extractor.output_jmespath_expression
            else None
        )
        
        log_probability = (
            self._extractor.extract_log_probability(data=model_output, num_records=1)
            if self._extractor.log_probability_jmespath_expression
            else None
        )
        return output, log_probability

    def __reduce__(self):
        """
        Custom serializer method used by Ray when it serializes instances of this
        class in eval_algorithms.util.generate_model_predict_response_for_dataset.
        """
        serialized_data = (
            self._model_id,
            self._content_template,
            self._output,
            self._log_probability,
            self._content_type,
            self._accept_type,
        )
        return self.__class__, serialized_data

Summarization Use Case:

A typical Evaluation require a json file with .jsonl extension that contains few samples. Each sample contains a document key with some text as value, and summary key with expected output. Each sample is fed into the model and its output is evaluated with the expected summary and provides 3 different metrics such as METEOR, ROUGE and BERTScore.

METEOR

METEOR (Metric for Evaluation of Translation with Explicit ORdering) is a metric used to evaluate the quality of machine-generated translations. It extends beyond simple word overlap by incorporating stemming, synonym matching, and exact matches. The primary features of METEOR include:

• Stemming: Reducing words to their base or root form.
• Synonym Matching: Recognizing synonyms to account for different words with similar meanings.
• Precision and Recall: Balancing the measure of exact matches (precision) and relevant matches retrieved (recall).
• F-score: A weighted harmonic mean of precision and recall, providing a single score that balances both.

METEOR aims to improve correlation with human judgment over other metrics by considering linguistic phenomena like synonyms and stemming.

ROUGE

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics for evaluating automatic summarization and machine translation. The most common variants include:

• ROUGE-N: Measures the overlap of n-grams (sequences of N words) between the generated summary and a reference summary. For example, ROUGE-1 measures unigram (single word) overlap, while ROUGE-2 measures bigram (two-word sequence) overlap.
• ROUGE-L: Measures the longest common subsequence (LCS) between the generated and reference summaries, capturing sentence-level structure similarity.
• ROUGE-S: Measures the overlap of skip-bigrams, which are pairs of words in their sentence order allowing gaps between them.

ROUGE primarily focuses on recall, making it useful for evaluating summaries where capturing all relevant information is crucial.

BERTScore

BERTScore is a metric that leverages deep learning models, specifically BERT (Bidirectional Encoder Representations from Transformers), to evaluate text similarity. It calculates the similarity of the embeddings (vector representations) of words in the generated and reference texts. Key aspects of BERTScore include:

• Contextual Embeddings: Uses BERT to generate context-aware embeddings for words, capturing semantic nuances.
• Cosine Similarity: Measures the cosine similarity between the embeddings of the generated text and reference text.
• Precision, Recall, and F1: Similar to traditional metrics, BERTScore provides precision, recall, and an F1 score, offering a balanced view of the model’s performance.

BERTScore aims to capture semantic similarity more effectively than surface-level metrics, accounting for the meaning and context of the text.

#summarize_accuracy.jsonl

{"document": "five-time world champion michelle kwan withdrew from the 2006 us figure skating championships on wednesday, but will petition us skating officials for the chance to compete at the 2006 turin olympics.","summary": "injury leaves kwan's olympic hopes in limbo","idx": "0"}
{"document": "the company announced a new range of products in a press release today, highlighting their commitment to innovation and customer satisfaction. the new line includes several high-tech gadgets aimed at improving daily life.","summary": "company unveils innovative product line","idx": "1"}
{"document": "scientists have discovered a new species of fish in the deep waters of the pacific ocean. the fish, which has unique characteristics, adds to the biodiversity of the region and provides new opportunities for research.","summary": "new fish species discovered in pacific ocean","idx": "2"}
{"document": "the mayor held a press conference to address the recent rise in crime rates in the city. she outlined new measures that will be implemented to improve public safety and reduce criminal activities.","summary": "mayor announces new measures to tackle crime","idx": "3"}
{"document": "a major earthquake struck the coastal region early this morning, causing significant damage to buildings and infrastructure. emergency services are on the scene, and the government has declared a state of emergency.","summary": "earthquake causes widespread damage on coast","idx": "4"}
{"document": "the tech giant reported record profits for the third quarter of the fiscal year, driven by strong sales of its flagship products. the company's stock prices surged following the announcement.","summary": "tech giant reports record third-quarter profits","idx": "5"}
{"document": "researchers at the university have developed a new vaccine that shows promise in protecting against the zika virus. initial trials have been successful, and further testing is planned.","summary": "new zika virus vaccine shows promise","idx": "6"}
{"document": "the local sports team clinched the championship title after a thrilling final match. fans celebrated the victory, marking the team's first championship win in over a decade.","summary": "local team wins championship after thrilling final","idx": "7"}
{"document": "a new law aimed at reducing carbon emissions was passed by the legislature today. the law includes measures to promote renewable energy sources and improve energy efficiency.","summary": "legislature passes new law to reduce carbon emissions","idx": "8"}
{"document": "the famous artist's latest exhibition opened to the public today, featuring a collection of new works that explore themes of nature and humanity. the exhibition will run for the next three months.","summary": "artist's new exhibition explores nature and humanity","idx": "9"}

#summarization_bedrock.py

import boto3
import os
import glob
import json
import pandas as pd

from fmeval.data_loaders.data_config import DataConfig
from bedrock_claud3_model_runner import BedrockClaude3ModelRunner
from fmeval.constants import MIME_TYPE_JSONLINES
from fmeval.eval_algorithms.summarization_accuracy import SummarizationAccuracy

os.environ["PARALLELIZATION_FACTOR"] = "1"
os.environ["DISABLE_FMEVAL_TELEMETRY"] = "true"

# Bedrock clients for model inference
bedrock = boto3.client(service_name='bedrock')
bedrock_runtime = boto3.client(service_name='bedrock-runtime')

model_id = 'anthropic.claude-3-haiku-20240307-v1:0'
# model_id = 'anthropic.claude-3-sonnet-20240229-v1:0'

accept = "application/json"
contentType = "application/json"

config = DataConfig(
    dataset_name="summarize_accuracy_sample",
    dataset_uri="data/summarize_accuracy.jsonl",
    dataset_mime_type=MIME_TYPE_JSONLINES,
    model_input_location="document",
    target_output_location="summary"
)
    
content_template = '{"anthropic_version": "bedrock-2023-05-31", "system": "Summarise the following text in one sentence and less than 10 words", "prompt": $prompt, "max_tokens": 500}'

bedrock_model_runner = BedrockClaude3ModelRunner(
    model_id=model_id,
    output='text',
    content_template=content_template
)

eval_algo = SummarizationAccuracy()
eval_output = eval_algo.evaluate(model=bedrock_model_runner, dataset_config=config, 
                                 prompt_template="$model_input", save=True)
                                 
eval_response = json.dumps(eval_output, default=vars, indent=4)
eval_response_json = json.loads(eval_response)
output_path = eval_response_json[0]["output_path"]

with open(output_path, "r") as file:
    for line in file:
        print(line)

Summarization Evaluation Result – Haiku Model

Summarization Evaluation Result – Sonnet Model

Factual Knowledge Use Case:

A typical Evaluation require a json file with .jsonl extension that contains few samples. Each sample contains a question key with some text as value, answer key with expected output (It can have multiple answers with OR), and knowledge_category key to indicate category. Each sample is fed into the model and its output is evaluated with the expected summary and provides 1 metric score with either 0 or 1 for no-match or match respectively.

# factual_knowledge.jsonl

{"question": "Paracetamol is commonly used to treat","answers": "pain<OR>fever<OR>headache<OR>flu","knowledge_category": "Pharmaceuticals"}
{"question": "The Great Wall of China was primarily built to","answers": "protect against invasions<OR>defend from nomadic tribes<OR>defend from invasions<OR>protect from nomadic tribes","knowledge_category": "History"}
{"question": "The heart is part of the","answers": "circulatory system<OR>cardiovascular system<OR>human circulatory system<OR>human cardiovascular system","knowledge_category": "Biology"}
{"question": "COVID-19 is caused by","answers": "coronavirus<OR>SARS-CoV-2<OR>the novel coronavirus<OR>COVID-19 virus","knowledge_category": "Healthcare"}
{"question": "Penicillin was discovered by","answers": "Alexander Fleming<OR>Fleming<OR>Sir Alexander Fleming","knowledge_category": "Scientists"}
{"question": "The speed of light in vacuum is approximately","answers": "299,792 kilometers per second<OR>299,792 km/s<OR>300,000 km/s","knowledge_category": "Physics"}
{"question": "The periodic table was created by","answers": "Dmitri Mendeleev<OR>Mendeleev<OR>Dmitri Ivanovich Mendeleev","knowledge_category": "Chemistry"}
{"question": "The first woman to win a Nobel Prize was","answers": "Marie Curie<OR>Curie<OR>Madame Curie","knowledge_category": "History"}
{"question": "The longest reigning British monarch is","answers": "Queen Elizabeth II<OR>Elizabeth II<OR>Queen Elizabeth the Second","knowledge_category": "History"}
{"question": "The author of '1984' is","answers": "George Orwell<OR>Orwell<OR>Eric Arthur Blair","knowledge_category": "Literature"}

# factual_knowledge_bedrock.py

import boto3
import os
import glob
import json

from fmeval.data_loaders.data_config import DataConfig
from bedrock_claud3_model_runner import BedrockClaude3ModelRunner
from fmeval.constants import MIME_TYPE_JSONLINES
from fmeval.eval_algorithms.factual_knowledge import FactualKnowledge, FactualKnowledgeConfig

# Bedrock clients for model inference
bedrock = boto3.client(service_name='bedrock')
bedrock_runtime = boto3.client(service_name='bedrock-runtime')

model_id = 'anthropic.claude-3-haiku-20240307-v1:0'
# model_id = 'anthropic.claude-3-sonnet-20240229-v1:0'

config = DataConfig(
    dataset_name="factual_knowledge",
    dataset_uri="data/factual_knowledge.jsonl",
    dataset_mime_type=MIME_TYPE_JSONLINES,
    model_input_location="question",
    target_output_location="answers"
)

content_template = '{"anthropic_version": "bedrock-2023-05-31", "system": "Provide the answer in one sentence and less than 5 words", "prompt": $prompt, "max_tokens": 500}'

bedrock_model_runner = BedrockClaude3ModelRunner(
    model_id=model_id,
    output='text',
    content_template=content_template
)

eval_algo = FactualKnowledge(FactualKnowledgeConfig(target_output_delimiter="<OR>"))
eval_output = eval_algo.evaluate(model=bedrock_model_runner, dataset_config=config, 
                                 prompt_template="$model_input", save=True)
                                 

eval_response = json.dumps(eval_output, default=vars, indent=4)
eval_response_json = json.loads(eval_response)
output_path = eval_response_json[0]["output_path"]

with open(output_path, "r") as file:
    for line in file:
        print(line)

Factual Knowledge Evaluation Result – Haiku Model

Conclusion

FM evaluations empower customers to ensure the LLM they choose is suitable for their specific use case and operates responsibly. This extensible responsible AI framework is seamlessly integrated into Amazon SageMaker, enhancing the transparency of language models by simplifying the evaluation and communication of risks throughout the ML lifecycle. It represents a significant advancement in building trust and promoting the adoption of LLMs on AWS.

FMEval and AWS Bedrock further support this process by providing powerful tools for evaluating and managing LLMs. FMEval offers a vast set of built-in algorithms across many NLP tasks and serves as a scalable and flexible tool for large-scale evaluations of your own models, datasets, and algorithms. AWS Bedrock simplifies deploying and operationalizing these evaluations, enabling organizations to efficiently manage and scale their AI models. Together, FMEval and AWS Bedrock help ensure that language models perform optimally and responsibly, facilitating their seamless integration into production environments.

References

• https://aws.amazon.com/bedrock/
• https://github.com/aws/fmeval

Author: Raghavan Madabusi