Mäta feedbackkvalitet och relevans

Reinforcement Learning from Human Feedback (RLHF)

Mina Parham

AI Engineer

Att identifiera avvikande feedback

Till exempel:

  • Positiv recension:
    • "Jag älskade den här produkten!"
  • Negativ recension:
    • "Usel service."
  • Neutral recension:
    • "Gör vad den ska."
  • Avvikande recension:
    • "Himlen är blå."

En bedömning på fyra av fem stjärnor med en hand som lägger till en femte stjärna

Reinforcement Learning from Human Feedback (RLHF)

Identifiera avvikande feedback

import numpy as np

def least_confidence(prob_dist):
    simple_least_conf = np.nanmax(prob_dist) 
    num_labels = float(prob_dist.size)  # number of labels
    least_conf = (1 - simple_least_conf) * (num_labels / (num_labels - 1))
    return least_conf

def filter_low_confidence_predictions(prob_dists, threshold=0.5):
    filtered_indices = [i for i, prob_dist in enumerate(prob_dists) 
                        if least_confidence(prob_dist) > threshold]
    return filtered_indices
Reinforcement Learning from Human Feedback (RLHF)

Identifiera avvikande feedback

prob_distribution_array = np.array([
    [0.1, 0.1, 0.2],   # Low confidence (0.2)
    [0.6, 0.2, 0.1],   # High confidence (0.6)
    [0.3, 0.3, 0.4]   # Medium confidence (0.4)
])

# Filter function with 0.5 threshold
filtered_feedback_indices, filtered_confidences = 
filter_low_confidence_predictions(prob_distribution_array, threshold=0.5)

print(f"Filtered Confidence Scores: {filtered_confidences}")

Filtered Confidence Scores: [0.6]
Reinforcement Learning from Human Feedback (RLHF)

K-means

  • Bra för att identifiera avvikelser och enkel att implementera
  • Använd domänkunskap eller analytiska metoder för att bestämma antal kluster

Ett diagram som illustrerar k-means-algoritmen.

Reinforcement Learning from Human Feedback (RLHF)

Avvikelsedetektering med k-means

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans


def detect_anomalies(data, n_clusters=3):
    kmeans = KMeans(n_clusters=n_clusters, random_state=42)
    clusters = kmeans.fit_predict(data)
    centers = kmeans.cluster_centers_

    # Calculate distances from cluster centers
    distances = np.linalg.norm(data - centers[clusters], axis=1)
    return distances
Reinforcement Learning from Human Feedback (RLHF)

Avvikelsedetektering med k-means

feedback_data = np.array([
    [4.0],  # Close to center of cluster
    [4.5],  # Close to center of cluster
    [1.0],  # Anomaly - far from main group
    [4.1],  # Close to center of cluster
    [3.9]  # Close to center of cluster
])

anomalies = detect_anomalies(confidences, n_clusters=1)

print(anomalies)

[0.5 1.  2.5   0.6 0.4]
Reinforcement Learning from Human Feedback (RLHF)

Nu kör vi en övning!

Reinforcement Learning from Human Feedback (RLHF)

Preparing Video For Download...