My roleIndependent engineerStackPython · Streamlit · scikit-learn · Plotly
Problem
Prompt wording can affect evaluation results.
Models can respond differently when a request keeps the same meaning but uses different wording. The application tests those variations under consistent settings and retains the response evidence used to calculate each metric.
Implementation
Controlled prompt variations and consistent generation settings.
The engine applies seven prompt variations, records every response, calculates violation and refusal metrics, and groups similar outputs using TF-IDF and K-means clustering.
01ConfigureSelect models and generation settings.
02Run testsApply each prompt variation to every target.
03ReviewCompare metrics, clusters, and responses.
04ExportDownload CSV, JSON, or PDF reports.
Technical work
Evaluation, analysis, testing, and reporting.
Multi-model evaluation engine with deterministic offline fixtures and OpenAI-compatible providers
Response evidence, behavioral clustering, and violation and refusal metrics
CSV, JSON, and PDF report generation
Automated tests and a GitHub Actions continuous integration workflow