Artificial intelligence

AI response evaluation bench

Compare responses against references, check explicit constraints and optionally connect a local Ollama server.

My library
Resource type
Downloadable pack
Version
1.0.0
Licence
MIT
FreeGet this resource

Contents and terms

A reproducible evaluation runner using JSON files. It measures exact match, lexical F1, required or forbidden phrases and maximum length. Missing responses remain visible failures. An optional adapter calls a local Ollama IP address without proxies or automatic redirects. These metrics do not establish truth, safety or overall response quality.

Included capabilities

  • Offline response-file evaluation
  • Lexical F1 and explicit constraints
  • CI-friendly exit status
  • Loopback-only Ollama adapter
  • Local HTTP tests without model downloads

Licence and getting started

Original code under the MIT licence, copyright 2026 STELLARR STUDIO. Personal and commercial use permitted with the licence notice retained. French and English guides included. Trademarks and dependencies retain their own rights.

Read the README for requirements, start commands, tests and deployment limits. Hosting, compute, external APIs and bespoke support are not included.

Included files

  • .gitignore
  • evaluate.py
  • examples/cases.json
  • examples/responses-failing.json
  • examples/responses.json
  • kit.json
  • LICENSE
  • README.en.md
  • README.md
  • test_evaluate.py
  • TEST_REPORT.md
  • THIRD_PARTY_NOTICES.md
  • verification.json

Getting started

01

Compatibility

The technologies on this page provide a starting point. Check exact versions, dependencies and requirements in the included guides before installation.

  • Python
  • JSON
  • Ollama (optionnel)
02

Set up at your own pace

Download the archive from your account, extract it into a dedicated directory and follow README.md or README.en.md. Review commands and try the provided examples before using your own data.

03

An explicit scope

This page describes the delivered contents. Integration with your product or bespoke assistance is arranged separately with the studio. Usage rights are specified in the licence below.

Discuss a specific requirement

This resource’s licence

Read the complete terms before ordering. They define permitted uses and redistribution obligations.

Show the full textMIT
MIT License

Copyright (c) 2026 STELLARR STUDIO

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
A first conversation

Let’s talk about your project.

Tell us about your project in a few lines. Your message will be sent directly to our team.

PhoneMonday to Friday, 10:00 to 22:00

Saturday and Sunday: email only

Paris time (Europe/Paris)

An anti-bot check protects this form. It also runs when you choose to send your message.

Your contact details and message are used to handle your enquiry and reply to you. Learn more about your data.

You can also email contact@stellarrstudio.com.

Explore the studio

What are you looking for?

Enter a few words to find a page.

    Cookies and external content
    Remembering your preferences

    Your choice is stored on this device for six months. This local storage is not used to track you.

    Necessary

    Short videos, fonts and illustrations are hosted with the website. No external service consent is needed to display them. Learn more