A new open-source tool is trying to make LLM bad behavior measurable.

iFixAi is an open-source test kit (Github link in comment) for checking whether an AI agent behaves reliably before a company ships it to real users.

You get a repeatable, fixture-driven diagnostic with 32 inspection-tests across 5 risk categories.

Those tests ask practical questions: does the AI invent facts, follow unsafe instructions, change answers for no good reason, hide uncertainty, or act differently when the wording slightly changes?

A fixture is just the test scene, meaning the fake users, roles, tools, permissions, documents, and rules that mimic the company’s real product.

The big deal is repeatability, because iFixAi saves the exact model, prompts, rubrics, judge setup, and test inputs, so another engineer can rerun the same test and verify the same result.

The point is not to prove an AI is “safe,” but to catch regressions, compare vendors fairly, and give compliance teams evidence instead of screenshots and opinions.

Run the same fixtures, compare the same categories, and see whether the system’s behavior has moved.