AI QA Engineer, LLM Evaluation and Test Automation
Test the AI features in Verasoft’s software for accuracy and safety, and use AI to make all of our testing faster and broader.
Services
AI features need a different kind of testing. We’re looking for AI QA engineers to make sure the LLM-powered features and agents in Verasoft’s software are accurate, consistent and safe, before they ship and after.
You’ll also use AI tools to make testing across our wider product suite faster and broader.
What you’ll do
- Plan and run evaluations of AI features for accuracy, consistency, relevance and safety, using automated checks and human review.
- Build and maintain evaluation datasets, including edge cases and adversarial inputs, with synthetic data where it helps.
- Build automated evaluation harnesses and regression suites that run in CI, so model, prompt and code changes are measured before release.
- Test for hallucinations, prompt injection, data leakage, bias and unsafe actions by agents.
- Define and track quality metrics such as factual accuracy, task success, latency and cost.
- Monitor AI features in production for drift and new failure modes, and turn every incident into a new test case.
- Use AI tools to generate test cases, write test automation and triage results across our products.
- Work closely with AI engineers to bring testing into the earliest stages of development.
What you’ll bring
- At least 3 years in software QA or test automation, including testing AI or ML features.
- Experience with LLM evaluation methods and tools, such as promptfoo, DeepEval, Ragas, LangSmith or similar.
- Experience with Python and a test framework such as pytest, and with Playwright or a similar UI automation tool.
- A strong understanding of how LLMs fail, and how to design tests that catch those failures.
- Strong debugging and root-cause analysis skills.
- Excellent written English, for judging AI-generated text and writing clear reports.
Nice to have
- Experience red-teaming or security testing LLM applications.
- Experience with human annotation workflows.
- Familiarity with retrieval (RAG) and agent architectures.
Verasoft is an equal opportunity employer. If you need an accommodation at any stage of the hiring process, please let us know. To learn how we handle applicant information, see our privacy notice.
Not quite the right role?
See every opening, or send a general application and tell us about the work you’re looking for.