AI Evaluations Engineer – Healthcare

Ellipsis Health • Remote (Remote (US)) • $150k - $180k / year • 1m ago

Responsibilities

Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams, including automated testing platform for AI voice agents, debugging and observability tools.
Develop and productionalize evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails.
Partner with ML, engineering and QA teams to translate evaluation requirements into robust, maintainable infrastructure and tooling.
Improve developer experience by making evaluation systems easy to extend, well-documented, and reliable in day-to-day use.
Ensure evaluation tooling meets production standards for reliability, performance, and maintainability.

Qualifications

5+ years of professional software engineering experience, with a strong focus on building backend systems, platforms, or developer tooling.
Proven experience designing and maintaining production-grade infrastructure with code, including APIs, services, and data pipelines.
Strong proficiency in at least one general-purpose programming language (e.g., Python, Typescript/Javascript, Java, or similar).
Experience using test automation frameworks, evaluation pipelines, or CI/CD-integrated testing systems.
Familiarity with observability and debugging tools (logging, metrics, tracing) and building internal tools that improve developer and QA workflows.
Strong debugging skills and a methodical approach to diagnosing production and evaluation issues.
Ability to collaborate effectively across engineering, QA, and operations teams, translating requirements into reliable, maintainable systems.
Product-minded approach to infrastructure, with attention to usability, documentation, and long-term maintainability.

Preferred

Experience working with complex, multi-component systems (e.g., ASR, LLMs, TTS, or other ML-powered services)
Experience working in healthcare or other regulated environments, including awareness of HIPAA and PHI handling.
Familiarity with conversational AI or voice agents, including multi-turn dialogue, latency constraints, and error recovery.
Familiarity with LLM observability or evaluation tools (e.g., Langfuse, prompt eval frameworks).
Background in digital health, care coordination, or patient-facing systems.

Background Checks

As a health technology company, we reserve the right to run background checks on candidates to whom we extend offers, in compliance with applicable laws. We evaluate candidates holistically and comply with all “ban the box” regulations.

Salary and Benefits

We offer competitive salary and benefits, including 401(k) matching, health, vision, and dental insurance, and very flexible paid time off.
The typical salary range for this role is $150,000 to $180,000 USD, depending on skills, qualifications, and relevant experience.

Assistance

If you have a disability or require accommodations during the application or recruitment process, please contact careers@ellipsishealth.com.

Related Jobs

Lead Product Designer - Mobile at Somethings

SOMETHINGS • Hybrid (New York City, NY) • $150k - $220k / year • 2w ago

2w ago

Lead Product Designer at Somethings

SOMETHINGS • Hybrid (New York City, NY) • $140k - $210k / year • 2w ago

2w ago

Marketing Lead — Brand and Cultural Strategy at Somethings

SOMETHINGS • In Office (New York City, NY) • $140k - $180k / year • 2w ago

2w ago

Government Affairs Lead at Somethings

SOMETHINGS • In Office (New York, NY) • $160k - $220k / year • 2w ago

2w ago

VP of Sales at Somethings

SOMETHINGS • In Office (New York, NY) • $180k - $220k / year • 2w ago

2w ago

Senior Full Stack Software Engineer at Somethings

SOMETHINGS • In Office (New York, NY) • $160k - $225k / year • 2w ago

2w ago

Chief Science & Innovation Officer (CSIO)

Radicle Science • Remote (U.S. required) with travel for key events (up to 50%) • 2w ago

2w ago

Sales Director

Radicle Science • Remote (U.S. required) with travel for key events (up to 50%) • 2w ago

2w ago

Founding Machine Learning Engineer

Orbit • San Francisco • $150k - $225k / year • 2w ago

2w ago

Expression of Interest

Orbit • San Francisco • 2w ago

2w ago