Note: The job is a remote job and is open to candidates in USA. Olli Health is a Series A funded healthtech startup modernizing home healthcare with advanced AI tools. The Senior AI Engineer will play a crucial role in shaping the product by focusing on LLM-driven systems and ensuring the reliability of language models in a regulated customer-facing environment.
Responsibilities
- Own an LLM problem area end-to-end: drive the roadmap, design, and shipping of one or more LLM-driven product surfaces, collaborating with product, clinicians, and engineering partners
- Run high-leverage LLM experiments to advance the quality, response reliability (consistency, refusal, tool-use correctness), latency, and cost of our LLM workflows, and make calibrated model, prompt, retrieval, and agent design decisions
- Evolve our evaluation and observability stack to better measure model behavior locally, in CI, and in production, and extend instrumentation and analytics so we can better track cost, latency, usage, and quality as our products scale
- Discover, pilot, and integrate new quality signals, and build the human-in-the-loop workflows that feed our evaluation and training pipelines
- Raise the team’s LLM bar through code/design review, pattern-setting, mentorship, and written design docs
- Proactively identify areas of high risk in an ever-changing LLM landscape and integrate countermeasures into our workflows
- Partner with customers and internal clinical users to identify high-leverage opportunities, observe real usage, and ship customer-facing LLM features through to production under our HIPAA and clinical-safety constraints
Skills
- Hands-on experience shipping and iterating production LLM features (prompting, retrieval, tools/agents, failure modes), grounded in relevant NLP fundamentals (e.g. embeddings, retrieval, tokenization, extraction/classification) to make informed tradeoffs
- Define and run offline and/or online evaluation; use outcomes to drive releases; operate LLM-backed features after deployment (monitoring, regressions, drift, closing gaps between eval and production)
- 5+ years demonstrating strong programming skills; disciplined software practices suitable for shared production codebases (design clarity, review participation, testability where appropriate). You are capable of using LLMs to accelerate development, but require an understanding of the code at a deep level before trusting the output
- An enjoyment of that early startup balance between efficiency, scalability, and how to prioritize competing and complex demands meet customer needs
- Comfort working on applications subject to HIPAA and real clinical or operational consequences; eagerness to treat privacy, safe defaults, and proportionate controls as non-negotiable design inputs
- Must be authorized to work in the US and CA (no visa sponsorship available)
- Proven success in the Seed/Series A stage startups
- Experience training, evaluating, and iterating on supervised learning models in PyTorch or TensorFlow, including transformer-based architectures
- Advanced degree (PhD or comparable industry/research depth) in ML, NLP, CS, or a related field
- Experience with model observability, cost/latency optimization, LLM serving infrastructure, and the trade-offs between different frontier model providers
- Experience with our stack: AWS, PostgreSQL, Redis, Python, React
- Experience building data pipelines for analytics and ML-ready datasets
- Experience working with healthcare datasets (e.g. payer claims data, clinical notes, EHR charts, ICD-10 coding, etc.)
- Experience building secure systems with HIPAA compliance as a core principle
Benefits
- A key early engineering role with equity vesting options.
- A chance to be at the forefront of healthtech innovation, building real AI tools that meet compliance standards while solving real problems (we don't build marketing gimmicks).
- A collaborative and inclusive work environment that prioritizes debate, ethical decision-making, and is allergic to inflated egos.
- Flexible remote work environment with travel to team onsite meetings 1-2 times per year.
- Opportunities for professional growth and leadership.
- 100% coverage of Health, Vision, Dental, and Life insurance premiums for employees
- Unlimited PTO
- 401K plan
- Individual budget for your preferred laptop and monitor setup
- Fully remote
- Great colleagues working on a challenging problem for meaningful customers (older adults)
Company Overview