Prosus
Senior Machine Learning Engineer, Evals
Опубликовано
- География
- Amsterdam
- Уровень
- Senior
Доступ проверим при нажатии.
О вакансии
Вакансия «Senior Machine Learning Engineer, Evals» в компании Prosus. Место работы: Amsterdam.
Чем предстоит заниматься
- Design evaluations with scientists and domain experts, translating intended capabilities into representative tasks, datasets and success criteria. Cover current applications and capabilities we want future models to acquire.
- Develop programmatic checks, outcome-based scoring and LLM-based graders. Calibrate automated judgments against expert reviews, investigate disagreements and test whether graders reward superficial answers or reject valid solutions.
- Maintain versioned tasks, datasets, graders and environments, with protected held-out tests and contamination checks. Expand coverage as models improve, retaining useful regression tests and adding harder tasks where existing tests no longer distinguish performance.
- Build tools and agents that help generate candidate tasks, run evaluations and investigate failures, validating their outputs before adoption. Review contributions and mentor engineers so teams can extend the suite consistently.
Что нужно для работы
- At least five years of ML or software engineering experience, including substantial hands-on work building and evaluating ML systems.
- Experience owning evaluation frameworks, benchmarks or model-quality systems that informed training or deployment decisions.
- Strong Python and experience with PyTorch, Transformers or comparable ML libraries, alongside reliable production software development.
- Practical knowledge of experimental design and statistics, including sampling, uncertainty, repeated trials and comparisons across model versions.
- Experience running inference or evaluation workloads at scale and debugging failures across models, data, scoring code and infrastructure.
- Clear communication and the ability to turn ambiguous results into defensible recommendations.
- Preferred qualifications
- Будет плюсом: Experience with language-model training or post-training.
- Будет плюсом: Experience with agent or gaming environments.
- Будет плюсом: Experience with human evaluation, LLM-as-judge calibration, recommendation metrics or online experimentation.
- Будет плюсом: Contributions to evaluation frameworks, public benchmarks or research.
Что предлагает компания
- Ownership of the evaluation platform and the evaluations that guide LCM's research, training and deployment.
- Direct collaboration with researchers, engineers and portfolio teams, with scope to publish methods and benchmarks, including through ProLLM, where the work can be shared.
- Colleagues who have released Hugging Face models with more than 100 million downloads, authored papers at NeurIPS, created benchmarks and built multiple production AI systems.
- Current ML frameworks and unlimited access to coding assistants.
- A base at our Amsterdam office and AI House, with relocation support. We also consider remote arrangements for candidates with particularly strong experience relevant to the role.
- At Prosus, your colleagues include 60,000 AI agents… and your job is to figure out what that actually means. We're one of the largest operators of agentic AI on the planet, inside a $100B+ portfolio across 100+ companies and two billion people. The humans here own what AI can't do alone: judgment, creativity, and the call on what gets built next.
- Own the impact. Work the ecosystem. Earn your growth. Build with backing.
- We respect the dignity and human rights of individuals and communities wherever we operate in the world. Building an inclusive workplace where everyone feels welcome and can thrive is critical for us. We provide access to education, which helps everyone understand the important role they play and the positive impact they can have.
Источник: Arbeitnow. Условия и актуальность вакансии подтверждает работодатель.
Подходит вакансия?
Перейдите к исходному объявлению или контакту работодателя.
Доступ проверим при нажатии.