Descripción de la vacante
Mindrift
Up to US$70,000 a year
Spain
AI Evaluation Engineer (Python, QA or Security) en Spain, España is listed on Jobeax. Browse 80,000+ vacancies available.
Up to $40/hr equivalent, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.
Please submit your CV in English and indicate your level of English proficiency.
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.
What this opportunity involves We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.
You'll create challenging tasks and evaluation criteria within realistic simulated environments:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid
Vacantes similares
Up to US$262,500 a year
... approaches and reject incorrect ones, neither too strict nor too lenient - Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust What we're looking for - 8+ years in software development - Core stack: Python (FastAPI), JavaScript/TypeScript ...
Spain
Full-time hours
... Strong software engineering background with QA ownership - Familiarity with RAG & Graph frameworks - 3+ years of experience as a software engineer (preferably 5), or QA engineer in production systems, preferably including at least 1 year experience in GenAI systems - Advanced Python skills - Experience working with LLM-based ...
Spain
Up to US$40 an hour
... that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient - Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust - Not data labeling - Not prompt engineering - Not writing ...
Spain
Plexus Tech
... desarrollo en Python. - Experiencia con LangChain o LangGraph para la orquestación de IA Generativa. - Comprensión profunda de los fundamentos de los LLM: Transformers, mecanismos de atención, embeddings y tokenización. - Dominio de al menos una plataforma cloud (AWS, Azure o GCP). - Experiencia con Azure AI Studio o Amazon ...
Spain, Valencia
... while integrating advanced AI capabilities. What we're looking for ✅ Must-haves for this challenge - 3–5 years of professional experience. - Strong command of Python for backend and API development. - Proven experience with OpenAI or similar AI models. - Experience with Gen AI models APIs, AI based application and patterns ...
Spain, Madrid
Up to US$70,000 a year
... approaches and reject incorrect ones, neither too strict nor too lenient - Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust What we're looking for - 5+ years in software development - Core stack: Python (FastAPI), JavaScript/TypeScript ...
Spain
Up to US$70,000 a year
... for Academic and/or Professional Experience Candidates should have a minimum of 3 years of professional experience in related roles or domain - specifically for QA-automation/testing or cybersecurity roles Education & certifications Educational qualifications - A Master's Degree in Computer Science, Software Engineering, ...
Spain
Up to US$70,000 a year
... approaches and reject incorrect ones, neither too strict nor too lenient - Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust What we're looking for - 5+ years in software development - Core stack: Python (FastAPI), JavaScript/TypeScript ...
Spain
Up to US$30,625 a year
... Code, or similar, on non-trivial work. You can cite a specific time the AI was confidently wrong and how you caught it. - English — B2+ written. Nice to have - Domain depth in Security, System Administration (nginx / systemd / cron), Scientific Computing (NumPy / PyTorch / SciPy), DevOps, or Git internals. - Modern Python tooling ...
Spain, Barcelona
Up to US$30,625 a year
... solve the task 20-60% of the time. - Iterate based on feedback from expert QA reviewers. - Later: review other authors' tasks as a QA reviewer. Nice to have - Domain depth in Security, System Administration (nginx / systemd / cron), Scientific Computing (NumPy / PyTorch / SciPy), DevOps, or Git internals. - Modern Python tooling ...
Spain, Barcelona
Fully remote
... open-source evaluation frameworks - Experience designing evaluations for agentic systems: tool use, multi-turn reasoning, environment-based benchmarks (SWE-bench, GAIA, WebArena-style), or interactive evaluation settings. - Track record of publishing or contributing to evaluation research, new benchmark design, methodology papers, ...
Spain
... incorporation. How do we envision you? - You hold a degree or Master's in Computer Engineering, Data Engineering, Data Science, or a related field. - You have more than 2 years (Data Scientist) or more than 5 years (AI Engineer) of professional experience developing solutions based on AI and Generative AI. - Experience in ...
Spain, Sevilla
Fully remote
... and C++-specific AI capabilities. - Build practical benchmarks and evaluation frameworks for AI-assisted C++ development scenarios. - Prototype and validate new AI workflows, including debugging, code understanding, refactoring, and project navigation. - Collaborate with product managers, ML engineers, QA engineers, stakeholders, ...
Spain
Fully remote
... and C++-specific AI capabilities. - Build practical benchmarks and evaluation frameworks for AI-assisted C++ development scenarios. - Prototype and validate new AI workflows, including debugging, code understanding, refactoring, and project navigation. - Collaborate with product managers, ML engineers, QA engineers, stakeholders, ...
Spain
JetBrains
... and C++-specific AI capabilities. - Build practical benchmarks and evaluation frameworks for AI-assisted C++ development scenarios. - Prototype and validate new AI workflows, including debugging, code understanding, refactoring, and project navigation. - Collaborate with product managers, ML engineers, QA engineers, stakeholders, ...
Spain, Madrid
CARTO
... self-healing selectors, AI-assisted triage - Nice to have: experience testing GIS or mapping software, CI/CD pipelines (ideally GitHub Actions), BDD/TDD practice or Gherkin, and querying data warehouses like BigQuery, Snowflake, Databricks or Redshift - +3 years in QA or test automation on a real product - A track record of ...
Spain, Madrid
Grid Dynamics
Grid Dynamics busca un Senior Machine Learning Engineer enfocado en sistemas LLM y evaluación para liderar proyectos de IA de última generación, incluidas técnicas de RAG, agentes y evaluación de seguridad AI. El candidato ideal tiene 5+ años en ML engineering, fuerte Python y experiencia con PyTorch, TensorFlow o JAX, ...
Spain, Castro-Urdiales
SD Worx
... AI & Azure Foundry Engineering Leadership -Design and build automation solutions leveraging Azure AI Foundry capabilities, including model orchestration, vector search, secure prompt pipelines, and evaluation tooling -Develop and maintain Python-based AI services integrated with Foundry workflows, AI hubs, models, and ...
Spain, Madrid
Accenture
... we're looking for Requirements: - University degree in Engineering, Computer Science, Mathematics, Physics, or equivalent. - Professional experience in Data Engineering, Software Engineering, or AI Engineering in production environments. - Advanced use of Python and SQL as primary work tools. - Experience with AWS services ...
Spain