Cargando descripción…

Software Engineering Evaluation Specialist en Barcelona,… - Jobeax

Software Engineering Evaluation Specialist en Barcelona, España is listed on Jobeax. Browse 80,000+ vacancies available.

About the role

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

About the Role

You’ll design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome. Your deliverable is the full task package: broken code, tests, instructions, and a reference solution proving the task is solvable.

What you'll do

Invent a realistic developer scenario — a real bug, a broken ETL, a missing feature — not a toy problem.
Build a reproducible Docker environment with pinned dependencies.
Write a pytest that verifies outcomes, not specific commands — deterministic, non-flaky, and does not leak the fix.
Write an https://jobeax.com/link/HkFvsHT8ADjCqmv9 that reads like a Jira ticket a developer would receive.
Write a reference https://jobeax.com/link/CUdYvpFg5T5uOD03 proving the task is solvable.
Calibrate difficulty so current state-of-the-art agents solve the task 20–60% of the time.
Iterate based on feedback from expert QA reviewers.
Later: review other authors’ tasks as a QA reviewer.

What we're looking for

3+ years of production software development in one backend stack — Python, Go, https://jobeax.com/link/JSd7UYCSg572jJDY, Java, or Rust. Depth in one stack beats breadth.
Python + pytest fluency — required regardless of primary stack. The task harness is pytest-based even when the broken app is in another language. Fixtures, parametrize, monkeypatch, timeouts, conftest.py.
Docker authoring — reproducible Dockerfiles, pinned dependencies, multi-stage builds when needed, non-root user.
Linux & Bash — comfort debugging inside containers (strace, lsof, journalctl); shell beyond set -euo pipefail.
AI coding agent experience — Claude Code, Cursor, Roo Code, or similar, on non-trivial work. You can cite a specific time the AI was confidently wrong and how you caught it.
English — B2+ written.

Nice to have

Domain depth in Security, System Administration (nginx / systemd / cron), Scientific Computing (NumPy / PyTorch / SciPy), DevOps, or Git internals.
Modern Python tooling (uv, poetry, https://jobeax.com/link/YRJr577iA0aMx5co).
Coverage tooling (pytest-cov, https://jobeax.com/link/khUWQLUnflUiPBzz, gcov, llvm-cov, kcov).
Fuzzing or property-based testing (Hypothesis).
Prior contribution to agent-evaluation benchmarks or related frameworks.

What you'll get

Paid contributions, rates up to $35/hour*.
Task-based compensation equivalent to hourly rate, depending on performance and volume.
Some projects include incentive payments.

*Rates vary based on expertise, skills assessment, location, project needs, and other factors. Higher rates may be provided to highly specialized experts. Lower rates may apply during onboarding or non-core project phases. Payment details are shared per project.

How you'll work

Onboarding: ~10 hours per first task.
Steady state: ~5 hours per task, 2–4 parallel tasks per author.
Realistic weekly load: 8–20 hours. Higher volume available for top performers.
You choose when and how to contribute; tasks must be submitted by the deadline and meet acceptance criteria.

Hiring process

Process

Apply → Pass qualification (90-minute sample-task screen + short behavioral interview) → Join a project → Complete tasks → Get paid.

Apply

Submit your CV via the Mindrift platform. Indicate your English level, note this role (Software Engineering Evaluation Specialist — Terminal Bench), and include a GitHub profile link if available.

Not in scope

Data labeling, prompt engineering.
Production code to ship — you design problems and verification for AI agents.
Leetcode puzzles — scenarios must look like real developer work.
Not every candidate task ships — quality over quantity.

Not a fit

Data Science, ML, or Computer Vision engineers without backend-engineering output.
Manual QA testers without automation or test authoring.
Frontend-only, low-code / no-code, IT Support, or Business Analysts.
Engineers who have never written pytest from scratch.
Junior, intern, or assistant as the most recent role.

Additional information

Please submit your CV in English and indicate your level of English proficiency.

Postularse a la vacante

Vacantes similares

Más como este puesto

Mindrift

Software Engineering Evaluation Specialist

Hace 13 días

Remoto

Up to US$30,625 a year

... must be submitted by the deadline and meet acceptance criteria. Apply Submit your CV via the Mindrift platform. Indicate your English level, note this role (Software Engineering Evaluation Specialist - Terminal Bench), and include a GitHub profile link if available. - Up to US$30,625 a year Gross, as stated in the ad. - ...

Spain, Barcelona

Manufacturing Process Evaluation Specialist

Hace 4 días

Remoto
Contrato

Have the right to work in Spain

... our teams and our culture. We are committed to equal employment opportunities that foster an inclusive environment. We are looking for Manufacturing Process Evaluation Specialist within the Product Quality area of the Quality Assurance division of Instituto Grifols SA, a company specialized in the research, development and ...

Spain

Software Engineering Intern

Hace 7 días

Híbrido
Tiempo completo
Tiempo parcial

Hybrid

About the role We're looking for Software Engineering Interns to help build and scale the systems that power Datadog's observability and security platform. Interns contribute directly to real-world engineering challenges across backend, frontend, infrastructure, data engineering, and developer tooling while working alongside ...

Spain

Bain & Company

Lead, Software Engineering

Hace 8 días

Remoto

... components to efficiently address common software engineering challenges across various industries and business functions - Champion best demonstrated practices in software engineering, and share learnings with team members in AIS about theoretical and technical developments in software engineering - Lead industry-leading innovations ...

Spain, Madrid

Thales

Software Engineering Intern

Hace 13 días

Híbrido
Tiempo completo

€16,100 a year

... training in a Master's in Adaptive Management and Innovation delivered by the Universidad-Empresa Foundation (distance learning). During your internship as a Software Engineering Intern: - Software development ( Develop software ) - Design ( Design ) - Command and control applications ( Command and control applications ) ...

Spain, Madrid

IVD Systems Engineering Specialist

Hace 5 días

Híbrido
Presencial
Contrato

Hybrid

... leader in plasma-derived medicines with a presence in more than 100 countries, and a growing global team of over 20.000 people. That's why we need a Systems Engineering Specialist like you to join IVD Systems Department of the Instrument Development & Life Cycle Management (LCM) Area at Diagnostic Grifols S.A. The Systems ...

Spain

ALSTOM Gruppe

Industrial Engineering Specialist

Hace 8 días

... production lines. The role focuses on developing working instructions aligned with EHS policies and coordinating with production teams. The candidate should have an engineering degree and 3–5 years of related experience, with knowledge of Lean Manufacturing and industrialization processes to drive cost-efficient improvements.

Spain, Madrid

Senior Software Engineer, Quality Engineering

Hace 7 días

Remoto

Fully remote

About the role As a Senior Software Engineer on the Platform team, you will build and own the frameworks, tooling, and CI/CD infrastructure that let Alpaca's engineers ship new features quickly and reliably. This is a software engineering role at its core: you will spend most of your time writing Go, designing automated ...

Spain

Grupo Santander

AI Software Engineering Analyst

Hace 8 días

Remoto
Híbrido
Presencial
Tiempo completo

... professionals at all levels are expected to take a proactive and responsible approach toward risk management. THE DIFFERENCE YOU MAKECDAIO/ AI TECH is looking for a AI Software Engineering Specialistbased out of https://jobeax.com/link/slJ0s7jTHeLh4lha will be responsible for designing, developing, and implementing AI-driven solutions ...

Spain

Jobgether

Software Engineer - Solutions Engineering

Hace 13 días

Contrato

$2k USD / year

... manages all applications and next steps. Our partner is looking for a Software Engineer - Solutions Engineering based in Spain. Join a globally distributed engineering team focused on transforming how modern infrastructure is built, operated, and automated. You will develop open source software that enables efficient DevOps ...

Spain, España

Asistencia Dental Especializada (ADE)

Software Engineer - Data Engineering

Hace 13 días

Remoto

... internacional líder en ofrecer soluciones integrales a entidades aseguradoras en el ámbito de la salud y el bienestar de las personas y las mascotas, buscamos un/a Software Engineer – Data Engineering para liderar la evolución de nuestra plataforma de datos, reporting y automatización de procesos. La persona trabajará estrechamente ...

Spain

Affirm

Manager, Software Engineering, Fullstack (Repayment UX Engineering)

Hace 13 días

Remoto
Híbrido
Presencial

€103,000 - €139,000 a year

... relocation We are able to offer visa sponsorship for this role, but do require that someone is based in Spain for the role. What we look for - You have 7+ years of software engineering experience, including 3+ years managing and growing engineering teams. - You have proven experience delivering consumer-facing software and driving ...

Spain

Full Stack Software Engineering Team Lead

Hace 5 días

Remoto
Tiempo completo

€60,000 - €75,000 a year

... accountability, knowledge sharing, continuous learning, and Mitek’s values of Delivering, Learning, and Caring. What we're looking for - Significant professional software engineering experience, including experience leading technical work or mentoring other engineers. - Strong back-end development experience. Production experience ...

Spain

Thales

Intern Work-Study "Software Engineering"

Hace 13 días

Híbrido
Tiempo completo
Contrato

€16,100 a year

... of more than 100 people, most of them engineers that participate in the following activities: Competences Technical/functional skills: - Degree in Computer Engineering, Telecommunications, or Aeronautics associated with software development - Interest in the software world, software engineering processes, and a desire to ...

Spain, Madrid

Freelance Agent Evaluation Engineer

Hace 7 días

Remoto
Tiempo parcial
Contrato

Up to US$40 an hour

... tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust - Not data labeling - Not prompt engineering - Not writing code from scratch - the agent writes most of the code; you guide and evaluate What we're looking for - 5+ years in software development - Core ...

Spain

ISDIN

Software Engineer Specialist

Hace 13 días

Remoto
Híbrido

About the role The Software Engineer Specialist is a high-level individual contributor responsible for the end-to-end development of robust and scalable applications. Reporting to the Head of Solutions Architecture, this role focuses on translating complex architectural designs into high-quality code. The position requires ...

Spain, Barcelona

Asistencia Dental Especializada (ADE)

Software Engineer - Data Engineering

Hace 13 días

Remoto

Spain

Generative Audio Evaluation - Spanish (Spain)

Hace 7 días

Remoto
Tiempo parcial
Contrato

US$12 an hour

About the role We are looking for Generative Audio Evaluation Specialists ! A great entry point into ongoing work within one of our most active AI markets! Job Type: Freelance Location: Remote (Spain) Work Schedule: Part-time - 10+ hours per week. Flexible - work whenever you want! Rate: 12 USD/hour We're currently hiring ...

Spain

Mindrift

Freelance Agent Evaluation Engineer

Hace 13 días

Remoto
Tiempo parcial
Contrato

Up to US$70,000 a year

... the role We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. You'll create challenging tasks and evaluation criteria within realistic simulated environments: What this is NOT - Not data labeling - Not prompt engineering - Not writing code from scratch - the agent ...

Spain