AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you’re looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
We are looking for a Data Engineer with strong Databricks and GCP experience to develop scalable ETL/ELT pipelines and productionize agentic workflows for data engineering automation.
MUST HAVES
5+ years of strong hands-on experience with Databricks and PySpark.
Advanced SQL and data-processing skills.
Hands-on experience with GCP , particularly BigQuery.
Experience with Delta Lake and modern data lake/lakehouse architectures.
Strong understanding of ETL/ELT, data pipeline design, performance optimization, and data quality.
Experience building reliable, scalable, production-grade data solutions.
Strong analytical and troubleshooting skills.
Understanding of software engineering practices, including testing, version control, deployment, monitoring, and production support.
NICE TO HAVES
Experience developing or integrating AI/agentic workflows, AI agents, or workflow automation solutions.
Experience applying AI to automate data engineering, validation, troubleshooting, or operational processes.
Familiarity with orchestration and automation frameworks.
Experience developing reusable data engineering frameworks and platform components.
Exposure to productionizing AI-enabled solutions with appropriate validation, monitoring, and human oversight.
WHAT YOU WILL DO
Design, develop, and optimize scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
Build and operationalize agentic workflows to automate data engineering and operational processes such as data validation, issue identification, troubleshooting, and workflow execution.
Integrate agentic capabilities with existing Databricks, GCP, BigQuery, and Delta Lake environments.
Develop data pipelines and processing solutions to support new business requirements and datasets.
Build reusable frameworks and components that can be leveraged across multiple data engineering and business use cases.
Implement data quality checks, monitoring, validation, exception handling, and production controls.
Optimize PySpark and SQL workloads for performance, reliability, and scalability.
Support testing, deployment, productionization, and ongoing enhancement of data and agentic solutions.
Troubleshoot complex data and production issues and implement sustainable solutions.
Collaborate with business, data engineering, and platform teams to identify further automation opportunities.
PERKS AND BENEFITS
Professional growth : Mentorship, TechTalks, and personalized growth roadmaps.
Competitive compensation : USD-based pay with education, fitness, and team activity budgets.
Exciting projects : Modern solutions with Fortune 500 and top product companies.
Flextime : Flexible schedule with remote and office options.
Meet Our Recruitment Process
Each step helps us understand your skills and overall fit.
Data Engineer - Málaga | KRELL Consulting Buscamos incorporar un/a Data Engineer para participar en un proyecto de integración de plataformas, trabajando en el desarrollo y mantenimiento de soluciones de datos escalables y eficientes. Buscamos un perfil con experiencia en Python, Spark y Apache Airflow, valorándose especialmente ...
What you’ll do - Build the data-serving layer: curated datasets, marts, and product-ready tables - Develop incremental / micro-batch pipelines and support CDC near-real-time ingestion (AWS DMS) - Design BI-friendly data models (star schema) and manage schemas - Build ETL/ELT in Python (Polars) and serve/query via Athena ...
... are: - Migrate the existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service. - Implement data transformation and quality to ensure data consistency and accuracy. Utilize programming languages such as Scala and SQL and tools like Spark for data transformation ...
Overview Importante consultora en tecnología busca un Data Engineer con mínimo 3 años de experiencia como desarrollador y mínimo 1 año de experiencia trabajando en proyectos de Big Data. Requisitos técnicos - Experiencia con las tecnologías Spark, YARN, Hadoop, Hive. - Experiencia con bases de datos relacionales (MySQL). ...
... the team alongside you, and grow into more of the platform as you go. You'll also ship with agentic AI tools as a real part of how we work — this is AI-native data engineering. If you like owning data end to end, from a Pub/Sub event to a Looker explore, and want scale you can't get just anywhere (hundreds of millions of ...
We are looking for an experienced Data Engineer with a strong technical background in data engineering, data architecture, and data pipeline development. The successful candidate will work within an Agile team to design, develop, test, and deploy data solutions that meet business requirements while adhering to the client's ...
... capacidades: - Experiencia mínima de 3 años como Data Engineer o en roles similares. - Conocimiento sólido del ecosistema de datos Microsoft Azure y Microsoft Fabric: Data Factory, Synapse, Databricks, Fabric (Lakehouse/Warehouse), Power BI y Purview. - Experiencia con servicios de almacenamiento en Azure: Data Lake Storage Gen2, ...
... cloud-based data services (AWS, Azure, or GCP). - Solid understanding of Data Lakes, Data Warehouses, and modern data architectures. - Experience with Python data engineering and analytics libraries, such as Pandas, NumPy, and PySpark and Spark. - Hands-on experience with Databricks, including data engineering, analytics, ...
... importance of our data is growing fast, and we're looking for a Data Engineer to build the infrastructure that keeps up. Your profile You'll design and build the data infrastructure that carries Bookboost from hundreds of gigabytes to terabytes of guest data, moving, transforming and making it available to our frontend application ...
Misión del rol El rol de Data Engineer es eminentemente técnico y de ejecución. Buscamos a alguien que disfrute escribiendo código, construyendo pipelines robustos y resolviendo con velocidad las necesidades cambiantes del negocio. El Data Engineer será el propietario técnico de Azure Data Factory (ADF) y dbt — las herramientas ...