Build and optimize production data pipelines on Databricks for enterprise customers, owning engagement components from ingestion and transformation through production quality, performance, and downstream analytics readiness.
Senior Data Engineer
📜 Description
- Design, build, and maintain scalable ETL/ELT pipelines processing petabytes of data.
- Architect data solutions using AWS, Databricks, and Snowflake.
- Develop real-time and batch workflows with Apache Airflow and optimize analytical storage and retrieval.
- Implement data quality frameworks, monitoring, automated testing, lineage, and metadata management.
- Establish data governance standards and drive CI/CD and other modern development practices.
- Lead architecture reviews, mentor junior engineers, and apply AI tools to data engineering and quality checks.
🛠️ Requirements
- 3-8 years of data engineering experience;
- Expert-level SQL - Complex queries, performance optimization, window functions;
- Advanced Python - Object-oriented programming, async/await, data processing libraries;
- Apache Airflow - DAG development, custom operators, workflow orchestration;
- Databricks - Spark optimization, Delta Lake, Unity Catalog governance;
- Cloud Platforms - AWS (S3, EMR, Lambda, Glue) or equivalent;
- Data Modeling - Dimensional modeling, data vault, modern data stack patterns;
- Experience mentoring junior engineers and leading technical initiatives;
- Strong communication skills for cross-functional collaboration;
- Experience with Agile methodologies and project management;
✨ Benefits
- 20 days of vacation;
- Public holidays;
- Mobilization reservation available upon successful completion of the probation period (for Ukrainian colleagues);
- The exchange of experience and work with talented colleagues;
- Last but not least - valuable compensation for your efforts.
Full job description
M Science is a data-driven research and analytics firm, uncovering new insights for leading financial institutions and corporations. We come to work every day to create actionable insights from alternative data. M Science is revolutionizing research, discovering new data sets and pioneering methodologies to provide actionable intelligence. Our research teams have decades of experience working with massive amounts of unstructured data in near real-time to discern critical insights that help clients make smarter, more informed decisions. We combine the best of finance, data and technology to create a truly unique value proposition for both financial services firms and major corporations.
We're looking for a Senior Data Engineer who is passionate about building robust, scalable data infrastructure and pipelines. You'll work with petabytes of data daily using cutting-edge technologies including Databricks, SQL, Python, Airflow, and modern AI development tools. This role requires someone who can architect solutions, mentor junior engineers, and drive technical excellence across our data platform.
NOTE: This role requires flexible working hours to ensure at least 2 hours of overlap with colleagues based on both the US East Coast and West Coast. Expected working schedule: 12:00–20:00 CET.
Responsibilities:
- Design, build, and maintain scalable ETL/ELT pipelines processing petabytes of data;
- Architect data solutions using modern cloud technologies (AWS, Databricks, Snowflake);
- Implement real-time and batch processing workflows using Apache Airflow;
- Optimize data storage and retrieval patterns for analytical workloads;
- Implement comprehensive data quality frameworks and monitoring systems;
- Establish data governance standards and best practices;
- Build automated testing and validation for data pipelines;
- Develop data lineage and metadata management solutions;
- Lead technical design reviews and architecture decisions;
- Mentor junior data engineers and promote best practices;
- Drive adoption of modern development practices including CI/CD for data pipelines;
- Leverage AI coding assistants (Cursor, Claude, GitHub Copilot) to accelerate development;
- Implement AI-driven data quality checks and anomaly detection;
- Stay current with emerging AI tools and their applications in data engineering.
Requirements:
- 3-8 years of data engineering experience;
- Expert-level SQL - Complex queries, performance optimization, window functions;
- Advanced Python - Object-oriented programming, async/await, data processing libraries;
- Apache Airflow - DAG development, custom operators, workflow orchestration;
- Databricks - Spark optimization, Delta Lake, Unity Catalog governance;
- Cloud Platforms - AWS (S3, EMR, Lambda, Glue) or equivalent;
- Data Modeling - Dimensional modeling, data vault, modern data stack patterns;
- Experience mentoring junior engineers and leading technical initiatives;
- Strong communication skills for cross-functional collaboration;
- Experience with Agile methodologies and project management;
- Ability to translate business requirements into technical solutions.
Will be a plus:
- Experience in financial services, fintech, or data intelligence companies;
- Background in consumer retail, digital media, or technology verticals;
- Advanced degree in Computer Science, Statistics, Mathematics, or related field.
What we offer:
- 20 days of vacation;
- Public holidays;
- Mobilization reservation available upon successful completion of the probation period (for Ukrainian colleagues);
- The exchange of experience and work with talented colleagues;
- Last but not least - valuable compensation for your efforts.
Similar jobs
Search more Data Engineer jobsSenior Engineer – Data Engineering (Databricks) - Contractor - India
Lead delivery of Databricks data engineering solutions, designing and optimizing scalable batch and streaming pipelines while contributing to data platform architecture and reusable solution accelerators.
Senior Data Engineer (Databricks) - Brazil
Lead Databricks data engineering work to design and implement scalable lakehouse platforms and batch and streaming pipelines, while contributing to solution architecture, data quality, and Databricks accelerators.
Build and maintain Celonis data models, integrations, and pipelines for process intelligence and automation, while defining technical standards and guiding delivery for an automotive client.
Design, build, and maintain reliable, scalable data pipelines and infrastructure, partnering with analysts and stakeholders to support analytics and business initiatives in banking.
