🕒 6 days ago
Design and maintain scalable Azure Databricks data pipelines, strengthen data quality and monitoring, and use AI-assisted tools to automate code reviews, compliance checks, and routine engineering work.
la26.042 - Data Engineer
📅 Mar 18
Azure DatabricksMedallion ArchitectureCloud Data PipelinesPyspark
📜 Description
- Create, deploy, and maintain production-grade data pipelines.
- Develop and manage Lakehouse pipelines across Bronze, Silver, and Gold layers.
- Optimize Delta Lake performance and query costs using features such as Z-Ordering, liquid clustering, vacuuming, and partitioning.
- Deploy cloud infrastructure and automate pipeline releases using Terraform, Git, and Azure DevOps.
- Implement data governance, access controls, and lineage tracking with Unity Catalog.
- Monitor and tune distributed Spark clusters, including sizing, memory allocation, and execution plans.
🛠️ Requirements
- Over four years of experience with modern data stack technologies, including Azure Databricks and Medallion Architecture.
- Experience designing, developing, and optimizing scalable cloud-based data pipelines.
- Proficiency with PySpark, Azure Data Factory, Delta Lake, Terraform, Git, Azure DevOps, and Unity Catalog.
- Proficiency in spoken and written English.
✨ Benefits
- A collaborative and innovative work environment.
- Opportunities to work on impactful projects with global clients.
- Competitive salary and benefits tailored to mid to senior professionals.
- Continuous learning and career development opportunities.
- Flexible working arrangements based in Mexico.
Full job description
Tiger Analytics is a leading AI and advanced analytics consulting firm dedicated to delivering impactful data-driven solutions to Fortune 100 companies worldwide. We specialize in helping organizations solve complex business challenges through innovative Data Engineering, Machine Learning, and AI techniques. We are currently seeking a highly skilled Data Engineering Lead to manage and lead complex data engineering projects, driving excellence and innovation in our delivery.
Requirements
• We are seeking a highly experienced Data Engineer with over four years of expertise in modern data stack technologies, specifically focusing on Azure Databricks and the Medallion Architecture encompassing the Bronze, Silver, and Gold processing layers.
• Proven experience in designing, developing, and optimizing scalable, cloud-based data pipelines is essential.
• Pipeline Development & Orchestration: Responsible for creating, deploying, and maintaining production-grade data pipelines utilizing Azure Databricks, PySpark, and Azure Data Factory (ADF).
• Medallion Framework Management: Accountable for developing and managing the comprehensive Lakehouse pipeline architecture across the Bronze, Silver, and Gold layers.
• Delta Lake Optimization: Expertise in applying advanced Delta Lake features such as Z-Ordering, liquid clustering, vacuuming, and partitioning techniques to minimize query costs and improve read/write efficiency.
• Infrastructure as Code & CI/CD: Collaborate in managing and deploying cloud infrastructure using Terraform, Git, and Azure DevOps, automating the deployment of data pipelines.
• Data Governance & Security: Implement enterprise-level data governance frameworks, enforce access controls, and manage data lineage tracking utilizing Unity Catalog.
• Performance Tuning: Monitor, analyze, and address performance bottlenecks in distributed Spark clusters by optimizing cluster sizing, memory allocation, and execution plans.
• Proficiency in English, both spoken and written, is required for effective communication.
• Working hours are from 8:00 AM to 5:00 PM PST.
Benefits
- A collaborative and innovative work environment.
- Opportunities to work on impactful projects with global clients.
- Competitive salary and benefits tailored to mid to senior professionals.
- Continuous learning and career development opportunities.
- Flexible working arrangements based in Mexico.
Similar jobs
Search more Data Engineer jobs
🕒 4 days ago
Design and build scalable data pipelines using Python, Apache Spark, and Databricks, while modernizing legacy systems and ensuring data quality and reliability.
🕒 4 days ago
Build and maintain batch and streaming data pipelines, ETL workflows, and data warehouse solutions, improving data quality, reliability, and performance through Python, SQL, BigQuery, and Airflow.
Mid-Level Data Engineer (Ontologist)
🕒 5 days ago
CESAR👥 1001 - 5000 employees🏢 Information Technology And Services
🕒 5 days ago
Desenvolve pipelines de dados e modela ontologias, taxonomias e grafos de conhecimento, conectando engenharia de dados, recuperação de informação e IA generativa para viabilizar ingestão e inferência por máquinas.
🕒 7 days ago
Build and optimize cloud-native data platforms and distributed pipelines for scientific research, analytics, and machine learning, with a focus on data quality, governance, reliability, and cost efficiency.