Build and maintain batch and streaming data pipelines, ETL workflows, and data warehouse solutions, improving data quality, reliability, and performance through Python, SQL, BigQuery, and Airflow.
GenAI Data Engineer
๐ Description
- Design and maintain pipelines to ingest, process, and store unstructured data at scale.
- Use Snowflake Directory Tables, Scoped URLs, and Snowpark to make unstructured data queryable.
- Build cloud-native ETL/ELT workflows with BigQuery, Cloud Storage, and Dataflow, integrating GCP with Snowflake.
- Integrate OCR, NLP entity extraction, and Document AI into data workflows to produce structured insights.
- Tune SQL queries and Python processing jobs for petabyte-scale environments.
๐ ๏ธ Requirements
- Design and implement robust data pipelines that ingest, process, and store unstructured data formats at scale within Snowflake and GCP .
- Leverage Snowflake's unstructured data capabilities (Directory Tables, Scoped URLs, Snowpark) to make "dark data" queryable and actionable.
- Build and maintain cloud-native ETL/ELT processes using BigQuery, Cloud Storage, and Dataflow, ensuring seamless integration between GCP and Snowflake.
- Instead of just using LLMs, you will integrate AI tools (OCR, NLP entities, Document AI) into the engineering flow to transform unstructured blobs into structured insights.
- Tune complex SQL queries and Python-based processing jobs to handle petabyte-scale environments efficiently.
Full job description
Tiger Analytics is a fast-growing advanced analytics consulting firm. Our consultants bring deep expertise in Data Science, Machine Learning and AI. We are the trusted analytics partner for several Fortune 100 companies, enabling them to generate business value from data. Our business value and leadership has been recognized by various market research firms, including Forrester and Gartner. We are looking for top-notch talent as we continue to build the best analytics global consulting team in the world.
We are seeking an experienced Data Engineer with expertise in Dataiku to join our data team. As a Data Engineer, you will be responsible for designing, building, and maintaining data pipelines, data integration processes, and data infrastructure. You will collaborate closely with data scientists, analysts, and other stakeholders to ensure efficient data flow and support data-driven decision making across the organization.
Requirements
- Design and implement robust data pipelines that ingest, process, and store unstructured data formats at scale within Snowflake and GCP.
- Leverage Snowflakeโs unstructured data capabilities (Directory Tables, Scoped URLs, Snowpark) to make "dark data" queryable and actionable.
- Build and maintain cloud-native ETL/ELT processes using BigQuery, Cloud Storage, and Dataflow, ensuring seamless integration between GCP and Snowflake.
- Instead of just using LLMs, you will integrate AI tools (OCR, NLP entities, Document AI) into the engineering flow to transform unstructured blobs into structured insights.
- Tune complex SQL queries and Python-based processing jobs to handle petabyte-scale environments efficiently.
Benefits
Significant career development opportunities exist as the company grows. The position offers a unique opportunity to be part of a small, challenging, and entrepreneurial environment, with a high degree of individual responsibility.
Similar jobs
Search more Data Engineer jobsBuild reusable Python data ingestion and real-time streaming pipelines that connect customer systems with PostgreSQL and Snowflake, while improving schema mapping, onboarding, and production reliability.
Design and maintain scalable Azure Databricks data pipelines, strengthen data quality and monitoring, and use AI-assisted tools to automate code reviews, compliance checks, and routine engineering work.
Design and maintain reliable data pipelines, warehouse models, and curated datasets that transform complex life sciences and healthcare data into trusted resources for analytics and decision-making.
Own the operational backbone of a finance data platform, provisioning Azure and Snowflake environments, engineering secure CI/CD, monitoring replication and platform health, and governing costs through handoff.
