Full-Stack Data Engineer
Job Description
We are seeking a Full-Stack Data Engineer to design, develop, and manage scalable data pipelines, storage, and transformation solutions. This role requires expertise in cloud-based data platforms, data warehouse / data lake house design and development, workflow automation, and data integration to support business intelligence and analytics. The ideal candidate will have a strong background in data engineering, cloud technologies, and software development, with a focus on performance, security (especially data segregation), and automation.
Key Responsibilities
1. Data Platform Design & Implementation
· Architect and deploy scalable, secure, and high-performing Snowflake environments in line with data segregation policies.
· Automate infrastructure provisioning, testing, and deployment for seamless operations.
2. Data Integration & Pipeline Development
· Develop, optimize, and maintain data pipelines (ETL/ELT) to ensure efficient data ingestion, transformation, and migration.
· Implement best practices for data consistency, quality, and performance across cloud and on-premises systems.
3. Data Transformation & Modeling
· Design and implement data models that enable efficient reporting and analytics.
· Develop data transformation processes using Snowflake, DBT, and Python to enhance usability and accessibility.
4. Networking, Security & Compliance
· Configure and manage secure network connectivity for data ingestion.
· Ensure compliance with GDPR, CISO policies, and industry security standards.
5. Data Quality & Governance
· Ensure the Data Segregation Policy is firmly followed for the data sets enabled.
· Implement data validation, anomaly detection, and quality assurance frameworks.
· Collaborate with the Data Governance team to maintain compliance and integrate quality checks into data pipelines.
6. Real-Time & Batch Data Processing
· Build and optimize real-time streaming and batch processing solutions using Kafka, Kinesis, or Apache Airflow.
· Ensure high-throughput, low-latency data processing efficiency.
7. Stakeholder Collaboration & Business Alignment
· Work closely with business stakeholders, analysts, and data teams to deliver tailored solutions.
· Translate complex technical insights for both technical and non-technical audiences.
8. Performance Optimization & Continuous Improvement
· Identify and implement automation, cost optimization, and efficiency improvements in data pipelines.
Qualifications & Experience
Education:
· Bachelor’s degree in Computer Science, Data Science, Information Systems, or related field (Master’s preferred).
Experience:
· 5+ years in data engineering, analytics engineering, or related fields.
· Proven experience with Snowflake, DBT, Snaplogic, and modern data technologies.
· Expertise in cloud data platforms, real-time & batch processing, and CI/CD automation.
· Strong background in data modeling, architecture, and cost-efficient pipeline management.
· Experience in deploying data segregation policies, especially logical segregation. Ideally being able to design these policies or support the teams with the design.
Technical Skills
· Data Engineering & Warehousing: Snowflake (must have), DBT (must have), Snaplogic, ETL/ELT, APIs, Data Warehousing & Lakehouse Architecture.
· Programming & Scripting: Advanced SQL, Python, DBT, Bash/Shell scripting.
· Cloud & Infrastructure: AWS/Azure/GCP, Terraform, CloudFormation, Security (IAM, VPN, Encryption).
· Data Processing: Kafka, Kinesis, Apache Airflow, Dagster, Prefect.
· DevOps & CI/CD: Git, GitHub Actions, Jenkins, Docker, Kubernetes.
· Data Governance & Quality: Data validation, metadata management, GDPR, CCPA compliance.
