We are seeking an experienced Data Engineer with deep expertise in Databricks to design, build, and maintain scalable data pipelines and analytics solutions. This role requires 5 years of hands-on experience in data engineering with a strong focus on the Databricks platform.
Key Responsibilities:
- Data Pipeline Development & Management
- Design and implement robust, scalable ETL/ELT pipelines using Databricks and Apache Spark
- Process large volumes of structured and unstructured data
- Develop and maintain data workflows using Databricks workflows, Apache Airflow, or similar orchestration tools
- Optimize data processing jobs for performance, cost efficiency, and reliability
- Implement incremental data processing patterns and change data capture (CDC) mechanisms
Databricks Platform Engineering:
- Build and maintain Delta Lake tables and implement medallion architecture (bronze, silver, gold layers)
- Develop streaming data pipelines using Structured Streaming and Delta Live Tables
- Manage and optimize Databricks clusters for various workloads
- Implement Unity Catalog for data governance, security, and metadata management
- Configure and maintain Databricks workspace environments across development, staging, and production
Data Architecture & Modeling:
- Design and implement data models optimized for analytical workloads
- Create and maintain data warehouses and data lakes on cloud platforms (Azure, AWS, or GCP)
- Implement data partitioning, indexing, and caching strategies for optimal query performance
- Collaborate with data architects to establish best practices for data storage and retrieval patterns
Performance Optimization & Monitoring:
- Monitor and troubleshoot data pipeline performance issues
- Optimize Spark jobs through proper partitioning, caching, and broadcast strategies
- Implement data quality checks and automated testing frameworks
- Manage cost optimization through efficient resource utilization and cluster management
- Establish monitoring and alerting systems for data pipeline health and performance
Collaboration & Best Practices:
- Work closely with data scientists, analysts, and business stakeholders to understand data requirements
- Implement version control using Git and follow CI/CD best practices for code deployment
- Document data pipelines, data flows, and technical specifications
- Mentor junior engineers on Databricks and data engineering best practices
- Participate in code reviews and contribute to establishing team standards
Required Qualifications
Experience & Skills:
- 5+ years of experience in data engineering with hands-on Databricks experience
- Strong proficiency in Python and/or Scala for Spark application development
- Expert-level knowledge of Apache Spark, including Spark SQL, DataFrames, and RDDs
- Deep understanding of Delta Lake and Lakehouse architecture concepts
- Experience with SQL and database optimization techniques
- Solid understanding of distributed computing concepts and data processing frameworks
- Proficiency with cloud platforms (Azure, AWS, or GCP) and their data services
- Experience with data orchestration tools (Databricks Workflows, Apache Airflow, Azure Data Factory)
- Knowledge of data modeling concepts for both OLTP and OLAP systems
- Familiarity with data governance principles and tools like Unity Catalog
- Understanding of streaming data processing and real-time analytics
- Experience with version control systems (Git) and CI/CD pipelines
Preferred Qualifications
- Databricks Certified Data Engineer certification (Associate or Professional)
- Experience with machine learning pipelines and MLOps on Databricks
- Knowledge of data visualization tools (Power BI, Tableau, Looker)
- Experience with infrastructure as code (Terraform, CloudFormation)
- Familiarity with containerization technologies (Docker, Kubernetes)