Lead Data Engineer

Our company:

    We are seeking a highly skilled Lead Data Engineer to design, implement, and maintain data solutions on the Microsoft Azure Data platform and SQL Server (SSIS, SSAS, UC4 Atomic). The selected candidate will lead a team of data engineers, collaborate with various stakeholders, and ensure the efficient processing, storage, and retrieval of large volumes of data. This role requires both technical expertise and leadership capabilities to drive data engineering projects aligned with business goals.

What we are looking from an ideal candidate?

    • Design, build, and maintain scalable and reliable data pipelines.
    • Develop and maintain solutions in Azure Data Factory and Databricks to extract, transform, and load data into different source and target systems.
    • Design and build solutions in SSIS.
    • Analyze and understand the existing data landscape, providing recommendations for rearchitecting, optimizing, and streamlining to improve efficiency and scalability.
    • Collaborate with onshore counterparts to address technical gaps, requirements challenges, and other complex scenarios.
    • Monitor and troubleshoot data systems to ensure high performance and reliability.
    • Utilize strong analytical skills and attention to detail, with extensive familiarity with database management principles.
    • Optimize data processes for speed and efficiency.
    • Ensure the data architecture supports business requirements and data governance policies.
    • Define and execute the data engineering strategy in alignment with the company’s goals.
    • Integrate data from various sources, ensuring data quality and consistency.
    • Stay updated with emerging technologies and industry trends.
    • Translate big picture business processes, utilizing deep knowledge in the banking industry, into data requirements.
    • Enable and manage data migrations across different databases and servers.
    • Perform thorough testing and validation to ensure the accuracy of data transformations and data verification used in machine learning models.
    • Define data mapping by working with business, digital, and data teams.
    • Maintain, test, and validate data pipelines for performance.
    • Assemble large, complex data sets that meet functional and non-functional business requirements.
    • Analyze and identify gaps in data needs, working with business and IT to align on data requirements.
    • Understand the impact of data conversions on servicing operations.
    • Manage higher volume and more complex cases with accuracy and efficiency.

Preferred Skills:

What skills do you need?

    Leadership Responsibilities:

    • Provide technical direction and guidance to the development team.
    • Review and approve code, ensuring it meets quality standards and best practices.
    • Troubleshoot and resolve technical issues as they arise.
    • Foster a culture of collaboration, innovation, and continuous improvement within the team.
    • Optimize data flow and collection for cross-functional teams.
    • Work closely with data counterparts onshore, product owners, and business stakeholders to understand data needs and strategies.
    • Collaborate with IT and DevOps teams to ensure data infrastructure aligns with overall IT architecture.
    • Implement best practices for data security and privacy.
    • Drive continuous improvement initiatives within the data engineering function.

    Candidate Expectations:

    • Design and develop warehouse solutions using Azure Synapse Analytics, ADLS, ADF, Databricks, Power BI, and Azure Analysis Services.
    • Proficient in SSIS, SQL, and query optimization.
    • Experience as a technical lead, mentoring data engineers and driving data-oriented projects.
    • Experience in an onshore-offshore model, managing challenging scenarios.
    • Lead data architecture and implementation efforts for optimal accuracy and usage.
    • Expertise in working with large amounts of data (structured and unstructured), building data pipelines for ETL workloads, and generating insights using data science and analytics.
    • Expertise in Azure, AWS cloud services, and DevOps/CI/CD frameworks.
    • Ability to work with ambiguity and vague requirements, transforming them into deliverables.
    • Strong technical and interpersonal skills with excellent written and verbal communication; detail-oriented with the ability to work independently.
    • Drive automation efforts using Infrastructure as Code (IaC) with tools like Terraform, and CI/CD tools such as Jenkins.
    • Help define architecture frameworks, best practices, and processes. Collaborate on data warehouse architecture and technical design discussions.
    • Expertise in Azure Data Factory, building pipelines for ETL projects.
    • Strong SQL knowledge and experience working with relational databases.
    • Proficient in Python and ETL projects.
    • Experience with Databricks is advantageous.
    • Expertise in data lifecycle, data ingestion, transformation, loading, validation, and performance tuning.