Data Engineer (SQL, Pyspark, Databricks)

Summary
We 're looking for a dynamic Data Engineer with Apache Spark and AWS experience to join the data analytics team at GainwellTechnologies.You will have the opportunity towork as part of a cross:functional team to define, design and deploy frameworks for data collection, normalization, transformation, storage and reportingon AWStosupport the analytic missions of Gainwelland its clients
Your role in our mission
:
Design, develop and deploy data pipelines including ETL:processes for getting, processing and delivering data using Apache Spark Framework.
:
Monitor, manage, validate and test data extraction, movement, transformation, loading, normalization, cleansing and updating processes. Build complex databases that are useful, accessible, safe and secure.
:
Coordinates with users to understand data needs and delivery of data with a focus on data quality, data reuse, consistency, security, and regulatory compliance.
:
Collaborate with team:members on data models and schemas in our data warehouse.
:
Collaborate with team:members on documenting source:to:target mapping.
:
Conceptualize and visualize data frameworks
:
Communicate effectively with various internal and external stakeholders.
What were looking for
:
Bachelors degree in computer sciences or related field
:
3years of experience working with big data technologies on AWS/Azure/GCP
:
2years of experience in the Apache Spark/DataBricks framework (Python/Scala)
:
Experience working with different between database structures (e.g., transaction based vs. data warehouse)
:
Databricks and AWS developer/architect certifications a big plus
What you should expect in this role
:
Design, develop and deploy data pipelines including ETL:processes for getting, processing and delivering data using Apache Spark Framework.
:
Monitor, manage, validate and test data extraction, movement, transformation, loading, normalization, cleansing and updating processes. Build complex databases that are useful, accessible, safe and secure.
:
Coordinates with users to understand data needs and delivery of data with a focus on data quality, data reuse, consistency, security, and regulatory compliance.
:
Collaborate with team:members on data models and schemas in our data warehouse.
:
Collaborate with team:members on documenting source:to:target mapping.
:
Conceptualize and visualize data frameworks
:
Communicate effectively with various internal and external stakeholders.
Job Type: Permanent