Data Engineering

About the role :

As a Software Engineer, you will work on teams building a variety of big data analytics solutions including big data lakes. More specifically, you will work on:

  • Scalable data ingestion pipelines to handle real time streams, CDC events, and batch data
  • High-performance data processing for structured and unstructured data, and data harmonization
  • Scheduling, orchestrating, and validating pipelines
  • Exception handling and log monito ring for debugging
  • Collaborate with business consultants, data scientists, engineers, and application developers to develop analytics solutions.

Required Experience, Skills & Competencies:

  • Hands-on experience with: Hadoop ecosystem - HDFS, Hive, Sqoop, Kafka, ELK Stack etc Spark, Scala, Python and core/advance Java NOSQL databases
    e.g. Hbase, Cassandra, MongoDB Relevant AWS or Azure components required to build big data solutions
  • Relevant AWS or Azure components required to build big data solutions
  • Ability to develop and manage scalable Hadoop cluster environments
  • Good understanding of data warehousing concepts, distributed systems, data pipelines , ETL
  • Good to know: Databricks, Snowflake.