You are viewing a preview of this job. Log in or register to view more details about this job.

Data Engineer

Contriva Consulting Inc
Email

Job Brief

We are looking for an experienced Data Engineer / Python Developer with 7-9 years of hands-on experience in developing data pipelines, backend applications, and scalable data processing solutions. The ideal candidate should have strong expertise in Python, SQL, ETL/ELT, PySpark, cloud platforms, and data engineering frameworks.

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using Python and SQL.
  • Develop Python-based applications, scripts, APIs, and automation solutions.
  • Build and optimize data processing pipelines using PySpark/Apache Spark.
  • Work with relational and NoSQL databases for data ingestion, transformation, and storage.
  • Implement data quality, validation, monitoring, and error-handling processes.
  • Develop and maintain workflows using tools such as Airflow, Azure Data Factory, or Databricks.
  • Integrate data from APIs, databases, files, and cloud-based sources.
  • Optimize SQL queries and data pipelines for performance and scalability.
  • Collaborate with data scientists, analysts, software engineers, and business stakeholders.
  • Follow CI/CD, Git, Agile, and data engineering best practices.

 

Required Skills

  • 4–5 years of professional experience in Data Engineering/Python Development.
  • Strong proficiency in Python and SQL.
  • Hands-on experience with Pandas, NumPy, PySpark, and REST APIs.
  • Strong understanding of ETL/ELT, data warehousing, data modeling, and database concepts.
  • Experience with Apache Spark and distributed data processing.
  • Experience with at least one cloud platform: AWS, Azure, or GCP.
  • Knowledge of Databricks, Airflow, Azure Data Factory, or similar orchestration tools.
  • Experience with PostgreSQL, MySQL, SQL Server, Snowflake, or similar databases.
  • Strong knowledge of Git, CI/CD, testing, and Agile methodologies.

Tools & Technologies:

Python, SQL, PySpark, Pandas, NumPy, Apache Spark, Airflow, Databricks, Azure Data Factory/Microsoft Fabric, AWS/Azure/GCP, PostgreSQL, Snowflake, REST APIs, Git, Docker, CI/CD.

Preferred Certifications: If the candidate has any of these certifications, it'll be good

  • Microsoft Certified: Fabric Data Engineer Associate (DP-700)
  • Databricks Certified Data Engineer Associate
  • Google Cloud Professional Data Engineer
  • AWS Certified Data Engineer – Associate