Senior Data Engineer (GCP)

Job description

About us

Who we are? We are Big Data experts, working with international clients, creating and leading innovative projects related to the Big Data environment. We offer tailor-made solutions. It does not matter whether we are talking about building a Data Lake, conducting training in the field of data management, or performing detailed Big Data analysis. We don’t just focus on one technology, instead we specialize in a whole range of open-source and public cloud tools. Our team brings together over 130 specialists in their fields. We have participated in dozens of conferences, written countless amounts of code, we are the organizers of Big Data Tech Summit Warsaw, the largest Polish conference related to Big Data topics. We run webinars, share knowledge on blogs, creating whitepapers and more. Why? Because we believe that Big Data is an indispensable future of business.

Thanks to that, we always select the most optimal Big Data solutions.

Customer

We are working with one of the Polish eCommerce leaders who plan to fully migrate to GCP from on-premise Hadoop till the end of 2022. This initiative is delivered in phases, and currently, the need is to migrate the data from the existing Oracle system to a Google Cloud Platform, without the mediation of Hadoop.


Project

Data migration from Oracle system to Google Cloud Platform (BigQuery, DataProc, Cloud Composer), without the mediation of Hadoop.

Responsibilities

  • Spark Job adaptation during migration to GCP
  • Performance optimization for large data volumes (over 1kkk rows)
  • Refactoring of the business logic within the queries per needs
  • Cost optimization

Technologies used:

  • GCP (BigQuery, DataProc, Cloud Composer)
  • Apache Spark
  • Apache Airflow
  • Scala
  • Python
  • Terraform

Requirements

  • Proficiency with Google Cloud Platform
  • Experience in Scala
  • Good knowledge of BigQuery
  • Good software development practices e.g. clean code, CI/CD
  • Experience and ideally contribution to open source frameworks like Spark

We offer

  • Salary : 150-190 PLN net + VAT/h B2B (depending on knowledge and experience)
  • 100% remote work with autonomy, trust and transparency
  • Elastic working hours
  • Possibility to work from the office located in the heart of Warsaw
  • Opportunity to learn and develop with the best Big Data specialists in Poland
  • International projects
  • Possibility of conducting workshops and training
  • Clear career path and certifications
  • Co-financing sport card
  • Co-financing health care
  • All equipment needed for work