Senior Data Engineer (Cloud)

Job description

About us

Who we are? We are Big Data experts, working with international clients, creating and leading innovative projects related to the Big Data environment. We offer tailor-made solutions. It does not matter whether we are talking about building a Data Lake, conducting training in the field of data management, or performing detailed Big Data analysis. We don’t just focus on one technology, instead, we specialize in a whole range of open-source and public cloud tools. Our team brings together over 130 specialists in their fields. We have participated in dozens of conferences, and written countless amounts of code, we are the organizers of Big Data Tech Summit Warsaw, the largest Polish conference related to Big Data topics. We run webinars, share knowledge on blogs, create whitepapers, and more. Why? Because we believe that Big Data is an indispensable future of business.
Thanks to that, we always select the most optimal Big Data solutions.


Customer

We are working on the project with a startup-like organization that builds a trusted, safe, and highly-popular mobile communication app. It provides a lot of features with positive social impacts like protecting people from fraud, harassment, and scams, calls, and messages, identifying new numbers from anywhere in the world. The app is used by over 250 million people around the world and is growing very rapidly across regions. This generates petabytes of data, almost 3M events per second, and a lot of questions to be answered with data with the use of a mix of scalable open-source and cloud (GCP) technologies.

Project

As a Senior Data Engineer in the data platform team, you will play a crucial role in implementing, improving, and maintaining the Data platform and core data pipelines in our client’s organization. This includes partnering and consulting with business and analytics teams to provide guidance and assistance in creating and streamlining ETL pipelines and building data tools. You’ll help develop and propagate working with data best practices across the company.


Responsibilities

  • Maintaining and improving the ingestion pipeline to reliability deliver billions of events daily in defined SLA.

  • Provide support for all teams in building and optimizing their complex pipelines

  • Work closely with other teams to identify pain points and problems around the platform

  • Develop new tools and frameworks to improve the data platform

  • Facilitate the company-wide to be data-driven

  • Work in close collaboration with data scientists and data analysts to help support their work going to production

  • Work with vast projects such as building an ML platform and streaming use cases

  • Setup best practices and processes around software and data development

Technologies used:

  • Spark
  • BigQuery
  • Flink
  • Airflow
  • Kubernetes
  • GCP
  • Kubeflow
  • Kafka

Requirements

  • Experience working with orchestration tools (ex: Airflow)
  • Experience working in a big data environment (ex: Hadoop)
  • Experience building complex ETL pipelines
  • Experience working with cloud providers such as GCP, AWS, or Azure
  • Strong understanding of Software Engineering practices and principles
  • Have worked on building CI/CD pipelines
  • Excellent problem-solving and communication skills
  • Self-motivated and have a proven ability to take initiative to own the problems that come up and solve them 
  • Strong programming skills in Spark with Scala and Python
  • Deep understanding of internal Spark with experience in optimizing spark jobs

Nice to have:

  • Hands-on experience on the Google Cloud Platform
  • Working with messaging systems like Kafka
  • Knowledge of Kubernetes
  • Experience managing data warehouse in BigQuery
  • Have implemented or worked with microservices architecture
We offer
  • Salary: 150-190 PLN net + VAT/h B2B (depending on knowledge and experience)

  • 100% remote work

  • Elastic working hours

  • Possibility to work from the office located in the heart of Warsaw

  • Opportunity to learn and develop with the best Big Data specialists in Poland

  • International projects

  • Possibility of conducting workshops and training.

  • Clear career path and certifications

  • Co-financing sport card

  • Co-financing health care

  • All equipment needed for work