Data engineer

Software Engineering
654636

If you love working on complicated network pipelines, you understand the importance of reliable data and have felt the pain of big data inconsistencies, and you're the type who thinks of great solutions and want to bring them to life, BigaBid is your best challenge.

Bigabid is afast-growing startup founded in 2015 by an experienced team of serialentrepreneurs and backed by some of the most prominent angel investors inIsrael.

 

One of our goals at Bigabid is to disrupt the $200B App Marketing industry with an innovative user-Acquisition and Re-engagement platform powered by machine learning.

Our platform helpsleading App developers worldwide to acquire new loyal users for their business,whether It’s Games, e-Commerce or other utility Apps.  

 

As a data engineer youwill be working on our massive (PBs) data pipeline, making sure the data isclean, whole, and accessible. Your team's goal is to create amazinggroundbreaking tools to make the data scientists more productive and agile. Ifyou love working on complicated network pipelines, you understand theimportance of reliable data and have felt the pain of big data inconsistencies,and you're the type who thinks of great solutions and want to bring them tolife, BigaBid is your best challenge.

 

Responsibilities:

·      Create andmaintain optimal data pipeline architecture

·      Build and maintain our `featurestore` and machine learning orchestration mechanism

·      Identify,design, and implement internal process improvements: automating manualprocesses, optimizing data delivery, re-designing infrastructure for greaterscalability, etc.

·      Build theinfrastructure required for optimal extraction, transformation, and loading ofdata from a wide variety of data sources using SQL, Spark and AWS ‘big data’technologies.

·      Work withstakeholders including the executive, product and marketing teams to assistwith data-related technical issues and support their data infrastructure needs.

·      Create datatools for analytics and data scientist team members

·      Work withdata and analytics experts to strive for greater functionality in our datasystems

 

Required Skills:

·      5+ Yearscoding (preferably Python)

·      2+ Yearsexperience with big data tools: Hadoop, Spark, Kafka, Presto, EMR etc.

·      Experiencebuilding and optimizing ‘big data’ data pipelines; including - message queuing,stream processing, and highly scalable data sets

·      Experienceperforming root cause analysis on internal and external data and processes.

·      Strongorganizational skills with ability to juggle multiple tasks within constraintsand timelines

 

Preferred qualifications:

·      Experiencewith Airflow or other workflow management software

·      Familiar withthe Linux environment and bash scripting

·      Familiar withMachine Learning techniques

Apply Now