Vi manages a petabyte scale data lakehouse that drives data and ML pipelines across all our products. Vi is looking for an engineer with deep expertise building and scaling big-data and ML pipelines. The role will be a hybrid between an FDE that owns integrations of customer data with Vi’s data lakehouse and a platform engineer responsible for packaging Vi’s products into repeatable, scalable, automated pipelines.
Tech Stack
Apache PySpark, Apache Iceberg, AWS EMR, Glue, S3
Python, PyTorch, sklearn, xgboost, catboost, mlflow
AWS SageMaker, Airflow
What you'll do
Own the design and implementation of data and ML pipelines based on a combination of customer first party data and Vi’s internal data lakehouse.
Prototype predictive insight products and run rapid feasibility studies to assess new commercial opportunities for Vi’s capabilities built on petabyte scale data lakehouse.
Identify commonalities between customer engagements to synthesize reusable data capabilities.
What We’re Looking For
Deep expertise building large-scale data analytics pipelines with Apache Spark.
Comfortable using Python and AWS technologies for data engineering and ML.
Strong communication skills to interact with technical counterparts across customer accounts to coordinate data integrations.
Nice to have
Experience in the healthcare and life sciences domain.
Familiarity with ML and statistical modeling methodologies and experimental design.