Key Notes
The Data Transfer Hub is designed to move partner video, LiDAR and sensor data at up to petabyte-per-day scale. The engineer builds distributed pipelines, integrity checks and operational support using Flyte, Kafka, Azure, Terraform and Python. Large-scale pipeline and production-system experience is essential. The London role is full time and hybrid, with an out-of-hours support rotation once established.
What You'll Work On
- Build and operate Data Transfer Hub pipelines for partner video, LiDAR and sensor data at up to petabyte-per-day scale.
- Design distributed transfer workflows using Flyte, Kafka or event-driven patterns and Azure storage infrastructure.
- Implement the global hub-and-spoke model for retrieving and sharing autonomy data with external partners.
- Write Terraform infrastructure and Python pipeline logic, including cloud-to-cloud networking and object-storage transfer.
- Develop retries, backfills, integrity checks, lifecycle tracking, dashboards and alerts; join an out-of-hours support rotation once established.
Why This Role Matters
- Wayve’s Data Transfer Hub must move video, LiDAR and sensor data at up to petabyte-per-day scale; pipeline reliability determines whether model-training data arrives complete and traceable.
- Global partner programmes with Uber, Stellantis and Nissan depend on cloud-to-cloud transfer, retries and integrity checks working across evolving delivery dates and infrastructure constraints.
What They Are Looking For
- Experience: Building and operating large-scale data-transfer, ingestion or processing pipelines is essential; multi-terabyte or petabyte scale is ideal.
- Technical: Production software engineering, Python, Terraform and cloud-platform skills are essential; Azure is preferred.
- Technical: Workflow orchestration is essential; Flyte is preferred, with Airflow, Prefect or similar tools also relevant.
- Technical: Observable data systems, retries, backfills, integrity checks, lifecycle tracking and operational alerting are essential.
- Technical: Object storage, large-scale file movement, networking or cloud-to-cloud data-transfer experience is essential.
- Preferred: External partner integrations, data-intensive domains, Kafka, status dashboards, multiple clouds or cost, throughput and reliability improvements are desirable.