Senior ML Data Engineer (San Francisco)

We're the data team behind Midjourney's image generation models. We handle the dataset side : processing, filtering, scoring, captioning, and all the distributed compute that makes high-quality training data possible.

What youd be working on :

• Large-scale dataset processing and filtering pipelines
• Training classifiers for content moderation and quality assessment
• Models for data quality and aesthetic evaluation
• Data visualization tools for experimenting on dataset samples
• Testing / simulating distributed inference pipelines
• Monitoring dashboards for data quality and pipeline health
• Performance optimization and infrastructure scaling
• Occasionally jumping into inference optimization and other cross-team projects

Our current stack : PySpark, Slurm, distributed batch processing across hybrid cloud setup. We're pragmatic about tools - if there's something better, we'll switch.

We're looking for someone strong in either :

Don't need exact tech matches - comfort with adjacent technologies and willingness to learn matters more. We work with our own hardware plus GCP and other providers, so adaptability across different environments is valuable.

Location : SF office a few times per week (we may make exceptions on location for truly exceptional candidates)

The role offers variety, our team members often get pulled into different projects across the company, from dataset work to inference optimization. If you're interested in the intersection of large-scale data processing and cutting-edge generative AI, we'd love to hear from you.