Software Engineer, Ingestion Platform

Skills this job asks for

About the role

Reddit is a community of communities. It’s built on shared interests, passion, and trust, and is home to the most open and authentic conversations on the internet. Every day, Reddit users submit, vote, and comment on the topics they care most about. With 100,000+ active communities and approximately 130 million daily active unique visitors, Reddit is one of the internet’s largest sources of information. For more information, visit www.redditinc.com. The Data Platform team is looking to hire a Software Engineer who is excited to solve large scale data platform and efficiency challenges. Reddit’s mission is to bring community and belonging to everyone in the world. Reddit is a community of communities where people can dive into anything through experiences built around their interests, hobbies, and passions. With more than 50 million people visiting 100,000+ communities daily, it is home to the most open and authentic conversations on the internet. From pets to parenting, skincare to stocks, there’s a community for everybody on Reddit. For more information, visit redditinc.com. Our community of users generates over 100B events per day, each of which is ingested into a data warehouse that sees 55,000+ daily queries. We utilize this data to enable both batch and streaming based ML and BI workloads at the company. Critical teams such as ads, feed generation, and ML experimentation rely on the Data Platform to generate revenue for Reddit. The Ingestion Platform team within this group is specifically responsible for empowering the business to more efficiently process data and orchestrate these data workloads. As a software engineer, you will work with teammates and partner teams to create and improve scalable, fault tolerant, self-serve systems. You will: Design, write, and deliver reliable software for distributed data movement across streaming and batch workloads, with a focus on availability, scalability, latency, correctness, and cost efficiency. Contribute to the architecture and evolution of the platform’s control plane and data plane, including pipeline APIs, controllers, connectors, target managers, schemas, and sink integrations. Expand the platform beyond Kafka-to-BigQuery by helping deliver production-ready paths to S3/GCS and Apache Iceberg, as well as transformations, deduplication, dead-letter queues, and other reusable capabilities. Build high-quality connectors and abstractions for systems such as Kafka, BigQuery, S3, GCS, Iceberg, Flink, and other data stores, while keeping the platform modular and easy to extend. Improve the self-service experience for platform users through clear APIs, safe defaults, automated provisioning, documentation, onboarding workflows, and actionable observability and alerting. Lead migrations from bespoke and legacy ingestion systems to the Ingestion Platform, partnering with teams such as Ads Data Platform and ML Indexing to make migrations safe, incremental, and low-touch. Establish strong reliability, secu...

A summary from the original listing. Read the full details on their site.