Job Description
Job Description:
Our client is seeking a highly skilled and experienced Senior PySpark Developer to join our dynamic data engineering team. This role is ideal for a motivated professional with a strong background in big data technologies, ETL processes, and database systems. The successful candidate will play a key role in designing, developing, and optimizing data pipelines and analytics solutions that support critical business operations.
📌 Note: Only candidates eligible to work on a W2 basis will be considered.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines using PySpark and Spark SQL.
- Collaborate with data architects, analysts, and business stakeholders to understand data requirements and deliver robust solutions.
- Optimize and troubleshoot complex ETL workflows and data transformation processes.
- Write efficient SQL queries for data extraction, transformation, and loading.
- Ensure data quality, integrity, and compliance with enterprise standards.
- Participate in code reviews, testing, and deployment activities.
- Monitor and maintain production workflows, including job scheduling and performance tuning.
Required Skills & Qualifications:
- 5+ years of experience in data engineering or software development.
- Strong proficiency in PySpark, Python, and Spark SQL.
- Solid experience with SQL and relational database systems.
- Proven expertise in ETL development and data pipeline orchestration.
- Familiarity with job scheduling tools; Autosys experience is a plus.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration abilities.
Preferred Qualifications:
- Experience working in the financial services industry.
- Familiarity with cloud platforms and big data ecosystems.
Job Tags