Pyspark Developer

The position is suited for individuals who have strong PySpark programming skills and have demonstrated ability to work effectively in a fast paced, high volume, deadline driven environment.

Education and Experience

Education: B.Tech/M.Tech/MCA/MS/MBA

Experience in design and implementation of Big Data systems using PySpark, database migration, transformation and integration solutions for any Data warehousing project.

Required Skills

• Must have excellent knowledge in Apache Spark and Python programming experience
• Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations.
• Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred
• Working experience on Cloud technology architecture like AWS ecosystem, Google Cloud, BigQuery etc. is an added advantage
• Understanding of Unix/Linux + Shell Scripting
• Data modelling experience using advanced statistical analysis,unstructured data processing
• Experience with building APIs for provisioning data to downstream systems by leveraging different frameworks.
• Hands on project experience on Jupyter notebook/ Zeppelin/ PyCharm etc. IDEs
• Hands on experience with AWS S3 Filesystem operations
• Good knowledge of Hadoop, Hive and Cloudera/ Hortonworks Data Platform
• Experience handling CDC operations for huge volume of data
• Should understand and have operating experience with Agile delivery methodologies
• Should have hands-on experience in the following: data validation, writing unit test cases
• Should have experience in integrating PySpark with downstream and upstream applications through a batch/real-time interface
• Should have experience in fine tuning process and troubleshooting performance issues
• Should have demonstrated expertise in development of design documents like HLD, LLD etc.
• Should have experience in leading requirements gathering and developing solution architecture for Data migration/integration initiatives
• Should have experience in handling client interactions at different phases of the projects
• Should have experience in leading a team in a project or a module
• Should be well versed with onsite/offshore model and its challenges

Preferred Skills

• Exposure to any ETL/Reporting tool (Informatica, Jasper, QlikView, Tableau) is desirable
• Exposure to Jenkins or equivalent CICD tool & Git repository is preferred
• Design & Develop AI/ML model using PySpark on cloud environment