Data Engineer - Python / Big Data

Job Title: Data Engineer - Python / Big Data

About the Role

We are looking for a skilled Data Engineer – Python / Big Data to design, build, and optimize scalable, high-performance data systems in cloud environments. In this role, you will be responsible for developing robust data pipelines, managing data infrastructure, and enabling reliable, high-quality data availability for analytics, product, and business teams.

You will work closely with cross-functional teams to build scalable data products such as data lakes, data warehouses, and real-time data systems that power critical business decisions.

What You’ll Do

Data Engineering & Platform Development

• Design, develop, and maintain scalable data products including data warehouses, data lakes, and databases

• Build and optimize ETL/ELT pipelines for batch and real-time data processing

• Develop robust and reusable data models to support analytics and reporting needs

Pipeline Development & Optimization

• Build and maintain automated, scalable data pipelines

• Monitor, troubleshoot, and optimize pipeline performance for latency, cost, and reliability

• Ensure high availability and data integrity across systems

Cloud & Big Data Engineering

• Work with AWS services including EC2, S3, Lambda, Redshift, Glue, EMR, and DynamoDB

• Implement distributed data processing solutions using Spark, Flink, or Kafka

• Design cloud-native data architectures for scalability and efficiency

Technical Leadership & Collaboration

• Provide technical guidance on data architecture and engineering best practices

• Collaborate with data scientists, analysts, and engineering teams to deliver data-driven solutions

• Contribute to CI/CD processes and engineering automation

Data Quality & Governance

• Ensure accuracy, consistency, and reliability of data systems

• Implement data validation and monitoring frameworks

• Support secure and compliant data handling practices

What We’re Looking For

Technical Expertise

• 3+ years of experience in data engineering, preferably in cloud and big data environments

• Strong proficiency in Python and SQL (MySQL preferred)

• Hands-on experience with ETL/ELT pipelines and data modeling

• Strong understanding of scalable data architectures and distributed systems

Big Data & Cloud Skills

• Experience with AWS cloud services (EC2, S3, Lambda, Redshift, Glue, EMR, DynamoDB)

• Hands-on experience with Spark, Kafka, Flink, or similar distributed processing tools

• Familiarity with real-time and batch data processing systems

Engineering Practices

• Experience with Git-based version control systems

• Understanding of CI/CD pipelines and deployment automation

• Strong debugging and performance optimization skills

Education

• BE / BTech in Computer Science or related field

• MCA graduates also eligible

Soft Skills

• Strong analytical and problem-solving mindset

• Excellent communication and collaboration skills

• Ability to manage multiple priorities in a fast-paced environment

• Strong attention to detail and quality-focused approach

Nice to Have

• Experience in fintech or high-scale consumer platforms

• Exposure to data governance or data quality frameworks

• Familiarity with orchestration tools like Airflow or similar

• Experience working in startup or product-based environments

Why You’ll Love Working With Us

• Opportunity to build large-scale, high-impact data systems

• Work on modern cloud and big data technologies

• Fast-paced and collaborative engineering culture

• Strong ownership and learning opportunities

• Exposure to real-time, high-volume data platforms

Role Details

Position: Data Engineer - Python / Big Data
Employment Type: Full-Time
Experience Level: Mid-Level
Location: Bengaluru, India (Onsite)

Recruitment Fraud Warning & Disclaimer

KreditBee is committed to maintaining a transparent, fair, and secure recruitment process. We have become aware of fraudulent individuals and entities falsely claiming to represent KreditBee and offering fake employment opportunities.

Please note that KreditBee never asks candidates to pay any money for registration, training, certifications, security deposits, or any other purpose at any stage of the recruitment process. All official recruitment communications are sent through verified company email domains, primarily @kreditbee.in, @krazybee.com, and @kbnbfc.in. Candidates are advised to verify the sender's email address before responding to any recruitment-related communication. KreditBee does not conduct recruitment through unofficial email addresses or request payments.

If you receive a suspicious job offer, interview invitation, or recruitment-related communication claiming to be from KreditBee, do not make any payment or share sensitive personal or financial information. We encourage you to verify the authenticity of the communication through our official careers page or by contacting us at hr@kreditbee.in.

Disclaimer: KreditBee shall not be responsible for any loss, damage, or claims arising from fraudulent job offers or recruitment communications made by unauthorized individuals or entities falsely representing the company. Candidates should exercise due diligence and verify the authenticity of all recruitment-related communications before taking any action.

This job description is intended to outline the general nature and key responsibilities of the position. It is not intended to be an exhaustive list of all duties, responsibilities, or qualifications associated with the role. The responsibilities and qualifications described may be subject to change, and other duties may be assigned as needed. Employment is at-will, meaning the employee or the employer may terminate the employment relationship at any time, with or without cause, and with or without notice.

Data Utilization Disclaimer

By applying for this position, you acknowledge and agree that any personal data you provide may be used for recruitment and employment purposes. The data collected will be stored and processed in accordance with our privacy policy and applicable data protection laws. Your information will only be shared with relevant internal stakeholders and will not be disclosed to third parties without your consent, unless required by law.