Abhiraj Karpe

Data Engineer · Python, Spark, Kafka, Snowflake

Pune, Maharashtra, India

@abhiraj_karpe

LinkedIn GitHub

0 followers

🎓 B.Tech, Computer Science (Software Engineering) at Ajeenkya D.Y. Patil University · Graduating 2026

About

Data Engineer with hands-on experience building batch, streaming, and CDC pipelines using Apache Spark, Kafka, Databricks, Airflow, dbt, and Snowflake. Experience includes production engineering work at Jio Platforms, with a focus on incremental processing, data modeling, lakehouse architecture, and pipeline optimization.

Experience

Software Engineering Intern

Jio Platforms Limited · Pune, Maharashtra, India

Jan 2026 – Jul 2026

Converted legacy ADB script outputs from raw text to structured JSON across OTT device/app profiling workflows, reducing storage overhead and downstream parsing time. Optimized memory and CPU profiling workflows, designed and deployed a production ingestion API, built automated collection of device-level metrics, and debugged data-loss and parsing issues to improve pipeline reliability.

Education

2026

B.Tech

Ajeenkya D.Y. Patil University

Computer Science & Software Engineering · Sep 2022 – Jun 2026

Computer Science & Software Engineering · Sep 2022 – Jun 2026

Skills

6 to 12 months

SQL Python

<3 months

ETL RAG IAM CDC Java CI/CD MinIO Node.js PySpark debezium Snowpipe LangChain OpenAI API Delta Lake AWS Athena Apache Spark Data Modeling Unity Catalog Apache Iceberg Prompt Engineering Medallion Architecture API Integration Understanding

Tools / apps / platforms

<3 months

S3 dbt Git Linux MySQL Docker GitHub Lambda MongoDB Postman BigQuery ChromaDB Snowflake CloudWatch Databricks PostgreSQL Apache Kafka Apache Airflow Amazon Redshift Databricks Workflows

Projects

Real-Time Crypto Data Engineering Pipeline ↗

Apache Kafka, AWS S3, Snowpipe, Snowflake, dbt, Airflow

Built a real-time pipeline streaming CoinGecko market events through Kafka into Amazon S3 and Snowflake via Snowpipe. Developed incremental dbt transformations with MERGE-based upserts across RAW, STAGING, and GOLD layers, orchestrated with Airflow.
Databricks Taxi Analytics Pipeline ↗

Databricks, PySpark, Delta Lake, Unity Catalog, AWS S3

Built an incremental Bronze–Silver–Gold pipeline processing 355M+ taxi trip records with schema normalization, data quality filtering, and deduplication. Designed partition-aware incremental queries that cut data scanned by 89% and execution time by 33%, producing 3 analytics-ready Gold datasets.
CDC Lakehouse Pipeline ↗

PostgreSQL, Debezium, Apache Kafka, Apache Iceberg, MinIO, Hive Metastore, Trino

Built an end-to-end CDC pipeline streaming PostgreSQL inserts, updates, and deletes through Debezium and Kafka into Apache Iceberg tables with 1.03s P95 latency. Integrated MinIO, Hive Metastore, and Trino for queryable lakehouse analytics; containerized a 7-service stack for reproducible deployment.

Courses & certifications

Introduction to Data Engineering · DeepLearning.AI & Amazon Web Services (Coursera)
AWS re/Start Graduate · Amazon Web Services Training and Certification

🏆 Achievements & awards

LeetCode top 0.5% · 2025

Solved 700+ problems on LeetCode and reached contest rating 1575.
GeeksforGeeks Coding Score 313

Reported coding score on GeeksforGeeks.

Explore the Broxer community

💬 Q&A Ask about roles, salaries and interviews — and help others. 👥 Circles Join discussions on skills, companies and industries. 📰 Feed Follow updates and insights from the community.