About
B.Tech student in Information Technology at Dwarkadas J. Sanghvi College of Engineering with research experience in transformer geometry, representation learning, and spatiotemporal forecasting. Also builds PyTorch-based ML systems and participates in student research and coding communities.
Experience
-
Research CollaboratorUniversity of Central FloridaOct 2026 – Present
Working under the guidance of Dr. Niloofar Yousefi on a paper generalizing a continuous query-key initialization technique for Transformer attention, targeting an ICLR 2027 workshop. Designing experiments to decouple the effects of width and context length on cross-scale transfer of initial attention regimes, and to remap regimes in context-length-aware sharpness coordinates. Setting up a matched-sharpness training comparison of initialization geometries against a mimetic-initialization baseline, with multi-seed 2-layer decoders and an associative-recall probe.
-
Research CollaboratorUniversity of GävleAug 2026 – Present
Collaborating with Dr. R. P. Durga Prasad Bavirisetti on pedestrian intent prediction with occlusion-aware reasoning for autonomous driving. Developing novel prediction models and implementing architectures from recent literature, spanning attention-based, vision-language, and JEPA-based approaches. Surveying work on pedestrian behavior prediction, visual occlusion, and uncertainty-aware reasoning to guide model design choices.
-
Collaborative ResearcherAdvanced Vision LabsAug 2026 – Present
Collaborating on research in representation learning, investigating the geometry and structure of learned representations in modern deep neural networks. Reproducing and experimentally analyzing recent representation-learning methods, studying the effects of normalization and feature alignment across pretrained models.
-
Research AssistantIIT RoorkeeMay 2026 – Present
Conducting research on geospatial time-series forecasting for district-level drought prediction across India using a proprietary multi-year dataset. Surveyed recent literature on spatiotemporal forecasting and drought prediction, analyzing Transformer-, graph-, and attention-based architectures to identify suitable modeling strategies. Designed candidate deep learning architectures integrating spatial attention with temporal sequence modeling for nationwide drought forecasting.
Education
-
2028
Bachelor of Technology in Information TechnologyDwarkadas J. Sanghvi College of EngineeringInformation Technology · Sep 2024 – 2028
- 2024
Skills
Tools / apps / platforms
Projects
-
Torch-IsoFLOPs: Chinchilla-Style Scaling Law FrameworkPyTorch, Scaling Laws
Developed a reusable framework for IsoFLOP scaling-law experiments under fixed compute budgets using Chinchilla-style methodology. Implemented automatic FLOP estimation, experiment scheduling, Huber regression, RANSAC-based curve fitting, and resumable experiment logging. Enabled analytical estimation of optimal parameter-data tradeoffs across compute budgets with automated visualization.
-
Representation Geometry: Hidden State AnalysisPyTorch, Hugging Face
Developed and published a model-agnostic PyTorch library for transformer representation analysis using activation hooks, streaming covariance estimation, and reproducible artifact generation. Implemented spectral, statistical, and subspace metrics including participation ratio, stable rank, entropy rank, linear CKA, and activation-space SVD analysis. Supported Hugging Face and native PyTorch models through automatic transformer-block discovery with modular analysis and bounded-memory statistical backends.
-
Language Model Stack: From Training to AgentsPyTorch, LLM Systems
Trained a 50M-parameter decoder-only transformer from scratch with custom BPE tokenization, RoPE, RMSNorm, grouped-query attention, SwiGLU Mixture-of-Experts, weight tying, and cosine learning-rate decay. Built a token-level inference runtime for Qwen3 models implementing KV cache, greedy, beam, top-k, nucleus, min-p, and locally typical sampling without relying on model.generate(). Extended the inference runtime into a tool-using agent capable of interrupting decoding, invoking external tools, preserving KV cache across tool calls, and resuming generation.
-
Attention Geometry & Training Dynamics in TransformersPyTorch, Mech Interp
Instrumented a 4M-parameter GPT-style transformer with 18 geometric metrics tracking attention entropy, effective rank, covariance spectra, Q/K/V statistics, and gradient dynamics across 87 epochs. Observed divergent query/key geometry despite shared projections, spectral redistribution in Wq, and increasing pre-softmax rank while post-softmax attention remained low-rank due to softmax concentration. Developed a reproducible measurement stack generating per-epoch JSONL logs and automated visualization for transformer geometry analysis.
Extracurricular activities
-
DJ InIT.ai
Research Mentee → Vice Chairperson (Research); lead a student research club focused on machine learning, organizing technical sessions, paper reading groups, and research discussions. Coordinate research collaborations and mentor members in exploring and implementing machine learning research.
-
DJS Codestars
Technical Member; authored competitive programming problems for weekly contests engaging 150+ participants and contributed to organizing CodeUncode, collaborating with teams on contest preparation and website development.
-
GDG DJSCE
ML Research Member; first author of a research paper under review on redundancy reduction in transformer hidden-state representations and participant in research paper reading sessions and technical discussions on modern machine learning.
Publications
-
A Continuous Space of Attention Projection Geometries at Initialization
-
Query Expansion and Key Specialization in Transformer Attention Geometry
-
Residual-Stream Geometry in Transformer Language Model