Available for Opportunities

Sonu Ansari.

Senior Data Engineer & Software Engineer

Building production ETL pipelines, real-time data systems, cloud platforms, and AI powered tools at scale. 4+ years of experience cutting costs, reducing latency, and shipping reliable engineering solutions.

Get in Touch LinkedIn
4+
Years Experience
80%
Cost Reduction
1K+
Vehicles Tracked
<30s
Telemetry Latency
01.

About Me

Sonu Ansari, Senior Data Engineer & Software Engineer
Goa, India
Reveal email
linkedin.com/in/sonu-ansari
github.com/Sonu-Ansari

I'm a Data & Cloud Engineer who enjoys turning complex engineering problems into systems that are reliable, measurable, and easy to evolve. My work sits at the intersection of data engineering, cloud systems, real-time processing, and software development, with a strong focus on building things that solve practical problems.

I build ETL pipelines, streaming systems, IoT platforms, data products, and cloud-native applications. I care about what happens beyond simply making a pipeline run: performance, observability, data quality, scalability, and the experience of the people consuming the data. I've achieved up to 80% reductions in processing time and infrastructure cost through PySpark profiling and optimization, and built real-time systems capable of handling telemetry from 1,000+ vehicles per fleet.

Outside traditional data platforms, I enjoy experimenting with AI assistants, automation, Linux, Android, and developer tooling. Whether I'm designing an architecture, optimizing a query, debugging a distributed workflow, or building a small tool from scratch, I like understanding the problem deeply and turning it into a clean, working solution.

I'm also passionate about open source, knowledge sharing, and mentoring. I believe that great engineering is not just about writing code — it's about enabling others to do their best work too.

02.

Experience

Senior Software Engineer, Cloud & Data

Zimetrics Technology Official site ↗

Aug 2022 – Present Goa, India
Current Healthcare IoT AWS PySpark

Senior engineering role focused on data platforms, cloud systems, real-time processing, automation, and quality engineering. I work across architecture, implementation, optimization, testing, and production operations, turning high-volume data and device signals into reliable products and decision-ready data.

Healthcare data platform

✓

Multiple concurrent ETL pipelines

Delivering audit-ready data across production systems with zero missed SLAs.

✓

80% cost reduction

Cut pipeline processing time and infrastructure cost by profiling and rewriting PySpark jobs, $500/month saving and unblocking downstream teams.

✓

Fault-tolerant orchestration layer

Built with AWS Step Functions, taking manual-intervention incidents from weekly to zero across 12 months.

✓

Automated data-quality validation

Eliminated an entire class of silent failures on PySpark transforms that previously required manual remediation.

IoT & real-time systems

✓

End-to-end IoT telemetry platform

Tracking 1K+ live vehicles across fleets, ingesting per-second data with zero data loss.

✓

Serverless streaming pipeline

AWS IoT Core → Kinesis → Lambda achieving <30-second end-to-end latency, beating the original 5-minute SLA by 10×.

✓

Geofencing & real-time alerting engine

Shipped for boundary violations and emergencies, cutting incident response time from minutes to seconds.

✓

Google Maps integration

Live vehicle visualization and route optimization, reducing driver idle time and improving on-time fleet performance.

Engineering toolkit & delivery

✓

Data engineering

PySpark, Databricks, AWS Glue, Redshift, PostgreSQL, Apache Airflow, and Delta Lake for batch and streaming data products.

✓

Time-series & telemetry

Designed around high-frequency telemetry and time-based workloads using services such as Amazon Timestream alongside event streaming and object storage.

✓

Quality engineering

Robot Framework, automated validation, data-quality checks, CI/CD, and workflow testing to catch failures before they reach downstream consumers.

✓

AI & intelligent automation

Applied AI tooling, LLM workflows, automation, and developer assistants to reduce repetitive work and accelerate engineering tasks.

✓

Agile delivery & Jira workflows

Run sprint planning, daily standups, sprint reviews, and retrospectives in Jira and Confluence, breaking epics into a continuously groomed backlog that ships through CI/CD on a predictable delivery cadence.

Impact at a glance

80%

Cost cut

$500

Saved/month

1K+

Vehicles

<30s

Latency

0

Missed SLAs

12mo

Zero incidents

03.

Skills & Technologies

Data Engineering
PySpark Apache Kafka Apache Airflow Databricks Delta Lake dbt Pandas NumPy
Cloud & Infrastructure
AWS S3 Lambda Kinesis Step Functions Glue Redshift IoT Core EC2
Databases & Storage
PostgreSQL MySQL MongoDB Redis Amazon Redshift Delta Lake Amazon Timestream
Programming
Python SQL Bash JavaScript TypeScript Java
DevOps & Delivery
Docker Git CI/CD Continuous Delivery Terraform Linux Termux Ubuntu Robot Framework Jira Confluence Agile / Scrum Kanban Sprint Planning Retrospectives
AI & Automation
AI / ML Computer Vision OpenCV LLMs OpenAI API LangChain RAG Web Scraping Browser Automation ADB
04.

Personal Projects

Featured
🤖
Android AI Assistant
A fully autonomous AI assistant running locally on Android via Termux + Ubuntu. Capable of surfing the internet, controlling device settings, launching apps, and executing complex multi-step tasks through natural language.
  • Web browsing and real-time information retrieval
  • Device control: volume, brightness, WiFi, Bluetooth
  • App interaction: launch, query, automate workflows
  • File system navigation and management
  • Terminal command execution with safety validation
LocalOn-Device
100%Privacy
ZeroCloud Cost
Python Termux Ubuntu LLM ADB Linux
📂 View Code ▶ Demo
📷
Raspberry Pi Face Recognition Surveillance
A personal home surveillance prototype using a Raspberry Pi webcam and OpenCV to recognize people at the door or inside the home, record activity, and notify the owner when an unfamiliar or unauthorized person is detected.
  • Real-time face detection and recognition with OpenCV
  • Known-face enrollment and recognition history
  • Automatic video recording for detected activity
  • Owner alerts for unknown or unauthorized access
  • Remote access to recorded events from a phone
EdgeProcessing
24/7Monitoring
Real-timeAlerts
Raspberry Pi Python OpenCV Computer Vision Linux
🤖
Autonomous Robot Navigation
A robotics project focused on autonomous movement through constrained environments, combining shortest-path algorithms, sensor feedback, and control logic for reliable navigation.
  • Shortest-path routing and obstacle avoidance
  • Grid-based path planning experiments
  • Sensor-driven movement decisions
  • Control-loop tuning for stable motion
Python Algorithms Robotics Computer Vision Embedded Systems
05.

Systems, Robotics & Algorithms

Interactive engineering demos, from robot control and shortest-path routing to real-time geofencing, GPS correction with Google Maps Platform, and a high-throughput event-driven ETL architecture.

Shortest-Path Routing

Build obstacles, choose the algorithm, and watch the search frontier expand. This mirrors the core routing problem behind an autonomous transport robot.

Click cells to add/remove obstacles
StartGoalExploredShortest path

Line-Follower Control Loop

A small PID-style feedback loop: sensors estimate lateral error, the controller adjusts motor speeds, and the robot converges back to the track.

PID feedback active
0.00
Lateral error
0.00
Correction
50%
Left motor
50%
Right motor
Kp1.20
Ki0.08
Kd0.45

Live Geofencing

Point-in-shape boundary detection for fleet tracking, switch the fence geometry and watch the vehicle trigger a live breach alert.

5 sides
⚠ Geofence breach detected

GPS Correction & Road Matching

Raw GPS from a moving vehicle can jump off the road because of signal noise, urban canyons, or multipath effects. In a production flow, Google Maps Platform can snap the noisy points to the road network before they are used for tracking, ETA, geofencing, or routing.

Noisy GPS projected onto the visible road
Raw GPSRoad-matched
1. Vehicle telemetryGPS latitude/longitude arrives every few seconds with heading, speed and timestamp.AWS IoT / Kinesis
2. Coordinate correctionSend a batch of GPS points to Google Roads API → Snap to Roads to map noisy coordinates onto the road network.Google Roads API
3. Route intelligenceUse corrected positions for route display, ETA, geofencing, replay and downstream analytics.Routes API / Maps JS

ETL Pipeline, From User Activity to Monthly Reports

Imagine a watch company collecting activity and purchase telemetry from a large user base. Every month, we first identify customers who are actively subscribed to the insight service. Only that eligible cohort is processed: their raw events are read, parsed, transformed into performance metrics, turned into observations and motivational badges, and finally delivered by email.

Ready, filtering active subscribers
👥 User pool100,000registered customers
✓ Policy filter18,420active subscribers
📈 Monthly reports0eligible users processed
✉ Delivery0%reports emailed
👥
User Pool
100K users
→
🔐
Subscription Filter
active policy only
→
📡
Kinesis
10K events/sec
→
🗃️
DynamoDB
raw activity
→
λ
Lambda
read + parse
→
⚙
Glue / Spark
metrics + features
→
🔀
Step Functions
orchestration
→
🥉
Bronze → Silver → Gold
trusted report data
→
🧠
Insights Engine
observations + badges
→
✉
Email Delivery
monthly report
0Events read
0Users validated
0Gold report rows
,Pipeline status
MONTHLY REPORT PREVIEW
SA
Subscriber #A10482Active plan, July report
Consistency92%↑ 8%
Goal completion87%↑ 12%
Active days24 / 31Strong
OBSERVATIONS & MOTIVATIONAL BADGES
🏆
Consistency ChampionMaintained activity on 24 of 31 days.
🔥
Momentum BuilderImproved goal completion by 12%.
💡
ObservationYour strongest performance was during weekday mornings.
Why this is ETL: subscription eligibility is applied before expensive processing; raw events are ingested and retained, Lambda handles parsing/validation, Glue/Spark creates reusable metrics, Step Functions coordinates the monthly workflow, and Bronze/Silver/Gold tables separate raw, trusted and business-ready data. The Gold layer feeds the observation/badge engine and APIs, while the final delivery stage sends each subscriber a personalized monthly report by email. Non-subscribers remain outside the reporting workflow.
06.

Certifications

☁️

AWS Certified Cloud Practitioner

Amazon Web Services, Validated cloud fundamentals, architecture, and best practices

📊

Databricks Certified Associate Data Engineer

Databricks, Spark, Delta Lake, and production data pipeline development

07.

Education

🎓

Bachelor's Degree

Goa University, Computer Science & Engineering

08.

Recognition

🏆

National-Level Robotics Competition, IIT Bombay

Selected among hundreds of teams nationwide to compete in IIT Bombay's engineering challenge. Designed shortest-path routing logic for an autonomous construction-material transport robot.

9.

Latest Articles

Data Engineering

Optimizing PySpark: 5 Techniques That Cut Our Cost by 80%

Practical strategies for profiling, partitioning, and rewriting Spark jobs to reduce processing time and infrastructure spend.

📅 15 min read · 3 min ago IoT

Building a Serverless IoT Pipeline with AWS Lambda and Kinesis

From device telemetry to real-time alerts: how we built a sub-30-second streaming pipeline that processes 10,000 events per second.

📅 12 min read · 1 week ago AI

Running a Fully Local AI Assistant on Android with Termux and LLMs

How I built a privacy-first AI assistant that runs entirely on-device — zero cloud, zero cost, 100% open source.

📅 20 min read · 2 weeks ago

Let's Build Something Together

I'm always open to discussing data engineering, cloud architecture, IoT systems, or AI automation projects. Whether you have a challenge or an opportunity — let's talk.

📧 Email Me 💼 LinkedIn 🐙 GitHub