Available for AI/Data architecture/Leadership roles · Bangalore, India

Hardik Goel

Bridging enterprise-scale data platforms and production-grade GenAI systems — from MLOps governance to agentic AI orchestration. Cut payment SLA 15d → 3d, own a £3.5B AI forecasting platform, and ship open-source AI tooling. Open-source contributor, published author, and AI educator with a decade plus of community work alongside corporate career.

~1.5Decades experience
£3.5BPlatform ownership
Decade+Years as AI educator
3OSS tools shipped
Hardik Goel — AI & Data Architect, GenAI systems builder, Bangalore
Taught, spoke & published across
UpGrad Analytics Vidhya GUVI DataCouch Interview Kickstart Ducat Keynote · ElasticSearch CXO Fusion Times of India ToolsQA CleverFox Publishing
About

The Enterprise ↔ GenAI Bridge

Most engineers are either enterprise-only or GenAI-only. I sit at the intersection — ~1.5 decades owning scalable data platforms, MLOps governance frameworks, and production pipelines, while simultaneously building agentic AI systems, intelligent orchestration frameworks, and LLM-powered engineering workflows.

At Tesco, I architect a £3.5B commodity AI platform and pioneered a persona-based agentic engineering framework — one of the earliest enterprise implementations of role-based AI agents in production data engineering. At Paytm, I led a 14-member team and drove transformative metrics: payment SLA from 15 days → 3 days, attrition cut by 40%.

Parallel to my corporate career, I've been an AI educator for 10+ years, a published author, open-source contributor, and content creator — building community while building systems.

Agentic AI Systems MLOps Governance AI-Augmented Engineering Intelligent Orchestration Platform Strategy
🏗️
£3.5B
Procurement platform at Tesco
15→3d
Payment SLA reduction at Paytm
🎓
10 +
Years as AI educator across 8 platforms
📦
PyPI
Published open-source AI library
📖
Author
"Forgotten By Design" — published 2025
👥
14×
Engineering team led at Paytm
Experience

Career Timeline

Click any role to expand full details.

🛒 Tesco · Bangalore
Technical Architect — SDE3
Jan 2025 – Present
£3.5B platform Commodity onboarding ↓ 65% Deploy failures ↓ 80% Delivery velocity ↑ 40% MLOps governance Agentic AI framework Azure ML · MLflow · SHAP

    After leading a 14-engineer org at Paytm, chose a deep individual-contributor architect track — to design and own AI platforms hands-on rather than manage at a distance.

    End-to-end architecture ownership of an AI-driven commodity forecasting platform supporting a £3.5B procurement domain, spanning MLOps governance, platform engineering, and AI-augmented engineering enablement.

    Platform Architecture & MLOps Governance

    • Architected and delivered a production-grade AI commodity forecasting platform supporting £3.5B in annual procurement across multiple commodities and prediction horizons.
    • Designed scalable MLOps workflows on Azure ML using MLflow, SHAP, and PSI drift monitoring; established end-to-end governed deployment pipelines with DQ gates, model metadata registries, security scanning, audit logs, Microsoft Purview lineage, and Maker-Checker controls.
    • Defined and enforced engineering governance standards across Data Engineering and ML — modular architecture, typed interfaces, reusable pipeline templates, and automated pytest coverage gates — improving deployment consistency and platform reliability.
    • Built reusable metadata-driven data and ML pipelines integrating LSEG and enterprise RDBMS sources, reducing commodity onboarding effort and improving scalability across new forecasting use cases.
    • Revamped fragmented forecasting workflows into a scalable, production-ready architecture optimised for reliability, maintainability, auditability, and developer velocity.
    • Delivered stakeholder analytics dashboards and data enrichment solutions, improving commercial visibility into commodity trends and model outputs.

    AI-Augmented Engineering & Agentic Framework Design

    • Pioneered AI-augmented engineering workflows using Claude and GPT agents for automated code review, test generation, documentation, and pipeline scaffolding — significantly accelerating delivery velocity.
    • Designed a persona-based agentic framework with specialised AI roles for Data Engineering, ML Review, Security, and MLOps Governance — one of the earliest enterprise implementations of role-based AI agents in a production data platform.
    • Authored AI-enforceable markdown standards enabling automated enforcement of coding practices, security guardrails, and quality gates through structured context orchestration and RAG workflows.
    • Contributed architecture consulting and platform engineering across enterprise PoCs and production adoption initiatives, accelerating AI adoption across cross-functional teams.
💳 Paytm · Bangalore & Noida
Data Engineering Manager
Jan 2022 – Jan 2025
SLA 15d → 3d Attrition ↓ 40% Cost ↓ 50%+ 14-member team Built FSEBuddy → Product self-serve, cut eng dependency Dream11 scale (1.8B events/day)
  • Led 14-member engineering team across incentive management, merchant analytics, and GenAI product initiatives.
  • Incentive platform: payment SLA 15 → 3 days, accuracy >97%, attrition ↓ 40% — using Airflow, AWS EMR, PySpark, MongoDB, DBT.
  • Drove >50% cost reduction on Incentives; >70% cost cut on PGOLAP DBR at 110M daily events (Dream11 scale — 1.8B daily events).
  • Merchant analytics: >40% offline merchant sales growth, >30% cross-sell rate uplift — Kafka + NiFi + Spark + Apache Hudi + ElasticSearch.
  • Built FSEBuddy — self-service platform enabling Product team autonomy, reducing engineering dependency.
  • Delivered CustomerEchoEase — GenAI-powered customer insights solution as an internal innovation initiative.
  • Implemented observability frameworks, Maker-Checker processes, and process-level governance controls across the platform.
✈️ Impetus Technologies · Gurgaon
Senior Software Engineer
Feb 2020 – Jan 2022
Ingestion TAT ↓ 40% Error rates ↓ 90% Latency 2s → 20ms
  • ICON framework (US airline): data lake governance — ingestion TAT ↓ 40%, error rates ↓ 90% — AWS Lambda, Glue, Athena, RDS, PySpark.
  • Search engine optimisation (intelligence client): latency ↓ 99% (2s → 20ms) using Solr + ELK; Cassandra for cold-path storage; Kafka as data backbone.
  • Semiconductor sensor analytics: real-time anomaly detection — Kafka + Flink + QuickSight + Redshift Spectrum + DBT + MWAA + Apache Beam.
  • Built internal NLP sentiment analyser using NLTK, outperforming VADER accuracy benchmarks.
🏦 Cognizant · Gurgaon
Associate — Big Data Engineer
Apr 2018 – Jan 2020
AML false positives ↓ from 91% European bank
  • AML false positive reduction from 91% — LightGBM (Paylay model) + Spark + Hive/Impala pipeline; Oracle/Teradata sources via Sqoop.
  • Mainframe CDC pipeline: Kafka + PySpark + MongoDB as Operational Data Store (ODS) for European bank.
📡 Accenture · Pune & Gurgaon
Software Engineering Analyst
Oct 2014 – Apr 2018
US/UK telecom clients Kafka · Spark · Hive · Selenium · Appium
  • CDR/UDR ingestion pipelines for US telecom (postpaid) — Kafka → Spark → Hive data stack.
  • MVNO mobile automation frameworks — Selenium WebDriver, Appium, TestNG, Jenkins (UK telecom).
  • Led cross-functional sprint and UAT testing programs — Agile/Scrum across US prepaid client teams.
Open Source & Projects

What I Build

Production AI tooling, open-source frameworks, and stealth initiatives.

🔀
Open Source
Switchboard
Provider-agnostic AI model routing and orchestration framework. Dynamic fallback chains, cost-optimised routing, latency-aware model selection across OpenAI, Anthropic, and Google — designed for enterprise AI platform integration.
PythonLLM RoutingOrchestrationOpenAIAnthropic
🎙️
PyPI Published
AudioSummariser
GenAI audio intelligence pipeline: transcription → embedding → chunking → summarisation → conversational RAG retrieval. Published as a PyPI package — production-ready for enterprise audio intelligence use cases.
RAGEmbeddingsVector DBPyPIWhisper
🌾
AI Product
CommodityBuddy
GenAI assistant for commodity forecasting and procurement intelligence. Natural language querying over enterprise forecasting outputs — structured data → embeddings → RAG → conversational interface for non-technical stakeholders.
GenAIRAGProcurement AILLM
🤖
Active Build
Agentic AI SaaS
Founder-led SaaS applying multi-agent orchestration to enterprise workflow automation. Specialised AI personas, MCP-based integrations, and autonomous enterprise execution. Hands-on with image-gen LLMs, voiceovers, and multimodal agents.
Agentic AIMCPMultimodalSaaS
🔒
Stealth
Enterprise AI Governance Platform
Autonomous compliance monitoring, alerting, and remediation for enterprise AI adoption in regulated industries. Agentic governance frameworks for finance, retail, and healthcare — built with RAG, Vector DBs, and LLMs.
GovernanceComplianceAgenticRAG
🔎
Open Source
http-queryable
Node/TypeScript implementation of the HTTP QUERY method (RFC 10008) with body-aware caching — safe, idempotent reads that carry complex payloads without URL-length limits or POST's caching side effects. Express, Fastify and raw http adapters, an isomorphic client with auto-retry, and RFC-compliant cache keying that never yields false hits.
TypeScriptNode.jsHTTP CachingRFC 10008ExpressFastify
🚚
Advisory · FDE
Distribution Management System
Forward-deployed engineering on a distribution-management platform (DMS) — order-to-delivery flows, inventory and route data pipelines, and analytics for a distribution client. An occasional advisory engagement delivered via a boutique technology consultancy.
FDEDMSData PipelinesAnalytics
Blog

Latest Writing, Right Here

Recent Substack and Medium posts rendered directly on this page for a smoother reading path. Front-page links to both platforms remain available.

Substack
Visit
  • Loading latest posts...
Medium
Visit
  • Loading latest posts...

Read every post on the blog — canonical home for all essays, also on Substack & Medium.

Skills

Technical Stack

🧠
AI & GenAI
Agentic AIRAGLLMsMCPVector DatabasesEmbeddingsMultimodal AgentsImage-Gen LLMsVoice AIOpenAI / Claude / GeminiChromaDB / FAISS
⚙️
MLOps & Governance
MLflowSHAPPSI Drift MonitoringAzure MLModel RegistryFeature StoresDQ GatesPurview LineageChampion-ChallengerAudit Frameworks
🔧
Data Engineering
Apache SparkKafkaAirflowDBTDatabricksApache HudiFlinkNiFiDelta LakeCDCData MeshData LakehouseElasticSearch / Solr
☁️
Cloud & Infra
AWS EMRLambdaGlueAthenaRedshiftS3OpenSearchECSAzure AZ-900DockerKubernetesCI/CDMWAA
💻
Languages
PythonPySparkSQLJava / SpringBootScalaPandas · NumPy · scikit-learn
🏛️
Architecture & Leadership
Platform StrategyReference ArchitectureHLD / LLDData ContractsSchema EvolutionGovernance FrameworksTeam BuildingStakeholder ManagementIntrapreneurship
Beyond Engineering

Community, Content & Creation

A decade of parallel work — educating, writing, speaking, and building community alongside the corporate career.

📖
Published Author
Forgotten By Design — The Silent Epidemic of Digital Dementia (CleverFox Publishing, 2025). Explores how digital dependency and system design erode human cognition. Available on Kindle and paperback globally.
🎓
AI / Data Educator — 10+ Years
Designed and delivered full-stack curricula in Data Engineering, Data Science, and Generative AI. Mentored professionals, built end-to-end courses, and conducted PAN-India workshops for enterprises transitioning into AI-driven organisations.
UpGradAnalytics VidhyaDucatGUVIDataCouchIKToolsQA
🎤
Speaker & Panelist
Keynote panelist at ElasticSearch CXO Fusion on Generative AI adoption. Guest lecturer at graduate universities. Regular podcast guest. Representing practical AI leadership in enterprise contexts.
📹
Content Creator
YouTube and Instagram creator simplifying Data Engineering and Generative AI for global audiences. Currently producing a video tutorial series on Data & AI. Focus: practical, applied content for practitioners and learners.
✍️
Technical Writer & Editor
Freelance contributions to leading tech publications — in-depth features on AI, data engineering, and digital transformation. Covers technology from practitioner perspective, not just theoretical.
Times of IndiaToolsQATech journalse-magazines
🏗️
Corporate Trainer & Workshop Lead
Delivered PAN-India enterprise workshops and college masterclasses. Helped organisations transition from traditional data stacks to AI-driven architectures. Trained enterprise teams (Upward & Onward clients) and academic cohorts at graduate universities.
Endorsements

What Colleagues Say

Verbatim LinkedIn recommendations from managers, direct reports, and peers across Paytm, JPMorgan Chase, Walmart, and Impetus.

Hardik is a highly skilled and knowledgeable manager. He is not only friendly and helpful but goes above and beyond to maintain team spirit. His understanding of people's needs is outstanding, and he instills positive energy in every team member.
Srivatsav Veesam
Software Engineer III, Walmart · Ex-Paytm
Reported to Hardik
A focused and seasoned data engineer who has set things in motion for many time-sensitive deliverables. Very dependable — he doesn't hesitate to go the extra mile or collaborate across teams to get things done. Gladly recommend Hardik for all things data engineering.
Abhranshu Bagchi
Founder, Autonmis · Ex-leadership @ Paytm
Managed Hardik directly
Hardik is always the go-to person for any team member. He stays positive and adds positive energy to the team. Looking forward to working with him again.
Sri Harsha Atmuri
VP, Lead Data Engineer, JPMorgan Chase
Worked with Hardik
His leadership qualities are the best — the way he manages the team and creates a good environment for everyone to do their best work is amazing. He guides the team, shares strong ideas, and is a good leader in true terms.
Tejashri Marne
Data Engineer — PySpark, AWS, ETL
Reported to Hardik
Certifications

Professional Credentials

Amazon Q Developer Fundamentals
Amazon Web Services
Oct 2025
DAG Authoring for Apache Airflow 3
Astronomer
Oct 2025
Apache Airflow Fundamentals
Astronomer
Jul 2025
Generative AI Fundamentals
Databricks
Jul 2023
Lakehouse Fundamentals
Databricks
Aug 2022
Certified Associate Developer — Spark 3.0
Databricks
Jul 2022
Apache Kafka Fundamentals
Confluent
Feb 2021
Azure Fundamentals (AZ-900)
Microsoft
Aug 2020
Teradata Vantage Associate
Teradata
Jun 2020
Spark Level-1
IBM
Oct 2019
Contact

Let's Connect

Open to senior AI/Data architecture roles, advisory conversations, speaking engagements, and collaboration on open-source AI projects.

Hardik Goel
HG
Hardik Goel
AI-powered · responds as Hardik
Hi! Ask me about my projects, architecture decisions, what I'm building, or anything in my background. Powered by AI.