Data Engineering

A personal story that made me pause and think. Last year, I started building an AI app called Jarvis as a side project, hosted on GCP. I leaned heavily on Gemini to handle the DevOps side - I didn’t want to go deep on deployment when the real value was in the app itself. Gemini consistently pushed me toward App Engine. Every time I tried to pivot to Cloud Run (which, in hindsight, was the right fit - a low-traffic app that needs to scale to zero), the code never quite worked. Eventually Gemini told me: “For your use case, App Engine is the right choice.” So I went with it. It also recommended Cloud SQL with pgvector for my RAG functionality. I followed that suggestion too. ...

During 2019-2020, long before generative AI became a boardroom talking point, a team of data engineers and data scientists built something quietly remarkable inside one of Australia’s largest banks. There were no large language models involved. No prompts. No ChatGPT. Just a well-defined problem, a disciplined team, and a willingness to let the data do the heavy lifting. This is that story. What Worked - Until It Didn’t! One of Australia’s Big Four banks was undergoing a significant shift in how it handled customer complaints. Complaints were arriving through multiple channels - phone, branch, online, third-party services - and all of them were being funnelled into a central database. So far, so good. ...

Let me paint you a picture. A mid-sized accounting firm in Sydney. Twelve staff. Busy tax season. Every year, the same story - hundreds of client emails, document requests flying back and forth, data manually keyed from PDFs into spreadsheets, and partners spending Friday nights and at times weekends reviewing work that, frankly, a well-configured AI could handle in minutes. They know AI exists. They’ve heard about ChatGPT. A few of them have even played with it. But turning that curiosity into something that actually works inside their business? That’s where it stops. ...