Ai

Do we still need RAG if context windows hit 1M tokens? Long context windows have made the debate legitimate. The answer still depends on what problem is actually being solved. The problem every LLM has LLMs are trained on a snapshot of the world. Products built on top of them - ChatGPT, Claude, Perplexity - can search the web, but that is a tool injecting retrieved content into the context window, not the model learning anything new. The model itself cannot reliably access or recall information beyond its training data, especially when it is private, recent, or highly specific. Also nothing about internal documents, proprietary codebases, private wikis, or anything that was never in the training corpus to begin with. ...

A personal story that made me pause and think. Last year, I started building an AI app called Jarvis as a side project, hosted on GCP. I leaned heavily on Gemini to handle the DevOps side - I didn’t want to go deep on deployment when the real value was in the app itself. Gemini consistently pushed me toward App Engine. Every time I tried to pivot to Cloud Run (which, in hindsight, was the right fit - a low-traffic app that needs to scale to zero), the code never quite worked. Eventually Gemini told me: “For your use case, App Engine is the right choice.” So I went with it. It also recommended Cloud SQL with pgvector for my RAG functionality. I followed that suggestion too. ...

During 2019-2020, long before generative AI became a boardroom talking point, a team of data engineers and data scientists built something quietly remarkable inside one of Australia’s largest banks. There were no large language models involved. No prompts. No ChatGPT. Just a well-defined problem, a disciplined team, and a willingness to let the data do the heavy lifting. This is that story. What Worked - Until It Didn’t! One of Australia’s Big Four banks was undergoing a significant shift in how it handled customer complaints. Complaints were arriving through multiple channels - phone, branch, online, third-party services - and all of them were being funnelled into a central database. So far, so good. ...

Let me paint you a picture. A mid-sized accounting firm in Sydney. Twelve staff. Busy tax season. Every year, the same story - hundreds of client emails, document requests flying back and forth, data manually keyed from PDFs into spreadsheets, and partners spending Friday nights and at times weekends reviewing work that, frankly, a well-configured AI could handle in minutes. They know AI exists. They’ve heard about ChatGPT. A few of them have even played with it. But turning that curiosity into something that actually works inside their business? That’s where it stops. ...