GenAI - 13

LLM Access : Running Models Locally

Business Scenario

Welcome!

Previously, you used Chain of Thought prompting for product and price comparisons, enforced valid JSON and JsonOutputParser in a LangChain, converted JSON responses into product cards, and checked budgets. You also added memory and built SmartCart AI v2, successfully passing all tests that v1 failed.

Today, your manager has assigned you a task to run SmartCart AI on a local open-source model as an alternative to GroqCloud. You will compare the cost, speed, & response quality of both options, understand privacy and reliability benefits of running a model locally, & update Kart so you can switch between cloud and local models without changing its features.

Git Pull

git pull origin branchName

Pre-Lab Preparation

Dataset :

Topic : LLM Access - Running Models Locally

1) Running LLM Locally

2) Open-weight vs API Models

3) Swapping the provider

Task 1: Install Ollama and Start the Server

!sudo apt-get install -y -qq zstd # needed by the Ollama installer
!curl -fsSL https://ollama.com/install.sh | sh
import subprocess, time
subprocess.Popen(["ollama", "serve"],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
time.sleep(5) # give the server a moment to start
!curl -s http://localhost:11434

Task 2 : Chain of Thought Prompting

Download model & check if it is installed

!ollama pull qwen2.5:3b
!ollama list

!ollama run qwen2.5:3b "Recommend 3 gym earbuds under Rs. 2,000, one line each."

Test with command

Output

Task 3: Call the Local Model with Langchain

!pip install -q -r requirements.txt langchain-ollama
!echo "langchain-ollama" >> requirements.txt

import os
from google.colab import userdata
os.environ["GROQ_API_KEY"] = userdata.get("GROQ_API_KEY") # still needed for the cloudtests
from langchain_ollama import ChatOllama

!pip install -q -r requirements.txt langchain-ollama
!echo "langchain-ollama" >> requirements.txt

import os
from google.colab import userdata
os.environ["GROQ_API_KEY"] = userdata.get("GROQ_API_KEY") # still needed for the cloudtests
from langchain_ollama import ChatOllama

local_llm = ChatOllama(
 model="qwen2.5:3b",
 temperature=0.7,
 top_p=0.95,
 num_predict=300, # Ollama's name for max_tokens)
response = local_llm.invoke("Recommend 3 gym earbuds under Rs. 2,000, one line each.")
print(response.content)
print("\nToken usage :", response.usage_metadata)
print("Finish :", response.response_metadata["done_reason"])

Output

Task 4 : Measure Latency

import time
from smartcart_v1 import llm as cloud_llm

import time
from smartcart_v1 import llm as cloud_llm

def time_call(model, prompt):
 """Return how many seconds one call takes."""
 start = time.time()
 model.invoke(prompt)
 return round(time.time() - start, 2)

prompt = "Recommend 3 gym earbuds under Rs. 2,000, one line each."

!ollama stop qwen2.5:3b

print("Local cold start:", time_call(local_llm, prompt), "s")
for name, model in [("Cloud", cloud_llm), ("Local", local_llm)]:
 times = [time_call(model, prompt) for run in range(3)]
 print(f"{name} (warm): {times} -> average {sum(times) / 3:.2f} s")

Output

Task 5 : Cost & Privacy Trade-offs

usage = cloud_llm.invoke(prompt).usage_metadata
tokens_per_request = usage["total_tokens"]

REQUESTS_PER_MONTH = 300_000 # SmartCart estimate: 10,000 chats a day
PRICE_PER_MILLION_TOKENS = 0.30 # USD -- example only, check groq.com/pricing

usage = cloud_llm.invoke(prompt).usage_metadata
tokens_per_request = usage["total_tokens"]

REQUESTS_PER_MONTH = 300_000 # SmartCart estimate: 10,000 chats a day
PRICE_PER_MILLION_TOKENS = 0.30 # USD -- example only, check groq.com/pricing

monthly_tokens = tokens_per_request * REQUESTS_PER_MONTH
monthly_cost = monthly_tokens / 1_000_000 * PRICE_PER_MILLION_TOKENS

print(f"Tokens per request : {tokens_per_request}")
print(f"Tokens per month : {monthly_tokens:,}")
print(f"Cloud cost / month : ${monthly_cost:,.2f}")
print("Local cost / month : $0 in tokens -- but you pay for the machine")

Output

Task 6 : Make SmartCart v2 Backend Swappable

%%writefile smartcart_backends.py
import os
from langchain_groq import ChatGroq
from langchain_ollama import ChatOllama

CLOUD_MODEL = "qwen/qwen3.8-27b"
LOCAL_MODEL = "qwen2.5:3b"

Save the backend file

%%writefile smartcart_backends.py
import os
from langchain_groq import ChatGroq
from langchain_ollama import ChatOllama

CLOUD_MODEL = "qwen/qwen3.8-27b"
LOCAL_MODEL = "qwen2.5:3b"

def get_llm(backend="cloud"):
    """Return the chat model for 'cloud' (GroqCloud) or 'local' (Ollama)."""
    if backend == "local":
        return ChatOllama(
            model=LOCAL_MODEL,
            temperature=0.3,
            num_predict=800,
            format="json",  # JSON mode in Ollama)
    if backend == "cloud":
        if not os.environ.get("GROQ_API_KEY"):
            raise SystemExit("GROQ_API_KEY not found. Add it to Colab Secrets (Lab 9, Task 2).")
        return ChatGroq(
            model=CLOUD_MODEL,
            temperature=0.3,
            max_tokens=800,
            model_kwargs={"response_format": {"type": "json_object"}},
            reasoning_effort="none",)
    raise ValueError(f"Unknown backend '{backend}'. Use 'cloud' or 'local'.")

Update SmartCart v2

%%writefile smartcart_v2.py
import os
import json
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_backends import get_llm  # NEW

BACKEND = os.environ.get("SMARTCART_BACKEND", "cloud")  # NEW: "cloud" or "local"
llm = get_llm(BACKEND)  # NEW

%%writefile smartcart_v2.py
import os
import json
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_backends import get_llm  # NEW

BACKEND = os.environ.get("SMARTCART_BACKEND", "cloud")  # NEW: "cloud" or "local"
llm = get_llm(BACKEND)  # NEW

EXIT_WORDS = ["quit", "exit", "bye"]
MAX_CHARS = 500
FRIENDLY_ERROR = "Sorry, I'm having trouble right now. Please try again in a moment."

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:
   Step 1 list what the shopper needs, Step 2 check each product, Step 3 give a verdict.
Always reply in JSON with exactly these keys:
{"steps": ["reasoning steps, or an empty list if this is not a comparison"],
 "reply": "a short friendly message, under 60 words",
 "products": [{"name": "text", "price_rs": number,
               "category": "Electronics or Grocery or Apparel",
               "reason": "one short sentence"}]}"""

MEMORY_SIZE = 10
history = []

def check_input(text):  # NEW: copied from Lab 10
    """Return (cleaned_text, problem). problem is "" when the input is fine."""
    cleaned = " ".join(text.split())
    if cleaned == "":
        return cleaned, "Please type what you are shopping for."
    if len(cleaned) > MAX_CHARS:
        return cleaned, f"Please keep your request under {MAX_CHARS} characters."
    return cleaned, ""

%%writefile smartcart_v2.py
import os
import json
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_backends import get_llm  # NEW

BACKEND = os.environ.get("SMARTCART_BACKEND", "cloud")  # NEW: "cloud" or "local"
llm = get_llm(BACKEND)  # NEW

EXIT_WORDS = ["quit", "exit", "bye"]
MAX_CHARS = 500
FRIENDLY_ERROR = "Sorry, I'm having trouble right now. Please try again in a moment."

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:
   Step 1 list what the shopper needs, Step 2 check each product, Step 3 give a verdict.
Always reply in JSON with exactly these keys:
{"steps": ["reasoning steps, or an empty list if this is not a comparison"],
 "reply": "a short friendly message, under 60 words",
 "products": [{"name": "text", "price_rs": number,
               "category": "Electronics or Grocery or Apparel",
               "reason": "one short sentence"}]}"""

MEMORY_SIZE = 10
history = []

def check_input(text):  # NEW: copied from Lab 10
    """Return (cleaned_text, problem). problem is "" when the input is fine."""
    cleaned = " ".join(text.split())
    if cleaned == "":
        return cleaned, "Please type what you are shopping for."
    if len(cleaned) > MAX_CHARS:
        return cleaned, f"Please keep your request under {MAX_CHARS} characters."
    return cleaned, ""

def ask_smartcart_v2(text):
    """Send the message with memory; return the reply as a dictionary."""
    messages = ([SystemMessage(content=SYSTEM_PROMPT)]
                + history[-MEMORY_SIZE:]
                + [HumanMessage(content=text)])
    response = llm.invoke(messages)
    data = json.loads(response.content)
    history.append(HumanMessage(content=text))
    history.append(AIMessage(content=response.content))
    return data

def show(data):
    """Print the reasoning steps, the message and the product cards."""
    for step in data.get("steps", []):
        print("  (thinking)", step)
    print("Kart:", data.get("reply", ""))
    for p in data.get("products", []):
        print(f"  [{p['category']}] {p['name']} – Rs. {p['price_rs']:,} – {p['reason']}")

def chat():
    """Run SmartCart AI v2 until the shopper types an exit word."""
    global llm  # NEW: lets chat() swap the model
    print(f"SmartCart AI v2 [{BACKEND}] -- 'quit' to exit, 'new' to restart, "
          "'use cloud' / 'use local' to switch")
    while True:
        user_input = input("\nYou: ")
        command = user_input.strip().lower()
        if command in EXIT_WORDS:
            print("Kart: Thanks for shopping with SmartCart. Bye!")
            break

%%writefile smartcart_v2.py
import os
import json
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_backends import get_llm  # NEW

BACKEND = os.environ.get("SMARTCART_BACKEND", "cloud")  # NEW: "cloud" or "local"
llm = get_llm(BACKEND)  # NEW

EXIT_WORDS = ["quit", "exit", "bye"]
MAX_CHARS = 500
FRIENDLY_ERROR = "Sorry, I'm having trouble right now. Please try again in a moment."

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:
   Step 1 list what the shopper needs, Step 2 check each product, Step 3 give a verdict.
Always reply in JSON with exactly these keys:
{"steps": ["reasoning steps, or an empty list if this is not a comparison"],
 "reply": "a short friendly message, under 60 words",
 "products": [{"name": "text", "price_rs": number,
               "category": "Electronics or Grocery or Apparel",
               "reason": "one short sentence"}]}"""

MEMORY_SIZE = 10
history = []

def check_input(text):  # NEW: copied from Lab 10
    """Return (cleaned_text, problem). problem is "" when the input is fine."""
    cleaned = " ".join(text.split())
    if cleaned == "":
        return cleaned, "Please type what you are shopping for."
    if len(cleaned) > MAX_CHARS:
        return cleaned, f"Please keep your request under {MAX_CHARS} characters."
    return cleaned, ""

def ask_smartcart_v2(text):
    """Send the message with memory; return the reply as a dictionary."""
    messages = ([SystemMessage(content=SYSTEM_PROMPT)]
                + history[-MEMORY_SIZE:]
                + [HumanMessage(content=text)])
    response = llm.invoke(messages)
    data = json.loads(response.content)
    history.append(HumanMessage(content=text))
    history.append(AIMessage(content=response.content))
    return data

def show(data):
    """Print the reasoning steps, the message and the product cards."""
    for step in data.get("steps", []):
        print("  (thinking)", step)
    print("Kart:", data.get("reply", ""))
    for p in data.get("products", []):
        print(f"  [{p['category']}] {p['name']} – Rs. {p['price_rs']:,} – {p['reason']}")

def chat():
    """Run SmartCart AI v2 until the shopper types an exit word."""
    global llm  # NEW: lets chat() swap the model
    print(f"SmartCart AI v2 [{BACKEND}] -- 'quit' to exit, 'new' to restart, "
          "'use cloud' / 'use local' to switch")
    while True:
        user_input = input("\nYou: ")
        command = user_input.strip().lower()
        if command in EXIT_WORDS:
            print("Kart: Thanks for shopping with SmartCart. Bye!")
            break
        if command == "new":
            history.clear()
            print("Kart: Okay, starting fresh! What are you shopping for?")
            continue
        if command in ["use cloud", "use local"]:  # NEW
            backend = command.split()[1]
            llm = get_llm(backend)
            print(f"Kart: Switched to the {backend} model. I still remember our chat.")
            continue
        cleaned, problem = check_input(user_input)
        if problem:
            print("Kart:", problem)
            continue
        try:
            show(ask_smartcart_v2(cleaned))
        except json.JSONDecodeError:
            print("Kart: Sorry, something went wrong with my answer. Please ask again.")
        except Exception as error:
            print("[log]", type(error).__name__, "-", str(error)[:200])
            print("Kart:", FRIENDLY_ERROR)

if __name__ == "__main__":
    chat()

Switch Model to cloud in middle of conversation

os.environ["SMARTCART_BACKEND"] = "local"
%run smartcart_v2.py

Output

Task 7 : Test both Backends

import time
import importlib
import pandas as pd
import smartcart_backends
import smartcart_v2
importlib.reload(smartcart_backends)   # pick up any re-written files
importlib.reload(smartcart_v2)
from smartcart_backends import get_llm

import time
import importlib
import pandas as pd
import smartcart_backends
import smartcart_v2
importlib.reload(smartcart_backends)   # pick up any re-written files
importlib.reload(smartcart_v2)
from smartcart_backends import get_llm

tests = [
    "gym earbuds under Rs. 2,000",
    "make the second one cheaper",
    "who won the cricket match yesterday",
    "What is SmartCart's return policy for opened electronics?",
    "Compare boAt Airdopes 141 and Noise Buds VS104 for the gym",
    "office ke liye formal shirt chahiye under 1500 rupees",]

rows = []
for backend in ["cloud", "local"]:
    smartcart_v2.llm = get_llm(backend)
    smartcart_v2.history.clear()
    for text in tests:
        start = time.time()
        error = ""
        try:
            data = smartcart_v2.ask_smartcart_v2(text)
            if not isinstance(data, dict):
                raise ValueError("JSON was not an object")
            valid = "Yes"
        except Exception as e:
            data, valid = {}, "No"
            error = f"{type(e).__name__}: {str(e)[:60]}"
        rows.append({
            "backend": backend,
            "input": text[:28],
            "seconds": round(time.time() - start, 2),
            "valid_json": valid,
            "products": len(data.get("products", [])),
            "steps": len(data.get("steps", [])),
            "reply": data.get("reply", ""),
            "error": error,})

import time
import importlib
import pandas as pd
import smartcart_backends
import smartcart_v2
importlib.reload(smartcart_backends)   # pick up any re-written files
importlib.reload(smartcart_v2)
from smartcart_backends import get_llm

tests = [
    "gym earbuds under Rs. 2,000",
    "make the second one cheaper",
    "who won the cricket match yesterday",
    "What is SmartCart's return policy for opened electronics?",
    "Compare boAt Airdopes 141 and Noise Buds VS104 for the gym",
    "office ke liye formal shirt chahiye under 1500 rupees",]

rows = []
for backend in ["cloud", "local"]:
    smartcart_v2.llm = get_llm(backend)
    smartcart_v2.history.clear()
    for text in tests:
        start = time.time()
        error = ""
        try:
            data = smartcart_v2.ask_smartcart_v2(text)
            if not isinstance(data, dict):
                raise ValueError("JSON was not an object")
            valid = "Yes"
        except Exception as e:
            data, valid = {}, "No"
            error = f"{type(e).__name__}: {str(e)[:60]}"
        rows.append({
            "backend": backend,
            "input": text[:28],
            "seconds": round(time.time() - start, 2),
            "valid_json": valid,
            "products": len(data.get("products", [])),
            "steps": len(data.get("steps", [])),
            "reply": data.get("reply", ""),
            "error": error,})

results = pd.DataFrame(rows)
results.to_csv("smartcart_v2_backend_comparison.csv", index=False)
print(results.drop(columns="reply").to_string(index=False))
print("\nAverage seconds:",
      results.groupby("backend")["seconds"].mean().round(2).to_dict())

Output

Great job!

You have successfully installed Ollama, ran an open-weight model locally, integrated it through LangChain and compared cloud and local latency. You estimated cloud token costs, examined shopper data privacy, and made SmartCart AI v2 backend-swappable using get_llm(), testing both backends on the same conversation.

Checkpoint

Git Push

git push origin branchName

Next-Lab Preparation

Topic : Giving Models Memory Embedding & Similarity

1) Text Embeddings

2) Embedding Models

3) Cosine Similarity