LLM Access : Running Models Locally

Business Scenario

Welcome!

Previously, you used Chain of Thought prompting for product and price comparisons, enforced valid JSON and JsonOutputParser in a LangChain, converted JSON responses into product cards, and checked budgets. You also added memory and built SmartCart AI v2, successfully passing all tests that v1 failed.​

Today, your manager has assigned you a task to​ run SmartCart AI on a local open-source model as an alternative to GroqCloud. You will compare the cost, speed, & response quality of both options, understand privacy and reliability benefits of running a model locally, & update Kart so you can switch between cloud and local models without changing its features.

Git Pull

git pull origin branchName

Pre-Lab Preparation

Dataset :

Topic : LLM Access - Running Models Locally

1) Running LLM Locally

2) Open-weight vs API Models

3) Swapping the provider

Task 1: Install Ollama and Start the Server

!sudo apt-get install -y -qq zstd # needed by the Ollama installer
!curl -fsSL https://ollama.com/install.sh | sh
import subprocess, time
subprocess.Popen(["ollama", "serve"],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
time.sleep(5) # give the server a moment to start
!curl -s http://localhost:11434

Task 2 : Chain of Thought Prompting

Download model & check if it is installed

1

!ollama pull qwen2.5:3b
!ollama list
!ollama run qwen2.5:3b "Recommend 3 gym earbuds under Rs. 2,000, one line each."

Test with command

2

Output

Task 3: Call the Local Model with Langchain

!pip install -q -r requirements.txt langchain-ollama
!echo "langchain-ollama" >> requirements.txt

import os
from google.colab import userdata
os.environ["GROQ_API_KEY"] = userdata.get("GROQ_API_KEY") # still needed for the cloudtests
from langchain_ollama import ChatOllama


!pip install -q -r requirements.txt langchain-ollama
!echo "langchain-ollama" >> requirements.txt

import os
from google.colab import userdata
os.environ["GROQ_API_KEY"] = userdata.get("GROQ_API_KEY") # still needed for the cloudtests
from langchain_ollama import ChatOllama

local_llm = ChatOllama(
 model="qwen2.5:3b",
 temperature=0.7,
 top_p=0.95,
 num_predict=300, # Ollama's name for max_tokens)
response = local_llm.invoke("Recommend 3 gym earbuds under Rs. 2,000, one line each.")
print(response.content)
print("\nToken usage :", response.usage_metadata)
print("Finish :", response.response_metadata["done_reason"])

Output

Task 4 : Measure Latency

import time
from smartcart_v1 import llm as cloud_llm


import time
from smartcart_v1 import llm as cloud_llm

def time_call(model, prompt):
 """Return how many seconds one call takes."""
 start = time.time()
 model.invoke(prompt)
 return round(time.time() - start, 2)

prompt = "Recommend 3 gym earbuds under Rs. 2,000, one line each."

!ollama stop qwen2.5:3b

print("Local cold start:", time_call(local_llm, prompt), "s")
for name, model in [("Cloud", cloud_llm), ("Local", local_llm)]:
 times = [time_call(model, prompt) for run in range(3)]
 print(f"{name} (warm): {times} -> average {sum(times) / 3:.2f} s")

Output

Task 5 : Cost & Privacy Trade-offs

usage = cloud_llm.invoke(prompt).usage_metadata
tokens_per_request = usage["total_tokens"]

REQUESTS_PER_MONTH = 300_000 # SmartCart estimate: 10,000 chats a day
PRICE_PER_MILLION_TOKENS = 0.30 # USD -- example only, check groq.com/pricing
usage = cloud_llm.invoke(prompt).usage_metadata
tokens_per_request = usage["total_tokens"]

REQUESTS_PER_MONTH = 300_000 # SmartCart estimate: 10,000 chats a day
PRICE_PER_MILLION_TOKENS = 0.30 # USD -- example only, check groq.com/pricing

monthly_tokens = tokens_per_request * REQUESTS_PER_MONTH
monthly_cost = monthly_tokens / 1_000_000 * PRICE_PER_MILLION_TOKENS

print(f"Tokens per request : {tokens_per_request}")
print(f"Tokens per month : {monthly_tokens:,}")
print(f"Cloud cost / month : ${monthly_cost:,.2f}")
print("Local cost / month : $0 in tokens -- but you pay for the machine")

Output

Task 6 : Make SmartCart v2 Backend Swappable

%%writefile smartcart_backends.py
import os
from langchain_groq import ChatGroq
from langchain_ollama import ChatOllama

CLOUD_MODEL = "qwen/qwen3.8-27b"
LOCAL_MODEL = "qwen2.5:3b"

Save the backend file

1

%%writefile smartcart_backends.py
import os
from langchain_groq import ChatGroq
from langchain_ollama import ChatOllama

CLOUD_MODEL = "qwen/qwen3.8-27b"
LOCAL_MODEL = "qwen2.5:3b"

def get_llm(backend="cloud"):
    """Return the chat model for 'cloud' (GroqCloud) or 'local' (Ollama)."""
    if backend == "local":
        return ChatOllama(
            model=LOCAL_MODEL,
            temperature=0.3,
            num_predict=800,
            format="json",  # JSON mode in Ollama)
    if backend == "cloud":
        if not os.environ.get("GROQ_API_KEY"):
            raise SystemExit("GROQ_API_KEY not found. Add it to Colab Secrets (Lab 9, Task 2).")
        return ChatGroq(
            model=CLOUD_MODEL,
            temperature=0.3,
            max_tokens=800,
            model_kwargs={"response_format": {"type": "json_object"}},
            reasoning_effort="none",)
    raise ValueError(f"Unknown backend '{backend}'. Use 'cloud' or 'local'.")

Update SmartCart v2

2

%%writefile smartcart_v2.py
import os
import json
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_backends import get_llm  # NEW

BACKEND = os.environ.get("SMARTCART_BACKEND", "cloud")  # NEW: "cloud" or "local"
llm = get_llm(BACKEND)  # NEW
%%writefile smartcart_v2.py
import os
import json
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_backends import get_llm  # NEW

BACKEND = os.environ.get("SMARTCART_BACKEND", "cloud")  # NEW: "cloud" or "local"
llm = get_llm(BACKEND)  # NEW

EXIT_WORDS = ["quit", "exit", "bye"]
MAX_CHARS = 500
FRIENDLY_ERROR = "Sorry, I'm having trouble right now. Please try again in a moment."

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:
   Step 1 list what the shopper needs, Step 2 check each product, Step 3 give a verdict.
Always reply in JSON with exactly these keys:
{"steps": ["reasoning steps, or an empty list if this is not a comparison"],
 "reply": "a short friendly message, under 60 words",
 "products": [{"name": "text", "price_rs": number,
               "category": "Electronics or Grocery or Apparel",
               "reason": "one short sentence"}]}"""

MEMORY_SIZE = 10
history = []

def check_input(text):  # NEW: copied from Lab 10
    """Return (cleaned_text, problem). problem is "" when the input is fine."""
    cleaned = " ".join(text.split())
    if cleaned == "":
        return cleaned, "Please type what you are shopping for."
    if len(cleaned) > MAX_CHARS:
        return cleaned, f"Please keep your request under {MAX_CHARS} characters."
    return cleaned, ""
%%writefile smartcart_v2.py
import os
import json
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_backends import get_llm  # NEW

BACKEND = os.environ.get("SMARTCART_BACKEND", "cloud")  # NEW: "cloud" or "local"
llm = get_llm(BACKEND)  # NEW

EXIT_WORDS = ["quit", "exit", "bye"]
MAX_CHARS = 500
FRIENDLY_ERROR = "Sorry, I'm having trouble right now. Please try again in a moment."

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:
   Step 1 list what the shopper needs, Step 2 check each product, Step 3 give a verdict.
Always reply in JSON with exactly these keys:
{"steps": ["reasoning steps, or an empty list if this is not a comparison"],
 "reply": "a short friendly message, under 60 words",
 "products": [{"name": "text", "price_rs": number,
               "category": "Electronics or Grocery or Apparel",
               "reason": "one short sentence"}]}"""

MEMORY_SIZE = 10
history = []

def check_input(text):  # NEW: copied from Lab 10
    """Return (cleaned_text, problem). problem is "" when the input is fine."""
    cleaned = " ".join(text.split())
    if cleaned == "":
        return cleaned, "Please type what you are shopping for."
    if len(cleaned) > MAX_CHARS:
        return cleaned, f"Please keep your request under {MAX_CHARS} characters."
    return cleaned, ""

def ask_smartcart_v2(text):
    """Send the message with memory; return the reply as a dictionary."""
    messages = ([SystemMessage(content=SYSTEM_PROMPT)]
                + history[-MEMORY_SIZE:]
                + [HumanMessage(content=text)])
    response = llm.invoke(messages)
    data = json.loads(response.content)
    history.append(HumanMessage(content=text))
    history.append(AIMessage(content=response.content))
    return data

def show(data):
    """Print the reasoning steps, the message and the product cards."""
    for step in data.get("steps", []):
        print("  (thinking)", step)
    print("Kart:", data.get("reply", ""))
    for p in data.get("products", []):
        print(f"  [{p['category']}] {p['name']} – Rs. {p['price_rs']:,} – {p['reason']}")

def chat():
    """Run SmartCart AI v2 until the shopper types an exit word."""
    global llm  # NEW: lets chat() swap the model
    print(f"SmartCart AI v2 [{BACKEND}] -- 'quit' to exit, 'new' to restart, "
          "'use cloud' / 'use local' to switch")
    while True:
        user_input = input("\nYou: ")
        command = user_input.strip().lower()
        if command in EXIT_WORDS:
            print("Kart: Thanks for shopping with SmartCart. Bye!")
            break


%%writefile smartcart_v2.py
import os
import json
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_backends import get_llm  # NEW

BACKEND = os.environ.get("SMARTCART_BACKEND", "cloud")  # NEW: "cloud" or "local"
llm = get_llm(BACKEND)  # NEW

EXIT_WORDS = ["quit", "exit", "bye"]
MAX_CHARS = 500
FRIENDLY_ERROR = "Sorry, I'm having trouble right now. Please try again in a moment."

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:
   Step 1 list what the shopper needs, Step 2 check each product, Step 3 give a verdict.
Always reply in JSON with exactly these keys:
{"steps": ["reasoning steps, or an empty list if this is not a comparison"],
 "reply": "a short friendly message, under 60 words",
 "products": [{"name": "text", "price_rs": number,
               "category": "Electronics or Grocery or Apparel",
               "reason": "one short sentence"}]}"""

MEMORY_SIZE = 10
history = []

def check_input(text):  # NEW: copied from Lab 10
    """Return (cleaned_text, problem). problem is "" when the input is fine."""
    cleaned = " ".join(text.split())
    if cleaned == "":
        return cleaned, "Please type what you are shopping for."
    if len(cleaned) > MAX_CHARS:
        return cleaned, f"Please keep your request under {MAX_CHARS} characters."
    return cleaned, ""

def ask_smartcart_v2(text):
    """Send the message with memory; return the reply as a dictionary."""
    messages = ([SystemMessage(content=SYSTEM_PROMPT)]
                + history[-MEMORY_SIZE:]
                + [HumanMessage(content=text)])
    response = llm.invoke(messages)
    data = json.loads(response.content)
    history.append(HumanMessage(content=text))
    history.append(AIMessage(content=response.content))
    return data

def show(data):
    """Print the reasoning steps, the message and the product cards."""
    for step in data.get("steps", []):
        print("  (thinking)", step)
    print("Kart:", data.get("reply", ""))
    for p in data.get("products", []):
        print(f"  [{p['category']}] {p['name']} – Rs. {p['price_rs']:,} – {p['reason']}")

def chat():
    """Run SmartCart AI v2 until the shopper types an exit word."""
    global llm  # NEW: lets chat() swap the model
    print(f"SmartCart AI v2 [{BACKEND}] -- 'quit' to exit, 'new' to restart, "
          "'use cloud' / 'use local' to switch")
    while True:
        user_input = input("\nYou: ")
        command = user_input.strip().lower()
        if command in EXIT_WORDS:
            print("Kart: Thanks for shopping with SmartCart. Bye!")
            break


BUDGET = 2000
for product in data["products"]:
 within = "Yes" if product["price_rs"] <= BUDGET else "NO"
 print("+-------------------------------------------")
 print("| " + product["name"])
 print(f"| Rs. {product['price_rs']:,} | {product['category']}")
 print("| " + product["reason"])
 print(f"| Within budget: {within}")
print("+-------------------------------------------")

Output

Use a data like app would print products cards and check budget

2

Task 5 : Add Conversational Memory

history = [] # grows by 2 messages every turn
MEMORY_SIZE = 10 # send only the last 10 messages (5 turns)
def chat_with_memory(text):
 """Send the new message plus recent history; save both sides."""
 messages = ([SystemMessage(content=SYSTEM_PROMPT)]
 + history[-MEMORY_SIZE:]
 + [HumanMessage(content=text)])
 reply = llm.invoke(messages).content
 history.append(HumanMessage(content=text))
 history.append(AIMessage(content=reply))
 return reply
print("Kart:", chat_with_memory("running shoes for women under 3000"), "\n")
print("Kart:", chat_with_memory("make the second one cheaper"), "\n")
print("Messages in memory:", len(history))

Output

Task 6 : Build SmartCart v2

%%writefile smartcart_v2.py
import os
import json
from langchain_groq import ChatGroq
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_v1_chat import check_input, EXIT_WORDS, FRIENDLY_ERROR

if not os.environ.get("GROQ_API_KEY"):
    raise SystemExit("GROQ_API_KEY not found. Add it to Colab Secrets (Lab 9, Task 2).")

llm = ChatGroq(
    model="qwen/qwen3.8-27b",
    temperature=0.3,
    max_tokens=800,
    model_kwargs={"response_format": {"type": "json_object"}},  # JSON mode
    reasoning_effort="none",)

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:

Save the Script

1

%%writefile smartcart_v2.py
import os
import json
from langchain_groq import ChatGroq
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_v1_chat import check_input, EXIT_WORDS, FRIENDLY_ERROR

if not os.environ.get("GROQ_API_KEY"):
    raise SystemExit("GROQ_API_KEY not found. Add it to Colab Secrets (Lab 9, Task 2).")

llm = ChatGroq(
    model="qwen/qwen3.8-27b",
    temperature=0.3,
    max_tokens=800,
    model_kwargs={"response_format": {"type": "json_object"}},  # JSON mode
    reasoning_effort="none",)

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:
   Step 1 list what the shopper needs, Step 2 check each product, Step 3 give a verdict.
Always reply in JSON with exactly these keys:
{"steps": ["reasoning steps, or an empty list if this is not a comparison"],
 "reply": "a short friendly message, under 60 words",
 "products": [{"name": "text", "price_rs": number,
               "category": "Electronics or Grocery or Apparel",
               "reason": "one short sentence"}]}"""

MEMORY_SIZE = 10
history = []


def ask_smartcart_v2(text):
    """Send the message with memory; return the reply as a dictionary."""
    messages = ([SystemMessage(content=SYSTEM_PROMPT)]
                + history[-MEMORY_SIZE:]
                + [HumanMessage(content=text)])
    response = llm.invoke(messages)
    data = json.loads(response.content)  # JSON text -> dictionary
    history.append(HumanMessage(content=text))
    history.append(AIMessage(content=response.content))
    return data


def show(data):
    """Print the reasoning steps, the message and the product cards."""
    for step in data.get("steps", []):
        print("  (thinking)", step)
    print("Kart:", data.get("reply", ""))
    for p in data.get("products", []):
        print(f"  [{p['category']}] {p['name']} – Rs. {p['price_rs']:,} – {p['reason']}")
%%writefile smartcart_v2.py
import os
import json
from langchain_groq import ChatGroq
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
from smartcart_v1_chat import check_input, EXIT_WORDS, FRIENDLY_ERROR

if not os.environ.get("GROQ_API_KEY"):
    raise SystemExit("GROQ_API_KEY not found. Add it to Colab Secrets (Lab 9, Task 2).")

llm = ChatGroq(
    model="qwen/qwen3.8-27b",
    temperature=0.3,
    max_tokens=800,
    model_kwargs={"response_format": {"type": "json_object"}},  # JSON mode
    reasoning_effort="none",)

SYSTEM_PROMPT = """You are Kart, SmartCart's friendly shopping assistant for shoppers in India.
Rules:
1. Only help with shopping. For anything else, politely say you can only help with shopping.
2. Recommend up to 3 products. Prices are in rupees.
3. Reply in the same style as the shopper: English stays English, Hinglish stays Hinglish.
4. Never make up SmartCart store policies. Say: "Please check the Help page in the SmartCart app."
5. Use the earlier conversation to understand words like "it", "that one" or "the second one".
6. When the shopper compares products, think step by step:
   Step 1 list what the shopper needs, Step 2 check each product, Step 3 give a verdict.
Always reply in JSON with exactly these keys:
{"steps": ["reasoning steps, or an empty list if this is not a comparison"],
 "reply": "a short friendly message, under 60 words",
 "products": [{"name": "text", "price_rs": number,
               "category": "Electronics or Grocery or Apparel",
               "reason": "one short sentence"}]}"""

MEMORY_SIZE = 10
history = []


def ask_smartcart_v2(text):
    """Send the message with memory; return the reply as a dictionary."""
    messages = ([SystemMessage(content=SYSTEM_PROMPT)]
                + history[-MEMORY_SIZE:]
                + [HumanMessage(content=text)])
    response = llm.invoke(messages)
    data = json.loads(response.content)  # JSON text -> dictionary
    history.append(HumanMessage(content=text))
    history.append(AIMessage(content=response.content))
    return data


def show(data):
    """Print the reasoning steps, the message and the product cards."""
    for step in data.get("steps", []):
        print("  (thinking)", step)
    print("Kart:", data.get("reply", ""))
    for p in data.get("products", []):
        print(f"  [{p['category']}] {p['name']} – Rs. {p['price_rs']:,} – {p['reason']}")

def chat():
    """Run SmartCart AI v2 until the shopper types an exit word."""
    print("SmartCart AI v2 -- type 'quit' to exit, 'new' to start a new conversation")
    while True:
        user_input = input("\nYou: ")
        command = user_input.strip().lower()
        if command in EXIT_WORDS:
            print("Kart: Thanks for shopping with SmartCart. Bye!")
            break
        if command == "new":
            history.clear()
            print("Kart: Okay, starting fresh! What are you shopping for?")
            continue
        cleaned, problem = check_input(user_input)
        if problem:
            print("Kart:", problem)
            continue
        try:
            show(ask_smartcart_v2(cleaned))
        except json.JSONDecodeError:
            print("Kart: Sorry, something went wrong with my answer. Please ask again.")
        except Exception as error:
            print("[log]", type(error).__name__, "-", str(error)[:200])
            print("Kart:", FRIENDLY_ERROR)


if __name__ == "__main__":
    chat()

Run it, try a request, a follow up & a comparision

2

%run smartcart_v2.py

Output

Task 7 : Final Test v1 vs v2

import pandas as pd
import smartcart_v2
tests = [
 {"id": "T1", "input": "gym earbuds under Rs. 2,000"},
 {"id": "T6", "input": "make the second one cheaper"},
 {"id": "T7", "input": "who won the cricket match yesterday"},
 {"id": "T8", "input": "What is SmartCart's return policy for opened electronics?"},
 {"id": "T9", "input": "Compare boAt Airdopes 141 and Noise Buds VS104 for the gym"},
 {"id": "T10", "input": "office ke liye formal shirt chahiye under 1500 rupees"},]
smartcart_v2.history.clear()
for case in tests:
 data = smartcart_v2.ask_smartcart_v2(case["input"])
 case["steps"] = len(data.get("steps", []))
 case["products"] = len(data.get("products", []))
 case["reply"] = data.get("reply", "")
results = pd.DataFrame(tests)
results.to_csv("smartcart_v2_test_results.csv", index=False)
print(results[["id", "steps", "products", "reply"]].to_string(max_colwidth=50))

Output

 

Great job!

You have successfully used Chain of Thought prompting for checkable product and price comparisons, enforced valid JSON and JsonOutputParser in a LangChain, converted JSON responses into product cards, and checked budgets with one line of code. You also added memory using the last 10 messages and built SmartCart AI v2, successfully passing all tests that v1 failed.​

Checkpoint

   Git Push

git push origin branchName

Next-Lab Preparation

Topic : LLM Access - Running Models Locally

1) Running LLM Locally

2) Open-weight vs API Models

3) Swapping the provider

GenAI - 13

By Content ITV

GenAI - 13

  • 5

More from Content ITV