Efficient programming with Scala and LLMs
Tomasz Godzik
VirtusLab
Maintainer of multiple Scala tools including Metals, Bloop, Scalameta, Munit, Mdoc and parts of the Scala 3 compiler.
Release officer for Scala LTS versions
Part of the Scala Core team as coordinator and VirtusLab representative
Part of the moderation team
What do I do?
How smart are LLMs really
01
02
03
04
Prompting 101
Context Engineering
Your new work environment
05
Approaches to working with LLMs
06
Questions
Presentation is based on the work by myself, Łukasz Biały and Piotr Oramus
I will not cover everything that you might need, but just give you a taste of what is currently happening
How smart are LLMs really?
For sure it's easy to fool them!
That was a few months ago
You can still fool it.
LLMs has fallen into well known paths
What is it that modern LLMs really generate?
text that COULD be the answer
Is AGI already here?
Is AGI already here?
Opinions differ, but LLMs are getting better.
What modern LLMs really are?
What modern LLMs really are?
autocomplete on steroids super-soldier serum
What modern LLMs really are?
autocomplete on steroids super-soldier serum
Something more?
The hype can get too much at times though.
The hype can get too much at times though.
It probably is
One thing is certain, AI will not replace good engineers.
We need to understand what LLMs are capable and how to make them work for us
Prompting 101
> Create a simple todo app
Is this a good prompt?
> Create a simple todo app
Is this a good prompt?
Obviously not
You are a javafx / scalafx expert tasked with creating a simple todo app using Scala 3. Use scala-cli to set up, run and test the project.
Structure of a prompt
You are a javafx / scalafx expert tasked with creating a simple todo app using Scala 3. Use scala-cli to set up, run and test the project.
Structure of a prompt
persona
task
context
You are a javafx / scalafx expert tasked with creating a simple todo app using Scala 3. Use scala-cli to setup, run and test the project. Here’s a snippet of Scalafx specific APIs you can use:
// Aggregate operator
val r1, r2, r3 = new Rectangle()
r1.width <== max(r2.width, r3.width)
// Conditional binding
r2.fill <== when (r2.hover) choose Red otherwise Blue
// Complex boolean + string concat
val tf = new TextField { text = "Hello" }
val lbl = new Label()
lbl.text <== when (r1.hover || r2.hover)
...
Structure of a prompt
-
Persona competence sizing: expert vs regular - allows to limit the amount of concerns the LLM will try to juggle with.
-
In-context learning: few-shot prompting > zero-shot prompting for output fitting the precise requirements
-
Bias management: it’s always good to watch how your prompt is biasing the model - maybe the solution you have in mind is not the best one?
Properties
- Multimodal prompting: Different types of input, not just text, but voice images etc.
-
Prompt chaining: Each conversation produces artifacts for the next prompt
-
Reasoning models: Ask the model to explain it's reasoning, default in some models.
Prompting techniques
- Tree of thought: Explore possible options, make model offer them and choose each time one that suits you better
-
Meta prompting: Prompt to create new prompts for later models
-
Agentic simulations: Make agents fulfill some persona and simulate things like recruitment calls etc.
Prompting techniques
It's a bit of an alchemy
Some of the existing techniques made it into tools.
The more people come up with the more the tooling catches up.
The ideas are basically endless.
Prompting
Context engineering
Context one is king.
Should we just put a lot of information
into the prompt’s context?
Obviously not
Should we just put all the information
into the prompt’s context?
Context drift
When the conversation gets reused for new tasks, the previous results end up confusing the model’s attention mechanisms
Context overload
When the input provided with the initial prompt is big enough, it will confuse the model and lead it astray from the original task.
Costs!
More data in context, just gets attached before the prompt and uses more tokens, which cost money.
Rules
1. Provide only the information (and capabilities) necessary to execute the task at hand
Rules
2. Prefer to work in shorter conversations that don’t reach the full size of the available context window
Rules
3. Use compaction and external progress tracking to distill and preserve important decisions and insightsContext compaction
> Summarize the current decisions and progress so that we can use it in another conversationProgress tracking
Use local files for tracking specification and TODOs for the agents to pick up.
Optimizing the costs
-
Always provide the same prefix, each LLM query is cached
-
If you change it will need to recalculate everything
-
This cache will only be available for a time
Your new work envronment
ask: does not change anything, used for exploration
plan: create documents need for further work, any specification timeline etc.
agent: actually apply changes, test code, compile until the results are satifactory
Typical usages of LLMs
ask: does not change anything, used for exploration
plan: create documents need for further work, any specification timeline etc.
agent: actually apply changes, test code, compile until the results are satifactory
Typical usages of LLMs
Most of the time you will use
Agents
They can use a variety of tools to enrich their context and verify the results of their work.
Every development tool might be useful for you, but some might need work.
Model Context Protocol
One of the more popular tools to improve the output of LLMs
MCP uses Json RPC, so each possible request and response is represented by json schema
Model Context Protocol
Model Context Protocol is very similar to Language Server Protocol, but instead of people it's for agents.
Host with MCP Client
MCP protocol
MCP protocol
MCP protocol
MCP Server 1
MCP Server 2
MCP Server 2
Local Data Source 1
Local Data Source 2
Web API
Model Context Protocol
- tools: longer running tasks that agents can schedule
- resources: static context available to load lazily
- prompts: possible meta prompts to use by the users in the chat
Metals is a model context protocol server.
compilation: compile-full, compile-module, compile-file test: test analysis: glob-search, inspect, get-docs, get-usages utility: import-build, format-file, find-dep, run-scalafix-rule, list-scalafix-rules
Available tools
Problems with MCP
- The more tools you have the more context is taken up.
- Model efficiency might go down and it might start hallucinating more.
- Data returned from the tools might go through the model adding to the context.
Alternative: Executable tools
Instead of ingesting the entire context, create smaller tool to change codebase or extract information.
Using typed languages like Scala makes it significantly more efficient.
Alternative: Executable tools
- The agent queries each toolset it needs and creates executable code
// ./skills/getDocsFor.scala
import metals.mcp.*
@main
def main(nameToLookFor: String) =
val allMatching = globSearch(fullyQualifiedName, None)
val fqcns = allMatching.split("\n")
val result = fqcns.map{
fullName => getDocs(fullName)
}.mkString("\n")
println(result)
}
Security
- You still need to check the code and make sure it does what you want it to do.
- You cannot YOLO mode it for sure.
- Martin already told you about safe capabilities, which could make it much safer.
- You can also sandbox it! github.com/VirtusLab/sandcat
Tools for agents: Skills
---
name: your-skill-name
description: Brief description of what this Skill does and when to use it
---
# Your Skill Name
## Instructions
[Clear, step-by-step guidance for Claude to follow]
## Examples
[Concrete examples of using this Skill]Tools for agents: Skills
- Each skill could contain scripts or mini libraries that the agents can use
# Glob search
## Quick start
Run the script if you want to return users entire
documentation based on specific glob
(part of the name user is looking for)
```bash
scala-cli search-script.scala
```
Approaches to working with LLMs
These scripts could be added as skills to you company repo.
Some of the scripts could improve CI as it doesn't require a lot of code.
Automize any manual work even if normally wrting a script would take longer than the work
Creating small usability scripts
Brownfield projects
These are projects that we previously written using normal engineering practises and we want to add feature or fix bugs.
Brownfield projects
Might be harder to use LLMs in, tasks should be scoped, all data available that the LLM afent might need.
Provide as many tools as needed to make sure the context is spot on.
A lot depends on the complexity and the existing quality of the code.
We can also build something from scratch with LLMs in mind.
Makes it possible to create small to mid project almost automatically with guidelines from the developer
Greenfield projects
Exploration
Greenfield projects
Product planning
Component analysis
Implementation
Full product ideaFull project specificationSpecification with all modules plannedHalf a year ago was not feasable, but becoming more and more
It is more expensive and complex to monitor
But can for example be used to implement multiple components at the same time or make product exploration faster
Multiagent orchestration
Summary
Verify if the code is maintainable, extendable and optimal.
Make sure your prompts are sound and context provided is well scoped.
You need to know your domain to be able to verify the output, use tools to improve that
Thank You
Tomasz Godzik
Bluesky: @tgodzik.bsky.social
Mastodon: fosstodon.org/@tgodzik
Discord: tgodzik