Let's use LLMs from Ruby

Leaner Technologies, Inc.
Kokuyou (黒曜) / Shunsuke Mori

@kokuyouwind

~ Refine RBS types using LLM ~

Talk: Japanese
Slides: English (+ Japanese)

トーク: 日本語
スライド: 日英併記

$ whoami

Kokuyou (黒曜) / Shunsuke Mori

Twitter or X: @kokuyouwind
Work at: Leaner Technologies, Inc.

Platinum / Drinkup Sponsor
Day 2: Leaner YAKINIKU Party
Procurement Domain BtoB Startup

調達領域のBtoBスタートアップ

We're Hiring!!!

エンジニア絶賛採用中！

Feel free to talk/DM us!
気軽に話しかけたり
DMしてください！

https://careers.leaner.co.jp/

I'll talk about one that has
no relation to my job at all.

業務とは全然関係のない話をします

Lightning Talk at RubyKaigi 2023

RubyKaigi 2023 でのライトニングトーク

https://rubykaigi.org/2023/presentations/lt/

Matz Keynote at RubyKaigi 2023

RubyKaigi 2023 でのMatzの基調講演

https://speakerdeck.com/matz/30-years-of-ruby?slide=79

Oh no, Matz presented on the same topic

I was planning to cover!

https://twitter.com/kokuyouwind/status/1656488513666453505​

This Year...

今年は…

😌

Large Language Model (LLM)

Machine learning models trained on large amounts of text data.

OpenAI

ChatGPT 4

Anthropic

Claude 3

Microsoft
GitHub Copilot

大量のテキストデータを使ってトレーニングされた機械学習モデル

大規模言語モデル

LLM Example

LLMの例

https://claude.ai/

Evolution of LLM since last year

昨年からのLLMの進化

Price Reduction

💰

値下げ

Sentence Length Expansion

扱える文章長の拡大

x 1/8

📚

x 480

Improvement of coding skills

コーディング能力の向上

+ 23.2 pt

(OpenAI gpt-3.5-turbo-0301 to
Anthropic Claude3 Haiku)

(OpenAI gpt-3.5-turbo-0301 to
Google Gemini 1.5 Pro 2M)

(GPT-4 to GPT-4o,

HumanEval)

4096 → 2M

$2.0 → $.025 / 1MTok

67.0% → 90.2%

📝

🤔

Now that LLM's capabilities have increased so much,

can we even guess RBS types for the entire project?

これだけLLMの能力が上がったなら、
プロジェクト全体のRBSの型推測もワンチャンいけるのでは…！？

🤨

Now that LLM's capabilities have increased so much,

can we even guess RBS types for the entire project?

これだけLLMの能力が上がったなら、
プロジェクト全体のRBSの型推測もワンチャンいけるのでは…！？

😲

Now that LLM's capabilities have increased so much,

can we even guess RBS types for the entire project?

Goose

I made

as a tool to guess RBS types.

Duck

quacking like geese (Duck Goose Typing)

Duck

I talk about the creation of RBS Goose,

a tool to guess RBS types from Ruby.

RubyコードからRBS型を推測するツール、

RBS Gooseを作った話をします

Headline

目次

Purpose of creating RBS Goose

How RBS Goose works

Tips for Development with LLM

Performance Evaluation of RBS Goose

Conclusion

RBS Gooseを作った目的

RBS Gooseの仕組み

LLMを使った開発のTips

RBS Gooseの性能評価

まとめ

Headline

目次

Purpose of creating RBS Goose

How RBS Goose works

Tips for Development with LLM

Performance Evaluation of RBS Goose

Conclusion

RBS Gooseを作った目的

RBS Gooseの仕組み

LLMを使った開発のTips

RBS Gooseの性能評価

まとめ

I thought it would be interesting

if I could guess the type with LLM.

LLMで型が推測できたら面白そうだったから。

Fin.

完

Let me give you

a little more background.

もうすこし背景の話をします。

What is RBS?

RBSとは？

Language for defining Ruby type structures

Rubyの型構造を定義するための言語

class Person
  attr_reader :name

  def initialize(name:)
    @name = name
  end

  def name=(name)
    @name = name
  end
end

class Person
  @name: String

  attr_reader name: String

  def initialize: (name: String) -> void

  def name=: (String name) -> void
end

person.rb

person.rbs

https://github.com/ruby/rbs​

Why RBS is needed

RBSがなぜ必要か

  • Safe development through type checking

    • Detect invalid method calls, etc., before they are executed

    • use steep check, etc

  • ​Improved development experience, including complements

    • More accurate completion based on type

    • use steep-vscode or TypeProf for IDE, etc

型検査による安全な開発

不正なメソッド呼び出しなどを実行前に検知できる

steep checkなどが利用できる

補完などの開発体験の向上

型に基づいてより正確に補完できる

steep-vscodeやTypeProf for IDE が利用できる

Existing way to generate RBS from Ruby

RubyからRBSを生成する既存手法

rbs prototype rb

rbs prototype runtime

typeprof

Ruby

RBS

static parsing

静的構文解析

Ruby

RBS

dynamic load

動的ロード

Ruby

type level execution

型レベル実行

RBS

Each method has its own strengths and weaknesses,

and it's difficult to generate a perfect RBS in one shot.

手法ごとに一長一短があり、一発で完璧なRBSを生成するのは難しい

rbs prototype rb config.rb

Ruby

RBS

static parsing

静的構文解析

class Config
  def self.configure(&block)
    new.tap(&block)
  end

  %w[hoge fuga piyo].each do |v|
    attr_accessor v
  end
end

# config = Config.configure do |c|
#   c.hoge = 1
#   c.fuga = 'a'
#   c.piyo = :piyo
# end

config.rb

class Config
  def self.configure: () \
      { () -> untyped } -> untyped
end

# All untyped, and
# No attribute accessors

config.rbs

rbs prototype runtime -R config.rb Config

class Config
  def self.configure(&block)
    new.tap(&block)
  end

  %w[hoge fuga piyo].each do |v|
    attr_accessor v
  end
end

# config = Config.configure do |c|
#   c.hoge = 1
#   c.fuga = 'a'
#   c.piyo = :piyo
# end

config.rb

class Config
  def self.configure: () \
      { (*untyped) -> untyped } -> untyped

  public

  def fuga: () -> untyped
  def fuga=: (untyped) -> untyped

  def hoge: () -> untyped
  def hoge=: (untyped) -> untyped

  def piyo: () -> untyped
  def piyo=: (untyped) -> untyped
end

# All Untyped

config.rbs

Ruby

RBS

dynamic load

動的ロード

typeprof config.rb

class Config
  def self.configure(&block)
    new.tap(&block)
  end

  %w[hoge fuga piyo].each do |v|
    attr_accessor v
  end
end

# Required for Type Level Exec
config = Config.configure do |c|
  c.hoge = 1
  c.fuga = 'a'
  c.piyo = :piyo
end

config.rb

class Config
  def self.configure: { (Config) -> :piyo } -> Config
end

# Typed, but
# No attribute accessors 
# (and I want the return value to be void.)

config.rbs

Ruby

type level execution

型レベル実行

RBS

It's pretty hard to fix untyped

or make up for what's missing.

untypedを直したり、足りないものを補うのは結構大変

What I want to do with RBS Goose

Ruby

Existing Tools

既存ツール

RBS

RBS Gooseで何をしたいか

Guessing by LLM

LLMによる推測

Refined
RBS

Untyped becomes concrete type

The missing methods are compensated for

untypedが具体型になり、不足メソッドが補われる

https://github.com/kokuyouwind/rbs_goose​

I must first say that

it has not been realized to a practical level.

先に断っておくと、実用レベルまでは実現できていません

Headline

目次

Purpose of creating RBS Goose

How RBS Goose works

Tips for Development with LLM

Performance Evaluation of RBS Goose

Conclusion

RBS Gooseを作った目的

RBS Gooseの仕組み

LLMを使った開発のTips

RBS Gooseの性能評価

まとめ

Example for explanation

説明のための例

class Person
  attr_reader :name

  def initialize(name:)
    @name = name
  end

  def name=(name)
    @name = name
  end
end

class Person
  @name: String

  attr_reader name: String

  def initialize: (name: String) -> void

  def name=: (String name) -> void
end

lib/person.rb

sig/person.rbs

RBS Goose Architecture(Infer Type)

RBS Goose の構成 (型の推測)

Ruby

RBS

Refined RBS

rbs prototype
(or other tools)

examples

Prompt

LLM
(e.g. ChatGPT)

RbsGoose::TypeInferrer#infer

Ruby

RBS

Refined RBS

examples

Prompt

LLM
(e.g. ChatGPT)

class Person
  @name: untyped

  attr_reader name: untyped

  def initialize: (name: untyped) -> void

  def name=: (untyped name) -> void
end

sig/person.rbs

RBS Goose Architecture(Infer Type)

RBS Goose の構成 (型の推測)

rbs prototype
(or other tools)

Ruby

RBS

Refined RBS

examples

Prompt

LLM
(e.g. ChatGPT)

class Example1
  attr_reader :quantity

  def initialize(quantity:)
    @quantity = quantity
  end

  def quantity=(quantity)
    @quantity = quantity
  end
end

lib/example1.rb

class Example1
  @quantity: untyped

  attr_reader quantity: untyped

  def initialize: (quantity: untyped) -> void

  def quantity=: (untyped quantity) -> void
end

sig/example1.rbs

class Example1
  @quantity: Integer

  attr_reader quantity: Integer

  def initialize: (quantity: Integer) -> void

  def quantity=: (Integer quantity) -> void
end

refined/sig/example1.rbs

RBS Goose Architecture(Infer Type)

RBS Goose の構成 (型の推測)

rbs prototype
(or other tools)

Ruby

RBS

Refined RBS

examples

Prompt

LLM
(e.g. ChatGPT)

When ruby source codes and 
RBS type signatures are given, 
refine each RBS type signatures. 

======== Input ========

```lib/example1.rb
...
```

```sig/example1.rbs
...
```

======== Output ========

```sig/example1.rbs
...
```

======== Input ========

```lib/person.rb
...
```

```sig/person.rbs
...
```

======== Output ========

Examples

Ruby Code

LLM Infer

RBS
Prototype

RBS Goose Architecture(Infer Type)

RBS Goose の構成 (型の推測)

rbs prototype
(or other tools)

Ruby

RBS

Refined RBS

steep prototype

examples

Prompt

LLM
(e.g. ChatGPT)

```sig/person.rbs
class Person
  @name: String

  attr_reader name: String

  def initialize: (name: String) -> void

  def name=: (String name) -> void
end
```

RBS Goose Architecture(Infer Type)

RBS Goose の構成 (型の推測)

Ruby

RBS

Refined RBS

examples

Prompt

LLM
(e.g. ChatGPT)

RBS Goose Architecture(Infer Type)

RBS Goose の構成 (型の推測)

rbs prototype
(or other tools)

Multiple File Handling

複数ファイルの扱い

class Person
  attr_reader :name
end

lib/person.rb

class PersonName
  attr_reader :value
end

lib/person_name.rb

class Person
  # Not a String
  @name: PersonName
end

sig/person.rbs

Multiple File Handling: Strategy

複数ファイルの扱い: 戦略

  • Pass all constants list

  • Combine and pass on files that may be related by RAG

  • Infer all Ruby files at once

すべての定数のリストを渡す

RAGで関連しそうなファイルを組み合わせて渡す

すべてのRubyファイルをまとめて1度に推論させる

Multiple File Handling: Adopted

複数ファイルの扱い: 選択したもの

  • Infer all Ruby files at once

    • AI can make comprehensive decisions from all codes

    • Small Project can be stores in 128K tokens

      • Unrealistic as of last year, as 4K was the max​

すべてのRubyファイルをまとめて1度に推論させる

小さなプロジェクトなら128Kトークンに収まる

昨年時点では4Kトークンが最大だったため非現実的だった

AIが全てのコードを見て総合的に判断できる

Act as Ruby type inferrer.
When ruby source codes and RBS type signatures are given, 
refine each RBS type signatures. 
Each file should be split in markdown code format.
Use class names, variable names, etc., to infer type.


========Input========
```ruby:lib/email.rb
class Email
  # @dynamic address
  attr_reader :address

  def initialize(address:)
    @address = address
  end

  def ==(other)
    other.is_a?(self.class) && other.address == address
  end

  def hash
    self.class.hash ^ address.hash
  end
end
```

```rbs:sig/email.rbs
class Email
  @address: untyped

  attr_reader address: untyped

  def initialize: (address: untyped) -> void

  def ==: (untyped other) -> untyped

  def hash: () -> untyped
end
```

```ruby:lib/person.rb
class Person
  # @dynamic name, contacts
  attr_reader :name
  attr_reader :contacts

  def initialize(name:)
    @name = name
    @contacts = []
  end

  def name=(name)
    @name = name
  end

  def guess_country()
    contacts.map do |contact|
      case contact
      when Phone
        contact.country
      end
    end.compact.first
  end
end
```
```rbs:sig/person.rbs
class Person
  @name: untyped

  @contacts: untyped

  attr_reader name: untyped

  attr_reader contacts: untyped

  def initialize: (name: untyped) -> void

  def name=: (untyped name) -> void

  def guess_country: () -> untyped
end
```

```ruby:lib/phone.rb
class Phone
  # @dynamic country, number
  attr_reader :country, :number

  def initialize(country:, number:)
    @country = country
    @number = number
  end

  def ==(other)
    if other.is_a?(Phone)
      # @type var other: Phone
      other.country == country && other.number == number
    else
      false
    end
  end

  def hash
    self.class.hash ^ country.hash ^ number.hash
  end
end
```

```rbs:sig/phone.rbs
class Phone
  @country: untyped

  @number: untyped

  attr_reader country: untyped

  attr_reader number: untyped

  def initialize: (country: untyped, number: untyped) -> void

  def ==: (untyped other) -> (untyped | nil)

  def hash: () -> untyped
end
```

========Output========
```rbs:sig/email.rbs
class Email
  @address: String

  attr_reader address: String

  def initialize: (address: String) -> void

  def ==: (Object other) -> bool

  def hash: () -> Integer
end
```

```rbs:sig/person.rbs
class Person
  @name: String

  @contacts: Array[(Email | Phone)]

  attr_reader name: String

  attr_reader contacts: Array[(Email | Phone)]

  def initialize: (name: String) -> void

  def name=: (String name) -> void

  def guess_country: () -> (String | nil)
end
```

```rbs:sig/phone.rbs
class Phone
  @country: String

  @number: String

  attr_reader country: String

  attr_reader number: String

  def initialize: (country: String, number: String) -> void

  def ==: (Object other) -> (bool | nil)

  def hash: () -> Integer
end
```

========Input========
```ruby:lib/user.rb
class User
  def initialize(name:)
    @name = name
  end

  attr_reader :name
end
```

```rbs:sig/user.rbs
class User
  @name: untyped

  def initialize: (name: untyped) -> void

  attr_reader name: untyped
end
```

```ruby:lib/user_factory.rb
class UserFactory
  def name(name)
    @name = name
    self
  end

  def build
    User.new(name: @name)
  end
end
```

```rbs:sig/user_factory.rbs
class UserFactory
  @name: untyped

  def name: (untyped name) -> self

  def build: () -> untyped
end
```


========Output========

Real Prompt

実際のプロンプト

Most of the time, one shot doesn't work.

We need to look at type errors

and eventually correct them.

たいてい、一発ではうまくいかない。

発生した型エラーを見ながら修正していく必要がある。

Ruby

RBS

Fixed RBS

examples

Prompt

LLM
(e.g. ChatGPT)

RBS Goose Architecture(Fix Error)

RBS Goose の構成 (エラーの修正)

❌

Errors

Steep Check

❌

(Still experimental)

まだ実験的

RbsGoose::TypeInferrer#fix_error

Headline

目次

Purpose of creating RBS Goose

How RBS Goose works

Tips for Development with LLM

Performance Evaluation of RBS Goose

Conclusion

RBS Gooseを作った目的

RBS Gooseの仕組み

LLMを使った開発のTips

RBS Gooseの性能評価

まとめ

RBS Goose Configuration

RBS Goose の設定

I want to allow users to choose which LLM to use.

どの LLM を使うか、ユーザーが選べるようにしたい

Using the LLM framework as an adapter

LLMフレームワークをアダプタとして利用

Use Langchain.rb gem by @andreibondarev

@andreibondarev 氏の Langchain.rb gem を利用する

Ruby

RBS Goose Configuration Example

RBS Goose の設定例

api_key = ENV.fetch('OPENAI_ACCESS_TOKEN')

RbsGoose.configure do |c|
  # Use the provided configuration methods
  c.use_open_ai(api_key)

  # or directly configure an instance of Langchain::LLM
  c.llm.client = ::Langchain::LLM::OpenAI.new(api_key: )

  # or Local Server such as Ollama
  c.llm.client = ::Langchain::LLM::Ollama.new(
    url: "http://localhost:11434"
  )
end

ref: RbsGooseTest Rakefile Setups

RBS Goose Testing

RBS Goose のテスト

  • LLM API is Expensive and High latency

    • Critically unsuitable for CI.

  • Web mocks such as VCR gem can be used

    • Make it an exact match, including Request Body

    • Temperature should be set to 0 for reproducibility

LLM APIは費用が高く応答も遅い

CIと致命的に相性が悪い

VCR gem などの Webモックを利用すると良い

リクエストボディを含めた厳密一致を指定する

再現性のために、temperatureは0にする

RBS Goose Testing - VCR Setup

RBS Goose のテスト - VCR セットアップ

# spec/spec_helper.rb
VCR.configure do |config|
  config.cassette_library_dir = 
    'spec/fixtures/vcr_cassettes'
  config.hook_into :webmock
  config.default_cassette_options = {
    match_requests_on: %i[method uri body],
    record: ENV.fetch('RECORD', :once).to_sym
  }
  config.filter_sensitive_data('<openai_access_token>') { 
    ENV.fetch('OPENAI_ACCESS_TOKEN')
  }
end

ref: spec/spec_helper.rb

RBS Goose Testing - VCR Usage

RBS Goose のテスト - VCRの利用

# spec/rbs_goose/type_inferrer_spec.rb
RSpec.describe RbsGoose::TypeInferrer, :configure do
  it 'returns refined rbs' do
    VCR.use_cassette('openai/infer') do
      expect(described_class.new.infer).to
        eq(refined_rbs_list)
    end
  end
end

ref: spec/rbs_goose/type_inferrer_spec.rb

Recorded Request Example

記録されたリクエストの例

---
http_interactions:
- request:
    method: post
    uri: https://api.openai.com/v1/chat/completions
    body:
      encoding: UTF-8
      string: 
        '{"messages":[
          {"role":"user","content":"Act as Ruby type inferrer..."}],
           "model":"gpt-3.5-turbo-1106","n":1,
           "temperature":0.0}'
    headers:
      Content-Type:
      - application/json
      Authorization:
      - Bearer <openai_access_token>
      ...
  response:
   ...

ref: spec/fixtures/vcr_cassettes/ollama_codegemma_chat/infer_user_factory.yml

Headline

目次

Purpose of creating RBS Goose

How RBS Goose works

Tips for Development with LLM

Performance Evaluation of RBS Goose

Conclusion

RBS Gooseを作った目的

RBS Gooseの仕組み

LLMを使った開発のTips

RBS Gooseの性能評価

まとめ

Evaluation 1: Config & Runner

評価 1: Config & Runner

class Config
  def self.configure(&block)
    new.tap(&block)
  end

  %w[client role prompt].each do
    attr_accessor _1.to_sym
  end
end
class Runner
  def initialize(config)
    @config = config
  end

  def run
    config.client.chat(
      messages: [{
        role: config.role,
        content: config.prompt
      }]
    ).chat_completion
  end

  private

  attr_reader :config
end

lib/config.rb

lib/runner.rb

kokuyouwind/rbs_goose_test case1/lib

Evaluation 1: Config & Runner

評価 1: Config & Runner

  • Let RBS Goose guess a small example involving metaprogramming

    • The base RBS is generated by each of the three methods

    • Tried OpenAI and Anthropic models + CodeGemma (local LLM)

  • ​steep check + Quality checks by read the RBS

    • Check if there are any untyped left that can be detailed, etc.

メタプログラミングを含む小さな例を推測させた

ベースとなるRBSは、事前に解説した3種類の手法で生成

OpenAIとAnthropicの各モデル + CodeGemma(ローカルLLM)を試した

kokuyouwind/rbs_goose_test case1

steep checkの確認に加えて、目視での品質確認を実施

まだ具体化できるuntypedが残されていないか、などを確認

Result 1: Generated RBS Quality

結果 1: 生成されたRBSの質

Platform Model Size prototype rb base prototype runtime base Typeprof
base
OpenAI GPT-3.5 Turbo Small Perfect Perfect Almost
OpenAI GPT-4 Turbo Large Perfect Perfect Perfect
OpenAI GPT-4 Omni Large Perfect Perfect Perfect
Anthropic Claude 3 Haiku Small Perfect Almost Almost
Anthropic Claude 3 Sonnet Medium Almost Not Good Perfect
Anthropic Claude 3 Opus Large Almost Almost Almost
Ollama(Local) CodeGemma Small Not Good Not Good Not Good

Result 1: Attribute Accessors

結果 1: アトリビュートアクセサ

rbs prototype runtime + OpenAI gpt-3.5-turbo​

Regardless of the base, the output was the same.

ベースのRBSを問わず、同じような出力になった

Result 1: Almost Example

結果 1: Almostの例

rbs prototype rb + Anthropic Claude 3 Sonnet

rbs prototype rb + Anthropic Claude 3 Opus

There was a case of fabricating the return type

of LangChain::LLM::OpenAI#chat

LangChain::LLM::OpenAI#chat の返り値の型を捏造することがあった

Result 1: Interesting Case

結果 1: 興味深いケース

In one case, gpt-4-turbo commented

on why it was left untyped

1例だけ、 gpt-4-turbo が
「なぜuntypedのまま残したか」をコメントしているものがあった

rbs prototype runtime + OpenAI gpt-4-turbo​

Result 1: Execution Time [sec]

結果 1: 実行時間 [秒]

Platform Model Size prototype rb base prototype runtime base Typeprof
base
OpenAI GPT-3.5 Turbo Small 2.2 2.2 4.8
OpenAI GPT-4 Turbo Large 7.6 11.4 7.2
OpenAI GPT-4 Omni Large 1.8 1.7 1.9
Anthropic Claude 3 Haiku Small 3.3 3.3 2.9
Anthropic Claude 3 Sonnet Medium 3.5 8.6 3.0
Anthropic Claude 3 Opus Large 14.6 13.0 13.1
Ollama(Local) CodeGemma Small 7.1 7.4 4.1
Platform Model Size prototype rb base prototype runtime base Typeprof
base
OpenAI GPT-3.5 Turbo Small Perfect (2.2) Perfect (2.2) Almost (4.8)
OpenAI GPT-4 Turbo Large Perfect (7.6) Perfect (11.4) Perfect (7.2)
OpenAI GPT-4 Omni Large Perfect (1.8) Perfect (1.7) Perfect (1.9)
Anthropic Claude 3 Haiku Small Perfect (3.3) Almost (3.3) Almost (2.9)
Anthropic Claude 3 Sonnet Medium Almost (3.5) - Perfect (3.0)
Anthropic Claude 3 Opus Large Almost (14.6) Almost (7.4) Almost (4.1)

Result 1: Time (Perfect or Almost)

結果 1: 実行時間(PerfectかAlmostのもののみ)

Evaluation 1: Consideration

実験 1: 考察

元となるRBS生成手法はどれにしても大差なかった

GPT-4 Omni が最速なのに理想的な出力だった

実行が手軽で速い rbs prototype rb に絞っても良さそう

rbs prototype rb + GPT-4 Omni の組み合わせが良さそう

  • The base is much the same for all methods

    • Looks good to focus on rbs prototype rb

  • ​For the model, the GPT system clearly performed better

    • ​GPT-4 Omni was the fastest but ideal output

  • ​rbs prototype rb + GPT-4 Omni combination looks good

モデルは GPT系の成績が明らかに良かった

Evaluation 2: RbsGoose

評価 2: RbsGoose

  • Infer RBS from Ruby code in whole RbsGoose

    • The base used only rbs prototype rb

RbsGooseのRubyコード全体からRBSを推測する

ベースはrbs prototype rbのみを用いた

# File Count
❯ find lib -type f | wc -l
      17

# Line Count
❯ find lib -type f | xargs cat | wc -l
     698

# Size Count
❯ du -sh lib
 68K	lib
Platform model Model Size Quality time[sec] cost[¢]
OpenAI GPT-3.5 Turbo Small Poor 4.3 0.44
OpenAI GPT-4 Turbo Large Almost 69.2 12.6
OpenAI GPT-4 Omni Large Almost 52.5 7.86
Anthropic Claude 3 Haiku Small Poor 33.4 0.65
Anthropic Claude 3 Sonnet Medium Almost 55.5 7.88
Anthropic Claude 3 Opus Large Almost 90.7 35.72
Ollama(Local) codegemma Small Subtle 95.9 N/A

Result 2: Generated RBS Quality

結果 2: 生成されたRBSの質

Result 2: Almost Summary

結果 2: Almostな出力の概要

Overall, well guessed, including generics.

全体的にはジェネリクスも含めてよく推測されている

class RbsGoose::IO::ExampleGroup < ::Array[RbsGoose::IO::Example]
  self.@default_examples: Hash[Symbol, RbsGoose::IO::ExampleGroup]
  attr_accessor error_messages: String?
  def self.load_from: 
    (String base_path, ?code_dir: String, ?sig_dir: String, ?refined_dir: String) 
      -> RbsGoose::IO::ExampleGroup
  def self.default_examples: () -> Hash[Symbol, RbsGoose::IO::ExampleGroup]
  private def self.load_example:
    (String base_path, String code_dir, String path, String refined_dir, String sig_dir) 
      -> RbsGoose::IO::Example
  private def self.to_rbs_path: (String path, String sig_dir) -> String
  def to_target_group: () -> RbsGoose::IO::TargetGroup
  def to_refined_rbs_list: () -> Array[RbsGoose::IO::File]
end

sig/rbs_goose/io/example_group.rbs

ref: OpenAI GPT-4-omni でのsig:refine結果

Evaluation 2: Failure Point

評価 2: 失敗していたポイント

Failure Description: Syntax Error in Struct or delegator

失敗内容: Struct や def_delegator で Syntax Error

class RbsGoose::Configuration
  LLMConfig: Struct[client: ::Langchain::LLM::Base, ...
  TemplateConfig: Struct[instruction: String, ...
      
  def_delegator llm, :client, :llm_client
  def_delegator llm, :mode, :llm_mode
  ...

sig/rbs_goose/configuration.rbs

Evaluation 2: What happens when I fix it

評価 2: それらを直したらどうなるか

🙃

Evaluation 2: Consideration

実験 2: 考察

Structなどの特殊ケースのRBSをうまく扱えない

rbs_railsやtypeprofなどはトップレベルにRBSを生成するので対応が取れない

exampleに含めるか, Fine Tuningを行う必要がありそう

やっぱり型エラーの自動修正が欲しい

  • Cannot handle RBS for special cases such as Struct well

    • Necessary to include it in the example, or require Fine Tuning

  • The 1:1 assumption of ruby and rbs was not a good

    • ​rbs_rails, typeprof, etc. generate RBS at the top level

  • I still want a fix for type errors

rubyとrbsを一対一の前提にしたのはあまり良くなかった

Headline

目次

Purpose of creating RBS Goose

How RBS Goose works

Tips for Development with LLM

Performance Evaluation of RBS Goose

Conclusion

RBS Gooseを作った目的

RBS Gooseの仕組み

LLMを使った開発のTips

RBS Gooseの性能評価

まとめ

Conclusion

まとめ

  • Introduced a case study of the creation of the RBS Goose

    • Explained how to compose the prompt and the intent

    • Some tips for development with LLM were presented

  • ​RBS Goose is still experimental

    • LLM could be used to do some interesting things

RBS Goose を作った事例を紹介した

プロンプトの構成方法と、その意図について解説した

LLMを使った開発のTipsをいくつか紹介した

LLM を使うと面白いことができるかも、というのが伝わると嬉しい

RBS Goose はまだ実験段階

Concerns

気になっていること

Tomorrow's Session Schedule

明日のセッションスケジュール

rbs-inline

https://github.com/soutaro/rbs-inline

Seems to work well with AI completion

AI補完と相性が良さそう

I tried GitHub Copilot and it completes quite well.

GitHub Copilot を試したら、結構補完してくれそう

Completion

Editing entire projects with AI could work well

like Open Interpreter or Copilot Workspace

Open Interpreter や Copilot Workspace など、

AIでプロジェクト全体を編集する戦略もやりやすくなりそう

Is RBS Goose dead?

RBS Goose 死亡のお知らせ？

🪦

I'm not sure yet whether

the RBS Goose will become a dead duck

or a goose that lays golden eggs.

So I'll keep at it a little longer
before I cooks my own goose.

RBS Gooseが Dead Duckになる (失敗に終わる) のか、

それとも金の卵を生むガチョウになるのかはまだわからないので、

Cook my own goose(自分で成功の機会を捨てる)前に

もう少し続けてみたいと思う。