AddING Rusty-ness

> Swarnim Arun

DeepSource

QR & LINK

> whoami ?

  • Swarnim Arun, a Rustacean,
    Coffee + Games + Light Novel addict (@swarnimarun)
     
  • Currently,
    Software Engineer (Rust & C++) at DeepSource
     
  • Systems, Applications, Language and Games Developer
     
  • Haskell-er and FP enthusiast
     
  • Been using Rust since pre-2018.
    Currently quite interested in Roc and Zig.
     
  • Feel free to ask questions about any of the above.
    All opinions are my own.

> where rust?

We(@DeepSourceHQ) have an analysis engine for Rust.

 

Think `clippy` + `coverage` + lots of other cool things and much easier to integrate into CI workflows.

 

So Rust itself was the obvious choice.

DeepSOURCE   RUST

VS    CLIPPY

~10-30x*** FASTER

doesn't require the code to be built on our servers, so no setup required

works with any build system with zero setup, cargo is good but we like & should have options**

one click analysis

awesome dashboard and much much more...

***

# with about ~1700 lines of Rust code to be linted against

# us
1.96s user 0.11s system 101% cpu 2.043 total

# this is cause you need to build the project for clippy
66.18s user 8.03s system 315% cpu 23.517 total

a serious part of our code involves resolving rust types and its type system things which is largely single-threaded atm.

also we don't cache for rust(atm) because certain scaling constraints

sometimes compute is cheaper than storage

**

Clippy Driver(a way to run clippy without cargo) exists and works without cargo but is a pain to set up if your build system doesn't have in-built support for it.

HOW?

use rust analyzer duh.

> "MORE" rust?

Well yes, indeed we added more Rust,

More parsing and code analysis.
Including significant parts of some of our other analyzers.

HOW MUCH?

3.1x in 10 MONTHS

we will have 100K+ Lines of Code soon(tm)

WHY?

> MORE

> SAFETY

  • Immutable by Default
  • Bound Checking By Default
  • Memory Safety with `borrowck` & Ownership
  • No More Data Races with `Send` + `Sync` types
  • ...

Let's DISCUSS AN EXAMPLE OF,

Safety As THE DEFAULT

GIST GITHUB

WHERE IT SAVED OUR BACON?

fn get_src<'a'>(
    vfs: &'a Vfs,
    file_id: FileId
) -> Result<&'a str, Utf8Error>
{
    std::str::from_utf8(
        virtual_fs.read_contents(file_id)
    ) // guarantees mem safety
}
auto get_src(
    const Vfs& virtual_fs,
    FileId file_id
) -> std::string_view
{
    auto [src, length] =
    	virtual_fs.read_contents(file_id);
    return std::string_view{
        src,
        length
    }; // this constructor can raise exception
    // ----- no guarantee of mem safety -----
    // so if src is dangling after above call
    // it will be a `use-after-free`
}

> BETTER

> PERFORMANCE

> WITH

> "LESS" CODE

BETTER SEMANTICS & MORE PERFORMANCE

GODBOLT: LINK

> AWESOME

  • cargo
  • rust-analyzer
  • rustup
  • ...

> TOOLING

> FRIENDLY

> "ERRORS"

   Compiling test v0.0.1 (/test)
error[E0433]: failed to resolve: use of undeclared type `HashMap`
 --> src/main.rs:2:13
  |
2 |     let _ = HashMap::<u32, String>::new();
  |             ^^^^^^^ not found in this scope
  |
help: consider importing one of these items
  |
1 | use hashbrown::HashMap;
  |
1 | use hashbrown_0_12_3::HashMap;
  |
1 | use nom::lib::std::collections::HashMap;
  |
1 | use std::collections::HashMap;
  |

For more information about this error, try `rustc --explain E0433`.
error: could not compile `test` due to previous error

> SOME THINGS  

> WE DO &
> RECOMMEND

> IMPROVING BUILD TIMES

  • sccache
  • cargo-chef
  • dynamic dispatch, aka smart pointers, `Box<T>`

> Maybe its not

> YOU

Use cargo-bloat with cargo-tree to figure out and remove any dependencies that waste too much time, space or otherwise.

 

> NO-ASYNC

> JUTSU

Sometimes async is essential, but often you can get away with just using threads & pools to dispatch functions as tasks.

Maintaining the simplicity of your application.

CODE WITH ASYNC

// this not nearly enough code to write good async
// this had to be fit into this slide
use futures::*; // this might not be enough of an import but I don't have
// a code slice that will fit here so ask me later about everything else
fn process_vfs_paths(paths: impl Iterator<Item = (FileId, Vfs)>) -> Vec<Info> {
    let pool = ThreadPool::new()
    	.expect("Failed to build thread pool"); // pls don't use `expect` in prod
    let (tx, mut rx) = mpsc::unbounded::<Info>();
    // some shared state
    let shared_state = SharedStateProvider();
	
    pool.spawn_ok(
        stream::iter(paths.zip(repeat(move || tx.clone()))).for_each_concurrent(
            10,
            |((file_id, ref vfs), tx)| async move {
                // TODO: retry? if receiver not dropped!
                let sn = __process(file_id, vfs, &shared_state);
                let Err(e) = tx().unbounded_send(sn) else { return; };
                eprintln!("error sending, {e:?}");
            },
        ),
    );
    let mut ret = Vec::new();
    loop {
        match rx.try_next() {
            Ok(Some(Ok(x))) => ret.push(x),
            Ok(None) => break,
            _ => {}
        }
    }
    ret
}

CODE WITHOUT ASYNC

// this is pseudo code to explain how something
// with shared state would work with rayon
use rayon::prelude::*;

fn process_vfs_paths(paths: impl Iterator<Item = (FileId, Vfs)>) -> Vec<Info> {
    paths
    .par_bridge()
    .map_with(SharedStateProvider(), |ref shared_state, (file_id, ref vfs)| {
        __process(file_id, vfs, shared_state)
    })
    .filter_map(|issues| issues.ok())
    .collect() // this will run our stack in parallel then build Vec
}

THANKS!

Made with Slides.com