More Transport, Please!

June 9-12, 2020

iRODS User Group Meeting 2020

Virtual Event

Kory Draughn

Software Developer

iRODS Consortium

More Transport, Please!

The Interface

namespace irods::experimental::io
{
    template <typename CharT, typename Traits>
    class transport
    {
    public:
    	virtual ~transport();
        
        virtual auto open(path, openmode) -> bool;
        
        virtual auto open(path, replica_number, openmode) -> bool;
        
        virtual auto open(path, resource_name, openmode) -> bool;
        
        virtual auto close() -> bool;
        
        virtual auto send(buffer, buffer_size) -> streamsize;
        
        virtual auto receive(buffer, buffer_size) -> streamsize;
        
        virtual auto seekpos(offset, seekdir) -> pos_type;
        
        virtual auto is_open() -> bool;
        
        virtual auto file_descriptor() -> int;
    };
}

Why is it interesting?

It enables wizardly things!

 

For example:

  • Compression
  • Encryption
  • Statistics / Diagnostics
  • New Protocol Support (e.g. RDMA)

 

And more ...

Example: Compression w/ Snappy!

#include <snappy-c.h>
#include <vector>

// ... Boilerplate ...

auto send(char_type* buffer, streamsize buffer_size) -> streamsize override
{
    auto output_length = snappy_max_compressed_length(_buffer_size);
    std::vector<char> output(output_length);
    snappy_compress(_buffer, _buffer_size, output.data(), &output_length);
    return tp_->send(output.data(), output_length);
}

auto receive(char_type* buffer, streamsize buffer_size) -> streamsize override
{
    const auto bytes_read = tp_->receive(_buffer, _buffer_size);

    // Uncompress the buffer if the buffer can be uncompressed.
    if (snappy_validate_compressed_buffer(_buffer, bytes_read) == SNAPPY_OK) {
        std::size_t output_length;
        snappy_uncompressed_length(_buffer, bytes_read, &output_length);

        std::vector<char_type> compressed(bytes_read);
        std::copy(_buffer, _buffer + bytes_read, std::begin(compressed));
        snappy_uncompress(compressed.data(), compressed.size(), _buffer, &output_length);

        return output_length;
    }

    return bytes_read;
}

// ... Boilerplate ...

From snappy_transport.hpp

Example: Compression w/ Snappy! (cont.)

#include <irods/dstream.hpp>
#include <irods/transport/default_transport.hpp>
#include <irods/transport/snappy_transport.hpp>

int main()
{
    namespace io = irods::experimental::io;
    
    auto large_buffer = read_a_whole_lot_of_data();
    
    // No compression here.
    io::client::default_transport dtp{conn};
    
    if (io::odstream out{dtp, "/tempZone/home/rods/foo.txt"}; out) {
    	out.write(large_buffer.data(), large_buffer.size());
    }
    
    // Want compression? Just decorate the original transport with
    // another transport that provides compression!
    io::client::snappy_transport stp{conn, dtp};
    
    if (io::odstream out{stp, "/tempZone/home/rods/foo.txt"}; out) {
    	out.write(large_buffer.data(), large_buffer.size());
    }
}

Example: Compression w/ Snappy! (cont.)

Using tc to emulate a delay of 200ms RTT.

 

  • File Size      = 72971843 bytes (73MB)
  • Buffer Size = 4000000 bytes (4MB)
  • Single Stream Object
  • Custom iRODS server supporting compression and decompression

 

  • Write Speeds roughly 40% faster
  • Read Speeds roughly 36% faster

 

Results vary depending on the network, file size, and buffer size.

UGM 2020 - More Transport, Please

By korydraughn

UGM 2020 - More Transport, Please

iRODS User Group Meeting 2020 - Transports

  • 1,058