REX Migrating from a legacy system

April -> July 27th

The Core Team

Demo 

Why revamping conversations?

Specification

Group conversation

Specifications

Conversation unique to a relation

Project A

Project B

Project C

Relation A

Except...

Architecture

Architecture

Migrating

Strategy to preserve retro-compatibility

Is it retro-compatible?

  • Zero downtime
  • Easy Rollback
  • Verification at each step essential

The model

Translating the requests

The challenge

Mongo request

db.conversations.find({date: {$lte: "12 mars", $gte: "34 avril"}}).as(Conversation.class)

Mongo compliant Postgres

select *
from conversation
left outer join conversations_participants
    	on conversations_participants.conversation_id = conversation.id
    left outer join participant
    	on participant.identity_id = conversations_participants.participant_id
    left outer join message
    	on message.conversation_id = conversation.id
    left outer join message_read_by
    	on participant.identity_id = message_read_by.participant_id
    	and message.id = message_read_by.message_id
    left outer join temps_attachment
    	on temps_attachment.message_id = message.id
    left outer join stored_file
    	on stored_file.message_id = message.id
    where conversation.id in conversation_ids
    order by
    	array_position(conversation_ids, conversation.id),
        message.created_at,
        stored_file.index,
        temp_attachment.index

Keep the result in order

select conversation.id
from conversation
where conversation.date between "12 avril 2014" and "14 mars 2019"
order by conversation.date

select * from conversation
-- ...
where conversation.id in conversation_ids
order by array_position(conversation_ids, conversation.id)

Golden Master

  • Replace with exactly the same behavior
  • Understand the intricacies only when necessary
  • Secure the migration

Treat the existing as a black box

@Test
void golden_master_simple_test() {
    Conversation conversation = givenAConversation();
    
    mongoRepository.save(conversation);
    jooqRepository.save(conversation);

    Conversation expected = mongoRepository.findById(conversation.getKey());
    assertThat(jooqRepository.findById(conversation.getKey())).isEqualTo(expected);
}

Simplest case

Golden Master over conversations

  • Sampling a 100 conversations
  • 13 writing requests -> 15 tests
  • 49 reading requests -> 40 tests

Fixing the data while migrating

  • Took us 2 weeks worth of time
  • Not guided by data analysis
  • Completely blind to date whether it was useful

Migration done!

Concurrency issues

findById

process

change data

save

throw event B

other stuff

findById

change data

save

Event A

Event B

Concurrency issues

findById

process

change data

save

throw event B

other stuff

findById

change data

save

Event A

Event B

throw transactional event B

Solution: TransactionalAwareEventService

Concurrency issues

findById

process

change data

save

other stuff

findById

change data

save

Event A

Event A

some stuff

Concurrency issues

findById

process

change data

save

other stuff

findById

change data

save

Event A

Event A

some stuff

select for update

 

select for update

 

Solution: Locking

select conversationId from conversation where conversationId = "1234" for update

Was it worth it?

KPI

  • +64% message by month
  • +40% message by user
  • - 1h30 answer delay

Question?

Thank you <3

REX Migrating from a legacy system

By Thomas Bracher

REX Migrating from a legacy system

  • 126