REX Migrating from a legacy system

April -> July 27th






The Core Team

Demo



Why revamping conversations?

Specification
Group conversation



Specifications
Conversation unique to a relation



Project A
Project B
Project C



Relation A







Except...


Architecture




Architecture




Migrating

Strategy to preserve retro-compatibility
















Is it retro-compatible?
- Zero downtime
- Easy Rollback
- Verification at each step essential

The model


Translating the requests



The challenge











Mongo request
db.conversations.find({date: {$lte: "12 mars", $gte: "34 avril"}}).as(Conversation.class)

Mongo compliant Postgres
select *
from conversation
left outer join conversations_participants
on conversations_participants.conversation_id = conversation.id
left outer join participant
on participant.identity_id = conversations_participants.participant_id
left outer join message
on message.conversation_id = conversation.id
left outer join message_read_by
on participant.identity_id = message_read_by.participant_id
and message.id = message_read_by.message_id
left outer join temps_attachment
on temps_attachment.message_id = message.id
left outer join stored_file
on stored_file.message_id = message.id
where conversation.id in conversation_ids
order by
array_position(conversation_ids, conversation.id),
message.created_at,
stored_file.index,
temp_attachment.index
Keep the result in order
select conversation.id
from conversation
where conversation.date between "12 avril 2014" and "14 mars 2019"
order by conversation.date
select * from conversation
-- ...
where conversation.id in conversation_ids
order by array_position(conversation_ids, conversation.id)

Golden Master

- Replace with exactly the same behavior
- Understand the intricacies only when necessary
- Secure the migration


Treat the existing as a black box

@Test
void golden_master_simple_test() {
Conversation conversation = givenAConversation();
mongoRepository.save(conversation);
jooqRepository.save(conversation);
Conversation expected = mongoRepository.findById(conversation.getKey());
assertThat(jooqRepository.findById(conversation.getKey())).isEqualTo(expected);
}
Simplest case

Golden Master over conversations
- Sampling a 100 conversations
- 13 writing requests -> 15 tests
- 49 reading requests -> 40 tests







Fixing the data while migrating

- Took us 2 weeks worth of time
- Not guided by data analysis
- Completely blind to date whether it was useful

Migration done!

Concurrency issues
findById
process
change data
save
throw event B
other stuff
findById
change data
save


Event A
Event B



Concurrency issues
findById
process
change data
save
throw event B
other stuff
findById
change data
save


Event A
Event B

throw transactional event B
Solution: TransactionalAwareEventService

Concurrency issues
findById
process
change data
save
other stuff
findById
change data
save


Event A
Event A
some stuff



Concurrency issues
findById
process
change data
save
other stuff
findById
change data
save


Event A
Event A
some stuff

select for update
select for update
Solution: Locking
select conversationId from conversation where conversationId = "1234" for update

Was it worth it?

KPI
- +64% message by month
- +40% message by user
- - 1h30 answer delay



Question?

Thank you <3
REX Migrating from a legacy system
By Thomas Bracher
REX Migrating from a legacy system
- 257