REX Migrating from a legacy system
April -> July 27th
The Core Team
Demo
Why revamping conversations?
Specification
Group conversation
Specifications
Conversation unique to a relation
Project A
Project B
Project C
Relation A
Except...
Architecture
Architecture
Migrating
Strategy to preserve retro-compatibility
Is it retro-compatible?
- Zero downtime
- Easy Rollback
- Verification at each step essential
The model
Translating the requests
The challenge
Mongo request
db.conversations.find({date: {$lte: "12 mars", $gte: "34 avril"}}).as(Conversation.class)
Mongo compliant Postgres
select *
from conversation
left outer join conversations_participants
on conversations_participants.conversation_id = conversation.id
left outer join participant
on participant.identity_id = conversations_participants.participant_id
left outer join message
on message.conversation_id = conversation.id
left outer join message_read_by
on participant.identity_id = message_read_by.participant_id
and message.id = message_read_by.message_id
left outer join temps_attachment
on temps_attachment.message_id = message.id
left outer join stored_file
on stored_file.message_id = message.id
where conversation.id in conversation_ids
order by
array_position(conversation_ids, conversation.id),
message.created_at,
stored_file.index,
temp_attachment.index
Keep the result in order
select conversation.id
from conversation
where conversation.date between "12 avril 2014" and "14 mars 2019"
order by conversation.date
select * from conversation
-- ...
where conversation.id in conversation_ids
order by array_position(conversation_ids, conversation.id)
Golden Master
- Replace with exactly the same behavior
- Understand the intricacies only when necessary
- Secure the migration
Treat the existing as a black box
@Test
void golden_master_simple_test() {
Conversation conversation = givenAConversation();
mongoRepository.save(conversation);
jooqRepository.save(conversation);
Conversation expected = mongoRepository.findById(conversation.getKey());
assertThat(jooqRepository.findById(conversation.getKey())).isEqualTo(expected);
}
Simplest case
Golden Master over conversations
- Sampling a 100 conversations
- 13 writing requests -> 15 tests
- 49 reading requests -> 40 tests
Fixing the data while migrating
- Took us 2 weeks worth of time
- Not guided by data analysis
- Completely blind to date whether it was useful
Migration done!
Concurrency issues
findById
process
change data
save
throw event B
other stuff
findById
change data
save
Event A
Event B
Concurrency issues
findById
process
change data
save
throw event B
other stuff
findById
change data
save
Event A
Event B
throw transactional event B
Solution: TransactionalAwareEventService
Concurrency issues
findById
process
change data
save
other stuff
findById
change data
save
Event A
Event A
some stuff
Concurrency issues
findById
process
change data
save
other stuff
findById
change data
save
Event A
Event A
some stuff
select for update
select for update
Solution: Locking
select conversationId from conversation where conversationId = "1234" for update
Was it worth it?
KPI
- +64% message by month
- +40% message by user
- - 1h30 answer delay
Question?
Thank you <3
REX Migrating from a legacy system
By Thomas Bracher
REX Migrating from a legacy system
- 200