Logs are not enough
...or why traces are worth the setup.
About me
Author of uniforms.
Everything is at radekmie.dev.
Core contributor of Meteor.js.
What are logs?
A log is a timestamped text record, either structured (recommended) or unstructured, with optional metadata. [...]
A log is a timestamped text record,
Most programming languages have built-in logging capabilities or well-known, widely used logging libraries.
A log is a timestamped text record,
How to work with logs?
async function installPayments(c: Config, r: Id) {
logger.info('Install started.');
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info('Install finished.');
}
async function createStripeAccount(c: Config) {
const response = await fetch(...);
if (!response.ok)
throw new Error('StripeError');
const result = await response.json();
return result.id;
}How to work with logs?
async function installPayments(c: Config, r: Id) {
logger.info('Install started.');
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info('Install finished.');
}
async function createStripeAccount(c: Config) {
const response = await fetch(...);
if (!response.ok)
throw new Error('StripeError');
const result = await response.json();
return result.id;
}Given these logs, answer these questions:
- # of installations last week.
- Average duration.
- % of fails.
How to work with logs?
async function installPayments(c: Config, r: Id) {
logger.info('Install started.');
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info('Install finished.');
}
async function createStripeAccount(c: Config) {
const response = await fetch(...);
if (!response.ok)
throw new Error('StripeError');
const result = await response.json();
return result.id;
}Given these logs, answer these questions:
- When was the last attempt from X?
- Did they succeed?
- What was the Stripe error?
We need context!
async function installPayments(c: Config, r: Id) {
logger.info(`Install started for ${r}.`);
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info(`Install finished for ${r}.`);
}
async function createStripeAccount(c: Config) {
const response = await fetch(...);
if (!response.ok)
throw new Error(`StripeError${response.status}`);
const result = await response.json();
return result.id;
}Unstructured!
Actually, lot's of it...
async function checkIfChargeAlreadyExists(...) {
const existingTransactions =
await transactions.search(spaceId, { ... });
if (existingTransactions.body.length > 0) {
logger.warn(
'Duplicate charge transaction detected.',
{ referenceId, restaurantId, spaceId, token },
);
throw Errors.DUPLICATE_CHARGE_DETECTED;
}
}Structured!
CloudWatch Logs Insights
filter @message like /error/
| parse @message /"restaurantId":"(?<id>[^"]+)"/
| stats count(*) as total by message, id
| sort total descLots of operators, including pattern detection.
Can I somehow link logs?
async function installPayments(c: Config, r: Id) {
logger.info(`Install started for ${r}.`);
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info(`Install finished for ${r}.`);
}[2026-06-11 18:25:10] Install started for X.
Which one is which?
[2026-06-11 18:25:11] Install started for X.
[2026-06-11 18:25:19] Install finished for X.
[2026-06-11 18:25:20] Install finished for X.Execution ID
async function installPayments(c: Config, r: Id) {
logger.info(`Install started for ${r}.`);
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info(`Install finished for ${r}.`);
}
Execution ID
async function installPayments(c: Config, r: Id) {
logger.info(`Install started for ${r}.`);
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info(`Install finished for ${r}.`);
}
Execution ID
async function installPayments(c: Config, r: Id) {
const x = getRandomExecutionId();
logger.info(`Install started for ${r}.`);
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info(`Install finished for ${r}.`);
}
Execution ID
async function installPayments(c: Config, r: Id) {
const x = getRandomExecutionId();
logger.info( `Install started for ${r}.`);
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info( `Install finished for ${r}.`);
}
Execution ID
async function installPayments(c: Config, r: Id) {
const x = getRandomExecutionId();
logger.info(`(${x}) Install started for ${r}.`);
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
logger.info(`(${x}) Install finished for ${r}.`);
}
[2026-06-11 18:25:10] (A) Install started for X.
[2026-06-11 18:25:11] (B) Install started for X.
[2026-06-11 18:25:19] (B) Install finished for X.
[2026-06-11 18:25:20] (A) Install finished for X.Execution ID
Every "operation" could generate its own ID and add it to all of its logs.
Nothing is stopping us from passing IDs down to child functions, so they could correlate with parent's logs...
...even if they are executed on a different machine. We just need to send the ID.
Congrats, we just invented tracing!
What is a trace?
Traces give us the big picture of what happens when a request is made to an application. Whether your application is a monolith with a single database or a sophisticated mesh of services, traces are essential to understanding the full “path” a request takes in your application.
Traces give us the big picture of what happens when a request is made to an application. Whether your application is a monolith with a single database or a sophisticated mesh of services, traces are essential to understanding the full “path” a request takes in your application.
Traces give us the big picture of what happens when a request is made to an application. Whether your application is a monolith with a single database or a sophisticated mesh of services, traces are essential to understanding the full “path” a request takes in your application.
How to work with traces?
async function installPayments(c: Config, r: Id) {
await trace('installPayments', { r }, async () => {
if (await database.getById(r))
throw new Error('PaymentsAlreadyInstalled');
const account = await createStripeAccount(c);
await database.insert(r, account);
});
}
function createStripeAccount(c: Config) {
return trace('createStripeAccount', async () => {
const response = await fetch(...);
if (!response.ok)
throw new Error('StripeError');
const result = await response.json();
return result.id;
});
}trace() calls can be nested!
How does it look?
How does it look?
What about the costs?
| Provider | Read | Write | Storage |
|---|---|---|---|
| Dynatrace | $0.0035/GB | $0.20/GB | $0.21/GB |
| Grafana Labs | $0.05/GB | $0.40/GB | $0.10/GB |
| Signoz | (included) | $0.40/GB | (included) |
Self hosting is a viable option, too! Just make sure to account for the time spent on maintaining it.
What else is there!?
- Span kinds, links, and events.
- Automatic metrics generation.
- Service graph.
- Logs correlation.
- TraceQL.
