Article 0460

How to Compare Deliverability Before and After an ESP Migration

This guide addresses before-and-after deliverability measurement. The practical objective is to make the change observable and reversible: define the source of truth, document the destination behavior, test representative records, and retain evidence that the result matches the intended business process.

What must be controlled

Deliverability comparison after migration is not complete simply because a setting was copied or an import reported success. The important question is whether the downstream user, system, or recipient experiences the intended result. Identify the record or event that starts the process, the system that owns the data, the rule that changes it, and the evidence that proves the result.

Before making changes, record the current behavior and the intended behavior side by side. Include the affected audience, relevant fields, timing, exceptions, and dependencies. Preserve a dated source export or configuration snapshot where appropriate. Limit access to personal data and secrets, and do not place API credentials or private subscriber information in public files or screenshots.

Implementation procedure

1. Compare deliverability before and after migration using consistent definitions, audiences, time windows, and mailbox-provider segments. Distinguish accepted, delivered, inbox placement, and engagement; they are not interchangeable.
2. Create a baseline table with send volume, hard bounce, soft bounce, complaint, deferral, and available inbox-placement evidence. Annotate changes in audience, content, and sending volume.

Worked example

A provider’s “delivered” metric generally means accepted by the receiving server, not guaranteed placement in the inbox.

For a controlled test, use synthetic or internal test records wherever possible. Give each test a unique identifier, write down the starting conditions, and capture the observed result. This makes it possible to distinguish a real platform defect from a test record that had unexpected history or stale data.

Verification and acceptance criteria

Accept the change only when the expected result can be reproduced. At minimum, check one normal case, one boundary or missing-data case, one duplicate or retry case where relevant, and one case that must be excluded. Compare the source and destination values rather than relying only on a dashboard status message.

• Correctness: the destination value, message, route, or event matches the approved rule.
• Completeness: required records and dependencies are accounted for, including rejected or unresolved items.
• Safety: suppression, access restrictions, privacy controls, and rollback arrangements remain effective.
• Traceability: a reviewer can find the test input, timestamp, observed output, exception, and approval.

If the test fails

Pause the affected rollout or workflow before making repeated edits. Preserve the error message, request or event identifier, relevant timestamp, sanitized input, and current configuration. Classify the issue as a data-mapping problem, permission or authentication failure, timing issue, provider limitation, or unexpected business rule. Correct one cause at a time and rerun the same test case. If the failure affects consent, unsubscribe handling, security, or live transactional messages, follow the organization’s incident escalation process.

Completion checklist

• Purpose, owner, scope, and source of truth are documented.
• Current and destination behavior have been compared.
• Relevant records, dependencies, and exception cases are accounted for.
• Normal, boundary, and failure tests have recorded results.
• Personal data and credentials are handled with appropriate access controls.
• Unresolved risks have an owner, due date, and explicit decision.
• A reviewer has approved the evidence before production use.

Frequently asked questions

Can this task be completed by copying the old platform settings?

Not safely in every case. Platform features, field types, event timing, and defaults can differ. Translate the intended business rule, then verify it in the destination with test records.

What evidence should be retained?

Keep the approved mapping or configuration, test inputs and outputs, timestamps, error details, reconciliation totals where relevant, and reviewer sign-off. Store only the personal data needed for the purpose and follow the organization’s retention policy.

When should the change go live?

Only after the acceptance criteria pass, high-impact exceptions are resolved, and the accountable owner approves the rollout. If a safe rollback is not available, treat that as an unresolved risk rather than assuming the change is ready.