We've been experiencing an issue that impacts the count of unique viewers in A/B tests.
The issue first occurred on December 3rd at 10am UTC and has been fully resolved on December 6th at 5pm UTC.

20% of our customers were affected and will be contacted individually. No transaction has been lost and none of your app users were impacted.

Incident description

A production outage of one of the component of our platform has led to a partial loss of UI events (generated by the SDK) for some of our customers between the 3rd and the 6th of December.

During the outage, the platform was still functioning and processing transactions but UI events such as PRESENTATION_VIEWED (on which the number of paywalls displayed relies) were lost.

The incident started on Sunday 3rd of December, 10am UTC and was fully fixed on Wednesday 6th 6pm UTC.

The platform is functioning nominally since that time.

Impact

The loss of UI events has the following impact:

Ongoing A/B test

• Number of unique viewed
• Statistical significance
• Conversion rate Viewed to paid
• ARPU

Other A/B tests KPIs related to transaction processing are NOT impacted:

• TRIAL STARTED
• SUBSCRIPTION STARTED
• TRIAL CONVERTED
• 1ST RENEWAL
• REVENUE
• ARPPU

How to read the data

• The bias on the KPIs listed above depends on the moment where the A/B test was started. The closer to the incident dates, the bigger the impact. The bias will fade out in time as the A/B test will go-on.
• All variants were impacted the same way, so the end result should not change. However the statistical significance is likely to be overestimated.

Other minor impacts

• Conversion rates for Placements and Paywalls & screens over the last 30 days are slightly impacted (overestimated)
• Conversion rates of Paywalls and Placements that were activated shortly before or during the outage are likely to be more impacted on the short term but the impact will fade out in time.

What can you do

You have 2 possibilities:

1. do nothing and just be more cautious in the interpretation of the results of your A/B tests
2. if you just started your A/B tests, you can stop them and restart them to have a clean set of data. Note that the Multi-placement A/B tests were just released in production this morning (December 7th): they allow you to run an A/B test on several Placements.

Root cause

A non-critical component of the Purchasely infrastructure stopped working, which saturated the queueing system receiving UI events, and had the consequence to saturate it make it unable to receive new UI events.

Actions taken to avoid this in the future

We took the following actions to reduce the occurrences and the impacts:

1. Added controls and alarms on the components responsible for the issue
2. Added controls and alarms on upstream components to identify risks before it occurs
3. Documented the process to resolve the issue to solve it faster and avoid it before it even occurs