This incident is now resolved. The recovery of all wrongly deleted mirrored entities has been completed.
Where we have found some minor abnormalities our Skai Customer Success team will reach out to share further information.
Posted Sep 02, 2021 - 20:18 UTC
Update
We have completed the recovery of all mirrored entities that were deleted, and are now QAing and validating that everything is back to normal and in the correct status.
We will soon close this incident once all validations are completed fully.
Posted Aug 30, 2021 - 14:48 UTC
Monitoring
The recovery process has been completed. All mirrored entities that were deleted by this issue have now been recreated.
We are still monitoring to verify the results and will keep an eye for any unusual behavior.
Posted Aug 25, 2021 - 12:37 UTC
Update
Our engineers are working to complete the recovery process of mirrored entities that were deleted. The process is still in progress, so you may still see deleted entities at this time.
Please note - campaigns that were mirrored using the relatively new “Mirroring powered by Microsoft” feature were not affected by this incident, and their entities were not deleted.
The campaigns that were affected by this issue are ones that were mirrored using our legacy campaign mirroring tool.
Posted Aug 24, 2021 - 22:26 UTC
Update
The final recovery process is now running. This recovery process is for any mirrored entities that were deleted in Skai, and then also uploaded to Microsoft/Google/Pinterest as deleted.
Please note - this did not impact all campaigns mirrored from Google, only some ad groups, keywords, ads, and product groups. The deleted entities will be recreated by the mirroring process as if they are just mirrored today for the first time.
We expect a full resolution of this incident today (Aug 24). We will continue to update on progress.
Posted Aug 24, 2021 - 13:21 UTC
Update
We have completed the first and most important step of this recovery process - all non-mirrored entities that were wrongly set to “Deleted” status in Skai are now back to their original status. If you find such entities that might still be incorrect, please let your Skai Customer Success team know.
As for entities in campaigns mirrored from Google (to Microsoft or to other Google campaigns) that were also deleted on the publisher accounts, our engineers are still working on reverting those changes.
In some cases, where ‘Un-Deleting’ is not allowed by the publisher, we will recreate entities identical to the ones that were deleted. This process is now being tested and is expected to run early on Aug 24th. If you wish to be excluded from this process (and prefer to manually recreate or adjust those mirrored entities), please let your Skai Customer Success team know as soon as possible.
Posted Aug 23, 2021 - 20:04 UTC
Update
Our engineers are still working on a full resolution for this issue - the vast majority of entities (~95%) that were wrongly set to Deleted status in Skai are now set back to their correct status. Some additional entities are in progress and are expected to be reverted during the day.
As a result of this issue, some mirrored entities were deleted as well. Our engineers are working to revert these deletions where possible or recreate the deleted entities if they can’t be reactivated. We will continue to update you on progress.
Posted Aug 23, 2021 - 12:52 UTC
Update
The recovery process is currently in progress.
Posted Aug 23, 2021 - 00:07 UTC
Identified
We have identified an issue with our Nightly Sync process between Skai and Google which has resulted in some Google entities appear as deleted in Skai when they should not be.
This has started at different points in time for different clients, and some remain unaffected. For those who are, the change has happened between Thursday (Aug 19) and Sunday (Aug 22).
The issue itself is already fixed - no additional entities in Skai will turn to Deleted status.
As for the entities (such as keywords, ads, ad groups, etc.) that have been affected, we are working on identifying and fixing them. Clients who have been severely impacted may be contacted by their Skai team with a more detailed analysis and possible actions.
We have paused some sync operations between Google/Microsoft and Skai while we work on a resolution, and will resume those when all has been fixed (or any remaining issues will not be exasperated by the sync).