Unexpected Increase in Exchange Incremental Data Processed By Veeam Backup for Microsoft 365
Issue Summary
First Occurrence — August 7 → August 21, 2023
Some customers using Veeam Backup for Microsoft 365 to back up Microsoft 365 Exchange data may have experienced an unexpected increase in the volume of data being backed up between August 7 and August 21, 2023. Microsoft confirmedMicrosoft has confirmed this issue in the message EX6700041, which will be displayed in the Microsoft 365 Admin Center of some organizations with affected mailboxes. at that time that this was due to a limited incident, which caused older emails to resurface unexpectedly in some Microsoft Exchange Online environments, resulting in increased email data in some Exchange Online mailboxes.
Second Occurrence — March 8 → April 23, 2024
Beginning on March 8th, 2024, Veeam Support started to receive reports from customers about an unexpected increase in the amount of Exchange data being backed up. Based on the experience with the first occurrence of this issue, Veeam's development team promptly engaged with Microsoft Support to begin investigating the root cause. On April 23rd, Microsoft provided the statement below to address this instance.
Third Occurrence — May 28 → June 12, 2024
Around May 28th, Veeam Support received reports from customers observing an unexpected increase in the amount of Exchange data being backed up. Veeam's development team engaged with Microsoft Support, and we believe the core issue has been resolved and that no further impact should occur. Microsoft has released a statement regarding this occurrence.
Fourth Occurrence — Late August 2024 → (ongoing)
Around Aug 27th, Veeam Support received initial reports from customers observing an unexpected increase in the amount of Exchange data being backed up. Veeam's development team is engaged with Microsoft Support to learn if this issue is the same as the previous occurrences and is working towards a mitigation strategy.
This article will be updated when further information becomes available. As with every occurrence, the Advised Actions below remain the same; the jobs must be allowed to complete.
Backup Impact
Emails that were changed, causing them to resurface unexpectedly, must be captured by Veeam Backup for Microsoft 365 to ensure backup continuity. Even though these emails may have been previously backed up, their modified state means they must be backed up again. There is no way to reclaim the space utilized by the backup of these resurfaced emails, as it is technically changed data.
This situation is discussed in further detail in Veeam's RND forums.
Statements from Microsoft
Microsoft provided the following statements to Veeam for us to share with our mutual customers.
Microsoft statement regarding March 2024 incidence:
| Due to changes we [Microsoft] are making to optimize search and compatibility functionality within our service, this action led to re-indexing for some of our customers' data. The re-indexing is not indicative of a change to fields material to our BCDR and storage partners, so we will be working on a change to minimize the re-indexing that is prompting another backup of customer data. We made changes over the weekend [of 2024-04-20] that should pause the unexpected behavior and those changes will be in place until Microsoft and our partners are able to make longer term enhancements to avoid future issues from taking place. Thank you for your patience. |
Microsoft provided the following statement regarding the June 2024 incidence:
| Microsoft routinely performs system maintenance on items within the storage system. A recent system maintenance task on items with attachments unexpectedly created EWS SyncFolderItems events to be triggered. The task finished processing about 70% of environments worldwide before it was postponed. The necessary fixes were rolled out to help prevent unexpected sync events for the remaining 30% of environments that need to be processed. |
Advised Actions
- Check the reports and monitoring activity around the dates indicated in the Issue Summary to determine if your setup is affected by this issue.
- Any impacted Veeam Backup for Microsoft 365 jobs must be allowed to be completed.
- If you have opened a Microsoft Support Case on behalf of your contract and would like more Veeam details regarding the issue to help address it with Microsoft, or should you have any questions, please don’t hesitate to contact Veeam Customer Support.
More Information
Veeam Backup for Microsoft 365 uses officially supported methods to back up Exchange Online data, specifically using EWS API's SyncFolderItems to identify changed items. As such, Veeam Backup for Microsoft 365 is affected by these internal changes made by Microsoft's maintenance.
The problem described in this article relates to internal changes on the Microsoft side that triggered:
- An item to be resynced via the EWS SyncFolderItems API, which is the standard way to perform backup operations.
- An item to have a new LastModifiedDateTime, as this can change for any Exchange item that has its internal properties modified.
Note: Veeam Backup for Microsoft 365 does not use the LastModifiedTime or ChangeKey as a primary indicator of change but can use them as a secondary indicator to avoid duplicate items with the same values when storing data for backup purposes.
Microsoft has said they have made the necessary fixes to prevent unexpected sync events going forward.
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.