Affected product versions

• Libraesva EA Backend from version 21.3.4020
• Libraesva EA Backend from version 21.8.4619

The current backend version can be found in Settings > General > Backend

Brief description

A bug has been discovered in the volume cleanup service which performs the hard deletion of emails that have been soft-deleted manually by the administrator or automatically by configured retention rules.

Additional conditions

In the mentioned product versions the bug arises when emails are deleted either manually by the administrator or by retention rules. The bug does not arise if no email is deleted.

If you don’t have configured any retention rule and did not perform any manual deletion you are not affected by this issue.

Consequences

Email bodies and attachments may have been accidentally deleted. Such emails are still indexed and can be searched for, but the email body is not available.

This is limited to emails stored in proximity of emails that have been deleted manually by the administrator or automatically by configured retention rules.

Detailed description

Email deletion is performed through a soft-delete followed by an asynchronous hard-delete.

When the administrator or a retention rule deletes an email, the email is flagged as deleted making it invisible to the searches but the email metadata is still available in the archiver and the email content is still available in the volumes. 

At a later time a scheduled cleanup process performs the hard deletion by removing the email content from the archive volumes and the email metadata from the archiver.

A bug in this cleanup process caused some emails, which were not been soft-deleted, to be deleted from the volumes. Metadata about these emails is not deleted from the archiver.

The result is that such emails can be searched for in the archiver but the content of the email is not available.

For such email the metadata which includes sender, recipients, subject, date and message-id is still available while the email body and attachments are not.

Mitigation

Libraesva released an “emergency update” to stop the cleanup process on all affected product versions until a fix of the software bug will be developed and released.

The “emergency update” is a mechanism through which Libraesva can release a software update to be installed immediately on the appliances. The process has been designed to handle unexpected and urgent issues like this.

The emergency update brings the backend version to 21.3.4945 or 21.8.4833. Starting from these versions the problem is not present.

Assessment

The Libraesva support service can assess whether the customer is affected by this problem or not.

The assessment can be done through the “remote support” feature of the product.

Should the customer be affected, the Libraesva support will perform an in-depth assessment and will provide the following information:

• how many emails have been deleted (phase 1)
• how many have been recovered by the mail service connector (phase 2)
• the list of email that eventually have not been recovered (phase 3)

Customers can contact the Libraesva support through the usual channels. The issue will be managed with high priority.

Remediation

The engineering team disabled the cleanup service through an “emergency update”, automatically installed on all appliances, thus no more email loss can happen. 

The cleanup service will be re-enabled by the engineering team when the issue will have been fixed.

Timeline

• Engineering: Investigation started on the 2nd of October at 9:15 CET
• Engineering: Bug discovered on the 2nd of October 2021 at 10:58 CET
• Engineering: Mitigation action through an “emergency update” released on the 2nd of October at 11:18 CET
• Technical Support: Initial assessment on all appliances hosted in Libraesva’s cloud completed on the 2nd of October at 16:30 CET (phase 1)
• Technical Support: In-depth assessment on all appliances hosted in Libraesva’s cloud completed on the 4th of October at 18:13 (phase 2)
• Engineering: Released the new cleanup process through standard automatic update on the 30th of November  at 06:00
• Engineering: Recovery completed confirmation from the last customer affected by this issue, Jan 27 2022

Incident closed