Standard Shadowserver reports cover a 24 hour period (the previous day) of events. Every single day we send out a full day of observed events from the previous day to report subscribers. This means that whenever we detect an event within a network and this event persists, we will continue to report that same event every single day. For some networks, that means a lot of events get sent daily. Additional effort is required by the recipient to be able to detect any daily changes in the reported events, which is often of interest to many report consumers.
A new opt-in feature in our reporting mechanism will allow for reporting only the changes of the data from day to day: the report delta mode option. In this mode, every Sunday we will continue to deliver a full set of reports on all events observed on a report recipient’s network. For the rest of the week, for every distinct report type we will report only the difference between events seen on that day relative to the Sunday report. This will continue throughout the week until the following Sunday, when everything is reset and a full report is delivered again.
Deltas are calculated based on a row hash excluding the timestamp and source port columns. Not all report types support the delta option. For report types that do not support the delta option, a full report will be created.
The new report delta feature is opt-in only. If you would like to enable this mode please request it explicitly. You can do so by using the contact us form. Of course, if you are not interested in using this mode you will continue to receive the full data set every day as always.
If you have not subscribed yet but are about to do so and would like to have access to our reports using the report delta mode, method please mention this when subscribing to our reports.
When using our new Report API and downloading the daily reports, the reports will be the delta version if this option is enabled. Note that the report statistics from the API will also be affected and reflect the data in the daily reports. All the changes are reflected on the actual data being shared daily.
This will also impact the statistics that are sent in the quarterly review report from Shadowserver. The total number of reported events will be less for any period since only unique events are being reported over a week period.
We hope that the new report delta mode option will allow for more options for processing of our data for some of our report consumers.
For any feedback, questions and comments, please contact us.