Last Updated: 2021-11-30
Reports API
An API to query the different reports received as well as to do basic queries of the data itself. This is meant as an optional replacement to the emails received with the report URL’s. In all cases the queries and the data that is delivered is only from the reports that you would have normally received. You only get the data on the networks you are responsible for. You will not be able to get data on other networks or systems. Note refer to the API: Documentation pages for testing details and examples.
Modules
- reports/subscribed – List of reports that the user is subscribed to
- reports/types – List of all the types of reports that are available for the subscriber
- reports/list – List of actual reports that could be downloaded
- reports/download – Download specific report
- reports/query – Query the stored data
- reports/stats – Output stats
- reports/device-info – Obtain device details for a given IP
- reports/schema – Obtain JSON schema for a given report type
Reports methods
reports/subscribed
Note that most organizations will only have a single list they are subscribed to and can get data on.
Fields:
apikey : string : Your API key
Response:
array of lists you are subscribed to
Example:
freed0@pit:~$ ./call-api.py reports/subscribed {} pretty
[
"united-states",
"california",
]
reports/types
Fields:
apikey : string : your api key
date : string : date range matching query interface
detail : boolean : include details (default false)
Response:
List of available types of reports.
Example 1:
freed0@pit:~$ ./call-api.py reports/types {} pretty
[
"blocklist",
"device_id",
"device_id6",
"event4_honeypot_brute_force",
"event4_honeypot_ddos_amp",
"event4_honeypot_http_scan",
"event4_microsoft_sinkhole_http",
"event4_sinkhole",
"event4_sinkhole_http",
"scan6_telnet"
]
Example 2:
freed0@pit:~$ ./call-api.py reports/types '{ "detail": true }' pretty
[
{
"url" : "https://www.shadowserver.org/what-we-do/network-reporting/accessible-ftp-report/",
"type" : "scan_ftp",
"description" : "Accessible FTP Service Results"
},
{
"url" : "https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-report/",
"type" : "scan_http_vulnerable",
"description" : "Vulnerable HTTP Results"
},
{
"url" : "https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-isakmp-report/",
"type" : "scan_isakmp",
"description" : "ISAKMP Vulnerability Scan Results"
}
]
reports/list
Fields:
apikey : string : Your API key
reports : list : Report types to return (optional) - note that this is a list and must be bracketed ** [ ] **
date : string : Date (YYYY-MM-DD) or range (YYYY-MM-DD:YYYY-MM-DD) (optional)
type : string : Specific report to get download ID for
limit : number : Limit the query to a specific number of records
Response:
array of up to 1,000 matching records
Example request data:
freed0@pit:~$ ./call-api.py reports/list '{"reports":["united-states"], "limit":3}' pretty
[
{
"report": "united-states@shadowserver.org",
"timestamp": "2020-09-28",
"file": "2020-09-28-blocklist-united_states_of_america-geo.csv",
"id": "uN6n7yZK90sdflkjdlLKTOkspksg?HjgX1lI_hAdsKGmVanG_Og"
},
{
"id": "ZjM0guFABt8KLHohis67lk23vjosu09lns934I5uqbPbKW6AHg",
"file": "2020-09-28-botnet_drone-united_states_of_america-geo.csv",
"timestamp": "2020-09-28",
"report": "united-states@shadowserver.org"
},
{
"file": "2020-09-28-caida_ip_spoofer-united_states_of_america-geo.csv",
"id": "3kkOMxJknsfe098yuweklnsvdoi0-9uwljIHOFR3ebOyfOYovjQ",
"timestamp": "2020-09-28",
"report": "united-states@shadowserver.org"
}
]
An example specifying the report type:
freed0@pit:~$ ./call-api.py reports/list '{"reports":["united-states", "california"], "type":"hp_ics_scan", "date":"2020-10-27"}' pretty
[
{
"timestamp": "2020-10-27",
"report": "united-states@shadowserver.org",
"file": "2020-10-27-hp_ics_scan-united_states_of_america-geo.csv",
"id": "60EL63IKLH90uLKJ3xcmXW93403mnkdsfiuIje-0nmdsseTXNP5EgDog"
},
{
"report": "california@shadowserver.org",
"timestamp": "2020-10-27",
"id": "3lG4u-cHIOY9083oihdkUGIkhdsjuj38sdlkn0u92345AHyO9g5H5PK7A",
"file": "2020-10-27-hp_ics_scan-california-region.csv"
}
]
reports/download
Fields:
apikey : string : Your API key
id : string : Report id
report : string : Specific report name
limit : number : Limit the number of records that are returned
Response:
The requested data is handed back in JSON format.
Example request data:
freed0@pit:~$ ./call-api.py reports/download '{"id":"uN6n7yZK90sdflkjdlLKTOkspksg?HjgX1lI_hAdsKGmVanG_Og", "limit":3}' pretty
[
{
"sic" : "",
"source" : "Alien Vault",
"geo" : "US",
"asn" : "20115",
"region" : "SOUTH CAROLINA",
"sector" : "Information Technology",
"reason" : "Malicious Host US",
"city" : "DUNCAN",
"ip" : "35.131.161.166",
"hostname" : "",
"timestamp" : "2020-10-26 07:00:12",
"naics" : "517311"
},
{
"ip" : "35.188.187.206",
"timestamp" : "2020-10-26 07:00:12",
"hostname" : "",
"naics" : "519130",
"sic" : "",
"geo" : "US",
"asn" : "15169",
"source" : "Alien Vault",
"sector" : "Information Technology",
"region" : "IOWA",
"city" : "COUNCIL BLUFFS",
"reason" : "Malicious Host US"
},
{
"city" : "THE DALLES",
"reason" : "Malicious Host US",
"region" : "OREGON",
"sector" : "Information Technology",
"source" : "Alien Vault",
"geo" : "US",
"asn" : "15169",
"sic" : "",
"naics" : "519130",
"hostname" : "",
"timestamp" : "2020-10-26 07:00:12",
"ip" : "35.233.182.140"
}
]
reports/query
Fields:
apikey : string : Your API key
query : dictionary : Search fields [any of the available fields from the reports].
sort : string : Ascending | descending
date : string : Date (YYYY-MM-DD) or range (YYYY-MM-DD:YYYY-MM-DD) now may be used in place of a date
facet : string : Returns the cardinality of each value of the given field sorted from highest to lowest
limit : number : Specify the number of records to pull
page : number : Default is 1; used to obtain additional pages of results
The query must contain one attribute that matches your organization’s report filters.
Examples:
- The Swiss CERT (GovCERT.ch) receives reports for all events for the country code ‘CH’. In order for their query to be valid it must contain a matching attribute of
"geo": "CH"
. - Cisco Systems receives reports filtered by ASN (109) and by domain name (com.cisco.). In order for their query to be valid it must contain
"asn": 109
and/or"domain": "*.cisco.com"
. - Universität Konstanz receives reports filtered by IP (134.34.0.0/16). In order for their query to be valid it must contain an IP that matches their CIDR block
"ip": "134.34.0.1"
or the CIDR block"ip": "134.34.0.0/16"
.
Help:
freed0@pit:~$ ./call-api.py reports/query '{"help":true}' pretty
[
"agent",
"application",
"asn",
"asn_name",
"banner",
"city",
"county_fips",
"county_name",
"device_model",
"device_sector",
"device_type",
"device_vendor",
"device_version",
"domain",
"dst_asn",
"dst_asn_name",
"dst_city",
"dst_county_fips",
"dst_county_name",
"dst_geo",
"dst_ip",
"dst_isp_name",
"dst_latitude",
"dst_longitude",
"dst_naics",
"dst_port",
"dst_region",
"dst_sector",
"family",
"geo",
"infection",
"ip",
"isp_name",
"latitude",
"longitude",
"md5",
"naics",
"port",
"protocol",
"referer",
"region",
"registrar",
"sector",
"sha1",
"sha256",
"sha512",
"sid",
"source",
"source_url",
"tag",
"text",
"timestamp",
"tld",
"type",
"version"
]
Response:
array of up to 1,000 matching records
Example SSL Query by port
:
freed0@pit:~$ ./call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "port":443}, "limit":1}' pretty
[
{
"port": "443",
"tag": [
"ssl"
],
"region": "VIRGINIA",
"protocol": "tcp",
"isp_name": "MICROSOFT CORP",
"county_fips": "51107",
"naics": "334111",
"infection": "ssl",
"type": "scan",
"asn": "8075",
"asn_name": "MICROSOFT-CORP-MSN-A",
"latitude": "39.03",
"county_name": "LOUDOUN",
"ip": "13.82.60.109",
"rip": "109.60.82.13",
"timestamp": "2020-10-27 02:59:32",
"sid": "4096627",
"geo": "US",
"city": "ASHBURN",
"longitude": "-77.49",
"sector": "Critical Manufacturing"
}
]
Example SSL query by tag
:
freed0@pit:~$ ./call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "tag":"ssl"}, "limit":1}' pretty
[
{
"region": "VIRGINIA",
"protocol": "tcp",
"tag": [
"ssl"
],
"port": "443",
"county_fips": "51107",
"naics": "334111",
"isp_name": "MICROSOFT CORP",
"infection": "ssl",
"type": "scan",
"asn_name": "MICROSOFT-CORP-MSN-A",
"latitude": "39.03",
"asn": "8075",
"county_name": "LOUDOUN",
"rip": "109.60.82.13",
"timestamp": "2020-10-27 02:59:32",
"sid": "4096627",
"ip": "13.82.60.109",
"sector": "Critical Manufacturing",
"geo": "US",
"longitude": "-77.49",
"city": "ASHBURN"
}
]
Example using IP address:
freed0@pit:~$ ./call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "ip":"13.82.60.109"}, "limit":1}' pretty
[
{
"geo": "US",
"longitude": "-77.49",
"city": "ASHBURN",
"sector": "Critical Manufacturing",
"ip": "13.82.60.109",
"timestamp": "2020-10-27 02:59:32",
"rip": "109.60.82.13",
"sid": "4096627",
"county_name": "LOUDOUN",
"asn": "8075",
"latitude": "39.03",
"asn_name": "MICROSOFT-CORP-MSN-A",
"type": "scan",
"infection": "ssl",
"isp_name": "MICROSOFT CORP",
"county_fips": "51107",
"naics": "334111",
"tag": [
"ssl"
],
"port": "443",
"region": "VIRGINIA",
"protocol": "tcp"
}
]
Example by City:
freed0@pit:~$ ./call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "city":"ashburn"}, "limit":1}' pretty
[
{
"geo": "US",
"rip": "193.74.6.52",
"tag": [
"http"
],
"longitude": "-77.49",
"infection": "http",
"latitude": "39.03",
"isp_name": "AMAZON TECHNOLOGIES INC.",
"county_fips": "51107",
"county_name": "LOUDOUN",
"protocol": "tcp",
"asn": "14618",
"port": "8080",
"sid": "4096627",
"asn_name": "AMAZON-AES",
"tld": "com",
"city": "ASHBURN",
"type": "scan",
"ip": "52.6.74.193",
"region": "VIRGINIA",
"sector": "Communications",
"naics": "454110",
"timestamp": "2020-10-27 00:02:21",
"domain": "crmapp005.anterasoftware.com"
}
]
Example requiring a tag
:
freed0@pit:~$ ./call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "require":"tag"}, "limit":1}' pretty
[
{
"rip": "229.156.120.34",
"tag": [
"http"
],
"geo": "US",
"county_fips": "29047",
"isp_name": "GOOGLE INC.",
"county_name": "CLAY",
"latitude": "39.22",
"infection": "http",
"longitude": "-94.57",
"asn_name": "GOOGLE",
"sid": "4096627",
"port": "8080",
"asn": "15169",
"protocol": "tcp",
"city": "KANSAS CITY",
"tld": "com",
"region": "MISSOURI",
"sector": "Defense Industrial Base",
"ip": "34.120.156.229",
"type": "scan",
"naics": "519130",
"timestamp": "2020-10-27 00:02:19",
"domain": "229.156.120.34.bc.googleusercontent.com"
}
]
Example of getting counts of different tags
:
freed0@pit:~$ ./call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "require":"tag"}, "facet":"tag", "limit":9}' pretty
[
{
"count": 15203935,
"tag": "ssl"
},
{
"tag": "cwmp",
"count": 11129661
},
{
"tag": "ftp",
"count": 2781725
},
{
"tag": "http",
"count": 2672067
},
{
"tag": "ssl-poodle",
"count": 1866075
},
{
"count": 1519186,
"tag": "isakmp"
},
{
"tag": "quic",
"count": 1212766
},
{
"tag": "rdp",
"count": 851989
},
{
"count": 571483,
"tag": "portmapper"
}
]
Example getting tag
counts over time. Note that since most of the data is repeated daily a count like this has little value:
freed0@pit:~$ ./call-api.py reports/query '{"date":"2020-10-01:2020-10-28", "query":{"geo":"us", "require":"tag"}, "facet":"tag", "limit":9}' pretty
[
{
"tag": "ssl",
"count": 322512588
},
{
"count": 238483485,
"tag": "cwmp"
},
{
"tag": "ftp",
"count": 57667266
},
{
"count": 57308759,
"tag": "http"
},
{
"count": 39666402,
"tag": "ssl-poodle"
},
{
"tag": "isakmp",
"count": 32107711
},
{
"count": 30016842,
"tag": "quic"
},
{
"count": 16919098,
"tag": "rdp"
},
{
"tag": "portmapper",
"count": 12755184
}
]
reports/stats
An API option to allow looking through the history of the statistics of the different reports. Note that if the consumer is subscribed to more than one filter or report set, you will need to specify a filter using the report
filter for that report set of statistics.
Fields:
apikey : string : your api key
Optional Fields:
date : string : date or date range; default is the previous date
report : string or list : filter by report name
type : string or list : filter by report type
Help
freed0@pit:~$ ./call-api.py reports/stats '{"help":true}' pretty
[
"date",
"report",
"type"
]
Returns
The date, type, and number of events for each report type as a two dimensional array.
Example:
freed0@pit:~$ ./call-api.py reports/stats '{"report":"united-states","type":"scan-exchange", "date":"2021-04-24:2021-04-26"}' pretty
[
[
"date",
"type",
"events"
],
[
"2021-04-24",
"scan-exchange",
2055
],
[
"2021-04-25",
"scan-exchange",
2034
],
[
"2021-04-26",
"scan-exchange",
2030
]
]
Different example of date filtering:
freed0@pit:~$ ./call-api.py reports/stats '{"report":"united-states","type":"scan-exchange", "date":"-3:now"}' pretty
[
[
"date",
"type",
"events"
],
[
"2021-04-24",
"scan-exchange",
2055
],
[
"2021-04-25",
"scan-exchange",
2034
],
[
"2021-04-26",
"scan-exchange",
2030
]
]
reports/device-info
Provides device details for a given IP.
Fields:
- apikey : string : your api key
- query : dictionary : search fields [ ip, asn, geo ]
The query must contain one attribute that matches your organization’s report filters.
For example the CERT-LT who has a report filter for the country code ‘LT’ might request:
"query": { "ip": "92.62.137.155", "geo", "LT" }
Response:
{
"device_type" : "router",
"device_sector" : "consumer",
"device_vendor" : "TP-Link"
}
reports/schema
Provides JSON schema for a given report type.
Fields:
apikey : string : your api key
type : string : report type
Example:
freed0@pit:~$ python3 call-api.py reports/schema '{"type":"scan-snmp"}' pretty
{
"protocol" : "string",
"device_version" : "string",
"naics" : "string",
"ip" : "string",
"hostname" : "string",
"asn" : "string",
"sic" : "string",
"city" : "string",
"device_type" : "string",
"sysname" : "string",
"timestamp" : "string",
"device_model" : "string",
"geo" : "string",
"version" : "string",
"sector" : "string",
"region" : "string",
"port" : "string",
"device_sector" : "string",
"sysdesc" : "string",
"device_vendor" : "string"
}
CHANGELOG
- 2021-11-30 – Added reports/schema and changes to reports/types
- 2021-07-28 – Removed alternative example from reports/download since that feature was removed once reports/query was added.
- 2021-04-25 – Added stats method.
- 2021-03-11 – Updated documents for clarity as well as adding the help were applicable. reports/query now requires a tighter search pattern to return results. Queries no longer need to include a specify a report but instead must include an applicable filter.
Old Example:
freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-27", "query":{"port":443}, "limit":1}' pretty
New Example:
freed0@pit:~$ ./call-api.py reports/query '{"geo":"us", "date":"2020-10-27", "query":{"port":443}, "limit":1}' pretty