API: Reports Query

Last Updated: 2020-10-29

Reports API

An API to query the different reports received as well as to do basic queries of the data itself. This is meant as an optional replacement to the emails received with the report URL’s. In all cases the queries and the data that is delivered is only from the reports that you would have normally received. You only get the data on the networks you are responsible for. You will not be able to get data on other networks or systems. Note refer to the API: Documentation pages for testing details and examples.

Modules

  • reports/subscribed – List of reports that the user is subscribed to
  • reports/types – List of all the types of reports that are available for the subscriber
  • reports/list – List of actual reports that could be downloaded
  • reports/download – Download specific report
  • reports/query – Query the stored data

Reports methods

reports/subscribed

Note that most organizations will only have a single list they are subscribed to and can get data on.

Fields:

apikey : string : Your API key

Response:

  • array of lists you are subscribed to

Example:

freed0@pit:~$ ./call-api.py reports/subscribed {} pretty
[
    "united-states",
    "california",
]

reports/types

Fields:

apikey : string : your api key

Response:

  • List of available types of reports.

Example:

freed0@pit:~$ ./call-api.py reports/types {} pretty
[
    "blocklist",
    "botnet_drone",
    "cisco_smart_install",
    "compromised_website",
    "ddos_amplification",
    "drone_brute_force",
    "hp_http_scan",
    "hp_ics_scan",
    "netis_router",
    "scan_adb",
    "scan_afp",
    "scan_ard",
    "scan_chargen",
    "scan_coap",
    "scan_cwmp",
    "scan_db2",
    ...
    "scan_ssl_freak",
    "scan_ssl_poodle",
    "scan_telnet",
    "scan_tftp",
    "scan_ubiquiti",
    "scan_vnc",
    "scan_xdmcp",
    "sinkhole_http_drone",
    "spam_url"
]

reports/list

Fields:

apikey  : string : Your API key
reports : list   : Report types to return (optional) - note that this is a list and must be bracketed ** [ ] **
date    : string : Date (YYYY-MM-DD) or range (YYYY-MM-DD:YYYY-MM-DD) (optional)
type    : string : Specific report to get download ID for
limit   : number : Limit the query to a specific number of records

Response:

  • array of up to 1,000 matching records

Example request data:

freed0@pit:~$ ./call-api.py reports/list '{"reports":["united-states"], "limit":3}' pretty
[
    {
        "report": "united-states@shadowserver.org",
        "timestamp": "2020-09-28",
        "file": "2020-09-28-blocklist-united_states_of_america-geo.csv",
        "id": "uN6n7yZK90sdflkjdlLKTOkspksg?HjgX1lI_hAdsKGmVanG_Og"
    },
    {
        "id": "ZjM0guFABt8KLHohis67lk23vjosu09lns934I5uqbPbKW6AHg",
        "file": "2020-09-28-botnet_drone-united_states_of_america-geo.csv",
        "timestamp": "2020-09-28",
        "report": "united-states@shadowserver.org"
    },
    {
        "file": "2020-09-28-caida_ip_spoofer-united_states_of_america-geo.csv",
        "id": "3kkOMxJknsfe098yuweklnsvdoi0-9uwljIHOFR3ebOyfOYovjQ",
        "timestamp": "2020-09-28",
        "report": "united-states@shadowserver.org"
    }
]

An example specifying the report type:

freed0@pit:~$ ./call-api.py reports/list '{"reports":["united-states", "california"], "type":"hp_ics_scan", "date":"2020-10-27"}' pretty
[
    {
        "timestamp": "2020-10-27",
        "report": "united-states@shadowserver.org",
        "file": "2020-10-27-hp_ics_scan-united_states_of_america-geo.csv",
        "id": "60EL63IKLH90uLKJ3xcmXW93403mnkdsfiuIje-0nmdsseTXNP5EgDog"
    },
    {
        "report": "california@shadowserver.org",
        "timestamp": "2020-10-27",
        "id": "3lG4u-cHIOY9083oihdkUGIkhdsjuj38sdlkn0u92345AHyO9g5H5PK7A",
        "file": "2020-10-27-hp_ics_scan-california-region.csv"
    }
]

reports/download

Fields:

apikey  : string : Your API key
id      : string : Report id
report  : string : Specific report name 
limit   : number : Limit the number of records that are returned

Response:

  • The requested data is handed back in JSON format.

Example request data:

freed0@pit:~$ ./call-api.py reports/download '{"id":"uN6n7yZK90sdflkjdlLKTOkspksg?HjgX1lI_hAdsKGmVanG_Og", "limit":3}' pretty
[
   {
      "sic" : "",
      "source" : "Alien Vault",
      "geo" : "US",
      "asn" : "20115",
      "region" : "SOUTH CAROLINA",
      "sector" : "Information Technology",
      "reason" : "Malicious Host US",
      "city" : "DUNCAN",
      "ip" : "35.131.161.166",
      "hostname" : "",
      "timestamp" : "2020-10-26 07:00:12",
      "naics" : "517311"
   },
   {
      "ip" : "35.188.187.206",
      "timestamp" : "2020-10-26 07:00:12",
      "hostname" : "",
      "naics" : "519130",
      "sic" : "",
      "geo" : "US",
      "asn" : "15169",
      "source" : "Alien Vault",
      "sector" : "Information Technology",
      "region" : "IOWA",
      "city" : "COUNCIL BLUFFS",
      "reason" : "Malicious Host US"
   },
   {
      "city" : "THE DALLES",
      "reason" : "Malicious Host US",
      "region" : "OREGON",
      "sector" : "Information Technology",
      "source" : "Alien Vault",
      "geo" : "US",
      "asn" : "15169",
      "sic" : "",
      "naics" : "519130",
      "hostname" : "",
      "timestamp" : "2020-10-26 07:00:12",
      "ip" : "35.233.182.140"
   }
]

Alternative query example:

freed0@pit:~$ ./call-api.py reports/download '{"report":"california", "date":"2020-10-26", "type":"scan_ssl_freak", "limit":1}' pretty
[
    {
        "timestamp": "2020-10-26 00:11:30",
        "handshake": "TLSv1.0",
        "issuer_given_name": "",
        "issuer_business_category": "",
        "issuer_surname": "",
        "city": "LOS ANGELES",
        "issuer_street_address": "",
        "issuer_postal_code": "",
        "validation_level": "unknown",
        ...
        "issuer_organization_name": "Snake Oil, Ltd",
        "browser_trusted": "N",
        "http_reason": "OK",
        "server_type": "Apache/2",
        "browser_error": "x509: certificate signed by unknown authority",
        "subject_locality_name": "Snake Town",
        "hostname": "45-35-38-141.static.secserverpros.com",
        "subject_organization_unit_name": "Webserver Team",
        "cert_issue_date": "1999-10-21 18:21:51",
        "transfer_encoding": "",
        "subject_business_category": "",
        "subject_state_or_province_name": "Snake Desert",
        "signature_algorithm": "md5WithRSAEncryption",
        "tag": "ssl-freak",
        "sic": "",
        "subject_surname": "",
        "key_algorithm": "rsaEncryption",
        "sha1_fingerprint": "165931466980620243E0DB952900D7587A80307E",
        "issuer_email_address": "ca@snakeoil.dom",
        "md5_fingerprint": "BAEC163027CA9917FFDFA44CBCBF1B98",
        "http_response_type": "HTTP/1.1"
    }
]

reports/query

Fields:

apikey : string     : Your API key 
report : string     : Specific report name 
query  : dictionary : Search fields [any of the available fields from the reports].
sort   : string     : Ascending | descending 
date   : string     : Date (YYYY-MM-DD) or range (YYYY-MM-DD:YYYY-MM-DD) now may be used in place of a date 
facet  : string.    : Returns the cardinality of each value of the given field sorted from highest to lowest
limit  : number     : Specify the number of records to pull
page   : number     : Default is 1; used to obtain additional pages of results

Response:

  • array of up to 1,000 matching records

Example SSL Query by port:

freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-27", "query":{"port":443}, "limit":1}' pretty
[
    {
        "port": "443",
        "tag": [
            "ssl"
        ],
        "region": "VIRGINIA",
        "protocol": "tcp",
        "isp_name": "MICROSOFT CORP",
        "county_fips": "51107",
        "naics": "334111",
        "infection": "ssl",
        "type": "scan",
        "asn": "8075",
        "asn_name": "MICROSOFT-CORP-MSN-A",
        "latitude": "39.03",
        "county_name": "LOUDOUN",
        "ip": "13.82.60.109",
        "rip": "109.60.82.13",
        "timestamp": "2020-10-27 02:59:32",
        "sid": "4096627",
        "geo": "US",
        "city": "ASHBURN",
        "longitude": "-77.49",
        "sector": "Critical Manufacturing"
    }
]

Example SSL query by tag:

freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-27", "query":{"tag":"ssl"}, "limit":1}' pretty
[
    {
        "region": "VIRGINIA",
        "protocol": "tcp",
        "tag": [
            "ssl"
        ],
        "port": "443",
        "county_fips": "51107",
        "naics": "334111",
        "isp_name": "MICROSOFT CORP",
        "infection": "ssl",
        "type": "scan",
        "asn_name": "MICROSOFT-CORP-MSN-A",
        "latitude": "39.03",
        "asn": "8075",
        "county_name": "LOUDOUN",
        "rip": "109.60.82.13",
        "timestamp": "2020-10-27 02:59:32",
        "sid": "4096627",
        "ip": "13.82.60.109",
        "sector": "Critical Manufacturing",
        "geo": "US",
        "longitude": "-77.49",
        "city": "ASHBURN"
    }
]

Example using IP address:

freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-27", "query":{"ip":"13.82.60.109"}, "limit":1}' pretty
[
    {
        "geo": "US",
        "longitude": "-77.49",
        "city": "ASHBURN",
        "sector": "Critical Manufacturing",
        "ip": "13.82.60.109",
        "timestamp": "2020-10-27 02:59:32",
        "rip": "109.60.82.13",
        "sid": "4096627",
        "county_name": "LOUDOUN",
        "asn": "8075",
        "latitude": "39.03",
        "asn_name": "MICROSOFT-CORP-MSN-A",
        "type": "scan",
        "infection": "ssl",
        "isp_name": "MICROSOFT CORP",
        "county_fips": "51107",
        "naics": "334111",
        "tag": [
            "ssl"
        ],
        "port": "443",
        "region": "VIRGINIA",
        "protocol": "tcp"
    }
]

Example by City:

freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-27", "query":{"city":"ashburn"}, "limit":1}' pretty
[
    {
        "geo": "US",
        "rip": "193.74.6.52",
        "tag": [
            "http"
        ],
        "longitude": "-77.49",
        "infection": "http",
        "latitude": "39.03",
        "isp_name": "AMAZON TECHNOLOGIES INC.",
        "county_fips": "51107",
        "county_name": "LOUDOUN",
        "protocol": "tcp",
        "asn": "14618",
        "port": "8080",
        "sid": "4096627",
        "asn_name": "AMAZON-AES",
        "tld": "com",
        "city": "ASHBURN",
        "type": "scan",
        "ip": "52.6.74.193",
        "region": "VIRGINIA",
        "sector": "Communications",
        "naics": "454110",
        "timestamp": "2020-10-27 00:02:21",
        "domain": "crmapp005.anterasoftware.com"
    }
]

Example requiring a tag:

freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-27", "query":{"require":"tag"}, "limit":1}' pretty
[
    {
        "rip": "229.156.120.34",
        "tag": [
            "http"
        ],
        "geo": "US",
        "county_fips": "29047",
        "isp_name": "GOOGLE INC.",
        "county_name": "CLAY",
        "latitude": "39.22",
        "infection": "http",
        "longitude": "-94.57",
        "asn_name": "GOOGLE",
        "sid": "4096627",
        "port": "8080",
        "asn": "15169",
        "protocol": "tcp",
        "city": "KANSAS CITY",
        "tld": "com",
        "region": "MISSOURI",
        "sector": "Defense Industrial Base",
        "ip": "34.120.156.229",
        "type": "scan",
        "naics": "519130",
        "timestamp": "2020-10-27 00:02:19",
        "domain": "229.156.120.34.bc.googleusercontent.com"
    }
]

Example of getting counts of different tags:

freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-27", "query":{"require":"tag"}, "facet":"tag", "limit":9}' pretty
[
    {
        "count": 15203935,
        "tag": "ssl"
    },
    {
        "tag": "cwmp",
        "count": 11129661
    },
    {
        "tag": "ftp",
        "count": 2781725
    },
    {
        "tag": "http",
        "count": 2672067
    },
    {
        "tag": "ssl-poodle",
        "count": 1866075
    },
    {
        "count": 1519186,
        "tag": "isakmp"
    },
    {
        "tag": "quic",
        "count": 1212766
    },
    {
        "tag": "rdp",
        "count": 851989
    },
    {
        "count": 571483,
        "tag": "portmapper"
    }
]

Example getting tag counts over time. Note that since most of the data is repeated daily a count like this has little value:

freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", "date":"2020-10-01:2020-10-28", "query":{"require":"tag"}, "facet":"tag", "limit":9}' pretty
[
    {
        "tag": "ssl",
        "count": 322512588
    },
    {
        "count": 238483485,
        "tag": "cwmp"
    },
    {
        "tag": "ftp",
        "count": 57667266
    },
    {
        "count": 57308759,
        "tag": "http"
    },
    {
        "count": 39666402,
        "tag": "ssl-poodle"
    },
    {
        "tag": "isakmp",
        "count": 32107711
    },
    {
        "count": 30016842,
        "tag": "quic"
    },
    {
        "count": 16919098,
        "tag": "rdp"
    },
    {
        "tag": "portmapper",
        "count": 12755184
    }
]

Our 86 Report Types