internetdata_result (internetdata v1.17.1)

View Source

Turns a decoded JSON body into the maps the client hands back.

Only a name the SPEC defines, at the POSITION the spec defines it, becomes an atom. Everything else keeps its binary key. That is not a style preference: schema, sample and size are keyed by format, the rows under sample by the dataset's own column names, and an AS answer's whois by the registry's own field names, all of which are the server's to choose. Erlang's atom table is capped at 1,048,576 entries and is never garbage collected, so minting an atom per key off a body like that is a remote VM kill rather than a leak.

The consequence to read the results by: maps:get(base, Db) for a documented field, maps:get(<<"csvgz">>, Size) for anything keyed by format or by a dataset column.

Absent is not false. A key that is ABSENT from a lookup answer is one your plan does not include, the paid members inside a shared object (location, asn, company, abuse) included. It never means "we could not check", so an absent key and false are genuinely different answers, and maps:get(is_hosting, Result, undefined) is how you tell them apart. A detail map that is present but empty (#{}, Web IP's []) means the flag above it is false. raw holds the answer as the wire carried it.

Summary

Functions

An AS answer. whois is keyed by the registry's own field names, so its keys stay binaries.

The answer a bogon gets, in the full shape the API serves on a paid plan: every flag false, every detail empty (Web IP's list empty), every key of location, asn, company, abuse, whois and rwhois empty or 0, no hostname and no hosted domains, and is_bogon true.

Every digest published for one dataset file.

The catalog, one entry per dataset family.

A domain answer.

Recent download attempts, refusals included.

What a key may spend, and what it has.

Translate a decoded lookup answer into a result.

One dataset's build document: schema, samples, row count, sizes.

Types

abuse/0

-type abuse() ::
          #{email := binary(),
            name => binary(),
            phone => binary(),
            address => binary(),
            country => binary(),
            network => binary(),
            validated => boolean(),
            unvalidated_since => binary()}.

asn/0

-type asn() ::
          #{asn := integer(),
            name := binary(),
            domain := binary(),
            type := binary(),
            prefix := binary(),
            country => binary(),
            rpki_status => binary()}.

asn_prefix/0

-type asn_prefix() ::
          #{prefix := binary(),
            peer_count := integer(),
            rpki_status := binary(),
            first_seen := binary(),
            country := binary(),
            size := integer(),
            other_origins := [integer()]}.

asn_result/0

-type asn_result() ::
          #{asn := integer(),
            name := binary(),
            country := binary(),
            registry := binary(),
            allocated := binary(),
            domain := binary(),
            type => binary(),
            num_ips => integer(),
            num_prefixes4 => integer(),
            num_prefixes6 => integer(),
            rpki_valid => integer(),
            rpki_invalid => integer(),
            rpki_not_found => integer(),
            whois => #{binary() => binary()},
            prefixes => [asn_prefix()],
            rank => integer(),
            tier1 => boolean(),
            num_upstreams => integer(),
            upstreams => [integer()],
            num_peers => integer(),
            peers => [integer()],
            num_downstreams => integer(),
            downstreams => [integer()],
            hosted_domains => hosted_domains(),
            trial_expired => boolean(),
            cap_reached => boolean()}.

bittorrent/0

-type bittorrent() ::
          #{verified => boolean(),
            id_matches_ip => boolean(),
            first_seen => binary(),
            last_seen => binary(),
            last_reachable => binary(),
            days_seen => integer(),
            num_ports => integer(),
            peak_num_ports => integer(),
            indexer => boolean(),
            clients => [binary()]}.

carrier/0

-type carrier() ::
          #{name => binary(),
            mcc => binary(),
            mnc => binary(),
            country => binary(),
            confidence => binary()}.

class/0

-type class() ::
          #{provider => binary(),
            provider_name => binary(),
            confidence => binary(),
            last_seen => binary()}.

company/0

-type company() :: #{name := binary(), domain := binary(), type => binary(), country => binary()}.

disposable/0

-type disposable() ::
          #{confidence => binary(),
            providers => [binary()],
            first_seen => binary(),
            last_seen => binary()}.

dns/0

-type dns() ::
          #{a := [dns_address()],
            aaaa := [dns_address()],
            mx := [dns_mx()],
            ns := [dns_ns()],
            cname := [dns_cname()]}.

dns_address/0

-type dns_address() :: #{ip := binary(), last_seen := binary()}.

dns_cname/0

-type dns_cname() :: #{target := binary(), last_seen := binary()}.

dns_mx/0

-type dns_mx() :: #{priority := integer(), host := binary(), last_seen := binary()}.

dns_ns/0

-type dns_ns() :: #{host := binary(), last_seen := binary()}.

dns_resolver/0

-type dns_resolver() ::
          #{port => integer(),
            recursion_available => boolean(),
            rcode => integer(),
            answer_correct => boolean(),
            first_seen => binary(),
            last_seen => binary()}.

domain_result/0

-type domain_result() ::
          #{domain := binary(),
            is_registered := boolean(),
            is_disposable => boolean(),
            has_a := boolean(),
            has_aaaa := boolean(),
            has_mx := boolean(),
            has_ns := boolean(),
            has_cname := boolean(),
            dns => dns(),
            ip_ranges => ip_ranges(),
            disposable => disposable(),
            trial_expired => boolean(),
            cap_reached => boolean()}.

entitlement/0

-type entitlement() ::
          #{org_id := binary(),
            apikey := entitlement_apikey(),
            plan := entitlement_plan(),
            usage := entitlement_usage()}.

entitlement_apikey/0

-type entitlement_apikey() :: #{id := binary(), expires := binary() | null, allowed_cidrs := [binary()]}.

entitlement_plan/0

-type entitlement_plan() :: #{key := binary(), name := binary(), tier := binary()}.

entitlement_usage/0

-type entitlement_usage() ::
          #{requests := integer(),
            quota := integer(),
            hard_limit := integer() | null,
            window_start := binary(),
            window_end := binary()}.

hosted_domains/0

-type hosted_domains() :: #{total := integer(), domains => [binary()]}.

ip_range/0

-type ip_range() :: #{range := binary(), name := binary(), country := binary()}.

ip_ranges/0

-type ip_ranges() ::
          #{domain := binary(),
            redirects_to := binary(),
            total := integer(),
            ipv4 := integer(),
            ranges := [ip_range()]}.

location/0

-type location() ::
          #{continent := binary(),
            continent_name := binary(),
            country := binary(),
            country_name := binary(),
            timezone := binary(),
            currency_code := binary(),
            currency_name := binary(),
            currency_symbol := binary(),
            flag_emoji := binary(),
            flag_unicode := binary(),
            region => binary(),
            region_code => binary(),
            city => binary(),
            postal_code => binary(),
            latitude => number(),
            longitude => number(),
            accuracy_radius => integer(),
            geoname_id => binary(),
            dma_code => binary(),
            last_changed => binary()}.

proxy/0

-type proxy() ::
          #{provider => binary(),
            provider_name => binary(),
            first_seen => binary(),
            last_seen => binary(),
            hits => integer(),
            hits_days_pct => integer(),
            providers_num => integer()}.

pubproxy/0

-type pubproxy() ::
          #{entry => boolean(),
            exit => boolean(),
            protocols => [binary()],
            num_ports => integer(),
            num_proxies => integer(),
            last_seen => binary(),
            last_usable => binary()}.

result/0

-type result() ::
          #{raw := map(),
            ip := binary(),
            is_bogon := boolean(),
            is_vpn => boolean(),
            is_tor := boolean(),
            is_relay := boolean(),
            is_resproxy => boolean(),
            is_dcproxy => boolean(),
            is_mobproxy => boolean(),
            is_pubproxy => boolean(),
            is_hosting => boolean(),
            is_cdn => boolean(),
            is_anycast := boolean(),
            is_carrier := boolean(),
            is_satellite := boolean(),
            is_ssh := boolean(),
            is_smtp := boolean(),
            is_web := boolean(),
            is_dns_resolver := boolean(),
            is_bittorrent := boolean(),
            vpn => class(),
            tor := class(),
            relay := class(),
            resproxy => proxy(),
            dcproxy => proxy(),
            mobproxy => proxy(),
            pubproxy => pubproxy(),
            hosting => class(),
            cdn => class(),
            carrier => carrier(),
            satellite => satellite(),
            ssh => ssh(),
            smtp => smtp(),
            web => [web()],
            dns_resolver => dns_resolver(),
            bittorrent => bittorrent(),
            location := location(),
            asn := asn(),
            company := company(),
            hostname := binary(),
            abuse := abuse(),
            whois => whois(),
            rwhois => rwhois(),
            hosted_domains => hosted_domains(),
            trial_expired => boolean(),
            cap_reached => boolean()}.

rwhois/0

-type rwhois() ::
          #{id := binary(),
            name := binary(),
            country := binary(),
            source := binary(),
            domain := binary(),
            abuse_email := binary()}.

satellite/0

-type satellite() ::
          #{provider => binary(),
            provider_name => binary(),
            orbit => binary(),
            segment => binary(),
            country => binary(),
            asn => integer(),
            confidence => binary()}.

smtp/0

-type smtp() ::
          #{port => integer(),
            banner => binary(),
            session_ok => boolean(),
            first_seen => binary(),
            last_seen => binary()}.

ssh/0

-type ssh() ::
          #{port => integer(),
            banner => binary(),
            host_key_fp => binary(),
            session_ok => boolean(),
            first_seen => binary(),
            last_seen => binary()}.

web/0

-type web() ::
          #{port := integer(),
            tls_ok := boolean(),
            session_ok := boolean(),
            status_code := integer(),
            server := binary(),
            location := binary(),
            title := binary(),
            cert_subject_cn := binary(),
            cert_subject_o := binary(),
            cert_issuer_dn := binary(),
            cert_browser_trusted := boolean(),
            tls_version := binary(),
            first_seen := binary(),
            last_seen := binary()}.

whois/0

-type whois() ::
          #{id := binary(),
            name := binary(),
            org := binary(),
            country := binary(),
            status := binary(),
            source := binary(),
            domain := binary(),
            abuse_email := binary(),
            asn := integer()}.

Functions

asn(Wire)

-spec asn(map()) -> asn_result().

An AS answer. whois is keyed by the registry's own field names, so its keys stay binaries.

bogon(Ip)

-spec bogon(binary()) -> result().

The answer a bogon gets, in the full shape the API serves on a paid plan: every flag false, every detail empty (Web IP's list empty), every key of location, asn, company, abuse, whois and rwhois empty or 0, no hostname and no hosted domains, and is_bogon true.

This is deliberately the WIDEST shape whatever your plan is, so a caller must not infer which fields they are entitled to from a bogon answer.

Built as a WIRE body and then mapped, so raw carries the same answer the mapped keys do.

checksums(Wire)

-spec checksums(map()) -> map().

Every digest published for one dataset file.

databases(Wire)

-spec databases([map()]) -> [map()].

The catalog, one entry per dataset family.

domain(Wire)

-spec domain(map()) -> domain_result().

A domain answer.

downloads(Wire)

-spec downloads([map()]) -> [map()].

Recent download attempts, refusals included.

entitlement(Wire)

-spec entitlement(map()) -> entitlement().

What a key may spend, and what it has.

from_wire(Body)

-spec from_wire(map()) -> result().

Translate a decoded lookup answer into a result.

Folding over what the body HAS is what preserves absent-versus-false: a key the server did not send cannot appear in the fold, so no plan-gated member can be invented as false.

metadata(Wire)

-spec metadata(map()) -> map().

One dataset's build document: schema, samples, row count, sizes.