Turns out that awk lends itself very nicely to writing OpenSMTPD filters with its line-based filter protocol. Below is a simple example of how to implement a simple DNSBL check.

Note, the snippet below is an example, only, and has some shortcomings for simplicity:

• only handles IPv4 addresses
• does not do any version detection of the filter protocol, and doesn't work on filter protocol < 0.6 (which was used in OpenSMTPD < 6.7; see comments in the script for details)
• SMTP error returned on blacklisting is hardcoded, but should be configurable
• has hardcoded logging of every filter action, people might want to silence it

#!/usr/bin/awk -f
#
# Usage in smtpd.conf:
#   filter <filter-name> proc-exec "/path/to/filter-dnsbl <resolve_cmd> <dnsbl> <ip_bl>"
#
# Where:
# - <resolve_cmd> is a string used to resolve the DNSBL query, returning
#   only the response, use %s for assembled request, escape % with %%, e.g.:
#     "dig +short %s"
#     "host -t A %s | sed 's/^.*has address //'"
# - <dnsbl> is the DNSBL address-suffix to look up, e.g.:
#     "ix.dnsbl.manitu.net"
# - <ip_bl> is a regex, if IP(s) returned by the lookup match they are
#   considered "blacklisted", e.g.:
#     "^127\.0\.0\.[234]$"
#
# Examples (for smtpd.conf):
#   filter dnsbl_nixspam proc-exec "filter-dnsbl.awk \"host -t A %s | sed 's/^.*has address //'\" ix.dnsbl.manitu.net '^127\.0\.0\.2$'"
#   filter dnsbl_nixspam proc-exec "filter-dnsbl.awk \"dig +short %s A\" bl.spamcop.net '^127\.0\.0\.2$'"

BEGIN {
    if (ARGC != 4) {
        printf("Error, 4 args expected, got %d\n", ARGC) > "/dev/stderr"
        exit 1  # note, this will terminate smtpd
    }
    RESOLVE_CMD = ARGV[1]
    DNSBL = ARGV[2]
    IP_BL = ARGV[3]
    ARGC = 0 # no more input args / files
    FS = "|"
}

"config|ready" == $0 {
    print("register|filter|smtp-in|connect") > "/dev/stdin"
    print("register|ready") > "/dev/stdin"
    next # don't exit as this will stop smtpd
}
"filter" == $1 {
    if (NF < 9) {
        printf("Error, filter line not having enough fields, 9+ expected, got %d\n", NF) > "/dev/stderr"
        next # don't exit as this will stop smtpd
    }
    sess_id = $6
    resp_token = $7
    # reverse on the fly and trim port
    # !NOTE!: with version < 0.6 (e.g. $2 == "0.5") the connecting ip is in $10 - not handling version detection, here
    split($9, x, "[.:]")
    req = x[4]"."x[3]"."x[2]"."x[1]"."DNSBL  # !NOTE!: works only with ipv4

    ret = "proceed"
    cmd = sprintf(RESOLVE_CMD, req)
    while((cmd | getline r) > 0) {
        if(r ~ IP_BL) {
            ret = "reject|550 connecting server is blacklisted"
            break
        }
    }
    close(cmd)

    print(sess_id" DNSBL check: "$8" ["$9"]: "cmd" => "ret) > "/dev/stderr"
    print("filter-result|"sess_id"|"resp_token"|"ret) > "/dev/stdin"
}