ipsetmgr - a tool for management and deployment of ipset data

Lots needs to be documented, but let's get some basics down.

System structure

ipsetmgr is based around maintaining sets of IP data to be used for access/blocking via the ipset command or other kernel firewalls mechanisms like nftables or application level checking tools like [https://code.quatermass.co.uk/jgh/nmh](nmh - Net-Masked-Hero).

ipsetmgr makes no distinction between IPV4 and IPV6; it is only within the consumer modules where any splitting into family-specific requirements are handled.

General command structure

    ipsetmgr --command something --setname nameofset --filename /var/tmp/somestuff

Examples

Initialise the database.

ipsetmgr --command init

Create a set or two

ipsetmgr --command addset --setname friends
ipsetmgr --command addset --setname enemies

Load an existing list of IPs into a set

ipsetmgr --command bulkload --setname friends --filename friends_ips
ipsetmgr --command bulkload --setname enemies --filename enemies_ips

Assign a consumer of a set as IPSets

ipsetmgr --command consumer --setname friends --consumer IPSets
ipsetmgr --command consumer --setname enemies --consumer IPSets

Apply the set data to the assigned consumers

ipsetmgr --command apply --setname friends --timings
ipsetmgr --command apply --setname enemies --timings

Additionally there is an --apply option which can be used along with another command (assuming a consumer is already defined, or being defined).

ipsetmgr --command bulkload --setname friends --filename friends_ips --apply
ipsetmgr --command consumer --setname friends --consumer IPSets --apply

And since there is no conflict in parameter use for the previous ideas we can wrap everything up in one go

ipsetmgr --command addset --setname friends --consumer IPSets --filename friends_ips --apply --timings
ipsetmgr --command addset --setname enemies --consumer IPSets --filename enemies_ips --apply --timings

Not into using IPSets ? No problem, try nftables.

ipsetmgr --command addset --setname friends --consumer NFTables --filename friends_ips --apply --timings
ipsetmgr --command addset --setname enemies --consumer NFTables --filename enemies_ips --apply --timings

Land of plenty, multiple consumers

ipsetmgr --command consumer --setname friends --consumer NFTables --apply

Load ASN data (repeat when needed)

curl -O https://iptoasn.com/data/ip2asn-combined.tsv.gz
ipsetmgr --command asnload --filename ip2asn-combined.tsv.gz

Use the ASN data in a set

ipsetmgr --command addset --setname asnblock
ipsetmgr --command consumer --setname asnblock --consumer IPSets

ipsetmgr --command asnsetentry --asn 44477 --setname asnblock --comment "stark spammers"
ipsetmgr --command asnsetentry --asn 0 --setname asnblock --comment "block non-routed"

ipsetmgr --command apply --setname asnblock

Use the ASN data in a different set

ipsetmgr --command addset --setname asnlocal
ipsetmgr --command consumer --setname asnlocal --consumer IPSets

ipsetmgr --command asnsetentry --asn 64499 --setname asnlocal --comment "local AS ips"

ipsetmgr --command apply --setname asnlocal

Updating the ASN data and applying to applicable sets

curl -O https://iptoasn.com/data/ip2asn-combined.tsv.gz
ipsetmgr --command asnload --filename ip2asn-combined.tsv.gz --apply

Load the Spamhaus SBLDrop lists (repeat when needed)

curl -O https://www.spamhaus.org/drop/drop_v4.json
curl -O https://www.spamhaus.org/drop/drop_v6.json
ipsetmgr --command sblload --setname enemies --apply --filename drop_v4.json drop_v6.json

With both the ASN and SBLDrop features, the system will automatically replace existing entries with the new data.

Not only, but also

Each entry in a set can also be tagged with a source value which can be used to track where the data came from. This is used by the ASN and SBLDrop features to allow removal of items in a set without touching other data. This means that if it is the most convenient method for final deployment, a single set can be used for all data with personal lists, AS blocks and SBLDrops all merged into the single list.

This feature can also be used the flush option to allow deletion and replacement of a bulkload source. So, our previous example

ipsetmgr --command bulkload --setname enemies --filename enemies_ips --apply

could become a repeated operation

ipsetmgr --command bulkload --setname enemies --source enemy_file_1 -flush --filename enemies_ips --apply

and all previous entries with source enemy_file_1 get flushed, the new file loaded with the same source value and then the whole set being loaded by the defined consumers.

Hold on, did you says consumers plural ?

Why yes, I did.

Whilst I currently can't think of a sane use of, say, two different ipsets using the same set data it is possible. Just a subtle adjustment when defining each consumer.

ipsetmgr --command consumer --setname enemies --consumer IPSets --name 'enemy_mine'
ipsetmgr --command consumer --setname enemies --consumer IPSets --name  no_friend_of_mine --params '{"setname_v4":"fail32","setname_v6":"badbadbad666"}'

Also you can use different consumer modules, so should you want an NFTables and an NMH to use the same set.

ipsetmgr --command consumer --setname enemies --consumer NFTables
ipsetmgr --command consumer --setname enemies --consumer NMH

ASN Handling

The ASN handling is designed to provide end-points (servers and the like) with a method of acting on IP at an AS level, rather than being utilised by networking equipment (routers, firewalls). This is not to say it cannot be used by such network kit, merely that it is not the primary focus.

The approach with ASN-based data is to maintain a nominally full set of ASN data regardless of whether or which specific ASN we are interested in using in sets. When a new set of ASN data is loaded and changes that need to be applied to in-use ASN will automatically be propogated. This allows for easy communication/distribution of which ASN are bad without having to cover details of actual ranges which might have changed by the time the ASN numbers are acted on.

Currently the code is designed to load the tab-separated data from [https://iptoasn.com/](https://iptoasn.com/ IPtoASN Database) This data has some interesting properties which are directly handled by the current loader but are noted here for clarity.

The data is a set of tab-separated fields in the format

    range_start range_end AS_number country_code AS_description

Observations

• Currently the AS listed appear to only be those that appear at the original collection point.
• The source includes items like RFC1918 space, sometimes as ASN0, sometimes as an ASN visible at the source collection point.
• non-routed IP are listed as ASN 0
• Some different type of IP (RFC5735,RFC6598,RFC4291,RFC4048,RFC4193,RFC3879 et al) are merged as ASN0.

Loading Rules and Consequences

The above entails some work be performed at load time to provide the most useful resultant data. The sequence is something like the following

• If a range has different IP types (see ip_iptypev6/ip_iptypev6 in Net::IP) it is split into CIDRs and each entry lower/upper checked until they are matching types.

• If both the IP types are not both either PUBLIC or GLOBAL-UNICAST then the range is skipped.

Assumptions

• Access/routing based on decisions like ownership of an ASN, RFC1918 spaces is already handled at a proper network border device or directly specified in some other set.

Current Issues

• cannot currently handle blocking ASN that are not mentioned in the bulk data. To be fixed.