Ok. Removed from v0.4.1. Thanks for the assistance - and the experience.
For me, it also meant that PayPal payment notification code couldn’t contact Paypal until I put a long list of IP address ranges in there, which could change at any time without warning, of course.
I’ve hit a new snag. People seem to be unable to connect using ftp, whether ftp:// or ftps:// or ftpes:// each time it ends badly, with:
| Status: | Resolving address of ****.uk |
|---|---|
| Status: | Connecting to ..*:21… |
| Status: | Connection established, waiting for welcome message… |
| Status: | Initializing TLS… |
| Status: | Verifying certificate… |
| Status: | TLS connection established. |
| Status: | Logged in |
| Status: | Retrieving directory listing… |
| Command: | PWD |
| Response: | 257 “/” is your current location |
| Command: | TYPE I |
| Response: | 200 TYPE is now 8-bit binary |
| Command: | PASV |
| Response: | 227 Entering Passive Mode (,,,,214,197) |
| Command: | MLSD |
| Error: | The data connection could not be established: ECONNREFUSED - Connection refused by server |
Have you updated to the current version recently? I got FTP wrong in the early releases - the documentation on how to do this is less than sparse.
I made some changes for FTP in version 0.2.2. The current version 0.4.5 has not had any significant changes for some time now. It seems to be running OK on all my machines.
a) You need to have the recent version of nftfw_init.nft - which contains:
table ip myhelpers {
…
which is incorrectly defined in the early releases.
b) Check that rule/ftp-helper.sh is current
c) Add
06-ftp-helper to incoming
c) and add to incoming
20-ftp
to allow the port in.
I think that pure only allows ssh connections when logging in - but I seem to remember that it switches automatically so ftps isn’t needed. I did test all this and it seemed to work.
It’s a brand new build, from Sunday.
It said it uninstalled and reinstalled 0.5.0 when I ran sudo pip3 install . so I guess it is current
And there are 06-ftp-helper and 20-ftp 22-ftps 22-ftps-data in /usr/local/etc/nftfw/incoming.d
It seems that Pure-ftpd needs a range of inbound ports to operate correctly. Version 0.5.1 now includes a new rule:
ftp-passive.sh
which looks in Pure’s /etc/pure-ftpd/conf/PassivePortRange to obtain the range and then include it in the incoming nftables rule. This is included by adding:
21-ftp-passive
to incoming.d.
When you pull the new version, you’ll need to run Install.sh to put the new files in /usr/local/etc/nftfw. Then copy original/rule/ftp-passive.sh to rule.d, and
touch incoming.d/21-ftp-passive
to create the file to load the rules.
Run nftfw -x load to check that the tables will be OK, and then nftfw -f load to reload.
All should then be well.
You appear to be correct. All does seem to be well!
Goody…
Getting a little bit less beta as time goes on.
I think I may have uncovered another bug. A user had some wrong settings in their Thunderbird setup and managed to get themseves into the blacklist
I deleted the ip.ad.dr.ess.auto file from blacklist.d and it reappeared again, so I deleted it again and made a new file ip.ad.dr.ess file in whitelist.d
Subsequent attempts to connect haven’t resulted in a new blacklist entry, but they also haven’t managed to connect. Even web browser access to the server is blocked.
So perhaps the whitelist isn’t quite as powerful as it should be. Wassup, doc?
Removing .auto entries is not sufficient, the blacklist.d directory will be rebuilt from the database. To remove an ip address from the blacklist database use
nftfwedit -d a.b.c.d
this will also delete any entries in blacklist.d.
nftfwedit a.b.c.d
will also give you information about the ip address.
If the ip address is in the whitelist, it will be accepted immediately before any checks.
It maybe that the whitelisted address hasn’t made it into the actual nftables setup.
Check that nftables contains the address by using:
sudo nft list ruleset ip
or
sudo nft list ruleset ip6
if it’s an IPv6 address. I use a shell alias for this:
alias nfl='sudo nft list ruleset ip|less'
The set w_all should contain all whitelisted addresses.
It’s possible for the caching system to get out of sync, use
sudo nftfw -f load
to ensure a full reload of the tables. If you are not using incron, this is needed when you add/remove files by hand, and occasionally I find it necessary - or I am just being prudent.
Today, from release 0.6, I’ve removed the use of incron for nftfw. It’s replaced by a systemd path watching unit which does the job. The Updating nftfw document summarises the steps you need to take to unwind the previous incron use and install the systemd version.
Using an ‘active’ directory is not needed to make the basic system update itself, but means that when firewall information is manually changed then you don’t need to remember to reload the firewall.
I am now thinking very seriously about providing bulk block lists so I can keep all the IPs from a certain country out of my system. I now have 2097 distinct IP addresses in my database, and 66% of them are from one country. There was a certain excitement when the firewall system was initially published, and activity is now slowly reducing.
As promised, from release 0.7.1, I’ve added the ability to create large blacklist sets from lists of network addresses entered in CIDR format. This adds a new control directory blacknets.d containing files called something.nets.
It’s now possible to easily block countries and also other ranges of IP addresses.
The ‘Getting CIDR Lists’ document contains two methods of getting lists, including how to use the scripts I’ve provided to download files derived from the GeoIP2 databases regularly. There are, of course, other providers. The Updating nftfw document also includes some comments on updating the system files because of the change.
It turned out that this was not actually a massive alteration to the system, it just needed a new way to read files but could then use all the existing code that manages firewall sets without any change.
Trying this on a test server. Looks good.
But why not put the recommended settings into the distribution .ini file? The only changes I made to it were the ones you suggested, so it would make sense to make them the default, wouldn’t it?
Am I right in assuming that the settings on lines starting with ; in the config.ini file are the default settings? I’m assumed so, but it doesn’t actually say so anywhere, as far as I can see.
I don’t want to zap any changes you might have made.
Yes - the values with semi-colons are the default settings. This is a common convention, which is obviously not common enough not to need documenting 
Version 0.7.2 has a new flag to nftfw
$ sudo nftfw -a
lists config sections and values that differ from the compiled in default versions.
You can never assume that people understand what you assume to be a common convention.
People can’t be sure that yoir idea of the norm is the same as theirs.
Try turning off a light switch in the USA to see what I mean.
Well as it happens it’s explained in Para 2 of the nftfw-config man page…
Version 0.7.3 has enhanced Install.sh so it can be run from a config file. See the Autoinstall.default file for how to set this up. This was an issue request on GitHub.
Suddenly today, I’m getting a slew of errors, one email each fifteen minutes:
cron /usr/local/bin/nftfw -q blacklist
Traceback (most recent call last):
File "/usr/local/bin/nftfw", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/nftfw/__main__.py", line 128, in main
sc.run(args.action)
File "/usr/local/lib/python3.7/dist-packages/nftfw/scheduler.py", line 65, in run
self.execute(command)
File "/usr/local/lib/python3.7/dist-packages/nftfw/scheduler.py", line 143, in execute
changes = bl.blacklist()
File "/usr/local/lib/python3.7/dist-packages/nftfw/blacklist.py", line 87, in blacklist
work = log_reader(self.cf)
File "/usr/local/lib/python3.7/dist-packages/nftfw/logreader.py", line 57, in log_reader
res = one_log_reader(cf, filename, patinfo, update_position)
File "/usr/local/lib/python3.7/dist-packages/nftfw/logreader.py", line 167, in one_log_reader
out = scanlog(patinfo, fhandle)
File "/usr/local/lib/python3.7/dist-packages/nftfw/logreader.py", line 209, in scanlog
for ip, patinfo in active:
File "/usr/local/lib/python3.7/dist-packages/nftfw/logreader.py", line 205, in <genexpr>
active = (f for f in found if f)
File "/usr/local/lib/python3.7/dist-packages/nftfw/logreader.py", line 203, in <genexpr>
found = (matchline(allregex, line) for line in lines)
File "/usr/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1298-1299: invalid continuation byteand the blacklist hasn’t been updated since the early hours of this morning.
How do I fix this? Any ideas?