Looks like there’s something odd - an unicode error - in one of the log files you are scanning. I am guessing that you haven’t made a change to the pattern files… can you confirm this?
You may be able to isolate which file by settting up a test pattern and running blacklist test.
See nftfw/How_do_I.md at master · pcollinson/nftfw · GitHub.
I’ll see what can be done to make this more resiliant.
I should also say that it would be a great help to have the offending file, so I can prove it’s fixed.
You significantly underestimate my cluelessness. I’d not have the first idea of how to create or change a pattern file.
The document you refer to says:
You can use the *nftfw* command to see if your expression works. Create a new pattern file and set
ports = test
and add your regular expression. Now say:
$ sudo nftfw -x -p PATTERN blacklist
but I don’t know where to “Create a new pattern file”
or how to set “ports = test”
I’m sure I could send you the offending file, if I knew which it was
OK. No problem.
I’m looking.
Version v0.7.7 should fix this problem.
After a few days of being somewhere with almost no internet connectivity, I see that the error message had stopped, but I’ve now been able to update to 0.7.7
Probably the offending line moved out of the logs being scanned. 0.7.7 now won’t have problems with decoding ‘bad’ unicode.
Ah, found a problem. Updated nftfw to see if that fixed the problem. But I updated one server, and the problem was on the other server. BOTH now updated!
Well on examination of my change in v0.7.7, it seems that probably it wasn’t a fix for your original problem - which was bad UTF-8 coding in a log file. This time I found a UTF-8 test file stuffed with coding errors to confirm the fix is correct.
Recommend updating to v0.7.10 so it shouldn’t happen again.
Oh, I had properly forgotten toupdatethat server…it was on 0.6.0 I think.
v0.7.10 was installed this am (12 May) so I would recommend that you upgrade to it.
I am working towards getting Debian packaging created for nftfw 1.0. Getting it into the debian release is likely to be a longer journey, and sorting out how to get the binary package out is on my list to do. Anyway, I am making some simplifications (and error detection) for a broader user base.
For example, nftfw post 0.7.14 will use the owner/group of the etc/nftfw file to set file ownership when writing files - losing the user setting from the config.ini - and asking it at install time. So once it’s live: chown -R on the directory will just work, no need to tell nftfw.
I want to insist that all control files are in /usr/local/etc/nftfw (or /etc/nftfw in the package) losing the nftfw_base config setting - and the ability to use extant /etc/sympl/firewall files. It’s probably better not to do this anyway, it won’t work with home grown rules for example. I used this when I was bootstrapping, but not since.
I can provide an import script that will setup the nftfw files from the current /etc/S*/firewall files, this will have to flag any rules that are not supported. I doubt that many people have written their own rules, and if they have, they should be capable of creating replacements. But a live conversion script will work in most cases, so seems a good idea.
Can I have some thoughts on this?
a) OK to lose ability to access /etc/sympl/firewall?
b) Do I need to write an import script?
I have never really understood the complexities of all this. There seem to be setups and rules all over the place. My suggestion is that you should keep it as simple as possible.
That’s not just simple for users, but also simple in that everything should be in one place.
However, what does that mean?
Would you end up with firewall rules in /etc/sympl/firewall that are ignored, because only /usr/local/etc/nftfw ones are used?
If your default rules are the same as (or equivalent to) sympl default rules then that means only rules that someone has added would need to be imported.
So why not not just search for non-standard sympl rules and if any are found, alert the user of the need to set up their special rules in nftfw?
The new package will install with files in /etc/nftfw - because that is the Debian way. So it’s all in one place. Once /etc/nftfw exists, nftfw will automatically switch to /etc/nftfw and /var/lib/nftfw in preference to files under /usr/local/. This has been there since the start.
Everything in /etc/sympl/firewall will be ignored. Sympl rules are written in Ruby and nftfw rules are small shell scripts. Automatic conversion is not easy and I’ve avoided doing that, it’s possibly an AI task. However, nftfw has mirrored all the Sympl rules.
There is however a new python script in the distribution called import_tool which is standalone and will scan for all settings in /etc/sympl/firewall and install them in /etc/nftfw (or /usr/local/etc/nftfw if that exists and the /etc/ file doesn’t). It will ignore the myriad Sympl aliases for rules and will rename files in incoming.d if needed.
It will also flag local rules that need porting into nftfw and will also ignore various rules from sympl that are not needed for nftfw. It tells you what it’s doing and why - you have to tell it to write files. So it’s safe to run, if you want to look at what will happen. Look at the README file, and run the import_tool with no arguments to get a bunch of how to do stuff information. Oh - the help assumes that it’s installing in /etc/nftfw, but it will update things in /usr/local/etc/nftfw. The first thing it says is where it’s found the source and destination.
There is also a new Uninstall.sh script. It can remove a manually installed version. It assumes that all control files are in /usr/local/etc/nftfw. It also asks what to do, so you can retain control files and just move them into /etc and /var/lib before installing the new package. Assuming that the control files are in /usr/local/etc/nftfw and rules, config.ini and nftfw_init.nft match - then it should be a simple change to switch. This will be tested.
The new stuff is actually on GitHub now - version v0.8.1. The new version implies small changes to the APIs. See the Changelog file for what’s altered. It has the changes to config.ini - removing the Owner section and the pointer to /etc/sympl/firewall. There is also a change to nftfw_init.nft - to move the essential ipv6 rules into the template, because I recently found that they were needed there to allow selective blocking of IPv6 services while allowing the local IPv6 essential packets to get through. There are changes to files in rule.d which will need installing, mostly to make them no-ops.
There are slight changes to the Install.sh script that will install updated rules into rule.d - and will create local.d which is the place for local rules. It will also comment on things that need updating by hand. The rule.d directory will now be updated on new releases.
Finally, installing the Debian package is so much easier than the manual installation. I’m putting the final touches to it, and when ready it will be version 1.0.0 or thereabouts of nftfw.
The Debian package for installing and updating nftfw is now online at the GitHub site. The source code contains the debian directory for building the package, and binary versions of the package for installation. Updating the installation is considerably simpler with the package. I think that this is now my preferred way of updating things.
There is a document Install nftfw from the Debian Package aimed at installing the package, which starts with installation on a vanilla debian system. There are then links to more complex installation scenarios. The source distribution includes a new Uninstall.sh script, and both the source and the binary package include import_to_nftfw.py, a script to process and install settings from the /etc/sympl/firewall directory.
If you use this do check that your config.ini and nftfw_init.nft are updated. Use diff to see what the differences between your versions and the distributed ones.
I’ve submitted the package to Debian for inclusion. However, if you have experience with Debian packaging and can spare the time to look at what I’ve done, then I’d be grateful for any assistance and comments you may have.
It seems that Maxmind has changed the way they are zipping their databases and the getgeocountry script is not functioning properly. You can get a new copy of the script by pulling the source from the GitHub site either by a git pull - or downloading the latest Debian package if you’ve upgraded.
To fix things: go to /var/lib/GeoIP and install the new script. Use ls -l to see where GeoLite2-Country-CSV is pointing. It should be pointing at a directory, and not into a directory. If it’s pointing to a file, then
$ sudo rm GeoLite2-Country-CSV
$ sudo ln -s <MOST RECENT> GeoLite2-Country-CSV
Then run getgeocountry to make sure things work. Then you can delete all but the most recent directory.
Nftfw is now on release 0.9.12 - Maxmind changed their download format again… and I had a request to include fail2ban support which I’ve done. Then some bugs appeared and were fixed.
What is the state of play with this now? I’ve moved one server to Mythic Beasts and it is on vanilla Sympl 11.
The release is stable. 0.9.13 made one change that I nicked from @Kelduum which changes IPv6 behaviour.
Downloading the latest package from the latest GitHub package
and installing it should work.
Reading through the description, it is still slightly ambiguous to a newcomer. For an example of what I mean, here’s a paragraph:
When installing nftfw , you will be asked if you want to change the ownership of the /etc/nftfw directory to allow configuration by a non-root user. When nftfw writes files under the directory it will take the ownership from the owner of /etc/nftfw . Debian’s debconf is used to remember this setting for later updates, and you can change ownership after installation using:
You can stick your hand into a mangle. The question is whether or not you should.
In my opinion, the docs should offer advice on how to answer a question, not just say that there is a question. If the appropriate answer depends on unknown factors, list them and give decision advice. If there is a “right” answer, say what it is.