Michael Tremer

About why we are looking for testers
by Michael Tremer, July 28, 2014

Good Monday morning to you all!

We are hoping to be able to release Core Update 80 this week. Our usual measurement is that we want so see about 1% of the systems that are in the fireinfo database who got the update installed. So we assume that there are enough testers out there who checked that everything is working alright.

Now there are more and more IPFire systems around. The user base of IPFire is constantly growing. Therefore the number that make that one percent must be growing as well – but it isn’t.

We are not getting very much feedback at all on the pre-released updates which is extremely essential for the development process. There are sometimes some suggestions that something might go wrong on some forum posts, but for me it is very hard to crawl to the forums every day and find those bits. So we would rather like to rely on the community and I am hoping that it is possible you can confirm your bugs and then open bug reports on our bugtracker.

Why don’t we do that on our own? Well, we do. We do lots of testing. However, when a developer tests his or hers own code, that does not really count. That is the reason why we have those pre-releases out there to get a huge amount of people using IPFire in various different environments – because we need them to verify that the changes we did are working in the real world and not only in the lab.

Two very simple examples from the current Core Update 80 are the DNSSEC validating DNS proxy and the new dynamic DNS updater. The new dynamic DNS updater supports over two dozens of dynamic DNS providers. No one has got an account at all of them to test the code. Some of them even cost a huge amount of money and that makes it pretty easy to understand that one person testing them all is quite infeasible. The other example is even more convincing to me: The DNSSEC resolvers rely on the upstream DNS servers from your ISP. There are some which caused a bit of trouble with DNSSEC signed zones. It is crystal clear that one person testing all DNS servers by all ISPs in the world is impossible. We do the best we can and construct the weirdest things in the lab, which does not necessarily has to have any similarity with the real world. Hence we need your help for verifying that we did a good job.

I hope that these lines help understanding why we are constantly begging for supporters. It’s not because we are lazy, but some things are just infeasible for us to do. If everyone takes a couple of minutes and checks out the things they are using we all would just need to spend a little time and we will get a great result out of it, which is a much more stable firewall distribution.

Posted: July 28, 2014 • 696 views