Improving the validation process

[DavidIQ]

Hello Extension Writers,

As you know phpBB has a long history of having user code provided in MODs and now Extensions validated by a team (MOD Team and now Extensions Team) before it gets published on our site. That process hasn't changed a whole lot, but maybe that's because feedback on how it's going is not easily shared by you or is able to be seen by our team. This topic is so that you, the Extension Writers, can share your feedback on the validation process and share ideas that you think can improve the process.

Thank you in advance for your feedback.

[Ger]

Well, main points would be: first up, provide a complete editorconfig file that enforces the coding standards. That would probably help the second point: the time between submitting and validation. If there isn't any fuzz about newlines, spaces, braces, etc validation would be much faster I would think

[DavidIQ]

For many years now (10+?) anything regarding spacing or coding guidelines issues has been mostly informational and, for a while now, the validators have made this even more clear by marking things as "Warning". So I assume that by "fuzz" you mean that this can be seen as noise that can detract attention from the items marked as "Fail"? That's certainly a fair point that can be remediated by providing an editorconfig file for phpStorm and maybe one for VSCode as well.

We have this one in the main phpBB repo that can be a basis for an extensions one, at least where phpStorm is concerned.

[Ger]

Well, it's detracting to say the least. And like I wrote before, my last submission almost 3 years ago seems to be denied solely for formatting reasons. I've re-read it just now, I don't see anything that fits in another category.
Now I don't want to get into details about this extension per se, but if policy is that such points shouldn't result in denying extensions or even be relevant for the validation result, it should be much, much clearer.

I've looked to all my earlier submissions, almost every time there were mostly code style issues.

[DavidIQ]

Ok let's take a look at your last denied extension. There were some validator comments about dead code still being present, which isn't against coding guidelines, but hinders the validation process and is just a bit messy tbh. As a developer that has to review other developer's merge requests, I wouldn't merge code in that state. There is also an invalid json format issue with composer.json as well as an issue reported with the uninstall process.

I can see though why you would think that the other comments about where commas were placed or there being extra spaces or lines are part of the denial reason. The format of reports were already adjusted in order to make it clear what is and isn't a denial reason after some feedback was received. In the case of this denial the issue with the uninstall could have been made clearer as well so we could also improve on that.

[david63]

The number one problem with submitting extensions is the length of time it takes to get them validated - and yes, of course, I know that everyone is a volunteer and they that have other areas of their lives that take precedence. But this all comes back to one of my other points that phpBB in general is not building for the future by involving more members in this type of activity. At one time, in the days of MODS, there were at least 10, if not more, actively involved in the validation process - now there are two or at best three. At this current time there is at least one extension that has been in the validation queue for almost three months, which in my opinion is totally unacceptable - extensions should, again in my opinion, be validated/denied in no more than a month. It would also help if extension developers knew where their extensions were in the queue.

Next there are the inconsistencies with validations where in one submission the code is fine but on another submission code that has already passed validation is denied. I have also known of extensions being denied when code that has been copied from the core is considered to be incorrect!

Why can there not be a "fast track" for minor amendments? An extension may literally only require a single character change but the whole extension has to go through the full validation process which can take a month or more. There are quite a few extension developers who can be "trusted" with minor amendments.

Next why can extensions not be pre validated for when a new version of phpBB is released? Currently an extension cannot be submitted until the relevant phpBB version is released yet where there is a significant change users have to wait for months before they can get validated updated extensions.

By keeping extensions out of the validation system extension developers have more freedom to update their extensions quickly - if there is a bug in one of my extensions then outside of the validation process I can fix that bug and release a new version sometimes within minutes rather than months,

The bottom line is that the whole process is too slow and needs a radical overhaul but there are some who are living in the past and want to continue in the old ways.

As an aside, but somewhat relevant, when are phpBB's coding guidelines going to be updated to use industry standards?

[oxpus]

As an aside, but somewhat relevant, when are phpBB's coding guidelines going to be updated to use industry standards?

I can only agree.

I find a full validation of the extension nonsensical if something has only been changed in a few places on an update.
When submitting a new version, it could determine what was changed and only these files should be checked.
This is also extremely important with regard on possible security risks to distribute a new version immediately!

A complete and freely accessible test environment should also be provided for pre-validation.
For me, EPV just covers a tiny part, the test suite for Git Actions is systematically flawed in some points (e. g. line breaks) and also incomplete.
In addition, not every developer uses GitHub.

In my opinion, a better and freely accessible test environment can accelerate the validation process enormously if the results are reliable and these are also used in the validation process.

[DavidIQ]

As an aside, but somewhat relevant, when are phpBB's coding guidelines going to be updated to use industry standards?

What standards are these exactly that are outdated?

In my opinion, a better and freely accessible test environment can accelerate the validation process enormously if the results are reliable and these are also used in the validation process.

Quick Install already exists to help with setting up a phpBB environment quickly. Other than that I can't envision how some system that takes an extension and somehow tests it would exist.

[rxu]

Whatever the denial reason is, the main factor is reviewing speed.
As per the current validation stats, the current unvalidated revisions have spent an average of 56 days in the queue, the average validation time for revisions validated in the past year is 41 days, the oldest unvalidated revision was submitted 83 days ago.
So, just for example, after "41 days" of waiting you get your ext denied for some reason which you need a couple of minutes/hours to fix, then you resubmit the ext and have to wait another "41 days". To the end of validation process the revision submitted can be outdated a lot in comparison with the extension's current development state.
It seems that extensions validation is a PITA not just for extension authors but for extensions validators as well. The longer every ext takes to be reviewed the longer overall queue you get, so it looks like the process has to be simplified somehow and restricted to a several (3-5) simple steps for extensions validator to get any ext reviewed (validated or denied) for 2-3 days.
Also to speed it up, it would probably make sense to consider new validation type like "Validated with known issues" where validator might consider extension validated with the list of known issues posted on extension support page (possibly even in new dedicated tab "Known issues"). That could work when those issues aren't critical, don't break anything and can be fixed in the next extension release.

[Ger]

As an aside, but somewhat relevant, when are phpBB's coding guidelines going to be updated to use industry standards?

What standards are these exactly that are outdated?

https://www.php-fig.org/psr/
Has been around since 2009 and is considered industry standard for many years now.

[oxpus]

Quick Install already exists to help with setting up a phpBB environment quickly.

I wasn't talking about installing a phpBB for testing purposes.
Every developer should do that.

But my suggestion relates to a more extensive test suite for extensions, with which developers could run more automatic tests and not only to check the phpBB Coding Standard (e. g. using EPV).

A lot of points can be tested automatically in an extension without having to use an expensive development environment like phpStorm.
For example CodeSniffer can be used in a more extensive test suite like it is currently done by EPV to find significantly more errors in extensions than in a manual validation process and these rules can also provide reliable results in a few seconds or minutes, too:

• PHP_CodeSniffer (https://www.squiz.net)
• PHPCompatibility (https://github.com/PHPCompatibility)
• PHP_CodeSniffer VariableAnalysis (https://github.com/sirbrillig)

And at least using rules like the phpBB PHP Strict Standard Extensions (https://github.com/phpbb) and the phpBB 3.3 YAML check by LukeWCS.

I would just suggest a simple but extensive test suite for extensions that can be used by any developer without having to spend a lot of money on software or having to set up their own extensive test environments.

[MattF]

The number one problem with submitting extensions is the length of time it takes to get them validated - and yes, of course, I know that everyone is a volunteer and they that have other areas of their lives that take precedence. But this all comes back to one of my other points that phpBB in general is not building for the future by involving more members in this type of activity. At one time, in the days of MODS, there were at least 10, if not more, activly involved in the validation process - now there are two or at best three. At this current time there is at least one extension that has been in the validaion queue for almost three months, which in my opinion is totally unacceptable - extensions shoared, again in my opinion, be validated/denied in no more than a month. It would also help if extension developers knew where their extensions were in the queue.

I agree with you, and I'm sure every one our team does as well. However, this is no longer the days of MODS, and phpBB - forums actually - are so much more irrelevant these days now that users have Reddit, Twitter, YouTube, Discord and all the other platforms for immediate and more live and interactive communities. PHP as a programming language, while still the basis of much of the web, is also no longer as relevant as it was in the days of MODS, with so many new developers jumping immediately into JS based language and platforms like React to build content for the web.

All that is to say, our teams are stretched very thin these days. But our doors are also always open. Anyone can volunteer to join us, provided they meet our qualifications. And we are always on the look out for people to recruit. Unfortunately the pond hasn't many fish left in it here who are willing to be a part of the effort, and we are trying to make do the best we can with who and what we've got and the time they have to devote to this project.

I find a full validation of the extension nonsensical if something has only been changed in a few places on an update.
When submitting a new version, it could determine what was changed and only these files should be checked.
This is also extremely important with regard on possible security risks to distribute a new version immediately!

Normally this is what happens. When an extension is updated, generally the changes are reviewed. Of course I'm sure some validators may skim over other bits of code or find something they flag as an issue that a previous validation may have missed (we're only humans). What is, however, fully tested during each submission is the functionality of the extension - that is works and does everything it claims to do, without bugs - by the Jr. Extension Validators.

I guess what I'm trying to say is, for those who think the process takes too long, consider joining us. Help us fix this problem by becoming part of its solution.

[MattF]

Quick Install already exists to help with setting up a phpBB environment quickly.

A lot of points can be tested automatically in an extension without having to use an expensive development environment like phpStorm.
For example CodeSniffer can be used in a more extensive test suite like it is currently done by EPV to find significantly more errors in extensions than in a manual validation process and these rules can also provide reliable results in a few seconds or minutes, too:

• PHP_CodeSniffer (https://www.squiz.net)
• PHPCompatibility (https://github.com/PHPCompatibility)
• PHP_CodeSniffer VariableAnalysis (https://github.com/sirbrillig)

phpBB has already, through Github, provided an extensive test suite that while meant for phpBB developers, can be harnessed easily by extension developers, for free.

Every extension I write for myself (and our official phpBB one's) is tested on GitHub with:

• Unit testing to make sure every function performs as expected
• Functional testing to make sure the board performs as expected
• Code sniffer is run (courtesy of phpBB and using phpBB's ruleset)
• EPV is run, to make sure Validation rules are not violated
• ICC image profiler is run to make sure images are clean
• Executable files are checked to make sure there are none
• Additional integrations with Scrutinizer or Code Climate include many more PHP, CSS, and JS fixers and checkers too.

And these tests are run in Ubuntu, Windows environments on multiple versions of PHP 7 & 8, MySQL, MariaDB, Postgresql, SQLite3, MSSQL, etc. And in this forum we provide the config file for your extension to harness all these Github Actions.

You can't really test it any more than that.

Also, if you can't afford PHPStorm, you can use VSCode which is free, and full of plugins that will help clean up mess, like removing empty spaces and newlines, using tabs over spaces, etc and integrating with dozens of code validators/sniffers. We have a topic in this forum listing dozens of free code and IDE platforms.

As DavidIQ said, I'm not really sure what more you think we can do for you than that, but we are not going to try to build some kind of A.I. super smart extension tester validator robot with all the tools we already provide and use as mentioned above.

[david63]

So far in this discussion the overall consensus is that the validation process is too slow.

As I see it there are only two options:

1. Recruit more members onto the team, but

   Unfortunately the pond hasn't many fish left in it here who are willing to be a part of the effort

   and I suspect that those that are left "in the pond" have either been approached and declined or are too disenfranchised.
2. Change the validation process.
   
   This option has, in my opinion, two ways to go:
   1. Change the current process.
      
      Currently there are two ways in which the process could be speeded up. The first is that instead of waiting for complete validation (extension testing and code validation) if an extension fails the first (whichever that may be) then the extension is denied - this saves time in waiting another few weeks/months. I know that the arguments against this are that overall it makes the process quicker but I do not believe that to be the case, certainly not as things stand currently.
      
      The second change would be rather than deal with extensions in the order in which they are submitted any small quick to validate extensions could be dealt with first and the bigger ones validated later with a proviso on the team that they cannot be left in the queue longer than x days.
   2. Have a radical overhaul of the whole validation process, which to be honest is basically the same process that has been around for the last 15+ years.
      
      My suggestion here would be to effectively discontinue the validation process for all but "official" extensions and allow extension developers if they wish to apply for their extension to be "official" and released under the phpbb vendor. I accept that there might be an issue with ongoing support with this idea.
      
      I would then lock the Extensions in Development forum and create a new forum "Extensions Available" where an extension developer can have a one post topic the details the extension and a link to where it is supported - be that their own site, a communal site or GitHub - which is, in effect, what is happening now with many extensions. The forum would of course have an appropriate disclaimer about all such extensions

There may some other ideas but as the current system is not working, at least not working to the benefit of extension developers, then something has to change - sooner rather than later.

[3Di]

So what have we come to? Which points could be useful and which less so? The purpose of this topic is clear but I don't see any progress. Shall we make an ordered list and summarise the highlights whether they are positive or negative for the main cause referred to in the subject?

I personally have my reasons and they are few, I had already expressed them in May 2021 on discord but they went unnoticed, so I'm waiting to see the progress before expressing myself again. The one month queue has never scared me, as an insider I can say it fits. Of course the revisions can't be at the same pace.