Testing ecj<->javac compatibility

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Testing ecj<->javac compatibility

Stephan Herrmann-2
Hi Team,

Here's a New Year's Resolution of mine: "Always run all tests!" :)

Background:
When recently I worked on a bug about a difference between ecj & javac,
I had a patch that would cause a number of regressions in our suite,
but I was quite uncertain, whether the 'broken' tests actually expected
the 'correct' outcome. At that point I remembered, that all our compiler
tests can be run in a mode to compare against javac, but I'm afraid this
mode has seen a lot of bit rot over the years.

As a first counter measure and to make that test mode useful again,
I created a new Jenkins job [1], which currently runs once per week
(on a weekend night). I plan to add a similar job for Java 9, too.

The first run of this job (after configuration issues) was quite
devastating: close to 1000 failures.

While inching along fixing the most obvious cases, where ecj has
more diagnostics than javac (null annotations, optional warnings
configured as error etc.) I once again struggled with the overload
overkill in AbstractRegressionTest and 'invented' a style which I
hope to be better usable and maintainable, for details see [2] and
if you have any comments regarding the new style please drop them in
the bug (I know this style may look like a riddle on first glance :)).
For existing tests, this style facilitates adding a JavacTestOption
argument with minimal code changes (for minimal impact on blame history).
For new tests I'm planning to completely switch to this style
(unless s.o. raises any objections) (with somewhat more conventional
formatting :) ).

With technical issues out of the way, the *run.javac-1.8 job is
currently at 281 failures and I would greatly appreciate if we could
split this round of clean-up among the team. My idea is that in this
round we only address obvious "Excuses" and I expect that after this
we have a residuum of some tens of tests that perhaps signal relevant
differences between ecj & javac - to be addressed in a second round.

Finally, when/if we reach the state where these builds are error-free,
I suggest that everybody working on the compiler enter their email
in the job configuration (or ask me to do so) to be notified of any
failures.

let me know what you think,
Stephan


[1] https://ci.eclipse.org/jdt/job/eclipse.jdt.core-run.javac-1.8/
[2] https://bugs.eclipse.org/bugs/show_bug.cgi?id=404648#c45
_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testing ecj<->javac compatibility

Daniel Megert
Hi Stephan

Great work and thanks for doing this!

 The approach looks good to me. JDT Core team, please help Stephan to get the failure count down.

Dani



From:        Stephan Herrmann <[hidden email]>
To:        Eclipse JDT Core developers list <[hidden email]>
Date:        07.01.2018 12:35
Subject:        [jdt-core-dev] Testing ecj<->javac compatibility
Sent by:        [hidden email]




Hi Team,

Here's a New Year's Resolution of mine: "Always run all tests!" :)

Background:
When recently I worked on a bug about a difference between ecj & javac,
I had a patch that would cause a number of regressions in our suite,
but I was quite uncertain, whether the 'broken' tests actually expected
the 'correct' outcome. At that point I remembered, that all our compiler
tests can be run in a mode to compare against javac, but I'm afraid this
mode has seen a lot of bit rot over the years.

As a first counter measure and to make that test mode useful again,
I created a new Jenkins job [1], which currently runs once per week
(on a weekend night). I plan to add a similar job for Java 9, too.

The first run of this job (after configuration issues) was quite
devastating: close to 1000 failures.

While inching along fixing the most obvious cases, where ecj has
more diagnostics than javac (null annotations, optional warnings
configured as error etc.) I once again struggled with the overload
overkill in AbstractRegressionTest and 'invented' a style which I
hope to be better usable and maintainable, for details see [2] and
if you have any comments regarding the new style please drop them in
the bug (I know this style may look like a riddle on first glance :)).
For existing tests, this style facilitates adding a JavacTestOption
argument with minimal code changes (for minimal impact on blame history).
For new tests I'm planning to completely switch to this style
(unless s.o. raises any objections) (with somewhat more conventional
formatting :) ).

With technical issues out of the way, the *run.javac-1.8 job is
currently at 281 failures and I would greatly appreciate if we could
split this round of clean-up among the team. My idea is that in this
round we only address obvious "Excuses" and I expect that after this
we have a residuum of some tens of tests that perhaps signal relevant
differences between ecj & javac - to be addressed in a second round.

Finally, when/if we reach the state where these builds are error-free,
I suggest that everybody working on the compiler enter their email
in the job configuration (or ask me to do so) to be notified of any
failures.

let me know what you think,
Stephan


[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.eclipse.org_jdt_job_eclipse.jdt.core-2Drun.javac-2D1.8_&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=1UITCR5rxUZHSFczvfaNFK4ymEbEiccRX7VKchpqz0Y&m=4Kj8AcrlvGg1LN0vYgJCT03EmPFI8nnSRUufQfSE6ic&s=Q6RM6f4okcKi-qxABIxR2Nqd8rA31-iIyFxmu4M-vAQ&e=
[2]
https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.eclipse.org_bugs_show-5Fbug.cgi-3Fid-3D404648-23c45&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=1UITCR5rxUZHSFczvfaNFK4ymEbEiccRX7VKchpqz0Y&m=4Kj8AcrlvGg1LN0vYgJCT03EmPFI8nnSRUufQfSE6ic&s=Eyqvbb4SCh1B2OxOVqIT3L-HLxT6S5Q_cT98XN8zamk&e=
_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jdt-2Dcore-2Ddev&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=1UITCR5rxUZHSFczvfaNFK4ymEbEiccRX7VKchpqz0Y&m=4Kj8AcrlvGg1LN0vYgJCT03EmPFI8nnSRUufQfSE6ic&s=tBgF42vOyQm9pSQZsR7QVQOJVxcvYG-0Qgnq82neB2c&e=





_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testing ecj<->javac compatibility

Sasikanth B Ammanamanchi
In reply to this post by Stephan Herrmann-2
Thanks Stephan, great work.
I will start looking into lambda expression related test failures if you have not already looked at those (LET, NLET and SLT) and proceed from there
Let me know if you'd rather I looked into some other failures first

Regards
Sasikanth

[hidden email] wrote: -----
To: Eclipse JDT Core developers list <[hidden email]>
From: Stephan Herrmann
Sent by: [hidden email]
Date: 01/07/2018 05:05PM
Subject: [jdt-core-dev] Testing ecj<->javac compatibility

Hi Team,

Here's a New Year's Resolution of mine: "Always run all tests!" :)

Background:
When recently I worked on a bug about a difference between ecj & javac,
I had a patch that would cause a number of regressions in our suite,
but I was quite uncertain, whether the 'broken' tests actually expected
the 'correct' outcome. At that point I remembered, that all our compiler
tests can be run in a mode to compare against javac, but I'm afraid this
mode has seen a lot of bit rot over the years.

As a first counter measure and to make that test mode useful again,
I created a new Jenkins job [1], which currently runs once per week
(on a weekend night). I plan to add a similar job for Java 9, too.

The first run of this job (after configuration issues) was quite
devastating: close to 1000 failures.

While inching along fixing the most obvious cases, where ecj has
more diagnostics than javac (null annotations, optional warnings
configured as error etc.) I once again struggled with the overload
overkill in AbstractRegressionTest and 'invented' a style which I
hope to be better usable and maintainable, for details see [2] and
if you have any comments regarding the new style please drop them in
the bug (I know this style may look like a riddle on first glance :)).
For existing tests, this style facilitates adding a JavacTestOption
argument with minimal code changes (for minimal impact on blame history).
For new tests I'm planning to completely switch to this style
(unless s.o. raises any objections) (with somewhat more conventional
formatting :) ).

With technical issues out of the way, the *run.javac-1.8 job is
currently at 281 failures and I would greatly appreciate if we could
split this round of clean-up among the team. My idea is that in this
round we only address obvious "Excuses" and I expect that after this
we have a residuum of some tens of tests that perhaps signal relevant
differences between ecj & javac - to be addressed in a second round.

Finally, when/if we reach the state where these builds are error-free,
I suggest that everybody working on the compiler enter their email
in the job configuration (or ask me to do so) to be notified of any
failures.

let me know what you think,
Stephan


[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.eclipse.org_jdt_job_eclipse.jdt.core-2Drun.javac-2D1.8_&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=o963LG-v40yVK_FFyn8Arh6uH3v-wImwnF-gopa6FwA&m=y6THPytyUXgGekuEsnRkmXV2cZ8MF47MngY3ZRuBS3A&s=O7I48fWV1cpZBJ-f1uV2dj5WiuA1Gaw6mMaHBfE6ovA&e=
[2] https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.eclipse.org_bugs_show-5Fbug.cgi-3Fid-3D404648-23c45&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=o963LG-v40yVK_FFyn8Arh6uH3v-wImwnF-gopa6FwA&m=y6THPytyUXgGekuEsnRkmXV2cZ8MF47MngY3ZRuBS3A&s=tSs1oLy2nA-k7TaCNGmCaHsk2ihUQzze_qEOL-V2KV4&e=
_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jdt-2Dcore-2Ddev&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=o963LG-v40yVK_FFyn8Arh6uH3v-wImwnF-gopa6FwA&m=y6THPytyUXgGekuEsnRkmXV2cZ8MF47MngY3ZRuBS3A&s=lhzfZf9mR9U3Dee0Li4sbWZO0uJG8bw98dq1t78pa48&e=



_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
Reply | Threaded
Open this post in threaded view
|

Re: Testing ecj<->javac compatibility

Stephan Herrmann-2

I am not currently working on any of the tests you mention,
so your contribution there is much appreciated!

For the future I suggest, whoever picks up a suite for clean-up
just drops a comment into bug 404648 for coordination.
I will do the same in order to avoid duplicate work.

thanks,
Stephan

Am 2018-01-22 10:00, schrieb Sasikanth B Ammanamanchi:

Thanks Stephan, great work.
I will start looking into lambda expression related test failures if you have not already looked at those (LET, NLET and SLT) and proceed from there
Let me know if you'd rather I looked into some other failures first

Regards
Sasikanth

[hidden email] wrote: -----
To: Eclipse JDT Core developers list <[hidden email]>
From: Stephan Herrmann
Sent by: [hidden email]
Date: 01/07/2018 05:05PM
Subject: [jdt-core-dev] Testing ecj<->javac compatibility

Hi Team,

Here's a New Year's Resolution of mine: "Always run all tests!" :)

Background:
When recently I worked on a bug about a difference between ecj & javac,
I had a patch that would cause a number of regressions in our suite,
but I was quite uncertain, whether the 'broken' tests actually expected
the 'correct' outcome. At that point I remembered, that all our compiler
tests can be run in a mode to compare against javac, but I'm afraid this
mode has seen a lot of bit rot over the years.

As a first counter measure and to make that test mode useful again,
I created a new Jenkins job [1], which currently runs once per week
(on a weekend night). I plan to add a similar job for Java 9, too.

The first run of this job (after configuration issues) was quite
devastating: close to 1000 failures.

While inching along fixing the most obvious cases, where ecj has
more diagnostics than javac (null annotations, optional warnings
configured as error etc.) I once again struggled with the overload
overkill in AbstractRegressionTest and 'invented' a style which I
hope to be better usable and maintainable, for details see [2] and
if you have any comments regarding the new style please drop them in
the bug (I know this style may look like a riddle on first glance :)).
For existing tests, this style facilitates adding a JavacTestOption
argument with minimal code changes (for minimal impact on blame history).
For new tests I'm planning to completely switch to this style
(unless s.o. raises any objections) (with somewhat more conventional
formatting :) ).

With technical issues out of the way, the *run.javac-1.8 job is
currently at 281 failures and I would greatly appreciate if we could
split this round of clean-up among the team. My idea is that in this
round we only address obvious "Excuses" and I expect that after this
we have a residuum of some tens of tests that perhaps signal relevant
differences between ecj & javac - to be addressed in a second round.

Finally, when/if we reach the state where these builds are error-free,
I suggest that everybody working on the compiler enter their email
in the job configuration (or ask me to do so) to be notified of any
failures.

let me know what you think,
Stephan


[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.eclipse.org_jdt_job_eclipse.jdt.core-2Drun.javac-2D1.8_&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=o963LG-v40yVK_FFyn8Arh6uH3v-wImwnF-gopa6FwA&m=y6THPytyUXgGekuEsnRkmXV2cZ8MF47MngY3ZRuBS3A&s=O7I48fWV1cpZBJ-f1uV2dj5WiuA1Gaw6mMaHBfE6ovA&e=
[2] https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.eclipse.org_bugs_show-5Fbug.cgi-3Fid-3D404648-23c45&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=o963LG-v40yVK_FFyn8Arh6uH3v-wImwnF-gopa6FwA&m=y6THPytyUXgGekuEsnRkmXV2cZ8MF47MngY3ZRuBS3A&s=tSs1oLy2nA-k7TaCNGmCaHsk2ihUQzze_qEOL-V2KV4&e=
_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jdt-2Dcore-2Ddev&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=o963LG-v40yVK_FFyn8Arh6uH3v-wImwnF-gopa6FwA&m=y6THPytyUXgGekuEsnRkmXV2cZ8MF47MngY3ZRuBS3A&s=lhzfZf9mR9U3Dee0Li4sbWZO0uJG8bw98dq1t78pa48&e=



_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev



_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev