RFR (S) 8007434: Write tests for 8006298

[David Chase]

On 2013-02-06, at 6:54 PM, Coleen Phillimore <coleen.phillimore at oracle.com> wrote:
> If we change the error output even slightly though, we'll have to change the tests.  Maybe this is okay.


Coleen,

Depends on the nature of the error messages.
If they have path names embedded in them, we might need a way to introduce variables into the pattern-match.
For example (real example, test excerpts from Project Fortress):

STATIC_TESTS_DIR=${FORTRESS_AUTOHOME}/ProjectFortress/compiler_tests
...
compile_err_equals=\
${STATIC_TESTS_DIR}/Compiled0.a.fss:11:11-15:\n\
\ Component/API names must match their enclosing file names.\n\
\ File name: ${STATIC_TESTS_DIR}/Compiled0.a.fss\n\
\ Component/API name: Hello\n\
Turn on \"-debug interpreter\" for Java-level stack trace.\n
compile_out_equals=

That system was all driven by key-value bindings, which meant the verb was embedded in the name.
In practice the only verbs that mattered were "equals" and "contains", optionally whitespace insensitive.
I don't much like the embedded verb approach anymore, and it was my idea, so it's probably a real stinker.
I was leveraging an existing config-file mechanism that handled all the variables nicely.

We also supported "matches" (regular expression) and whitespace-and-case-insensitive modifiers,
but those were not used much.

And sometimes we did change error output, and it was a major headache, but what are you supposed to do?

In the category of "learn from our mistakes":

1) You need to get serious about canonicalizing path names, meaning that they need to actually be "canonical".
Otherwise, the test harness and the compiler may disagree, or different parts of the compiler may disagree with each other.

2) Avoid use of non-ordered collection classes (that is, hash tables/maps) to store things for processing, because that can affect your output order, else you'll need to collect error messages and standardize, sometimes including the internal parts of error messages.

For example, these messages have to come out in the same order, every time:

compile_err_equals=\
${STATIC_TESTS_DIR}/Compiled6.at.fss:24:5-8:\n\
\ O has no getter called foo\n\
${STATIC_TESTS_DIR}/Compiled6.at.fss:25:5-12:\n\
\ Could not check call to operator +\n\
\ - (IntLiteral, IntLiteral)->IntLiteral is not applicable to an argument of type (String, IntLiteral).\n\
\ - (NN32, NN32)->NN32 is not applicable to an argument of type (String, IntLiteral).\n\
\ - (NN64, NN64)->NN64 is not applicable to an argument of type (String, IntLiteral).\n\
\ - (RR64, RR64)->RR64 is not applicable to an argument of type (String, IntLiteral).\n\
\ - (ZZ, ZZ)->ZZ is not applicable to an argument of type (String, IntLiteral).\n\
\ - (ZZ32, ZZ32)->ZZ32 is not applicable to an argument of type (String, IntLiteral).\n\
\ - (ZZ32Vector, ZZ32)->ZZ32Vector is not applicable to an argument of type (String, IntLiteral).\n\
\ - (ZZ64, ZZ64)->ZZ64 is not applicable to an argument of type (String, IntLiteral).\n\
${STATIC_TESTS_DIR}/Compiled6.at.fss:26:5-21:\n\
\ O has no setter called bar\n\
${STATIC_TESTS_DIR}/Compiled6.at.fss:29:5-49:\n\
\ Could not check method invocation O._[_]:=\n\
\ - (String, ZZ32)->() is not applicable to an argument of type (IntLiteral, IntLiteral).\n\
${STATIC_TESTS_DIR}/Compiled6.at.fss:29:21-27:\n\
\ Could not assign an expression of type IntLiteral to field O.s of type String.\n\
${STATIC_TESTS_DIR}/Compiled6.at.fss:33:5-21:\n\
\ Could not check call to operator ||\n\
\ - (String, Object)->String is not applicable to an argument of type ((String, String), (IntLiteral, IntLiteral)).\n\
File Compiled6.at.fss has 6 errors.\n

David