ECMAScript 5 Conformance Suite

Updated Wiki: Home

dfugate — Mon, 08 Aug 2011 17:57:03 GMT

A New Home for ES5Conform

The goal of the ES5Conform CodePlex project was to build a suite of tests for new features added in the 5th edition of the ECMAScript language specification. As this test suite matured, it became desirable to extend it with tests for features added in earlier versions of the ECMAScript standard that are still present in the 5th edition today. A comprehensive test suite for the 5th edition in its entirety is extremely useful for JavaScript implementers.

To achieve this goal, the tests from ES5Conform (which were focused on ECMAScript 5) have been contributed along with Google’s Sputnik tests (focused on ECMAScript 3) to Test262, a test suite supported by the ECMAScript committee. To this end, all active development on ES5Conform has been migrated to test262 and test defects are now tracked on http://bugs.ecmascript.org. Other improvements were made in the transition to test262, and the best example of this is that the entire test suite is conveniently runnable directly from the test262 website. Also worth pointing out is that development of test262 is now an actual project of the TC39 committee, and test262 resides in the ecmascript.org web domain. You can find more info about test262 on its official website, http://test262.ecmascript.org, or via the test262 mailing list - see https://mail.mozilla.org/listinfo/test262-discuss. Finally, the original version of ES5Conform's CodePlex project page has been archived here.

We thank everyone whose contributed to ES5Conform over the years, and hope you’ll bring the same passion for a standardized web to the test262 effort.

The ES5Conform Team

Updated Wiki: Documentation

dfugate — Mon, 08 Aug 2011 17:44:22 GMT

This page is deprecated and all ES5Conform tests have been contributed to ECMA's test262 project.

Project Description

The goal of this project was to develop a test suite that could be used to determine how closely a JavaScript implementation conforms to the ECMAScript, 5th Edition Language Specification. We hope that the availability of a common conformance test suite will help browser implementers achieve a very high degree of interoperability for their ECMAScript 5 implementations.

What is ECMAScript 5

"ECMAScript" is the name under which the language more commonly known as "JavaScript" is standardized. Development of the ECMAScript standard is the responsibility of Technical Committee 39 (TC39) of Ecma International. The ECMAScript standard is officially known as ECMA-262. ECMAScript 5 (or just ES5) is short hand for the "ECMA-262, 5th Edition ECMAScript Language Specification" the official name of the standard.

ECMAScript 5 was approved as an offical Ecma standard by the the Ecma General Assembly on December 3, 2009. The ECMAScript 5 Specification (PDF) is available from the Ecma International web site.

Project Coordination and Participation

The coordinators of this project represent organizations that are members of ECMA and participants in TC39. This project is being carried out in cooperation with ECMA TC39 but the actual operation of this development project is not currently an official ECMA activity. It is possible this some or all of this test suite may be formally adopted by TC39 as a formal ECMA publication.

TC39 members and other with an active interest in test ECMAScript implementations are invited to participate by running tests, identify issues, contributing tests, or becoming active developers on this project. Path submissions are welcomed from everyone.

Contributors

Microsoft contributed the original set of approximately 900 tests.

License

All files in this project are licensed using the New BSD license. See the license tab of this project for the actual license text.

Downloading and Running the Tests

Simply go to the Downloads tab, download the latest release, unzip it, and open runtests.html in the browser you want to test. If you want to run the tests in a non-browser environment you could contribute to this project by writing an appropriate test driver.
.

Test Organization and Conventions

The tests are arranged in a hierarchy that reflects the organization of the ECMAScript 5 Specification. The organization is explained here.

Updated Wiki: Home

dfugate — Mon, 08 Aug 2011 17:38:33 GMT

A New Home for ES5Conform

The goal of the ES5Conform CodePlex project was to build a suite of tests for new features added in the 5th edition of the ECMAScript language specification. As this test suite matured, it became desirable to extend it with tests for features added in earlier versions of the ECMAScript standard that are still present in the 5th edition today. A comprehensive test suite for the 5th edition in its entirety is extremely useful for JavaScript implementers.

To achieve this goal, the tests from ES5Conform (which were focused on ECMAScript 5) have been contributed along with Google’s Sputnik tests (focused on ECMAScript 3) to Test262, a test suite supported by the ECMAScript committee. To this end, all active development on ES5Conform has been migrated to test262 and test defects are now tracked on http://bugs.ecmascript.org. Other improvements were made in the transition to test262, and the best example of this is that the entire test suite is conveniently runnable directly from the test262 website. Also worth pointing out is that development of test262 is now an actual project of the TC39 committee, and test262 resides in the ecmascript.org web domain. You can find more info about test262 on its official website, http://test262.ecmascript.org, or via the test262 mailing list - see https://mail.mozilla.org/listinfo/test262-discuss.

We thank everyone whose contributed to ES5Conform over the years, and hope you’ll bring the same passion for a standardized web to the test262 effort.

The ES5Conform Team

Created Issue: 15.12.3-11-6.js - typo precodition [30361]

efreeti — Thu, 17 Mar 2011 09:35:57 GMT

I'm emulating the run of tests in my unit test system and do not take into account situation when the precondition is not defined. This test - TestCases/chapter15/15.12/15.12.3/15.12.3-11-6.js - fails and after looking at the code I see that there is a precondition function just there is the typo in the property name - 'precodition'.

Commented Issue: Test 7.8.4-1-s does not correctly check whether octal escape sequences are allowed [28578]

mmaly — Fri, 04 Feb 2011 22:36:48 GMT

The spec does not allow octal escape sequences within strict mode code. Test 7.8.4-1-s is designed to check this - however, I believe it fails to do so.

The code in question is this: eval(' "asterisk: \052"; "use strict";');

The eval function takes a single string and by the time it reaches that point, the escape sequences have been eliminated. Thus the octal escape sequence appears *outside* strict mode code (i.e. in the context of the testcase function) and thus it should be allowed.

Also debatable is whether octal escape sequences should be disallowed before the "use strict" directive is encountered. This seems like it would require the lexer to do a second pass over the code solely to resolve escape sequences.
Comments: ** Comment from web user: mmaly **

We should commit the relevant part of the patch attached to http://es5conform.codeplex.com/workitem/29141.

Commented Issue: Deleting a variable in strict mode produces a SyntaxError, not a ReferenceError (tests 11.4.1-5-1-s, 11.4.1-5-2-s, 11.4.1-5-3-s) [29084]

jwalden — Mon, 03 Jan 2011 22:29:24 GMT

The tests 11.4.1-5-1-s, 11.4.1-5-2-s and 11.4.1-5-3-s all assume that deleting a variable in strict mode produces a ReferenceError exception. Section 11.4.1 of the specification contradicts this as it specifies a SyntaxError should be thrown instead.
Comments: ** Comment from web user: jwalden **

I sent that es5-discuss mail:

https://mail.mozilla.org/pipermail/es5-discuss/2011-January/003862.html

Commented Issue: Deleting a variable in strict mode produces a SyntaxError, not a ReferenceError (tests 11.4.1-5-1-s, 11.4.1-5-2-s, 11.4.1-5-3-s) [29084]

jwalden — Mon, 03 Jan 2011 21:12:48 GMT

The colorable case involves a host object with a [[HasProperty]] hook that has side effects or may throw an exception other than a SyntaxError, then supplied as the object for a with clause, thus making the SyntaxError for deleting a name potentially not reachable if the delete expression is reached at runtime. ES6's proposed proxy specification would enable creation of such an object. The spec says, "An early error is an error that can be detected and reported prior to the evaluation of any construct in the Program containing the error." "can be detected and reported prior to..." is arguably under-defined enough to permit the host object snafu. As I said, I don't like this case, and I think it worth killing. But there is a case that deleting a name is not an early error.

Commented Issue: Tests 10.6-13-b-3-s and 10.6-13-c-3-s reference nonexistent property "put" [29141]

jwalden — Mon, 03 Jan 2011 21:10:36 GMT

The tests 10.6-13-b-3-s and 10.6-13-c-3-s call Object.getOwnPropertyDescriptor to determine whether "arguments.callee" and "arguments.caller" are configurable. However, they check for the existence of the property "put". The correct property to check for is "set".
Comments: ** Comment from web user: jwalden **

Here are SpiderMonkey's results, for what it's worth, with all my patches (all attached to bugs, most waiting to be reviewed, one almost at the ready-to-attach stage, one very hackish patch that probably needs more work):

We fail these valid tests (tho I believe there's a patch, awaiting my review, to fix all these):

FAIL: 15.4.4.14-1-1 Array.prototype.indexOf applied to undefined throws a TypeError
FAIL: 15.4.4.14-1-2 Array.prototype.indexOf applied to null throws a TypeError
FAIL: 15.5.4.20-1-2 String.prototype.trim throws TypeError when string is null
FAIL: 15.5.4.20-1-1 String.prototype.trim throws TypeError when string is undefined

These tests seem invalid to me if you don't extend ES5:

FAIL: 15.2.3.3-4-188 Object.getOwnPropertyDescriptor returns undefined for non-existent properties on built-ins (Function (instance).name)
FAIL: 15.2.3.3-4-184 Object.getOwnPropertyDescriptor returns undefined for non-existent properties on built-ins (Function.caller)
FAIL: 15.2.3.3-4-183 Object.getOwnPropertyDescriptor returns undefined for non-existent properties on built-ins (Function.arguments)

These tests are nearly-valid, but I just didn't fix them in the patch:

FAIL: 11.4.1-4.a-4-s delete operator throws TypeError when when deleting a non-configurable data property in strict mode (Global.NaN)
FAIL: 11.13.1-4-2-s simple assignment throws TypeError if LeftHandSide is a readonly property in strict mode (Global.NaN)
FAIL: 11.13.1-4-3-s simple assignment throws TypeError if LeftHandSide is a readonly property in strict mode (Global.Infinity)
FAIL: 11.13.1-4-4-s simple assignment throws TypeError if LeftHandSide is a readonly property in strict mode (Global.length)
FAIL: 11.13.1-1-7-s simple assignment throws TypeError if LeftHandSide is a property reference with a primitive base value (this is undefined)
FAIL: 11.13.1-4-27-s simple assignment throws TypeError if LeftHandSide is a readonly property in strict mode (Global.undefined)
PRECONDITION FAILED: 15.4.4.14-9.a-1 Need to define a quality of implementation test that makes sure that it doesn't take forever to process very large but very sparse arrays

These tests are the delete-name-is-early-error disagreement:

TestCases/chapter11/11.4/11.4.1/11.4.1-5-2-s.js Missing test: either misidentifed or could not load because of unexpected syntax error
TestCases/chapter11/11.4/11.4.1/11.4.1-5-3-s.js Missing test: either misidentifed or could not load because of unexpected syntax error
TestCases/chapter11/11.4/11.4.1/11.4.1-5-1-s.js Missing test: either misidentifed or could not load because of unexpected syntax error

This test probably works in a browser, but I generally don't test browser until I have a working patch for a much smaller, less hairy shell environment (plus browser takes a lot longer to build):

FAILED WITH EXCEPTION: WINDOW IS NOT DEFINED: 11.13.1-4-1 simple assignment creates property on the global object if LeftHandSide is an unresolvable reference

So aside from the two tests that expect no Function.caller/Function.arguments (this is outside the spec, which permits additional properties here and doesn't specify that these exist, as Function is not a function created through 13.something), we are in agreement as to what tests are bad and what tests are good. Or, rather, our implementations plus a little manual testing are in agreement.

Commented Issue: Tests 10.6-13-b-3-s and 10.6-13-c-3-s reference nonexistent property "put" [29141]

paulbartrum — Sun, 02 Jan 2011 13:06:11 GMT

Thanks Jeff. With your patch applied my ES5 implementation (http://jurassic.codeplex.com) fails 13 tests (down from 45). Two are known bugs with my implementation - the rest are currently outstanding issues on this issue tracker (6 issues covering 11 tests). By the way, your patch exposed a bug in my strict mode implementation, so thanks for your hard work!

In case you are interested, here are the eleven tests that I believe are still buggy (of course I am only picking up false negatives, not false positives):
11.4.1-4.a-4-s // assumes this refers to global object - http://es5conform.codeplex.com/workitem/29151
11.4.1-5-1-s // assumes delete var produces ReferenceError - http://es5conform.codeplex.com/workitem/29084
11.4.1-5-2-s // assumes delete var produces ReferenceError - http://es5conform.codeplex.com/workitem/29084
11.4.1-5-3-s // assumes delete var produces ReferenceError - http://es5conform.codeplex.com/workitem/29084
11.13.1-1-7-s // assumes this is undefined - http://es5conform.codeplex.com/workitem/29152
11.13.1-4-2-s // gets global object incorrectly - http://es5conform.codeplex.com/workitem/29087
11.13.1-4-27-s // gets global object incorrectly - http://es5conform.codeplex.com/workitem/29087
11.13.1-4-3-s // gets global object incorrectly - http://es5conform.codeplex.com/workitem/29087
11.13.1-4-4-s // gets global object incorrectly - http://es5conform.codeplex.com/workitem/29087
15.2.3.3-4-188 // assumes Function.prototype.name does not exist - http://es5conform.codeplex.com/workitem/28594
15.4.4.14-9.a-1 // placeholder test - http://es5conform.codeplex.com/workitem/29102

Commented Issue: Deleting a variable in strict mode produces a SyntaxError, not a ReferenceError (tests 11.4.1-5-1-s, 11.4.1-5-2-s, 11.4.1-5-3-s) [29084]

paulbartrum — Sun, 02 Jan 2011 06:20:56 GMT

For the record, my reasoning for believing it is an early error is as follows:
* Section 11.4.1 step 5 indicates that deleting a variable in strict mode is a SyntaxError.
* Section 16 says an implementation must treat any syntax error as an early error.
Seems pretty straightforward to me.

Commented Issue: Test 7.8.4-1-s does not correctly check whether octal escape sequences are allowed [28578]

jwalden — Sat, 01 Jan 2011 23:01:30 GMT

http://es5conform.codeplex.com/workitem/29141 has a patch that also fixes this.

I proposed a fix for your suggestion (directives must not contain escapes, ergo "\052"; "use strict"; would not enter strict mode and require reparsing, or carrying seen-octal state forward when parsing the directive prologue -- SpiderMonkey does the latter because it eagerly throws away source) to the es5-discuss list, but discussion fizzled, and eventually I became fed up on waiting for a definitive yes/no as to making that change and just implemented what the spec requires in SpiderMonkey. https://bugzilla.mozilla.org/show_bug.cgi?id=601262 and https://mail.mozilla.org/pipermail/es5-discuss/2010-October/003745.html are the relevant bug and mailing list discussion. I'd still love it if the spec eliminated the need for this pedantry and reparsing!

Commented Issue: The test 15.2.3.4-4-1 assumes a fixed list of properties on the global object [28595]

jwalden — Sat, 01 Jan 2011 22:58:34 GMT

The test 15.2.3.4-4-1 checks the property names of the global object against the following list:
["eval", "parseInt", "parseFloat", "isNaN", "isFinite", "decodeURI", "decodeURIComponent", "encodeURIComponent", "escape", "unescape", "NaN", "Infinity", "undefined"]
This test has two problems:
1. The list above is missing quite a few standard properties (e.g. "encodeURI", "Array", "JSON", "Math", etc)
2. Implementations are allowed to extend the global object and many do (including all the major browsers).
Comments: ** Comment from web user: jwalden **

I fixed the subsetting issue awhile ago, but the "incomplete properties" complaint is still valid.

Commented Issue: Array iteration methods may continue after a reference to the array has been deleted [25430]

jwalden — Sat, 01 Jan 2011 22:56:30 GMT

I think these tests are making an incorrect assertion:

15.4.4.16-7-7
15.4.4.17-7-7
15.4.4.18-7-6
15.4.4.19-8-7
15.4.4.20-9-7
15.4.4.21-9-7
15.4.4.22-9-7

In these tests the object property is being deleted, but that doesn't mean that the value itself is deleted - the method being called (e.g. every or filter) still has a reference to the array it's operating on.
Comments: ** Comment from web user: jwalden **

http://es5conform.codeplex.com/workitem/29141 has a patch that fixes this.

Commented Issue: All "stops calling callbackfn once the array is deleted during the call" tests are wrong [26872]

jwalden — Sat, 01 Jan 2011 22:54:12 GMT

These tests (15.4.4.16-7-7, 15.4.4.17-7-7, 15.4.4.18-7-6, 15.4.4.19-8-7, 15.4.4.20-9-7, 15.4.4.21-9-7, 15.4.4.22-9-7) follow the same pattern (this is reduce) :
function testcase() {
function callbackfn(prevVal, curVal, idx, obj) {
delete o.arr;
return prevVal + curVal;
}

var o = new Object;
o.arr = ["1", 2, 3, 4, 5];
if (o.arr.reduce(callbackfn) === "12") {
return true;
}
}

I think that, in this case, "delete" is misunderstood. The delete operator (ES5 11.4.1) calls the [[delete]] internal method (ES5 8.12.7) which removes from an object a property (if configurable). If this property happens to be an object, there is no reason that this object itself is deleted as long as a reference to this object is kept somewhere. In our case, the "obj" argument of callbackfn and the "this" keyword of reduce are both a reference to the array being traversed.

For that reason, I think that this these tests are wrong and should be removed from the test suite.
Comments: ** Comment from web user: jwalden **

This seems a dup of 25430, fwiw.

Commented Issue: All "stops calling callbackfn once the array is deleted during the call" tests are wrong [26872]

jwalden — Sat, 01 Jan 2011 22:52:58 GMT

http://es5conform.codeplex.com/workitem/29141 has a patch that fixes this, but by adjusting the test to pass, not by deleting it. (I have no particular objection to deleting the test, but since it's basically written, better to coopt it for a useful purpose if possible, I think, even if it's a completely different purpose from the original one.)

Commented Issue: The preconditions for tests 15.12.2-0-3 and 15.12.3-0-3 do not return true [28581]

jwalden — Sat, 01 Jan 2011 22:50:07 GMT

The preconditions for tests 15.12.2-0-3 and 15.12.3-0-3 both return a function and not a boolean.

For example, in test 15.12.2-0-3 the precondition is: "return JSON && Object.getOwnPropertyDescriptor". getOwnPropertyDescriptor is a function and therefore the whole expression returns a function. Since the test driver expects "true" the precondition fails. I believe that in this case the problem is that "fnExists" is missing and the precondition should be "return JSON && fnExists(Object.getOwnPropertyDescriptor)".
Comments: ** Comment from web user: jwalden **

http://es5conform.codeplex.com/workitem/29141 has a patch that also fixes this, although the fix is slightly different from what's suggested here.

Commented Issue: Deleting a variable in strict mode produces a SyntaxError, not a ReferenceError (tests 11.4.1-5-1-s, 11.4.1-5-2-s, 11.4.1-5-3-s) [29084]

jwalden — Sat, 01 Jan 2011 22:44:39 GMT

http://es5conform.codeplex.com/workitem/29141 has a patch that fixes many test suite issues but doesn't address this, because there's a somewhat colorable case that this is not necessarily an early error, and I haven't taken it to es5-discuss yet. But it's worth noting here that SpiderMonkey and (soon, I believe) Nitro will implement this as an early error, even if perhaps that shouldn't play into the calculation of whether these tests should or shouldn't change.

Commented Issue: Tests 15.4.4.21-9-c-ii-4-s and 15.4.4.22-9-c-ii-4-s assert that null should be passed as the this value (should be undefined) [29085]

jwalden — Sat, 01 Jan 2011 22:41:00 GMT

The tests 15.4.4.21-9-c-ii-4-s and 15.4.4.22-9-c-ii-4-s assert that null should be passed as the this value to the callback function for reduce and reduceRight. The spec is inconsistent (it says undefined for reduce and null for reduceRight) but in the latest errata it clarifies that both functions should pass undefined as the this value.
Comments: ** Comment from web user: jwalden **

http://es5conform.codeplex.com/workitem/29141 has a patch that also fixes this, although I believe it's not fixed quite as suggested here.

Commented Issue: The precondition for test 15.4.4.19-9-3 references an undefined variable [29088]

jwalden — Sat, 01 Jan 2011 22:39:49 GMT

The precondition for test 15.4.4.19-9-3 references "a":
return fnExists(Array.prototype.map, Array.isArray(a));
The variable "a" is not yet defined. This causes the precondition to fail.
Comments: ** Comment from web user: jwalden **

http://es5conform.codeplex.com/workitem/29141 has a patch that also fixes this.

Commented Issue: Tests 12.2.1-1-s to 12.2.1-10-s expect an EvalError when it should be a SyntaxError [29089]

jwalden — Sat, 01 Jan 2011 22:39:39 GMT

Tests 12.2.1-1-s to 12.2.1-10-s all expect that declaring a variable called "eval" will throw an EvalError. This is not correct - the spec states that SyntaxError should be thrown instead.
Comments: ** Comment from web user: jwalden **

http://es5conform.codeplex.com/workitem/29141 has a patch that also fixes this.