James McParlane's Blog

[UPDATE] I have updated this with information gleaned from http://jibbering.com/faq/faq_notes/closures.html on the true nature of closures and performed an experiment. Its worse than I thought. The ‘shotgun’ nature of scope closure in JavaScript means that its very easy to write leaky code. One of the problems with JavaScript is that you can make an effective closure over objects that are not explicitly obvious if you look at the code within the closure itself.

The very general rule of thumb seems to be The deeper you are in scope when you create your closures, the more likely you are to have leakage.

Updates are in italics.[/UPDATE]

Why does IE leak when closures are used in event listeners?

1. When a closure causes a circular reference. 
2. A blind spot in the IE garbage collector for circular references involving DOM elements. IE uses uses “mark and sweep” for JavaScript objects, but it has a blind spot in that it uses COM “reference counting” for DOM objects.

Now, when I say closure, I don’t just mean an anonymous function. I mean a real closure, although the way that JavaScript handles scope closure this becomes effectively a moot point.

The following is not a closure, because it does not reference any variable out of scope of the function.

function(){var x = 0; return x;}

The following is a closure in the traditional sense, because the function references l_a, which is declared outside the scope of the function.

var l_a = 1;
var l_f = function(){var x = l_a; return x;}

In JavaScript the following is effectively a closure too, because when you declare a function in JavaScript it gets a reference to the scope that it was declared in, and that scope contains references to all the objects that are declared in that scope. Even if you didn’t intend to maintain a reference to l_a from the closure, you effectively are.

var l_a = 1;
var l_f = function(){var x = 0; return x;}

Now here is an important bit, every closure that contains an intended or unintended out of scope reference to the same variable in the same scope will be referencing the same object. And any change to that referenced object will be visible to all closures.

So…

var l_a = 1;
var l_f1 = function(){return l_a;}

l_a = 2;
var l_f2 = function(){return l_a;}

alert(l_f1());
alert(l_f2());

Both l_f1() and l_f2() will return 2 [TEST] because they both reference l_a in the same scope.

Closures declared in the same scope referencing the same variable.

This is because all closures declared in the same scope maintain a reference to the same scope, so any changes you make to any variables declared in that scope or any other sub-scope are visible to the closure.

In the case of a memory leak pattern, if these closures were event handlers, then your leak would include l_a and because of scope closure l_f1 and l_f2

But a benign looking improvement to this code can cause many more leaks and change the behavior of the code.

function MakeClosure(p_a)
{
    return function(){return p_a;}
}

var l_a = 1;
var l_f1 = MakeClosure(l_a);

l_a = 2;
var l_f2 = MakeClosure(l_a);

l_a = 3;

alert(l_f1());
alert(l_f2());

In this case, l_f1() will return 1.. but l_f2() will return 2 [TEST]

Why?

l_a is a scalar value, it is being copied onto the stack in each scope that is created when you call MakeClosure, so p_a is not a reference to the original l_a, it is a copy of the value of l_a at the point of which MakeClosure() is called in two different scopes.

Closures declared in different scopes referencing different copies of a variable

Now here is another important bit..

If the above pattern was involved in a leak pattern, you would leak one copy of p_a for every time MakeClosure() is called because the scope of MakeClosure(), which includes the declaration of p_a is referenced by the closure.

This does not apply to object references. See the following.

function MakeClosure(p_o)
{
 return function(){return p_o.m_a;}
}

var l_o = new Object();
l_o.m_a = 1;
var l_f1 = MakeClosure(l_o);

l_o.m_a = 2;
var l_f2 = MakeClosure(l_o);
l_o.m_a = 3;

alert(l_f1());
alert(l_f2());

In this case both l_f1 and l_f2 return 3 – because p_o is a reference pointer to the original l_o. [TEST]

Closures declared in different scopes using different references to the same variable

So if the above pattern was involved in a leak pattern you would leak three objects, l_o, l_f1 and l_f1 no matter how many times you call MakeClosure();

So why does the following code leak so badly? [TEST]

window.onload = init;

function init()
{
 // Start timing
 var T1 = (new Date()).getTime();

 createLinks();

 var x = document.getElementsByTagName('a');

 for (var i=0;i<x.length;i++)
 {
  assignListener(x[i]);
 }

 // How long did that take?
 alert("it took " + ((new Date()).getTime() - T1) / 1000 +
 "seconds to render thisrnHit F5 and see "+
 "what happens to the render time.");

}

function assignListener(p_e)
{
 p_e.onclick = function ()
 {
  this.firstChild.nodeValue = 'CLICK';
 }
}

Every time we call assignListener(p_e) we are creating a new scope that contains a new and unique reference, so we are leaking multiple scopes and because each p_e is a references to a different DOM object, it comes under the garbage collectors blind spot.

The following leak pattern is established for every element reference passed as a parameter to assignedListener.

Every time we call assignListener() we create a new scope that contains a reference to a DOM element.

If we were to be sensible and use ‘this’ instead of p_e, then we would avoid the issue all together.

The deeper you are in scope when you create your closures, the more likely you are to have leakage.

But – these leaks are very subtle. Its not always obvious that they are going to occur, even when we do something innocent looking. One of the problems with JavaScript is that you can make an effective closure over objects that are not implicitly obvious if you look at the code in the closure itself.

There are four possible solutions.

1. Ban closures as event listeners – but they are very useful and they are just one part of the IE leakage issue.
2. Force programmers to go through a complicated framework that has no leak patterns. Lets face it, You can’t force a programmer to do anything and its not going to stop them from accidentally writing leaky code.
3. Wait for IE7 and hope it does not leak.
4. Write your own garbage collector. This is what I did in my solution, but a very focused one. Any element that gets a listener assigned gets very carefully scrubbed after I am finished with it. Any element that is attached to a structure but is not part of the DOM is ‘garbage collected’

Full points to anyone who gets the twisted ‘Cabs’ reference in the name of this topic. 🙂

[UPDATE]

Here is my updated element scrubber.

/*!
    @fn         MetaWrap.Page.Element.scrub = function(p_element)
    @param      p_element A reference to the element we want to scrub
    @return     void
    @brief      Nulls element references that commonly cause circular references.
    @author     James Mc Parlane
    @date       23 October 2005

    Called for every Element that is garbage collected by
    MetaWrap.Page.garbageCollectListeners() or by the teardown code
    MetaWrap.Page.deleteListeners()

    As a generic leak pattern cleanup, this solution is only
    effective for elements have had a listener added via
    MetaWrap.Page.Element.addEventListener

    Works by nulling event listeners and custom expanded members.

    This will only work if you use the following naming standard
    for your custom expansions of the element object.

    m_XXX eg. m_flipover
    $XXX eg. $flipover
*/
MetaWrap.Page.Element.scrub = function(p_element)
{
    // For each member of the element
    for(var l_member in p_element)
    {
        // If its an event listener or one of my user assigned
        // members m_XXX, $XXX (self imposed naming standard)
        if ((l_member.indexOf("on") == 0) ||
            (l_member.indexOf("m_") == 0) ||
            (l_member.indexOf("$") == 0))
        {
            // Break the potential circular reference
            p_element[l_member] = null;
        }
    }
}

Here is my entry for the QuirksMode addEvent() recoding contest.

Its not the smallest or simplest, but this was a project that I started before the competition was announced and its main purpose is to solve a much larger problem.

My solution provides an implementation of addEventListener and removeEventListener that is consistent across today’s most popular browsers, supporting event bubbling and event capture (both the strict and non strict variants).

It..

• Implements full W3C emulation layer for browsers that don’t have a native addEventListener.
• Emulates both event capture and event bubbling and provides a way to emulate the strict W3C and ‘consensual’ modes by dealing with some quirks in current browsers.
• Plays well with other code and the mixing of pre-existing inline and assigned event listeners.
• Allows event cancelling.
• Does not leak in itself but is aware of the IE leak patterns and implements a focused scrubbing garbage collector to deal with the common and not so common leakage issues. The GC can be set to to agressive or lazy.

Read the full blog entry if you want to get the whole story.

I have tested this under IE5 (thanks Emanuele Aina) , IE5.5, IE6, Safari1.3.1, Firefox1.0.5, Firefox1.5beta, Opera8.5, Netscape 6.2.3, Netscape 7.2, Netscape 8, Mozilla 1.7.12.

I have not tested it under v4 browsers, but I see no reason why it would not work in emulation mode with some minor tweaks.

It won’t work under IE5.0 unless I can somehow implement call or apply and splice in JavaScript for IE5 🙂

You can view the actual implementation in mw_lib_page_element_addhandler.js and there is some supporting code in mw_lib_page_element.js 

And if you just want the js files, here they are.

mw_lib_page_element_addhandler.js
mw_lib_page_element.js
mw_lib_page.js
mw_lib.js

How To Use

Download the JavaScript library files

Then include them using the following script includes

<script language="JavaScript" src="mw_lib.js"></script>
<script language="JavaScript" src="mw_lib_page.js"></script>
<script language="JavaScript" src="mw_lib_page_element.js"></script>
<script language="JavaScript" src="mw_lib_page_element_addhandler.js"></script>

Four functions are provided

addEventListener

MetaWrap.Page.Element.addEventListener(p_element,p_event_type,p_function,p_capture);

This is a complete replacement for W3C element.addEventListener. The parameters are the same as the W3C version of the function with the same name

removeEventListener

MetaWrap.Page.Element.removeEventListener(p_element,p_event_type,p_function,p_capture);

This is a complete replacement for W3C element.removeEventListener.The parameters are the same as the W3C version of the function with the same name

Here is a testcase that tests both addEventListener and removeEventListener.

deleteListeners

MetaWrap.Page.deleteListeners();

This function implements the classic IE memory leak fix and should only be executed at window.onunload

garbageCollectListeners

MetaWrap.Page.garbageCollectListeners();

This function will garbage collect any dangling element references. There is a global variable MetaWrap.Page.m_listeners_auto_garbage that if set to true will force a garbage collect after every event. Its default value is false.

Here is a testcase for garbageCollectListeners and here is another.

If requested by enough people, I will release a ‘compact’ stand alone version of the code. I have an explanation for why my code is so flowery at the end of this post with the heading “My Strange Coding Style”. 🙂

How It Works And Why

Hot on the heels of my last JavaScript project, I decided to get back into my JavaScript behavior library and found myself needing an implementation of addEventListener and removeEventListener that was consistent across all browsers. I researched the current state of the art and found an article on how you should not do it.

Taking this on board I decided to write own implementation, my aim being to fully support in IE everything that the W3C specification defines, so I ended up developing a solution that supports event bubbling and both the strict and non strict variants of event capture.

Now its debatable whether or not event capture is of any use, but in the past I have found it very useful in writing clean minimal code and I personally have a requirement for it.

The rules for how addEventListener (and the listeners themselves) should behave are very clearly defined, but perhaps not perfectly clear as it seems that only one of the major browsers (Opera) has followed the spec to the letter, and the rest seem to have either emulated each other, forming a de facto standard (Mozilla / Firefox / Netscape and Safari) and last but not least the browser with the most market share simply does not implement it at all (IE). Here is previous post about this issue that also points to a testcase.

The 6 Golden Rules Of W3C Events

Capture vs Bubble

Rule 1 describes the order in which the listeners are triggered and Rule 5 describes what should be passed to the listener. I try to think of the top level of a document, the <HTML> element for example, as the surface of a pond with all the other elements being below the surface at depths determined by parent/child relationships. Imagine you throw a rock into the pool at a particular element, the water captures the rock and down it falls till it hits that element. The element then bubbles and the bubbles rise to the surface. 

That is essentially what is happening during the capture and bubbling phases.

The added step is that as the rock passes each element, in the capture phase, if the element has events listening on capture, those events are triggered, and in the bubbling phase, only those listeners not registered to capture are triggered. The most shallow capture event fires first, and the most shallow bubbling event fires last.

Down And Up

I wrote a series of test cases and then I implemented MetaWrap.Page.Element.addEventListener.  When I completed the first version, I went looking for some critical test cases, I came across this competition.

“Hence I’d like to take the opportunity to launch an addEvent() recoding contest. Write your own version of addEvent() and removeEvent(), submit it by adding a comment to this page, and win JavaScript fame.”

When I first saw it, there were no comments and thus no entries submitted, so I thought to myself.. hmmm.. “First Post?” 🙂 It was 11pm, and approaching the dark side of breakfast, but I decided to go for it.

I downloaded the test page and substituted my code. I was disappointed, It kind of worked in IE, but didn’t have the same behavior in the sub menu elements as it did under W3C browsers, do I decided that there must be something wrong with my implementation, which was odd given the number of test-cases that I threw at it.

I decided to give first post a miss because I had to get some sleep and another project deadline was looming and I needed my wits about me.

A week later I audited my code and found something that was in theory, an issue, but it did not look like it would affect the competition code. The issue was so do with what happened when you registered event listeners during an event (see Rule 2). Eg. on element X, onclick is fired and then then modifies the onclick for element X and Y.

My interpretation of Rule 2 is that, if during an event of a given event type, you modify the listeners for that event type on the current element, you won’t see the new listeners executed till the next time that event is triggered. If you modify listeners of that event type on other elements that the event will be visiting as part of the bubbling or capture flow for this event, these listeners will be executed.

My code was violating this, you could in theory add a new handler on the event which would get fired and then add a new but different handler that removed the first and so on, so it was possible to throw the browser into an infinite loop of event handlers that leap-frogged each other into oblivion, which in retrospect makes Rule 2 a Good Thing.

Here is is a set of tests case for Rule 2. one, two, three, four.

Rule 3 describes what should happen when a listener is registered and how to deal with duplicates my implemenation obeys this, so no problem there. 

So that was a dead end. So still no joy. So I decided to see if a solution to the competition actually existed, maybe I was missing something fundamental. I recoded the competition page with the added events in-lined in the correct order and it behaved exactly the same as my code.

So something else was obviously up.

I wrote a test case so that I could log exactly which event was being fired and in what order and compared results for IE6 and Firefox when from above the elements dragging the mouse down over “Item 1” then up off it again.

IE6

Firefox

Looks like the issue was the mouseover/mouseout was targeted on the UL in Firefox, but on the A element in IE6.

Now the events are only listening on the LI elements but in IE the bubbled event is targeted by the A. So the mouseover/mouseouts are bubbling down to the parent LI and are being executed, but they are only reacting to movement on the A element. The trick was to make the LI wider.

To clarify this, from the point of view of the events, in IE, you are rolling the mouse over the A, but in Firefox, you are rolling the mouse over the LI which has plenty of area to roll around and activate the sub menu items.

So the competition has a sting, its not a simple case of sorting out addEvent, the competition example itself suffers from a CSS compatibility error which is going to mean its always a little flaky under IE even if you restyle the LI to be wider.

After making this width adjustment, my solution worked perfectly!

Later on after re-reading the competition page It dawned on me that I had been warned about this but had mis-interpreted the warning.

Your entry should exhibit the same behaviour as this example page, which uses Scott Andrew’s old functions. The example page doesn’t work in Explorer, but your entry should. The mouseout functions on the example page don’t work perfectly, but for the contest that doesn’t matter.

 

Browser Quirks

Mixed Mode Events

The following code results in different behavior in IE6 and Firefox 1.5

MetaWrap.Page.Element.addEventListener(l_a1,"click",f1,true);
MetaWrap.Page.Element.addEventListener(l_a1,"click",f2,true);        
l_a1.onclick = f3;
l_a1.onclick = f4;

In IE or Opera if you click on the element, just the f4 listener is executed.

In Firefox and Safari if you click on the element, just the f1,f2 and f4 listener are executed, in that order.

I have created a test case that tests mixed mode event listener assignment in various orders.

It looks like Firefox treats the assignment

element.onevent = listener;

as the following sequence

removeEventListener(element,event,element.onevent,false); 
addEventListener(element,event,listener,false); 

It would be ‘possible’ to emulate this behavior, but it would be computationally expensive, not 100% reliable and probably not very practical. A possible method would be to, on each event, walk the list of all elements event types and check for consistency eg. event with listener stacks with a different master event handler than MetaWrap.Page.Element.listenerShimFunction.

[UPDATE….

With very little change I have emulated two issues which were naging at me.

The first is an issue when listeners were added with the following code.

l_a1.onclick = f1;
MetaWrap.Page.Element.addEventListener(l_a1,"click",f2,true);

Under IE emulation mode and Opera, f1 will never fire because it is overwritten by f2.  In Safari, they fire in the orfer f1,f2. In Mozilla based browsers f1 is preserved so both fire in the order f1,f2. This is because the Mozilla based browsers do something extra in addEventListener, they preserve the existing inline/assigned listeners as a captured event. Here is a testcase that shows the default behavior using the standard addEventListener reacting with different mixed mode events.

In the latest version of my library I emulate the default behavior of modern Mozilla based browsers for all browsers. 

I am only trialing this emulation, but it seems to make a lot of sense. It could however be effort for very little gain – my only justification for this is.

1. It is low cost and standardises behaviors (and yes I choose Mozilla as the default behavior because of point 2)
2. Its going to make more robust code. When using the library in existing code I don’t want to inadvertently clobber the existing inline listeners so I have decided to (hopefuly) ‘do no harm’.

In Safari, they fire in the order f1,f2 – from what I can tell Safari events are rather broken.

The following code in Safari (tested against version 1.3.1)

var l_a5 = document.getElementById("a5");
l_a5.onclick = f1; 
l_a5.onclick = null; 
l_a5.addEventListener("click",f2,true); 

Clicking on the element referenced by l_a5 will result in both events triggering in the order f2,f1

On every other browser that supports element.addEventListener, only f2 triggers.

Here is a testcase – try the last test and you will see what I mean.

The second is for the following sequence

  MetaWrap.Page.Element.addEventListener(l_a3,"click",f1,true);
  l_a3.onclick = f3;
  l_a3.onclick = f4;
  MetaWrap.Page.Element.addEventListener(l_a3,"click",f2,true);

Which now works consistently across all browsers, even in emulation mode.

I’m not sure if I will keep these patches (they are clearly labeled in the source code). They make the code ugly, and they are little more then a token effort to fix a whole family of problems that occur if you start mixing the way you add event listeners. See this testcase.

They do however fix some obvious collisions that occur when adding this library to pre-existing code. The majority of the remaining problems would be an issue in the existing code anyway.

so..

I am of two minds on this one.

…]

Deleted Elements

Rule 4 covers listener removal. When an element is removed, what happens to its registered events? At the moment these are going to stick around in memory. Until the DOM mutation notification event becomes widely available and all other browsers are long dead (5 years?) we can’t really deal with this efficiently, so the best we can do is have a simple garbage collect phase. To this effect the following method is made available.

MetaWrap.Page.garbageCollectListeners();

This will perform a garbage collect pass and remove all listeners for elements that are no longer part of the document.

There are two testcases for this method.. one tests elements removed by overwriting using innerHTML and the other tests elements that have been removed surgically with removeNode. The resulting element states are different in each case which garbage collector has to deal with cleanly.

Browser Compliance To W3 Spec

Opera seems to be the only browser that follows Rule 6, which complicates things. I can emulate the right way, or I can emulate the way that everyone else has implemented it. I have decided to do both, to which end the following global variable exists.

MetaWrap.Page.m_listeners_strict_w3c = false;

Its default behavior (false) is to force all browsers to behave like Netscape/Mozilla/Firefox and Safari. If we are running Opera, we execute the same emulation mode that we use for IE.

If you set its value to true however, it will emulate for all other browsers the way that Opera handles its event propagation.

The logic behind this decision is rational yet complicated but it comes down to providing a choice for the developer who can choose performance vs compliance but still maintain consistency across all browsers.

Here is previous post about this issue about the Rest Of The world vs Opera.

Here is a testcase that demonstrates the library with MetaWrap.Page.m_listeners_strict_w3c set to false.

Here is a testcase that demonstrates the library with MetaWrap.Page.m_listeners_strict_w3c set to true.

My Strange Coding Style

I work at a web development company called Massive Interactive in Sydney Australia where part of my role is to be a technology storm-trooper and develop code and libraries that can be passed on to others for use or further development.

I too can code strange JavaScript built with nothing but single character variables, ternary statements, object hash tricks and K&R styling but I choose not to because much of what I develop needs to be maintained by other people. I code to be understandable and maintainable. I am painfully aware of the cost of name spacing, the cost of the size of variable names and even the load time cost of comments , luckily the run time cost for comments and formatting is close to zero. But I choose to code in this way because I would rather have the performance hit than have code that is hard to maintain. 

I’m a firm believer in keeping the primary JavaScript code base neat and tidy, when it comes to rationalising the code for production – that is where the automatic composition and compression utilities come in.

If enough people want a ‘compressed’ version of the functions, I will happily provide some, but I’m sure there is going to be some bugfixes as soon as people start playing with this.

At last a resource that describes exactly how memory leaks occur in IE, written by the only people qualified to know, Microsoft.

“In the past, memory leaks haven’t posed huge problems for Web developers. Pages were kept relatively simple and navigation between different locations within a site was a great way to clean up any loose memory. If there was a leak, it was most likely small enough to go unnoticed.

New Web applications live up to higher standards. A page might run for hours without being navigated and retrieve updated information dynamically through Web services. Language features are pushed to the breaking point by combining complex event schemes, object-oriented JScript, and closures to produce entire applications. With these and other changes, certain memory leak patterns are becoming more prominent, especially those previously hidden by navigation.”

Glad to see that my own mental model of the issue was spot on for the first two patterns so all my current cleanup code seems correct. The last two patterns are new to me; I have never encountered them, but are worth good hard study.

Rather shocked to find out that IE uses COM “reference counting” instead of the modern “mark and sweep” method which is immune to circular references.

I used to believe that the issue was simply a blind-spot in IEs garbage collector with event listeners, but from this article it seems that all circular references that include DOM elements are potentially harmful and the only antidote is effective cleanup code.

Also.. a handy dandy IE leak detector

http://www.outofhanwell.com/ieleak/

Try the detector against these testcases

test_19_ie_leak.html
test_20_ie_leak_noclosure.html
test_21_ie_leak_fix.html

Not me – and there is a slight twang of disappointment, but its hardly so very surprising – the competition was simply to recode addEvent, and my solution was for the entire class of event registration problems that plague all the current browsers and the associated memory leakage issues in IE.

The competition drove me to make my solution as thorough as it could be, and its happily powering the MetaWrap WireWrap behavior library and providing a reliable rock solid base. No matter what browser, no matter what event propagation mode I use, even when I mix them, its consistent across all the browsers I’m targeting.

And that makes me a very happy camper 🙂

Someone else who is on the same wavelength but can make much better diagrams 🙂

http://particletree.com/features/4-layers-of-separation/

Glad that someone has gone off and done this.

http://h3h.net/2005/04/faulty-firing/

This is part of the knowledge that I need to acquire if I am ever to get the JavaScript macro recorder to work consistently. The next stage of development is to complete the  event simulation and translation functions. A macro recorded on one browser needs to be translated into the language of another browser or it won’t work.