The Tor Browser: comparison js/src/jsgc.cpp

--1:000000000000
+:9abf47182107
+/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 4 -*-
+* vim: set ts=8 sts=4 et sw=4 tw=99:
+* This Source Code Form is subject to the terms of the Mozilla Public
+* License, v. 2.0. If a copy of the MPL was not distributed with this
+* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
+/*
+* This code implements an incremental mark-and-sweep garbage collector, with
+* most sweeping carried out in the background on a parallel thread.
+*
+* Full vs. zone GC
+* ----------------
+*
+* The collector can collect all zones at once, or a subset. These types of
+* collection are referred to as a full GC and a zone GC respectively.
+*
+* The atoms zone is only collected in a full GC since objects in any zone may
+* have pointers to atoms, and these are not recorded in the cross compartment
+* pointer map. Also, the atoms zone is not collected if any thread has an
+* AutoKeepAtoms instance on the stack, or there are any exclusive threads using
+* the runtime.
+*
+* It is possible for an incremental collection that started out as a full GC to
+* become a zone GC if new zones are created during the course of the
+* collection.
+*
+* Incremental collection
+* ----------------------
+*
+* For a collection to be carried out incrementally the following conditions
+* must be met:
+*  - the collection must be run by calling js::GCSlice() rather than js::GC()
+*  - the GC mode must have been set to JSGC_MODE_INCREMENTAL with
+*    JS_SetGCParameter()
+*  - no thread may have an AutoKeepAtoms instance on the stack
+*  - all native objects that have their own trace hook must indicate that they
+*    implement read and write barriers with the JSCLASS_IMPLEMENTS_BARRIERS
+*    flag
+*
+* The last condition is an engine-internal mechanism to ensure that incremental
+* collection is not carried out without the correct barriers being implemented.
+* For more information see 'Incremental marking' below.
+*
+* If the collection is not incremental, all foreground activity happens inside
+* a single call to GC() or GCSlice(). However the collection is not complete
+* until the background sweeping activity has finished.
+*
+* An incremental collection proceeds as a series of slices, interleaved with
+* mutator activity, i.e. running JavaScript code. Slices are limited by a time
+* budget. The slice finishes as soon as possible after the requested time has
+* passed.
+*
+* Collector states
+* ----------------
+*
+* The collector proceeds through the following states, the current state being
+* held in JSRuntime::gcIncrementalState:
+*
+*  - MARK_ROOTS - marks the stack and other roots
+*  - MARK       - incrementally marks reachable things
+*  - SWEEP      - sweeps zones in groups and continues marking unswept zones
+*
+* The MARK_ROOTS activity always takes place in the first slice. The next two
+* states can take place over one or more slices.
+*
+* In other words an incremental collection proceeds like this:
+*
+* Slice 1:   MARK_ROOTS: Roots pushed onto the mark stack.
+*            MARK:       The mark stack is processed by popping an element,
+*                        marking it, and pushing its children.
+*
+*          ... JS code runs ...
+*
+* Slice 2:   MARK:       More mark stack processing.
+*
+*          ... JS code runs ...
+*
+* Slice n-1: MARK:       More mark stack processing.
+*
+*          ... JS code runs ...
+*
+* Slice n:   MARK:       Mark stack is completely drained.
+*            SWEEP:      Select first group of zones to sweep and sweep them.
+*
+*          ... JS code runs ...
+*
+* Slice n+1: SWEEP:      Mark objects in unswept zones that were newly
+*                        identified as alive (see below). Then sweep more zone
+*                        groups.
+*
+*          ... JS code runs ...
+*
+* Slice n+2: SWEEP:      Mark objects in unswept zones that were newly
+*                        identified as alive. Then sweep more zone groups.
+*
+*          ... JS code runs ...
+*
+* Slice m:   SWEEP:      Sweeping is finished, and background sweeping
+*                        started on the helper thread.
+*
+*          ... JS code runs, remaining sweeping done on background thread ...
+*
+* When background sweeping finishes the GC is complete.
+*
+* Incremental marking
+* -------------------
+*
+* Incremental collection requires close collaboration with the mutator (i.e.,
+* JS code) to guarantee correctness.
+*
+*  - During an incremental GC, if a memory location (except a root) is written
+*    to, then the value it previously held must be marked. Write barriers
+*    ensure this.
+*
+*  - Any object that is allocated during incremental GC must start out marked.
+*
+*  - Roots are marked in the first slice and hence don't need write barriers.
+*    Roots are things like the C stack and the VM stack.
+*
+* The problem that write barriers solve is that between slices the mutator can
+* change the object graph. We must ensure that it cannot do this in such a way
+* that makes us fail to mark a reachable object (marking an unreachable object
+* is tolerable).
+*
+* We use a snapshot-at-the-beginning algorithm to do this. This means that we
+* promise to mark at least everything that is reachable at the beginning of
+* collection. To implement it we mark the old contents of every non-root memory
+* location written to by the mutator while the collection is in progress, using
+* write barriers. This is described in gc/Barrier.h.
+*
+* Incremental sweeping
+* --------------------
+*
+* Sweeping is difficult to do incrementally because object finalizers must be
+* run at the start of sweeping, before any mutator code runs. The reason is
+* that some objects use their finalizers to remove themselves from caches. If
+* mutator code was allowed to run after the start of sweeping, it could observe
+* the state of the cache and create a new reference to an object that was just
+* about to be destroyed.
+*
+* Sweeping all finalizable objects in one go would introduce long pauses, so
+* instead sweeping broken up into groups of zones. Zones which are not yet
+* being swept are still marked, so the issue above does not apply.
+*
+* The order of sweeping is restricted by cross compartment pointers - for
+* example say that object |a| from zone A points to object |b| in zone B and
+* neither object was marked when we transitioned to the SWEEP phase. Imagine we
+* sweep B first and then return to the mutator. It's possible that the mutator
+* could cause |a| to become alive through a read barrier (perhaps it was a
+* shape that was accessed via a shape table). Then we would need to mark |b|,
+* which |a| points to, but |b| has already been swept.
+*
+* So if there is such a pointer then marking of zone B must not finish before
+* marking of zone A.  Pointers which form a cycle between zones therefore
+* restrict those zones to being swept at the same time, and these are found
+* using Tarjan's algorithm for finding the strongly connected components of a
+* graph.
+*
+* GC things without finalizers, and things with finalizers that are able to run
+* in the background, are swept on the background thread. This accounts for most
+* of the sweeping work.
+*
+* Reset
+* -----
+*
+* During incremental collection it is possible, although unlikely, for
+* conditions to change such that incremental collection is no longer safe. In
+* this case, the collection is 'reset' by ResetIncrementalGC(). If we are in
+* the mark state, this just stops marking, but if we have started sweeping
+* already, we continue until we have swept the current zone group. Following a
+* reset, a new non-incremental collection is started.
+*/
+#include "jsgcinlines.h"
+#include "mozilla/ArrayUtils.h"
+#include "mozilla/DebugOnly.h"
+#include "mozilla/MemoryReporting.h"
+#include "mozilla/Move.h"
+#include <string.h>     /* for memset used when DEBUG */
+#ifndef XP_WIN
+# include <unistd.h>
+#endif
+#include "jsapi.h"
+#include "jsatom.h"
+#include "jscntxt.h"
+#include "jscompartment.h"
+#include "jsobj.h"
+#include "jsscript.h"
+#include "jstypes.h"
+#include "jsutil.h"
+#include "jswatchpoint.h"
+#include "jsweakmap.h"
+#ifdef XP_WIN
+# include "jswin.h"
+#endif
+#include "prmjtime.h"
+#include "gc/FindSCCs.h"
+#include "gc/GCInternals.h"
+#include "gc/Marking.h"
+#include "gc/Memory.h"
+#ifdef JS_ION
+# include "jit/BaselineJIT.h"
+#endif
+#include "jit/IonCode.h"
+#include "js/SliceBudget.h"
+#include "vm/Debugger.h"
+#include "vm/ForkJoin.h"
+#include "vm/ProxyObject.h"
+#include "vm/Shape.h"
+#include "vm/String.h"
+#include "vm/TraceLogging.h"
+#include "vm/WrapperObject.h"
+#include "jsobjinlines.h"
+#include "jsscriptinlines.h"
+#include "vm/Stack-inl.h"
+#include "vm/String-inl.h"
+using namespace js;
+using namespace js::gc;
+using mozilla::ArrayEnd;
+using mozilla::DebugOnly;
+using mozilla::Maybe;
+using mozilla::Swap;
+/* Perform a Full GC every 20 seconds if MaybeGC is called */
+static const uint64_t GC_IDLE_FULL_SPAN = 20 * 1000 * 1000;
+/* Increase the IGC marking slice time if we are in highFrequencyGC mode. */
+static const int IGC_MARK_SLICE_MULTIPLIER = 2;
+#if defined(ANDROID) || defined(MOZ_B2G)
+static const int MAX_EMPTY_CHUNK_COUNT = 2;
+#else
+static const int MAX_EMPTY_CHUNK_COUNT = 30;
+#endif
+/* This array should be const, but that doesn't link right under GCC. */
+const AllocKind gc::slotsToThingKind[] = {
+/* 0 */  FINALIZE_OBJECT0,  FINALIZE_OBJECT2,  FINALIZE_OBJECT2,  FINALIZE_OBJECT4,
+/* 4 */  FINALIZE_OBJECT4,  FINALIZE_OBJECT8,  FINALIZE_OBJECT8,  FINALIZE_OBJECT8,
+/* 8 */  FINALIZE_OBJECT8,  FINALIZE_OBJECT12, FINALIZE_OBJECT12, FINALIZE_OBJECT12,
+/* 12 */ FINALIZE_OBJECT12, FINALIZE_OBJECT16, FINALIZE_OBJECT16, FINALIZE_OBJECT16,
+/* 16 */ FINALIZE_OBJECT16
+};
+static_assert(JS_ARRAY_LENGTH(slotsToThingKind) == SLOTS_TO_THING_KIND_LIMIT,
+"We have defined a slot count for each kind.");
+const uint32_t Arena::ThingSizes[] = {
+sizeof(JSObject),           /* FINALIZE_OBJECT0             */
+sizeof(JSObject),           /* FINALIZE_OBJECT0_BACKGROUND  */
+sizeof(JSObject_Slots2),    /* FINALIZE_OBJECT2             */
+sizeof(JSObject_Slots2),    /* FINALIZE_OBJECT2_BACKGROUND  */
+sizeof(JSObject_Slots4),    /* FINALIZE_OBJECT4             */
+sizeof(JSObject_Slots4),    /* FINALIZE_OBJECT4_BACKGROUND  */
+sizeof(JSObject_Slots8),    /* FINALIZE_OBJECT8             */
+sizeof(JSObject_Slots8),    /* FINALIZE_OBJECT8_BACKGROUND  */
+sizeof(JSObject_Slots12),   /* FINALIZE_OBJECT12            */
+sizeof(JSObject_Slots12),   /* FINALIZE_OBJECT12_BACKGROUND */
+sizeof(JSObject_Slots16),   /* FINALIZE_OBJECT16            */
+sizeof(JSObject_Slots16),   /* FINALIZE_OBJECT16_BACKGROUND */
+sizeof(JSScript),           /* FINALIZE_SCRIPT              */
+sizeof(LazyScript),         /* FINALIZE_LAZY_SCRIPT         */
+sizeof(Shape),              /* FINALIZE_SHAPE               */
+sizeof(BaseShape),          /* FINALIZE_BASE_SHAPE          */
+sizeof(types::TypeObject),  /* FINALIZE_TYPE_OBJECT         */
+sizeof(JSFatInlineString),  /* FINALIZE_FAT_INLINE_STRING   */
+sizeof(JSString),           /* FINALIZE_STRING              */
+sizeof(JSExternalString),   /* FINALIZE_EXTERNAL_STRING     */
+sizeof(jit::JitCode),       /* FINALIZE_JITCODE             */
+};
+#define OFFSET(type) uint32_t(sizeof(ArenaHeader) + (ArenaSize - sizeof(ArenaHeader)) % sizeof(type))
+const uint32_t Arena::FirstThingOffsets[] = {
+OFFSET(JSObject),           /* FINALIZE_OBJECT0             */
+OFFSET(JSObject),           /* FINALIZE_OBJECT0_BACKGROUND  */
+OFFSET(JSObject_Slots2),    /* FINALIZE_OBJECT2             */
+OFFSET(JSObject_Slots2),    /* FINALIZE_OBJECT2_BACKGROUND  */
+OFFSET(JSObject_Slots4),    /* FINALIZE_OBJECT4             */
+OFFSET(JSObject_Slots4),    /* FINALIZE_OBJECT4_BACKGROUND  */
+OFFSET(JSObject_Slots8),    /* FINALIZE_OBJECT8             */
+OFFSET(JSObject_Slots8),    /* FINALIZE_OBJECT8_BACKGROUND  */
+OFFSET(JSObject_Slots12),   /* FINALIZE_OBJECT12            */
+OFFSET(JSObject_Slots12),   /* FINALIZE_OBJECT12_BACKGROUND */
+OFFSET(JSObject_Slots16),   /* FINALIZE_OBJECT16            */
+OFFSET(JSObject_Slots16),   /* FINALIZE_OBJECT16_BACKGROUND */
+OFFSET(JSScript),           /* FINALIZE_SCRIPT              */
+OFFSET(LazyScript),         /* FINALIZE_LAZY_SCRIPT         */
+OFFSET(Shape),              /* FINALIZE_SHAPE               */
+OFFSET(BaseShape),          /* FINALIZE_BASE_SHAPE          */
+OFFSET(types::TypeObject),  /* FINALIZE_TYPE_OBJECT         */
+OFFSET(JSFatInlineString),  /* FINALIZE_FAT_INLINE_STRING   */
+OFFSET(JSString),           /* FINALIZE_STRING              */
+OFFSET(JSExternalString),   /* FINALIZE_EXTERNAL_STRING     */
+OFFSET(jit::JitCode),       /* FINALIZE_JITCODE             */
+};
+#undef OFFSET
+/*
+* Finalization order for incrementally swept things.
+*/
+static const AllocKind FinalizePhaseStrings[] = {
+FINALIZE_EXTERNAL_STRING
+};
+static const AllocKind FinalizePhaseScripts[] = {
+FINALIZE_SCRIPT,
+FINALIZE_LAZY_SCRIPT
+};
+static const AllocKind FinalizePhaseJitCode[] = {
+FINALIZE_JITCODE
+};
+static const AllocKind * const FinalizePhases[] = {
+FinalizePhaseStrings,
+FinalizePhaseScripts,
+FinalizePhaseJitCode
+};
+static const int FinalizePhaseCount = sizeof(FinalizePhases) / sizeof(AllocKind*);
+static const int FinalizePhaseLength[] = {
+sizeof(FinalizePhaseStrings) / sizeof(AllocKind),
+sizeof(FinalizePhaseScripts) / sizeof(AllocKind),
+sizeof(FinalizePhaseJitCode) / sizeof(AllocKind)
+};
+static const gcstats::Phase FinalizePhaseStatsPhase[] = {
+gcstats::PHASE_SWEEP_STRING,
+gcstats::PHASE_SWEEP_SCRIPT,
+gcstats::PHASE_SWEEP_JITCODE
+};
+/*
+* Finalization order for things swept in the background.
+*/
+static const AllocKind BackgroundPhaseObjects[] = {
+FINALIZE_OBJECT0_BACKGROUND,
+FINALIZE_OBJECT2_BACKGROUND,
+FINALIZE_OBJECT4_BACKGROUND,
+FINALIZE_OBJECT8_BACKGROUND,
+FINALIZE_OBJECT12_BACKGROUND,
+FINALIZE_OBJECT16_BACKGROUND
+};
+static const AllocKind BackgroundPhaseStrings[] = {
+FINALIZE_FAT_INLINE_STRING,
+FINALIZE_STRING
+};
+static const AllocKind BackgroundPhaseShapes[] = {
+FINALIZE_SHAPE,
+FINALIZE_BASE_SHAPE,
+FINALIZE_TYPE_OBJECT
+};
+static const AllocKind * const BackgroundPhases[] = {
+BackgroundPhaseObjects,
+BackgroundPhaseStrings,
+BackgroundPhaseShapes
+};
+static const int BackgroundPhaseCount = sizeof(BackgroundPhases) / sizeof(AllocKind*);
+static const int BackgroundPhaseLength[] = {
+sizeof(BackgroundPhaseObjects) / sizeof(AllocKind),
+sizeof(BackgroundPhaseStrings) / sizeof(AllocKind),
+sizeof(BackgroundPhaseShapes) / sizeof(AllocKind)
+};
+#ifdef DEBUG
+void
+ArenaHeader::checkSynchronizedWithFreeList() const
+{
+/*
+* Do not allow to access the free list when its real head is still stored
+* in FreeLists and is not synchronized with this one.
+*/
+JS_ASSERT(allocated());
+/*
+* We can be called from the background finalization thread when the free
+* list in the zone can mutate at any moment. We cannot do any
+* checks in this case.
+*/
+if (IsBackgroundFinalized(getAllocKind()) && zone->runtimeFromAnyThread()->gcHelperThread.onBackgroundThread())
+return;
+FreeSpan firstSpan = FreeSpan::decodeOffsets(arenaAddress(), firstFreeSpanOffsets);
+if (firstSpan.isEmpty())
+return;
+const FreeSpan *list = zone->allocator.arenas.getFreeList(getAllocKind());
+if (list->isEmpty() || firstSpan.arenaAddress() != list->arenaAddress())
+return;
+/*
+* Here this arena has free things, FreeList::lists[thingKind] is not
+* empty and also points to this arena. Thus they must the same.
+*/
+JS_ASSERT(firstSpan.isSameNonEmptySpan(list));
+}
+#endif
+/* static */ void
+Arena::staticAsserts()
+{
+static_assert(JS_ARRAY_LENGTH(ThingSizes) == FINALIZE_LIMIT, "We have defined all thing sizes.");
+static_assert(JS_ARRAY_LENGTH(FirstThingOffsets) == FINALIZE_LIMIT, "We have defined all offsets.");
+}
+void
+Arena::setAsFullyUnused(AllocKind thingKind)
+{
+FreeSpan entireList;
+entireList.first = thingsStart(thingKind);
+uintptr_t arenaAddr = aheader.arenaAddress();
+entireList.last = arenaAddr | ArenaMask;
+aheader.setFirstFreeSpan(&entireList);
+}
+template<typename T>
+inline bool
+Arena::finalize(FreeOp *fop, AllocKind thingKind, size_t thingSize)
+{
+/* Enforce requirements on size of T. */
+JS_ASSERT(thingSize % CellSize == 0);
+JS_ASSERT(thingSize <= 255);
+JS_ASSERT(aheader.allocated());
+JS_ASSERT(thingKind == aheader.getAllocKind());
+JS_ASSERT(thingSize == aheader.getThingSize());
+JS_ASSERT(!aheader.hasDelayedMarking);
+JS_ASSERT(!aheader.markOverflow);
+JS_ASSERT(!aheader.allocatedDuringIncremental);
+uintptr_t thing = thingsStart(thingKind);
+uintptr_t lastByte = thingsEnd() - 1;
+FreeSpan nextFree(aheader.getFirstFreeSpan());
+nextFree.checkSpan();
+FreeSpan newListHead;
+FreeSpan *newListTail = &newListHead;
+uintptr_t newFreeSpanStart = 0;
+bool allClear = true;
+DebugOnly<size_t> nmarked = 0;
+for (;; thing += thingSize) {
+JS_ASSERT(thing <= lastByte + 1);
+if (thing == nextFree.first) {
+JS_ASSERT(nextFree.last <= lastByte);
+if (nextFree.last == lastByte)
+break;
+JS_ASSERT(Arena::isAligned(nextFree.last, thingSize));
+if (!newFreeSpanStart)
+newFreeSpanStart = thing;
+thing = nextFree.last;
+nextFree = *nextFree.nextSpan();
+nextFree.checkSpan();
+} else {
+T *t = reinterpret_cast<T *>(thing);
+if (t->isMarked()) {
+allClear = false;
+nmarked++;
+if (newFreeSpanStart) {
+JS_ASSERT(thing >= thingsStart(thingKind) + thingSize);
+newListTail->first = newFreeSpanStart;
+newListTail->last = thing - thingSize;
+newListTail = newListTail->nextSpanUnchecked(thingSize);
+newFreeSpanStart = 0;
+}
+} else {
+if (!newFreeSpanStart)
+newFreeSpanStart = thing;
+t->finalize(fop);
+JS_POISON(t, JS_SWEPT_TENURED_PATTERN, thingSize);
+}
+}
+}
+if (allClear) {
+JS_ASSERT(newListTail == &newListHead);
+JS_ASSERT(!newFreeSpanStart ||
+newFreeSpanStart == thingsStart(thingKind));
+JS_EXTRA_POISON(data, JS_SWEPT_TENURED_PATTERN, sizeof(data));
+return true;
+}
+newListTail->first = newFreeSpanStart ? newFreeSpanStart : nextFree.first;
+JS_ASSERT(Arena::isAligned(newListTail->first, thingSize));
+newListTail->last = lastByte;
+#ifdef DEBUG
+size_t nfree = 0;
+for (const FreeSpan *span = &newListHead; span != newListTail; span = span->nextSpan()) {
+span->checkSpan();
+JS_ASSERT(Arena::isAligned(span->first, thingSize));
+JS_ASSERT(Arena::isAligned(span->last, thingSize));
+nfree += (span->last - span->first) / thingSize + 1;
+JS_ASSERT(nfree + nmarked <= thingsPerArena(thingSize));
+}
+nfree += (newListTail->last + 1 - newListTail->first) / thingSize;
+JS_ASSERT(nfree + nmarked == thingsPerArena(thingSize));
+#endif
+aheader.setFirstFreeSpan(&newListHead);
+return false;
+}
+/*
+* Insert an arena into the list in appropriate position and update the cursor
+* to ensure that any arena before the cursor is full.
+*/
+void ArenaList::insert(ArenaHeader *a)
+{
+JS_ASSERT(a);
+JS_ASSERT_IF(!head, cursor == &head);
+a->next = *cursor;
+*cursor = a;
+if (!a->hasFreeThings())
+cursor = &a->next;
+}
+template<typename T>
+static inline bool
+FinalizeTypedArenas(FreeOp *fop,
+ArenaHeader **src,
+ArenaList &dest,
+AllocKind thingKind,
+SliceBudget &budget)
+{
+/*
+* Finalize arenas from src list, releasing empty arenas and inserting the
+* others into dest in an appropriate position.
+*/
+/*
+* During parallel sections, we sometimes finalize the parallel arenas,
+* but in that case, we want to hold on to the memory in our arena
+* lists, not offer it up for reuse.
+*/
+bool releaseArenas = !InParallelSection();
+size_t thingSize = Arena::thingSize(thingKind);
+while (ArenaHeader *aheader = *src) {
+*src = aheader->next;
+bool allClear = aheader->getArena()->finalize<T>(fop, thingKind, thingSize);
+if (!allClear)
+dest.insert(aheader);
+else if (releaseArenas)
+aheader->chunk()->releaseArena(aheader);
+else
+aheader->chunk()->recycleArena(aheader, dest, thingKind);
+budget.step(Arena::thingsPerArena(thingSize));
+if (budget.isOverBudget())
+return false;
+}
+return true;
+}
+/*
+* Finalize the list. On return al->cursor points to the first non-empty arena
+* after the al->head.
+*/
+static bool
+FinalizeArenas(FreeOp *fop,
+ArenaHeader **src,
+ArenaList &dest,
+AllocKind thingKind,
+SliceBudget &budget)
+{
+switch(thingKind) {
+case FINALIZE_OBJECT0:
+case FINALIZE_OBJECT0_BACKGROUND:
+case FINALIZE_OBJECT2:
+case FINALIZE_OBJECT2_BACKGROUND:
+case FINALIZE_OBJECT4:
+case FINALIZE_OBJECT4_BACKGROUND:
+case FINALIZE_OBJECT8:
+case FINALIZE_OBJECT8_BACKGROUND:
+case FINALIZE_OBJECT12:
+case FINALIZE_OBJECT12_BACKGROUND:
+case FINALIZE_OBJECT16:
+case FINALIZE_OBJECT16_BACKGROUND:
+return FinalizeTypedArenas<JSObject>(fop, src, dest, thingKind, budget);
+case FINALIZE_SCRIPT:
+return FinalizeTypedArenas<JSScript>(fop, src, dest, thingKind, budget);
+case FINALIZE_LAZY_SCRIPT:
+return FinalizeTypedArenas<LazyScript>(fop, src, dest, thingKind, budget);
+case FINALIZE_SHAPE:
+return FinalizeTypedArenas<Shape>(fop, src, dest, thingKind, budget);
+case FINALIZE_BASE_SHAPE:
+return FinalizeTypedArenas<BaseShape>(fop, src, dest, thingKind, budget);
+case FINALIZE_TYPE_OBJECT:
+return FinalizeTypedArenas<types::TypeObject>(fop, src, dest, thingKind, budget);
+case FINALIZE_STRING:
+return FinalizeTypedArenas<JSString>(fop, src, dest, thingKind, budget);
+case FINALIZE_FAT_INLINE_STRING:
+return FinalizeTypedArenas<JSFatInlineString>(fop, src, dest, thingKind, budget);
+case FINALIZE_EXTERNAL_STRING:
+return FinalizeTypedArenas<JSExternalString>(fop, src, dest, thingKind, budget);
+case FINALIZE_JITCODE:
+#ifdef JS_ION
+{
+// JitCode finalization may release references on an executable
+// allocator that is accessed when requesting interrupts.
+JSRuntime::AutoLockForInterrupt lock(fop->runtime());
+return FinalizeTypedArenas<jit::JitCode>(fop, src, dest, thingKind, budget);
+}
+#endif
+default:
+MOZ_ASSUME_UNREACHABLE("Invalid alloc kind");
+}
+}
+static inline Chunk *
+AllocChunk(JSRuntime *rt)
+{
+return static_cast<Chunk *>(MapAlignedPages(rt, ChunkSize, ChunkSize));
+}
+static inline void
+FreeChunk(JSRuntime *rt, Chunk *p)
+{
+UnmapPages(rt, static_cast<void *>(p), ChunkSize);
+}
+inline bool
+ChunkPool::wantBackgroundAllocation(JSRuntime *rt) const
+{
+/*
+* To minimize memory waste we do not want to run the background chunk
+* allocation if we have empty chunks or when the runtime needs just few
+* of them.
+*/
+return rt->gcHelperThread.canBackgroundAllocate() &&
+emptyCount == 0 &&
+rt->gcChunkSet.count() >= 4;
+}
+/* Must be called with the GC lock taken. */
+inline Chunk *
+ChunkPool::get(JSRuntime *rt)
+{
+JS_ASSERT(this == &rt->gcChunkPool);
+Chunk *chunk = emptyChunkListHead;
+if (chunk) {
+JS_ASSERT(emptyCount);
+emptyChunkListHead = chunk->info.next;
+--emptyCount;
+} else {
+JS_ASSERT(!emptyCount);
+chunk = Chunk::allocate(rt);
+if (!chunk)
+return nullptr;
+JS_ASSERT(chunk->info.numArenasFreeCommitted == 0);
+}
+JS_ASSERT(chunk->unused());
+JS_ASSERT(!rt->gcChunkSet.has(chunk));
+if (wantBackgroundAllocation(rt))
+rt->gcHelperThread.startBackgroundAllocationIfIdle();
+return chunk;
+}
+/* Must be called either during the GC or with the GC lock taken. */
+inline void
+ChunkPool::put(Chunk *chunk)
+{
+chunk->info.age = 0;
+chunk->info.next = emptyChunkListHead;
+emptyChunkListHead = chunk;
+emptyCount++;
+}
+/* Must be called either during the GC or with the GC lock taken. */
+Chunk *
+ChunkPool::expire(JSRuntime *rt, bool releaseAll)
+{
+JS_ASSERT(this == &rt->gcChunkPool);
+/*
+* Return old empty chunks to the system while preserving the order of
+* other chunks in the list. This way, if the GC runs several times
+* without emptying the list, the older chunks will stay at the tail
+* and are more likely to reach the max age.
+*/
+Chunk *freeList = nullptr;
+int freeChunkCount = 0;
+for (Chunk **chunkp = &emptyChunkListHead; *chunkp; ) {
+JS_ASSERT(emptyCount);
+Chunk *chunk = *chunkp;
+JS_ASSERT(chunk->unused());
+JS_ASSERT(!rt->gcChunkSet.has(chunk));
+JS_ASSERT(chunk->info.age <= MAX_EMPTY_CHUNK_AGE);
+if (releaseAll || chunk->info.age == MAX_EMPTY_CHUNK_AGE ||
+freeChunkCount++ > MAX_EMPTY_CHUNK_COUNT)
+{
+*chunkp = chunk->info.next;
+--emptyCount;
+chunk->prepareToBeFreed(rt);
+chunk->info.next = freeList;
+freeList = chunk;
+} else {
+/* Keep the chunk but increase its age. */
+++chunk->info.age;
+chunkp = &chunk->info.next;
+}
+}
+JS_ASSERT_IF(releaseAll, !emptyCount);
+return freeList;
+}
+static void
+FreeChunkList(JSRuntime *rt, Chunk *chunkListHead)
+{
+while (Chunk *chunk = chunkListHead) {
+JS_ASSERT(!chunk->info.numArenasFreeCommitted);
+chunkListHead = chunk->info.next;
+FreeChunk(rt, chunk);
+}
+}
+void
+ChunkPool::expireAndFree(JSRuntime *rt, bool releaseAll)
+{
+FreeChunkList(rt, expire(rt, releaseAll));
+}
+/* static */ Chunk *
+Chunk::allocate(JSRuntime *rt)
+{
+Chunk *chunk = AllocChunk(rt);
+if (!chunk)
+return nullptr;
+chunk->init(rt);
+rt->gcStats.count(gcstats::STAT_NEW_CHUNK);
+return chunk;
+}
+/* Must be called with the GC lock taken. */
+/* static */ inline void
+Chunk::release(JSRuntime *rt, Chunk *chunk)
+{
+JS_ASSERT(chunk);
+chunk->prepareToBeFreed(rt);
+FreeChunk(rt, chunk);
+}
+inline void
+Chunk::prepareToBeFreed(JSRuntime *rt)
+{
+JS_ASSERT(rt->gcNumArenasFreeCommitted >= info.numArenasFreeCommitted);
+rt->gcNumArenasFreeCommitted -= info.numArenasFreeCommitted;
+rt->gcStats.count(gcstats::STAT_DESTROY_CHUNK);
+#ifdef DEBUG
+/*
+* Let FreeChunkList detect a missing prepareToBeFreed call before it
+* frees chunk.
+*/
+info.numArenasFreeCommitted = 0;
+#endif
+}
+void
+Chunk::init(JSRuntime *rt)
+{
+JS_POISON(this, JS_FRESH_TENURED_PATTERN, ChunkSize);
+/*
+* We clear the bitmap to guard against xpc_IsGrayGCThing being called on
+* uninitialized data, which would happen before the first GC cycle.
+*/
+bitmap.clear();
+/*
+* Decommit the arenas. We do this after poisoning so that if the OS does
+* not have to recycle the pages, we still get the benefit of poisoning.
+*/
+decommitAllArenas(rt);
+/* Initialize the chunk info. */
+info.age = 0;
+info.trailer.location = ChunkLocationTenuredHeap;
+info.trailer.runtime = rt;
+/* The rest of info fields are initialized in PickChunk. */
+}
+static inline Chunk **
+GetAvailableChunkList(Zone *zone)
+{
+JSRuntime *rt = zone->runtimeFromAnyThread();
+return zone->isSystem
+? &rt->gcSystemAvailableChunkListHead
+: &rt->gcUserAvailableChunkListHead;
+}
+inline void
+Chunk::addToAvailableList(Zone *zone)
+{
+insertToAvailableList(GetAvailableChunkList(zone));
+}
+inline void
+Chunk::insertToAvailableList(Chunk **insertPoint)
+{
+JS_ASSERT(hasAvailableArenas());
+JS_ASSERT(!info.prevp);
+JS_ASSERT(!info.next);
+info.prevp = insertPoint;
+Chunk *insertBefore = *insertPoint;
+if (insertBefore) {
+JS_ASSERT(insertBefore->info.prevp == insertPoint);
+insertBefore->info.prevp = &info.next;
+}
+info.next = insertBefore;
+*insertPoint = this;
+}
+inline void
+Chunk::removeFromAvailableList()
+{
+JS_ASSERT(info.prevp);
+*info.prevp = info.next;
+if (info.next) {
+JS_ASSERT(info.next->info.prevp == &info.next);
+info.next->info.prevp = info.prevp;
+}
+info.prevp = nullptr;
+info.next = nullptr;
+}
+/*
+* Search for and return the next decommitted Arena. Our goal is to keep
+* lastDecommittedArenaOffset "close" to a free arena. We do this by setting
+* it to the most recently freed arena when we free, and forcing it to
+* the last alloc + 1 when we allocate.
+*/
+uint32_t
+Chunk::findDecommittedArenaOffset()
+{
+/* Note: lastFreeArenaOffset can be past the end of the list. */
+for (unsigned i = info.lastDecommittedArenaOffset; i < ArenasPerChunk; i++)
+if (decommittedArenas.get(i))
+return i;
+for (unsigned i = 0; i < info.lastDecommittedArenaOffset; i++)
+if (decommittedArenas.get(i))
+return i;
+MOZ_ASSUME_UNREACHABLE("No decommitted arenas found.");
+}
+ArenaHeader *
+Chunk::fetchNextDecommittedArena()
+{
+JS_ASSERT(info.numArenasFreeCommitted == 0);
+JS_ASSERT(info.numArenasFree > 0);
+unsigned offset = findDecommittedArenaOffset();
+info.lastDecommittedArenaOffset = offset + 1;
+--info.numArenasFree;
+decommittedArenas.unset(offset);
+Arena *arena = &arenas[offset];
+MarkPagesInUse(info.trailer.runtime, arena, ArenaSize);
+arena->aheader.setAsNotAllocated();
+return &arena->aheader;
+}
+inline ArenaHeader *
+Chunk::fetchNextFreeArena(JSRuntime *rt)
+{
+JS_ASSERT(info.numArenasFreeCommitted > 0);
+JS_ASSERT(info.numArenasFreeCommitted <= info.numArenasFree);
+JS_ASSERT(info.numArenasFreeCommitted <= rt->gcNumArenasFreeCommitted);
+ArenaHeader *aheader = info.freeArenasHead;
+info.freeArenasHead = aheader->next;
+--info.numArenasFreeCommitted;
+--info.numArenasFree;
+--rt->gcNumArenasFreeCommitted;
+return aheader;
+}
+ArenaHeader *
+Chunk::allocateArena(Zone *zone, AllocKind thingKind)
+{
+JS_ASSERT(hasAvailableArenas());
+JSRuntime *rt = zone->runtimeFromAnyThread();
+if (!rt->isHeapMinorCollecting() && rt->gcBytes >= rt->gcMaxBytes)
+return nullptr;
+ArenaHeader *aheader = MOZ_LIKELY(info.numArenasFreeCommitted > 0)
+? fetchNextFreeArena(rt)
+: fetchNextDecommittedArena();
+aheader->init(zone, thingKind);
+if (MOZ_UNLIKELY(!hasAvailableArenas()))
+removeFromAvailableList();
+rt->gcBytes += ArenaSize;
+zone->gcBytes += ArenaSize;
+if (zone->gcBytes >= zone->gcTriggerBytes) {
+AutoUnlockGC unlock(rt);
+TriggerZoneGC(zone, JS::gcreason::ALLOC_TRIGGER);
+}
+return aheader;
+}
+inline void
+Chunk::addArenaToFreeList(JSRuntime *rt, ArenaHeader *aheader)
+{
+JS_ASSERT(!aheader->allocated());
+aheader->next = info.freeArenasHead;
+info.freeArenasHead = aheader;
+++info.numArenasFreeCommitted;
+++info.numArenasFree;
+++rt->gcNumArenasFreeCommitted;
+}
+void
+Chunk::recycleArena(ArenaHeader *aheader, ArenaList &dest, AllocKind thingKind)
+{
+aheader->getArena()->setAsFullyUnused(thingKind);
+dest.insert(aheader);
+}
+void
+Chunk::releaseArena(ArenaHeader *aheader)
+{
+JS_ASSERT(aheader->allocated());
+JS_ASSERT(!aheader->hasDelayedMarking);
+Zone *zone = aheader->zone;
+JSRuntime *rt = zone->runtimeFromAnyThread();
+AutoLockGC maybeLock;
+if (rt->gcHelperThread.sweeping())
+maybeLock.lock(rt);
+JS_ASSERT(rt->gcBytes >= ArenaSize);
+JS_ASSERT(zone->gcBytes >= ArenaSize);
+if (rt->gcHelperThread.sweeping())
+zone->reduceGCTriggerBytes(zone->gcHeapGrowthFactor * ArenaSize);
+rt->gcBytes -= ArenaSize;
+zone->gcBytes -= ArenaSize;
+aheader->setAsNotAllocated();
+addArenaToFreeList(rt, aheader);
+if (info.numArenasFree == 1) {
+JS_ASSERT(!info.prevp);
+JS_ASSERT(!info.next);
+addToAvailableList(zone);
+} else if (!unused()) {
+JS_ASSERT(info.prevp);
+} else {
+rt->gcChunkSet.remove(this);
+removeFromAvailableList();
+JS_ASSERT(info.numArenasFree == ArenasPerChunk);
+decommitAllArenas(rt);
+rt->gcChunkPool.put(this);
+}
+}
+/* The caller must hold the GC lock. */
+static Chunk *
+PickChunk(Zone *zone)
+{
+JSRuntime *rt = zone->runtimeFromAnyThread();
+Chunk **listHeadp = GetAvailableChunkList(zone);
+Chunk *chunk = *listHeadp;
+if (chunk)
+return chunk;
+chunk = rt->gcChunkPool.get(rt);
+if (!chunk)
+return nullptr;
+rt->gcChunkAllocationSinceLastGC = true;
+/*
+* FIXME bug 583732 - chunk is newly allocated and cannot be present in
+* the table so using ordinary lookupForAdd is suboptimal here.
+*/
+GCChunkSet::AddPtr p = rt->gcChunkSet.lookupForAdd(chunk);
+JS_ASSERT(!p);
+if (!rt->gcChunkSet.add(p, chunk)) {
+Chunk::release(rt, chunk);
+return nullptr;
+}
+chunk->info.prevp = nullptr;
+chunk->info.next = nullptr;
+chunk->addToAvailableList(zone);
+return chunk;
+}
+#ifdef JS_GC_ZEAL
+extern void
+js::SetGCZeal(JSRuntime *rt, uint8_t zeal, uint32_t frequency)
+{
+if (rt->gcVerifyPreData)
+VerifyBarriers(rt, PreBarrierVerifier);
+if (rt->gcVerifyPostData)
+VerifyBarriers(rt, PostBarrierVerifier);
+#ifdef JSGC_GENERATIONAL
+if (rt->gcZeal_ == ZealGenerationalGCValue) {
+MinorGC(rt, JS::gcreason::DEBUG_GC);
+rt->gcNursery.leaveZealMode();
+}
+if (zeal == ZealGenerationalGCValue)
+rt->gcNursery.enterZealMode();
+#endif
+bool schedule = zeal >= js::gc::ZealAllocValue;
+rt->gcZeal_ = zeal;
+rt->gcZealFrequency = frequency;
+rt->gcNextScheduled = schedule ? frequency : 0;
+}
+static bool
+InitGCZeal(JSRuntime *rt)
+{
+const char *env = getenv("JS_GC_ZEAL");
+if (!env)
+return true;
+int zeal = -1;
+int frequency = JS_DEFAULT_ZEAL_FREQ;
+if (strcmp(env, "help") != 0) {
+zeal = atoi(env);
+const char *p = strchr(env, ',');
+if (p)
+frequency = atoi(p + 1);
+}
+if (zeal < 0 || zeal > ZealLimit || frequency < 0) {
+fprintf(stderr,
+"Format: JS_GC_ZEAL=N[,F]\n"
+"N indicates \"zealousness\":\n"
+"  0: no additional GCs\n"
+"  1: additional GCs at common danger points\n"
+"  2: GC every F allocations (default: 100)\n"
+"  3: GC when the window paints (browser only)\n"
+"  4: Verify pre write barriers between instructions\n"
+"  5: Verify pre write barriers between paints\n"
+"  6: Verify stack rooting\n"
+"  7: Collect the nursery every N nursery allocations\n"
+"  8: Incremental GC in two slices: 1) mark roots 2) finish collection\n"
+"  9: Incremental GC in two slices: 1) mark all 2) new marking and finish\n"
+" 10: Incremental GC in multiple slices\n"
+" 11: Verify post write barriers between instructions\n"
+" 12: Verify post write barriers between paints\n"
+" 13: Purge analysis state every F allocations (default: 100)\n");
+return false;
+}
+SetGCZeal(rt, zeal, frequency);
+return true;
+}
+#endif
+/* Lifetime for type sets attached to scripts containing observed types. */
+static const int64_t JIT_SCRIPT_RELEASE_TYPES_INTERVAL = 60 * 1000 * 1000;
+bool
+js_InitGC(JSRuntime *rt, uint32_t maxbytes)
+{
+InitMemorySubsystem(rt);
+if (!rt->gcChunkSet.init(INITIAL_CHUNK_CAPACITY))
+return false;
+if (!rt->gcRootsHash.init(256))
+return false;
+if (!rt->gcHelperThread.init())
+return false;
+/*
+* Separate gcMaxMallocBytes from gcMaxBytes but initialize to maxbytes
+* for default backward API compatibility.
+*/
+rt->gcMaxBytes = maxbytes;
+rt->setGCMaxMallocBytes(maxbytes);
+#ifndef JS_MORE_DETERMINISTIC
+rt->gcJitReleaseTime = PRMJ_Now() + JIT_SCRIPT_RELEASE_TYPES_INTERVAL;
+#endif
+#ifdef JSGC_GENERATIONAL
+if (!rt->gcNursery.init())
+return false;
+if (!rt->gcStoreBuffer.enable())
+return false;
+#endif
+#ifdef JS_GC_ZEAL
+if (!InitGCZeal(rt))
+return false;
+#endif
+return true;
+}
+static void
+RecordNativeStackTopForGC(JSRuntime *rt)
+{
+ConservativeGCData *cgcd = &rt->conservativeGC;
+#ifdef JS_THREADSAFE
+/* Record the stack top here only if we are called from a request. */
+if (!rt->requestDepth)
+return;
+#endif
+cgcd->recordStackTop();
+}
+void
+js_FinishGC(JSRuntime *rt)
+{
+/*
+* Wait until the background finalization stops and the helper thread
+* shuts down before we forcefully release any remaining GC memory.
+*/
+rt->gcHelperThread.finish();
+#ifdef JS_GC_ZEAL
+/* Free memory associated with GC verification. */
+FinishVerifier(rt);
+#endif
+/* Delete all remaining zones. */
+if (rt->gcInitialized) {
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+for (CompartmentsInZoneIter comp(zone); !comp.done(); comp.next())
+js_delete(comp.get());
+js_delete(zone.get());
+}
+}
+rt->zones.clear();
+rt->gcSystemAvailableChunkListHead = nullptr;
+rt->gcUserAvailableChunkListHead = nullptr;
+if (rt->gcChunkSet.initialized()) {
+for (GCChunkSet::Range r(rt->gcChunkSet.all()); !r.empty(); r.popFront())
+Chunk::release(rt, r.front());
+rt->gcChunkSet.clear();
+}
+rt->gcChunkPool.expireAndFree(rt, true);
+if (rt->gcRootsHash.initialized())
+rt->gcRootsHash.clear();
+rt->functionPersistentRooteds.clear();
+rt->idPersistentRooteds.clear();
+rt->objectPersistentRooteds.clear();
+rt->scriptPersistentRooteds.clear();
+rt->stringPersistentRooteds.clear();
+rt->valuePersistentRooteds.clear();
+}
+template <typename T> struct BarrierOwner {};
+template <typename T> struct BarrierOwner<T *> { typedef T result; };
+template <> struct BarrierOwner<Value> { typedef HeapValue result; };
+template <typename T>
+static bool
+AddRoot(JSRuntime *rt, T *rp, const char *name, JSGCRootType rootType)
+{
+/*
+* Sometimes Firefox will hold weak references to objects and then convert
+* them to strong references by calling AddRoot (e.g., via PreserveWrapper,
+* or ModifyBusyCount in workers). We need a read barrier to cover these
+* cases.
+*/
+if (rt->gcIncrementalState != NO_INCREMENTAL)
+BarrierOwner<T>::result::writeBarrierPre(*rp);
+return rt->gcRootsHash.put((void *)rp, RootInfo(name, rootType));
+}
+template <typename T>
+static bool
+AddRoot(JSContext *cx, T *rp, const char *name, JSGCRootType rootType)
+{
+bool ok = AddRoot(cx->runtime(), rp, name, rootType);
+if (!ok)
+JS_ReportOutOfMemory(cx);
+return ok;
+}
+bool
+js::AddValueRoot(JSContext *cx, Value *vp, const char *name)
+{
+return AddRoot(cx, vp, name, JS_GC_ROOT_VALUE_PTR);
+}
+extern bool
+js::AddValueRootRT(JSRuntime *rt, js::Value *vp, const char *name)
+{
+return AddRoot(rt, vp, name, JS_GC_ROOT_VALUE_PTR);
+}
+extern bool
+js::AddStringRoot(JSContext *cx, JSString **rp, const char *name)
+{
+return AddRoot(cx, rp, name, JS_GC_ROOT_STRING_PTR);
+}
+extern bool
+js::AddObjectRoot(JSContext *cx, JSObject **rp, const char *name)
+{
+return AddRoot(cx, rp, name, JS_GC_ROOT_OBJECT_PTR);
+}
+extern bool
+js::AddObjectRoot(JSRuntime *rt, JSObject **rp, const char *name)
+{
+return AddRoot(rt, rp, name, JS_GC_ROOT_OBJECT_PTR);
+}
+extern bool
+js::AddScriptRoot(JSContext *cx, JSScript **rp, const char *name)
+{
+return AddRoot(cx, rp, name, JS_GC_ROOT_SCRIPT_PTR);
+}
+extern JS_FRIEND_API(bool)
+js::AddRawValueRoot(JSContext *cx, Value *vp, const char *name)
+{
+return AddRoot(cx, vp, name, JS_GC_ROOT_VALUE_PTR);
+}
+extern JS_FRIEND_API(void)
+js::RemoveRawValueRoot(JSContext *cx, Value *vp)
+{
+RemoveRoot(cx->runtime(), vp);
+}
+void
+js::RemoveRoot(JSRuntime *rt, void *rp)
+{
+rt->gcRootsHash.remove(rp);
+rt->gcPoke = true;
+}
+typedef RootedValueMap::Range RootRange;
+typedef RootedValueMap::Entry RootEntry;
+typedef RootedValueMap::Enum RootEnum;
+static size_t
+ComputeTriggerBytes(Zone *zone, size_t lastBytes, size_t maxBytes, JSGCInvocationKind gckind)
+{
+size_t base = gckind == GC_SHRINK ? lastBytes : Max(lastBytes, zone->runtimeFromMainThread()->gcAllocationThreshold);
+double trigger = double(base) * zone->gcHeapGrowthFactor;
+return size_t(Min(double(maxBytes), trigger));
+}
+void
+Zone::setGCLastBytes(size_t lastBytes, JSGCInvocationKind gckind)
+{
+/*
+* The heap growth factor depends on the heap size after a GC and the GC frequency.
+* For low frequency GCs (more than 1sec between GCs) we let the heap grow to 150%.
+* For high frequency GCs we let the heap grow depending on the heap size:
+*   lastBytes < highFrequencyLowLimit: 300%
+*   lastBytes > highFrequencyHighLimit: 150%
+*   otherwise: linear interpolation between 150% and 300% based on lastBytes
+*/
+JSRuntime *rt = runtimeFromMainThread();
+if (!rt->gcDynamicHeapGrowth) {
+gcHeapGrowthFactor = 3.0;
+} else if (lastBytes < 1 * 1024 * 1024) {
+gcHeapGrowthFactor = rt->gcLowFrequencyHeapGrowth;
+} else {
+JS_ASSERT(rt->gcHighFrequencyHighLimitBytes > rt->gcHighFrequencyLowLimitBytes);
+uint64_t now = PRMJ_Now();
+if (rt->gcLastGCTime && rt->gcLastGCTime + rt->gcHighFrequencyTimeThreshold * PRMJ_USEC_PER_MSEC > now) {
+if (lastBytes <= rt->gcHighFrequencyLowLimitBytes) {
+gcHeapGrowthFactor = rt->gcHighFrequencyHeapGrowthMax;
+} else if (lastBytes >= rt->gcHighFrequencyHighLimitBytes) {
+gcHeapGrowthFactor = rt->gcHighFrequencyHeapGrowthMin;
+} else {
+double k = (rt->gcHighFrequencyHeapGrowthMin - rt->gcHighFrequencyHeapGrowthMax)
+/ (double)(rt->gcHighFrequencyHighLimitBytes - rt->gcHighFrequencyLowLimitBytes);
+gcHeapGrowthFactor = (k * (lastBytes - rt->gcHighFrequencyLowLimitBytes)
++ rt->gcHighFrequencyHeapGrowthMax);
+JS_ASSERT(gcHeapGrowthFactor <= rt->gcHighFrequencyHeapGrowthMax
+&& gcHeapGrowthFactor >= rt->gcHighFrequencyHeapGrowthMin);
+}
+rt->gcHighFrequencyGC = true;
+} else {
+gcHeapGrowthFactor = rt->gcLowFrequencyHeapGrowth;
+rt->gcHighFrequencyGC = false;
+}
+}
+gcTriggerBytes = ComputeTriggerBytes(this, lastBytes, rt->gcMaxBytes, gckind);
+}
+void
+Zone::reduceGCTriggerBytes(size_t amount)
+{
+JS_ASSERT(amount > 0);
+JS_ASSERT(gcTriggerBytes >= amount);
+if (gcTriggerBytes - amount < runtimeFromAnyThread()->gcAllocationThreshold * gcHeapGrowthFactor)
+return;
+gcTriggerBytes -= amount;
+}
+Allocator::Allocator(Zone *zone)
+: zone_(zone)
+{}
+inline void
+GCMarker::delayMarkingArena(ArenaHeader *aheader)
+{
+if (aheader->hasDelayedMarking) {
+/* Arena already scheduled to be marked later */
+return;
+}
+aheader->setNextDelayedMarking(unmarkedArenaStackTop);
+unmarkedArenaStackTop = aheader;
+markLaterArenas++;
+}
+void
+GCMarker::delayMarkingChildren(const void *thing)
+{
+const Cell *cell = reinterpret_cast<const Cell *>(thing);
+cell->arenaHeader()->markOverflow = 1;
+delayMarkingArena(cell->arenaHeader());
+}
+inline void
+ArenaLists::prepareForIncrementalGC(JSRuntime *rt)
+{
+for (size_t i = 0; i != FINALIZE_LIMIT; ++i) {
+FreeSpan *headSpan = &freeLists[i];
+if (!headSpan->isEmpty()) {
+ArenaHeader *aheader = headSpan->arenaHeader();
+aheader->allocatedDuringIncremental = true;
+rt->gcMarker.delayMarkingArena(aheader);
+}
+}
+}
+static inline void
+PushArenaAllocatedDuringSweep(JSRuntime *runtime, ArenaHeader *arena)
+{
+arena->setNextAllocDuringSweep(runtime->gcArenasAllocatedDuringSweep);
+runtime->gcArenasAllocatedDuringSweep = arena;
+}
+inline void *
+ArenaLists::allocateFromArenaInline(Zone *zone, AllocKind thingKind)
+{
+/*
+* Parallel JS Note:
+*
+* This function can be called from parallel threads all of which
+* are associated with the same compartment. In that case, each
+* thread will have a distinct ArenaLists.  Therefore, whenever we
+* fall through to PickChunk() we must be sure that we are holding
+* a lock.
+*/
+Chunk *chunk = nullptr;
+ArenaList *al = &arenaLists[thingKind];
+AutoLockGC maybeLock;
+#ifdef JS_THREADSAFE
+volatile uintptr_t *bfs = &backgroundFinalizeState[thingKind];
+if (*bfs != BFS_DONE) {
+/*
+* We cannot search the arena list for free things while the
+* background finalization runs and can modify head or cursor at any
+* moment. So we always allocate a new arena in that case.
+*/
+maybeLock.lock(zone->runtimeFromAnyThread());
+if (*bfs == BFS_RUN) {
+JS_ASSERT(!*al->cursor);
+chunk = PickChunk(zone);
+if (!chunk) {
+/*
+* Let the caller to wait for the background allocation to
+* finish and restart the allocation attempt.
+*/
+return nullptr;
+}
+} else if (*bfs == BFS_JUST_FINISHED) {
+/* See comments before BackgroundFinalizeState definition. */
+*bfs = BFS_DONE;
+} else {
+JS_ASSERT(*bfs == BFS_DONE);
+}
+}
+#endif /* JS_THREADSAFE */
+if (!chunk) {
+if (ArenaHeader *aheader = *al->cursor) {
+JS_ASSERT(aheader->hasFreeThings());
+/*
+* Normally, the empty arenas are returned to the chunk
+* and should not present on the list. In parallel
+* execution, however, we keep empty arenas in the arena
+* list to avoid synchronizing on the chunk.
+*/
+JS_ASSERT(!aheader->isEmpty() || InParallelSection());
+al->cursor = &aheader->next;
+/*
+* Move the free span stored in the arena to the free list and
+* allocate from it.
+*/
+freeLists[thingKind] = aheader->getFirstFreeSpan();
+aheader->setAsFullyUsed();
+if (MOZ_UNLIKELY(zone->wasGCStarted())) {
+if (zone->needsBarrier()) {
+aheader->allocatedDuringIncremental = true;
+zone->runtimeFromMainThread()->gcMarker.delayMarkingArena(aheader);
+} else if (zone->isGCSweeping()) {
+PushArenaAllocatedDuringSweep(zone->runtimeFromMainThread(), aheader);
+}
+}
+return freeLists[thingKind].infallibleAllocate(Arena::thingSize(thingKind));
+}
+/* Make sure we hold the GC lock before we call PickChunk. */
+if (!maybeLock.locked())
+maybeLock.lock(zone->runtimeFromAnyThread());
+chunk = PickChunk(zone);
+if (!chunk)
+return nullptr;
+}
+/*
+* While we still hold the GC lock get an arena from some chunk, mark it
+* as full as its single free span is moved to the free lits, and insert
+* it to the list as a fully allocated arena.
+*
+* We add the arena before the the head, not after the tail pointed by the
+* cursor, so after the GC the most recently added arena will be used first
+* for allocations improving cache locality.
+*/
+JS_ASSERT(!*al->cursor);
+ArenaHeader *aheader = chunk->allocateArena(zone, thingKind);
+if (!aheader)
+return nullptr;
+if (MOZ_UNLIKELY(zone->wasGCStarted())) {
+if (zone->needsBarrier()) {
+aheader->allocatedDuringIncremental = true;
+zone->runtimeFromMainThread()->gcMarker.delayMarkingArena(aheader);
+} else if (zone->isGCSweeping()) {
+PushArenaAllocatedDuringSweep(zone->runtimeFromMainThread(), aheader);
+}
+}
+aheader->next = al->head;
+if (!al->head) {
+JS_ASSERT(al->cursor == &al->head);
+al->cursor = &aheader->next;
+}
+al->head = aheader;
+/* See comments before allocateFromNewArena about this assert. */
+JS_ASSERT(!aheader->hasFreeThings());
+uintptr_t arenaAddr = aheader->arenaAddress();
+return freeLists[thingKind].allocateFromNewArena(arenaAddr,
+Arena::firstThingOffset(thingKind),
+Arena::thingSize(thingKind));
+}
+void *
+ArenaLists::allocateFromArena(JS::Zone *zone, AllocKind thingKind)
+{
+return allocateFromArenaInline(zone, thingKind);
+}
+void
+ArenaLists::wipeDuringParallelExecution(JSRuntime *rt)
+{
+JS_ASSERT(InParallelSection());
+// First, check that we all objects we have allocated are eligible
+// for background finalization. The idea is that we will free
+// (below) ALL background finalizable objects, because we know (by
+// the rules of parallel execution) they are not reachable except
+// by other thread-local objects. However, if there were any
+// object ineligible for background finalization, it might retain
+// a reference to one of these background finalizable objects, and
+// that'd be bad.
+for (unsigned i = 0; i < FINALIZE_LAST; i++) {
+AllocKind thingKind = AllocKind(i);
+if (!IsBackgroundFinalized(thingKind) && arenaLists[thingKind].head)
+return;
+}
+// Finalize all background finalizable objects immediately and
+// return the (now empty) arenas back to arena list.
+FreeOp fop(rt, false);
+for (unsigned i = 0; i < FINALIZE_OBJECT_LAST; i++) {
+AllocKind thingKind = AllocKind(i);
+if (!IsBackgroundFinalized(thingKind))
+continue;
+if (arenaLists[i].head) {
+purge(thingKind);
+forceFinalizeNow(&fop, thingKind);
+}
+}
+}
+void
+ArenaLists::finalizeNow(FreeOp *fop, AllocKind thingKind)
+{
+JS_ASSERT(!IsBackgroundFinalized(thingKind));
+forceFinalizeNow(fop, thingKind);
+}
+void
+ArenaLists::forceFinalizeNow(FreeOp *fop, AllocKind thingKind)
+{
+JS_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
+ArenaHeader *arenas = arenaLists[thingKind].head;
+arenaLists[thingKind].clear();
+SliceBudget budget;
+FinalizeArenas(fop, &arenas, arenaLists[thingKind], thingKind, budget);
+JS_ASSERT(!arenas);
+}
+void
+ArenaLists::queueForForegroundSweep(FreeOp *fop, AllocKind thingKind)
+{
+JS_ASSERT(!IsBackgroundFinalized(thingKind));
+JS_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
+JS_ASSERT(!arenaListsToSweep[thingKind]);
+arenaListsToSweep[thingKind] = arenaLists[thingKind].head;
+arenaLists[thingKind].clear();
+}
+inline void
+ArenaLists::queueForBackgroundSweep(FreeOp *fop, AllocKind thingKind)
+{
+JS_ASSERT(IsBackgroundFinalized(thingKind));
+#ifdef JS_THREADSAFE
+JS_ASSERT(!fop->runtime()->gcHelperThread.sweeping());
+#endif
+ArenaList *al = &arenaLists[thingKind];
+if (!al->head) {
+JS_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE);
+JS_ASSERT(al->cursor == &al->head);
+return;
+}
+/*
+* The state can be done, or just-finished if we have not allocated any GC
+* things from the arena list after the previous background finalization.
+*/
+JS_ASSERT(backgroundFinalizeState[thingKind] == BFS_DONE ||
+backgroundFinalizeState[thingKind] == BFS_JUST_FINISHED);
+arenaListsToSweep[thingKind] = al->head;
+al->clear();
+backgroundFinalizeState[thingKind] = BFS_RUN;
+}
+/*static*/ void
+ArenaLists::backgroundFinalize(FreeOp *fop, ArenaHeader *listHead, bool onBackgroundThread)
+{
+JS_ASSERT(listHead);
+AllocKind thingKind = listHead->getAllocKind();
+Zone *zone = listHead->zone;
+ArenaList finalized;
+SliceBudget budget;
+FinalizeArenas(fop, &listHead, finalized, thingKind, budget);
+JS_ASSERT(!listHead);
+/*
+* After we finish the finalization al->cursor must point to the end of
+* the head list as we emptied the list before the background finalization
+* and the allocation adds new arenas before the cursor.
+*/
+ArenaLists *lists = &zone->allocator.arenas;
+ArenaList *al = &lists->arenaLists[thingKind];
+AutoLockGC lock(fop->runtime());
+JS_ASSERT(lists->backgroundFinalizeState[thingKind] == BFS_RUN);
+JS_ASSERT(!*al->cursor);
+if (finalized.head) {
+*al->cursor = finalized.head;
+if (finalized.cursor != &finalized.head)
+al->cursor = finalized.cursor;
+}
+/*
+* We must set the state to BFS_JUST_FINISHED if we are running on the
+* background thread and we have touched arenaList list, even if we add to
+* the list only fully allocated arenas without any free things. It ensures
+* that the allocation thread takes the GC lock and all writes to the free
+* list elements are propagated. As we always take the GC lock when
+* allocating new arenas from the chunks we can set the state to BFS_DONE if
+* we have released all finalized arenas back to their chunks.
+*/
+if (onBackgroundThread && finalized.head)
+lists->backgroundFinalizeState[thingKind] = BFS_JUST_FINISHED;
+else
+lists->backgroundFinalizeState[thingKind] = BFS_DONE;
+lists->arenaListsToSweep[thingKind] = nullptr;
+}
+void
+ArenaLists::queueObjectsForSweep(FreeOp *fop)
+{
+gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_OBJECT);
+finalizeNow(fop, FINALIZE_OBJECT0);
+finalizeNow(fop, FINALIZE_OBJECT2);
+finalizeNow(fop, FINALIZE_OBJECT4);
+finalizeNow(fop, FINALIZE_OBJECT8);
+finalizeNow(fop, FINALIZE_OBJECT12);
+finalizeNow(fop, FINALIZE_OBJECT16);
+queueForBackgroundSweep(fop, FINALIZE_OBJECT0_BACKGROUND);
+queueForBackgroundSweep(fop, FINALIZE_OBJECT2_BACKGROUND);
+queueForBackgroundSweep(fop, FINALIZE_OBJECT4_BACKGROUND);
+queueForBackgroundSweep(fop, FINALIZE_OBJECT8_BACKGROUND);
+queueForBackgroundSweep(fop, FINALIZE_OBJECT12_BACKGROUND);
+queueForBackgroundSweep(fop, FINALIZE_OBJECT16_BACKGROUND);
+}
+void
+ArenaLists::queueStringsForSweep(FreeOp *fop)
+{
+gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_STRING);
+queueForBackgroundSweep(fop, FINALIZE_FAT_INLINE_STRING);
+queueForBackgroundSweep(fop, FINALIZE_STRING);
+queueForForegroundSweep(fop, FINALIZE_EXTERNAL_STRING);
+}
+void
+ArenaLists::queueScriptsForSweep(FreeOp *fop)
+{
+gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_SCRIPT);
+queueForForegroundSweep(fop, FINALIZE_SCRIPT);
+queueForForegroundSweep(fop, FINALIZE_LAZY_SCRIPT);
+}
+void
+ArenaLists::queueJitCodeForSweep(FreeOp *fop)
+{
+gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_JITCODE);
+queueForForegroundSweep(fop, FINALIZE_JITCODE);
+}
+void
+ArenaLists::queueShapesForSweep(FreeOp *fop)
+{
+gcstats::AutoPhase ap(fop->runtime()->gcStats, gcstats::PHASE_SWEEP_SHAPE);
+queueForBackgroundSweep(fop, FINALIZE_SHAPE);
+queueForBackgroundSweep(fop, FINALIZE_BASE_SHAPE);
+queueForBackgroundSweep(fop, FINALIZE_TYPE_OBJECT);
+}
+static void *
+RunLastDitchGC(JSContext *cx, JS::Zone *zone, AllocKind thingKind)
+{
+/*
+* In parallel sections, we do not attempt to refill the free list
+* and hence do not encounter last ditch GC.
+*/
+JS_ASSERT(!InParallelSection());
+PrepareZoneForGC(zone);
+JSRuntime *rt = cx->runtime();
+/* The last ditch GC preserves all atoms. */
+AutoKeepAtoms keepAtoms(cx->perThreadData);
+GC(rt, GC_NORMAL, JS::gcreason::LAST_DITCH);
+/*
+* The JSGC_END callback can legitimately allocate new GC
+* things and populate the free list. If that happens, just
+* return that list head.
+*/
+size_t thingSize = Arena::thingSize(thingKind);
+if (void *thing = zone->allocator.arenas.allocateFromFreeList(thingKind, thingSize))
+return thing;
+return nullptr;
+}
+template <AllowGC allowGC>
+/* static */ void *
+ArenaLists::refillFreeList(ThreadSafeContext *cx, AllocKind thingKind)
+{
+JS_ASSERT(cx->allocator()->arenas.freeLists[thingKind].isEmpty());
+JS_ASSERT_IF(cx->isJSContext(), !cx->asJSContext()->runtime()->isHeapBusy());
+Zone *zone = cx->allocator()->zone_;
+bool runGC = cx->allowGC() && allowGC &&
+cx->asJSContext()->runtime()->gcIncrementalState != NO_INCREMENTAL &&
+zone->gcBytes > zone->gcTriggerBytes;
+#ifdef JS_THREADSAFE
+JS_ASSERT_IF(cx->isJSContext() && allowGC,
+!cx->asJSContext()->runtime()->currentThreadHasExclusiveAccess());
+#endif
+for (;;) {
+if (MOZ_UNLIKELY(runGC)) {
+if (void *thing = RunLastDitchGC(cx->asJSContext(), zone, thingKind))
+return thing;
+}
+if (cx->isJSContext()) {
+/*
+* allocateFromArena may fail while the background finalization still
+* run. If we are on the main thread, we want to wait for it to finish
+* and restart. However, checking for that is racy as the background
+* finalization could free some things after allocateFromArena decided
+* to fail but at this point it may have already stopped. To avoid
+* this race we always try to allocate twice.
+*/
+for (bool secondAttempt = false; ; secondAttempt = true) {
+void *thing = cx->allocator()->arenas.allocateFromArenaInline(zone, thingKind);
+if (MOZ_LIKELY(!!thing))
+return thing;
+if (secondAttempt)
+break;
+cx->asJSContext()->runtime()->gcHelperThread.waitBackgroundSweepEnd();
+}
+} else {
+#ifdef JS_THREADSAFE
+/*
+* If we're off the main thread, we try to allocate once and
+* return whatever value we get. If we aren't in a ForkJoin
+* session (i.e. we are in a worker thread async with the main
+* thread), we need to first ensure the main thread is not in a GC
+* session.
+*/
+mozilla::Maybe<AutoLockWorkerThreadState> lock;
+JSRuntime *rt = zone->runtimeFromAnyThread();
+if (rt->exclusiveThreadsPresent()) {
+lock.construct();
+while (rt->isHeapBusy())
+WorkerThreadState().wait(GlobalWorkerThreadState::PRODUCER);
+}
+void *thing = cx->allocator()->arenas.allocateFromArenaInline(zone, thingKind);
+if (thing)
+return thing;
+#else
+MOZ_CRASH();
+#endif
+}
+if (!cx->allowGC() || !allowGC)
+return nullptr;
+/*
+* We failed to allocate. Run the GC if we haven't done it already.
+* Otherwise report OOM.
+*/
+if (runGC)
+break;
+runGC = true;
+}
+JS_ASSERT(allowGC);
+js_ReportOutOfMemory(cx);
+return nullptr;
+}
+template void *
+ArenaLists::refillFreeList<NoGC>(ThreadSafeContext *cx, AllocKind thingKind);
+template void *
+ArenaLists::refillFreeList<CanGC>(ThreadSafeContext *cx, AllocKind thingKind);
+JSGCTraceKind
+js_GetGCThingTraceKind(void *thing)
+{
+return GetGCThingTraceKind(thing);
+}
+/* static */ int64_t
+SliceBudget::TimeBudget(int64_t millis)
+{
+return millis * PRMJ_USEC_PER_MSEC;
+}
+/* static */ int64_t
+SliceBudget::WorkBudget(int64_t work)
+{
+/* For work = 0 not to mean Unlimited, we subtract 1. */
+return -work - 1;
+}
+SliceBudget::SliceBudget()
+: deadline(INT64_MAX),
+counter(INTPTR_MAX)
+{
+}
+SliceBudget::SliceBudget(int64_t budget)
+{
+if (budget == Unlimited) {
+deadline = INT64_MAX;
+counter = INTPTR_MAX;
+} else if (budget > 0) {
+deadline = PRMJ_Now() + budget;
+counter = CounterReset;
+} else {
+deadline = 0;
+counter = -budget - 1;
+}
+}
+bool
+SliceBudget::checkOverBudget()
+{
+bool over = PRMJ_Now() > deadline;
+if (!over)
+counter = CounterReset;
+return over;
+}
+void
+js::MarkCompartmentActive(InterpreterFrame *fp)
+{
+fp->script()->compartment()->zone()->active = true;
+}
+static void
+RequestInterrupt(JSRuntime *rt, JS::gcreason::Reason reason)
+{
+if (rt->gcIsNeeded)
+return;
+rt->gcIsNeeded = true;
+rt->gcTriggerReason = reason;
+rt->requestInterrupt(JSRuntime::RequestInterruptMainThread);
+}
+bool
+js::TriggerGC(JSRuntime *rt, JS::gcreason::Reason reason)
+{
+/* Wait till end of parallel section to trigger GC. */
+if (InParallelSection()) {
+ForkJoinContext::current()->requestGC(reason);
+return true;
+}
+/* Don't trigger GCs when allocating under the interrupt callback lock. */
+if (rt->currentThreadOwnsInterruptLock())
+return false;
+JS_ASSERT(CurrentThreadCanAccessRuntime(rt));
+/* GC is already running. */
+if (rt->isHeapCollecting())
+return false;
+JS::PrepareForFullGC(rt);
+RequestInterrupt(rt, reason);
+return true;
+}
+bool
+js::TriggerZoneGC(Zone *zone, JS::gcreason::Reason reason)
+{
+/*
+* If parallel threads are running, wait till they
+* are stopped to trigger GC.
+*/
+if (InParallelSection()) {
+ForkJoinContext::current()->requestZoneGC(zone, reason);
+return true;
+}
+/* Zones in use by a thread with an exclusive context can't be collected. */
+if (zone->usedByExclusiveThread)
+return false;
+JSRuntime *rt = zone->runtimeFromMainThread();
+/* Don't trigger GCs when allocating under the interrupt callback lock. */
+if (rt->currentThreadOwnsInterruptLock())
+return false;
+/* GC is already running. */
+if (rt->isHeapCollecting())
+return false;
+if (rt->gcZeal() == ZealAllocValue) {
+TriggerGC(rt, reason);
+return true;
+}
+if (rt->isAtomsZone(zone)) {
+/* We can't do a zone GC of the atoms compartment. */
+TriggerGC(rt, reason);
+return true;
+}
+PrepareZoneForGC(zone);
+RequestInterrupt(rt, reason);
+return true;
+}
+void
+js::MaybeGC(JSContext *cx)
+{
+JSRuntime *rt = cx->runtime();
+JS_ASSERT(CurrentThreadCanAccessRuntime(rt));
+if (rt->gcZeal() == ZealAllocValue || rt->gcZeal() == ZealPokeValue) {
+JS::PrepareForFullGC(rt);
+GC(rt, GC_NORMAL, JS::gcreason::MAYBEGC);
+return;
+}
+if (rt->gcIsNeeded) {
+GCSlice(rt, GC_NORMAL, JS::gcreason::MAYBEGC);
+return;
+}
+double factor = rt->gcHighFrequencyGC ? 0.85 : 0.9;
+Zone *zone = cx->zone();
+if (zone->gcBytes > 1024 * 1024 &&
+zone->gcBytes >= factor * zone->gcTriggerBytes &&
+rt->gcIncrementalState == NO_INCREMENTAL &&
+!rt->gcHelperThread.sweeping())
+{
+PrepareZoneForGC(zone);
+GCSlice(rt, GC_NORMAL, JS::gcreason::MAYBEGC);
+return;
+}
+#ifndef JS_MORE_DETERMINISTIC
+/*
+* Access to the counters and, on 32 bit, setting gcNextFullGCTime below
+* is not atomic and a race condition could trigger or suppress the GC. We
+* tolerate this.
+*/
+int64_t now = PRMJ_Now();
+if (rt->gcNextFullGCTime && rt->gcNextFullGCTime <= now) {
+if (rt->gcChunkAllocationSinceLastGC ||
+rt->gcNumArenasFreeCommitted > rt->gcDecommitThreshold)
+{
+JS::PrepareForFullGC(rt);
+GCSlice(rt, GC_SHRINK, JS::gcreason::MAYBEGC);
+} else {
+rt->gcNextFullGCTime = now + GC_IDLE_FULL_SPAN;
+}
+}
+#endif
+}
+static void
+DecommitArenasFromAvailableList(JSRuntime *rt, Chunk **availableListHeadp)
+{
+Chunk *chunk = *availableListHeadp;
+if (!chunk)
+return;
+/*
+* Decommit is expensive so we avoid holding the GC lock while calling it.
+*
+* We decommit from the tail of the list to minimize interference with the
+* main thread that may start to allocate things at this point.
+*
+* The arena that is been decommitted outside the GC lock must not be
+* available for allocations either via the free list or via the
+* decommittedArenas bitmap. For that we just fetch the arena from the
+* free list before the decommit pretending as it was allocated. If this
+* arena also is the single free arena in the chunk, then we must remove
+* from the available list before we release the lock so the allocation
+* thread would not see chunks with no free arenas on the available list.
+*
+* After we retake the lock, we mark the arena as free and decommitted if
+* the decommit was successful. We must also add the chunk back to the
+* available list if we removed it previously or when the main thread
+* have allocated all remaining free arenas in the chunk.
+*
+* We also must make sure that the aheader is not accessed again after we
+* decommit the arena.
+*/
+JS_ASSERT(chunk->info.prevp == availableListHeadp);
+while (Chunk *next = chunk->info.next) {
+JS_ASSERT(next->info.prevp == &chunk->info.next);
+chunk = next;
+}
+for (;;) {
+while (chunk->info.numArenasFreeCommitted != 0) {
+ArenaHeader *aheader = chunk->fetchNextFreeArena(rt);
+Chunk **savedPrevp = chunk->info.prevp;
+if (!chunk->hasAvailableArenas())
+chunk->removeFromAvailableList();
+size_t arenaIndex = Chunk::arenaIndex(aheader->arenaAddress());
+bool ok;
+{
+/*
+* If the main thread waits for the decommit to finish, skip
+* potentially expensive unlock/lock pair on the contested
+* lock.
+*/
+Maybe<AutoUnlockGC> maybeUnlock;
+if (!rt->isHeapBusy())
+maybeUnlock.construct(rt);
+ok = MarkPagesUnused(rt, aheader->getArena(), ArenaSize);
+}
+if (ok) {
+++chunk->info.numArenasFree;
+chunk->decommittedArenas.set(arenaIndex);
+} else {
+chunk->addArenaToFreeList(rt, aheader);
+}
+JS_ASSERT(chunk->hasAvailableArenas());
+JS_ASSERT(!chunk->unused());
+if (chunk->info.numArenasFree == 1) {
+/*
+* Put the chunk back to the available list either at the
+* point where it was before to preserve the available list
+* that we enumerate, or, when the allocation thread has fully
+* used all the previous chunks, at the beginning of the
+* available list.
+*/
+Chunk **insertPoint = savedPrevp;
+if (savedPrevp != availableListHeadp) {
+Chunk *prev = Chunk::fromPointerToNext(savedPrevp);
+if (!prev->hasAvailableArenas())
+insertPoint = availableListHeadp;
+}
+chunk->insertToAvailableList(insertPoint);
+} else {
+JS_ASSERT(chunk->info.prevp);
+}
+if (rt->gcChunkAllocationSinceLastGC || !ok) {
+/*
+* The allocator thread has started to get new chunks. We should stop
+* to avoid decommitting arenas in just allocated chunks.
+*/
+return;
+}
+}
+/*
+* chunk->info.prevp becomes null when the allocator thread consumed
+* all chunks from the available list.
+*/
+JS_ASSERT_IF(chunk->info.prevp, *chunk->info.prevp == chunk);
+if (chunk->info.prevp == availableListHeadp || !chunk->info.prevp)
+break;
+/*
+* prevp exists and is not the list head. It must point to the next
+* field of the previous chunk.
+*/
+chunk = chunk->getPrevious();
+}
+}
+static void
+DecommitArenas(JSRuntime *rt)
+{
+DecommitArenasFromAvailableList(rt, &rt->gcSystemAvailableChunkListHead);
+DecommitArenasFromAvailableList(rt, &rt->gcUserAvailableChunkListHead);
+}
+/* Must be called with the GC lock taken. */
+static void
+ExpireChunksAndArenas(JSRuntime *rt, bool shouldShrink)
+{
+if (Chunk *toFree = rt->gcChunkPool.expire(rt, shouldShrink)) {
+AutoUnlockGC unlock(rt);
+FreeChunkList(rt, toFree);
+}
+if (shouldShrink)
+DecommitArenas(rt);
+}
+static void
+SweepBackgroundThings(JSRuntime* rt, bool onBackgroundThread)
+{
+/*
+* We must finalize in the correct order, see comments in
+* finalizeObjects.
+*/
+FreeOp fop(rt, false);
+for (int phase = 0 ; phase < BackgroundPhaseCount ; ++phase) {
+for (Zone *zone = rt->gcSweepingZones; zone; zone = zone->gcNextGraphNode) {
+for (int index = 0 ; index < BackgroundPhaseLength[phase] ; ++index) {
+AllocKind kind = BackgroundPhases[phase][index];
+ArenaHeader *arenas = zone->allocator.arenas.arenaListsToSweep[kind];
+if (arenas)
+ArenaLists::backgroundFinalize(&fop, arenas, onBackgroundThread);
+}
+}
+}
+rt->gcSweepingZones = nullptr;
+}
+#ifdef JS_THREADSAFE
+static void
+AssertBackgroundSweepingFinished(JSRuntime *rt)
+{
+JS_ASSERT(!rt->gcSweepingZones);
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+for (unsigned i = 0; i < FINALIZE_LIMIT; ++i) {
+JS_ASSERT(!zone->allocator.arenas.arenaListsToSweep[i]);
+JS_ASSERT(zone->allocator.arenas.doneBackgroundFinalize(AllocKind(i)));
+}
+}
+}
+unsigned
+js::GetCPUCount()
+{
+static unsigned ncpus = 0;
+if (ncpus == 0) {
+# ifdef XP_WIN
+SYSTEM_INFO sysinfo;
+GetSystemInfo(&sysinfo);
+ncpus = unsigned(sysinfo.dwNumberOfProcessors);
+# else
+long n = sysconf(_SC_NPROCESSORS_ONLN);
+ncpus = (n > 0) ? unsigned(n) : 1;
+# endif
+}
+return ncpus;
+}
+#endif /* JS_THREADSAFE */
+bool
+GCHelperThread::init()
+{
+if (!rt->useHelperThreads()) {
+backgroundAllocation = false;
+return true;
+}
+#ifdef JS_THREADSAFE
+if (!(wakeup = PR_NewCondVar(rt->gcLock)))
+return false;
+if (!(done = PR_NewCondVar(rt->gcLock)))
+return false;
+thread = PR_CreateThread(PR_USER_THREAD, threadMain, this, PR_PRIORITY_NORMAL,
+PR_GLOBAL_THREAD, PR_JOINABLE_THREAD, 0);
+if (!thread)
+return false;
+backgroundAllocation = (GetCPUCount() >= 2);
+#endif /* JS_THREADSAFE */
+return true;
+}
+void
+GCHelperThread::finish()
+{
+if (!rt->useHelperThreads() || !rt->gcLock) {
+JS_ASSERT(state == IDLE);
+return;
+}
+#ifdef JS_THREADSAFE
+PRThread *join = nullptr;
+{
+AutoLockGC lock(rt);
+if (thread && state != SHUTDOWN) {
+/*
+* We cannot be in the ALLOCATING or CANCEL_ALLOCATION states as
+* the allocations should have been stopped during the last GC.
+*/
+JS_ASSERT(state == IDLE || state == SWEEPING);
+if (state == IDLE)
+PR_NotifyCondVar(wakeup);
+state = SHUTDOWN;
+join = thread;
+}
+}
+if (join) {
+/* PR_DestroyThread is not necessary. */
+PR_JoinThread(join);
+}
+if (wakeup)
+PR_DestroyCondVar(wakeup);
+if (done)
+PR_DestroyCondVar(done);
+#endif /* JS_THREADSAFE */
+}
+#ifdef JS_THREADSAFE
+#ifdef MOZ_NUWA_PROCESS
+extern "C" {
+MFBT_API bool IsNuwaProcess();
+MFBT_API void NuwaMarkCurrentThread(void (*recreate)(void *), void *arg);
+}
+#endif
+/* static */
+void
+GCHelperThread::threadMain(void *arg)
+{
+PR_SetCurrentThreadName("JS GC Helper");
+#ifdef MOZ_NUWA_PROCESS
+if (IsNuwaProcess && IsNuwaProcess()) {
+JS_ASSERT(NuwaMarkCurrentThread != nullptr);
+NuwaMarkCurrentThread(nullptr, nullptr);
+}
+#endif
+static_cast<GCHelperThread *>(arg)->threadLoop();
+}
+void
+GCHelperThread::wait(PRCondVar *which)
+{
+rt->gcLockOwner = nullptr;
+PR_WaitCondVar(which, PR_INTERVAL_NO_TIMEOUT);
+#ifdef DEBUG
+rt->gcLockOwner = PR_GetCurrentThread();
+#endif
+}
+void
+GCHelperThread::threadLoop()
+{
+AutoLockGC lock(rt);
+TraceLogger *logger = TraceLoggerForCurrentThread();
+/*
+* Even on the first iteration the state can be SHUTDOWN or SWEEPING if
+* the stop request or the GC and the corresponding startBackgroundSweep call
+* happen before this thread has a chance to run.
+*/
+for (;;) {
+switch (state) {
+case SHUTDOWN:
+return;
+case IDLE:
+wait(wakeup);
+break;
+case SWEEPING: {
+AutoTraceLog logSweeping(logger, TraceLogger::GCSweeping);
+doSweep();
+if (state == SWEEPING)
+state = IDLE;
+PR_NotifyAllCondVar(done);
+break;
+}
+case ALLOCATING: {
+AutoTraceLog logAllocating(logger, TraceLogger::GCAllocation);
+do {
+Chunk *chunk;
+{
+AutoUnlockGC unlock(rt);
+chunk = Chunk::allocate(rt);
+}
+/* OOM stops the background allocation. */
+if (!chunk)
+break;
+JS_ASSERT(chunk->info.numArenasFreeCommitted == 0);
+rt->gcChunkPool.put(chunk);
+} while (state == ALLOCATING && rt->gcChunkPool.wantBackgroundAllocation(rt));
+if (state == ALLOCATING)
+state = IDLE;
+break;
+}
+case CANCEL_ALLOCATION:
+state = IDLE;
+PR_NotifyAllCondVar(done);
+break;
+}
+}
+}
+#endif /* JS_THREADSAFE */
+void
+GCHelperThread::startBackgroundSweep(bool shouldShrink)
+{
+JS_ASSERT(rt->useHelperThreads());
+#ifdef JS_THREADSAFE
+AutoLockGC lock(rt);
+JS_ASSERT(state == IDLE);
+JS_ASSERT(!sweepFlag);
+sweepFlag = true;
+shrinkFlag = shouldShrink;
+state = SWEEPING;
+PR_NotifyCondVar(wakeup);
+#endif /* JS_THREADSAFE */
+}
+/* Must be called with the GC lock taken. */
+void
+GCHelperThread::startBackgroundShrink()
+{
+JS_ASSERT(rt->useHelperThreads());
+#ifdef JS_THREADSAFE
+switch (state) {
+case IDLE:
+JS_ASSERT(!sweepFlag);
+shrinkFlag = true;
+state = SWEEPING;
+PR_NotifyCondVar(wakeup);
+break;
+case SWEEPING:
+shrinkFlag = true;
+break;
+case ALLOCATING:
+case CANCEL_ALLOCATION:
+/*
+* If we have started background allocation there is nothing to
+* shrink.
+*/
+break;
+case SHUTDOWN:
+MOZ_ASSUME_UNREACHABLE("No shrink on shutdown");
+}
+#endif /* JS_THREADSAFE */
+}
+void
+GCHelperThread::waitBackgroundSweepEnd()
+{
+if (!rt->useHelperThreads()) {
+JS_ASSERT(state == IDLE);
+return;
+}
+#ifdef JS_THREADSAFE
+AutoLockGC lock(rt);
+while (state == SWEEPING)
+wait(done);
+if (rt->gcIncrementalState == NO_INCREMENTAL)
+AssertBackgroundSweepingFinished(rt);
+#endif /* JS_THREADSAFE */
+}
+void
+GCHelperThread::waitBackgroundSweepOrAllocEnd()
+{
+if (!rt->useHelperThreads()) {
+JS_ASSERT(state == IDLE);
+return;
+}
+#ifdef JS_THREADSAFE
+AutoLockGC lock(rt);
+if (state == ALLOCATING)
+state = CANCEL_ALLOCATION;
+while (state == SWEEPING || state == CANCEL_ALLOCATION)
+wait(done);
+if (rt->gcIncrementalState == NO_INCREMENTAL)
+AssertBackgroundSweepingFinished(rt);
+#endif /* JS_THREADSAFE */
+}
+/* Must be called with the GC lock taken. */
+inline void
+GCHelperThread::startBackgroundAllocationIfIdle()
+{
+JS_ASSERT(rt->useHelperThreads());
+#ifdef JS_THREADSAFE
+if (state == IDLE) {
+state = ALLOCATING;
+PR_NotifyCondVar(wakeup);
+}
+#endif /* JS_THREADSAFE */
+}
+void
+GCHelperThread::replenishAndFreeLater(void *ptr)
+{
+JS_ASSERT(freeCursor == freeCursorEnd);
+do {
+if (freeCursor && !freeVector.append(freeCursorEnd - FREE_ARRAY_LENGTH))
+break;
+freeCursor = (void **) js_malloc(FREE_ARRAY_SIZE);
+if (!freeCursor) {
+freeCursorEnd = nullptr;
+break;
+}
+freeCursorEnd = freeCursor + FREE_ARRAY_LENGTH;
+*freeCursor++ = ptr;
+return;
+} while (false);
+js_free(ptr);
+}
+#ifdef JS_THREADSAFE
+/* Must be called with the GC lock taken. */
+void
+GCHelperThread::doSweep()
+{
+if (sweepFlag) {
+sweepFlag = false;
+AutoUnlockGC unlock(rt);
+SweepBackgroundThings(rt, true);
+if (freeCursor) {
+void **array = freeCursorEnd - FREE_ARRAY_LENGTH;
+freeElementsAndArray(array, freeCursor);
+freeCursor = freeCursorEnd = nullptr;
+} else {
+JS_ASSERT(!freeCursorEnd);
+}
+for (void ***iter = freeVector.begin(); iter != freeVector.end(); ++iter) {
+void **array = *iter;
+freeElementsAndArray(array, array + FREE_ARRAY_LENGTH);
+}
+freeVector.resize(0);
+rt->freeLifoAlloc.freeAll();
+}
+bool shrinking = shrinkFlag;
+ExpireChunksAndArenas(rt, shrinking);
+/*
+* The main thread may have called ShrinkGCBuffers while
+* ExpireChunksAndArenas(rt, false) was running, so we recheck the flag
+* afterwards.
+*/
+if (!shrinking && shrinkFlag) {
+shrinkFlag = false;
+ExpireChunksAndArenas(rt, true);
+}
+}
+#endif /* JS_THREADSAFE */
+bool
+GCHelperThread::onBackgroundThread()
+{
+#ifdef JS_THREADSAFE
+return PR_GetCurrentThread() == getThread();
+#else
+return false;
+#endif
+}
+static bool
+ReleaseObservedTypes(JSRuntime *rt)
+{
+bool releaseTypes = rt->gcZeal() != 0;
+#ifndef JS_MORE_DETERMINISTIC
+int64_t now = PRMJ_Now();
+if (now >= rt->gcJitReleaseTime)
+releaseTypes = true;
+if (releaseTypes)
+rt->gcJitReleaseTime = now + JIT_SCRIPT_RELEASE_TYPES_INTERVAL;
+#endif
+return releaseTypes;
+}
+/*
+* It's simpler if we preserve the invariant that every zone has at least one
+* compartment. If we know we're deleting the entire zone, then
+* SweepCompartments is allowed to delete all compartments. In this case,
+* |keepAtleastOne| is false. If some objects remain in the zone so that it
+* cannot be deleted, then we set |keepAtleastOne| to true, which prohibits
+* SweepCompartments from deleting every compartment. Instead, it preserves an
+* arbitrary compartment in the zone.
+*/
+static void
+SweepCompartments(FreeOp *fop, Zone *zone, bool keepAtleastOne, bool lastGC)
+{
+JSRuntime *rt = zone->runtimeFromMainThread();
+JSDestroyCompartmentCallback callback = rt->destroyCompartmentCallback;
+JSCompartment **read = zone->compartments.begin();
+JSCompartment **end = zone->compartments.end();
+JSCompartment **write = read;
+bool foundOne = false;
+while (read < end) {
+JSCompartment *comp = *read++;
+JS_ASSERT(!rt->isAtomsCompartment(comp));
+/*
+* Don't delete the last compartment if all the ones before it were
+* deleted and keepAtleastOne is true.
+*/
+bool dontDelete = read == end && !foundOne && keepAtleastOne;
+if ((!comp->marked && !dontDelete) || lastGC) {
+if (callback)
+callback(fop, comp);
+if (comp->principals)
+JS_DropPrincipals(rt, comp->principals);
+js_delete(comp);
+} else {
+*write++ = comp;
+foundOne = true;
+}
+}
+zone->compartments.resize(write - zone->compartments.begin());
+JS_ASSERT_IF(keepAtleastOne, !zone->compartments.empty());
+}
+static void
+SweepZones(FreeOp *fop, bool lastGC)
+{
+JSRuntime *rt = fop->runtime();
+JSZoneCallback callback = rt->destroyZoneCallback;
+/* Skip the atomsCompartment zone. */
+Zone **read = rt->zones.begin() + 1;
+Zone **end = rt->zones.end();
+Zone **write = read;
+JS_ASSERT(rt->zones.length() >= 1);
+JS_ASSERT(rt->isAtomsZone(rt->zones[0]));
+while (read < end) {
+Zone *zone = *read++;
+if (zone->wasGCStarted()) {
+if ((zone->allocator.arenas.arenaListsAreEmpty() && !zone->hasMarkedCompartments()) ||
+lastGC)
+{
+zone->allocator.arenas.checkEmptyFreeLists();
+if (callback)
+callback(zone);
+SweepCompartments(fop, zone, false, lastGC);
+JS_ASSERT(zone->compartments.empty());
+fop->delete_(zone);
+continue;
+}
+SweepCompartments(fop, zone, true, lastGC);
+}
+*write++ = zone;
+}
+rt->zones.resize(write - rt->zones.begin());
+}
+static void
+PurgeRuntime(JSRuntime *rt)
+{
+for (GCCompartmentsIter comp(rt); !comp.done(); comp.next())
+comp->purge();
+rt->freeLifoAlloc.transferUnusedFrom(&rt->tempLifoAlloc);
+rt->interpreterStack().purge(rt);
+rt->gsnCache.purge();
+rt->scopeCoordinateNameCache.purge();
+rt->newObjectCache.purge();
+rt->nativeIterCache.purge();
+rt->sourceDataCache.purge();
+rt->evalCache.clear();
+if (!rt->hasActiveCompilations())
+rt->parseMapPool().purgeAll();
+}
+static bool
+ShouldPreserveJITCode(JSCompartment *comp, int64_t currentTime)
+{
+JSRuntime *rt = comp->runtimeFromMainThread();
+if (rt->gcShouldCleanUpEverything)
+return false;
+if (rt->alwaysPreserveCode)
+return true;
+if (comp->lastAnimationTime + PRMJ_USEC_PER_SEC >= currentTime)
+return true;
+return false;
+}
+#ifdef DEBUG
+class CompartmentCheckTracer : public JSTracer
+{
+public:
+CompartmentCheckTracer(JSRuntime *rt, JSTraceCallback callback)
+: JSTracer(rt, callback)
+{}
+Cell *src;
+JSGCTraceKind srcKind;
+Zone *zone;
+JSCompartment *compartment;
+};
+static bool
+InCrossCompartmentMap(JSObject *src, Cell *dst, JSGCTraceKind dstKind)
+{
+JSCompartment *srccomp = src->compartment();
+if (dstKind == JSTRACE_OBJECT) {
+Value key = ObjectValue(*static_cast<JSObject *>(dst));
+if (WrapperMap::Ptr p = srccomp->lookupWrapper(key)) {
+if (*p->value().unsafeGet() == ObjectValue(*src))
+return true;
+}
+}
+/*
+* If the cross-compartment edge is caused by the debugger, then we don't
+* know the right hashtable key, so we have to iterate.
+*/
+for (JSCompartment::WrapperEnum e(srccomp); !e.empty(); e.popFront()) {
+if (e.front().key().wrapped == dst && ToMarkable(e.front().value()) == src)
+return true;
+}
+return false;
+}
+static void
+CheckCompartment(CompartmentCheckTracer *trc, JSCompartment *thingCompartment,
+Cell *thing, JSGCTraceKind kind)
+{
+JS_ASSERT(thingCompartment == trc->compartment ||
+trc->runtime()->isAtomsCompartment(thingCompartment) ||
+(trc->srcKind == JSTRACE_OBJECT &&
+InCrossCompartmentMap((JSObject *)trc->src, thing, kind)));
+}
+static JSCompartment *
+CompartmentOfCell(Cell *thing, JSGCTraceKind kind)
+{
+if (kind == JSTRACE_OBJECT)
+return static_cast<JSObject *>(thing)->compartment();
+else if (kind == JSTRACE_SHAPE)
+return static_cast<Shape *>(thing)->compartment();
+else if (kind == JSTRACE_BASE_SHAPE)
+return static_cast<BaseShape *>(thing)->compartment();
+else if (kind == JSTRACE_SCRIPT)
+return static_cast<JSScript *>(thing)->compartment();
+else
+return nullptr;
+}
+static void
+CheckCompartmentCallback(JSTracer *trcArg, void **thingp, JSGCTraceKind kind)
+{
+CompartmentCheckTracer *trc = static_cast<CompartmentCheckTracer *>(trcArg);
+Cell *thing = (Cell *)*thingp;
+JSCompartment *comp = CompartmentOfCell(thing, kind);
+if (comp && trc->compartment) {
+CheckCompartment(trc, comp, thing, kind);
+} else {
+JS_ASSERT(thing->tenuredZone() == trc->zone ||
+trc->runtime()->isAtomsZone(thing->tenuredZone()));
+}
+}
+static void
+CheckForCompartmentMismatches(JSRuntime *rt)
+{
+if (rt->gcDisableStrictProxyCheckingCount)
+return;
+CompartmentCheckTracer trc(rt, CheckCompartmentCallback);
+for (ZonesIter zone(rt, SkipAtoms); !zone.done(); zone.next()) {
+trc.zone = zone;
+for (size_t thingKind = 0; thingKind < FINALIZE_LAST; thingKind++) {
+for (CellIterUnderGC i(zone, AllocKind(thingKind)); !i.done(); i.next()) {
+trc.src = i.getCell();
+trc.srcKind = MapAllocToTraceKind(AllocKind(thingKind));
+trc.compartment = CompartmentOfCell(trc.src, trc.srcKind);
+JS_TraceChildren(&trc, trc.src, trc.srcKind);
+}
+}
+}
+}
+#endif
+static bool
+BeginMarkPhase(JSRuntime *rt)
+{
+int64_t currentTime = PRMJ_Now();
+#ifdef DEBUG
+if (rt->gcFullCompartmentChecks)
+CheckForCompartmentMismatches(rt);
+#endif
+rt->gcIsFull = true;
+bool any = false;
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+/* Assert that zone state is as we expect */
+JS_ASSERT(!zone->isCollecting());
+JS_ASSERT(!zone->compartments.empty());
+for (unsigned i = 0; i < FINALIZE_LIMIT; ++i)
+JS_ASSERT(!zone->allocator.arenas.arenaListsToSweep[i]);
+/* Set up which zones will be collected. */
+if (zone->isGCScheduled()) {
+if (!rt->isAtomsZone(zone)) {
+any = true;
+zone->setGCState(Zone::Mark);
+}
+} else {
+rt->gcIsFull = false;
+}
+zone->scheduledForDestruction = false;
+zone->maybeAlive = false;
+zone->setPreservingCode(false);
+}
+for (CompartmentsIter c(rt, WithAtoms); !c.done(); c.next()) {
+JS_ASSERT(c->gcLiveArrayBuffers.empty());
+c->marked = false;
+if (ShouldPreserveJITCode(c, currentTime))
+c->zone()->setPreservingCode(true);
+}
+/*
+* Atoms are not in the cross-compartment map. So if there are any
+* zones that are not being collected, we are not allowed to collect
+* atoms. Otherwise, the non-collected zones could contain pointers
+* to atoms that we would miss.
+*
+* keepAtoms() will only change on the main thread, which we are currently
+* on. If the value of keepAtoms() changes between GC slices, then we'll
+* cancel the incremental GC. See IsIncrementalGCSafe.
+*/
+if (rt->gcIsFull && !rt->keepAtoms()) {
+Zone *atomsZone = rt->atomsCompartment()->zone();
+if (atomsZone->isGCScheduled()) {
+JS_ASSERT(!atomsZone->isCollecting());
+atomsZone->setGCState(Zone::Mark);
+any = true;
+}
+}
+/* Check that at least one zone is scheduled for collection. */
+if (!any)
+return false;
+/*
+* At the end of each incremental slice, we call prepareForIncrementalGC,
+* which marks objects in all arenas that we're currently allocating
+* into. This can cause leaks if unreachable objects are in these
+* arenas. This purge call ensures that we only mark arenas that have had
+* allocations after the incremental GC started.
+*/
+if (rt->gcIsIncremental) {
+for (GCZonesIter zone(rt); !zone.done(); zone.next())
+zone->allocator.arenas.purge();
+}
+rt->gcMarker.start();
+JS_ASSERT(!rt->gcMarker.callback);
+JS_ASSERT(IS_GC_MARKING_TRACER(&rt->gcMarker));
+/* For non-incremental GC the following sweep discards the jit code. */
+if (rt->gcIsIncremental) {
+for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_MARK_DISCARD_CODE);
+zone->discardJitCode(rt->defaultFreeOp());
+}
+}
+GCMarker *gcmarker = &rt->gcMarker;
+rt->gcStartNumber = rt->gcNumber;
+/*
+* We must purge the runtime at the beginning of an incremental GC. The
+* danger if we purge later is that the snapshot invariant of incremental
+* GC will be broken, as follows. If some object is reachable only through
+* some cache (say the dtoaCache) then it will not be part of the snapshot.
+* If we purge after root marking, then the mutator could obtain a pointer
+* to the object and start using it. This object might never be marked, so
+* a GC hazard would exist.
+*/
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_PURGE);
+PurgeRuntime(rt);
+}
+/*
+* Mark phase.
+*/
+gcstats::AutoPhase ap1(rt->gcStats, gcstats::PHASE_MARK);
+gcstats::AutoPhase ap2(rt->gcStats, gcstats::PHASE_MARK_ROOTS);
+for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
+/* Unmark everything in the zones being collected. */
+zone->allocator.arenas.unmarkAll();
+}
+for (GCCompartmentsIter c(rt); !c.done(); c.next()) {
+/* Reset weak map list for the compartments being collected. */
+WeakMapBase::resetCompartmentWeakMapList(c);
+}
+if (rt->gcIsFull)
+UnmarkScriptData(rt);
+MarkRuntime(gcmarker);
+if (rt->gcIsIncremental)
+BufferGrayRoots(gcmarker);
+/*
+* This code ensures that if a zone is "dead", then it will be
+* collected in this GC. A zone is considered dead if its maybeAlive
+* flag is false. The maybeAlive flag is set if:
+*   (1) the zone has incoming cross-compartment edges, or
+*   (2) an object in the zone was marked during root marking, either
+*       as a black root or a gray root.
+* If the maybeAlive is false, then we set the scheduledForDestruction flag.
+* At any time later in the GC, if we try to mark an object whose
+* zone is scheduled for destruction, we will assert.
+* NOTE: Due to bug 811587, we only assert if gcManipulatingDeadCompartments
+* is true (e.g., if we're doing a brain transplant).
+*
+* The purpose of this check is to ensure that a zone that we would
+* normally destroy is not resurrected by a read barrier or an
+* allocation. This might happen during a function like JS_TransplantObject,
+* which iterates over all compartments, live or dead, and operates on their
+* objects. See bug 803376 for details on this problem. To avoid the
+* problem, we are very careful to avoid allocation and read barriers during
+* JS_TransplantObject and the like. The code here ensures that we don't
+* regress.
+*
+* Note that there are certain cases where allocations or read barriers in
+* dead zone are difficult to avoid. We detect such cases (via the
+* gcObjectsMarkedInDeadCompartment counter) and redo any ongoing GCs after
+* the JS_TransplantObject function has finished. This ensures that the dead
+* zones will be cleaned up. See AutoMarkInDeadZone and
+* AutoMaybeTouchDeadZones for details.
+*/
+/* Set the maybeAlive flag based on cross-compartment edges. */
+for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next()) {
+for (JSCompartment::WrapperEnum e(c); !e.empty(); e.popFront()) {
+Cell *dst = e.front().key().wrapped;
+dst->tenuredZone()->maybeAlive = true;
+}
+}
+/*
+* For black roots, code in gc/Marking.cpp will already have set maybeAlive
+* during MarkRuntime.
+*/
+for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
+if (!zone->maybeAlive && !rt->isAtomsZone(zone))
+zone->scheduledForDestruction = true;
+}
+rt->gcFoundBlackGrayEdges = false;
+return true;
+}
+template <class CompartmentIterT>
+static void
+MarkWeakReferences(JSRuntime *rt, gcstats::Phase phase)
+{
+GCMarker *gcmarker = &rt->gcMarker;
+JS_ASSERT(gcmarker->isDrained());
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_MARK);
+gcstats::AutoPhase ap1(rt->gcStats, phase);
+for (;;) {
+bool markedAny = false;
+for (CompartmentIterT c(rt); !c.done(); c.next()) {
+markedAny |= WatchpointMap::markCompartmentIteratively(c, gcmarker);
+markedAny |= WeakMapBase::markCompartmentIteratively(c, gcmarker);
+}
+markedAny |= Debugger::markAllIteratively(gcmarker);
+if (!markedAny)
+break;
+SliceBudget budget;
+gcmarker->drainMarkStack(budget);
+}
+JS_ASSERT(gcmarker->isDrained());
+}
+static void
+MarkWeakReferencesInCurrentGroup(JSRuntime *rt, gcstats::Phase phase)
+{
+MarkWeakReferences<GCCompartmentGroupIter>(rt, phase);
+}
+template <class ZoneIterT, class CompartmentIterT>
+static void
+MarkGrayReferences(JSRuntime *rt)
+{
+GCMarker *gcmarker = &rt->gcMarker;
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_MARK);
+gcstats::AutoPhase ap1(rt->gcStats, gcstats::PHASE_SWEEP_MARK_GRAY);
+gcmarker->setMarkColorGray();
+if (gcmarker->hasBufferedGrayRoots()) {
+for (ZoneIterT zone(rt); !zone.done(); zone.next())
+gcmarker->markBufferedGrayRoots(zone);
+} else {
+JS_ASSERT(!rt->gcIsIncremental);
+if (JSTraceDataOp op = rt->gcGrayRootTracer.op)
+(*op)(gcmarker, rt->gcGrayRootTracer.data);
+}
+SliceBudget budget;
+gcmarker->drainMarkStack(budget);
+}
+MarkWeakReferences<CompartmentIterT>(rt, gcstats::PHASE_SWEEP_MARK_GRAY_WEAK);
+JS_ASSERT(gcmarker->isDrained());
+gcmarker->setMarkColorBlack();
+}
+static void
+MarkGrayReferencesInCurrentGroup(JSRuntime *rt)
+{
+MarkGrayReferences<GCZoneGroupIter, GCCompartmentGroupIter>(rt);
+}
+#ifdef DEBUG
+static void
+MarkAllWeakReferences(JSRuntime *rt, gcstats::Phase phase)
+{
+MarkWeakReferences<GCCompartmentsIter>(rt, phase);
+}
+static void
+MarkAllGrayReferences(JSRuntime *rt)
+{
+MarkGrayReferences<GCZonesIter, GCCompartmentsIter>(rt);
+}
+class js::gc::MarkingValidator
+{
+public:
+MarkingValidator(JSRuntime *rt);
+~MarkingValidator();
+void nonIncrementalMark();
+void validate();
+private:
+JSRuntime *runtime;
+bool initialized;
+typedef HashMap<Chunk *, ChunkBitmap *, GCChunkHasher, SystemAllocPolicy> BitmapMap;
+BitmapMap map;
+};
+js::gc::MarkingValidator::MarkingValidator(JSRuntime *rt)
+: runtime(rt),
+initialized(false)
+{}
+js::gc::MarkingValidator::~MarkingValidator()
+{
+if (!map.initialized())
+return;
+for (BitmapMap::Range r(map.all()); !r.empty(); r.popFront())
+js_delete(r.front().value());
+}
+void
+js::gc::MarkingValidator::nonIncrementalMark()
+{
+/*
+* Perform a non-incremental mark for all collecting zones and record
+* the results for later comparison.
+*
+* Currently this does not validate gray marking.
+*/
+if (!map.init())
+return;
+GCMarker *gcmarker = &runtime->gcMarker;
+/* Save existing mark bits. */
+for (GCChunkSet::Range r(runtime->gcChunkSet.all()); !r.empty(); r.popFront()) {
+ChunkBitmap *bitmap = &r.front()->bitmap;
+	ChunkBitmap *entry = js_new<ChunkBitmap>();
+if (!entry)
+return;
+memcpy((void *)entry->bitmap, (void *)bitmap->bitmap, sizeof(bitmap->bitmap));
+if (!map.putNew(r.front(), entry))
+return;
+}
+/*
+* Temporarily clear the lists of live weakmaps and array buffers for the
+* compartments we are collecting.
+*/
+WeakMapVector weakmaps;
+ArrayBufferVector arrayBuffers;
+for (GCCompartmentsIter c(runtime); !c.done(); c.next()) {
+if (!WeakMapBase::saveCompartmentWeakMapList(c, weakmaps) ||
+!ArrayBufferObject::saveArrayBufferList(c, arrayBuffers))
+{
+return;
+}
+}
+/*
+* After this point, the function should run to completion, so we shouldn't
+* do anything fallible.
+*/
+initialized = true;
+for (GCCompartmentsIter c(runtime); !c.done(); c.next()) {
+WeakMapBase::resetCompartmentWeakMapList(c);
+ArrayBufferObject::resetArrayBufferList(c);
+}
+/* Re-do all the marking, but non-incrementally. */
+js::gc::State state = runtime->gcIncrementalState;
+runtime->gcIncrementalState = MARK_ROOTS;
+JS_ASSERT(gcmarker->isDrained());
+gcmarker->reset();
+for (GCChunkSet::Range r(runtime->gcChunkSet.all()); !r.empty(); r.popFront())
+r.front()->bitmap.clear();
+{
+gcstats::AutoPhase ap1(runtime->gcStats, gcstats::PHASE_MARK);
+gcstats::AutoPhase ap2(runtime->gcStats, gcstats::PHASE_MARK_ROOTS);
+MarkRuntime(gcmarker, true);
+}
+{
+gcstats::AutoPhase ap1(runtime->gcStats, gcstats::PHASE_MARK);
+SliceBudget budget;
+runtime->gcIncrementalState = MARK;
+runtime->gcMarker.drainMarkStack(budget);
+}
+runtime->gcIncrementalState = SWEEP;
+{
+gcstats::AutoPhase ap(runtime->gcStats, gcstats::PHASE_SWEEP);
+MarkAllWeakReferences(runtime, gcstats::PHASE_SWEEP_MARK_WEAK);
+/* Update zone state for gray marking. */
+for (GCZonesIter zone(runtime); !zone.done(); zone.next()) {
+JS_ASSERT(zone->isGCMarkingBlack());
+zone->setGCState(Zone::MarkGray);
+}
+MarkAllGrayReferences(runtime);
+/* Restore zone state. */
+for (GCZonesIter zone(runtime); !zone.done(); zone.next()) {
+JS_ASSERT(zone->isGCMarkingGray());
+zone->setGCState(Zone::Mark);
+}
+}
+/* Take a copy of the non-incremental mark state and restore the original. */
+for (GCChunkSet::Range r(runtime->gcChunkSet.all()); !r.empty(); r.popFront()) {
+Chunk *chunk = r.front();
+ChunkBitmap *bitmap = &chunk->bitmap;
+ChunkBitmap *entry = map.lookup(chunk)->value();
+Swap(*entry, *bitmap);
+}
+for (GCCompartmentsIter c(runtime); !c.done(); c.next()) {
+WeakMapBase::resetCompartmentWeakMapList(c);
+ArrayBufferObject::resetArrayBufferList(c);
+}
+WeakMapBase::restoreCompartmentWeakMapLists(weakmaps);
+ArrayBufferObject::restoreArrayBufferLists(arrayBuffers);
+runtime->gcIncrementalState = state;
+}
+void
+js::gc::MarkingValidator::validate()
+{
+/*
+* Validates the incremental marking for a single compartment by comparing
+* the mark bits to those previously recorded for a non-incremental mark.
+*/
+if (!initialized)
+return;
+for (GCChunkSet::Range r(runtime->gcChunkSet.all()); !r.empty(); r.popFront()) {
+Chunk *chunk = r.front();
+BitmapMap::Ptr ptr = map.lookup(chunk);
+if (!ptr)
+continue;  /* Allocated after we did the non-incremental mark. */
+ChunkBitmap *bitmap = ptr->value();
+ChunkBitmap *incBitmap = &chunk->bitmap;
+for (size_t i = 0; i < ArenasPerChunk; i++) {
+if (chunk->decommittedArenas.get(i))
+continue;
+Arena *arena = &chunk->arenas[i];
+if (!arena->aheader.allocated())
+continue;
+if (!arena->aheader.zone->isGCSweeping())
+continue;
+if (arena->aheader.allocatedDuringIncremental)
+continue;
+AllocKind kind = arena->aheader.getAllocKind();
+uintptr_t thing = arena->thingsStart(kind);
+uintptr_t end = arena->thingsEnd();
+while (thing < end) {
+Cell *cell = (Cell *)thing;
+/*
+* If a non-incremental GC wouldn't have collected a cell, then
+* an incremental GC won't collect it.
+*/
+JS_ASSERT_IF(bitmap->isMarked(cell, BLACK), incBitmap->isMarked(cell, BLACK));
+/*
+* If the cycle collector isn't allowed to collect an object
+* after a non-incremental GC has run, then it isn't allowed to
+* collected it after an incremental GC.
+*/
+JS_ASSERT_IF(!bitmap->isMarked(cell, GRAY), !incBitmap->isMarked(cell, GRAY));
+thing += Arena::thingSize(kind);
+}
+}
+}
+}
+#endif
+static void
+ComputeNonIncrementalMarkingForValidation(JSRuntime *rt)
+{
+#ifdef DEBUG
+JS_ASSERT(!rt->gcMarkingValidator);
+if (rt->gcIsIncremental && rt->gcValidate)
+rt->gcMarkingValidator = js_new<MarkingValidator>(rt);
+if (rt->gcMarkingValidator)
+rt->gcMarkingValidator->nonIncrementalMark();
+#endif
+}
+static void
+ValidateIncrementalMarking(JSRuntime *rt)
+{
+#ifdef DEBUG
+if (rt->gcMarkingValidator)
+rt->gcMarkingValidator->validate();
+#endif
+}
+static void
+FinishMarkingValidation(JSRuntime *rt)
+{
+#ifdef DEBUG
+js_delete(rt->gcMarkingValidator);
+rt->gcMarkingValidator = nullptr;
+#endif
+}
+static void
+AssertNeedsBarrierFlagsConsistent(JSRuntime *rt)
+{
+#ifdef DEBUG
+bool anyNeedsBarrier = false;
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next())
+anyNeedsBarrier |= zone->needsBarrier();
+JS_ASSERT(rt->needsBarrier() == anyNeedsBarrier);
+#endif
+}
+static void
+DropStringWrappers(JSRuntime *rt)
+{
+/*
+* String "wrappers" are dropped on GC because their presence would require
+* us to sweep the wrappers in all compartments every time we sweep a
+* compartment group.
+*/
+for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next()) {
+for (JSCompartment::WrapperEnum e(c); !e.empty(); e.popFront()) {
+if (e.front().key().kind == CrossCompartmentKey::StringWrapper)
+e.removeFront();
+}
+}
+}
+/*
+* Group zones that must be swept at the same time.
+*
+* If compartment A has an edge to an unmarked object in compartment B, then we
+* must not sweep A in a later slice than we sweep B. That's because a write
+* barrier in A that could lead to the unmarked object in B becoming
+* marked. However, if we had already swept that object, we would be in trouble.
+*
+* If we consider these dependencies as a graph, then all the compartments in
+* any strongly-connected component of this graph must be swept in the same
+* slice.
+*
+* Tarjan's algorithm is used to calculate the components.
+*/
+void
+JSCompartment::findOutgoingEdges(ComponentFinder<JS::Zone> &finder)
+{
+for (js::WrapperMap::Enum e(crossCompartmentWrappers); !e.empty(); e.popFront()) {
+CrossCompartmentKey::Kind kind = e.front().key().kind;
+JS_ASSERT(kind != CrossCompartmentKey::StringWrapper);
+Cell *other = e.front().key().wrapped;
+if (kind == CrossCompartmentKey::ObjectWrapper) {
+/*
+* Add edge to wrapped object compartment if wrapped object is not
+* marked black to indicate that wrapper compartment not be swept
+* after wrapped compartment.
+*/
+if (!other->isMarked(BLACK) || other->isMarked(GRAY)) {
+JS::Zone *w = other->tenuredZone();
+if (w->isGCMarking())
+finder.addEdgeTo(w);
+}
+} else {
+JS_ASSERT(kind == CrossCompartmentKey::DebuggerScript ||
+kind == CrossCompartmentKey::DebuggerSource ||
+kind == CrossCompartmentKey::DebuggerObject ||
+kind == CrossCompartmentKey::DebuggerEnvironment);
+/*
+* Add edge for debugger object wrappers, to ensure (in conjuction
+* with call to Debugger::findCompartmentEdges below) that debugger
+* and debuggee objects are always swept in the same group.
+*/
+JS::Zone *w = other->tenuredZone();
+if (w->isGCMarking())
+finder.addEdgeTo(w);
+}
+}
+Debugger::findCompartmentEdges(zone(), finder);
+}
+void
+Zone::findOutgoingEdges(ComponentFinder<JS::Zone> &finder)
+{
+/*
+* Any compartment may have a pointer to an atom in the atoms
+* compartment, and these aren't in the cross compartment map.
+*/
+JSRuntime *rt = runtimeFromMainThread();
+if (rt->atomsCompartment()->zone()->isGCMarking())
+finder.addEdgeTo(rt->atomsCompartment()->zone());
+for (CompartmentsInZoneIter comp(this); !comp.done(); comp.next())
+comp->findOutgoingEdges(finder);
+}
+static void
+FindZoneGroups(JSRuntime *rt)
+{
+ComponentFinder<Zone> finder(rt->mainThread.nativeStackLimit[StackForSystemCode]);
+if (!rt->gcIsIncremental)
+finder.useOneComponent();
+for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
+JS_ASSERT(zone->isGCMarking());
+finder.addNode(zone);
+}
+rt->gcZoneGroups = finder.getResultsList();
+rt->gcCurrentZoneGroup = rt->gcZoneGroups;
+rt->gcZoneGroupIndex = 0;
+JS_ASSERT_IF(!rt->gcIsIncremental, !rt->gcCurrentZoneGroup->nextGroup());
+}
+static void
+ResetGrayList(JSCompartment* comp);
+static void
+GetNextZoneGroup(JSRuntime *rt)
+{
+rt->gcCurrentZoneGroup = rt->gcCurrentZoneGroup->nextGroup();
+++rt->gcZoneGroupIndex;
+if (!rt->gcCurrentZoneGroup) {
+rt->gcAbortSweepAfterCurrentGroup = false;
+return;
+}
+if (!rt->gcIsIncremental)
+ComponentFinder<Zone>::mergeGroups(rt->gcCurrentZoneGroup);
+if (rt->gcAbortSweepAfterCurrentGroup) {
+JS_ASSERT(!rt->gcIsIncremental);
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+JS_ASSERT(!zone->gcNextGraphComponent);
+JS_ASSERT(zone->isGCMarking());
+zone->setNeedsBarrier(false, Zone::UpdateIon);
+zone->setGCState(Zone::NoGC);
+zone->gcGrayRoots.clearAndFree();
+}
+rt->setNeedsBarrier(false);
+AssertNeedsBarrierFlagsConsistent(rt);
+for (GCCompartmentGroupIter comp(rt); !comp.done(); comp.next()) {
+ArrayBufferObject::resetArrayBufferList(comp);
+ResetGrayList(comp);
+}
+rt->gcAbortSweepAfterCurrentGroup = false;
+rt->gcCurrentZoneGroup = nullptr;
+}
+}
+/*
+* Gray marking:
+*
+* At the end of collection, anything reachable from a gray root that has not
+* otherwise been marked black must be marked gray.
+*
+* This means that when marking things gray we must not allow marking to leave
+* the current compartment group, as that could result in things being marked
+* grey when they might subsequently be marked black.  To achieve this, when we
+* find a cross compartment pointer we don't mark the referent but add it to a
+* singly-linked list of incoming gray pointers that is stored with each
+* compartment.
+*
+* The list head is stored in JSCompartment::gcIncomingGrayPointers and contains
+* cross compartment wrapper objects. The next pointer is stored in the second
+* extra slot of the cross compartment wrapper.
+*
+* The list is created during gray marking when one of the
+* MarkCrossCompartmentXXX functions is called for a pointer that leaves the
+* current compartent group.  This calls DelayCrossCompartmentGrayMarking to
+* push the referring object onto the list.
+*
+* The list is traversed and then unlinked in
+* MarkIncomingCrossCompartmentPointers.
+*/
+static bool
+IsGrayListObject(JSObject *obj)
+{
+JS_ASSERT(obj);
+return obj->is<CrossCompartmentWrapperObject>() && !IsDeadProxyObject(obj);
+}
+/* static */ unsigned
+ProxyObject::grayLinkSlot(JSObject *obj)
+{
+JS_ASSERT(IsGrayListObject(obj));
+return ProxyObject::EXTRA_SLOT + 1;
+}
+#ifdef DEBUG
+static void
+AssertNotOnGrayList(JSObject *obj)
+{
+JS_ASSERT_IF(IsGrayListObject(obj),
+obj->getReservedSlot(ProxyObject::grayLinkSlot(obj)).isUndefined());
+}
+#endif
+static JSObject *
+CrossCompartmentPointerReferent(JSObject *obj)
+{
+JS_ASSERT(IsGrayListObject(obj));
+return &obj->as<ProxyObject>().private_().toObject();
+}
+static JSObject *
+NextIncomingCrossCompartmentPointer(JSObject *prev, bool unlink)
+{
+unsigned slot = ProxyObject::grayLinkSlot(prev);
+JSObject *next = prev->getReservedSlot(slot).toObjectOrNull();
+JS_ASSERT_IF(next, IsGrayListObject(next));
+if (unlink)
+prev->setSlot(slot, UndefinedValue());
+return next;
+}
+void
+js::DelayCrossCompartmentGrayMarking(JSObject *src)
+{
+JS_ASSERT(IsGrayListObject(src));
+/* Called from MarkCrossCompartmentXXX functions. */
+unsigned slot = ProxyObject::grayLinkSlot(src);
+JSObject *dest = CrossCompartmentPointerReferent(src);
+JSCompartment *comp = dest->compartment();
+if (src->getReservedSlot(slot).isUndefined()) {
+src->setCrossCompartmentSlot(slot, ObjectOrNullValue(comp->gcIncomingGrayPointers));
+comp->gcIncomingGrayPointers = src;
+} else {
+JS_ASSERT(src->getReservedSlot(slot).isObjectOrNull());
+}
+#ifdef DEBUG
+/*
+* Assert that the object is in our list, also walking the list to check its
+* integrity.
+*/
+JSObject *obj = comp->gcIncomingGrayPointers;
+bool found = false;
+while (obj) {
+if (obj == src)
+found = true;
+obj = NextIncomingCrossCompartmentPointer(obj, false);
+}
+JS_ASSERT(found);
+#endif
+}
+static void
+MarkIncomingCrossCompartmentPointers(JSRuntime *rt, const uint32_t color)
+{
+JS_ASSERT(color == BLACK || color == GRAY);
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_MARK);
+static const gcstats::Phase statsPhases[] = {
+gcstats::PHASE_SWEEP_MARK_INCOMING_BLACK,
+gcstats::PHASE_SWEEP_MARK_INCOMING_GRAY
+};
+gcstats::AutoPhase ap1(rt->gcStats, statsPhases[color]);
+bool unlinkList = color == GRAY;
+for (GCCompartmentGroupIter c(rt); !c.done(); c.next()) {
+JS_ASSERT_IF(color == GRAY, c->zone()->isGCMarkingGray());
+JS_ASSERT_IF(color == BLACK, c->zone()->isGCMarkingBlack());
+JS_ASSERT_IF(c->gcIncomingGrayPointers, IsGrayListObject(c->gcIncomingGrayPointers));
+for (JSObject *src = c->gcIncomingGrayPointers;
+src;
+src = NextIncomingCrossCompartmentPointer(src, unlinkList))
+{
+JSObject *dst = CrossCompartmentPointerReferent(src);
+JS_ASSERT(dst->compartment() == c);
+if (color == GRAY) {
+if (IsObjectMarked(&src) && src->isMarked(GRAY))
+MarkGCThingUnbarriered(&rt->gcMarker, (void**)&dst,
+"cross-compartment gray pointer");
+} else {
+if (IsObjectMarked(&src) && !src->isMarked(GRAY))
+MarkGCThingUnbarriered(&rt->gcMarker, (void**)&dst,
+"cross-compartment black pointer");
+}
+}
+if (unlinkList)
+c->gcIncomingGrayPointers = nullptr;
+}
+SliceBudget budget;
+rt->gcMarker.drainMarkStack(budget);
+}
+static bool
+RemoveFromGrayList(JSObject *wrapper)
+{
+if (!IsGrayListObject(wrapper))
+return false;
+unsigned slot = ProxyObject::grayLinkSlot(wrapper);
+if (wrapper->getReservedSlot(slot).isUndefined())
+return false;  /* Not on our list. */
+JSObject *tail = wrapper->getReservedSlot(slot).toObjectOrNull();
+wrapper->setReservedSlot(slot, UndefinedValue());
+JSCompartment *comp = CrossCompartmentPointerReferent(wrapper)->compartment();
+JSObject *obj = comp->gcIncomingGrayPointers;
+if (obj == wrapper) {
+comp->gcIncomingGrayPointers = tail;
+return true;
+}
+while (obj) {
+unsigned slot = ProxyObject::grayLinkSlot(obj);
+JSObject *next = obj->getReservedSlot(slot).toObjectOrNull();
+if (next == wrapper) {
+obj->setCrossCompartmentSlot(slot, ObjectOrNullValue(tail));
+return true;
+}
+obj = next;
+}
+MOZ_ASSUME_UNREACHABLE("object not found in gray link list");
+}
+static void
+ResetGrayList(JSCompartment *comp)
+{
+JSObject *src = comp->gcIncomingGrayPointers;
+while (src)
+src = NextIncomingCrossCompartmentPointer(src, true);
+comp->gcIncomingGrayPointers = nullptr;
+}
+void
+js::NotifyGCNukeWrapper(JSObject *obj)
+{
+/*
+* References to target of wrapper are being removed, we no longer have to
+* remember to mark it.
+*/
+RemoveFromGrayList(obj);
+}
+enum {
+JS_GC_SWAP_OBJECT_A_REMOVED = 1 << 0,
+JS_GC_SWAP_OBJECT_B_REMOVED = 1 << 1
+};
+unsigned
+js::NotifyGCPreSwap(JSObject *a, JSObject *b)
+{
+/*
+* Two objects in the same compartment are about to have had their contents
+* swapped.  If either of them are in our gray pointer list, then we remove
+* them from the lists, returning a bitset indicating what happened.
+*/
+return (RemoveFromGrayList(a) ? JS_GC_SWAP_OBJECT_A_REMOVED : 0) |
+(RemoveFromGrayList(b) ? JS_GC_SWAP_OBJECT_B_REMOVED : 0);
+}
+void
+js::NotifyGCPostSwap(JSObject *a, JSObject *b, unsigned removedFlags)
+{
+/*
+* Two objects in the same compartment have had their contents swapped.  If
+* either of them were in our gray pointer list, we re-add them again.
+*/
+if (removedFlags & JS_GC_SWAP_OBJECT_A_REMOVED)
+DelayCrossCompartmentGrayMarking(b);
+if (removedFlags & JS_GC_SWAP_OBJECT_B_REMOVED)
+DelayCrossCompartmentGrayMarking(a);
+}
+static void
+EndMarkingZoneGroup(JSRuntime *rt)
+{
+/*
+* Mark any incoming black pointers from previously swept compartments
+* whose referents are not marked. This can occur when gray cells become
+* black by the action of UnmarkGray.
+*/
+MarkIncomingCrossCompartmentPointers(rt, BLACK);
+MarkWeakReferencesInCurrentGroup(rt, gcstats::PHASE_SWEEP_MARK_WEAK);
+/*
+* Change state of current group to MarkGray to restrict marking to this
+* group.  Note that there may be pointers to the atoms compartment, and
+* these will be marked through, as they are not marked with
+* MarkCrossCompartmentXXX.
+*/
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+JS_ASSERT(zone->isGCMarkingBlack());
+zone->setGCState(Zone::MarkGray);
+}
+/* Mark incoming gray pointers from previously swept compartments. */
+rt->gcMarker.setMarkColorGray();
+MarkIncomingCrossCompartmentPointers(rt, GRAY);
+rt->gcMarker.setMarkColorBlack();
+/* Mark gray roots and mark transitively inside the current compartment group. */
+MarkGrayReferencesInCurrentGroup(rt);
+/* Restore marking state. */
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+JS_ASSERT(zone->isGCMarkingGray());
+zone->setGCState(Zone::Mark);
+}
+JS_ASSERT(rt->gcMarker.isDrained());
+}
+static void
+BeginSweepingZoneGroup(JSRuntime *rt)
+{
+/*
+* Begin sweeping the group of zones in gcCurrentZoneGroup,
+* performing actions that must be done before yielding to caller.
+*/
+bool sweepingAtoms = false;
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+/* Set the GC state to sweeping. */
+JS_ASSERT(zone->isGCMarking());
+zone->setGCState(Zone::Sweep);
+/* Purge the ArenaLists before sweeping. */
+zone->allocator.arenas.purge();
+if (rt->isAtomsZone(zone))
+sweepingAtoms = true;
+if (rt->sweepZoneCallback)
+rt->sweepZoneCallback(zone);
+}
+ValidateIncrementalMarking(rt);
+FreeOp fop(rt, rt->gcSweepOnBackgroundThread);
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_FINALIZE_START);
+if (rt->gcFinalizeCallback)
+rt->gcFinalizeCallback(&fop, JSFINALIZE_GROUP_START, !rt->gcIsFull /* unused */);
+}
+if (sweepingAtoms) {
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_ATOMS);
+rt->sweepAtoms();
+}
+/* Prune out dead views from ArrayBuffer's view lists. */
+for (GCCompartmentGroupIter c(rt); !c.done(); c.next())
+ArrayBufferObject::sweep(c);
+/* Collect watch points associated with unreachable objects. */
+WatchpointMap::sweepAll(rt);
+/* Detach unreachable debuggers and global objects from each other. */
+Debugger::sweepAll(&fop);
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_COMPARTMENTS);
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_DISCARD_CODE);
+zone->discardJitCode(&fop);
+}
+bool releaseTypes = ReleaseObservedTypes(rt);
+for (GCCompartmentGroupIter c(rt); !c.done(); c.next()) {
+gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
+c->sweep(&fop, releaseTypes && !c->zone()->isPreservingCode());
+}
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
+// If there is an OOM while sweeping types, the type information
+// will be deoptimized so that it is still correct (i.e.
+// overapproximates the possible types in the zone), but the
+// constraints might not have been triggered on the deoptimization
+// or even copied over completely. In this case, destroy all JIT
+// code and new script addendums in the zone, the only things whose
+// correctness depends on the type constraints.
+bool oom = false;
+zone->sweep(&fop, releaseTypes && !zone->isPreservingCode(), &oom);
+if (oom) {
+zone->setPreservingCode(false);
+zone->discardJitCode(&fop);
+zone->types.clearAllNewScriptAddendumsOnOOM();
+}
+}
+}
+/*
+* Queue all GC things in all zones for sweeping, either in the
+* foreground or on the background thread.
+*
+* Note that order is important here for the background case.
+*
+* Objects are finalized immediately but this may change in the future.
+*/
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
+zone->allocator.arenas.queueObjectsForSweep(&fop);
+}
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
+zone->allocator.arenas.queueStringsForSweep(&fop);
+}
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
+zone->allocator.arenas.queueScriptsForSweep(&fop);
+}
+#ifdef JS_ION
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
+zone->allocator.arenas.queueJitCodeForSweep(&fop);
+}
+#endif
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+gcstats::AutoSCC scc(rt->gcStats, rt->gcZoneGroupIndex);
+zone->allocator.arenas.queueShapesForSweep(&fop);
+zone->allocator.arenas.gcShapeArenasToSweep =
+zone->allocator.arenas.arenaListsToSweep[FINALIZE_SHAPE];
+}
+rt->gcSweepPhase = 0;
+rt->gcSweepZone = rt->gcCurrentZoneGroup;
+rt->gcSweepKindIndex = 0;
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_FINALIZE_END);
+if (rt->gcFinalizeCallback)
+rt->gcFinalizeCallback(&fop, JSFINALIZE_GROUP_END, !rt->gcIsFull /* unused */);
+}
+}
+static void
+EndSweepingZoneGroup(JSRuntime *rt)
+{
+/* Update the GC state for zones we have swept and unlink the list. */
+for (GCZoneGroupIter zone(rt); !zone.done(); zone.next()) {
+JS_ASSERT(zone->isGCSweeping());
+zone->setGCState(Zone::Finished);
+}
+/* Reset the list of arenas marked as being allocated during sweep phase. */
+while (ArenaHeader *arena = rt->gcArenasAllocatedDuringSweep) {
+rt->gcArenasAllocatedDuringSweep = arena->getNextAllocDuringSweep();
+arena->unsetAllocDuringSweep();
+}
+}
+static void
+BeginSweepPhase(JSRuntime *rt, bool lastGC)
+{
+/*
+* Sweep phase.
+*
+* Finalize as we sweep, outside of rt->gcLock but with rt->isHeapBusy()
+* true so that any attempt to allocate a GC-thing from a finalizer will
+* fail, rather than nest badly and leave the unmarked newborn to be swept.
+*/
+JS_ASSERT(!rt->gcAbortSweepAfterCurrentGroup);
+ComputeNonIncrementalMarkingForValidation(rt);
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP);
+#ifdef JS_THREADSAFE
+rt->gcSweepOnBackgroundThread = !lastGC && rt->useHelperThreads();
+#endif
+#ifdef DEBUG
+for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next()) {
+JS_ASSERT(!c->gcIncomingGrayPointers);
+for (JSCompartment::WrapperEnum e(c); !e.empty(); e.popFront()) {
+if (e.front().key().kind != CrossCompartmentKey::StringWrapper)
+AssertNotOnGrayList(&e.front().value().get().toObject());
+}
+}
+#endif
+DropStringWrappers(rt);
+FindZoneGroups(rt);
+EndMarkingZoneGroup(rt);
+BeginSweepingZoneGroup(rt);
+}
+bool
+ArenaLists::foregroundFinalize(FreeOp *fop, AllocKind thingKind, SliceBudget &sliceBudget)
+{
+if (!arenaListsToSweep[thingKind])
+return true;
+ArenaList &dest = arenaLists[thingKind];
+return FinalizeArenas(fop, &arenaListsToSweep[thingKind], dest, thingKind, sliceBudget);
+}
+static bool
+DrainMarkStack(JSRuntime *rt, SliceBudget &sliceBudget, gcstats::Phase phase)
+{
+/* Run a marking slice and return whether the stack is now empty. */
+gcstats::AutoPhase ap(rt->gcStats, phase);
+return rt->gcMarker.drainMarkStack(sliceBudget);
+}
+static bool
+SweepPhase(JSRuntime *rt, SliceBudget &sliceBudget)
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP);
+FreeOp fop(rt, rt->gcSweepOnBackgroundThread);
+bool finished = DrainMarkStack(rt, sliceBudget, gcstats::PHASE_SWEEP_MARK);
+if (!finished)
+return false;
+for (;;) {
+/* Finalize foreground finalized things. */
+for (; rt->gcSweepPhase < FinalizePhaseCount ; ++rt->gcSweepPhase) {
+gcstats::AutoPhase ap(rt->gcStats, FinalizePhaseStatsPhase[rt->gcSweepPhase]);
+for (; rt->gcSweepZone; rt->gcSweepZone = rt->gcSweepZone->nextNodeInGroup()) {
+Zone *zone = rt->gcSweepZone;
+while (rt->gcSweepKindIndex < FinalizePhaseLength[rt->gcSweepPhase]) {
+AllocKind kind = FinalizePhases[rt->gcSweepPhase][rt->gcSweepKindIndex];
+if (!zone->allocator.arenas.foregroundFinalize(&fop, kind, sliceBudget))
+return false;  /* Yield to the mutator. */
+++rt->gcSweepKindIndex;
+}
+rt->gcSweepKindIndex = 0;
+}
+rt->gcSweepZone = rt->gcCurrentZoneGroup;
+}
+/* Remove dead shapes from the shape tree, but don't finalize them yet. */
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP_SHAPE);
+for (; rt->gcSweepZone; rt->gcSweepZone = rt->gcSweepZone->nextNodeInGroup()) {
+Zone *zone = rt->gcSweepZone;
+while (ArenaHeader *arena = zone->allocator.arenas.gcShapeArenasToSweep) {
+for (CellIterUnderGC i(arena); !i.done(); i.next()) {
+Shape *shape = i.get<Shape>();
+if (!shape->isMarked())
+shape->sweep();
+}
+zone->allocator.arenas.gcShapeArenasToSweep = arena->next;
+sliceBudget.step(Arena::thingsPerArena(Arena::thingSize(FINALIZE_SHAPE)));
+if (sliceBudget.isOverBudget())
+return false;  /* Yield to the mutator. */
+}
+}
+}
+EndSweepingZoneGroup(rt);
+GetNextZoneGroup(rt);
+if (!rt->gcCurrentZoneGroup)
+return true;  /* We're finished. */
+EndMarkingZoneGroup(rt);
+BeginSweepingZoneGroup(rt);
+}
+}
+static void
+EndSweepPhase(JSRuntime *rt, JSGCInvocationKind gckind, bool lastGC)
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_SWEEP);
+FreeOp fop(rt, rt->gcSweepOnBackgroundThread);
+JS_ASSERT_IF(lastGC, !rt->gcSweepOnBackgroundThread);
+JS_ASSERT(rt->gcMarker.isDrained());
+rt->gcMarker.stop();
+/*
+* Recalculate whether GC was full or not as this may have changed due to
+* newly created zones.  Can only change from full to not full.
+*/
+if (rt->gcIsFull) {
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+if (!zone->isCollecting()) {
+rt->gcIsFull = false;
+break;
+}
+}
+}
+/*
+* If we found any black->gray edges during marking, we completely clear the
+* mark bits of all uncollected zones, or if a reset has occured, zones that
+* will no longer be collected. This is safe, although it may
+* prevent the cycle collector from collecting some dead objects.
+*/
+if (rt->gcFoundBlackGrayEdges) {
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+if (!zone->isCollecting())
+zone->allocator.arenas.unmarkAll();
+}
+}
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_DESTROY);
+/*
+* Sweep script filenames after sweeping functions in the generic loop
+* above. In this way when a scripted function's finalizer destroys the
+* script and calls rt->destroyScriptHook, the hook can still access the
+* script's filename. See bug 323267.
+*/
+if (rt->gcIsFull)
+SweepScriptData(rt);
+/* Clear out any small pools that we're hanging on to. */
+if (JSC::ExecutableAllocator *execAlloc = rt->maybeExecAlloc())
+execAlloc->purge();
+/*
+* This removes compartments from rt->compartment, so we do it last to make
+* sure we don't miss sweeping any compartments.
+*/
+if (!lastGC)
+SweepZones(&fop, lastGC);
+if (!rt->gcSweepOnBackgroundThread) {
+/*
+* Destroy arenas after we finished the sweeping so finalizers can
+* safely use IsAboutToBeFinalized(). This is done on the
+* GCHelperThread if possible. We acquire the lock only because
+* Expire needs to unlock it for other callers.
+*/
+AutoLockGC lock(rt);
+ExpireChunksAndArenas(rt, gckind == GC_SHRINK);
+}
+}
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_FINALIZE_END);
+if (rt->gcFinalizeCallback)
+rt->gcFinalizeCallback(&fop, JSFINALIZE_COLLECTION_END, !rt->gcIsFull);
+/* If we finished a full GC, then the gray bits are correct. */
+if (rt->gcIsFull)
+rt->gcGrayBitsValid = true;
+}
+/* Set up list of zones for sweeping of background things. */
+JS_ASSERT(!rt->gcSweepingZones);
+for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
+zone->gcNextGraphNode = rt->gcSweepingZones;
+rt->gcSweepingZones = zone;
+}
+/* If not sweeping on background thread then we must do it here. */
+if (!rt->gcSweepOnBackgroundThread) {
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_DESTROY);
+SweepBackgroundThings(rt, false);
+rt->freeLifoAlloc.freeAll();
+/* Ensure the compartments get swept if it's the last GC. */
+if (lastGC)
+SweepZones(&fop, lastGC);
+}
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+zone->setGCLastBytes(zone->gcBytes, gckind);
+if (zone->isCollecting()) {
+JS_ASSERT(zone->isGCFinished());
+zone->setGCState(Zone::NoGC);
+}
+#ifdef DEBUG
+JS_ASSERT(!zone->isCollecting());
+JS_ASSERT(!zone->wasGCStarted());
+for (unsigned i = 0 ; i < FINALIZE_LIMIT ; ++i) {
+JS_ASSERT_IF(!IsBackgroundFinalized(AllocKind(i)) ||
+!rt->gcSweepOnBackgroundThread,
+!zone->allocator.arenas.arenaListsToSweep[i]);
+}
+#endif
+}
+#ifdef DEBUG
+for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next()) {
+JS_ASSERT(!c->gcIncomingGrayPointers);
+JS_ASSERT(c->gcLiveArrayBuffers.empty());
+for (JSCompartment::WrapperEnum e(c); !e.empty(); e.popFront()) {
+if (e.front().key().kind != CrossCompartmentKey::StringWrapper)
+AssertNotOnGrayList(&e.front().value().get().toObject());
+}
+}
+#endif
+FinishMarkingValidation(rt);
+rt->gcLastGCTime = PRMJ_Now();
+}
+namespace {
+/* ...while this class is to be used only for garbage collection. */
+class AutoGCSession
+{
+JSRuntime *runtime;
+AutoTraceSession session;
+bool canceled;
+public:
+explicit AutoGCSession(JSRuntime *rt);
+~AutoGCSession();
+void cancel() { canceled = true; }
+};
+} /* anonymous namespace */
+/* Start a new heap session. */
+AutoTraceSession::AutoTraceSession(JSRuntime *rt, js::HeapState heapState)
+: lock(rt),
+runtime(rt),
+prevState(rt->heapState)
+{
+JS_ASSERT(!rt->noGCOrAllocationCheck);
+JS_ASSERT(!rt->isHeapBusy());
+JS_ASSERT(heapState != Idle);
+#ifdef JSGC_GENERATIONAL
+JS_ASSERT_IF(heapState == MajorCollecting, rt->gcNursery.isEmpty());
+#endif
+// Threads with an exclusive context can hit refillFreeList while holding
+// the exclusive access lock. To avoid deadlocking when we try to acquire
+// this lock during GC and the other thread is waiting, make sure we hold
+// the exclusive access lock during GC sessions.
+JS_ASSERT(rt->currentThreadHasExclusiveAccess());
+if (rt->exclusiveThreadsPresent()) {
+// Lock the worker thread state when changing the heap state in the
+// presence of exclusive threads, to avoid racing with refillFreeList.
+#ifdef JS_THREADSAFE
+AutoLockWorkerThreadState lock;
+rt->heapState = heapState;
+#else
+MOZ_CRASH();
+#endif
+} else {
+rt->heapState = heapState;
+}
+}
+AutoTraceSession::~AutoTraceSession()
+{
+JS_ASSERT(runtime->isHeapBusy());
+if (runtime->exclusiveThreadsPresent()) {
+#ifdef JS_THREADSAFE
+AutoLockWorkerThreadState lock;
+runtime->heapState = prevState;
+// Notify any worker threads waiting for the trace session to end.
+WorkerThreadState().notifyAll(GlobalWorkerThreadState::PRODUCER);
+#else
+MOZ_CRASH();
+#endif
+} else {
+runtime->heapState = prevState;
+}
+}
+AutoGCSession::AutoGCSession(JSRuntime *rt)
+: runtime(rt),
+session(rt, MajorCollecting),
+canceled(false)
+{
+runtime->gcIsNeeded = false;
+runtime->gcInterFrameGC = true;
+runtime->gcNumber++;
+// It's ok if threads other than the main thread have suppressGC set, as
+// they are operating on zones which will not be collected from here.
+JS_ASSERT(!runtime->mainThread.suppressGC);
+}
+AutoGCSession::~AutoGCSession()
+{
+if (canceled)
+return;
+#ifndef JS_MORE_DETERMINISTIC
+runtime->gcNextFullGCTime = PRMJ_Now() + GC_IDLE_FULL_SPAN;
+#endif
+runtime->gcChunkAllocationSinceLastGC = false;
+#ifdef JS_GC_ZEAL
+/* Keeping these around after a GC is dangerous. */
+runtime->gcSelectedForMarking.clearAndFree();
+#endif
+/* Clear gcMallocBytes for all compartments */
+for (ZonesIter zone(runtime, WithAtoms); !zone.done(); zone.next()) {
+zone->resetGCMallocBytes();
+zone->unscheduleGC();
+}
+runtime->resetGCMallocBytes();
+}
+AutoCopyFreeListToArenas::AutoCopyFreeListToArenas(JSRuntime *rt, ZoneSelector selector)
+: runtime(rt),
+selector(selector)
+{
+for (ZonesIter zone(rt, selector); !zone.done(); zone.next())
+zone->allocator.arenas.copyFreeListsToArenas();
+}
+AutoCopyFreeListToArenas::~AutoCopyFreeListToArenas()
+{
+for (ZonesIter zone(runtime, selector); !zone.done(); zone.next())
+zone->allocator.arenas.clearFreeListsInArenas();
+}
+class AutoCopyFreeListToArenasForGC
+{
+JSRuntime *runtime;
+public:
+AutoCopyFreeListToArenasForGC(JSRuntime *rt) : runtime(rt) {
+JS_ASSERT(rt->currentThreadHasExclusiveAccess());
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next())
+zone->allocator.arenas.copyFreeListsToArenas();
+}
+~AutoCopyFreeListToArenasForGC() {
+for (ZonesIter zone(runtime, WithAtoms); !zone.done(); zone.next())
+zone->allocator.arenas.clearFreeListsInArenas();
+}
+};
+static void
+IncrementalCollectSlice(JSRuntime *rt,
+int64_t budget,
+JS::gcreason::Reason gcReason,
+JSGCInvocationKind gcKind);
+static void
+ResetIncrementalGC(JSRuntime *rt, const char *reason)
+{
+switch (rt->gcIncrementalState) {
+case NO_INCREMENTAL:
+return;
+case MARK: {
+/* Cancel any ongoing marking. */
+AutoCopyFreeListToArenasForGC copy(rt);
+rt->gcMarker.reset();
+rt->gcMarker.stop();
+for (GCCompartmentsIter c(rt); !c.done(); c.next()) {
+ArrayBufferObject::resetArrayBufferList(c);
+ResetGrayList(c);
+}
+for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
+JS_ASSERT(zone->isGCMarking());
+zone->setNeedsBarrier(false, Zone::UpdateIon);
+zone->setGCState(Zone::NoGC);
+}
+rt->setNeedsBarrier(false);
+AssertNeedsBarrierFlagsConsistent(rt);
+rt->gcIncrementalState = NO_INCREMENTAL;
+JS_ASSERT(!rt->gcStrictCompartmentChecking);
+break;
+}
+case SWEEP:
+rt->gcMarker.reset();
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next())
+zone->scheduledForDestruction = false;
+/* Finish sweeping the current zone group, then abort. */
+rt->gcAbortSweepAfterCurrentGroup = true;
+IncrementalCollectSlice(rt, SliceBudget::Unlimited, JS::gcreason::RESET, GC_NORMAL);
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_WAIT_BACKGROUND_THREAD);
+rt->gcHelperThread.waitBackgroundSweepOrAllocEnd();
+}
+break;
+default:
+MOZ_ASSUME_UNREACHABLE("Invalid incremental GC state");
+}
+rt->gcStats.reset(reason);
+#ifdef DEBUG
+for (CompartmentsIter c(rt, SkipAtoms); !c.done(); c.next())
+JS_ASSERT(c->gcLiveArrayBuffers.empty());
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+JS_ASSERT(!zone->needsBarrier());
+for (unsigned i = 0; i < FINALIZE_LIMIT; ++i)
+JS_ASSERT(!zone->allocator.arenas.arenaListsToSweep[i]);
+}
+#endif
+}
+namespace {
+class AutoGCSlice {
+public:
+AutoGCSlice(JSRuntime *rt);
+~AutoGCSlice();
+private:
+JSRuntime *runtime;
+};
+} /* anonymous namespace */
+AutoGCSlice::AutoGCSlice(JSRuntime *rt)
+: runtime(rt)
+{
+/*
+* During incremental GC, the compartment's active flag determines whether
+* there are stack frames active for any of its scripts. Normally this flag
+* is set at the beginning of the mark phase. During incremental GC, we also
+* set it at the start of every phase.
+*/
+for (ActivationIterator iter(rt); !iter.done(); ++iter)
+iter->compartment()->zone()->active = true;
+for (GCZonesIter zone(rt); !zone.done(); zone.next()) {
+/*
+* Clear needsBarrier early so we don't do any write barriers during
+* GC. We don't need to update the Ion barriers (which is expensive)
+* because Ion code doesn't run during GC. If need be, we'll update the
+* Ion barriers in ~AutoGCSlice.
+*/
+if (zone->isGCMarking()) {
+JS_ASSERT(zone->needsBarrier());
+zone->setNeedsBarrier(false, Zone::DontUpdateIon);
+} else {
+JS_ASSERT(!zone->needsBarrier());
+}
+}
+rt->setNeedsBarrier(false);
+AssertNeedsBarrierFlagsConsistent(rt);
+}
+AutoGCSlice::~AutoGCSlice()
+{
+/* We can't use GCZonesIter if this is the end of the last slice. */
+bool haveBarriers = false;
+for (ZonesIter zone(runtime, WithAtoms); !zone.done(); zone.next()) {
+if (zone->isGCMarking()) {
+zone->setNeedsBarrier(true, Zone::UpdateIon);
+zone->allocator.arenas.prepareForIncrementalGC(runtime);
+haveBarriers = true;
+} else {
+zone->setNeedsBarrier(false, Zone::UpdateIon);
+}
+}
+runtime->setNeedsBarrier(haveBarriers);
+AssertNeedsBarrierFlagsConsistent(runtime);
+}
+static void
+PushZealSelectedObjects(JSRuntime *rt)
+{
+#ifdef JS_GC_ZEAL
+/* Push selected objects onto the mark stack and clear the list. */
+for (JSObject **obj = rt->gcSelectedForMarking.begin();
+obj != rt->gcSelectedForMarking.end(); obj++)
+{
+MarkObjectUnbarriered(&rt->gcMarker, obj, "selected obj");
+}
+#endif
+}
+static void
+IncrementalCollectSlice(JSRuntime *rt,
+int64_t budget,
+JS::gcreason::Reason reason,
+JSGCInvocationKind gckind)
+{
+JS_ASSERT(rt->currentThreadHasExclusiveAccess());
+AutoCopyFreeListToArenasForGC copy(rt);
+AutoGCSlice slice(rt);
+bool lastGC = (reason == JS::gcreason::DESTROY_RUNTIME);
+gc::State initialState = rt->gcIncrementalState;
+int zeal = 0;
+#ifdef JS_GC_ZEAL
+if (reason == JS::gcreason::DEBUG_GC && budget != SliceBudget::Unlimited) {
+/*
+* Do the incremental collection type specified by zeal mode if the
+* collection was triggered by RunDebugGC() and incremental GC has not
+* been cancelled by ResetIncrementalGC.
+*/
+zeal = rt->gcZeal();
+}
+#endif
+JS_ASSERT_IF(rt->gcIncrementalState != NO_INCREMENTAL, rt->gcIsIncremental);
+rt->gcIsIncremental = budget != SliceBudget::Unlimited;
+if (zeal == ZealIncrementalRootsThenFinish || zeal == ZealIncrementalMarkAllThenFinish) {
+/*
+* Yields between slices occurs at predetermined points in these modes;
+* the budget is not used.
+*/
+budget = SliceBudget::Unlimited;
+}
+SliceBudget sliceBudget(budget);
+if (rt->gcIncrementalState == NO_INCREMENTAL) {
+rt->gcIncrementalState = MARK_ROOTS;
+rt->gcLastMarkSlice = false;
+}
+if (rt->gcIncrementalState == MARK)
+AutoGCRooter::traceAllWrappers(&rt->gcMarker);
+switch (rt->gcIncrementalState) {
+case MARK_ROOTS:
+if (!BeginMarkPhase(rt)) {
+rt->gcIncrementalState = NO_INCREMENTAL;
+return;
+}
+if (!lastGC)
+PushZealSelectedObjects(rt);
+rt->gcIncrementalState = MARK;
+if (rt->gcIsIncremental && zeal == ZealIncrementalRootsThenFinish)
+break;
+/* fall through */
+case MARK: {
+/* If we needed delayed marking for gray roots, then collect until done. */
+if (!rt->gcMarker.hasBufferedGrayRoots()) {
+sliceBudget.reset();
+rt->gcIsIncremental = false;
+}
+bool finished = DrainMarkStack(rt, sliceBudget, gcstats::PHASE_MARK);
+if (!finished)
+break;
+JS_ASSERT(rt->gcMarker.isDrained());
+if (!rt->gcLastMarkSlice && rt->gcIsIncremental &&
+((initialState == MARK && zeal != ZealIncrementalRootsThenFinish) ||
+zeal == ZealIncrementalMarkAllThenFinish))
+{
+/*
+* Yield with the aim of starting the sweep in the next
+* slice.  We will need to mark anything new on the stack
+* when we resume, so we stay in MARK state.
+*/
+rt->gcLastMarkSlice = true;
+break;
+}
+rt->gcIncrementalState = SWEEP;
+/*
+* This runs to completion, but we don't continue if the budget is
+* now exhasted.
+*/
+BeginSweepPhase(rt, lastGC);
+if (sliceBudget.isOverBudget())
+break;
+/*
+* Always yield here when running in incremental multi-slice zeal
+* mode, so RunDebugGC can reset the slice buget.
+*/
+if (rt->gcIsIncremental && zeal == ZealIncrementalMultipleSlices)
+break;
+/* fall through */
+}
+case SWEEP: {
+bool finished = SweepPhase(rt, sliceBudget);
+if (!finished)
+break;
+EndSweepPhase(rt, gckind, lastGC);
+if (rt->gcSweepOnBackgroundThread)
+rt->gcHelperThread.startBackgroundSweep(gckind == GC_SHRINK);
+rt->gcIncrementalState = NO_INCREMENTAL;
+break;
+}
+default:
+JS_ASSERT(false);
+}
+}
+IncrementalSafety
+gc::IsIncrementalGCSafe(JSRuntime *rt)
+{
+JS_ASSERT(!rt->mainThread.suppressGC);
+if (rt->keepAtoms())
+return IncrementalSafety::Unsafe("keepAtoms set");
+if (!rt->gcIncrementalEnabled)
+return IncrementalSafety::Unsafe("incremental permanently disabled");
+return IncrementalSafety::Safe();
+}
+static void
+BudgetIncrementalGC(JSRuntime *rt, int64_t *budget)
+{
+IncrementalSafety safe = IsIncrementalGCSafe(rt);
+if (!safe) {
+ResetIncrementalGC(rt, safe.reason());
+*budget = SliceBudget::Unlimited;
+rt->gcStats.nonincremental(safe.reason());
+return;
+}
+if (rt->gcMode() != JSGC_MODE_INCREMENTAL) {
+ResetIncrementalGC(rt, "GC mode change");
+*budget = SliceBudget::Unlimited;
+rt->gcStats.nonincremental("GC mode");
+return;
+}
+if (rt->isTooMuchMalloc()) {
+*budget = SliceBudget::Unlimited;
+rt->gcStats.nonincremental("malloc bytes trigger");
+}
+bool reset = false;
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+if (zone->gcBytes >= zone->gcTriggerBytes) {
+*budget = SliceBudget::Unlimited;
+rt->gcStats.nonincremental("allocation trigger");
+}
+if (rt->gcIncrementalState != NO_INCREMENTAL &&
+zone->isGCScheduled() != zone->wasGCStarted())
+{
+reset = true;
+}
+if (zone->isTooMuchMalloc()) {
+*budget = SliceBudget::Unlimited;
+rt->gcStats.nonincremental("malloc bytes trigger");
+}
+}
+if (reset)
+ResetIncrementalGC(rt, "zone change");
+}
+/*
+* Run one GC "cycle" (either a slice of incremental GC or an entire
+* non-incremental GC. We disable inlining to ensure that the bottom of the
+* stack with possible GC roots recorded in MarkRuntime excludes any pointers we
+* use during the marking implementation.
+*
+* Returns true if we "reset" an existing incremental GC, which would force us
+* to run another cycle.
+*/
+static MOZ_NEVER_INLINE bool
+GCCycle(JSRuntime *rt, bool incremental, int64_t budget,
+JSGCInvocationKind gckind, JS::gcreason::Reason reason)
+{
+AutoGCSession gcsession(rt);
+/*
+* As we about to purge caches and clear the mark bits we must wait for
+* any background finalization to finish. We must also wait for the
+* background allocation to finish so we can avoid taking the GC lock
+* when manipulating the chunks during the GC.
+*/
+{
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_WAIT_BACKGROUND_THREAD);
+rt->gcHelperThread.waitBackgroundSweepOrAllocEnd();
+}
+State prevState = rt->gcIncrementalState;
+if (!incremental) {
+/* If non-incremental GC was requested, reset incremental GC. */
+ResetIncrementalGC(rt, "requested");
+rt->gcStats.nonincremental("requested");
+budget = SliceBudget::Unlimited;
+} else {
+BudgetIncrementalGC(rt, &budget);
+}
+/* The GC was reset, so we need a do-over. */
+if (prevState != NO_INCREMENTAL && rt->gcIncrementalState == NO_INCREMENTAL) {
+gcsession.cancel();
+return true;
+}
+IncrementalCollectSlice(rt, budget, reason, gckind);
+return false;
+}
+#ifdef JS_GC_ZEAL
+static bool
+IsDeterministicGCReason(JS::gcreason::Reason reason)
+{
+if (reason > JS::gcreason::DEBUG_GC &&
+reason != JS::gcreason::CC_FORCED && reason != JS::gcreason::SHUTDOWN_CC)
+{
+return false;
+}
+if (reason == JS::gcreason::MAYBEGC)
+return false;
+return true;
+}
+#endif
+static bool
+ShouldCleanUpEverything(JSRuntime *rt, JS::gcreason::Reason reason, JSGCInvocationKind gckind)
+{
+// During shutdown, we must clean everything up, for the sake of leak
+// detection. When a runtime has no contexts, or we're doing a GC before a
+// shutdown CC, those are strong indications that we're shutting down.
+return reason == JS::gcreason::DESTROY_RUNTIME ||
+reason == JS::gcreason::SHUTDOWN_CC ||
+gckind == GC_SHRINK;
+}
+namespace {
+#ifdef JSGC_GENERATIONAL
+class AutoDisableStoreBuffer
+{
+JSRuntime *runtime;
+bool prior;
+public:
+AutoDisableStoreBuffer(JSRuntime *rt) : runtime(rt) {
+prior = rt->gcStoreBuffer.isEnabled();
+rt->gcStoreBuffer.disable();
+}
+~AutoDisableStoreBuffer() {
+if (prior)
+runtime->gcStoreBuffer.enable();
+}
+};
+#else
+struct AutoDisableStoreBuffer
+{
+AutoDisableStoreBuffer(JSRuntime *) {}
+};
+#endif
+} /* anonymous namespace */
+static void
+Collect(JSRuntime *rt, bool incremental, int64_t budget,
+JSGCInvocationKind gckind, JS::gcreason::Reason reason)
+{
+/* GC shouldn't be running in parallel execution mode */
+JS_ASSERT(!InParallelSection());
+JS_AbortIfWrongThread(rt);
+/* If we attempt to invoke the GC while we are running in the GC, assert. */
+JS_ASSERT(!rt->isHeapBusy());
+if (rt->mainThread.suppressGC)
+return;
+TraceLogger *logger = TraceLoggerForMainThread(rt);
+AutoTraceLog logGC(logger, TraceLogger::GC);
+#ifdef JS_GC_ZEAL
+if (rt->gcDeterministicOnly && !IsDeterministicGCReason(reason))
+return;
+#endif
+JS_ASSERT_IF(!incremental || budget != SliceBudget::Unlimited, JSGC_INCREMENTAL);
+AutoStopVerifyingBarriers av(rt, reason == JS::gcreason::SHUTDOWN_CC ||
+reason == JS::gcreason::DESTROY_RUNTIME);
+RecordNativeStackTopForGC(rt);
+int zoneCount = 0;
+int compartmentCount = 0;
+int collectedCount = 0;
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+if (rt->gcMode() == JSGC_MODE_GLOBAL)
+zone->scheduleGC();
+/* This is a heuristic to avoid resets. */
+if (rt->gcIncrementalState != NO_INCREMENTAL && zone->needsBarrier())
+zone->scheduleGC();
+zoneCount++;
+if (zone->isGCScheduled())
+collectedCount++;
+}
+for (CompartmentsIter c(rt, WithAtoms); !c.done(); c.next())
+compartmentCount++;
+rt->gcShouldCleanUpEverything = ShouldCleanUpEverything(rt, reason, gckind);
+bool repeat = false;
+do {
+MinorGC(rt, reason);
+/*
+* Marking can trigger many incidental post barriers, some of them for
+* objects which are not going to be live after the GC.
+*/
+AutoDisableStoreBuffer adsb(rt);
+gcstats::AutoGCSlice agc(rt->gcStats, collectedCount, zoneCount, compartmentCount, reason);
+/*
+* Let the API user decide to defer a GC if it wants to (unless this
+* is the last context). Invoke the callback regardless.
+*/
+if (rt->gcIncrementalState == NO_INCREMENTAL) {
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_GC_BEGIN);
+if (JSGCCallback callback = rt->gcCallback)
+callback(rt, JSGC_BEGIN, rt->gcCallbackData);
+}
+rt->gcPoke = false;
+bool wasReset = GCCycle(rt, incremental, budget, gckind, reason);
+if (rt->gcIncrementalState == NO_INCREMENTAL) {
+gcstats::AutoPhase ap(rt->gcStats, gcstats::PHASE_GC_END);
+if (JSGCCallback callback = rt->gcCallback)
+callback(rt, JSGC_END, rt->gcCallbackData);
+}
+/* Need to re-schedule all zones for GC. */
+if (rt->gcPoke && rt->gcShouldCleanUpEverything)
+JS::PrepareForFullGC(rt);
+/*
+* If we reset an existing GC, we need to start a new one. Also, we
+* repeat GCs that happen during shutdown (the gcShouldCleanUpEverything
+* case) until we can be sure that no additional garbage is created
+* (which typically happens if roots are dropped during finalizers).
+*/
+repeat = (rt->gcPoke && rt->gcShouldCleanUpEverything) || wasReset;
+} while (repeat);
+if (rt->gcIncrementalState == NO_INCREMENTAL) {
+#ifdef JS_THREADSAFE
+EnqueuePendingParseTasksAfterGC(rt);
+#endif
+}
+}
+void
+js::GC(JSRuntime *rt, JSGCInvocationKind gckind, JS::gcreason::Reason reason)
+{
+Collect(rt, false, SliceBudget::Unlimited, gckind, reason);
+}
+void
+js::GCSlice(JSRuntime *rt, JSGCInvocationKind gckind, JS::gcreason::Reason reason, int64_t millis)
+{
+int64_t sliceBudget;
+if (millis)
+sliceBudget = SliceBudget::TimeBudget(millis);
+else if (rt->gcHighFrequencyGC && rt->gcDynamicMarkSlice)
+sliceBudget = rt->gcSliceBudget * IGC_MARK_SLICE_MULTIPLIER;
+else
+sliceBudget = rt->gcSliceBudget;
+Collect(rt, true, sliceBudget, gckind, reason);
+}
+void
+js::GCFinalSlice(JSRuntime *rt, JSGCInvocationKind gckind, JS::gcreason::Reason reason)
+{
+Collect(rt, true, SliceBudget::Unlimited, gckind, reason);
+}
+static bool
+ZonesSelected(JSRuntime *rt)
+{
+for (ZonesIter zone(rt, WithAtoms); !zone.done(); zone.next()) {
+if (zone->isGCScheduled())
+return true;
+}
+return false;
+}
+void
+js::GCDebugSlice(JSRuntime *rt, bool limit, int64_t objCount)
+{
+int64_t budget = limit ? SliceBudget::WorkBudget(objCount) : SliceBudget::Unlimited;
+if (!ZonesSelected(rt)) {
+if (JS::IsIncrementalGCInProgress(rt))
+JS::PrepareForIncrementalGC(rt);
+else
+JS::PrepareForFullGC(rt);
+}
+Collect(rt, true, budget, GC_NORMAL, JS::gcreason::DEBUG_GC);
+}
+/* Schedule a full GC unless a zone will already be collected. */
+void
+js::PrepareForDebugGC(JSRuntime *rt)
+{
+if (!ZonesSelected(rt))
+JS::PrepareForFullGC(rt);
+}
+JS_FRIEND_API(void)
+JS::ShrinkGCBuffers(JSRuntime *rt)
+{
+AutoLockGC lock(rt);
+JS_ASSERT(!rt->isHeapBusy());
+if (!rt->useHelperThreads())
+ExpireChunksAndArenas(rt, true);
+else
+rt->gcHelperThread.startBackgroundShrink();
+}
+void
+js::MinorGC(JSRuntime *rt, JS::gcreason::Reason reason)
+{
+#ifdef JSGC_GENERATIONAL
+TraceLogger *logger = TraceLoggerForMainThread(rt);
+AutoTraceLog logMinorGC(logger, TraceLogger::MinorGC);
+rt->gcNursery.collect(rt, reason, nullptr);
+JS_ASSERT_IF(!rt->mainThread.suppressGC, rt->gcNursery.isEmpty());
+#endif
+}
+void
+js::MinorGC(JSContext *cx, JS::gcreason::Reason reason)
+{
+// Alternate to the runtime-taking form above which allows marking type
+// objects as needing pretenuring.
+#ifdef JSGC_GENERATIONAL
+TraceLogger *logger = TraceLoggerForMainThread(cx->runtime());
+AutoTraceLog logMinorGC(logger, TraceLogger::MinorGC);
+Nursery::TypeObjectList pretenureTypes;
+JSRuntime *rt = cx->runtime();
+rt->gcNursery.collect(cx->runtime(), reason, &pretenureTypes);
+for (size_t i = 0; i < pretenureTypes.length(); i++) {
+if (pretenureTypes[i]->canPreTenure())
+pretenureTypes[i]->setShouldPreTenure(cx);
+}
+JS_ASSERT_IF(!rt->mainThread.suppressGC, rt->gcNursery.isEmpty());
+#endif
+}
+void
+js::gc::GCIfNeeded(JSContext *cx)
+{
+JSRuntime *rt = cx->runtime();
+#ifdef JSGC_GENERATIONAL
+/*
+* In case of store buffer overflow perform minor GC first so that the
+* correct reason is seen in the logs.
+*/
+if (rt->gcStoreBuffer.isAboutToOverflow())
+MinorGC(cx, JS::gcreason::FULL_STORE_BUFFER);
+#endif
+if (rt->gcIsNeeded)
+GCSlice(rt, GC_NORMAL, rt->gcTriggerReason);
+}
+void
+js::gc::FinishBackgroundFinalize(JSRuntime *rt)
+{
+rt->gcHelperThread.waitBackgroundSweepEnd();
+}
+AutoFinishGC::AutoFinishGC(JSRuntime *rt)
+{
+if (JS::IsIncrementalGCInProgress(rt)) {
+JS::PrepareForIncrementalGC(rt);
+JS::FinishIncrementalGC(rt, JS::gcreason::API);
+}
+gc::FinishBackgroundFinalize(rt);
+}
+AutoPrepareForTracing::AutoPrepareForTracing(JSRuntime *rt, ZoneSelector selector)
+: finish(rt),
+session(rt),
+copy(rt, selector)
+{
+RecordNativeStackTopForGC(rt);
+}
+JSCompartment *
+js::NewCompartment(JSContext *cx, Zone *zone, JSPrincipals *principals,
+const JS::CompartmentOptions &options)
+{
+JSRuntime *rt = cx->runtime();
+JS_AbortIfWrongThread(rt);
+ScopedJSDeletePtr<Zone> zoneHolder;
+if (!zone) {
+zone = cx->new_<Zone>(rt);
+if (!zone)
+return nullptr;
+zoneHolder.reset(zone);
+zone->setGCLastBytes(8192, GC_NORMAL);
+const JSPrincipals *trusted = rt->trustedPrincipals();
+zone->isSystem = principals && principals == trusted;
+}
+ScopedJSDeletePtr<JSCompartment> compartment(cx->new_<JSCompartment>(zone, options));
+if (!compartment || !compartment->init(cx))
+return nullptr;
+// Set up the principals.
+JS_SetCompartmentPrincipals(compartment, principals);
+AutoLockGC lock(rt);
+if (!zone->compartments.append(compartment.get())) {
+js_ReportOutOfMemory(cx);
+return nullptr;
+}
+if (zoneHolder && !rt->zones.append(zone)) {
+js_ReportOutOfMemory(cx);
+return nullptr;
+}
+zoneHolder.forget();
+return compartment.forget();
+}
+void
+gc::MergeCompartments(JSCompartment *source, JSCompartment *target)
+{
+// The source compartment must be specifically flagged as mergable.  This
+// also implies that the compartment is not visible to the debugger.
+JS_ASSERT(source->options_.mergeable());
+JSRuntime *rt = source->runtimeFromMainThread();
+AutoPrepareForTracing prepare(rt, SkipAtoms);
+// Cleanup tables and other state in the source compartment that will be
+// meaningless after merging into the target compartment.
+source->clearTables();
+// Fixup compartment pointers in source to refer to target.
+for (CellIter iter(source->zone(), FINALIZE_SCRIPT); !iter.done(); iter.next()) {
+JSScript *script = iter.get<JSScript>();
+JS_ASSERT(script->compartment() == source);
+script->compartment_ = target;
+}
+for (CellIter iter(source->zone(), FINALIZE_BASE_SHAPE); !iter.done(); iter.next()) {
+BaseShape *base = iter.get<BaseShape>();
+JS_ASSERT(base->compartment() == source);
+base->compartment_ = target;
+}
+// Fixup zone pointers in source's zone to refer to target's zone.
+for (size_t thingKind = 0; thingKind != FINALIZE_LIMIT; thingKind++) {
+for (ArenaIter aiter(source->zone(), AllocKind(thingKind)); !aiter.done(); aiter.next()) {
+ArenaHeader *aheader = aiter.get();
+aheader->zone = target->zone();
+}
+}
+// The source should be the only compartment in its zone.
+for (CompartmentsInZoneIter c(source->zone()); !c.done(); c.next())
+JS_ASSERT(c.get() == source);
+// Merge the allocator in source's zone into target's zone.
+target->zone()->allocator.arenas.adoptArenas(rt, &source->zone()->allocator.arenas);
+target->zone()->gcBytes += source->zone()->gcBytes;
+source->zone()->gcBytes = 0;
+// Merge other info in source's zone into target's zone.
+target->zone()->types.typeLifoAlloc.transferFrom(&source->zone()->types.typeLifoAlloc);
+}
+void
+gc::RunDebugGC(JSContext *cx)
+{
+#ifdef JS_GC_ZEAL
+JSRuntime *rt = cx->runtime();
+int type = rt->gcZeal();
+if (rt->mainThread.suppressGC)
+return;
+if (type == js::gc::ZealGenerationalGCValue)
+return MinorGC(rt, JS::gcreason::DEBUG_GC);
+PrepareForDebugGC(cx->runtime());
+if (type == ZealIncrementalRootsThenFinish ||
+type == ZealIncrementalMarkAllThenFinish ||
+type == ZealIncrementalMultipleSlices)
+{
+js::gc::State initialState = rt->gcIncrementalState;
+int64_t budget;
+if (type == ZealIncrementalMultipleSlices) {
+/*
+* Start with a small slice limit and double it every slice. This
+* ensure that we get multiple slices, and collection runs to
+* completion.
+*/
+if (initialState == NO_INCREMENTAL)
+rt->gcIncrementalLimit = rt->gcZealFrequency / 2;
+else
+rt->gcIncrementalLimit *= 2;
+budget = SliceBudget::WorkBudget(rt->gcIncrementalLimit);
+} else {
+// This triggers incremental GC but is actually ignored by IncrementalMarkSlice.
+budget = SliceBudget::WorkBudget(1);
+}
+Collect(rt, true, budget, GC_NORMAL, JS::gcreason::DEBUG_GC);
+/*
+* For multi-slice zeal, reset the slice size when we get to the sweep
+* phase.
+*/
+if (type == ZealIncrementalMultipleSlices &&
+initialState == MARK && rt->gcIncrementalState == SWEEP)
+{
+rt->gcIncrementalLimit = rt->gcZealFrequency / 2;
+}
+} else {
+Collect(rt, false, SliceBudget::Unlimited, GC_NORMAL, JS::gcreason::DEBUG_GC);
+}
+#endif
+}
+void
+gc::SetDeterministicGC(JSContext *cx, bool enabled)
+{
+#ifdef JS_GC_ZEAL
+JSRuntime *rt = cx->runtime();
+rt->gcDeterministicOnly = enabled;
+#endif
+}
+void
+gc::SetValidateGC(JSContext *cx, bool enabled)
+{
+JSRuntime *rt = cx->runtime();
+rt->gcValidate = enabled;
+}
+void
+gc::SetFullCompartmentChecks(JSContext *cx, bool enabled)
+{
+JSRuntime *rt = cx->runtime();
+rt->gcFullCompartmentChecks = enabled;
+}
+#ifdef DEBUG
+/* Should only be called manually under gdb */
+void PreventGCDuringInteractiveDebug()
+{
+TlsPerThreadData.get()->suppressGC++;
+}
+#endif
+void
+js::ReleaseAllJITCode(FreeOp *fop)
+{
+#ifdef JS_ION
+# ifdef JSGC_GENERATIONAL
+/*
+* Scripts can entrain nursery things, inserting references to the script
+* into the store buffer. Clear the store buffer before discarding scripts.
+*/
+MinorGC(fop->runtime(), JS::gcreason::EVICT_NURSERY);
+# endif
+for (ZonesIter zone(fop->runtime(), SkipAtoms); !zone.done(); zone.next()) {
+if (!zone->jitZone())
+continue;
+# ifdef DEBUG
+/* Assert no baseline scripts are marked as active. */
+for (CellIter i(zone, FINALIZE_SCRIPT); !i.done(); i.next()) {
+JSScript *script = i.get<JSScript>();
+JS_ASSERT_IF(script->hasBaselineScript(), !script->baselineScript()->active());
+}
+# endif
+/* Mark baseline scripts on the stack as active. */
+jit::MarkActiveBaselineScripts(zone);
+jit::InvalidateAll(fop, zone);
+for (CellIter i(zone, FINALIZE_SCRIPT); !i.done(); i.next()) {
+JSScript *script = i.get<JSScript>();
+jit::FinishInvalidation<SequentialExecution>(fop, script);
+jit::FinishInvalidation<ParallelExecution>(fop, script);
+/*
+* Discard baseline script if it's not marked as active. Note that
+* this also resets the active flag.
+*/
+jit::FinishDiscardBaselineScript(fop, script);
+}
+zone->jitZone()->optimizedStubSpace()->free();
+}
+#endif
+}
+/*
+* There are three possible PCCount profiling states:
+*
+* 1. None: Neither scripts nor the runtime have count information.
+* 2. Profile: Active scripts have count information, the runtime does not.
+* 3. Query: Scripts do not have count information, the runtime does.
+*
+* When starting to profile scripts, counting begins immediately, with all JIT
+* code discarded and recompiled with counts as necessary. Active interpreter
+* frames will not begin profiling until they begin executing another script
+* (via a call or return).
+*
+* The below API functions manage transitions to new states, according
+* to the table below.
+*
+*                                  Old State
+*                          -------------------------
+* Function                 None      Profile   Query
+* --------
+* StartPCCountProfiling    Profile   Profile   Profile
+* StopPCCountProfiling     None      Query     Query
+* PurgePCCounts            None      None      None
+*/
+static void
+ReleaseScriptCounts(FreeOp *fop)
+{
+JSRuntime *rt = fop->runtime();
+JS_ASSERT(rt->scriptAndCountsVector);
+ScriptAndCountsVector &vec = *rt->scriptAndCountsVector;
+for (size_t i = 0; i < vec.length(); i++)
+vec[i].scriptCounts.destroy(fop);
+fop->delete_(rt->scriptAndCountsVector);
+rt->scriptAndCountsVector = nullptr;
+}
+JS_FRIEND_API(void)
+js::StartPCCountProfiling(JSContext *cx)
+{
+JSRuntime *rt = cx->runtime();
+if (rt->profilingScripts)
+return;
+if (rt->scriptAndCountsVector)
+ReleaseScriptCounts(rt->defaultFreeOp());
+ReleaseAllJITCode(rt->defaultFreeOp());
+rt->profilingScripts = true;
+}
+JS_FRIEND_API(void)
+js::StopPCCountProfiling(JSContext *cx)
+{
+JSRuntime *rt = cx->runtime();
+if (!rt->profilingScripts)
+return;
+JS_ASSERT(!rt->scriptAndCountsVector);
+ReleaseAllJITCode(rt->defaultFreeOp());
+ScriptAndCountsVector *vec = cx->new_<ScriptAndCountsVector>(SystemAllocPolicy());
+if (!vec)
+return;
+for (ZonesIter zone(rt, SkipAtoms); !zone.done(); zone.next()) {
+for (CellIter i(zone, FINALIZE_SCRIPT); !i.done(); i.next()) {
+JSScript *script = i.get<JSScript>();
+if (script->hasScriptCounts() && script->types) {
+ScriptAndCounts sac;
+sac.script = script;
+sac.scriptCounts.set(script->releaseScriptCounts());
+if (!vec->append(sac))
+sac.scriptCounts.destroy(rt->defaultFreeOp());
+}
+}
+}
+rt->profilingScripts = false;
+rt->scriptAndCountsVector = vec;
+}
+JS_FRIEND_API(void)
+js::PurgePCCounts(JSContext *cx)
+{
+JSRuntime *rt = cx->runtime();
+if (!rt->scriptAndCountsVector)
+return;
+JS_ASSERT(!rt->profilingScripts);
+ReleaseScriptCounts(rt->defaultFreeOp());
+}
+void
+js::PurgeJITCaches(Zone *zone)
+{
+#ifdef JS_ION
+for (CellIterUnderGC i(zone, FINALIZE_SCRIPT); !i.done(); i.next()) {
+JSScript *script = i.get<JSScript>();
+/* Discard Ion caches. */
+jit::PurgeCaches(script);
+}
+#endif
+}
+void
+ArenaLists::normalizeBackgroundFinalizeState(AllocKind thingKind)
+{
+volatile uintptr_t *bfs = &backgroundFinalizeState[thingKind];
+switch (*bfs) {
+case BFS_DONE:
+break;
+case BFS_JUST_FINISHED:
+// No allocations between end of last sweep and now.
+// Transfering over arenas is a kind of allocation.
+*bfs = BFS_DONE;
+break;
+default:
+JS_ASSERT(!"Background finalization in progress, but it should not be.");
+break;
+}
+}
+void
+ArenaLists::adoptArenas(JSRuntime *rt, ArenaLists *fromArenaLists)
+{
+// The other parallel threads have all completed now, and GC
+// should be inactive, but still take the lock as a kind of read
+// fence.
+AutoLockGC lock(rt);
+fromArenaLists->purge();
+for (size_t thingKind = 0; thingKind != FINALIZE_LIMIT; thingKind++) {
+#ifdef JS_THREADSAFE
+// When we enter a parallel section, we join the background
+// thread, and we do not run GC while in the parallel section,
+// so no finalizer should be active!
+normalizeBackgroundFinalizeState(AllocKind(thingKind));
+fromArenaLists->normalizeBackgroundFinalizeState(AllocKind(thingKind));
+#endif
+ArenaList *fromList = &fromArenaLists->arenaLists[thingKind];
+ArenaList *toList = &arenaLists[thingKind];
+while (fromList->head != nullptr) {
+// Remove entry from |fromList|
+ArenaHeader *fromHeader = fromList->head;
+fromList->head = fromHeader->next;
+fromHeader->next = nullptr;
+// During parallel execution, we sometimes keep empty arenas
+// on the lists rather than sending them back to the chunk.
+// Therefore, if fromHeader is empty, send it back to the
+// chunk now. Otherwise, attach to |toList|.
+if (fromHeader->isEmpty())
+fromHeader->chunk()->releaseArena(fromHeader);
+else
+toList->insert(fromHeader);
+}
+fromList->cursor = &fromList->head;
+}
+}
+bool
+ArenaLists::containsArena(JSRuntime *rt, ArenaHeader *needle)
+{
+AutoLockGC lock(rt);
+size_t allocKind = needle->getAllocKind();
+for (ArenaHeader *aheader = arenaLists[allocKind].head;
+aheader != nullptr;
+aheader = aheader->next)
+{
+if (aheader == needle)
+return true;
+}
+return false;
+}
+AutoMaybeTouchDeadZones::AutoMaybeTouchDeadZones(JSContext *cx)
+: runtime(cx->runtime()),
+markCount(runtime->gcObjectsMarkedInDeadZones),
+inIncremental(JS::IsIncrementalGCInProgress(runtime)),
+manipulatingDeadZones(runtime->gcManipulatingDeadZones)
+{
+runtime->gcManipulatingDeadZones = true;
+}
+AutoMaybeTouchDeadZones::AutoMaybeTouchDeadZones(JSObject *obj)
+: runtime(obj->compartment()->runtimeFromMainThread()),
+markCount(runtime->gcObjectsMarkedInDeadZones),
+inIncremental(JS::IsIncrementalGCInProgress(runtime)),
+manipulatingDeadZones(runtime->gcManipulatingDeadZones)
+{
+runtime->gcManipulatingDeadZones = true;
+}
+AutoMaybeTouchDeadZones::~AutoMaybeTouchDeadZones()
+{
+runtime->gcManipulatingDeadZones = manipulatingDeadZones;
+if (inIncremental && runtime->gcObjectsMarkedInDeadZones != markCount) {
+JS::PrepareForFullGC(runtime);
+js::GC(runtime, GC_NORMAL, JS::gcreason::TRANSPLANT);
+}
+}
+AutoSuppressGC::AutoSuppressGC(ExclusiveContext *cx)
+: suppressGC_(cx->perThreadData->suppressGC)
+{
+suppressGC_++;
+}
+AutoSuppressGC::AutoSuppressGC(JSCompartment *comp)
+: suppressGC_(comp->runtimeFromMainThread()->mainThread.suppressGC)
+{
+suppressGC_++;
+}
+AutoSuppressGC::AutoSuppressGC(JSRuntime *rt)
+: suppressGC_(rt->mainThread.suppressGC)
+{
+suppressGC_++;
+}
+bool
+js::UninlinedIsInsideNursery(JSRuntime *rt, const void *thing)
+{
+return IsInsideNursery(rt, thing);
+}
+#ifdef DEBUG
+AutoDisableProxyCheck::AutoDisableProxyCheck(JSRuntime *rt
+MOZ_GUARD_OBJECT_NOTIFIER_PARAM_IN_IMPL)
+: count(rt->gcDisableStrictProxyCheckingCount)
+{
+MOZ_GUARD_OBJECT_NOTIFIER_INIT;
+count++;
+}
+JS_FRIEND_API(void)
+JS::AssertGCThingMustBeTenured(JSObject *obj)
+{
+JS_ASSERT((!IsNurseryAllocable(obj->tenuredGetAllocKind()) || obj->getClass()->finalize) &&
+obj->isTenured());
+}
+JS_FRIEND_API(size_t)
+JS::GetGCNumber()
+{
+JSRuntime *rt = js::TlsPerThreadData.get()->runtimeFromMainThread();
+if (!rt)
+return 0;
+return rt->gcNumber;
+}
+JS::AutoAssertNoGC::AutoAssertNoGC()
+: runtime(nullptr), gcNumber(0)
+{
+js::PerThreadData *data = js::TlsPerThreadData.get();
+if (data) {
+/*
+* GC's from off-thread will always assert, so off-thread is implicitly
+* AutoAssertNoGC. We still need to allow AutoAssertNoGC to be used in
+* code that works from both threads, however. We also use this to
+* annotate the off thread run loops.
+*/
+runtime = data->runtimeIfOnOwnerThread();
+if (runtime)
+gcNumber = runtime->gcNumber;
+}
+}
+JS::AutoAssertNoGC::AutoAssertNoGC(JSRuntime *rt)
+: runtime(rt), gcNumber(rt->gcNumber)
+{
+}
+JS::AutoAssertNoGC::~AutoAssertNoGC()
+{
+if (runtime)
+MOZ_ASSERT(gcNumber == runtime->gcNumber, "GC ran inside an AutoAssertNoGC scope.");
+}
+#endif

The Tor Browser / file comparison

comparison: js/src/jsgc.cpp

js/src/jsgc.cpp