layout/doc/regression_tests.html@7e26c7da4463
layout/doc/regression_tests.html
Wed, 31 Dec 2014 06:55:50 +0100
- author
- Michael Schloh von Bennewitz <michael@schloh.com>
- date
- Wed, 31 Dec 2014 06:55:50 +0100
- changeset 2
- 7e26c7da4463
- permissions
- -rw-r--r--
Added tag UPSTREAM_283F7C6 for changeset ca08bd8f51b2
1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
2 <!-- This Source Code Form is subject to the terms of the Mozilla Public
3 - License, v. 2.0. If a copy of the MPL was not distributed with this
4 - file, You can obtain one at http://mozilla.org/MPL/2.0/. -->
6 <html>
7 <head>
8 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
9 <title>Layout Regression tests</title>
10 <style type="text/css">
11 body {font-family:arial}
12 .screen {background-color:silver; color:blue}
13 pre.screen {display:table-cell}
14 </style>
16 </head>
17 <body>
19 <h1>Table Regression Tests</h1>
20 <h6>Bernd Mielke</h6>
21 <h2>About This Document</h2>
22 <p>
23 This document describes my guerrilla approach to successfully running the table layout regression tests. It should encourage more people to run those tests.
25 <h2>Subject Overview</h2>
26 <p>
27 The table element and its children are one of the major layout components that can create some structure on the screen. They are widely (ab)used by page authors. Every small change in the table code can generate a bunch of bugzilla entries and a lot of seemingly unrelated dupes. With the high frequency of checkins in to the source tree it is difficult and time consuming to track the regression down to a single checkin. The risk of becoming very infamous can be reduced by running the layout regression tests.
29 <h2>Prerequisites</h2>
30 <p>
31 In order to run these tests, you will need to have:
32 <ol>
33 <li>a debug build with a working viewer, an optimized build will not work!
34 <li>a mozilla tree with the testfiles at <code>%MOZ_SRC%/layout/html/tests</code>,
35 <li> <code>DISABLE_TESTS</code> should not be defined,
36 <li>patience and time.
37 </ol>
39 <h2>Test Runs</h2>
41 <ul>
42 <li>Go to the <code>%MOZ_SRC%/layout/html/tests/block</code> directory.
43 <li> Avoid the nasty assertions to pop up by <code>set XPCOM_DEBUG_BREAK=warn</code>, the world would be a much better place if we could <a href="http://bugzilla.mozilla.org/show_bug.cgi?id=152015">first fix those assertions</a>.
44 <li>The following shell scripts need a <code>sh</code> before the command under WinXX.
45 <li>Execute <code>rtest.sh baseline</code>
46 <li>Execute <code>rtest.sh verify>outputfile.txt</code>
47 <li>Make a note which tests have failed (these are false positives. If you are in bad mood file a bug against Chris Waterson about them)
48 <li>you can grep the failures by:<br>
49 <code>grep 'rgd failed' <outputfile.txt >false_positive.txt</code>
50 <li> Make a copy of <code>gklayout.dll</code> and <code>gkcontent.dll</code> in the <code>dist</code> directory. If you would like to switch back, you will need them.
51 <li>Make your changes to the source, compile and install.
52 <li>execute in the <code>layout/html/tests/block</code> directory <code>rtest.sh verify >regression.txt</code>
53 <li>Check how many tests have failed and analyse your results:
54 <ul>
55 <li>If the regression tests have failed at the same place as the previous diagnosed false positives => ignore
56 <ol>
57 <li>copy <code>rtest.sh</code> to <code>my_rtest.sh</code>
58 <li>create a new directory <code>my_files</code>.
59 <li>create in this directory a file <code>file_list.txt</code>, where you put the url's of your remaining test failures
60 <li>change <code>my_rtest.sh</code> to use only directory <code>my_files</code>
61 <li>go back to the original gklayout.dll and gkcontent.dll and repeat the regression test procedure on a much smaller set now.
62 </ol>
63 <li>Try to figure out what the other regressions are: Are they
64 improving the picture or are they regressions?
65 <li>Note the difficult to explain testfailures.
66 </ul>
67 <li> Change the source if necessary until the regression tests indicate that your patch does not hork the block and table layout
68 <li> Submit your patch and lists the remaining differences in the bug
69 </ul>
71 <p>Be prepared to cycle.
73 <h2>How the layout regression tests work</h2>
75 <p>First the URL's to test are red from <code>file_list.txt</code>.<p>
76 <table><caption>snippet from file_list.txt</caption><tr><td>
77 <pre>file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html
78 file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html
79 file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html
80 file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html
81 file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html
82 file:///s|/mozilla/layout/html/tests/table/dom/appendRowsExpand1.html</pre>
83 </td></tr></table>
84 <p>The regression tests compare the frame tree dumps recorded at two different times. The URL's to test are red from <code>file_list.txt</code>
85 For every file in <code>file_list.txt</code>viewer takes the root frame and dumps the whole tree via
86 <a href="http://lxr.mozilla.org/seamonkey/search?string=nsFrame%3A%3ADumpRegression">
87 nsFrame::DumpRegression</a>.
88 <p>
89 The frame tree dumps are invoked by the <code>-o</code> switch to the <code>viewer</code> (see <a href="http://lxr.mozilla.org/seamonkey/ident?i=PrintHelpInfo">commandline arguments </a>).
90 When <code>-o</code> is specified together with a output directory for instance (<code>-o s:\mozilla\layout\html\tests\table\dom\</code>),
91 the frame trees are dumped to separate files in that directory for all URL's in <code>file_list.txt</code> (see:
92 <a href="http://lxr.mozilla.org/seamonkey/search?string=nsWebCrawler%3A%3ADumpRegression">
93 nsWebCrawler::DumpRegressionData</a>) .
94 <p>Their name is generated by replacing the old extension <code>.html</code> or <code>.xml</code> with <code>.rgd</code>. For <code>file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html</code> a file <code>appendCells1.rgd</code> will be created in the <code>s:\mozilla\layout\html\tests\table\dom\</code> directory.
96 <p>
97 There is a special type of frame dumps - the printing regression tests, they are invoked by the <code>-Prt</code> command line argument to the <code>viewer</code> and include first a display then a frame dump taking into account the printer pages. A typical beginning of a dump (<code>*.rgd</code> file) looks like:<p>
98 <pre class="screen">
99 <frame va="15022440" type="Viewport(-1)" state="270340" parent="0">
100 <view va="47171904">
101 </view>
102 <stylecontext va="15022232">
103 <font serif 240 240 0 />
104 <color data="-16777216"/>
105 <background data="0 2 3 -1 0 0 "/>
106 <spacing data="left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0" />
107 <list data="100 100 " />
108 <position data="left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 0 Auto " />
109 <text data="0 0 0 Normal Normal 0[0x0]tw Normal " />
110 <textreset data="0 10[0xa]enum " />
111 <display data="0 1 0 0 0 0 0 0 0 0 0 0 " />
112 <visibility data="0 1 1.000000" />
113 <table data="0 0 4 -1 1 " />
114 <tableborder data="1 Null Null 0 2 " />
115 <content data="0 0 0 Null " />
116 <quotes data="0 " />
117 <ui data="3 0 0 1 " />
118 <uireset data="7 0 4" />
119 <xul data="0 0 0 0 0 1 <svg data="0 1.000000 1.000000 0 1.000000" />
120 </stylecontext></pre>
121 <p>The first run of the regression tests is started with the argument <code>baseline</code> and it creates the reference frame dumps.
122 <p>The <code>baseline</code> log will look like:
123 <pre class="screen">
124 Type Manifest File: C:\MOZ_SOUR\MOZILLA\MOZILLA\DIST\WIN32_D.OBJ\BIN\components\xpti.dat
125 nsNativeComponentLoader: autoregistering begins.
126 nsNativeComponentLoader: autoregistering succeeded
127 Going to create the event queue
128 WEBSHELL+ = 1
129 Note: verifyreflow is disabled
130 Note: styleverifytree is disabled
131 Note: frameverifytree is disabled
132 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html: done loading (11810 msec)
133 appendCells1.rgd - being written
134 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html: done loading (1260 msec)
135 appendCellsRebuild1.rgd - being written
136 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html: done loading (270 msec)
137 appendCol1.rgd - being written
138 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html: done loading (220 msec)
139 appendCol2.rgd - being written
140 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html: done loading (390 msec)
141 appendColGroup1.rgd - being written
142 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendRowsExpand1.html: done loading (220 msec)
143 appendRowsExpand1.rgd - being written
144 </pre>
145 <p>
146 The second run with the argument <code>verify</code> creates first the frame dumps in a subdirectory (<code>verify</code>) and <a href="http://lxr.mozilla.org/seamonkey/search?string=nsFrameUtil%3A%3ACompareTrees">compares</a> each frame dump with the corresponding reference frame dump. If they differ in critical points the test fails.
147 <p>A typical part of the <code>verify</code> log would look like:
148 <pre class="screen">
149 Type Manifest File: C:\MOZ_SOUR\MOZILLA\MOZILLA\DIST\WIN32_D.OBJ\BIN\components\xpti.dat
150 nsNativeComponentLoader: autoregistering begins.
151 nsNativeComponentLoader: autoregistering succeeded
152 Going to create the event queue
153 WEBSHELL+ = 1
154 Note: verifyreflow is disabled
155 Note: styleverifytree is disabled
156 Note: frameverifytree is disabled
157 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html: done loading (2200 msec)
158 frame bbox mismatch: 0,285,930,435 vs. 0,285,1305,435
159 Node 1:
160 TableOuter(table)(1) 0x10004 0,285,930,435, |null attr|-16777216|left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Inherit top: Inherit right: Inherit bottom: Inherit 0[0x0]tw 0[0x0]tw Null 0[0x0]tw 0[0x0]tw Null 0 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 8 0 0 0 0 0 1 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
161 Node 2:
162 TableOuter(table)(1) 0x10004 0,285,1305,435, |null attr|-16777216|left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Inherit top: Inherit right: Inherit bottom: Inherit 0[0x0]tw 0[0x0]tw Null 0[0x0]tw 0[0x0]tw Null 0 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 8 0 0 0 0 0 1 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
163 frame bbox mismatch: 0,0,930,435 vs. 0,0,1305,435
164 Node 1:
165 Table(table)(1) 0x10004 0,0,930,435, |null attr|-16777216|left: Null top: 0[0x0]tw right: Null bottom: 0[0x0]tw left: Null top: Null right: Null bottom: Null left: 15[0xf]tw top: 15[0xf]tw right: 15[0xf]tw bottom: 15[0xf]tw left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 2 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 8 0 0 0 0 0 0 0 0 0 0 |0 8 4 -1 1 |0 0 0 Null
166 Node 2:
167 Table(table)(1) 0x10004 0,0,1305,435, |null attr|-16777216|left: Null top: 0[0x0]tw right: Null bottom: 0[0x0]tw left: Null top: Null right: Null bottom: Null left: 15[0xf]tw top: 15[0xf]tw right: 15[0xf]tw bottom: 15[0xf]tw left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 2 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 8 0 0 0 0 0 0 0 0 0 0 |0 8 4 -1 1 |0 0 0 Null
168 frame bbox mismatch: 45,15,840,405 vs. 45,15,1215,405
169 Node 1:
170 TableColGroup(table)(1) 0x80010004 45,15,840,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 0 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 12 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
171 Node 2:
172 TableColGroup(table)(1) 0x80010004 45,15,1215,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 0 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 12 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
173 frame state mismatch: 0x30010406 vs. 0x30010004
174 Node 1:
175 TableCol(table)(1) 0x30010406 0,0,0,0, |null attr|-16777216|left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 0 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
176 Node 2:
177 TableCol(table)(1) 0x30010004 870,0,345,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 0 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
178 frame bbox mismatch: 0,0,0,0 vs. 870,0,345,405
179 Node 1:
180 TableCol(table)(1) 0x30010406 0,0,0,0, |null attr|-16777216|left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 0 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
181 Node 2:
182 TableCol(table)(1) 0x30010004 870,0,345,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum left: Null top: Null right: Null bottom: Null left: Null top: Null right: Null bottom: Null 1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto Auto 0[0x0]tw Null Auto 0[0x0]tw Null 0 Auto |0 0 0 Normal Normal 0[0x0]tw Normal |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
183 regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCells1.rgd failed
184 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html: done loading (390 msec)
185 044regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCellsRebuild1.rgd passed
186 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html: done loading (280 msec)
187 regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCol1.rgd passed
188 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html: done loading (220 msec)
189 regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCol2.rgd passed
190 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html: done loading (220 msec)
191 regression test s:\mozilla\layout\html\tests\table\dom\verify\appendColGroup1.rgd passed
192 </pre>
193 <p>
194 The first regression test has failed and it is marked so.
197 <h2>Adding new regression tests</h2>
198 <p>
199 Once you have checked in the code:
200 <ul>
201 <li> please add your testcase for the bug to the regression tests.
202 <li>A testcase should only contain local files.
203 <li>Place the images in <code>table/images</code> (don't forget <code> cvs commit -kb </code>)
204 <li>and the testfile in the <code>table/bugs</code> directory.
205 <li>Update the <code>file_list<b>x</b>.txt</code> in order to include your file.
206 </ul>
207 <p>I would like to thank Chris Karnaze for his guidance and fantasai for his language support.
208 <div style="text-align:right"><i><regression.html> <Last updated:
209 2002-06-16 <a href="mailto:bernd.mielke@snafu.de">Bernd Mielke</a></i></div>
211 </body>
212 </html>
