The Tor Browser / file comparison
comparison: security/nss/tests/doc/qa_wrapper.html
security/nss/tests/doc/qa_wrapper.html
- branch
- TOR_BUG_9701
- changeset 15
- b8a032363ba2
equal
deleted
inserted
replaced
| -1:000000000000 | 0:154930f9eaba |
|---|---|
| 1 <!doctype html public "-//w3c//dtd html 4.0 transitional//en"> | |
| 2 <html> | |
| 3 <head> | |
| 4 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> | |
| 5 <meta name="GENERATOR" content="Mozilla/4.7 [en] (X11; U; SunOS 5.8 sun4u) [Netscape]"> | |
| 6 </head> | |
| 7 <body text="#000000" bgcolor="#FFFFFF" link="#0000EE" vlink="#551A8B" alink="#FF0000"> | |
| 8 | |
| 9 <h3> | |
| 10 <b><font face="Times New Roman,Times">Author Sonja Mirtitsch</font></b></h3> | |
| 11 | |
| 12 <h3> | |
| 13 <b><font face="Times New Roman,Times">Last updated: 4/4/2001</font></b></h3> | |
| 14 | |
| 15 <h1> | |
| 16 <b><font face="Times New Roman,Times">NSS 3.2.QA Wrapper</font></b></h1> | |
| 17 | |
| 18 <p><br>The QA wrapper tests the nightly builds of NSS. The actual | |
| 19 tests are being run are called from the QA script all.sh. I will add documentation | |
| 20 for the actual QA soon. The main purpose of the wrapper is: find out which | |
| 21 build (NSS version, date, Build Platform) to test on which machine (OS, | |
| 22 OS version) and construct a summary report, which is then mailed to the | |
| 23 nss developers (aka mailing list nss-qa-report@netscape.com). Please see | |
| 24 also the <a href="#advertisement">feature</a> section. | |
| 25 <p><a href="#nssqa">nssqa</a> - the script that calls the actual | |
| 26 qa script all.sh | |
| 27 <br><a href="#qa_stat">qa_stat</a> - sends out status reports | |
| 28 <br><a href="#qaclean">qaclean</a> - if everything else fails | |
| 29 <p>Sample <a href="/u/sonmi/doc/publish/glob_result.html">global result</a>, | |
| 30 <a href="/u/sonmi/doc/publish/results.html">individual result </a>and <a href="/u/sonmi/doc/publish/output.log">log | |
| 31 files</a> | |
| 32 <p>The QA wrapper consistst mainly of scripts, most located in security/nss/tests | |
| 33 and subdirectories, but run from /u/sonmi/bin | |
| 34 <p>nssqa and qa_stat, the main scripts both include a common header (<a href="../header">header</a>) | |
| 35 and a common environment (<a href="../set_environment">set_environment</a>). | |
| 36 <br>Also used is <a href="../mksymlinks">mksymlinks</a> and <a href="../path_uniq">path_uniq</a> | |
| 37 and <a href="#qaclean">qaclean</a>. | |
| 38 <p>The scripts that are used on a daily basis are located in /u/sonmi/bin | |
| 39 and checked into security/nss/tests | |
| 40 <p>Parameters and Options are the same for most scripts. | |
| 41 <p><a NAME="Parameters"></a><b><u><font size=+1>Parameters</font></u></b> | |
| 42 <br> nssversion (supported: 30b, 31, tip, default tip) | |
| 43 <br> builddate (default - today, format mmdd) | |
| 44 <p><a NAME="Options"></a><b><u><font size=+1>Options</font></u></b> | |
| 45 <br> -y answer all questions with y - use at your own | |
| 46 risk... ignores warnings | |
| 47 <br> -s silent (only usefull with -y) | |
| 48 <br> -h, -? -help you guessed right - displays the usage | |
| 49 <br> -d debug | |
| 50 <br> -f <filename> - write the (error)output to filename | |
| 51 <br> -fcron writes resultfile in the same location as | |
| 52 would the -cron | |
| 53 <br> -m <mailinglist> - send filename to mailinglist | |
| 54 (csl) only useful | |
| 55 <br> with -f on nssqa | |
| 56 <br> -l <mozroot> run on a local build - does not | |
| 57 work at this time | |
| 58 <br> -cron equivalient to -y -s -d -f $RESULTDIR/$HOST.<scriptname> | |
| 59 <br> | |
| 60 <p>Please be aware that some iPlanet specific environments and features | |
| 61 are being used. | |
| 62 <p>-d Debug option might be removed from cron in a few weeks - or maybe | |
| 63 not | |
| 64 <br>-l QA on local build is not fully implemented yet - will not be implemented, | |
| 65 all.sh can be called directly instead | |
| 66 <p>Please do not use on Windows 95 and 98, ME platforms yet. | |
| 67 <p>use -d if script behaves strange or exits unexpectedly | |
| 68 <p><b><font size=+1>How to use the QA-wrapper</font></b> | |
| 69 <br>To test a build, first run nssqa on the required QA platforms (some | |
| 70 buildplatforms require QA to be run on additional platforms - for example | |
| 71 Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has been run on | |
| 72 multiple or all required platforms it makes sense to run qa_stat on the | |
| 73 output of nssqa as well. | |
| 74 <br>Before used on a new system (even if the same platform has been tested | |
| 75 before) please use completely interactive, to see what the variables are | |
| 76 being initialized to, and read the warnings. Same is true if being run | |
| 77 from a different user account than svbld. | |
| 78 <p>In any case, if you are using it, please let me know the results. | |
| 79 <p><a NAME="nssqa"></a><b><u><font size=+1>nssqa:</font></u></b> | |
| 80 <p>the script that calls the actual qa script all.sh | |
| 81 <p>nssqa <a href="#Parameters">parameters</a> and <a href="#Options">options</a> | |
| 82 <p><a href="../nssqa">view the script</a> | |
| 83 <p><b><u><font size=+1>Pseudocode Description of nssqa</font></u></b> | |
| 84 <br>not quite up to date | |
| 85 <p> header:init (global) | |
| 86 <br> set flags and variables | |
| 87 to default values | |
| 88 <br> signal trap (for interupts | |
| 89 and kills) | |
| 90 <br> set HOST and DOMSUF variables | |
| 91 if running from cron | |
| 92 <br> parse parameters and options | |
| 93 <br> determine os and set up | |
| 94 the environment (espec. PATH) | |
| 95 <br> set the directories to run | |
| 96 in (influenced by parameters and -l option)<br> | |
| 97 set the directories for backward | |
| 98 compatibility testing | |
| 99 <br> set and initialize the tmp | |
| 100 / debugging / output files | |
| 101 <p> nssqa:init (local) | |
| 102 <br> locking: if nssqa is already | |
| 103 running on this systems (yes-exit, | |
| 104 <br> | |
| 105 no-lockfile) | |
| 106 <br> set HOST and DOMSUF variables | |
| 107 if running interavtively | |
| 108 <br> set flag to kill remaining | |
| 109 selfserv processes during cleanup | |
| 110 <br> if QA platform different | |
| 111 from build platform create neccessary | |
| 112 <br> | |
| 113 symbolic links | |
| 114 <br> wait for the build to finish | |
| 115 (max of 5h) | |
| 116 <p> main: | |
| 117 <br> repeated per test (optimized, | |
| 118 debug, 32, 64 bit) | |
| 119 <br> | |
| 120 set flags for this run of all.sh (optimized, debug, 32, 64 bit) | |
| 121 <br> | |
| 122 set the DIST directory (where the binaries reside) | |
| 123 <br> | |
| 124 kill running selfservers (sorry - just don't use the svbld | |
| 125 <br> | |
| 126 account if you need to do your own testing... I will fix | |
| 127 <br> | |
| 128 selfserv as soon as I can - but it hangs too often and | |
| 129 <br> | |
| 130 disturbs all following QA) | |
| 131 <br> | |
| 132 run all.sh | |
| 133 <p> header:exit (global) | |
| 134 <br> remove temporary files | |
| 135 <p> kill remaining selfservers | |
| 136 <br> send email to the list | |
| 137 <br> | |
| 138 <p> errorhandling | |
| 139 <br> Option / Parameter errors: | |
| 140 Exit with usage information | |
| 141 <p> Severe errors: Exit wit errormessage | |
| 142 <br> | |
| 143 example: directory in which all.sh resides does not exist | |
| 144 <br> | |
| 145 can't create files or directories | |
| 146 <br> | |
| 147 build not done after 5 hours | |
| 148 <br> | |
| 149 is already running | |
| 150 <p> Other errors: User is prompted | |
| 151 with the "errormessage - continue (y/n)?" | |
| 152 <br> | |
| 153 example: local DIST dir does not exist (continues with next all.sh) | |
| 154 <br> | |
| 155 outputdirectory does not exist (user can specify other) | |
| 156 <p> Signals 2, 3, 15 are treated | |
| 157 as severe errors | |
| 158 <br> | |
| 159 <br> | |
| 160 <br> | |
| 161 <p><img SRC="clean.gif" height=129 width=92 align=LEFT><a NAME="qaclean"></a><b><u><font size=+2>qaclean:</font></u></b>/u/sonmi/bin/qaclean | |
| 162 <br> | |
| 163 <p>Use qaclean as user "svbld" to get the propper permissions. It is supposed | |
| 164 to clean up after a "hanging" QA and will also brutally kill, interupt | |
| 165 and disturb any other nss related test or performance meassurement on the | |
| 166 named machine. NT and 2000 might require an additional reboot, since the | |
| 167 ps is not so good about telling us the actual programmname - so we can't | |
| 168 kill them... Please note that this is a brute force script, it should not | |
| 169 be used on a regular basis, file a bug whenever you have to use it, since | |
| 170 hanging QA is nothing that should occur frequently | |
| 171 <p> <a href="../qaclean">view the script</a> | |
| 172 <p>What it does: | |
| 173 <ol> | |
| 174 <li> | |
| 175 see if there is a lockfile (/tmp/nssqa.$$ or $TMP/nssqa.$$)</li> | |
| 176 | |
| 177 <br>if yes: | |
| 178 <ol>kill the process of the lockfile <font color="#666666">(future expansion | |
| 179 and if possible it's children )</font> | |
| 180 <br>rm the lockfile</ol> | |
| 181 | |
| 182 <li> | |
| 183 kill selfservers</li> | |
| 184 | |
| 185 <li> | |
| 186 kill whatever other qa related processes might be hanging</li> | |
| 187 | |
| 188 <li> | |
| 189 clean up tmp files</li> | |
| 190 </ol> | |
| 191 <b>QAClean Parameters:</b> | |
| 192 <br> machinename. | |
| 193 <br> for example | |
| 194 <br> qaclean kentuckyderby | |
| 195 <br> started on any machine, will clean up on kentuckyderby | |
| 196 <p><a NAME="qa_stat"></a><b><u><font size=+2>qa_stat</font></u></b> | |
| 197 <p>qa_stat is the script that is being started from the svbld cron on kentuckyderby | |
| 198 every morning at 10:00 and runs some (very primitive) analysis on the qa | |
| 199 results. | |
| 200 <br>I'd like to rewrite the whole thing in perl, and in a few weeks I might | |
| 201 just do this... | |
| 202 <p> <a href="../qa_stat">view the script</a> | |
| 203 <p>qa_stat <a href="#Parameters">parameters</a> and <a href="#Options">options</a> | |
| 204 <p><a NAME="advertisement"></a><b><u><font size=+1>Why we need the QA wrapper</font></u></b> | |
| 205 <p>We need the new QA wrapper, because we have to test on so many platforms, | |
| 206 that running the tests and evaluating the results for the nightly builds | |
| 207 took about an average workday. | |
| 208 <p><b><font size=+1>New Features:</font></b> | |
| 209 <ul> | |
| 210 <li> | |
| 211 runs from <b>cron</b> / rsh or <b>interactive</b> if desired</li> | |
| 212 | |
| 213 <li> | |
| 214 generates <b>summary</b> (no need to look through 60-90 directories)</li> | |
| 215 | |
| 216 <li> | |
| 217 sends <b>email</b> about results</li> | |
| 218 | |
| 219 <li> | |
| 220 automatically <b>recognizes common errors</b> and problems and conflicts | |
| 221 and corrects them</li> | |
| 222 | |
| 223 <br>(or attempts to correct them :-) | |
| 224 <li> | |
| 225 automatically determines <b>which build </b>to test (waits if build in | |
| 226 progress, exits if no build)</li> | |
| 227 | |
| 228 <li> | |
| 229 runs on <b>all required platforms</b> (Windows 98 and before not functional | |
| 230 yet)</li> | |
| 231 | |
| 232 <li> | |
| 233 Windows version runs on <b>free Cygnus</b> as well as on MKS</li> | |
| 234 | |
| 235 <li> | |
| 236 debug mode, normal mode and silent mode</li> | |
| 237 | |
| 238 <li> | |
| 239 <b>locking</b> mechanism so it won't run twice</li> | |
| 240 | |
| 241 <li> | |
| 242 <b>cleanup</b> after being killed and most errors (no remaining selfservers, | |
| 243 tmpfiles, lock files)</li> | |
| 244 </ul> | |
| 245 The 1st script is started via cron between 5:00 and 8:00 am on different | |
| 246 systems, and starts QA on the nightly build. At 10:00 the next script is | |
| 247 started, and sends a QA summary to the nss developers. | |
| 248 <p><b>Cygnus Advantages</b>: | |
| 249 <ul> | |
| 250 <li> | |
| 251 <b>free</b></li> | |
| 252 | |
| 253 <li> | |
| 254 better handling of <b>processes</b> (background, processIDs, Signals)</li> | |
| 255 | |
| 256 <li> | |
| 257 Unix / Linux <b>compatible</b> sh / bash</li> | |
| 258 </ul> | |
| 259 <b>Disadvantages</b> | |
| 260 <ul> | |
| 261 <li> | |
| 262 MKS functionality needs to be preserved (makes <b>8 Windows platforms</b> | |
| 263 instead of 4 for the QA suites - makes 32 testruns on Windows alone)</li> | |
| 264 | |
| 265 <br>In certain functionality's <b>slow</b> | |
| 266 <br><b></b> </ul> | |
| 267 <b>Porting the windows QA to Uwin as well is also being considered</b> | |
| 268 </body> | |
| 269 </html> |
