The Tor Browser: security/nss/tests/doc/qa

Cloned upstream origin tor-browser at tor-browser-31.3.0esr-4.5-1-build1
revision ID fc1c9ff7c1b2defdbc039f12214767608f46423f for hacking purpose.

     1 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">

     2 <html>

     3 <head>

     4    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

     5    <meta name="GENERATOR" content="Mozilla/4.7 [en] (X11; U; SunOS 5.8 sun4u) [Netscape]">

     6 </head>

     7 <body text="#000000" bgcolor="#FFFFFF" link="#0000EE" vlink="#551A8B" alink="#FF0000">

     9 <h3>

    10 <b><font face="Times New Roman,Times">Author Sonja Mirtitsch</font></b></h3>

    12 <h3>

    13 <b><font face="Times New Roman,Times">Last updated: 4/4/2001</font></b></h3>

    15 <h1>

    16 <b><font face="Times New Roman,Times">NSS 3.2.QA Wrapper</font></b></h1>

    18 <p><br>The QA&nbsp; wrapper tests the nightly builds of NSS. The actual

    19 tests are being run are called from the QA script all.sh. I will add documentation

    20 for the actual QA soon. The main purpose of the wrapper is: find out which

    21 build (NSS version, date, Build Platform) to test on which machine (OS,

    22 OS version) and construct a summary report, which is then mailed to the

    23 nss developers (aka mailing list nss-qa-report@netscape.com). Please see

    24 also the <a href="#advertisement">feature</a> section.

    25 <p><a href="#nssqa">nssqa</a>&nbsp; - the script that calls the actual

    26 qa script all.sh

    27 <br><a href="#qa_stat">qa_stat</a> - sends out status reports

    28 <br><a href="#qaclean">qaclean</a>&nbsp; - if everything else fails

    29 <p>Sample <a href="/u/sonmi/doc/publish/glob_result.html">global result</a>,

    30 <a href="/u/sonmi/doc/publish/results.html">individual result </a>and <a href="/u/sonmi/doc/publish/output.log">log

    31 files</a>

    32 <p>The QA wrapper consistst mainly of scripts, most located in security/nss/tests

    33 and subdirectories, but run from /u/sonmi/bin

    34 <p>nssqa and qa_stat, the main scripts both include a common header (<a href="../header">header</a>)

    35 and a common environment (<a href="../set_environment">set_environment</a>).

    36 <br>Also used is <a href="../mksymlinks">mksymlinks</a> and <a href="../path_uniq">path_uniq</a>

    37 and <a href="#qaclean">qaclean</a>.

    38 <p>The scripts that are used on a daily basis are located in /u/sonmi/bin

    39 and checked into security/nss/tests

    40 <p>Parameters and Options are the same for most scripts.

    41 <p><a NAME="Parameters"></a><b><u><font size=+1>Parameters</font></u></b>

    42 <br>&nbsp;&nbsp;&nbsp; nssversion (supported: 30b, 31, tip, default tip)

    43 <br>&nbsp;&nbsp;&nbsp; builddate (default - today, format mmdd)

    44 <p><a NAME="Options"></a><b><u><font size=+1>Options</font></u></b>

    45 <br>&nbsp;&nbsp;&nbsp; -y answer all questions with y - use at your own

    46 risk... ignores warnings

    47 <br>&nbsp;&nbsp;&nbsp; -s silent (only usefull with -y)

    48 <br>&nbsp;&nbsp;&nbsp; -h, -? -help you guessed right - displays the usage

    49 <br>&nbsp;&nbsp;&nbsp; -d debug

    50 <br>&nbsp;&nbsp;&nbsp; -f &lt;filename> - write the (error)output to filename

    51 <br>&nbsp;&nbsp;&nbsp; -fcron writes resultfile in the same location as

    52 would the -cron

    53 <br>&nbsp;&nbsp;&nbsp; -m &lt;mailinglist> - send filename to mailinglist

    54 (csl) only useful

    55 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; with -f on nssqa

    56 <br>&nbsp;&nbsp;&nbsp; -l &lt;mozroot> run on a local build - does not

    57 work at this time

    58 <br>&nbsp;&nbsp;&nbsp; -cron equivalient to -y -s -d -f $RESULTDIR/$HOST.&lt;scriptname>

    59 <br>&nbsp;

    60 <p>Please be aware that some iPlanet specific environments and features

    61 are being used.

    62 <p>-d Debug option might be removed from cron in a few weeks - or maybe

    63 not

    64 <br>-l QA on local build is not fully implemented yet - will not be implemented,

    65 all.sh can be called directly instead

    66 <p>Please do not use on Windows 95 and 98, ME platforms yet.

    67 <p>use -d if script behaves strange or exits unexpectedly

    68 <p><b><font size=+1>How to use the QA-wrapper</font></b>

    69 <br>To test a build, first run nssqa on the required QA platforms (some

    70 buildplatforms require QA to be run on additional platforms - for example

    71 Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has been run on

    72 multiple or all required platforms it makes sense to run qa_stat on the

    73 output of nssqa as well.

    74 <br>Before used on a new system (even if the same platform has been tested

    75 before) please use completely interactive, to see what the variables are

    76 being initialized to, and read the warnings. Same is true if being run

    77 from a different user account than svbld.

    78 <p>In any case, if you are using it, please let me know the results.

    79 <p><a NAME="nssqa"></a><b><u><font size=+1>nssqa:</font></u></b>

    80 <p>the script that calls the actual qa script all.sh

    81 <p>nssqa <a href="#Parameters">parameters</a> and&nbsp; <a href="#Options">options</a>

    82 <p><a href="../nssqa">view the script</a>

    83 <p><b><u><font size=+1>Pseudocode Description of nssqa</font></u></b>

    84 <br>not quite up to date

    85 <p>&nbsp;&nbsp;&nbsp; header:init (global)

    86 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set flags and variables

    87 to default values

    88 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; signal trap (for interupts

    89 and kills)

    90 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set HOST and DOMSUF variables

    91 if running from cron

    92 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parse parameters and options

    93 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; determine os and set up

    94 the environment (espec. PATH)

    95 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set the directories to run

    96 in (influenced by parameters and -l option)<br>

    97 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set the directories for backward

    98 compatibility testing

    99 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set and initialize the tmp

   100 / debugging / output files

   101 <p>&nbsp;&nbsp;&nbsp; nssqa:init (local)

   102 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; locking: if nssqa is already

   103 running on this systems (yes-exit,

   104 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   105 no-lockfile)

   106 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set HOST and DOMSUF variables

   107 if running interavtively

   108 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set flag to kill remaining

   109 selfserv processes during cleanup

   110 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if QA platform different

   111 from build platform create neccessary

   112 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   113 symbolic links

   114 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; wait for the build to finish

   115 (max of 5h)

   116 <p>&nbsp;&nbsp;&nbsp; main:

   117 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; repeated per test (optimized,

   118 debug, 32, 64 bit)

   119 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   120 set flags for this run of all.sh (optimized, debug, 32, 64 bit)

   121 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   122 set the DIST directory (where the binaries reside)

   123 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   124 kill running selfservers (sorry - just don't use the svbld

   125 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   126 account if you need to do your own testing... I will fix

   127 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   128 selfserv as soon as I can - but it hangs too often and

   129 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   130 disturbs all following QA)

   131 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   132 run all.sh

   133 <p>&nbsp;&nbsp;&nbsp; header:exit (global)

   134 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; remove temporary files

   135 <p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; kill remaining selfservers

   136 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; send email to the list

   137 <br>&nbsp;

   138 <p>&nbsp;&nbsp;&nbsp; errorhandling

   139 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Option / Parameter errors:

   140 Exit with usage information

   141 <p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Severe errors: Exit wit errormessage

   142 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   143 example: directory in which all.sh resides does not exist

   144 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   145 can't create files or directories

   146 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   147 build not done after 5 hours

   148 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   149 is already running

   150 <p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Other errors: User is prompted

   151 with the "errormessage - continue (y/n)?"

   152 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   153 example: local DIST dir does not exist (continues with next all.sh)

   154 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

   155 outputdirectory does not exist (user can specify other)

   156 <p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Signals 2, 3, 15 are treated

   157 as severe errors

   158 <br>&nbsp;

   159 <br>&nbsp;

   160 <br>&nbsp;

   161 <p><img SRC="clean.gif" height=129 width=92 align=LEFT><a NAME="qaclean"></a><b><u><font size=+2>qaclean:</font></u></b>/u/sonmi/bin/qaclean

   162 <br>&nbsp;

   163 <p>Use qaclean as user "svbld" to get the propper permissions. It is supposed

   164 to clean up after a "hanging" QA and will also brutally kill, interupt

   165 and disturb any other nss related test or performance meassurement on the

   166 named machine. NT and 2000 might require an additional reboot, since the

   167 ps is not so good about telling us the actual programmname - so we can't

   168 kill them... Please note that this is a brute force script, it should not

   169 be used on a regular basis, file a bug whenever you have to use it, since

   170 hanging QA is nothing that should occur frequently

   171 <p>&nbsp;<a href="../qaclean">view the script</a>

   172 <p>What it does:

   173 <ol>

   174 <li>

   175 see if there is a lockfile (/tmp/nssqa.$$ or $TMP/nssqa.$$)</li>

   177 <br>if yes:

   178 <ol>kill the process of the lockfile <font color="#666666">(future expansion

   179 and if possible it's children )</font>

   180 <br>rm the lockfile</ol>

   182 <li>

   183 kill selfservers</li>

   185 <li>

   186 kill whatever other qa related processes might be hanging</li>

   188 <li>

   189 clean up tmp files</li>

   190 </ol>

   191 <b>QAClean Parameters:</b>

   192 <br>&nbsp;&nbsp;&nbsp; machinename.

   193 <br>&nbsp;&nbsp;&nbsp; for example

   194 <br>&nbsp;&nbsp;&nbsp; qaclean kentuckyderby

   195 <br>&nbsp;&nbsp;&nbsp; started on any machine, will clean up on kentuckyderby

   196 <p><a NAME="qa_stat"></a><b><u><font size=+2>qa_stat</font></u></b>

   197 <p>qa_stat is the script that is being started from the svbld cron on kentuckyderby

   198 every morning at 10:00 and runs some (very primitive) analysis on the qa

   199 results.

   200 <br>I'd like to rewrite the whole thing in perl, and in a few weeks I might

   201 just do this...

   202 <p>&nbsp;<a href="../qa_stat">view the script</a>

   203 <p>qa_stat <a href="#Parameters">parameters</a> and&nbsp; <a href="#Options">options</a>

   204 <p><a NAME="advertisement"></a><b><u><font size=+1>Why we need the QA wrapper</font></u></b>

   205 <p>We need the new QA wrapper, because we have to test on so many platforms,

   206 that running the tests and evaluating the results for the nightly builds

   207 took about an average workday.

   208 <p><b><font size=+1>New Features:</font></b>

   209 <ul>

   210 <li>

   211 runs from <b>cron</b> / rsh or <b>interactive</b> if desired</li>

   213 <li>

   214 generates <b>summary</b> (no need to look through 60-90 directories)</li>

   216 <li>

   217 sends <b>email</b> about results</li>

   219 <li>

   220 automatically <b>recognizes common errors</b> and problems and conflicts

   221 and corrects them</li>

   223 <br>(or attempts to correct them :-)

   224 <li>

   225 automatically determines <b>which build </b>to test (waits if build in

   226 progress, exits if no build)</li>

   228 <li>

   229 runs on <b>all required platforms</b> (Windows 98 and before not functional

   230 yet)</li>

   232 <li>

   233 Windows version runs on <b>free Cygnus</b> as well as on MKS</li>

   235 <li>

   236 debug mode, normal mode and silent mode</li>

   238 <li>

   239 <b>locking</b> mechanism so it won't run twice</li>

   241 <li>

   242 <b>cleanup</b> after being killed and most errors (no remaining selfservers,

   243 tmpfiles, lock files)</li>

   244 </ul>

   245 The 1st script is started via cron between 5:00 and 8:00 am on different

   246 systems, and starts QA on the nightly build. At 10:00 the next script is

   247 started, and sends a QA summary to the nss developers.

   248 <p><b>Cygnus Advantages</b>:

   249 <ul>

   250 <li>

   251 <b>free</b></li>

   253 <li>

   254 better handling of <b>processes</b> (background, processIDs, Signals)</li>

   256 <li>

   257 Unix / Linux <b>compatible</b> sh / bash</li>

   258 </ul>

   259 <b>Disadvantages</b>

   260 <ul>

   261 <li>

   262 MKS functionality needs to be preserved (makes <b>8 Windows platforms</b>

   263 instead of 4 for the QA suites - makes 32 testruns on Windows alone)</li>

   265 <br>In certain functionality's <b>slow</b>

   266 <br><b></b>&nbsp;</ul>

   267 <b>Porting the windows QA&nbsp;to Uwin as well is also being considered</b>

   268 </body>

   269 </html>

The Tor Browser / file revision

security/nss/tests/doc/qa_wrapper.html@6474c204b198

security/nss/tests/doc/qa_wrapper.html