1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
1.2 +++ b/security/nss/tests/doc/qa_wrapper.html Wed Dec 31 06:09:35 2014 +0100
1.3 @@ -0,0 +1,269 @@
1.4 +<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
1.5 +<html>
1.6 +<head>
1.7 + <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
1.8 + <meta name="GENERATOR" content="Mozilla/4.7 [en] (X11; U; SunOS 5.8 sun4u) [Netscape]">
1.9 +</head>
1.10 +<body text="#000000" bgcolor="#FFFFFF" link="#0000EE" vlink="#551A8B" alink="#FF0000">
1.11 +
1.12 +<h3>
1.13 +<b><font face="Times New Roman,Times">Author Sonja Mirtitsch</font></b></h3>
1.14 +
1.15 +<h3>
1.16 +<b><font face="Times New Roman,Times">Last updated: 4/4/2001</font></b></h3>
1.17 +
1.18 +<h1>
1.19 +<b><font face="Times New Roman,Times">NSS 3.2.QA Wrapper</font></b></h1>
1.20 +
1.21 +<p><br>The QA wrapper tests the nightly builds of NSS. The actual
1.22 +tests are being run are called from the QA script all.sh. I will add documentation
1.23 +for the actual QA soon. The main purpose of the wrapper is: find out which
1.24 +build (NSS version, date, Build Platform) to test on which machine (OS,
1.25 +OS version) and construct a summary report, which is then mailed to the
1.26 +nss developers (aka mailing list nss-qa-report@netscape.com). Please see
1.27 +also the <a href="#advertisement">feature</a> section.
1.28 +<p><a href="#nssqa">nssqa</a> - the script that calls the actual
1.29 +qa script all.sh
1.30 +<br><a href="#qa_stat">qa_stat</a> - sends out status reports
1.31 +<br><a href="#qaclean">qaclean</a> - if everything else fails
1.32 +<p>Sample <a href="/u/sonmi/doc/publish/glob_result.html">global result</a>,
1.33 +<a href="/u/sonmi/doc/publish/results.html">individual result </a>and <a href="/u/sonmi/doc/publish/output.log">log
1.34 +files</a>
1.35 +<p>The QA wrapper consistst mainly of scripts, most located in security/nss/tests
1.36 +and subdirectories, but run from /u/sonmi/bin
1.37 +<p>nssqa and qa_stat, the main scripts both include a common header (<a href="../header">header</a>)
1.38 +and a common environment (<a href="../set_environment">set_environment</a>).
1.39 +<br>Also used is <a href="../mksymlinks">mksymlinks</a> and <a href="../path_uniq">path_uniq</a>
1.40 +and <a href="#qaclean">qaclean</a>.
1.41 +<p>The scripts that are used on a daily basis are located in /u/sonmi/bin
1.42 +and checked into security/nss/tests
1.43 +<p>Parameters and Options are the same for most scripts.
1.44 +<p><a NAME="Parameters"></a><b><u><font size=+1>Parameters</font></u></b>
1.45 +<br> nssversion (supported: 30b, 31, tip, default tip)
1.46 +<br> builddate (default - today, format mmdd)
1.47 +<p><a NAME="Options"></a><b><u><font size=+1>Options</font></u></b>
1.48 +<br> -y answer all questions with y - use at your own
1.49 +risk... ignores warnings
1.50 +<br> -s silent (only usefull with -y)
1.51 +<br> -h, -? -help you guessed right - displays the usage
1.52 +<br> -d debug
1.53 +<br> -f <filename> - write the (error)output to filename
1.54 +<br> -fcron writes resultfile in the same location as
1.55 +would the -cron
1.56 +<br> -m <mailinglist> - send filename to mailinglist
1.57 +(csl) only useful
1.58 +<br> with -f on nssqa
1.59 +<br> -l <mozroot> run on a local build - does not
1.60 +work at this time
1.61 +<br> -cron equivalient to -y -s -d -f $RESULTDIR/$HOST.<scriptname>
1.62 +<br>
1.63 +<p>Please be aware that some iPlanet specific environments and features
1.64 +are being used.
1.65 +<p>-d Debug option might be removed from cron in a few weeks - or maybe
1.66 +not
1.67 +<br>-l QA on local build is not fully implemented yet - will not be implemented,
1.68 +all.sh can be called directly instead
1.69 +<p>Please do not use on Windows 95 and 98, ME platforms yet.
1.70 +<p>use -d if script behaves strange or exits unexpectedly
1.71 +<p><b><font size=+1>How to use the QA-wrapper</font></b>
1.72 +<br>To test a build, first run nssqa on the required QA platforms (some
1.73 +buildplatforms require QA to be run on additional platforms - for example
1.74 +Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has been run on
1.75 +multiple or all required platforms it makes sense to run qa_stat on the
1.76 +output of nssqa as well.
1.77 +<br>Before used on a new system (even if the same platform has been tested
1.78 +before) please use completely interactive, to see what the variables are
1.79 +being initialized to, and read the warnings. Same is true if being run
1.80 +from a different user account than svbld.
1.81 +<p>In any case, if you are using it, please let me know the results.
1.82 +<p><a NAME="nssqa"></a><b><u><font size=+1>nssqa:</font></u></b>
1.83 +<p>the script that calls the actual qa script all.sh
1.84 +<p>nssqa <a href="#Parameters">parameters</a> and <a href="#Options">options</a>
1.85 +<p><a href="../nssqa">view the script</a>
1.86 +<p><b><u><font size=+1>Pseudocode Description of nssqa</font></u></b>
1.87 +<br>not quite up to date
1.88 +<p> header:init (global)
1.89 +<br> set flags and variables
1.90 +to default values
1.91 +<br> signal trap (for interupts
1.92 +and kills)
1.93 +<br> set HOST and DOMSUF variables
1.94 +if running from cron
1.95 +<br> parse parameters and options
1.96 +<br> determine os and set up
1.97 +the environment (espec. PATH)
1.98 +<br> set the directories to run
1.99 +in (influenced by parameters and -l option)<br>
1.100 + set the directories for backward
1.101 +compatibility testing
1.102 +<br> set and initialize the tmp
1.103 +/ debugging / output files
1.104 +<p> nssqa:init (local)
1.105 +<br> locking: if nssqa is already
1.106 +running on this systems (yes-exit,
1.107 +<br>
1.108 +no-lockfile)
1.109 +<br> set HOST and DOMSUF variables
1.110 +if running interavtively
1.111 +<br> set flag to kill remaining
1.112 +selfserv processes during cleanup
1.113 +<br> if QA platform different
1.114 +from build platform create neccessary
1.115 +<br>
1.116 +symbolic links
1.117 +<br> wait for the build to finish
1.118 +(max of 5h)
1.119 +<p> main:
1.120 +<br> repeated per test (optimized,
1.121 +debug, 32, 64 bit)
1.122 +<br>
1.123 +set flags for this run of all.sh (optimized, debug, 32, 64 bit)
1.124 +<br>
1.125 +set the DIST directory (where the binaries reside)
1.126 +<br>
1.127 +kill running selfservers (sorry - just don't use the svbld
1.128 +<br>
1.129 +account if you need to do your own testing... I will fix
1.130 +<br>
1.131 +selfserv as soon as I can - but it hangs too often and
1.132 +<br>
1.133 +disturbs all following QA)
1.134 +<br>
1.135 +run all.sh
1.136 +<p> header:exit (global)
1.137 +<br> remove temporary files
1.138 +<p> kill remaining selfservers
1.139 +<br> send email to the list
1.140 +<br>
1.141 +<p> errorhandling
1.142 +<br> Option / Parameter errors:
1.143 +Exit with usage information
1.144 +<p> Severe errors: Exit wit errormessage
1.145 +<br>
1.146 +example: directory in which all.sh resides does not exist
1.147 +<br>
1.148 +can't create files or directories
1.149 +<br>
1.150 +build not done after 5 hours
1.151 +<br>
1.152 +is already running
1.153 +<p> Other errors: User is prompted
1.154 +with the "errormessage - continue (y/n)?"
1.155 +<br>
1.156 +example: local DIST dir does not exist (continues with next all.sh)
1.157 +<br>
1.158 +outputdirectory does not exist (user can specify other)
1.159 +<p> Signals 2, 3, 15 are treated
1.160 +as severe errors
1.161 +<br>
1.162 +<br>
1.163 +<br>
1.164 +<p><img SRC="clean.gif" height=129 width=92 align=LEFT><a NAME="qaclean"></a><b><u><font size=+2>qaclean:</font></u></b>/u/sonmi/bin/qaclean
1.165 +<br>
1.166 +<p>Use qaclean as user "svbld" to get the propper permissions. It is supposed
1.167 +to clean up after a "hanging" QA and will also brutally kill, interupt
1.168 +and disturb any other nss related test or performance meassurement on the
1.169 +named machine. NT and 2000 might require an additional reboot, since the
1.170 +ps is not so good about telling us the actual programmname - so we can't
1.171 +kill them... Please note that this is a brute force script, it should not
1.172 +be used on a regular basis, file a bug whenever you have to use it, since
1.173 +hanging QA is nothing that should occur frequently
1.174 +<p> <a href="../qaclean">view the script</a>
1.175 +<p>What it does:
1.176 +<ol>
1.177 +<li>
1.178 +see if there is a lockfile (/tmp/nssqa.$$ or $TMP/nssqa.$$)</li>
1.179 +
1.180 +<br>if yes:
1.181 +<ol>kill the process of the lockfile <font color="#666666">(future expansion
1.182 +and if possible it's children )</font>
1.183 +<br>rm the lockfile</ol>
1.184 +
1.185 +<li>
1.186 +kill selfservers</li>
1.187 +
1.188 +<li>
1.189 +kill whatever other qa related processes might be hanging</li>
1.190 +
1.191 +<li>
1.192 +clean up tmp files</li>
1.193 +</ol>
1.194 +<b>QAClean Parameters:</b>
1.195 +<br> machinename.
1.196 +<br> for example
1.197 +<br> qaclean kentuckyderby
1.198 +<br> started on any machine, will clean up on kentuckyderby
1.199 +<p><a NAME="qa_stat"></a><b><u><font size=+2>qa_stat</font></u></b>
1.200 +<p>qa_stat is the script that is being started from the svbld cron on kentuckyderby
1.201 +every morning at 10:00 and runs some (very primitive) analysis on the qa
1.202 +results.
1.203 +<br>I'd like to rewrite the whole thing in perl, and in a few weeks I might
1.204 +just do this...
1.205 +<p> <a href="../qa_stat">view the script</a>
1.206 +<p>qa_stat <a href="#Parameters">parameters</a> and <a href="#Options">options</a>
1.207 +<p><a NAME="advertisement"></a><b><u><font size=+1>Why we need the QA wrapper</font></u></b>
1.208 +<p>We need the new QA wrapper, because we have to test on so many platforms,
1.209 +that running the tests and evaluating the results for the nightly builds
1.210 +took about an average workday.
1.211 +<p><b><font size=+1>New Features:</font></b>
1.212 +<ul>
1.213 +<li>
1.214 +runs from <b>cron</b> / rsh or <b>interactive</b> if desired</li>
1.215 +
1.216 +<li>
1.217 +generates <b>summary</b> (no need to look through 60-90 directories)</li>
1.218 +
1.219 +<li>
1.220 +sends <b>email</b> about results</li>
1.221 +
1.222 +<li>
1.223 +automatically <b>recognizes common errors</b> and problems and conflicts
1.224 +and corrects them</li>
1.225 +
1.226 +<br>(or attempts to correct them :-)
1.227 +<li>
1.228 +automatically determines <b>which build </b>to test (waits if build in
1.229 +progress, exits if no build)</li>
1.230 +
1.231 +<li>
1.232 +runs on <b>all required platforms</b> (Windows 98 and before not functional
1.233 +yet)</li>
1.234 +
1.235 +<li>
1.236 +Windows version runs on <b>free Cygnus</b> as well as on MKS</li>
1.237 +
1.238 +<li>
1.239 +debug mode, normal mode and silent mode</li>
1.240 +
1.241 +<li>
1.242 +<b>locking</b> mechanism so it won't run twice</li>
1.243 +
1.244 +<li>
1.245 +<b>cleanup</b> after being killed and most errors (no remaining selfservers,
1.246 +tmpfiles, lock files)</li>
1.247 +</ul>
1.248 +The 1st script is started via cron between 5:00 and 8:00 am on different
1.249 +systems, and starts QA on the nightly build. At 10:00 the next script is
1.250 +started, and sends a QA summary to the nss developers.
1.251 +<p><b>Cygnus Advantages</b>:
1.252 +<ul>
1.253 +<li>
1.254 +<b>free</b></li>
1.255 +
1.256 +<li>
1.257 +better handling of <b>processes</b> (background, processIDs, Signals)</li>
1.258 +
1.259 +<li>
1.260 +Unix / Linux <b>compatible</b> sh / bash</li>
1.261 +</ul>
1.262 +<b>Disadvantages</b>
1.263 +<ul>
1.264 +<li>
1.265 +MKS functionality needs to be preserved (makes <b>8 Windows platforms</b>
1.266 +instead of 4 for the QA suites - makes 32 testruns on Windows alone)</li>
1.267 +
1.268 +<br>In certain functionality's <b>slow</b>
1.269 +<br><b></b> </ul>
1.270 +<b>Porting the windows QA to Uwin as well is also being considered</b>
1.271 +</body>
1.272 +</html>
