security/nss/tests/doc/qa_wrapper.html@b8a032363ba2 (annotated)
security/nss/tests/doc/qa_wrapper.html
Thu, 22 Jan 2015 13:21:57 +0100
- author
- Michael Schloh von Bennewitz <michael@schloh.com>
- date
- Thu, 22 Jan 2015 13:21:57 +0100
- branch
- TOR_BUG_9701
- changeset 15
- b8a032363ba2
- permissions
- -rwxr-xr-x
Incorporate requested changes from Mozilla in review:
https://bugzilla.mozilla.org/show_bug.cgi?id=1123480#c6
| michael@0 | 1 | <!doctype html public "-//w3c//dtd html 4.0 transitional//en"> |
| michael@0 | 2 | <html> |
| michael@0 | 3 | <head> |
| michael@0 | 4 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> |
| michael@0 | 5 | <meta name="GENERATOR" content="Mozilla/4.7 [en] (X11; U; SunOS 5.8 sun4u) [Netscape]"> |
| michael@0 | 6 | </head> |
| michael@0 | 7 | <body text="#000000" bgcolor="#FFFFFF" link="#0000EE" vlink="#551A8B" alink="#FF0000"> |
| michael@0 | 8 | |
| michael@0 | 9 | <h3> |
| michael@0 | 10 | <b><font face="Times New Roman,Times">Author Sonja Mirtitsch</font></b></h3> |
| michael@0 | 11 | |
| michael@0 | 12 | <h3> |
| michael@0 | 13 | <b><font face="Times New Roman,Times">Last updated: 4/4/2001</font></b></h3> |
| michael@0 | 14 | |
| michael@0 | 15 | <h1> |
| michael@0 | 16 | <b><font face="Times New Roman,Times">NSS 3.2.QA Wrapper</font></b></h1> |
| michael@0 | 17 | |
| michael@0 | 18 | <p><br>The QA wrapper tests the nightly builds of NSS. The actual |
| michael@0 | 19 | tests are being run are called from the QA script all.sh. I will add documentation |
| michael@0 | 20 | for the actual QA soon. The main purpose of the wrapper is: find out which |
| michael@0 | 21 | build (NSS version, date, Build Platform) to test on which machine (OS, |
| michael@0 | 22 | OS version) and construct a summary report, which is then mailed to the |
| michael@0 | 23 | nss developers (aka mailing list nss-qa-report@netscape.com). Please see |
| michael@0 | 24 | also the <a href="#advertisement">feature</a> section. |
| michael@0 | 25 | <p><a href="#nssqa">nssqa</a> - the script that calls the actual |
| michael@0 | 26 | qa script all.sh |
| michael@0 | 27 | <br><a href="#qa_stat">qa_stat</a> - sends out status reports |
| michael@0 | 28 | <br><a href="#qaclean">qaclean</a> - if everything else fails |
| michael@0 | 29 | <p>Sample <a href="/u/sonmi/doc/publish/glob_result.html">global result</a>, |
| michael@0 | 30 | <a href="/u/sonmi/doc/publish/results.html">individual result </a>and <a href="/u/sonmi/doc/publish/output.log">log |
| michael@0 | 31 | files</a> |
| michael@0 | 32 | <p>The QA wrapper consistst mainly of scripts, most located in security/nss/tests |
| michael@0 | 33 | and subdirectories, but run from /u/sonmi/bin |
| michael@0 | 34 | <p>nssqa and qa_stat, the main scripts both include a common header (<a href="../header">header</a>) |
| michael@0 | 35 | and a common environment (<a href="../set_environment">set_environment</a>). |
| michael@0 | 36 | <br>Also used is <a href="../mksymlinks">mksymlinks</a> and <a href="../path_uniq">path_uniq</a> |
| michael@0 | 37 | and <a href="#qaclean">qaclean</a>. |
| michael@0 | 38 | <p>The scripts that are used on a daily basis are located in /u/sonmi/bin |
| michael@0 | 39 | and checked into security/nss/tests |
| michael@0 | 40 | <p>Parameters and Options are the same for most scripts. |
| michael@0 | 41 | <p><a NAME="Parameters"></a><b><u><font size=+1>Parameters</font></u></b> |
| michael@0 | 42 | <br> nssversion (supported: 30b, 31, tip, default tip) |
| michael@0 | 43 | <br> builddate (default - today, format mmdd) |
| michael@0 | 44 | <p><a NAME="Options"></a><b><u><font size=+1>Options</font></u></b> |
| michael@0 | 45 | <br> -y answer all questions with y - use at your own |
| michael@0 | 46 | risk... ignores warnings |
| michael@0 | 47 | <br> -s silent (only usefull with -y) |
| michael@0 | 48 | <br> -h, -? -help you guessed right - displays the usage |
| michael@0 | 49 | <br> -d debug |
| michael@0 | 50 | <br> -f <filename> - write the (error)output to filename |
| michael@0 | 51 | <br> -fcron writes resultfile in the same location as |
| michael@0 | 52 | would the -cron |
| michael@0 | 53 | <br> -m <mailinglist> - send filename to mailinglist |
| michael@0 | 54 | (csl) only useful |
| michael@0 | 55 | <br> with -f on nssqa |
| michael@0 | 56 | <br> -l <mozroot> run on a local build - does not |
| michael@0 | 57 | work at this time |
| michael@0 | 58 | <br> -cron equivalient to -y -s -d -f $RESULTDIR/$HOST.<scriptname> |
| michael@0 | 59 | <br> |
| michael@0 | 60 | <p>Please be aware that some iPlanet specific environments and features |
| michael@0 | 61 | are being used. |
| michael@0 | 62 | <p>-d Debug option might be removed from cron in a few weeks - or maybe |
| michael@0 | 63 | not |
| michael@0 | 64 | <br>-l QA on local build is not fully implemented yet - will not be implemented, |
| michael@0 | 65 | all.sh can be called directly instead |
| michael@0 | 66 | <p>Please do not use on Windows 95 and 98, ME platforms yet. |
| michael@0 | 67 | <p>use -d if script behaves strange or exits unexpectedly |
| michael@0 | 68 | <p><b><font size=+1>How to use the QA-wrapper</font></b> |
| michael@0 | 69 | <br>To test a build, first run nssqa on the required QA platforms (some |
| michael@0 | 70 | buildplatforms require QA to be run on additional platforms - for example |
| michael@0 | 71 | Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has been run on |
| michael@0 | 72 | multiple or all required platforms it makes sense to run qa_stat on the |
| michael@0 | 73 | output of nssqa as well. |
| michael@0 | 74 | <br>Before used on a new system (even if the same platform has been tested |
| michael@0 | 75 | before) please use completely interactive, to see what the variables are |
| michael@0 | 76 | being initialized to, and read the warnings. Same is true if being run |
| michael@0 | 77 | from a different user account than svbld. |
| michael@0 | 78 | <p>In any case, if you are using it, please let me know the results. |
| michael@0 | 79 | <p><a NAME="nssqa"></a><b><u><font size=+1>nssqa:</font></u></b> |
| michael@0 | 80 | <p>the script that calls the actual qa script all.sh |
| michael@0 | 81 | <p>nssqa <a href="#Parameters">parameters</a> and <a href="#Options">options</a> |
| michael@0 | 82 | <p><a href="../nssqa">view the script</a> |
| michael@0 | 83 | <p><b><u><font size=+1>Pseudocode Description of nssqa</font></u></b> |
| michael@0 | 84 | <br>not quite up to date |
| michael@0 | 85 | <p> header:init (global) |
| michael@0 | 86 | <br> set flags and variables |
| michael@0 | 87 | to default values |
| michael@0 | 88 | <br> signal trap (for interupts |
| michael@0 | 89 | and kills) |
| michael@0 | 90 | <br> set HOST and DOMSUF variables |
| michael@0 | 91 | if running from cron |
| michael@0 | 92 | <br> parse parameters and options |
| michael@0 | 93 | <br> determine os and set up |
| michael@0 | 94 | the environment (espec. PATH) |
| michael@0 | 95 | <br> set the directories to run |
| michael@0 | 96 | in (influenced by parameters and -l option)<br> |
| michael@0 | 97 | set the directories for backward |
| michael@0 | 98 | compatibility testing |
| michael@0 | 99 | <br> set and initialize the tmp |
| michael@0 | 100 | / debugging / output files |
| michael@0 | 101 | <p> nssqa:init (local) |
| michael@0 | 102 | <br> locking: if nssqa is already |
| michael@0 | 103 | running on this systems (yes-exit, |
| michael@0 | 104 | <br> |
| michael@0 | 105 | no-lockfile) |
| michael@0 | 106 | <br> set HOST and DOMSUF variables |
| michael@0 | 107 | if running interavtively |
| michael@0 | 108 | <br> set flag to kill remaining |
| michael@0 | 109 | selfserv processes during cleanup |
| michael@0 | 110 | <br> if QA platform different |
| michael@0 | 111 | from build platform create neccessary |
| michael@0 | 112 | <br> |
| michael@0 | 113 | symbolic links |
| michael@0 | 114 | <br> wait for the build to finish |
| michael@0 | 115 | (max of 5h) |
| michael@0 | 116 | <p> main: |
| michael@0 | 117 | <br> repeated per test (optimized, |
| michael@0 | 118 | debug, 32, 64 bit) |
| michael@0 | 119 | <br> |
| michael@0 | 120 | set flags for this run of all.sh (optimized, debug, 32, 64 bit) |
| michael@0 | 121 | <br> |
| michael@0 | 122 | set the DIST directory (where the binaries reside) |
| michael@0 | 123 | <br> |
| michael@0 | 124 | kill running selfservers (sorry - just don't use the svbld |
| michael@0 | 125 | <br> |
| michael@0 | 126 | account if you need to do your own testing... I will fix |
| michael@0 | 127 | <br> |
| michael@0 | 128 | selfserv as soon as I can - but it hangs too often and |
| michael@0 | 129 | <br> |
| michael@0 | 130 | disturbs all following QA) |
| michael@0 | 131 | <br> |
| michael@0 | 132 | run all.sh |
| michael@0 | 133 | <p> header:exit (global) |
| michael@0 | 134 | <br> remove temporary files |
| michael@0 | 135 | <p> kill remaining selfservers |
| michael@0 | 136 | <br> send email to the list |
| michael@0 | 137 | <br> |
| michael@0 | 138 | <p> errorhandling |
| michael@0 | 139 | <br> Option / Parameter errors: |
| michael@0 | 140 | Exit with usage information |
| michael@0 | 141 | <p> Severe errors: Exit wit errormessage |
| michael@0 | 142 | <br> |
| michael@0 | 143 | example: directory in which all.sh resides does not exist |
| michael@0 | 144 | <br> |
| michael@0 | 145 | can't create files or directories |
| michael@0 | 146 | <br> |
| michael@0 | 147 | build not done after 5 hours |
| michael@0 | 148 | <br> |
| michael@0 | 149 | is already running |
| michael@0 | 150 | <p> Other errors: User is prompted |
| michael@0 | 151 | with the "errormessage - continue (y/n)?" |
| michael@0 | 152 | <br> |
| michael@0 | 153 | example: local DIST dir does not exist (continues with next all.sh) |
| michael@0 | 154 | <br> |
| michael@0 | 155 | outputdirectory does not exist (user can specify other) |
| michael@0 | 156 | <p> Signals 2, 3, 15 are treated |
| michael@0 | 157 | as severe errors |
| michael@0 | 158 | <br> |
| michael@0 | 159 | <br> |
| michael@0 | 160 | <br> |
| michael@0 | 161 | <p><img SRC="clean.gif" height=129 width=92 align=LEFT><a NAME="qaclean"></a><b><u><font size=+2>qaclean:</font></u></b>/u/sonmi/bin/qaclean |
| michael@0 | 162 | <br> |
| michael@0 | 163 | <p>Use qaclean as user "svbld" to get the propper permissions. It is supposed |
| michael@0 | 164 | to clean up after a "hanging" QA and will also brutally kill, interupt |
| michael@0 | 165 | and disturb any other nss related test or performance meassurement on the |
| michael@0 | 166 | named machine. NT and 2000 might require an additional reboot, since the |
| michael@0 | 167 | ps is not so good about telling us the actual programmname - so we can't |
| michael@0 | 168 | kill them... Please note that this is a brute force script, it should not |
| michael@0 | 169 | be used on a regular basis, file a bug whenever you have to use it, since |
| michael@0 | 170 | hanging QA is nothing that should occur frequently |
| michael@0 | 171 | <p> <a href="../qaclean">view the script</a> |
| michael@0 | 172 | <p>What it does: |
| michael@0 | 173 | <ol> |
| michael@0 | 174 | <li> |
| michael@0 | 175 | see if there is a lockfile (/tmp/nssqa.$$ or $TMP/nssqa.$$)</li> |
| michael@0 | 176 | |
| michael@0 | 177 | <br>if yes: |
| michael@0 | 178 | <ol>kill the process of the lockfile <font color="#666666">(future expansion |
| michael@0 | 179 | and if possible it's children )</font> |
| michael@0 | 180 | <br>rm the lockfile</ol> |
| michael@0 | 181 | |
| michael@0 | 182 | <li> |
| michael@0 | 183 | kill selfservers</li> |
| michael@0 | 184 | |
| michael@0 | 185 | <li> |
| michael@0 | 186 | kill whatever other qa related processes might be hanging</li> |
| michael@0 | 187 | |
| michael@0 | 188 | <li> |
| michael@0 | 189 | clean up tmp files</li> |
| michael@0 | 190 | </ol> |
| michael@0 | 191 | <b>QAClean Parameters:</b> |
| michael@0 | 192 | <br> machinename. |
| michael@0 | 193 | <br> for example |
| michael@0 | 194 | <br> qaclean kentuckyderby |
| michael@0 | 195 | <br> started on any machine, will clean up on kentuckyderby |
| michael@0 | 196 | <p><a NAME="qa_stat"></a><b><u><font size=+2>qa_stat</font></u></b> |
| michael@0 | 197 | <p>qa_stat is the script that is being started from the svbld cron on kentuckyderby |
| michael@0 | 198 | every morning at 10:00 and runs some (very primitive) analysis on the qa |
| michael@0 | 199 | results. |
| michael@0 | 200 | <br>I'd like to rewrite the whole thing in perl, and in a few weeks I might |
| michael@0 | 201 | just do this... |
| michael@0 | 202 | <p> <a href="../qa_stat">view the script</a> |
| michael@0 | 203 | <p>qa_stat <a href="#Parameters">parameters</a> and <a href="#Options">options</a> |
| michael@0 | 204 | <p><a NAME="advertisement"></a><b><u><font size=+1>Why we need the QA wrapper</font></u></b> |
| michael@0 | 205 | <p>We need the new QA wrapper, because we have to test on so many platforms, |
| michael@0 | 206 | that running the tests and evaluating the results for the nightly builds |
| michael@0 | 207 | took about an average workday. |
| michael@0 | 208 | <p><b><font size=+1>New Features:</font></b> |
| michael@0 | 209 | <ul> |
| michael@0 | 210 | <li> |
| michael@0 | 211 | runs from <b>cron</b> / rsh or <b>interactive</b> if desired</li> |
| michael@0 | 212 | |
| michael@0 | 213 | <li> |
| michael@0 | 214 | generates <b>summary</b> (no need to look through 60-90 directories)</li> |
| michael@0 | 215 | |
| michael@0 | 216 | <li> |
| michael@0 | 217 | sends <b>email</b> about results</li> |
| michael@0 | 218 | |
| michael@0 | 219 | <li> |
| michael@0 | 220 | automatically <b>recognizes common errors</b> and problems and conflicts |
| michael@0 | 221 | and corrects them</li> |
| michael@0 | 222 | |
| michael@0 | 223 | <br>(or attempts to correct them :-) |
| michael@0 | 224 | <li> |
| michael@0 | 225 | automatically determines <b>which build </b>to test (waits if build in |
| michael@0 | 226 | progress, exits if no build)</li> |
| michael@0 | 227 | |
| michael@0 | 228 | <li> |
| michael@0 | 229 | runs on <b>all required platforms</b> (Windows 98 and before not functional |
| michael@0 | 230 | yet)</li> |
| michael@0 | 231 | |
| michael@0 | 232 | <li> |
| michael@0 | 233 | Windows version runs on <b>free Cygnus</b> as well as on MKS</li> |
| michael@0 | 234 | |
| michael@0 | 235 | <li> |
| michael@0 | 236 | debug mode, normal mode and silent mode</li> |
| michael@0 | 237 | |
| michael@0 | 238 | <li> |
| michael@0 | 239 | <b>locking</b> mechanism so it won't run twice</li> |
| michael@0 | 240 | |
| michael@0 | 241 | <li> |
| michael@0 | 242 | <b>cleanup</b> after being killed and most errors (no remaining selfservers, |
| michael@0 | 243 | tmpfiles, lock files)</li> |
| michael@0 | 244 | </ul> |
| michael@0 | 245 | The 1st script is started via cron between 5:00 and 8:00 am on different |
| michael@0 | 246 | systems, and starts QA on the nightly build. At 10:00 the next script is |
| michael@0 | 247 | started, and sends a QA summary to the nss developers. |
| michael@0 | 248 | <p><b>Cygnus Advantages</b>: |
| michael@0 | 249 | <ul> |
| michael@0 | 250 | <li> |
| michael@0 | 251 | <b>free</b></li> |
| michael@0 | 252 | |
| michael@0 | 253 | <li> |
| michael@0 | 254 | better handling of <b>processes</b> (background, processIDs, Signals)</li> |
| michael@0 | 255 | |
| michael@0 | 256 | <li> |
| michael@0 | 257 | Unix / Linux <b>compatible</b> sh / bash</li> |
| michael@0 | 258 | </ul> |
| michael@0 | 259 | <b>Disadvantages</b> |
| michael@0 | 260 | <ul> |
| michael@0 | 261 | <li> |
| michael@0 | 262 | MKS functionality needs to be preserved (makes <b>8 Windows platforms</b> |
| michael@0 | 263 | instead of 4 for the QA suites - makes 32 testruns on Windows alone)</li> |
| michael@0 | 264 | |
| michael@0 | 265 | <br>In certain functionality's <b>slow</b> |
| michael@0 | 266 | <br><b></b> </ul> |
| michael@0 | 267 | <b>Porting the windows QA to Uwin as well is also being considered</b> |
| michael@0 | 268 | </body> |
| michael@0 | 269 | </html> |
