This page serves as a gateway to the results of the production testing framework. It aims to provide the experiment with an overview of how a given NOvA software release performs in terms of a number of simple metrics.

This site indexes the contents of novagpvm10.fnal.gov:/nusoft/app/web/htdoc/nova/production/testing/ . Any folders that contain test results are then displayed. The plan is to eventually have tests be run (semi-)automatically. This page itself is generated using a cron job. In case of any issues contact Matthew Tamsett.

NOvA data processing can be divided into a number of logical chains based on the type of data being handled. The currently configured chains are:

• FD cosmics
• FD data NuMI
• FD data cosmics
• FD genie FHC nonswap
• FD genie FHC swap
• FD genie FHC tau
• FD genie RHC nonswap
• FD genie RHC swap
• FD genie RHC tau
• ND cosmics
• ND data NuMI
• ND data cosmics
• ND genie FHC nonswap
• ND genie RHC nonswap

Each of these chains is subdivided into a number of tiers. Each tier is a single NOvA art job configured using a single FHiCL file. The available chains and tiers are defined in TierConfigurations.py. Tier configurations are used so that the execution of multiple tiers can be processed inside a single loop. Tier-to-tier differences are small and can be configured based on a small number of options. The currently configured tiers are:

• FD cosmics: cry, pchits, reco, pid, recopid, caf, fullchain nolem
• FD data NuMI: raw2root, reco, pid, recopid, caf, fullchain nolem, recokeepup
• FD data cosmics: raw2root, pchits, reco, pid, recopid, caf, fullchain nolem
• FD genie FHC nonswap: genie, reco, pid, recopid, caf, fullchain nolem
• FD genie FHC swap: genie, reco, pid, recopid, caf, fullchain nolem
• FD genie FHC tau: genie, reco, pid, recopid, caf, fullchain nolem
• FD genie RHC nonswap: genie, reco, pid, recopid, caf, fullchain nolem
• FD genie RHC swap: genie, reco, pid, recopid, caf, fullchain nolem
• FD genie RHC tau: genie, reco, pid, recopid, caf, fullchain nolem
• ND cosmics: cry, pchits, reco, pid, recopid, caf, fullchain nolem
• ND data NuMI: raw2root, reco, pid, recopid, caf, fullchain nolem, recokeepup
• ND data cosmics: raw2root, pchits, reco, pid, recopid, fullchain nolem, caf
• ND genie FHC nonswap: rock, mix, reco, pid, recopid, caf, fullchain nolem
• ND genie RHC nonswap: rock, mix, reco, pid, recopid, caf, fullchain nolem

Tests are run on the requested chain and tiers using a python wrapper program to spawn NOvA ART jobs using the sub- process module. The performance of jobs is monitored during their run-time using the psutil module to periodically record metrics such as CPU and memory usage. Simultaneously the STDOUT and STDERR generated by the job is captured and interlaced with time stamps. After the completion of the job this log is then parsed to extract information about the ART process, such as when each event began processing and when calls were made to the database.

The resultant metrics and logs are stored in a pickle file by the python job. After the fact these metrics are visualised and formatted for web display using pickle, HTML parsing, Google charts and Bokeh.

---