cronlog make testing page

--- running make testing page
--- Start date: 
Wed Jul 24 10:35:01 CDT 2019
--- Check for locks
File /nova/app/users/novapro/prod_valid/python/locks/testing_page.lock does not exist.
--- setup
user: novapro

--- Setup NOvA production validation
User:           novapro
Use with SRT:   false

Base directory: /nova/app/users/novapro/prod_valid/python
Grid output:    /nova/ana/users/novapro/prod_valid/condor_output
Web output:     /nusoft/app/web/htdoc/nova/production

--- novapro
--- Sync folders
run  : Copying new pkl files
run  : Input folders
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-prod2reco.b_2016-03-21-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-06-29_2016-06-29-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-prod2reco.d_2016-04-05-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-12-06_2016-12-06-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-03-04_2016-03-04-19-46-10
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-prod2genie.b_2016-03-04-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-05-20_2016-05-20-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-06-28_2016-06-28-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-04-21_2016-04-26-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-09-12_2016-09-12-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-prod2genie.a_2016-02-11-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-geniepreview.a_2016-09-20-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-03-04_2016-03-14-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-03-10_2016-03-10-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-mixedcalib.a_2016-06-20-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-geantpreview.a_2016-09-21-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-07-06_2016-07-06-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-prod2reco.a_2016-03-03-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-04-08_2016-04-08-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-09-13_2016-09-13-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.d_2016-12-20-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-prod2reco.e_2016-05-03-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-07-06_2016-07-06-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-07-26_2016-07-26-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-geantpreview.b_2016-10-03-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-09-12_2016-09-12-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-06-21_2016-06-21-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-06-21_2016-06-21-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-06-01_2016-06-01-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-prod2genie.c_2016-05-24-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-03-25_2016-03-25-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-prod2reco.f_2016-05-20-19-46-10
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-prod2reco.h_2016-06-17-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-fiberbrightness.a_2016-06-14-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-geant4101p3.a_2016-05-06-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-prod2reco.g_2016-06-06-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/feature_caf_size_2016-10-18-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-05-10_2016-05-10-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-11-02_2016-11-02-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-prod2genie.d_2016-11-01-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-03-10_2016-03-14-19-46-20
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-12-feature_caf_size.a_2016-11-16-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-geantpreview.a_2016-09-21-19-45-07
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.i_2017-05-02-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-07-15-secondana.a_2016-07-28-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-05-17_2017-05-17-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.a_2016-11-22-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-08-04_2016-08-05-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-10-26_2016-10-27-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/CR16-08-23_2016-08-24-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-08-24_2016-08-24-19-45-09
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-08-24_2016-08-24-19-45-11
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-01-11_2017-01-11-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-12-07_2016-12-07-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-12-06_2016-12-06-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-07-18_2017-07-18-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-12-20-prod3recopreview.c_2017-01-19-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/CR16-10-07_2016-10-07-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/CR16-10-07_2016-10-07-19-45-06
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-10-07_2016-10-07-19-45-10
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.e_2017-02-04-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.a_2017-03-09-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-12-20-prod3recopreview.c_2017-01-19-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.d_2016-12-20-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-geniepreview.b_2016-10-12-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-geniepreview.b_2016-10-12-19-45-06
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-newgeo.a_2016-10-14-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.m_2017-08-31-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-08-22_2017-08-22-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.d_2017-03-31-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-05-17_2017-05-17-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-10-21_2016-10-21-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-02-prod3calibpreview.a_2016-11-04-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-01-11_2017-01-11-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-11-12_2016-11-12-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/feature_caf_size.a_2016-11-13-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-12-feature_caf_size.a_2016-11-16-19-45-08
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.a_2016-11-22-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.c_2017-03-22-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-06-19_2017-06-19-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.b_2016-11-30-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.b_2016-11-30-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.l_2017-07-13-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.e_2017-08-02-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.d_2017-03-31-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.i_2017-11-03-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-07-27_2017-07-27-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.c_2016-12-15-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-12-20-prod3recopreview.a_2016-12-20-19-45-09
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-04-03_2017-04-03-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-12-20-prod3recopreview.d_2017-01-22-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.a_2017-11-14-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-01-26_2017-01-27-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-01-26_2017-01-27-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-12-feature_caf_size.b_2017-01-28-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.e_2017-02-04-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.e_2017-09-18-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.f_2017-02-10-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-11-22-prod3geniepreview.f_2017-02-10-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-02-21_2017-02-21-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-10-10_2017-10-11-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-02-22-legacy-cafs-v1_2017-02-23-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-02-24-nutools_upgrade.b_2017-02-28-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-02-24-nutools_upgrade.b_2017-02-28-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.b_2017-03-02-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.a_2017-03-02-19-45-09
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.a_2017-03-02-19-45-10
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-11-18_2017-11-18-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.c_2017-03-07-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-04-07_2018-04-07-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.g_2017-10-23-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.b_2017-03-13-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.b_2017-03-13-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-06-19_2017-06-19-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-03-16_2017-03-16-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.c_2017-03-22-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.e_2017-04-05-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.e_2018-01-16-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.d_2017-04-10-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.f_2017-04-12-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.g_2017-04-13-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.g_2017-04-13-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.b_2017-11-27-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-04-19_2017-04-19-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.h_2017-04-21-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.i_2017-05-02-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.j_2017-05-03-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.j_2017-05-03-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-05-03_2017-05-03-19-45-42
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.k_2017-05-09-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-02-19_2018-02-19-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-11-22_2017-11-22-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.CVNprong-respin.a_2018-07-12-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.q_2018-01-26-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.neutron-respin.a_2018-04-20-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-06-30_2017-06-30-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-10-12_2018-10-12-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-09-02_2018-09-06-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.l_2017-07-13-19-45-06
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-07-13_2017-07-13-19-45-41
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-07-13_2017-07-13-19-45-43
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/CR17-07-15_2017-07-15-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/CR17-07-15_2017-07-15-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-01-05_2018-01-05-19-45-09
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-08-11_2017-08-11-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.a_2017-09-05-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.a_2017-09-05-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-09-05_2017-09-05-19-45-42
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.b_2017-09-08-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.b_2017-09-08-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.k_2018-02-22-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.c_2017-09-11-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-09-12_2017-09-12-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-04-20-art2.11.b_2018-05-16-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.d_2017-09-13-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.d_2017-09-13-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.e_2017-09-18-19-45-06
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-10-08-prod5calib.a_2018-10-08-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-09-20_2017-09-20-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-09-21_2017-09-21-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-09-21_2017-09-21-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-12-18_2018-12-18-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-09-24_2017-09-24-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-09-24_2017-09-24-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-03-03-prod2reco.i_2017-09-27-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.d_2017-12-21-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-09-05-prod4recopreview.f_2017-09-29-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.q_2018-01-26-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.f_2017-10-02-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-10-09_2017-10-09-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-10-09_2017-10-09-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-01-26-CCPi02017.a_2017-10-11-19-45-58
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.h_2017-10-27-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.h_2017-10-27-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.n_2017-10-27-19-45-27
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.n_2017-10-27-19-46-18
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-22-2017ana.a_2018-02-01-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-06-11-2018ana.a_2018-06-11-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-10-30_2017-10-30-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-10-30_2017-10-30-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-08-22-prod3nus17.b_2018-02-04-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.i_2017-11-03-19-45-06
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.j_2017-11-10-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.o_2017-11-15-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.o_2017-11-15-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-11-16_2017-11-16-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-11-16_2017-11-16-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-11-18_2017-11-18-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-02-09_2018-02-09-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.b_2017-11-27-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-12-08_2017-12-08-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.l.srt-hotfix_2017-12-08-19-45-44
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.l.srt-hotfix_2017-12-08-19-45-45
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-11-05-feature_novarwgt.a_2018-11-05-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S16-12-07.ifdb-hotfix_2017-12-13-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.p_2017-12-14-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.p_2017-12-14-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-04-23_2018-04-23-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.c_2017-12-15-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-12-15_2017-12-15-19-45-32
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S17-12-15_2017-12-15-19-45-41
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.remid-hotfix.c_2018-12-13-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S19-01-14_2019-01-14-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.neutron-respin.b_2018-04-21-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-03-09_2018-03-09-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-08-22-prod3nus17.a_2018-01-04-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-08-22-prod3nus17.a_2018-01-04-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-01-05_2018-01-05-19-45-12
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.f_2018-01-18-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.f_2018-01-18-19-45-07
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-01-19_2018-01-19-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-01-19_2018-01-19-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.g_2018-01-26-19-45-39
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.g_2018-01-26-19-45-53
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-02-01_2018-02-01-19-45-34
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-02-01_2018-02-01-19-45-48
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.r_2018-02-02-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-08-22-prod3nus17.b_2018-02-04-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-05-14_2018-05-14-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-06-14_2018-06-15-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.j_2018-12-12-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.neutron-respin.b_2018-04-21-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-12-10_2018-12-10-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-08-22-prod3nus17.c_2018-02-07-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-09-prod3genie.k_2018-02-22-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.h.remid-hotfix_2018-02-22-19-45-41
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.h.remid-hotfix_2018-02-22-19-45-45
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.remid-hotfix.a_2018-02-25-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-02-25_2018-02-25-19-45-35
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-02-25-art1.a_2018-02-26-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-05-06_2018-05-07-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-09-29_2018-09-29-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-06-28_2018-06-28-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-03-09_2018-03-09-19-45-07
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-03-15-feature_nc_legacy.b_2018-03-16-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-03-15-feature_nc_legacy.b_2018-03-16-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.remid-hotfix.b_2018-03-16-19-45-43
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.remid-hotfix.b_2018-03-16-19-45-44
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.i_2018-03-21-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.i_2018-03-21-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-03-21_2018-03-21-19-45-41
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-08-13_2018-08-13-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-11-12_2018-11-12-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-19-05-10-01
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-06-28_2018-06-28-19-45-06
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-09-29_2018-09-29-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-04-07_2018-04-07-19-45-06
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.neutron-respin.c_2018-05-02-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4mre.a_2018-05-02-19-45-44
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4mre.a_2018-05-02-19-45-45
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-08-27_2018-08-27-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S19-01-02_2019-01-02-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-09-15_2018-09-16-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-05-29_2018-05-29-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-14-05-10-01
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-11-19-feature_stan.a_2018-11-20-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-20-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-10-05-10-01
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-23-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-09-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-07-17_2018-07-17-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-03-01-prod3reco.s_2018-07-21-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-07-27_2018-07-27-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.neutron-respin.d_2018-07-27-19-45-45
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R17-11-14-prod4reco.neutron-respin.d_2018-07-27-19-45-46
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-15-05-10-01
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-24-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-12-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-13-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-08-13_2018-08-13-19-45-05
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-16-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-17-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-18-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-21-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-22-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R18-11-05-feature_novarwgt.b_2019-01-04-19-45-04
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-10-29_2018-10-29-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-11-05-10-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-11-29_2018-11-29-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S18-11-29_2018-11-29-19-45-03
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/S19-01-16_2019-01-16-19-45-02
run  :     /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R19-02-06-nutools-upgrade.a_2019-02-11-19-45-04
run  : Output folders
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-24-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-14-05-10-01
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-09-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-23-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S19-01-14_2019-01-14-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-15-05-10-01
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-13-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-11-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-08-13_2018-08-13-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-10-05-10-01
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4mre.a_2018-05-02-19-45-45
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-12-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-11-12_2018-11-12-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-04-07_2018-04-07-19-45-06
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-10-09_2017-10-09-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.j_2017-11-10-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-22-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-21-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-09-21_2017-09-21-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-09-24_2017-09-24-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.neutron-respin.d_2018-07-27-19-45-46
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-02-25-art1.a_2018-02-26-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-20-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-12-10_2018-12-10-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-01-05_2018-01-05-19-45-12
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-11-18_2017-11-18-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-17-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.i_2017-05-02-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.f_2017-10-02-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-08-13_2018-08-13-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.l.srt-hotfix_2017-12-08-19-45-45
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.d_2017-12-21-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-06-19_2017-06-19-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.d_2017-03-31-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-16-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.h_2017-10-27-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-06-19_2017-06-19-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.remid-hotfix.c_2018-12-13-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4mre.a_2018-05-02-19-45-44
run  :     /nusoft/app/web/htdoc/nova/production/testing/index.html
run  :     /nusoft/app/web/htdoc/nova/production/testing/config.html
run  :     /nusoft/app/web/htdoc/nova/production/testing/results.html
run  :     /nusoft/app/web/htdoc/nova/production/testing/projections.html
run  :     /nusoft/app/web/htdoc/nova/production/testing/log_book.html
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-10-29_2018-10-29-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.i_2017-11-03-19-45-06
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-03-15-feature_nc_legacy.b_2018-03-16-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-09-20_2017-09-20-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-06-30_2017-06-30-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-01-05_2018-01-05-19-45-09
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-09-15_2018-09-16-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.a_2017-09-05-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-07-27_2018-07-27-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.d_2017-03-31-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.remid-hotfix.b_2018-03-16-19-45-44
run  :     /nusoft/app/web/htdoc/nova/production/testing/CR17-07-15_2017-07-15-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-geniepreview.b_2016-10-12-19-45-06
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-04-23_2018-04-23-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.f_2017-02-10-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.d_2016-12-20-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.neutron-respin.c_2018-05-02-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S19-01-02_2019-01-02-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-09-29_2018-09-29-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.f_2018-01-18-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-03-21_2018-03-21-19-45-41
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.p_2017-12-14-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-19-05-10-01
run  :     /nusoft/app/web/htdoc/nova/production/testing/development_2019-07-18-05-10-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-prod2genie.d_2016-11-01-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-11-18_2017-11-18-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-02-21_2017-02-21-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/CR16-10-07_2016-10-07-19-45-06
run  :     /nusoft/app/web/htdoc/nova/production/testing/CR16-10-07_2016-10-07-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.i_2017-05-02-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-12-18_2018-12-18-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-09-12_2017-09-12-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-08-11_2017-08-11-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-06-28_2018-06-28-19-45-06
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-03-09_2018-03-09-19-45-07
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-06-28_2018-06-28-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-prod2reco.h_2016-06-17-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-09-24_2017-09-24-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-12-08_2017-12-08-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.l.srt-hotfix_2017-12-08-19-45-44
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-08-22_2017-08-22-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-08-22-prod3nus17.b_2018-02-04-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/index.html~
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.e_2017-02-04-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.CVNprong-respin.a_2018-07-12-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-01-19_2018-01-19-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-01-19_2018-01-19-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-11-16_2017-11-16-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-11-19-feature_stan.a_2018-11-20-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.h_2017-04-21-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-10-09_2017-10-09-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.b_2017-09-08-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.o_2017-11-15-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.d_2017-09-13-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-12-20-prod3recopreview.a_2016-12-20-19-45-09
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.e_2017-09-18-19-45-06
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.b_2017-09-08-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-10-12_2018-10-12-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-05-03_2017-05-03-19-45-42
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.j_2017-05-03-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-02-01_2018-02-01-19-45-48
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.e_2017-04-05-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.m_2017-08-31-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/CR17-07-15_2017-07-15-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.i_2017-11-03-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-08-22-prod3nus17.a_2018-01-04-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-11-16_2017-11-16-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.b_2016-11-30-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S19-01-16_2019-01-16-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.i_2018-03-21-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.i_2018-03-21-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-02-01_2018-02-01-19-45-34
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.c_2017-12-15-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-09-29_2018-09-29-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-03-09_2018-03-09-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.j_2018-12-12-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-09-02_2018-09-06-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.g_2018-01-26-19-45-53
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-03-25_2016-03-25-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.j_2017-05-03-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-03-16_2017-03-16-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-11-29_2018-11-29-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-12-feature_caf_size.a_2016-11-16-19-45-08
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.k_2017-05-09-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.r_2018-02-02-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.o_2017-11-15-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.e_2017-09-18-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.g_2017-04-13-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-11-22_2017-11-22-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.f_2017-04-12-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-10-08-prod5calib.a_2018-10-08-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-02-24-nutools_upgrade.b_2017-02-28-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-12-15_2017-12-15-19-45-41
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-04-07_2018-04-07-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-01-26_2017-01-27-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-02-24-nutools_upgrade.b_2017-02-28-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-geantpreview.a_2016-09-21-19-45-07
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-10-30_2017-10-30-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-12-15_2017-12-15-19-45-32
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.a_2017-11-14-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.e_2017-02-04-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-01-26_2017-01-27-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.g_2017-04-13-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-06-01_2016-06-01-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-10-07_2016-10-07-19-45-10
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-08-22-prod3nus17.a_2018-01-04-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-07-13_2017-07-13-19-45-43
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-07-13_2017-07-13-19-45-41
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-12-20-prod3recopreview.c_2017-01-19-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-12-20-prod3recopreview.c_2017-01-19-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-12-20-prod3recopreview.d_2017-01-22-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-11-12_2016-11-12-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.g_2018-01-26-19-45-39
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.q_2018-01-26-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.remid-hotfix.b_2018-03-16-19-45-43
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-09-05_2017-09-05-19-45-42
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-01-11_2017-01-11-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-01-11_2017-01-11-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-08-04_2016-08-05-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-11-29_2018-11-29-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-22-2017ana.a_2018-02-01-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-geniepreview.b_2016-10-12-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-10-30_2017-10-30-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.a_2017-03-09-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-02-22-legacy-cafs-v1_2017-02-23-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-02-prod3calibpreview.a_2016-11-04-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.e_2017-08-02-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-09-21_2017-09-21-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.a_2017-09-05-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.q_2018-01-26-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-03-15-feature_nc_legacy.b_2018-03-16-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-03-10_2016-03-10-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.neutron-respin.d_2018-07-27-19-45-45
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.p_2017-12-14-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.n_2017-10-27-19-45-27
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-06-11-2018ana.a_2018-06-11-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-07-15-secondana.a_2016-07-28-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-geantpreview.a_2016-09-21-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-geniepreview.a_2016-09-20-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-12-07.ifdb-hotfix_2017-12-13-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.a_2017-03-02-19-45-10
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-prod2reco.i_2017-09-27-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-07-17_2018-07-17-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.n_2017-10-27-19-46-18
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.h_2017-10-27-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.c_2017-09-11-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.f_2017-09-29-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.f_2018-01-18-19-45-07
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-11-05-feature_novarwgt.b_2019-01-04-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-08-22-prod3nus17.b_2018-02-04-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/feature_caf_size_2016-10-18-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-12-feature_caf_size.a_2016-11-16-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-prod2genie.a_2016-02-11-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-03-04_2016-03-04-19-46-10
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-09-05-prod4recopreview.d_2017-09-13-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-02-19_2018-02-19-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.c_2016-12-15-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.c_2017-03-22-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.d_2017-04-10-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-07-18_2017-07-18-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-12-06_2016-12-06-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.b_2017-11-27-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-04-19_2017-04-19-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.e_2018-01-16-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.a_2017-03-02-19-45-09
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.b_2017-03-02-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.g_2017-10-23-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.b_2016-11-30-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.l_2017-07-13-19-45-06
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-fiberbrightness.a_2016-06-14-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-11-02_2016-11-02-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-05-10_2016-05-10-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.d_2016-12-20-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-07-27_2017-07-27-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.c_2017-03-22-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-newgeo.a_2016-10-14-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.b_2017-03-13-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.f_2017-02-10-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-04-03_2017-04-03-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-08-24_2016-08-24-19-45-11
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-08-27_2018-08-27-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.l_2017-07-13-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-05-29_2018-05-29-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.h.remid-hotfix_2018-02-22-19-45-45
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.h.remid-hotfix_2018-02-22-19-45-41
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.k_2018-02-22-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-08-22-prod3nus17.c_2018-02-07-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-06-14_2018-06-15-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-05-14_2018-05-14-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-02-25_2018-02-25-19-45-35
run  :     /nusoft/app/web/htdoc/nova/production/testing/R19-02-06-nutools-upgrade.a_2019-02-11-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-geantpreview.b_2016-10-03-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-04-08_2016-04-08-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-05-06_2018-05-07-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.neutron-respin.b_2018-04-21-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-09-12_2016-09-12-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.c_2017-03-07-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.b_2017-03-13-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.a_2016-11-22-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-10-21_2016-10-21-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-05-17_2017-05-17-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-12-07_2016-12-07-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-08-24_2016-08-24-19-45-09
run  :     /nusoft/app/web/htdoc/nova/production/testing/CR16-08-23_2016-08-24-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-03-10_2016-03-14-19-46-20
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-06-21_2016-06-21-19-45-05
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-06-21_2016-06-21-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-07-06_2016-07-06-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-10-26_2016-10-27-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/feature_caf_size.a_2016-11-13-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-geant4101p3.a_2016-05-06-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-prod2genie.c_2016-05-24-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-04-21_2016-04-26-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-12-06_2016-12-06-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-prod2reco.g_2016-06-06-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-06-28_2016-06-28-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-22-prod3geniepreview.a_2016-11-22-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-05-17_2017-05-17-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-prod2genie.b_2016-03-04-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-01-26-CCPi02017.a_2017-10-11-19-45-58
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-prod2reco.e_2016-05-03-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-07-26_2016-07-26-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-09-prod3genie.k_2018-02-22-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-prod2reco.d_2016-04-05-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-11-05-feature_novarwgt.a_2018-11-05-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-prod2reco.b_2016-03-21-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.remid-hotfix.a_2018-02-25-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-06-29_2016-06-29-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-prod2reco.a_2016-03-03-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-07-06_2016-07-06-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-mixedcalib.a_2016-06-20-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-03-03-prod2reco.f_2016-05-20-19-46-10
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-03-01-prod3reco.s_2018-07-21-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S17-10-10_2017-10-11-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.neutron-respin.a_2018-04-20-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.neutron-respin.b_2018-04-21-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-03-04_2016-03-14-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-09-12_2016-09-12-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-09-13_2016-09-13-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/S18-02-09_2018-02-09-19-45-04
run  :     /nusoft/app/web/htdoc/nova/production/testing/S16-05-20_2016-05-20-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R17-11-14-prod4reco.b_2017-11-27-19-45-02
run  :     /nusoft/app/web/htdoc/nova/production/testing/R16-11-12-feature_caf_size.b_2017-01-28-19-45-03
run  :     /nusoft/app/web/htdoc/nova/production/testing/R18-04-20-art2.11.b_2018-05-16-19-45-04
run  : Output base names:
run  :     development_2019-07-24-05-10-02
run  :     development_2019-07-14-05-10-01
run  :     development_2019-07-09-05-10-02
run  :     development_2019-07-23-05-10-02
run  :     S19-01-14_2019-01-14-19-45-05
run  :     development_2019-07-15-05-10-01
run  :     development_2019-07-13-05-10-02
run  :     development_2019-07-11-05-10-02
run  :     S18-08-13_2018-08-13-19-45-05
run  :     development_2019-07-10-05-10-01
run  :     R17-11-14-prod4mre.a_2018-05-02-19-45-45
run  :     development_2019-07-12-05-10-02
run  :     S18-11-12_2018-11-12-19-45-04
run  :     S18-04-07_2018-04-07-19-45-06
run  :     S17-10-09_2017-10-09-19-45-04
run  :     R17-03-09-prod3genie.j_2017-11-10-19-45-04
run  :     development_2019-07-22-05-10-02
run  :     development_2019-07-21-05-10-02
run  :     S17-09-21_2017-09-21-19-45-03
run  :     S17-09-24_2017-09-24-19-45-04
run  :     R17-11-14-prod4reco.neutron-respin.d_2018-07-27-19-45-46
run  :     R18-02-25-art1.a_2018-02-26-19-45-04
run  :     development_2019-07-20-05-10-02
run  :     S18-12-10_2018-12-10-19-45-04
run  :     S18-01-05_2018-01-05-19-45-12
run  :     S17-11-18_2017-11-18-19-45-04
run  :     development_2019-07-17-05-10-02
run  :     R17-03-01-prod3reco.i_2017-05-02-19-45-04
run  :     R17-03-09-prod3genie.f_2017-10-02-19-45-03
run  :     S18-08-13_2018-08-13-19-45-04
run  :     R17-03-01-prod3reco.l.srt-hotfix_2017-12-08-19-45-45
run  :     R17-11-14-prod4reco.d_2017-12-21-19-45-03
run  :     S17-06-19_2017-06-19-19-45-03
run  :     R17-03-01-prod3reco.d_2017-03-31-19-45-02
run  :     development_2019-07-16-05-10-02
run  :     R17-03-09-prod3genie.h_2017-10-27-19-45-02
run  :     S17-06-19_2017-06-19-19-45-04
run  :     R17-11-14-prod4reco.remid-hotfix.c_2018-12-13-19-45-03
run  :     R17-11-14-prod4mre.a_2018-05-02-19-45-44
run  :     index.html
run  :     config.html
run  :     results.html
run  :     projections.html
run  :     log_book.html
run  :     S18-10-29_2018-10-29-19-45-03
run  :     R17-03-09-prod3genie.i_2017-11-03-19-45-06
run  :     R18-03-15-feature_nc_legacy.b_2018-03-16-19-45-04
run  :     S17-09-20_2017-09-20-19-45-03
run  :     S17-06-30_2017-06-30-19-45-04
run  :     S18-01-05_2018-01-05-19-45-09
run  :     S18-09-15_2018-09-16-19-45-04
run  :     R17-09-05-prod4recopreview.a_2017-09-05-19-45-03
run  :     S18-07-27_2018-07-27-19-45-05
run  :     R17-03-01-prod3reco.d_2017-03-31-19-45-03
run  :     R17-11-14-prod4reco.remid-hotfix.b_2018-03-16-19-45-44
run  :     CR17-07-15_2017-07-15-19-45-03
run  :     R16-02-11-geniepreview.b_2016-10-12-19-45-06
run  :     S18-04-23_2018-04-23-19-45-04
run  :     R16-11-22-prod3geniepreview.f_2017-02-10-19-45-03
run  :     R16-11-22-prod3geniepreview.d_2016-12-20-19-45-03
run  :     R17-11-14-prod4reco.neutron-respin.c_2018-05-02-19-45-03
run  :     S19-01-02_2019-01-02-19-45-02
run  :     S18-09-29_2018-09-29-19-45-03
run  :     R17-11-14-prod4reco.f_2018-01-18-19-45-05
run  :     S18-03-21_2018-03-21-19-45-41
run  :     R17-03-01-prod3reco.p_2017-12-14-19-45-04
run  :     development_2019-07-19-05-10-01
run  :     development_2019-07-18-05-10-02
run  :     R16-02-11-prod2genie.d_2016-11-01-19-45-02
run  :     S17-11-18_2017-11-18-19-45-02
run  :     S17-02-21_2017-02-21-19-45-03
run  :     CR16-10-07_2016-10-07-19-45-06
run  :     CR16-10-07_2016-10-07-19-45-03
run  :     R17-03-01-prod3reco.i_2017-05-02-19-45-03
run  :     S18-12-18_2018-12-18-19-45-04
run  :     S17-09-12_2017-09-12-19-45-03
run  :     S17-08-11_2017-08-11-19-45-04
run  :     S18-06-28_2018-06-28-19-45-06
run  :     S18-03-09_2018-03-09-19-45-07
run  :     S18-06-28_2018-06-28-19-45-03
run  :     R15-05-22-nuexsecana.a_2016-03-15-19-45-03
run  :     R16-03-03-prod2reco.h_2016-06-17-19-45-03
run  :     S17-09-24_2017-09-24-19-45-03
run  :     S17-12-08_2017-12-08-19-45-03
run  :     R17-03-01-prod3reco.l.srt-hotfix_2017-12-08-19-45-44
run  :     S17-08-22_2017-08-22-19-45-03
run  :     R17-08-22-prod3nus17.b_2018-02-04-19-45-04
run  :     index.html~
run  :     R16-11-22-prod3geniepreview.e_2017-02-04-19-45-04
run  :     R17-11-14-prod4reco.CVNprong-respin.a_2018-07-12-19-45-04
run  :     S18-01-19_2018-01-19-19-45-03
run  :     S18-01-19_2018-01-19-19-45-02
run  :     S17-11-16_2017-11-16-19-45-04
run  :     R18-11-19-feature_stan.a_2018-11-20-19-45-04
run  :     R17-03-01-prod3reco.h_2017-04-21-19-45-02
run  :     S17-10-09_2017-10-09-19-45-02
run  :     R17-09-05-prod4recopreview.b_2017-09-08-19-45-03
run  :     R17-03-01-prod3reco.o_2017-11-15-19-45-04
run  :     R17-09-05-prod4recopreview.d_2017-09-13-19-45-04
run  :     R16-12-20-prod3recopreview.a_2016-12-20-19-45-09
run  :     R17-09-05-prod4recopreview.e_2017-09-18-19-45-06
run  :     R17-09-05-prod4recopreview.b_2017-09-08-19-45-02
run  :     S18-10-12_2018-10-12-19-45-05
run  :     S17-05-03_2017-05-03-19-45-42
run  :     R17-03-01-prod3reco.j_2017-05-03-19-45-03
run  :     S18-02-01_2018-02-01-19-45-48
run  :     R17-03-01-prod3reco.e_2017-04-05-19-45-03
run  :     R17-03-01-prod3reco.m_2017-08-31-19-45-03
run  :     CR17-07-15_2017-07-15-19-45-02
run  :     R17-03-09-prod3genie.i_2017-11-03-19-45-03
run  :     R17-08-22-prod3nus17.a_2018-01-04-19-45-04
run  :     S17-11-16_2017-11-16-19-45-03
run  :     R16-11-22-prod3geniepreview.b_2016-11-30-19-45-03
run  :     S19-01-16_2019-01-16-19-45-02
run  :     R17-11-14-prod4reco.i_2018-03-21-19-45-05
run  :     R17-11-14-prod4reco.i_2018-03-21-19-45-04
run  :     S18-02-01_2018-02-01-19-45-34
run  :     R17-11-14-prod4reco.c_2017-12-15-19-45-03
run  :     S18-09-29_2018-09-29-19-45-02
run  :     S18-03-09_2018-03-09-19-45-05
run  :     R17-11-14-prod4reco.j_2018-12-12-19-45-04
run  :     S18-09-02_2018-09-06-19-45-04
run  :     R17-11-14-prod4reco.g_2018-01-26-19-45-53
run  :     S16-03-25_2016-03-25-19-45-02
run  :     R17-03-01-prod3reco.j_2017-05-03-19-45-02
run  :     S17-03-16_2017-03-16-19-45-02
run  :     S18-11-29_2018-11-29-19-45-03
run  :     R16-11-12-feature_caf_size.a_2016-11-16-19-45-08
run  :     R17-03-01-prod3reco.k_2017-05-09-19-45-02
run  :     R17-03-01-prod3reco.r_2018-02-02-19-45-03
run  :     R17-03-01-prod3reco.o_2017-11-15-19-45-03
run  :     R17-09-05-prod4recopreview.e_2017-09-18-19-45-03
run  :     R17-03-01-prod3reco.g_2017-04-13-19-45-02
run  :     S17-11-22_2017-11-22-19-45-03
run  :     R17-03-01-prod3reco.f_2017-04-12-19-45-02
run  :     R18-10-08-prod5calib.a_2018-10-08-19-45-04
run  :     R17-02-24-nutools_upgrade.b_2017-02-28-19-45-03
run  :     S17-12-15_2017-12-15-19-45-41
run  :     S18-04-07_2018-04-07-19-45-03
run  :     S17-01-26_2017-01-27-19-45-03
run  :     R17-02-24-nutools_upgrade.b_2017-02-28-19-45-02
run  :     R16-02-11-geantpreview.a_2016-09-21-19-45-07
run  :     S17-10-30_2017-10-30-19-45-05
run  :     S17-12-15_2017-12-15-19-45-32
run  :     R17-11-14-prod4reco.a_2017-11-14-19-45-05
run  :     R16-11-22-prod3geniepreview.e_2017-02-04-19-45-02
run  :     S17-01-26_2017-01-27-19-45-02
run  :     R17-03-01-prod3reco.g_2017-04-13-19-45-03
run  :     S16-06-01_2016-06-01-19-45-03
run  :     S16-10-07_2016-10-07-19-45-10
run  :     R17-08-22-prod3nus17.a_2018-01-04-19-45-03
run  :     S17-07-13_2017-07-13-19-45-43
run  :     S17-07-13_2017-07-13-19-45-41
run  :     R16-12-20-prod3recopreview.c_2017-01-19-19-45-02
run  :     R16-12-20-prod3recopreview.c_2017-01-19-19-45-03
run  :     R16-12-20-prod3recopreview.d_2017-01-22-19-45-02
run  :     S16-11-12_2016-11-12-19-45-02
run  :     R17-11-14-prod4reco.g_2018-01-26-19-45-39
run  :     R17-03-01-prod3reco.q_2018-01-26-19-45-05
run  :     R17-11-14-prod4reco.remid-hotfix.b_2018-03-16-19-45-43
run  :     S17-09-05_2017-09-05-19-45-42
run  :     S17-01-11_2017-01-11-19-45-02
run  :     S17-01-11_2017-01-11-19-45-03
run  :     S16-08-04_2016-08-05-19-45-03
run  :     S18-11-29_2018-11-29-19-45-02
run  :     R17-11-22-2017ana.a_2018-02-01-19-45-04
run  :     R16-02-11-geniepreview.b_2016-10-12-19-45-03
run  :     S17-10-30_2017-10-30-19-45-03
run  :     R17-03-09-prod3genie.a_2017-03-09-19-45-04
run  :     R17-02-22-legacy-cafs-v1_2017-02-23-19-45-02
run  :     R16-11-02-prod3calibpreview.a_2016-11-04-19-45-02
run  :     R17-03-09-prod3genie.e_2017-08-02-19-45-04
run  :     S17-09-21_2017-09-21-19-45-04
run  :     R17-09-05-prod4recopreview.a_2017-09-05-19-45-02
run  :     R17-03-01-prod3reco.q_2018-01-26-19-45-04
run  :     R18-03-15-feature_nc_legacy.b_2018-03-16-19-45-03
run  :     S16-03-10_2016-03-10-19-45-02
run  :     R17-11-14-prod4reco.neutron-respin.d_2018-07-27-19-45-45
run  :     R17-03-01-prod3reco.p_2017-12-14-19-45-02
run  :     R17-03-01-prod3reco.n_2017-10-27-19-45-27
run  :     R18-06-11-2018ana.a_2018-06-11-19-45-02
run  :     R16-07-15-secondana.a_2016-07-28-19-45-02
run  :     R16-02-11-geantpreview.a_2016-09-21-19-45-03
run  :     R16-02-11-geniepreview.a_2016-09-20-19-45-03
run  :     S16-12-07.ifdb-hotfix_2017-12-13-19-45-03
run  :     R17-03-01-prod3reco.a_2017-03-02-19-45-10
run  :     R16-03-03-prod2reco.i_2017-09-27-19-45-04
run  :     S18-07-17_2018-07-17-19-45-05
run  :     R17-03-01-prod3reco.n_2017-10-27-19-46-18
run  :     R17-03-09-prod3genie.h_2017-10-27-19-45-03
run  :     R17-09-05-prod4recopreview.c_2017-09-11-19-45-04
run  :     R17-09-05-prod4recopreview.f_2017-09-29-19-45-04
run  :     R17-11-14-prod4reco.f_2018-01-18-19-45-07
run  :     R18-11-05-feature_novarwgt.b_2019-01-04-19-45-04
run  :     R17-08-22-prod3nus17.b_2018-02-04-19-45-02
run  :     feature_caf_size_2016-10-18-19-45-03
run  :     R16-11-12-feature_caf_size.a_2016-11-16-19-45-02
run  :     R16-02-11-prod2genie.a_2016-02-11-19-45-02
run  :     S16-03-04_2016-03-04-19-46-10
run  :     R17-09-05-prod4recopreview.d_2017-09-13-19-45-03
run  :     S18-02-19_2018-02-19-19-45-04
run  :     R16-11-22-prod3geniepreview.c_2016-12-15-19-45-03
run  :     R17-03-09-prod3genie.c_2017-03-22-19-45-03
run  :     R17-03-09-prod3genie.d_2017-04-10-19-45-03
run  :     S17-07-18_2017-07-18-19-45-04
run  :     S16-12-06_2016-12-06-19-45-03
run  :     R17-11-14-prod4reco.b_2017-11-27-19-45-03
run  :     S17-04-19_2017-04-19-19-45-02
run  :     R17-11-14-prod4reco.e_2018-01-16-19-45-05
run  :     R17-03-01-prod3reco.a_2017-03-02-19-45-09
run  :     R17-03-01-prod3reco.b_2017-03-02-19-45-03
run  :     R17-03-09-prod3genie.g_2017-10-23-19-45-04
run  :     R16-11-22-prod3geniepreview.b_2016-11-30-19-45-02
run  :     R17-03-01-prod3reco.l_2017-07-13-19-45-06
run  :     R16-02-11-fiberbrightness.a_2016-06-14-19-45-02
run  :     S16-11-02_2016-11-02-19-45-03
run  :     S16-05-10_2016-05-10-19-45-02
run  :     R16-11-22-prod3geniepreview.d_2016-12-20-19-45-02
run  :     S17-07-27_2017-07-27-19-45-03
run  :     R17-03-09-prod3genie.c_2017-03-22-19-45-02
run  :     R16-02-11-newgeo.a_2016-10-14-19-45-03
run  :     R17-03-09-prod3genie.b_2017-03-13-19-45-02
run  :     R16-11-22-prod3geniepreview.f_2017-02-10-19-45-02
run  :     S17-04-03_2017-04-03-19-45-02
run  :     S16-08-24_2016-08-24-19-45-11
run  :     S18-08-27_2018-08-27-19-45-04
run  :     R17-03-01-prod3reco.l_2017-07-13-19-45-04
run  :     S18-05-29_2018-05-29-19-45-05
run  :     R17-11-14-prod4reco.h.remid-hotfix_2018-02-22-19-45-45
run  :     R17-11-14-prod4reco.h.remid-hotfix_2018-02-22-19-45-41
run  :     R17-03-09-prod3genie.k_2018-02-22-19-45-04
run  :     R17-08-22-prod3nus17.c_2018-02-07-19-45-03
run  :     S18-06-14_2018-06-15-19-45-04
run  :     S18-05-14_2018-05-14-19-45-04
run  :     S18-02-25_2018-02-25-19-45-35
run  :     R19-02-06-nutools-upgrade.a_2019-02-11-19-45-04
run  :     R16-02-11-geantpreview.b_2016-10-03-19-45-03
run  :     S16-04-08_2016-04-08-19-45-03
run  :     S18-05-06_2018-05-07-19-45-04
run  :     R17-11-14-prod4reco.neutron-respin.b_2018-04-21-19-45-02
run  :     S16-09-12_2016-09-12-19-45-04
run  :     R17-03-01-prod3reco.c_2017-03-07-19-45-03
run  :     R17-03-09-prod3genie.b_2017-03-13-19-45-04
run  :     R16-11-22-prod3geniepreview.a_2016-11-22-19-45-03
run  :     S16-10-21_2016-10-21-19-45-04
run  :     S17-05-17_2017-05-17-19-45-02
run  :     S16-12-07_2016-12-07-19-45-03
run  :     S16-08-24_2016-08-24-19-45-09
run  :     CR16-08-23_2016-08-24-19-45-03
run  :     S16-03-10_2016-03-14-19-46-20
run  :     S16-06-21_2016-06-21-19-45-05
run  :     S16-06-21_2016-06-21-19-45-03
run  :     S16-07-06_2016-07-06-19-45-03
run  :     S16-10-26_2016-10-27-19-45-02
run  :     feature_caf_size.a_2016-11-13-19-45-02
run  :     R16-02-11-geant4101p3.a_2016-05-06-19-45-03
run  :     R16-02-11-prod2genie.c_2016-05-24-19-45-02
run  :     S16-04-21_2016-04-26-19-45-02
run  :     S16-12-06_2016-12-06-19-45-02
run  :     R16-03-03-prod2reco.g_2016-06-06-19-45-03
run  :     S16-06-28_2016-06-28-19-45-02
run  :     R16-11-22-prod3geniepreview.a_2016-11-22-19-45-02
run  :     S17-05-17_2017-05-17-19-45-03
run  :     R16-02-11-prod2genie.b_2016-03-04-19-45-03
run  :     R17-01-26-CCPi02017.a_2017-10-11-19-45-58
run  :     R16-03-03-prod2reco.e_2016-05-03-19-45-03
run  :     S16-07-26_2016-07-26-19-45-03
run  :     R17-03-09-prod3genie.k_2018-02-22-19-45-02
run  :     R16-03-03-prod2reco.d_2016-04-05-19-45-02
run  :     R18-11-05-feature_novarwgt.a_2018-11-05-19-45-04
run  :     R16-03-03-prod2reco.b_2016-03-21-19-45-03
run  :     R17-11-14-prod4reco.remid-hotfix.a_2018-02-25-19-45-04
run  :     S16-06-29_2016-06-29-19-45-02
run  :     R16-03-03-prod2reco.a_2016-03-03-19-45-03
run  :     S16-07-06_2016-07-06-19-45-02
run  :     R16-03-03-mixedcalib.a_2016-06-20-19-45-02
run  :     R16-03-03-prod2reco.f_2016-05-20-19-46-10
run  :     R17-03-01-prod3reco.s_2018-07-21-19-45-04
run  :     S17-10-10_2017-10-11-19-45-03
run  :     R17-11-14-prod4reco.neutron-respin.a_2018-04-20-19-45-03
run  :     R17-11-14-prod4reco.neutron-respin.b_2018-04-21-19-45-04
run  :     S16-03-04_2016-03-14-19-45-02
run  :     S16-09-12_2016-09-12-19-45-03
run  :     S16-09-13_2016-09-13-19-45-03
run  :     S18-02-09_2018-02-09-19-45-04
run  :     S16-05-20_2016-05-20-19-45-03
run  :     R17-11-14-prod4reco.b_2017-11-27-19-45-02
run  :     R16-11-12-feature_caf_size.b_2017-01-28-19-45-03
run  :     R18-04-20-art2.11.b_2018-05-16-19-45-04
run  :     input folder R16-03-03-prod2reco.b_2016-03-21-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder S16-06-29_2016-06-29-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R16-03-03-prod2reco.d_2016-04-05-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder S16-12-06_2016-12-06-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-03-04_2016-03-04-19-46-10 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - info.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder R16-02-11-prod2genie.b_2016-03-04-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_data_cosmics.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl FD_data_cosmics.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder S16-05-20_2016-05-20-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder S16-06-28_2016-06-28-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder S16-04-21_2016-04-26-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder S16-09-12_2016-09-12-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-02-11-prod2genie.a_2016-02-11-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_data_cosmics.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl FD_data_cosmics.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :     input folder R16-02-11-geniepreview.a_2016-09-20-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-03-04_2016-03-14-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder S16-03-10_2016-03-10-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_data_cosmics.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_data_cosmics.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :     input folder R16-03-03-mixedcalib.a_2016-06-20-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R15-05-22-nuexsecana.a_2016-03-15-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_data_cosmics.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl FD_data_cosmics.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :     input folder R16-02-11-geantpreview.a_2016-09-21-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-07-06_2016-07-06-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R16-03-03-prod2reco.a_2016-03-03-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_data_cosmics.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_data_cosmics.pkl is in outputs
run  :     input folder S16-04-08_2016-04-08-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder S16-09-13_2016-09-13-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.d_2016-12-20-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-03-03-prod2reco.e_2016-05-03-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-07-06_2016-07-06-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-07-26_2016-07-26-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R16-02-11-geantpreview.b_2016-10-03-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-09-12_2016-09-12-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-06-21_2016-06-21-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder S16-06-21_2016-06-21-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-06-01_2016-06-01-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R16-02-11-prod2genie.c_2016-05-24-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder S16-03-25_2016-03-25-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder R16-03-03-prod2reco.f_2016-05-20-19-46-10 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder R16-03-03-prod2reco.h_2016-06-17-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R16-02-11-fiberbrightness.a_2016-06-14-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R16-02-11-geant4101p3.a_2016-05-06-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-03-03-prod2reco.g_2016-06-06-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder feature_caf_size_2016-10-18-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-05-10_2016-05-10-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-11-02_2016-11-02-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-02-11-prod2genie.d_2016-11-01-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-03-10_2016-03-14-19-46-20 is in outputs, checking for new files
run  :     output pkls:
run  :      - FD_cosmics.pkl
run  :      - FD_data_NuMI.pkl
run  :      - FD_genie_FHC_nonswap.pkl
run  :      - FD_genie_FHC_swap.pkl
run  :      - FD_genie_FHC_tau.pkl
run  :      - FD_genie_RHC_nonswap.pkl
run  :      - FD_genie_RHC_swap.pkl
run  :      - FD_genie_RHC_tau.pkl
run  :      - ND_cosmics.pkl
run  :      - ND_data_NuMI.pkl
run  :      - ND_data_cosmics.pkl
run  :      - ND_genie_FHC_nonswap.pkl
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :      - input pkl ND_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl ND_data_cosmics.pkl is in outputs
run  :      - input pkl FD_genie_FHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_RHC_nonswap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_tau.pkl is in outputs
run  :      - input pkl FD_genie_FHC_nonswap.pkl is in outputs
run  :      - input pkl FD_data_NuMI.pkl is in outputs
run  :      - input pkl FD_genie_FHC_swap.pkl is in outputs
run  :      - input pkl FD_genie_RHC_swap.pkl is in outputs
run  :      - input pkl FD_cosmics.pkl is in outputs
run  :      - input pkl ND_data_NuMI.pkl is in outputs
run  :      - input pkl ND_cosmics.pkl is in outputs
run  :     input folder R16-11-12-feature_caf_size.a_2016-11-16-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-02-11-geantpreview.a_2016-09-21-19-45-07 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.i_2017-05-02-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-07-15-secondana.a_2016-07-28-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder S17-05-17_2017-05-17-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.a_2016-11-22-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-08-04_2016-08-05-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-10-26_2016-10-27-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder CR16-08-23_2016-08-24-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-08-24_2016-08-24-19-45-09 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-08-24_2016-08-24-19-45-11 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-01-11_2017-01-11-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-12-07_2016-12-07-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-12-06_2016-12-06-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-07-18_2017-07-18-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-12-20-prod3recopreview.c_2017-01-19-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder CR16-10-07_2016-10-07-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder CR16-10-07_2016-10-07-19-45-06 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-10-07_2016-10-07-19-45-10 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.e_2017-02-04-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.a_2017-03-09-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-12-20-prod3recopreview.c_2017-01-19-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.d_2016-12-20-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-02-11-geniepreview.b_2016-10-12-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-02-11-geniepreview.b_2016-10-12-19-45-06 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-02-11-newgeo.a_2016-10-14-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.m_2017-08-31-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-08-22_2017-08-22-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.d_2017-03-31-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-05-17_2017-05-17-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-10-21_2016-10-21-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-02-prod3calibpreview.a_2016-11-04-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-01-11_2017-01-11-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-11-12_2016-11-12-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder feature_caf_size.a_2016-11-13-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-12-feature_caf_size.a_2016-11-16-19-45-08 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.a_2016-11-22-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.c_2017-03-22-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-06-19_2017-06-19-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.b_2016-11-30-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.b_2016-11-30-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.l_2017-07-13-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.e_2017-08-02-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.d_2017-03-31-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.i_2017-11-03-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-07-27_2017-07-27-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.c_2016-12-15-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-12-20-prod3recopreview.a_2016-12-20-19-45-09 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-04-03_2017-04-03-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-12-20-prod3recopreview.d_2017-01-22-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.a_2017-11-14-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-01-26_2017-01-27-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-01-26_2017-01-27-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-12-feature_caf_size.b_2017-01-28-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.e_2017-02-04-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.e_2017-09-18-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.f_2017-02-10-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-11-22-prod3geniepreview.f_2017-02-10-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-02-21_2017-02-21-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-10-10_2017-10-11-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-02-22-legacy-cafs-v1_2017-02-23-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-02-24-nutools_upgrade.b_2017-02-28-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-02-24-nutools_upgrade.b_2017-02-28-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.b_2017-03-02-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.a_2017-03-02-19-45-09 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.a_2017-03-02-19-45-10 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-11-18_2017-11-18-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.c_2017-03-07-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-04-07_2018-04-07-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.g_2017-10-23-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.b_2017-03-13-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.b_2017-03-13-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-06-19_2017-06-19-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-03-16_2017-03-16-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.c_2017-03-22-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.e_2017-04-05-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.e_2018-01-16-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R17-03-09-prod3genie.d_2017-04-10-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.f_2017-04-12-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.g_2017-04-13-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.g_2017-04-13-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.b_2017-11-27-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-04-19_2017-04-19-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.h_2017-04-21-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.i_2017-05-02-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.j_2017-05-03-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.j_2017-05-03-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-05-03_2017-05-03-19-45-42 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.k_2017-05-09-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-02-19_2018-02-19-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-11-22_2017-11-22-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.CVNprong-respin.a_2018-07-12-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.q_2018-01-26-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.neutron-respin.a_2018-04-20-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-06-30_2017-06-30-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-10-12_2018-10-12-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-09-02_2018-09-06-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.l_2017-07-13-19-45-06 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-07-13_2017-07-13-19-45-41 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-07-13_2017-07-13-19-45-43 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder CR17-07-15_2017-07-15-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder CR17-07-15_2017-07-15-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-01-05_2018-01-05-19-45-09 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-08-11_2017-08-11-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.a_2017-09-05-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.a_2017-09-05-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-09-05_2017-09-05-19-45-42 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.b_2017-09-08-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.b_2017-09-08-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.k_2018-02-22-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.c_2017-09-11-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-09-12_2017-09-12-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-04-20-art2.11.b_2018-05-16-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.d_2017-09-13-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.d_2017-09-13-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.e_2017-09-18-19-45-06 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-10-08-prod5calib.a_2018-10-08-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-09-20_2017-09-20-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-09-21_2017-09-21-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-09-21_2017-09-21-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-12-18_2018-12-18-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-09-24_2017-09-24-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-09-24_2017-09-24-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R16-03-03-prod2reco.i_2017-09-27-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.d_2017-12-21-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-09-05-prod4recopreview.f_2017-09-29-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.q_2018-01-26-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.f_2017-10-02-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-10-09_2017-10-09-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-10-09_2017-10-09-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-01-26-CCPi02017.a_2017-10-11-19-45-58 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.h_2017-10-27-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.h_2017-10-27-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R17-03-01-prod3reco.n_2017-10-27-19-45-27 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.n_2017-10-27-19-46-18 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R17-11-22-2017ana.a_2018-02-01-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-06-11-2018ana.a_2018-06-11-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-10-30_2017-10-30-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-10-30_2017-10-30-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-08-22-prod3nus17.b_2018-02-04-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.i_2017-11-03-19-45-06 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.j_2017-11-10-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.o_2017-11-15-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.o_2017-11-15-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-11-16_2017-11-16-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-11-16_2017-11-16-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-11-18_2017-11-18-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-02-09_2018-02-09-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.b_2017-11-27-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-12-08_2017-12-08-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.l.srt-hotfix_2017-12-08-19-45-44 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.l.srt-hotfix_2017-12-08-19-45-45 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-11-05-feature_novarwgt.a_2018-11-05-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S16-12-07.ifdb-hotfix_2017-12-13-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R17-03-01-prod3reco.p_2017-12-14-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.p_2017-12-14-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-04-23_2018-04-23-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.c_2017-12-15-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-12-15_2017-12-15-19-45-32 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S17-12-15_2017-12-15-19-45-41 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.remid-hotfix.c_2018-12-13-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S19-01-14_2019-01-14-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.neutron-respin.b_2018-04-21-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-03-09_2018-03-09-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-08-22-prod3nus17.a_2018-01-04-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-08-22-prod3nus17.a_2018-01-04-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-01-05_2018-01-05-19-45-12 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.f_2018-01-18-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :      - info.pkl
run  :      - input pkl info.pkl is in outputs
run  :     input folder R17-11-14-prod4reco.f_2018-01-18-19-45-07 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-01-19_2018-01-19-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-01-19_2018-01-19-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.g_2018-01-26-19-45-39 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.g_2018-01-26-19-45-53 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-02-01_2018-02-01-19-45-34 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-02-01_2018-02-01-19-45-48 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.r_2018-02-02-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-08-22-prod3nus17.b_2018-02-04-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-05-14_2018-05-14-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-06-14_2018-06-15-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.j_2018-12-12-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.neutron-respin.b_2018-04-21-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-12-10_2018-12-10-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-08-22-prod3nus17.c_2018-02-07-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-09-prod3genie.k_2018-02-22-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.h.remid-hotfix_2018-02-22-19-45-41 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.h.remid-hotfix_2018-02-22-19-45-45 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.remid-hotfix.a_2018-02-25-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-02-25_2018-02-25-19-45-35 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-02-25-art1.a_2018-02-26-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-05-06_2018-05-07-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-09-29_2018-09-29-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-06-28_2018-06-28-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-03-09_2018-03-09-19-45-07 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-03-15-feature_nc_legacy.b_2018-03-16-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-03-15-feature_nc_legacy.b_2018-03-16-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.remid-hotfix.b_2018-03-16-19-45-43 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.remid-hotfix.b_2018-03-16-19-45-44 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.i_2018-03-21-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.i_2018-03-21-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-03-21_2018-03-21-19-45-41 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-08-13_2018-08-13-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-11-12_2018-11-12-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-19-05-10-01 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-06-28_2018-06-28-19-45-06 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-09-29_2018-09-29-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-04-07_2018-04-07-19-45-06 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.neutron-respin.c_2018-05-02-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4mre.a_2018-05-02-19-45-44 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4mre.a_2018-05-02-19-45-45 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-08-27_2018-08-27-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S19-01-02_2019-01-02-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-09-15_2018-09-16-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-05-29_2018-05-29-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-14-05-10-01 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-11-19-feature_stan.a_2018-11-20-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-20-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-10-05-10-01 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-23-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-09-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-07-17_2018-07-17-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-03-01-prod3reco.s_2018-07-21-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-07-27_2018-07-27-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.neutron-respin.d_2018-07-27-19-45-45 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R17-11-14-prod4reco.neutron-respin.d_2018-07-27-19-45-46 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-15-05-10-01 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-24-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-12-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-13-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-08-13_2018-08-13-19-45-05 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-16-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-17-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-18-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-21-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-22-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R18-11-05-feature_novarwgt.b_2019-01-04-19-45-04 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-10-29_2018-10-29-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder development_2019-07-11-05-10-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-11-29_2018-11-29-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S18-11-29_2018-11-29-19-45-03 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder S19-01-16_2019-01-16-19-45-02 is in outputs, checking for new files
run  :     output pkls:
run  :     input folder R19-02-06-nutools-upgrade.a_2019-02-11-19-45-04 is in outputs, checking for new files
run  :     output pkls:
--- run
rm: cannot remove `/nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/embed/*.js': No such file or directory
run  : --- Make HTML Front Page
run  : --- Options
run  :     work dir:      ./
run  :     output dir:    /nusoft/app/web/htdoc/nova/production/testing/
run  :     local mode:    False
run  :     full refresh:  True
run  :     wildcard:      False
run  :     quick mode:    False
run  :     verbose mode:  True
run  : --- Making front page
svd page file name =  index.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/index.html
run  : --- Making config page
svd page file name =  config.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/config.html
run  : --- Making results page
PMS  :     Making results page for folder: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03
PMS  :     Making results page info: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/info.pkl
PMS  : --- Making batch log pages
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_cosmics 200 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194510.709327_8929', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194510.709327_8929/batch-job.sh_20160315_194511_2154550_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192986.', '', 'JobsubJobId of first job: 8192986.0@fifebatch2.fnal.gov', '', 'Use job id 8192986.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194510.709327_8929

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194510.709327_8929/batch-job.sh_20160315_194511_2154550_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192986.

JobsubJobId of first job: 8192986.0@fifebatch2.fnal.gov

Use job id 8192986.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192986.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.119681, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194511_2154550_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194511_2154550_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_cosmics_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_cosmics_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194511_2154550_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194511_2154550_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_cosmics_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_cosmics_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194511_2154550_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194511_2154550_0_1_cluster.8192986.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_cosmics_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_cosmics_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194511_2154550_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192986.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194511_2154550_0_1_cluster.8192986.0.err
tools: ---Parsing error log
tools:     read 17 lines
tools:         unfiltered error line[1/17]:
tools:           - 'Wed Mar 16 02:36:49 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_data_cosmics 2015 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194510.709327_8929': No such file or directory"
tools:         unfiltered error line[7/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194510.709327_8929': No such file or directory"
tools:         unfiltered error line[8/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194510.709327_8929': No such file or directory"
tools:         unfiltered error line[13/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194510.709327_8929': No such file or directory"
tools:         unfiltered error line[14/17]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_cosmics_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_cosmics_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_data_NuMI 138 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194515.249005_1783', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194515.249005_1783/batch-job.sh_20160315_194516_2154715_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192987.', '', 'JobsubJobId of first job: 8192987.0@fifebatch2.fnal.gov', '', 'Use job id 8192987.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194515.249005_1783

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194515.249005_1783/batch-job.sh_20160315_194516_2154715_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192987.

JobsubJobId of first job: 8192987.0@fifebatch2.fnal.gov

Use job id 8192987.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192987.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.152516, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194516_2154715_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194516_2154715_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_NuMI_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_NuMI_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194516_2154715_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194516_2154715_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_NuMI_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_NuMI_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194516_2154715_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194516_2154715_0_1_cluster.8192987.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_NuMI_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_NuMI_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194516_2154715_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192987.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194516_2154715_0_1_cluster.8192987.0.err
tools: ---Parsing error log
tools:     read 17 lines
tools:         unfiltered error line[1/17]:
tools:           - 'Wed Mar 16 02:56:20 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_data_NuMI 138 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194515.249005_1783': No such file or directory"
tools:         unfiltered error line[7/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194515.249005_1783': No such file or directory"
tools:         unfiltered error line[8/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194515.249005_1783': No such file or directory"
tools:         unfiltered error line[13/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194515.249005_1783': No such file or directory"
tools:         unfiltered error line[14/17]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_NuMI_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_NuMI_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_data_cosmics 2015 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194520.619836_6282', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194520.619836_6282/batch-job.sh_20160315_194521_2154924_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192988.', '', 'JobsubJobId of first job: 8192988.0@fifebatch2.fnal.gov', '', 'Use job id 8192988.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194520.619836_6282

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194520.619836_6282/batch-job.sh_20160315_194521_2154924_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192988.

JobsubJobId of first job: 8192988.0@fifebatch2.fnal.gov

Use job id 8192988.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192988.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.183684, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194521_2154924_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194521_2154924_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_cosmics_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_cosmics_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194521_2154924_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194521_2154924_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_cosmics_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_cosmics_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194521_2154924_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194521_2154924_0_1_cluster.8192988.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_cosmics_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_cosmics_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194521_2154924_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192988.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194521_2154924_0_1_cluster.8192988.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 02:57:59 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_cosmics 200 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194520.619836_6282': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194520.619836_6282': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194520.619836_6282': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194520.619836_6282': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_cosmics_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_data_cosmics_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_genie_FHC_nonswap 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194525.099420_5293', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194525.099420_5293/batch-job.sh_20160315_194526_2155094_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192989.', '', 'JobsubJobId of first job: 8192989.0@fifebatch2.fnal.gov', '', 'Use job id 8192989.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194525.099420_5293

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194525.099420_5293/batch-job.sh_20160315_194526_2155094_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192989.

JobsubJobId of first job: 8192989.0@fifebatch2.fnal.gov

Use job id 8192989.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192989.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.214216, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194526_2155094_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194526_2155094_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_nonswap_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_nonswap_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194526_2155094_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194526_2155094_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_nonswap_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_nonswap_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194526_2155094_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194526_2155094_0_1_cluster.8192989.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_nonswap_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_nonswap_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194526_2155094_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192989.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194526_2155094_0_1_cluster.8192989.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 02:58:22 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_genie_FHC_nonswap 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194525.099420_5293': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194525.099420_5293': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194525.099420_5293': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194525.099420_5293': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_nonswap_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_nonswap_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_genie_FHC_swap 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194529.635335_2740', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194529.635335_2740/batch-job.sh_20160315_194530_2155293_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192990.', '', 'JobsubJobId of first job: 8192990.0@fifebatch2.fnal.gov', '', 'Use job id 8192990.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194529.635335_2740

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194529.635335_2740/batch-job.sh_20160315_194530_2155293_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192990.

JobsubJobId of first job: 8192990.0@fifebatch2.fnal.gov

Use job id 8192990.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192990.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.243510, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194530_2155293_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194530_2155293_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_swap_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_swap_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194530_2155293_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194530_2155293_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_swap_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_swap_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194530_2155293_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194530_2155293_0_1_cluster.8192990.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_swap_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_swap_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194530_2155293_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192990.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194530_2155293_0_1_cluster.8192990.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 02:58:38 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_genie_FHC_swap 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194529.635335_2740': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194529.635335_2740': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194529.635335_2740': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194529.635335_2740': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_swap_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_swap_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_genie_FHC_tau 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194534.248305_3925', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194534.248305_3925/batch-job.sh_20160315_194535_2155598_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192992.', '', 'JobsubJobId of first job: 8192992.0@fifebatch2.fnal.gov', '', 'Use job id 8192992.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194534.248305_3925

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194534.248305_3925/batch-job.sh_20160315_194535_2155598_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192992.

JobsubJobId of first job: 8192992.0@fifebatch2.fnal.gov

Use job id 8192992.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192992.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.261349, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194535_2155598_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194535_2155598_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_tau_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_tau_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194535_2155598_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194535_2155598_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_tau_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_tau_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194535_2155598_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194535_2155598_0_1_cluster.8192992.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_tau_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_tau_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194535_2155598_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192992.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194535_2155598_0_1_cluster.8192992.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 02:58:52 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_genie_FHC_tau 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194534.248305_3925': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194534.248305_3925': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194534.248305_3925': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194534.248305_3925': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_tau_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_FHC_tau_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_genie_RHC_nonswap 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194538.807872_185', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194538.807872_185/batch-job.sh_20160315_194539_2155760_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192993.', '', 'JobsubJobId of first job: 8192993.0@fifebatch2.fnal.gov', '', 'Use job id 8192993.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194538.807872_185

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194538.807872_185/batch-job.sh_20160315_194539_2155760_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192993.

JobsubJobId of first job: 8192993.0@fifebatch2.fnal.gov

Use job id 8192993.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192993.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.287545, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194539_2155760_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194539_2155760_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_nonswap_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_nonswap_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194539_2155760_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194539_2155760_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_nonswap_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_nonswap_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194539_2155760_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194539_2155760_0_1_cluster.8192993.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_nonswap_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_nonswap_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194539_2155760_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192993.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194539_2155760_0_1_cluster.8192993.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 02:59:05 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_genie_RHC_nonswap 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194538.807872_185': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194538.807872_185': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194538.807872_185': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194538.807872_185': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_nonswap_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_nonswap_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_genie_RHC_swap 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194543.347252_2423', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194543.347252_2423/batch-job.sh_20160315_194544_2155929_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192994.', '', 'JobsubJobId of first job: 8192994.0@fifebatch2.fnal.gov', '', 'Use job id 8192994.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194543.347252_2423

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194543.347252_2423/batch-job.sh_20160315_194544_2155929_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192994.

JobsubJobId of first job: 8192994.0@fifebatch2.fnal.gov

Use job id 8192994.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192994.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.332560, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194544_2155929_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194544_2155929_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_swap_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_swap_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194544_2155929_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194544_2155929_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_swap_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_swap_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194544_2155929_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194544_2155929_0_1_cluster.8192994.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_swap_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_swap_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194544_2155929_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192994.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194544_2155929_0_1_cluster.8192994.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 02:59:18 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_genie_RHC_swap 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194543.347252_2423': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194543.347252_2423': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194543.347252_2423': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194543.347252_2423': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_swap_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_swap_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a FD_genie_RHC_tau 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194548.984006_5870', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194548.984006_5870/batch-job.sh_20160315_194549_2156095_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192995.', '', 'JobsubJobId of first job: 8192995.0@fifebatch2.fnal.gov', '', 'Use job id 8192995.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194548.984006_5870

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194548.984006_5870/batch-job.sh_20160315_194549_2156095_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192995.

JobsubJobId of first job: 8192995.0@fifebatch2.fnal.gov

Use job id 8192995.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192995.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.373204, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194549_2156095_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194549_2156095_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_tau_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_tau_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194549_2156095_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194549_2156095_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_tau_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_tau_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194549_2156095_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194549_2156095_0_1_cluster.8192995.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_tau_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_tau_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194549_2156095_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192995.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194549_2156095_0_1_cluster.8192995.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 02:59:28 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a FD_genie_RHC_tau 1000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194548.984006_5870': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194548.984006_5870': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194548.984006_5870': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194548.984006_5870': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_tau_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/FD_genie_RHC_tau_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a ND_cosmics 250000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194553.601658_5030', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194553.601658_5030/batch-job.sh_20160315_194554_2156266_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192996.', '', 'JobsubJobId of first job: 8192996.0@fifebatch2.fnal.gov', '', 'Use job id 8192996.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194553.601658_5030

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194553.601658_5030/batch-job.sh_20160315_194554_2156266_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192996.

JobsubJobId of first job: 8192996.0@fifebatch2.fnal.gov

Use job id 8192996.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192996.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.421982, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194554_2156266_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194554_2156266_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_cosmics_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_cosmics_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194554_2156266_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194554_2156266_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_cosmics_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_cosmics_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194554_2156266_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194554_2156266_0_1_cluster.8192996.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_cosmics_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_cosmics_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194554_2156266_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192996.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194554_2156266_0_1_cluster.8192996.0.err
tools: ---Parsing error log
tools:     read 17 lines
tools:         unfiltered error line[1/17]:
tools:           - 'Wed Mar 16 02:59:41 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a ND_data_cosmics 3599 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194553.601658_5030': No such file or directory"
tools:         unfiltered error line[7/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194553.601658_5030': No such file or directory"
tools:         unfiltered error line[8/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194553.601658_5030': No such file or directory"
tools:         unfiltered error line[13/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194553.601658_5030': No such file or directory"
tools:         unfiltered error line[14/17]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_cosmics_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_cosmics_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a ND_data_NuMI 1952 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194558.048506_3052', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194558.048506_3052/batch-job.sh_20160315_194559_2156590_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192998.', '', 'JobsubJobId of first job: 8192998.0@fifebatch2.fnal.gov', '', 'Use job id 8192998.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194558.048506_3052

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194558.048506_3052/batch-job.sh_20160315_194559_2156590_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192998.

JobsubJobId of first job: 8192998.0@fifebatch2.fnal.gov

Use job id 8192998.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192998.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.451396, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194559_2156590_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194559_2156590_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_NuMI_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_NuMI_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194559_2156590_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194559_2156590_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_NuMI_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_NuMI_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194559_2156590_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194559_2156590_0_1_cluster.8192998.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_NuMI_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_NuMI_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194559_2156590_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192998.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194559_2156590_0_1_cluster.8192998.0.err
tools: ---Parsing error log
tools:     read 17 lines
tools:         unfiltered error line[1/17]:
tools:           - 'Wed Mar 16 03:02:14 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a ND_data_NuMI 1952 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194558.048506_3052': No such file or directory"
tools:         unfiltered error line[7/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194558.048506_3052': No such file or directory"
tools:         unfiltered error line[8/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/17]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/17]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194558.048506_3052': No such file or directory"
tools:         unfiltered error line[13/17]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194558.048506_3052': No such file or directory"
tools:         unfiltered error line[14/17]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_NuMI_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_NuMI_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a ND_data_cosmics 3599 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194602.708792_4495', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194602.708792_4495/batch-job.sh_20160315_194603_2156768_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8192999.', '', 'JobsubJobId of first job: 8192999.0@fifebatch2.fnal.gov', '', 'Use job id 8192999.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194602.708792_4495

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194602.708792_4495/batch-job.sh_20160315_194603_2156768_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8192999.

JobsubJobId of first job: 8192999.0@fifebatch2.fnal.gov

Use job id 8192999.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8192999.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.481440, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194603_2156768_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194603_2156768_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_cosmics_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_cosmics_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194603_2156768_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194603_2156768_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_cosmics_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_cosmics_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194603_2156768_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194603_2156768_0_1_cluster.8192999.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_cosmics_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_cosmics_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194603_2156768_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8192999.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194603_2156768_0_1_cluster.8192999.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 03:03:43 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a ND_cosmics 250000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194602.708792_4495': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194602.708792_4495': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194602.708792_4495': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194602.708792_4495': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_cosmics_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_data_cosmics_err.html
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R15-05-22-nuexsecana.a ND_genie_FHC_nonswap 2000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194607.555473_5636', '', '/fife/local/scratch/uploads/nova/novapro/2016-03-15_194607.555473_5636/batch-job.sh_20160315_194608_2156991_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 8193000.', '', 'JobsubJobId of first job: 8193000.0@fifebatch2.fnal.gov', '', 'Use job id 8193000.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-03-15_194607.555473_5636

/fife/local/scratch/uploads/nova/novapro/2016-03-15_194607.555473_5636/batch-job.sh_20160315_194608_2156991_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 8193000.

JobsubJobId of first job: 8193000.0@fifebatch2.fnal.gov

Use job id 8193000.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 8193000.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-03-15 00:00:00, now: 2019-07-24 10:35:03.513524, days: 1226
PMS  :     using existing logs
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194608_2156991_0_1_*cmd
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194608_2156991_0_1_.cmd
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_genie_FHC_nonswap_cmd.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_genie_FHC_nonswap_cmd.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194608_2156991_0_1_*log
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194608_2156991_0_1_.log
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_genie_FHC_nonswap_log.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_genie_FHC_nonswap_log.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194608_2156991_0_1_*out
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194608_2156991_0_1_cluster.8193000.0.out
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_genie_FHC_nonswap_out.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_genie_FHC_nonswap_out.html
PMS  :     Finding log file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194608_2156991_0_1_*err
PMS  :       - using file: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/8193000.0@fifebatch2.fnal.gov/batch-job.sh_20160315_194608_2156991_0_1_cluster.8193000.0.err
tools: ---Parsing error log
tools:     read 18 lines
tools:         unfiltered error line[1/18]:
tools:           - 'Wed Mar 16 03:03:58 CDT 2016 BEGIN EXECUTION ./batch-job.sh R15-05-22-nuexsecana.a ND_genie_FHC_nonswap 2000 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ maxopt'
tools:         unfiltered error line[2/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[3/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[4/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[5/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[6/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194607.555473_5636': No such file or directory"
tools:         unfiltered error line[7/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194607.555473_5636': No such file or directory"
tools:         unfiltered error line[8/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[9/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[10/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro': No such file or directory"
tools:         unfiltered error line[11/18]:
tools:           - "mkdir: cannot create directory `/fife': Permission denied"
tools:         unfiltered error line[12/18]:
tools:           - "chgrp: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194607.555473_5636': No such file or directory"
tools:         unfiltered error line[13/18]:
tools:           - "chmod: cannot access `/fife/local/scratch/uploads/nova/novapro/2016-03-15_194607.555473_5636': No such file or directory"
tools:         unfiltered error line[14/18]:
tools:           - 'ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product sam_web_client: version v1_9 vs v2_0'
tools:         this is an error, returning 4
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_genie_FHC_nonswap_err.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/batch_logs/ND_genie_FHC_nonswap_err.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_cosmics.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: cry
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier cry
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_cry.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_cry.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_cry.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_cry.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_cry.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_cry.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_cry.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_cry.html
PMS  :     Making results page for tier: pchits
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pchits
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_pchits.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_cosmics_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_data_NuMI.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: raw2root
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier raw2root
vbokh:     parsing log
vbokh:     events found: 138
vbokh:     events used: 20
vbokh:      - making array for memory
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for memory_percent
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for vms_memory
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for cpu_percent
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for u_cpu_times
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/file_sizes/FD_data_NuMI_raw2root_fardet_r00018228_s31_t00_raw_root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/file_sizes/FD_data_NuMI_raw2root_fardet_r00018228_s31_t00_raw_root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_raw2root.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_fullchain_nolem.html
PMS  :     Making results page for tier: recokeepup
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recokeepup
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_recokeepup.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_NuMI_recokeepup.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_recokeepup.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_NuMI_recokeepup.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_recokeepup.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_NuMI_recokeepup.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_recokeepup.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_NuMI_recokeepup.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_data_cosmics.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: raw2root
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier raw2root
vbokh:     parsing log
vbokh:     events found: 2015
vbokh:     events used: 20
vbokh:      - making array for memory
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for memory_percent
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for vms_memory
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for cpu_percent
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for u_cpu_times
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/file_sizes/FD_data_cosmics_raw2root_fardet_r00018228_s31_t02_raw_root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/file_sizes/FD_data_cosmics_raw2root_fardet_r00018228_s31_t02_raw_root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_raw2root.html
PMS  :     Making results page for tier: pchits
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pchits
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_pchits.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_data_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_data_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_data_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_data_cosmics_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_genie_FHC_nonswap.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: genie
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier genie
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_genie.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_nonswap_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_genie_FHC_swap.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: genie
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier genie
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_genie.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_swap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_swap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_swap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_swap_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_genie_FHC_tau.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: genie
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier genie
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_genie.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_FHC_tau_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_FHC_tau_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_FHC_tau_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_FHC_tau_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_genie_RHC_nonswap.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: genie
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier genie
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_genie.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_nonswap_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_genie_RHC_swap.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: genie
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier genie
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_genie.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_swap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_swap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_swap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_swap_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/FD_genie_RHC_tau.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: genie
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier genie
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_genie.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_genie.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_genie.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/FD_genie_RHC_tau_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/FD_genie_RHC_tau_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/FD_genie_RHC_tau_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/FD_genie_RHC_tau_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ND_cosmics.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: cry
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier cry
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_cry.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_cry.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_cry.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_cry.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_cry.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_cry.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_cry.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_cry.html
PMS  :     Making results page for tier: pchits
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pchits
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_pchits.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016rm: cannot remove `/nusoft/app/web/htdoc/nova/production/testing/R16-02-11-fiberbrightness.a_2016-06-14-19-45-02/embed/*.js': No such file or directory
rm: cannot remove `/nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/11047481.0@fifebatch2.fnal.gov/*': No such file or directory
Cannot find credentials to use. Try the following:

- If you have an FNAL kerberized account, run 'kinit'.
- Otherwise, if you have an FNAL services account, run the following cigetcert command and which 
 will  prompt for your services password, then resubmit your job:
'cigetcert -s fifebatch.fnal.gov -o /tmp/x509up_u42417'
- Otherwise, follow the instructions at //fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0010798 to obtain a services and/or kerberized account. 
-03-15-19-45-03/logs/ND_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_cosmics_fullchain_nolem.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ND_data_NuMI.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: raw2root
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier raw2root
vbokh:     parsing log
vbokh:     events found: 1952
vbokh:     events used: 20
vbokh:      - making array for memory
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for memory_percent
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for vms_memory
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for cpu_percent
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for u_cpu_times
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/file_sizes/ND_data_NuMI_raw2root_neardet_r00010402_s54_t00_raw_root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/file_sizes/ND_data_NuMI_raw2root_neardet_r00010402_s54_t00_raw_root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_raw2root.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_fullchain_nolem.html
PMS  :     Making results page for tier: recokeepup
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recokeepup
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_recokeepup.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_NuMI_recokeepup.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_recokeepup.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_NuMI_recokeepup.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_recokeepup.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_NuMI_recokeepup.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_recokeepup.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_NuMI_recokeepup.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ND_data_cosmics.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: raw2root
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier raw2root
vbokh:     parsing log
vbokh:     events found: 3599
vbokh:     events used: 20
vbokh:      - making array for memory
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for memory_percent
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for vms_memory
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for cpu_percent
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
vbokh:      - making array for u_cpu_times
vbokh:      - making figure
vbokh:      - adding events
vbokh:      - making html snippet
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_raw2root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/file_sizes/ND_data_cosmics_raw2root_neardet_r00010402_s54_t02_raw_root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/file_sizes/ND_data_cosmics_raw2root_neardet_r00010402_s54_t02_raw_root.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_raw2root.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_raw2root.html
PMS  :     Making results page for tier: pchits
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pchits
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_pchits.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_pchits.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_pchits.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_recopid.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_fullchain_nolem.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_data_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_data_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_data_cosmics_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_data_cosmics_caf.html
PMS  :     Making results page for file: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/ND_genie_FHC_nonswap.pkl
PMS  :      - unpickling
PMS  :      - unpickled
PMS  :     Making results page for tier: rock
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier rock
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_rock.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_rock.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_rock.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_rock.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_rock.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_rock.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_rock.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_rock.html
PMS  :     Making results page for tier: mix
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier mix
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_mix.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_mix.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_mix.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_mix.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_mix.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_mix.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_mix.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_mix.html
PMS  :     Making results page for tier: reco
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier reco
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_reco.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_reco.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_reco.html
PMS  :     Making results page for tier: pid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier pid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_pid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_pid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_pid.html
PMS  :     Making results page for tier: recopid
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier recopid
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_recopid.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_recopid.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_recopid.html
PMS  :     Making results page for tier: caf
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier caf
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_caf.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_caf.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_caf.html
PMS  :     Making results page for tier: fullchain_nolem
PMS  :      - making bokeh metrics
vbokh: --- Writing html for tier fullchain_nolem
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/metrics/ND_genie_FHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/logs/ND_genie_FHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/fcls/ND_genie_FHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_fullchain_nolem.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/timings/ND_genie_FHC_nonswap_fullchain_nolem.html
svd page file name =  testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/index.html
PM   :     Writing page: /nusoft/app/web/htdoc/nova/production/testing/R15-05-22-nuexsecana.a_2016-03-15-19-45-03/index.html
PMS  :     Making results page for folder: /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-fiberbrightness.a_2016-06-14-19-45-02
PMS  :     Making results page info: /nusoft/app/web/htdoc/nova/production/testing/R16-02-11-fiberbrightness.a_2016-06-14-19-45-02/info.pkl
PMS  : --- Making batch log pages
PMS  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh R16-02-11-fiberbrightness.a FD_cosmics 200 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/R16-02-11-fiberbrightness.a_2016-06-14-19-45-02/ maxopt
['Credentials not found. Try running kinit first.', '/fife/local/scratch/uploads/nova/novapro/2016-06-14_194510.983841_1931', '', '/fife/local/scratch/uploads/nova/novapro/2016-06-14_194510.983841_1931/batch-job.sh_20160614_194511_2914054_0_1_.cmd', '', 'submitting....', '', 'Submitting job(s).', '', '1 job(s) submitted to cluster 11047481.', '', 'JobsubJobId of first job: 11047481.0@fifebatch2.fnal.gov', '', 'Use job id 11047481.0@fifebatch2.fnal.gov to retrieve output', '']
Credentials not found. Try running kinit first.
/fife/local/scratch/uploads/nova/novapro/2016-06-14_194510.983841_1931

/fife/local/scratch/uploads/nova/novapro/2016-06-14_194510.983841_1931/batch-job.sh_20160614_194511_2914054_0_1_.cmd

submitting....

Submitting job(s).

1 job(s) submitted to cluster 11047481.

JobsubJobId of first job: 11047481.0@fifebatch2.fnal.gov

Use job id 11047481.0@fifebatch2.fnal.gov to retrieve output

PMS  :     jobsub client job id: 11047481.0@fifebatch2.fnal.gov
PMS  :     client log dir: /nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/11047481.0@fifebatch2.fnal.gov
PMS  :     - log directory exists
PMS  :     folder datetime: 2016-06-14 00:00:00, now: 2019-07-24 10:35:08.455525, days: 1135
PMS  :     re-fetching logs
PMS  :     - fetching logs
PMS  :     fetch log command to be run: jobsub_fetchlog -G nova --job=11047481.0@fifebatch2.fnal.gov --unzipdir=/nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/11047481.0@fifebatch2.fnal.gov --role=Production
Traceback (most recent call last):
  File "makeHTMLFrontPage.py", line 33, in <module>
    messages = PM.makeResultsPage(options)
  File "/nova/app/users/novapro/prod_valid/python/testing/web/PageMakers.py", line 313, in makeResultsPage
    statuses = makeBatchLogPages(info,options,folder)
  File "/nova/app/users/novapro/prod_valid/python/testing/web/PageMakers.py", line 1010, in makeBatchLogPages
    subprocess.check_call(fetch_command.split())
  File "/nova/app/users/novapro/prod_valid/software/anaconda/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['jobsub_fetchlog', '-G', 'nova', '--job=11047481.0@fifebatch2.fnal.gov', '--unzipdir=/nova/ana/users/novapro/prod_valid/condor_output/production_log_archive/11047481.0@fifebatch2.fnal.gov', '--role=Production']' returned non-zero exit status 1
--- clean up
rm: cannot remove `*.js': No such file or directory
--- Removing lock
--- done
--- End date: 
Wed Jul 24 10:35:08 CDT 2019
---