--- running submit development tests --- Start date: Wed Jul 24 05:10:01 CDT 2019 --- setup user: novapro --- Setup NOvA production validation User: [01;34mnovapro[0m Use with SRT: [01;34mfalse[0m Base directory: [01;34m/nova/app/users/novapro/prod_valid/python[0m Grid output: [01;34m/nova/ana/users/novapro/prod_valid/condor_output[0m Web output: [01;34m/nusoft/app/web/htdoc/nova/production[0m --- novapro --- run rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.fcl': No such file or directory rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.root': No such file or directory rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.json': No such file or directory rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.db': No such file or directory rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.csv': No such file or directory rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/output/*': No such file or directory tar: Removing leading `/' from member names run : --- Launch batch jobs run : --- Options run : release: development run : dry run mode: False run : debug mode: False run : short mode: True run : message: Nightly development test: 2019-07-24 05:10:02 run : verbose mode: False run : output directory will be: /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-24-05-10-02/ run : making output directory run : tar command: tar czf /nova/app/users/novapro/prod_valid/python/../production_python.tgz /nova/app/users/novapro/prod_valid/python/ run : command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh development FD_data_cosmics 200 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-24-05-10-02/ maxopt Cannot find credentials to use. Try the following: - If you have an FNAL kerberized account, run 'kinit'. - Otherwise, if you have an FNAL services account, run the following cigetcert command and which will prompt for your services password, then resubmit your job: 'cigetcert -s fifebatch.fnal.gov -o /tmp/x509up_u42417' - Otherwise, follow the instructions at //fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0010798 to obtain a services and/or kerberized account. Traceback (most recent call last): File "/nova/app/users/novapro/prod_valid/python/testing/scripts/launch_batch_jobs.py", line 89, in <module> assert False, "run : ERROR in submission" AssertionError: run : ERROR in submission run : --- Submit development tests run : executing: python /nova/app/users/novapro/prod_valid/python/testing/scripts/launch_batch_jobs.py -s -r development -m "Nightly development test: 2019-07-24 05:10:02" run : --- done --- done --- End date: Wed Jul 24 05:10:41 CDT 2019 ---