--- running submit development tests
--- Start date: 
Wed Jul 24 05:10:01 CDT 2019
--- setup
user: novapro

--- Setup NOvA production validation
User:           novapro
Use with SRT:   false

Base directory: /nova/app/users/novapro/prod_valid/python
Grid output:    /nova/ana/users/novapro/prod_valid/condor_output
Web output:     /nusoft/app/web/htdoc/nova/production

--- novapro
--- run
rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.fcl': No such file or directory
rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.root': No such file or directory
rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.json': No such file or directory
rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.db': No such file or directory
rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/*.csv': No such file or directory
rm: cannot remove `/nova/app/users/novapro/prod_valid/python/testing/output/*': No such file or directory
tar: Removing leading `/' from member names
run  : --- Launch batch jobs
run  : --- Options
run  :     release:       development
run  :     dry run mode:  False
run  :     debug mode:    False
run  :     short mode:    True
run  :     message:       Nightly development test: 2019-07-24 05:10:02
run  :     verbose mode:  False
run  :     output directory will be: /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-24-05-10-02/
run  :     making output directory
run  :     tar command: tar czf /nova/app/users/novapro/prod_valid/python/../production_python.tgz /nova/app/users/novapro/prod_valid/python/
run  :     command: jobsub_submit -G nova --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC --expected-lifetime=3500s --OS=SL6 --memory=3GB --disk=10GB file:///nova/app/users/novapro/prod_valid/python/testing/scripts/batch-job.sh development FD_data_cosmics 200 /nova/ana/users/novapro/prod_valid/condor_output/production_testing/development_2019-07-24-05-10-02/ maxopt
Cannot find credentials to use. Try the following:

- If you have an FNAL kerberized account, run 'kinit'.
- Otherwise, if you have an FNAL services account, run the following cigetcert command and which
will  prompt for your services password, then resubmit your job:
'cigetcert -s fifebatch.fnal.gov -o /tmp/x509up_u42417'
- Otherwise, follow the instructions at //fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0010798 to obtain a services and/or kerberized account.
Traceback (most recent call last):
  File "/nova/app/users/novapro/prod_valid/python/testing/scripts/launch_batch_jobs.py", line 89, in <module>
    assert False, "run  : ERROR in submission"
AssertionError: run  : ERROR in submission
run  : --- Submit development tests
run  :     executing: python /nova/app/users/novapro/prod_valid/python/testing/scripts/launch_batch_jobs.py -s -r development -m "Nightly development test: 2019-07-24 05:10:02"
run  : --- done
--- done
--- End date: 
Wed Jul 24 05:10:41 CDT 2019
---