Classes | Functions | Variables
log_trawl Namespace Reference

Classes

class  FileViewer
 
class  Process
 
class  Project
 

Functions

def checkIsFile (path)
 
def setPermission (path, permission=PERMISSION_RW_R_R)
 
def tail (path, window=20)
 
def findLogInDirectory (logDir)
 
def retreiveLogFromJobSubClient (jobId)
 
def main (argv)
 

Variables

 PERMISSION_RW_R
 
 PERMISSION_RW_R_R
 
 states
 
 proj
 
 local
 
 banner
 

Function Documentation

def log_trawl.checkIsFile (   path)

Definition at line 14 of file log_trawl.py.

Referenced by log_trawl.FileViewer.__repr__(), log_trawl.Process.errLess(), log_trawl.Process.errPath(), log_trawl.Process.errTail(), log_trawl.Process.outLess(), log_trawl.Process.outPath(), log_trawl.Process.outTail(), and setPermission().

14 def checkIsFile(path):
15  if os.path.isfile(path):
16  return 1
17  print "Warning, not a file: ", path
18  return 0
19 
def checkIsFile(path)
Definition: log_trawl.py:14
def log_trawl.findLogInDirectory (   logDir)

Definition at line 343 of file log_trawl.py.

Referenced by retreiveLogFromJobSubClient().

343 def findLogInDirectory(logDir):
344  logDirContents = os.listdir(logDir)
345  for fileName in logDirContents:
346  if fileName.endswith(".log"):
347  return os.path.join(logDir,fileName)
348  return None
349 
350 
351 
def findLogInDirectory(logDir)
Definition: log_trawl.py:343
def log_trawl.main (   argv)

Definition at line 399 of file log_trawl.py.

References retreiveLogFromJobSubClient().

399 def main(argv):
400  parser = argparse.ArgumentParser(description="""
401  Loop over the ".log" file and find the final state of each
402  process, then find the .err and .out""")
403  parser.add_argument('logFile', help="""
404  The input file to run over, or a jobsub_client jobid.
405  In the jobid case, the files are fetched to CONDOR_TMP if it is defined,
406  or falls back to the current working directory if not defined. If the log
407  has already been fetcheded, it just finds it there.""", type=str)
408 
409  args = parser.parse_args()
410  logPath = args.logFile
411 
412  if not os.path.isfile(logPath) and "@" in logPath and ".fnal.gov" in logPath:
413  logPath = retreiveLogFromJobSubClient(logPath)
414 
415 
416  proj = Project(logPath)
417  return proj
418 
419 
420 
def retreiveLogFromJobSubClient(jobId)
Definition: log_trawl.py:352
def main(argv)
Definition: log_trawl.py:399
def log_trawl.retreiveLogFromJobSubClient (   jobId)

Definition at line 352 of file log_trawl.py.

References findLogInDirectory().

Referenced by main().

353  print "JobSub Client Job ID supplied, finding logs."
354  if "CONDOR_TMP" in os.environ and os.path.isdir(os.environ["CONDOR_TMP"]):
355  print "Found $CONDOR_TMP, logs will be stored there. "
356  fetchLocation = os.environ["CONDOR_TMP"]
357  else:
358  print "Failed to find $CONDOR_TMP, fetching logs to current directory."
359  fetchLocation = "./"
360 
361  logDir = os.path.join(fetchLocation, jobId)
362 
363  if not os.path.isdir(logDir):
364  os.mkdir(logDir)
365  print "Directory:", logDir
366  extantLogFile = findLogInDirectory(logDir)
367  if extantLogFile:
368  print "Log file previously fetched, location:", extantLogFile
369  return extantLogFile
370 
371  else:
372  print "Fetching log files..."
373  os.system("jobsub_fetchlog --jobid " + jobId + " --unzipdir " + logDir)
374 
375  fetchedLogFile = findLogInDirectory(logDir)
376  if fetchedLogFile:
377  print "Log file location:", fetchedLogFile
378  return fetchedLogFile
379  else:
380  raise Exception("Failed to filnd log file after fetching, " + \
381  "something went wrong.")
382 
def findLogInDirectory(logDir)
Definition: log_trawl.py:343
def retreiveLogFromJobSubClient(jobId)
Definition: log_trawl.py:352
cet::coded_exception< errors::ErrorCodes, ExceptionDetail::translate > Exception
Definition: Exception.h:66
def log_trawl.setPermission (   path,
  permission = PERMISSION_RW_R_R 
)

Definition at line 20 of file log_trawl.py.

References checkIsFile().

Referenced by log_trawl.Project.makeReadable().

20 def setPermission(path, permission=PERMISSION_RW_R_R):
21  if checkIsFile(path):
22  os.chmod(path, permission)
23 
def checkIsFile(path)
Definition: log_trawl.py:14
def setPermission(path, permission=PERMISSION_RW_R_R)
Definition: log_trawl.py:20
def log_trawl.tail (   path,
  window = 20 
)

Definition at line 24 of file log_trawl.py.

References confusionMatrixTree.count, and open().

Referenced by log_trawl.Process.errTail(), daqdataformats::RawMilliSlice.getMilliSliceIndex(), daqdataformats.getTailSize(), log_trawl.Process.outTail(), daqdataformats::VERSION_NAMESPACE::RawConfigurationBlock.print(), stan::mcmc::chains< RNG >.quantile(), stan::mcmc::chains< RNG >.quantiles(), and genie::TuneId.Tail().

24 def tail(path, window=20):
25  f = open(path, 'r')
26  BUFSIZ = 1024
27  f.seek(0, 2)
28  bytes = f.tell()
29  size = window
30  block = -1
31  data = []
32  while size > 0 and bytes > 0:
33  if (bytes - BUFSIZ > 0):
34  # Seek back one whole BUFSIZ
35  f.seek(block*BUFSIZ, 2)
36  # read BUFFER
37  data.append(f.read(BUFSIZ))
38  else:
39  # file too small, start from begining
40  f.seek(0,0)
41  # only read what was not read
42  data.append(f.read(bytes))
43  linesFound = data[-1].count('\n')
44  size -= linesFound
45  bytes -= BUFSIZ
46  block -= 1
47  return '\n'.join(''.join(data).splitlines()[-window:])
48 
49 
50 
procfile open("FD_BRL_v0.txt")
def tail(path, window=20)
Definition: log_trawl.py:24

Variable Documentation

log_trawl.banner

Definition at line 423 of file log_trawl.py.

log_trawl.local

Definition at line 423 of file log_trawl.py.

log_trawl.PERMISSION_RW_R

Definition at line 10 of file log_trawl.py.

log_trawl.PERMISSION_RW_R_R

Definition at line 12 of file log_trawl.py.

log_trawl.proj

Definition at line 422 of file log_trawl.py.

log_trawl.states

Definition at line 163 of file log_trawl.py.

Referenced by make_cosmic_eff_table().