Public Member Functions | Public Attributes | Private Attributes | List of all members
DefinitionData.DefinitionData Class Reference

Public Member Functions

def __init__ (self, name)
 
def queryDate (self)
 
def sameData (self, other)
 
def fetchData (self, maxCacheLifetime)
 
def printData (self)
 

Public Attributes

 name
 
 description
 
 summary
 
 state
 
 queryTime
 
 queryTimeStamp
 

Private Attributes

 __dict__
 we might want to add a fallback in case dataset exists but query fails More...
 

Detailed Description

Definition at line 48 of file DefinitionData.py.

Constructor & Destructor Documentation

def DefinitionData.DefinitionData.__init__ (   self,
  name 
)

Definition at line 50 of file DefinitionData.py.

50  def __init__(self,name):
51  self.name=name
52  self.description = { "defname":name }
53  self.summary = { "file_count":0, "total_event_count":0, "total_file_size":0 }
54  self.state = "noData"
55  self.queryTime = 0
56  self.queryTimeStamp = 0
57 

Member Function Documentation

def DefinitionData.DefinitionData.fetchData (   self,
  maxCacheLifetime 
)

Definition at line 69 of file DefinitionData.py.

References DefinitionData.definitionPickleName(), Definition.Definition.name, Table.name, legacymc_extractor.LegacyMC.name, Chain.Chain.name, ChainData.ChainData.name, nova_fhicl_metadata.NovaFhclExtractor.name, ana::HistDef.name, Controller.Controller.name, PageMaker.PageMaker.name, nova_log_metadata.NovaLogExtractor.name, Component.Component.name, ana::MultiHistDef.name, Plot.name, HistDef.name, CalibrationType.name, nova_h5_metadata.NovaH5.name, CutContainer.name, plotConfusion.HistSet.name, CutDef.name, ana::NuTruthHistDef.name, bp_plot.Plot.name, GENIEWeightLabel.name, generic.views.Pages.SubdirInfo.name, nova_caf_metadata.NovaCAFExtractor.name, nova_text_metadata.NovaText.name, novaddt::sm_cleaner.name, drawEvents.HistSet.name, ColumnDef.name, novaddt::MapSource.name, PlotInfo.name, Weight.name, ana::VarDef.name, VarContainer.name, galleryMaker.FileNBE.name, DefinitionData.DefinitionData.name, ana::CutDef.name, FileUp.name, ana::PIDCutDef.name, nova::dbi::DataEpoch.name, ana::WeightDef.name, VarContainer2D.name, PolyDef.name, IntConfig.name, VarContainer3D.name, nova_art_metadata.NovaArt.name, nuebarccinc::SelDef.name, object.name, nuebarccinc::SelDefST.name, generic.models.PlotInfo.CanvasSummary.name, generic.models.PlotInfo.PlotComparisonStatistic.name, EventContainer.name, nuebarccinc::mHistAxisDef.name, nuebarccinc::mHistAxisSTDef.name, gen_flatrecord.TypeName.name, generic.models.Organizational.PlotID.name, ana::GenieMultiverseSpectra::knob_sampling_mode.name, TB_WatchdogFx.Tier.name, WatchdogFx.Tier.name, TB_WatchdogFx.Path.name, WatchdogFx.Path.name, ana::GenieMultiverseParameters::knob_sampling_mode.name, ana::nueccinc::SelDef.name, ana::nueccinc::SelDefST.name, ana::SelDef.name, ana::nueccinc::mHistAxisDef.name, ExtrapDef.name, ana::nueccinc::mHistAxisSTDef.name, ana::predictions.name, ana::mHistAxisDef.name, ana::mHistAxisSTDef.name, generic.models.Organizational.Folder.name, std_candles.PlotConfig.name, and open().

69  def fetchData(self,maxCacheLifetime):
70  ## we might want to add a fallback in case dataset exists but query
71  ## fails
72  print "Fetching data for", self.name
73  cacheName=definitionPickleName(self.name, useCache=True)
74  cacheData=None
75  if not os.path.isfile(cacheName):
76  pass #no cache so don't even worry about it
77  else:
78  cacheData=pickle.load(open(cacheName,'r'))
79  #if cache data is recent, just use it, just use it,
80  # and that's the end of the story, but if the cache
81  # data is old, don't rely on it. But we do need to keep
82  # the cache around fo comparison purposes
83  if (time() - cacheData.queryTimeStamp) < maxCacheLifetime:
84  self.__dict__ = cacheData.__dict__
85  self.state="cached"
86  self.queryTimeStamp = cacheData.queryTimeStamp
87  return
88 
89  try:
90  self.description = sam.descDefinitionDict(self.name)
91 
92  startTime=time()
93  #try:
94  # self.snapshotID = sam.takeSnapshot(self.name)
95  # snapshotQuery = "dataset_def_name_newest_snapshot %s" % self.name
96  # self.summary = sam.listFilesSummary(dimensions=snapshotQuery)
97  # self.state = "good"
98  #except:
99  # self.summary = sam.listFilesSummary(defname=self.name)
100  # self.state = "snapshot failed"
101  self.summary = sam.listFilesSummary(defname=self.name)
102  self.state = "good"
103 
104  self.queryTime = time() - startTime
105  if self.summary["file_count"] == 0:
106  self.summary["total_event_count"]=0
107  self.summary["total_file_size"]=0
108 
110  print "ERROR, definition %s not found" % self.name
111  self.state="bad definition name error"
113  print "ERROR, Timeout getting data for %s" % self.name
114  self.state="timed out"
116  print "ERROR, HTTP: ", e
117  self.state="http error"
118  except IOError:
119  print "IOError", e
120  self.state="proxy error"
121  except KeyboardInterrupt:
122  print "Stopping because you say so."
123  sys.exit(99)
124  except:
125  print "ERROR, unknown exception fetching data for %s" % self.name
126  self.state="unknown error"
127  # Store the time, even if there is an exception
128  self.queryTimeStamp = time()
129 
130  #so here we have cache data that is stale (else we would have exited the function
131  # already), so we know we've fetched the data from SAM. Check if that matches the cache.
132  # If not print a warning, delete cache, as it's clearly no good. If they do match, dump to
133  # the cache as well.
134 
135  if cacheData != None:
136  if not self.sameData(cacheData):
137  os.remove(cacheName)
138  sys.stderr.write("Warning:: cache data and current data do not match for %s\n" %self.name)
139  else:
140  dumpDefinitionData(self,useCache=True)
141 
__dict__
we might want to add a fallback in case dataset exists but query fails
def definitionPickleName(defname, useCache=False)
def dumpDefinitionData(defData, useCache=False)
procfile open("FD_BRL_v0.txt")
def fetchData(self, maxCacheLifetime)
def DefinitionData.DefinitionData.printData (   self)

Definition at line 142 of file DefinitionData.py.

References ChainData.ChainData.description, DefinitionData.DefinitionData.description, DefinitionData.DefinitionData.summary, and fnex::CompareFNEXCAFEventListFiles.summary.

142  def printData(self):
143  for key in self.description.keys():
144  print key, ":", self.description[key]
145  for key in self.summary.keys():
146  print key, ":", self.summary[key]
147 
148 
def DefinitionData.DefinitionData.queryDate (   self)

Definition at line 58 of file DefinitionData.py.

References DefinitionData.DefinitionData.queryTimeStamp.

58  def queryDate(self):
59  return ctime(self.queryTimeStamp)
60 
def DefinitionData.DefinitionData.sameData (   self,
  other 
)

Definition at line 61 of file DefinitionData.py.

References ChainData.ChainData.description, DefinitionData.DefinitionData.description, DefinitionData.DefinitionData.summary, and fnex::CompareFNEXCAFEventListFiles.summary.

61  def sameData(self,other):
62 
63  if self.description != other.description:
64  return False
65 
66  if self.summary != other.summary:
67  return False
68 

Member Data Documentation

DefinitionData.DefinitionData.__dict__
private

we might want to add a fallback in case dataset exists but query fails

Definition at line 84 of file DefinitionData.py.

Referenced by site_stats_from_log.SiteStats.__repr__(), and samweb_client._exceptions.SAMWebHTTPError.__str__().

DefinitionData.DefinitionData.description
DefinitionData.DefinitionData.queryTime

Definition at line 55 of file DefinitionData.py.

DefinitionData.DefinitionData.queryTimeStamp

Definition at line 56 of file DefinitionData.py.

Referenced by DefinitionData.DefinitionData.queryDate().

DefinitionData.DefinitionData.state

Definition at line 54 of file DefinitionData.py.

DefinitionData.DefinitionData.summary

The documentation for this class was generated from the following file: