Public Types | Public Member Functions | Protected Member Functions | Protected Attributes | List of all members
ana::FileReducer Class Reference

Create smaller CAFs. More...

#include "/cvmfs/nova-development.opensciencegrid.org/novasoft/releases/N21-01-24/CAFAna/Core/FileReducer.h"

Inheritance diagram for ana::FileReducer:
ana::SpectrumLoaderBase

Public Types

typedef voidReductionFunc) (caf::StandardRecord *)
 
typedef voidReductionFuncWithProxy) (caf::StandardRecord *, const caf::SRProxy *)
 
typedef voidSpillReductionFunc) (caf::SRSpill *, const caf::SRSpillProxy *)
 
typedef voidNuTruthReductionFunc) (caf::SRSpillTruthBranch *, const caf::SRNeutrinoProxy *)
 

Public Member Functions

 FileReducer (const std::string &wildcard, const std::string &outfile, const bool copyNuTree=false)
 
 FileReducer (const std::vector< std::string > &fnames, const std::string &outfile, const bool copyNuTree=false)
 
virtual ~FileReducer ()
 
void AddEventCut (const Cut &cut)
 Only copy records to the output file if they pass this cut. More...
 
void AddNuTruthCut (const NuTruthCut &cut)
 Only copy nuTree neutrinos to the output file if they pass this cut. More...
 
void SetEventList (const std::string &fname)
 If called, only events whose run/subrun/event occur in fname will be retained. More...
 
void setOmitBadFiles (bool omit)
 Set an optional parameter to omit bad files (zombie or no POT) More...
 
void AddReductionStep (const std::function< ReductionFunc > &f)
 Run the specified reduction function over each event. More...
 
void AddReductionStep (const std::function< ReductionFuncWithProxy > &f)
 
void AddSpillReductionStep (const std::function< SpillReductionFunc > &f)
 Run the specified reduction function over each spill. More...
 
void AddNuTruthReductionStep (const std::function< NuTruthReductionFunc > &f)
 Run the specified reduction function over each spill. More...
 
void SetMetadata (const std::string &key, const std::string &val)
 Cause the output file to be systematically shifted. More...
 
void CopyMetadata (bool copy=true)
 
virtual void Go () override
 Load all the registered spectra. More...
 
virtual void AccumulateExposures (const caf::SRSpillProxy *spill) override
 

Protected Member Functions

void UpdateMetadata (std::map< std::string, std::string > &meta, const std::set< std::string > &mask, const std::vector< std::string > &fnames) const
 
void SetSpillCut (const SpillCut &cut)
 
virtual void AddSpectrum (Spectrum &spect, const Var &var, const Cut &cut, const SystShifts &shift, const Var &wei=kUnweighted)
 For use by the Spectrum constructor. More...
 
virtual void AddSpectrum (Spectrum &spect, const MultiVar &var, const Cut &cut, const SystShifts &shift, const Var &wei=kUnweighted)
 For use by the Spectrum constructor. More...
 
virtual void AddSpectrum (Spectrum &spect, const NuTruthVar &var, const NuTruthCut &cut, const SystShifts &shift, const NuTruthVar &wei=kNuTruthUnweighted)
 For use by the Spectrum constructor. More...
 
virtual void AddSpectrum (Spectrum &spect, const NuTruthMultiVar &var, const NuTruthCut &cut, const SystShifts &shift, const NuTruthVar &wei=kNuTruthUnweighted)
 
virtual void AddReweightableSpectrum (ReweightableSpectrum &spect, const Var &xvar, const Var &yvar, const Cut &cut, const SystShifts &shift, const Var &wei)
 For use by the constructors of ReweightableSpectrum subclasses. More...
 
virtual void AddSpillHistogram (TH1 *h, const SpillVar &var, const SpillCut &cut, const SpillVar &wei=kSpillUnweighted)
 Uses include counting the total POT or spills in a run. More...
 
virtual bool Gone () const
 Indicate whether or not Go has been called. More...
 
double GetWeightedPOT ()
 
IFileSourceWildcardOrSAMQuery (const std::string &str) const
 Figure out if str is a wildcard or SAM query and return a source. More...
 
int NFiles () const
 Forwards to fFileSource. More...
 
TFile * GetNextFile ()
 Forwards to fFileSource but also accumulates POT and livetime. More...
 
WeightApplierGetSinkSCW (const SystShifts &shift, const Cut &cut, const Var &wei)
 

Protected Attributes

std::string fOutfile
 
CutfEventCut
 
NuTruthCutfNuTruthCut
 
std::set< std::tuple< int, int, int > > fEventList
 
bool fOmitBadFiles
 
std::vector< std::function< ReductionFunc > > fReductionFuncs
 
std::vector< std::function< ReductionFuncWithProxy > > fReductionFuncsWithProxy
 
std::vector< std::function< SpillReductionFunc > > fSpillReductionFuncs
 
std::vector< std::function< NuTruthReductionFunc > > fNuTruthReductionFuncs
 
std::map< std::string, std::stringfMetaMap
 
bool fCopyMetadata
 
bool fCopyNuTree
 
std::string fWildcard
 
std::unique_ptr< IFileSourcefFileSource
 
bool fGone
 Has Go() been called? Can't add more histograms after that. More...
 
double fPOT
 Accumulated by calls to GetNextFile. More...
 
double fRunPOT
 Crude measure, not including spill cuts. More...
 
double fSinglePOT
 
double fBeamNue
 
double fSingleNue
 
double fWeightedPOT
 
std::set< IRecordSink * > fSpectrumSinks
 
std::unordered_map< std::tuple< int, int, int >, WeightApplier * > fSinksSCW
 
std::unordered_map< std::pair< int, int >, CutApplier * > fSinksSC
 
std::unordered_map< int, SystApplier * > fSinksS
 
RecordMultiSink fSink
 
ExposureSource fExposureSource
 
std::list< SpillHistDeffSpillHistDefs
 
std::list< NuHistDeffNuHistDefs
 
std::unique_ptr< SpillCutfSpillCut
 Cut applied to the spill branch for every event. More...
 

Detailed Description

Create smaller CAFs.

This class can do two seperate things. It can cause non-selected variables to be NaN in the output file, causing it to compress very well. It can also cut out slices that fail a cut.

Definition at line 56 of file FileReducer.h.

Member Typedef Documentation

typedef void( ana::FileReducer::NuTruthReductionFunc) (caf::SRSpillTruthBranch *, const caf::SRNeutrinoProxy *)

Definition at line 92 of file FileReducer.h.

typedef void( ana::FileReducer::ReductionFunc) (caf::StandardRecord *)

Definition at line 81 of file FileReducer.h.

typedef void( ana::FileReducer::ReductionFuncWithProxy) (caf::StandardRecord *, const caf::SRProxy *)

Definition at line 82 of file FileReducer.h.

typedef void( ana::FileReducer::SpillReductionFunc) (caf::SRSpill *, const caf::SRSpillProxy *)

Definition at line 88 of file FileReducer.h.

Constructor & Destructor Documentation

ana::FileReducer::FileReducer ( const std::string wildcard,
const std::string outfile,
const bool  copyNuTree = false 
)

Definition at line 333 of file FileReducer.cxx.

337  fOutfile(outfile),
338  fEventCut(nullptr),
339  fNuTruthCut(nullptr),
340  fOmitBadFiles(false),
341  fCopyMetadata(true),
342  fCopyNuTree(copyNuTree)
343  {
344  }
std::string fOutfile
Definition: FileReducer.h:119
NuTruthCut * fNuTruthCut
Definition: FileReducer.h:121
std::vector< std::string > wildcard(const std::string &wildcardString)
Definition: convert.C:9
SpectrumLoaderBase()
Component of other constructors.
FILE * outfile
Definition: dump_event.C:13
ana::FileReducer::FileReducer ( const std::vector< std::string > &  fnames,
const std::string outfile,
const bool  copyNuTree = false 
)

Definition at line 347 of file FileReducer.cxx.

350  : SpectrumLoaderBase(fnames),
351  fOutfile(outfile),
352  fEventCut(nullptr),
353  fNuTruthCut(nullptr),
354  fOmitBadFiles(false),
355  fCopyMetadata(true),
356  fCopyNuTree(copyNuTree)
357  {
358  }
std::string fOutfile
Definition: FileReducer.h:119
NuTruthCut * fNuTruthCut
Definition: FileReducer.h:121
SpectrumLoaderBase()
Component of other constructors.
FILE * outfile
Definition: dump_event.C:13
ana::FileReducer::~FileReducer ( )
virtual

Definition at line 361 of file FileReducer.cxx.

References fEventCut, and fNuTruthCut.

362  {
363  if(fEventCut) delete fEventCut;
364  if(fNuTruthCut) delete fNuTruthCut;
365  }
NuTruthCut * fNuTruthCut
Definition: FileReducer.h:121

Member Function Documentation

virtual void ana::FileReducer::AccumulateExposures ( const caf::SRSpillProxy spill)
inlineoverridevirtual

Implements ana::SpectrumLoaderBase.

Definition at line 112 of file FileReducer.h.

References lem_server::mask.

112 {};
void ana::FileReducer::AddEventCut ( const Cut cut)

Only copy records to the output file if they pass this cut.

Definition at line 368 of file FileReducer.cxx.

References cut, and fEventCut.

Referenced by concat_fd_events(), generate_fd_fake_events(), make_numu_decaf(), reduce_bendecomp2017(), reduce_nue_2018(), reduce_nue_fa(), reduce_nue_fd_fa(), reduce_nue_nd_fa(), reduce_nue_or_numu_fa(), reduce_nue_or_numu_or_nus(), reduce_nue_or_numu_sa(), reduce_nue_sa(), reduce_numu_ana2017(), reduce_numu_ana2017_prod(), reduce_numu_ana2018(), reduce_numu_ana2018_cosmics(), reduce_numuccinc(), reduce_nus(), reduce_nus_ana2019(), reduce_nus_ana2020(), reduce_nus_or_numu(), reduce_prod5_nue(), reduce_prod5_numu(), super_reduce(), and train_reduce().

369  {
370  if(fEventCut){
371  *fEventCut = *fEventCut && cut;
372  }
373  else{
374  fEventCut = new Cut(cut);
375  }
376  }
_Cut< caf::SRProxy > Cut
Representation of a cut (selection) to be applied to a caf::StandardRecord object.
Definition: Cut.h:96
const Cut cut
Definition: exporter_fd.C:30
void ana::FileReducer::AddNuTruthCut ( const NuTruthCut cut)

Only copy nuTree neutrinos to the output file if they pass this cut.

Definition at line 379 of file FileReducer.cxx.

References cut, and fNuTruthCut.

Referenced by reduce_numuccinc().

380  {
381  if(fNuTruthCut){
382  *fNuTruthCut = *fNuTruthCut && cut;
383  }
384  else{
385  fNuTruthCut = new NuTruthCut(cut);
386  }
387  }
NuTruthCut * fNuTruthCut
Definition: FileReducer.h:121
_Cut< caf::SRNeutrinoProxy > NuTruthCut
Cut designed to be used over the nuTree, ie all neutrinos, not just those that got slices...
Definition: Cut.h:104
const Cut cut
Definition: exporter_fd.C:30
void ana::FileReducer::AddNuTruthReductionStep ( const std::function< NuTruthReductionFunc > &  f)
inline

Run the specified reduction function over each spill.

Definition at line 94 of file FileReducer.h.

94 {fNuTruthReductionFuncs.push_back(f);}
std::vector< std::function< NuTruthReductionFunc > > fNuTruthReductionFuncs
Definition: FileReducer.h:127
void ana::FileReducer::AddReductionStep ( const std::function< ReductionFunc > &  f)
inline
void ana::FileReducer::AddReductionStep ( const std::function< ReductionFuncWithProxy > &  f)
inline

Definition at line 86 of file FileReducer.h.

86 {fReductionFuncsWithProxy.push_back(f);}
std::vector< std::function< ReductionFuncWithProxy > > fReductionFuncsWithProxy
Definition: FileReducer.h:125
void ana::SpectrumLoaderBase::AddReweightableSpectrum ( ReweightableSpectrum spect,
const Var xvar,
const Var yvar,
const Cut cut,
const SystShifts shift,
const Var wei 
)
virtualinherited

For use by the constructors of ReweightableSpectrum subclasses.

Reimplemented in ana::NullLoader.

Definition at line 199 of file SpectrumLoaderBase.cxx.

References ana::RecordMultiSink::AddSink(), ana::ExposureSource::AddSink(), om::cerr, allTimeWatchdog::endl, ana::SpectrumLoaderBase::fExposureSource, ana::SpectrumLoaderBase::fGone, ana::SpectrumLoaderBase::fSpectrumSinks, and ana::SpectrumLoaderBase::GetSinkSCW().

Referenced by ana::FluxReweight::FluxReweight(), ana::NoReweight::NoReweight(), and ana::RecoReweight::RecoReweight().

205  {
206  if(fGone){
207  std::cerr << "Error: can't add Spectra after the call to Go()" << std::endl;
208  abort();
209  }
210 
211  ReweightableSpectrumSink* sink = new ReweightableSpectrumSink(xvar, yvar, &spect);
212  fSpectrumSinks.insert(sink);
213  GetSinkSCW(shift, cut, wei)->AddSink(sink);
214  fExposureSource.AddSink(cut, sink);
215  }
bool fGone
Has Go() been called? Can&#39;t add more histograms after that.
OStream cerr
Definition: OStream.cxx:7
void AddSink(const Cut &cut, IExposureSink *s)
std::set< IRecordSink * > fSpectrumSinks
void AddSink(IRecordSink *s)
WeightApplier * GetSinkSCW(const SystShifts &shift, const Cut &cut, const Var &wei)
const Cut cut
Definition: exporter_fd.C:30
void ana::SpectrumLoaderBase::AddSpectrum ( Spectrum spect,
const Var var,
const Cut cut,
const SystShifts shift,
const Var wei = kUnweighted 
)
virtualinherited

For use by the Spectrum constructor.

Reimplemented in ana::NullLoader, and ana::SpectrumLoaderNuTreeSpillCuts.

Definition at line 141 of file SpectrumLoaderBase.cxx.

References ana::RecordMultiSink::AddSink(), ana::ExposureSource::AddSink(), om::cerr, allTimeWatchdog::endl, ana::SpectrumLoaderBase::fExposureSource, ana::SpectrumLoaderBase::fGone, ana::SpectrumLoaderBase::fSpectrumSinks, and ana::SpectrumLoaderBase::GetSinkSCW().

Referenced by ana::SpectrumLoaderNuTreeSpillCuts::AddSpectrum(), and ana::TrivialExtrap::TrivialExtrap().

146  {
147  if(fGone){
148  std::cerr << "Error: can't add Spectra after the call to Go()" << std::endl;
149  abort();
150  }
151 
152  SpectrumSink* sink = new SpectrumSink(var, &spect);
153  fSpectrumSinks.insert(sink);
154  GetSinkSCW(shift, cut, wei)->AddSink(sink);
155  fExposureSource.AddSink(cut, sink);
156  }
bool fGone
Has Go() been called? Can&#39;t add more histograms after that.
OStream cerr
Definition: OStream.cxx:7
void AddSink(const Cut &cut, IExposureSink *s)
std::set< IRecordSink * > fSpectrumSinks
void AddSink(IRecordSink *s)
WeightApplier * GetSinkSCW(const SystShifts &shift, const Cut &cut, const Var &wei)
const Cut cut
Definition: exporter_fd.C:30
void ana::SpectrumLoaderBase::AddSpectrum ( Spectrum spect,
const MultiVar var,
const Cut cut,
const SystShifts shift,
const Var wei = kUnweighted 
)
virtualinherited

For use by the Spectrum constructor.

Reimplemented in ana::NullLoader, and ana::SpectrumLoaderNuTreeSpillCuts.

Definition at line 159 of file SpectrumLoaderBase.cxx.

References ana::RecordMultiSink::AddSink(), ana::ExposureSource::AddSink(), om::cerr, allTimeWatchdog::endl, ana::SpectrumLoaderBase::fExposureSource, ana::SpectrumLoaderBase::fGone, ana::SpectrumLoaderBase::fSpectrumSinks, and ana::SpectrumLoaderBase::GetSinkSCW().

164  {
165  if(fGone){
166  std::cerr << "Error: can't add Spectra after the call to Go()" << std::endl;
167  abort();
168  }
169 
170  MultiVarSpectrumSink* sink = new MultiVarSpectrumSink(var, &spect);
171  fSpectrumSinks.insert(sink);
172  GetSinkSCW(shift, cut, wei)->AddSink(sink);
173  fExposureSource.AddSink(cut, sink);
174  }
bool fGone
Has Go() been called? Can&#39;t add more histograms after that.
OStream cerr
Definition: OStream.cxx:7
void AddSink(const Cut &cut, IExposureSink *s)
std::set< IRecordSink * > fSpectrumSinks
void AddSink(IRecordSink *s)
WeightApplier * GetSinkSCW(const SystShifts &shift, const Cut &cut, const Var &wei)
const Cut cut
Definition: exporter_fd.C:30
void ana::SpectrumLoaderBase::AddSpectrum ( Spectrum spect,
const NuTruthVar var,
const NuTruthCut cut,
const SystShifts shift,
const NuTruthVar wei = kNuTruthUnweighted 
)
virtualinherited

For use by the Spectrum constructor.

Reimplemented in ana::NullLoader, and ana::SpectrumLoaderNuTreeSpillCuts.

Definition at line 177 of file SpectrumLoaderBase.cxx.

References ana::SpectrumLoaderBase::fNuHistDefs.

182  {
183  // TODO - implement with the same source/sink deal as regular spectra
184  fNuHistDefs.emplace_back(spect, var, cut, shift, wei);
185  }
const Cut cut
Definition: exporter_fd.C:30
std::list< NuHistDef > fNuHistDefs
void ana::SpectrumLoaderBase::AddSpectrum ( Spectrum spect,
const NuTruthMultiVar var,
const NuTruthCut cut,
const SystShifts shift,
const NuTruthVar wei = kNuTruthUnweighted 
)
virtualinherited

Reimplemented in ana::NullLoader, and ana::SpectrumLoaderNuTreeSpillCuts.

Definition at line 188 of file SpectrumLoaderBase.cxx.

References ana::SpectrumLoaderBase::fNuHistDefs.

193  {
194  // TODO - implement with the same source/sink deal as regular spectra
195  fNuHistDefs.emplace_back(spect, multivar, cut, shift, wei);
196  }
const Cut cut
Definition: exporter_fd.C:30
std::list< NuHistDef > fNuHistDefs
void ana::SpectrumLoaderBase::AddSpillHistogram ( TH1 *  h,
const SpillVar var,
const SpillCut cut,
const SpillVar wei = kSpillUnweighted 
)
virtualinherited
void ana::FileReducer::AddSpillReductionStep ( const std::function< SpillReductionFunc > &  f)
inline

Run the specified reduction function over each spill.

Definition at line 90 of file FileReducer.h.

Referenced by reduce_bendecomp2017(), reduce_nue_or_numu_or_nus(), reduce_nue_or_numu_sa(), reduce_nue_sa(), reduce_nus(), reduce_nus_ana2019(), reduce_nus_ana2020(), and reduce_nus_or_numu().

90 {fSpillReductionFuncs.push_back(f);}
std::vector< std::function< SpillReductionFunc > > fSpillReductionFuncs
Definition: FileReducer.h:126
void ana::FileReducer::CopyMetadata ( bool  copy = true)
inline

Definition at line 107 of file FileReducer.h.

107 {fCopyMetadata = copy;};
TFile * ana::SpectrumLoaderBase::GetNextFile ( )
protectedinherited

Forwards to fFileSource but also accumulates POT and livetime.

Definition at line 234 of file SpectrumLoaderBase.cxx.

References ana::SpectrumLoaderBase::AccumulateExposures(), ana::assert(), caf::SRProxySystController::BeginTransaction(), om::cout, allTimeWatchdog::endl, MakeMiniprodValidationCuts::f, ana::SpectrumLoaderBase::fBeamNue, ana::SpectrumLoaderBase::fFileSource, ana::SpectrumLoaderBase::fNuHistDefs, genie::utils::style::Format(), ana::SpectrumLoaderBase::fPOT, ana::SpectrumLoaderBase::fRunPOT, ana::SpectrumLoaderBase::fSingleNue, ana::SpectrumLoaderBase::fSinglePOT, ana::SpectrumLoaderBase::fSpillCut, ana::SpectrumLoaderBase::fSpillHistDefs, ana::SpectrumLoaderBase::fWildcard, caf::GetCAFType(), hd, findDuplicateFiles::key, caf::kFlatMultiTree, caf::kNested, getGoodRuns4SAM::n, cacheDefinitionData::prog, caf::SRProxySystController::Rollback(), ana::Progress::SetProgress(), caf_analysis::spill, caf::Proxy< caf::SRSpill >::spillpot, string, and febshutoff_auto::val.

Referenced by ana::SpectrumLoader::Go(), and Go().

235  {
236  TFile* f = fFileSource->GetNextFile();
237  if(!f) return 0; // out of files
238 
239  if(f->IsZombie()){
240  std::cout << "Bad file (zombie): " << f->GetName() << std::endl;
241  abort();
242  }
243 
244  // Crude way to count POT, at a file level without referring to spill cuts
245  TH1* hPOT = (TH1*)f->Get("TotalPOT");
246 
247  if(!hPOT){
248  std::cout << "Bad file (no POT tree): " << f->GetName() << std::endl;
249  abort();
250  }
251 
252  fRunPOT += hPOT->Integral();
253 
254  TKey *key = f->FindKey("TotalSinglePOT");
255 
256  // Initialise and only fill these histos if they exist
257  // We know they exist for nue overlay files
258  TH1* hSinglePOT = 0;
259  TH1* hTotalTrueSingleNue = 0;
260  TH1* hTotalTrueNonswapNue = 0;
261  if(key != 0){
262  hSinglePOT = (TH1*)f->Get("TotalSinglePOT");
263  assert(hSinglePOT);
264  fSinglePOT = hSinglePOT->Integral();
265 
266  hTotalTrueNonswapNue = (TH1*)f->Get("TotalTrueNonswapNue");
267  assert(hTotalTrueNonswapNue);
268  fBeamNue = hTotalTrueNonswapNue->Integral();
269 
270  hTotalTrueSingleNue = (TH1*)f->Get("TotalTrueSingleNue");
271  assert(hTotalTrueSingleNue);
272  fSingleNue = hTotalTrueSingleNue->Integral();
273  }
274  else{
275  delete hSinglePOT;
276  delete hTotalTrueSingleNue;
277  delete hTotalTrueNonswapNue;
278  }
279 
280  TDirectory* spillDir = 0;
281  TTree* spillTree = 0;
282 
283  TObject* obj = f->Get("spillTree");
284  assert(obj); // Must exist in one form or the other
285 
286  // It might seem like you could use GetObject() to do the type-checking
287  // directly, but that method seems to have a memory leak in the case that
288  // the types don't match.
289  if(obj->ClassName() == std::string("TTree")){
290  spillTree = (TTree*)obj;
291  }
292  else{
293  spillDir = (TDirectory*)obj;
294  spillTree = (TTree*)spillDir->Get("spill");
295  assert(spillTree);
296  }
297 
298  const caf::CAFType type = caf::GetCAFType(spillDir, spillTree);
299 
300  long n = 0;
301  caf::SRSpillProxy spill(spillDir, spillTree, "spill", n, 0);
302 
303  // FloatingExceptionOnNaN fpnan;
304 
305  // With multiple files exposure counting alternates with histogram filling,
306  // but with one file they deserve to be reported as two separate steps.
307  Progress* prog = 0;
308  if(fFileSource->NFiles() == 1)
309  prog = new Progress(TString::Format("Counting exposure from 1 file matching '%s'", fWildcard.c_str()).Data());
310 
311  const int Nentries = spillTree->GetEntries();
312 
313  double totFilePOT = 0;
314 
315  for(n = 0; n < Nentries; ++n){
316  if(type != caf::kFlatMultiTree) spillTree->LoadTree(n); // for all single-tree modes
317 
318  if(!fSpillCut || (*fSpillCut)(&spill)){
320 
321  totFilePOT += spill.spillpot;
322 
323  for(const SpillHistDef& hd: fSpillHistDefs){
324  // No cut, or cut passes
325  if(hd.cut(&spill)){
326  const double wei = hd.wei(&spill);
327  if(wei){
328  const double val = hd.var(&spill);
329  hd.h->Fill(val, wei);
330  }
331  } // end if(cut passes)
332  } // end for hd
333  }
334 
335  if(prog && n%100 == 0) prog->SetProgress(double(n)/Nentries);
336  } // end for n
337 
338  // The POT we want to use is either the singles POT in case of overlay files
339  // i.e the singlepot histo exists, or the standard after spill cuts
340  // Note: must be sure to run spill cuts for the overlays otherwise this gets
341  // reset to the fRunPOT
342  if(key != 0) fPOT += fSinglePOT;
343  else fPOT += totFilePOT;
344 
345  delete prog;
346  prog = 0;
347 
348  if(!fNuHistDefs.empty()){
349  TDirectory* nuDir = 0;
350  TTree* nuTree = 0;
351 
352  TObject* obj = f->Get("nuTree");
353  assert(obj); // Must exist in one form or the other
354 
355  // It might seem like you could use GetObject() to do the type-checking
356  // directly, but that method seems to have a memory leak in the case that
357  // the types don't match.
358  if(obj->ClassName() == std::string("TTree")){
359  nuTree = (TTree*)obj;
360  }
361  else{
362  nuDir = (TDirectory*)obj;
363  nuTree = (TTree*)spillDir->Get("nu");
364  assert(nuTree);
365  }
366 
367  const caf::CAFType type = caf::GetCAFType(nuDir, nuTree);
368 
369  const int NNuEntries = nuTree->GetEntries();
370 
371  long n = 0;
372  caf::SRNeutrinoProxy nu(nuDir, nuTree, (type == caf::kNested) ? "mc.nu" : "nu", n, 0);
373 
374  if(fFileSource->NFiles() == 1)
375  prog = new Progress(TString::Format("Filling from nuTree from 1 file matching '%s'", fWildcard.c_str()).Data());
376 
377 
378  for(n = 0; n < NNuEntries; ++n){
379  if(type != caf::kFlatMultiTree) nuTree->LoadTree(n); // for all single-tree modes
380 
381  for(const NuHistDef& hd: fNuHistDefs){
383 
384  double systWeight = 1;
385  // Can special-case nominal to not pay cost of Shift()
386  if(!hd.shift.IsNominal()){
387  hd.shift.Shift(&nu, systWeight);
388  }
389 
390  if(hd.cut(&nu)){
391  const double wei = hd.wei(&nu)*systWeight;
392  if(wei){
393  if(hd.var){
394  const double val = (*hd.var)(&nu);
395  hd.spect.Fill(val, wei);
396  }
397  else{
398  for(double val: (*hd.multivar)(&nu)) hd.spect.Fill(val, wei);
399  }
400  }
401  }
402 
403  // Return SRNeutrino to its unshifted form ready for the next
404  // histogram.
406  } // end for hd
407 
408  if(prog && n%10000 == 0) prog->SetProgress(double(n)/NNuEntries);
409  } // end for n
410 
411  for(const NuHistDef& hd: fNuHistDefs) hd.spect.fPOT += totFilePOT;
412 
413  delete prog;
414  } // end if NuHistDefs
415 
416  return f;
417  }
Proxy for caf::SRNeutrino.
Definition: SRProxy.h:510
double fRunPOT
Crude measure, not including spill cuts.
CAFType GetCAFType(const TDirectory *dir, TTree *tr)
std::unique_ptr< SpillCut > fSpillCut
Cut applied to the spill branch for every event.
virtual void AccumulateExposures(const caf::SRSpillProxy *spill)=0
std::list< SpillHistDef > fSpillHistDefs
Proxy for caf::SRSpill.
Definition: SRProxy.h:1346
OStream cout
Definition: OStream.cxx:6
std::unique_ptr< IFileSource > fFileSource
assert(nhit_max >=nhit_nbins)
void Format(TGraph *gr, int lcol, int lsty, int lwid, int mcol, int msty, double msiz)
Definition: Style.cxx:154
double fPOT
Accumulated by calls to GetNextFile.
std::list< NuHistDef > fNuHistDefs
TH1F * hd
Definition: Xdiff_gwt.C:57
enum BeamMode string
WeightApplier * ana::SpectrumLoaderBase::GetSinkSCW ( const SystShifts shift,
const Cut cut,
const Var wei 
)
protectedinherited

Definition at line 110 of file SpectrumLoaderBase.cxx.

References ana::RecordMultiSink::AddSink(), PandAna.Demos.tute_pid_validation::cid, ana::SpectrumLoaderBase::fSink, ana::SpectrumLoaderBase::fSinksS, ana::SpectrumLoaderBase::fSinksSC, ana::SpectrumLoaderBase::fSinksSCW, ana::SystShifts::ID(), ana::_Var< T >::ID(), and ana::_Cut< T >::ID().

Referenced by ana::SpectrumLoaderBase::AddReweightableSpectrum(), and ana::SpectrumLoaderBase::AddSpectrum().

113  {
114  const int sid = shift.ID();
115  const int cid = cut.ID();
116  const int wid = wei.ID();
117 
118  const std::pair<int, int> sc(sid, cid);
119  const std::tuple<int, int, int> scw(sid, cid, wid);
120 
121  if(fSinksS.find(sid) == fSinksS.end()){
122  SystApplier* sa = new SystApplier(shift);
123  fSinksS[sid] = sa;
124  fSink.AddSink(sa);
125  }
126  if(fSinksSC.find(sc) == fSinksSC.end()){
127  CutApplier* ca = new CutApplier(cut);
128  fSinksS[sid]->AddSink(ca);
129  fSinksSC[sc] = ca;
130  }
131  if(fSinksSCW.find(scw) == fSinksSCW.end()){
132  WeightApplier* wa = new WeightApplier(wei);
133  fSinksSC[sc]->AddSink(wa);
134  fSinksSCW[scw] = wa;
135  }
136 
137  return fSinksSCW[scw];
138  }
void AddSink(IRecordSink *s)
std::unordered_map< int, SystApplier * > fSinksS
std::unordered_map< std::pair< int, int >, CutApplier * > fSinksSC
const Cut cut
Definition: exporter_fd.C:30
std::unordered_map< std::tuple< int, int, int >, WeightApplier * > fSinksSCW
double ana::SpectrumLoaderBase::GetWeightedPOT ( )
inherited
void ana::FileReducer::Go ( )
overridevirtual

Load all the registered spectra.

TODO potentially reenable systematically shifts via FileReducer

see if we want to omit the event

Implements ana::SpectrumLoaderBase.

Definition at line 405 of file FileReducer.cxx.

References ana::assert(), om::cerr, ana::CombineMetadata(), om::cout, ana::Progress::Done(), APDHVSetting::dummy, allTimeWatchdog::endl, caf::Proxy< caf::SRHeader >::evt, MakeMiniprodValidationCuts::f, fCopyMetadata, fCopyNuTree, fEventCut, fEventList, ana::SpectrumLoaderBase::fGone, fNuTruthCut, fNuTruthReductionFuncs, fOmitBadFiles, genie::utils::style::Format(), submit_syst::fout, fOutfile, fReductionFuncs, fReductionFuncsWithProxy, ana::SpectrumLoaderBase::fSpillCut, fSpillReductionFuncs, ana::SpectrumLoaderBase::fWildcard, ana::GetCAFMetadata(), ana::SpectrumLoaderBase::GetNextFile(), caf::Proxy< caf::StandardRecord >::hdr, getGoodRuns4SAM::n, ana::SpectrumLoaderBase::NFiles(), cacheDefinitionData::prog, make_template_knob_config::recTree, caf::Proxy< caf::SRHeader >::run, ana::Progress::SetProgress(), caf_analysis::spill, caf::Proxy< caf::StandardRecord >::spill, sr, caf::Proxy< caf::SRHeader >::subrun, UpdateMetadata(), and ana::WriteCAFMetadata().

Referenced by concat_dataset(), concat_fd_events(), generate_fd_fake_events(), make_numu_decaf(), reduce_bendecomp2017(), reduce_nue_2018(), reduce_nue_fa(), reduce_nue_fd_fa(), reduce_nue_nd_fa(), reduce_nue_or_numu_fa(), reduce_nue_or_numu_or_nus(), reduce_nue_or_numu_sa(), reduce_nue_sa(), reduce_numu_ana2017(), reduce_numu_ana2017_prod(), reduce_numu_ana2018(), reduce_numu_ana2018_cosmics(), reduce_numubarccpi0(), reduce_numuccinc(), reduce_nus(), reduce_nus_ana2019(), reduce_nus_ana2020(), reduce_nus_or_numu(), reduce_prod5_nue(), reduce_prod5_numu(), super_reduce(), and train_reduce().

406  {
407  // FloatingExceptionOnNaN fpnan;
408 
409  // Don't want overflow to happen. Set to 1 petabyte: effectively infinite.
410  TTree::SetMaxTreeSize(1e15);
411 
412  if(fGone){
413  std::cerr << "Error: can only call Go() once on a FileReducer" << std::endl;
414  abort();
415  }
416  fGone = true;
417 
418  const int Nfiles = NFiles();
419 
420  Progress* prog = 0;
421 
422  TFile fout(fOutfile.c_str(), "RECREATE");
423  TTree* trOut = new TTree("recTree", "recTree");
424  TTree* spillOut = new TTree("spillTree", "spillTree");
425  TTree* nuOut = new TTree("nuTree", "nuTree");
426  {
427  // FloatingExceptionOnNaN fpnan(false);
429  trOut->Branch("rec", &dummy);
430  caf::SRSpill dummySpill;
431  spillOut->Branch("spill", &dummySpill);
432  caf::SRSpillTruthBranch dummyNu;
433  nuOut->Branch("mc", &dummyNu);
434  }
435 
436  TH1* hPOTOut = new TH1F("TotalPOT", "", 1, 0, 1);
437  TH1* hEventsOut = new TH1F("TotalEvents", "", 1, 0, 1);
438  TH1* hLivetimeOut = new TH1F("TotalLivetime", "", 1, 0, 1);
439 
440  std::vector<std::string> fnames;
441 
442  std::map<std::string, std::string> meta;
443  std::set<std::string> meta_mask;
444 
445  caf::StandardRecord* oldsr = 0;
446 
447  long nRecSeen = 0;
448  long nRecPassed = 0;
449  long nNuSeen = 0;
450  long nNuPassed = 0;
451 
452  int fileIdx = -1;
453  while(TFile* f = GetNextFile()){
454  ++fileIdx;
455 
456  if(fOmitBadFiles){
457  if(f->IsZombie()) continue;
458  }
459  else{
460  assert(!f->IsZombie());
461  }
462 
463  TH1* hPOT = (TH1*)f->Get("TotalPOT");
464  if(fOmitBadFiles && !hPOT) continue;
465 
466  assert(hPOT);
467  hPOTOut->Add(hPOT);
468 
469  fnames.push_back(f->GetName());
470 
471  if(fCopyMetadata){
472  TDirectory* meta_dir = (TDirectory*)f->Get("metadata");
473  assert(meta_dir);
474  CombineMetadata(meta, GetCAFMetadata(meta_dir), meta_mask);
475  }
476 
477  TH1* hEvents = (TH1*)f->Get("TotalEvents");
478  assert(hEvents);
479  hEventsOut->Add(hEvents);
480 
481  TH1* hLivetime = (TH1*)f->Get("TotalLivetime");
482  assert(hLivetime);
483  hLivetimeOut->Add(hLivetime);
484 
485  TTree* recTree = (TTree*)f->Get("recTree");
486  assert(recTree);
487 
488  // Use this one for assessing cuts etc
489  caf::SRProxy srProxy(0, recTree, "rec", 0, 0);
490 
491  // And if they pass load into this one for writing out
492  caf::StandardRecord* sr = 0;
493  recTree->SetBranchAddress("rec", &sr);
494 
495  const int Nentries = recTree->GetEntries();
496  for(int n = 0; n < Nentries; ++n){
497  ++nRecSeen;
498  recTree->LoadTree(n);
499 
500  /// TODO potentially reenable systematically shifts via FileReducer
501  // Restorer restorer;
502  // double weight = 1;
503  // fSystShifts.Shift(restorer, sr, weight);
504  // if(weight != 1){
505  // std::cerr << "Warning: cannot apply weighting shift with FileReducer. Event's weight will be unaltered." << std::endl;
506  // }
507 
508  // First up, check this spill passes the global quality cuts
509  if(fSpillCut && !(*fSpillCut)(&srProxy.spill)) continue;
510 
511  // Apply EventList cut if it's been enabled
512  if(!fEventList.empty() &&
513  !fEventList.count(std::make_tuple(srProxy.hdr.run,
514  srProxy.hdr.subrun,
515  srProxy.hdr.evt))) continue;
516 
517  /// see if we want to omit the event
518  if(!fEventCut || (*fEventCut)(&srProxy)){
519  recTree->GetEntry(n);
520 
521  if(sr != oldsr){
522  trOut->SetBranchAddress("rec", &sr);
523  oldsr = sr;
524  }
525 
526  // Apply any additional reduction steps
527  for(const auto & f: fReductionFuncs) f(sr);
528  for(const auto & f: fReductionFuncsWithProxy) f(sr, &srProxy);
529 
530  ++nRecPassed;
531  trOut->Fill();
532  }
533 
534  if(Nfiles >= 0 && !prog) prog = new Progress(TString::Format("Filling from %d files matching '%s'", Nfiles, fWildcard.c_str()).Data());
535 
536  if(n%1000 == 0 && Nfiles == 1 && prog)
537  prog->SetProgress(double(n)/Nentries);
538  } // end for n
539 
540  // Copy over all the spillTree variables without modification
541  TTree* spillTree = (TTree*)f->Get("spillTree");
542  assert(spillTree);
543  spillTree->SetBranchStatus("*", 1);
545  caf::SRSpill* pspill = &spill;
546  spillTree->SetBranchAddress("spill", &pspill);
547  spillOut->SetBranchAddress("spill", &pspill);
548  caf::SRSpillProxy spillProxy(0, spillTree, "spill", 0, 0);
549  const int NspillEntries = spillTree->GetEntries();
550  for(int n = 0; n < NspillEntries; ++n){
551  spillTree->GetEntry(n);
552  spillTree->LoadTree(n);
553  for(const auto & f: fSpillReductionFuncs) f(pspill, &spillProxy);
554  spillOut->Fill();
555  }
556 
557  // Copy over all the nuTree variables without modification, unless spilltruthcuts are applied
558  TTree* nuTree = (TTree*)f->Get("nuTree");
559  if(nuTree && fCopyNuTree){
560  nuTree->SetBranchStatus("*", 1);
562  caf::SRSpillTruthBranch* pnu = &nu;
563  nuTree->SetBranchAddress("mc", &pnu);
564  nuOut->SetBranchAddress("mc", &pnu);
565 
566 
567  caf::SRNeutrinoProxy neutrinoProxy(0, nuTree, "mc.nu", 0, 0);
568  const int NNuEntries = nuTree->GetEntries();
569 
570  for(int n = 0; n < NNuEntries; ++n){
571  ++nNuSeen;
572  nuTree->LoadTree(n);
573 
574  if(!fNuTruthCut || (*fNuTruthCut)(&neutrinoProxy)){
575 
576  nuTree->GetEntry(n);
577  for(const auto & f: fNuTruthReductionFuncs) f(pnu, &neutrinoProxy);
578  ++nNuPassed;
579  nuOut->Fill();
580  }
581  }
582  }
583 
584  if(prog) prog->SetProgress((fileIdx+1.)/Nfiles);
585  } // end while GetNextFile
586 
587  fout.cd();
588  trOut->Write();
589  spillOut->Write();
590  if(nuOut->GetEntries() > 0) nuOut->Write();
591  hPOTOut->Write();
592  hEventsOut->Write();
593  hLivetimeOut->Write();
594 
595  UpdateMetadata(meta, meta_mask, fnames);
596  WriteCAFMetadata(fout.mkdir("metadata"), meta);
597 
598  fout.Close();
599 
600  if(prog){
601  prog->Done();
602  delete prog;
603  }
604 
605  std::cout << "Passed " << nRecPassed << " / " << nRecSeen << " records";
606  if(fCopyNuTree) std::cout << " and " << nNuPassed << " / " << nNuSeen << " from the nuTree";
607  std::cout << std::endl;
608  }
std::string fOutfile
Definition: FileReducer.h:119
TFile * GetNextFile()
Forwards to fFileSource but also accumulates POT and livetime.
NuTruthCut * fNuTruthCut
Definition: FileReducer.h:121
bool fGone
Has Go() been called? Can&#39;t add more histograms after that.
Proxy for caf::SRNeutrino.
Definition: SRProxy.h:510
Proxy for caf::StandardRecord.
Definition: SRProxy.h:2126
void CombineMetadata(std::map< std::string, std::string > &base, const std::map< std::string, std::string > &add, std::set< std::string > &mask)
base += add
Definition: Utilities.cxx:251
Truth info for all neutrinos in the spill.
OStream cerr
Definition: OStream.cxx:7
std::unique_ptr< SpillCut > fSpillCut
Cut applied to the spill branch for every event.
Class for parsing *.fcl files for their metadata information.
void WriteCAFMetadata(TDirectory *dir, const std::map< std::string, std::string > &meta)
Write map of metadata parameters into a CAF file.
Definition: Utilities.cxx:312
caf::StandardRecord * sr
Proxy for caf::SRSpill.
Definition: SRProxy.h:1346
OStream cout
Definition: OStream.cxx:6
std::vector< std::function< ReductionFunc > > fReductionFuncs
Definition: FileReducer.h:124
The StandardRecord is the primary top-level object in the Common Analysis File trees.
std::vector< std::function< NuTruthReductionFunc > > fNuTruthReductionFuncs
Definition: FileReducer.h:127
std::map< std::string, std::string > GetCAFMetadata(TDirectory *dir)
Extract map of metadata parameters from a CAF file.
Definition: Utilities.cxx:229
std::set< std::tuple< int, int, int > > fEventList
Definition: FileReducer.h:122
void UpdateMetadata(std::map< std::string, std::string > &meta, const std::set< std::string > &mask, const std::vector< std::string > &fnames) const
assert(nhit_max >=nhit_nbins)
void Format(TGraph *gr, int lcol, int lsty, int lwid, int mcol, int msty, double msiz)
Definition: Style.cxx:154
int NFiles() const
Forwards to fFileSource.
std::vector< std::function< SpillReductionFunc > > fSpillReductionFuncs
Definition: FileReducer.h:126
std::vector< std::function< ReductionFuncWithProxy > > fReductionFuncsWithProxy
Definition: FileReducer.h:125
virtual bool ana::SpectrumLoaderBase::Gone ( ) const
inlinevirtualinherited

Indicate whether or not Go has been called.

Definition at line 124 of file SpectrumLoaderBase.h.

References caf_analysis::spill, submit_syst::str, string, and wildcard().

Referenced by ana::TwoSampleDecomp::Decomp(), ana::CSVMaker::Go(), and ana::SpectrumHandler::Go().

124 {return fGone;}
bool fGone
Has Go() been called? Can&#39;t add more histograms after that.
int ana::SpectrumLoaderBase::NFiles ( ) const
protectedinherited

Forwards to fFileSource.

Definition at line 228 of file SpectrumLoaderBase.cxx.

References ana::SpectrumLoaderBase::fFileSource.

Referenced by ana::SpectrumLoader::Go(), and Go().

229  {
230  return fFileSource->NFiles();
231  }
std::unique_ptr< IFileSource > fFileSource
void ana::FileReducer::SetEventList ( const std::string fname)

If called, only events whose run/subrun/event occur in fname will be retained.

Definition at line 390 of file FileReducer.cxx.

References ana::assert(), MakeMiniprodValidationCuts::f, fclose(), fEventList, ana::run, and ana::subrun.

391  {
392  FILE* f = fopen(fname.c_str(), "r");
393  assert(f);
394 
395  while(!feof(f)){
396  int run, subrun, event;
397  fscanf(f, "%d %d %d", &run, &subrun, &event);
398  fEventList.insert(std::make_tuple(run, subrun, event));
399  }
400 
401  fclose(f);
402  }
fclose(fg1)
int run
Definition: NusVarsTemp.cxx:51
std::set< std::tuple< int, int, int > > fEventList
Definition: FileReducer.h:122
assert(nhit_max >=nhit_nbins)
int subrun
Definition: NusVarsTemp.cxx:52
void ana::FileReducer::SetMetadata ( const std::string key,
const std::string val 
)
inline
void ana::FileReducer::setOmitBadFiles ( bool  omit)
inline

Set an optional parameter to omit bad files (zombie or no POT)

Definition at line 79 of file FileReducer.h.

79 {fOmitBadFiles = omit;}
void ana::SpectrumLoaderBase::SetSpillCut ( const SpillCut cut)
inherited

Definition at line 64 of file SpectrumLoaderBase.cxx.

References ana::assert(), cut, ana::SpectrumLoaderBase::fGone, ana::SpectrumLoaderBase::fSpillCut, and ana::SpectrumLoaderBase::WildcardOrSAMQuery().

Referenced by Ana2017_box_opening_macro(), Ana2017_sb_opening_macro(), Ana2018_box_opening_macro(), AnaResultsLoad(), BlessedPlotsLoad(), caf_nue_data_mc(), caf_numu_fd_validation_data(), caf_numu_fd_validation_MC(), caf_numu_fd_validation_MC_no_tau(), caf_numu_nd_cutflow(), caf_numu_nuenergy_vs_xy(), caf_numu_reco_minus_true(), caf_numu_sensitivity(), caf_numu_sensitivity_no_tau(), caf_numu_validation(), caf_numu_vars(), CalcCutVals(), compare_cos_numi(), Compare_Spectra(), ComparisonPlots_Data(), CosmicPred(), CreateAndFillSpectra(), CutFlow_Cosmic(), CutFlow_Data(), CutFlow_NearDet(), CutTableLoad(), datamc_ND_numu_kinematics(), datamc_ND_numu_kinematics_FHC(), datamc_ND_numu_kinematics_FHC_pTBins(), datamc_ND_numu_kinematics_FHC_REW(), datamc_ND_numu_kinematics_FHC_REW_pTBins(), datamc_ND_numu_kinematics_REW(), datamc_ND_numu_kinematics_RHC(), datamc_ND_numu_kinematics_RHC_pTBins(), datamc_ND_numu_kinematics_RHC_REW(), datamc_ND_numu_kinematics_RHC_REW_pTBins(), dataprocess_numuccinc(), DataVtxDistributionLoad(), demo_trivial_xsec_analysis(), drawIntensityEffect_2019(), drawLongTerm(), drawLongTerm_2019(), drawShortTerm_2019(), drawTimePlots(), drawVsPOT(), EnergyCont_macro(), energyResolution(), estimate_energy(), Evaluate_BDTMLP_Algs_Spectra(), ExampleCode(), exporter_fd(), exporter_nd(), ExtendedAxesLoad(), FD_Data_PosComp(), FDDataMC(), FDDataMCSystBandLoad(), FillSpectra(), FitSystEffectsLoad(), genie_syst_make(), get_cosmic_sample(), get_cosmic_spectra(), get_data_and_cosmic(), get_data_histogram(), get_fd_dataspectrum(), get_numi_data_histogram(), get_numu_data_histogram(), get_spectra_2dplots(), getCrossSectionAnalysis_Spectra(), getCrossSectionAnalysis_Spectra_systematics(), getCrossSectionAnalysisSpectra(), getData(), getFitTemplates(), GetHistsFD(), GetHistsND(), getSpectra_ForFitting(), getStabilitySpectra(), hadEFrac_nd_data_mc_systs(), hyperon_nom_macro(), make_DataMCComp_numu(), make_decomp(), make_michel(), make_nue_ana2018_pot(), make_nue_ana2019_epoch7d_pot(), make_nue_ana2019_epoch8b_pot(), make_nue_ana2019_fhc_ub_pot(), make_nue_ana2019_rhc_ub_pot(), make_nue_thirdana_pot(), make_numu_thirdana_pot(), make_nus17_pot(), make_pi0_xcheck(), make_pid(), make_quantiles_histogram_2020(), make_RHC_WrongSign_Nue(), make_RHC_WrongSign_Numu_MC(), make_RHC_WrongSign_Numu_Pd4_6Data(), make_RHC_WrongSign_Numu_Pd7dData(), make_RHC_WrongSign_Numu_Pd8Data(), make_RHC_WrongSign_NumuQ1(), make_RHC_WrongSign_NumuQ2(), make_RHC_WrongSign_NumuQ3(), make_RHC_WrongSign_NumuQ4(), make_rhcpred_2017(), make_rockpred_2017(), make_xsec_tuning_hists_mp5(), make_xsec_wgts_2018_hists(), MakeCosmics(), MakeNus17CosBkgd(), MakeNus17Prediction(), MakeNus18CosBkgd(), MakeNus18ExtrapPred(), MakeNus18SidebandPred(), MakeNusPrediction(), ana::MakeQuantileHistogram(), ana::MakeTextListFileHelper(), MakeUnoscPlots(), mrbrem_get_initial_spectra(), mrbrem_get_reweighted_spectra(), MRDiFStudy_FHC_Step1(), MRDiFStudy_FHC_Step2(), MRDiFStudy_RHC_Step1(), MRDiFStudy_RHC_Step2(), mre_blessed(), mre_comp_split(), ND_DataMC(), ND_DataMC_Comp_Systs(), NDDataMC(), NDDataMCSystBandLoad(), neutKEsyst(), neutronE_macro(), nue_data_mc_validation(), nue_decomp_scales(), nue_fd_mc_validation(), nue_michelDataMC(), NuMu2019_BasicPIDPlots_ND(), NuMu2019_BasicPIDPlots_Spectrum(), NuMu2020_BasicPIDPlots_FD(), NuMu2020_BasicPIDPlots_ND(), NuMu2020_BasicPIDPlots_Spectrum(), NuMu2020_TrimCAFs(), numu_cut_flow(), numu_data_mc_validation(), numu_validation_numuvars(), NumuCosmic(), nus17_box_opening(), nus17_fd_cut_tables(), nus17_fd_cut_tables2D(), nus17_fiducial_accounting(), nus18_box_opening(), nus_ana01_box_opening(), nus_ana01_sideband_box_opening(), pi0_xcheck(), PositionComparison(), ppfx_make_systs(), ppfx_smooth_weights_save(), preselection_cutflow(), resolutionscript(), resolutionspectrums(), rock(), saveSpectraForUnf(), sidebandfittest(), SideBandLoad(), signal_count(), specprod_numuccinc(), specprod_systematics(), Syst(), SystematicComp(), SystsBirksLoad(), SystsCalibAbsLoad(), SystsCalibAbsLoad17(), SystsCalibAbsMethod17(), SystsCalibRelLoad(), SystsCalibRelLoad17(), SystsNoiseLoad(), SystsPPFXLoad17(), test_genieweights(), test_predictionscalecomp(), test_xsecanalysis(), TrimCAFs(), uncertainty1png(), uncertainty2png(), and validation_numu_nd().

65  {
66  assert(!fGone);
67  fSpillCut = std::make_unique<SpillCut>(cut);
68  }
bool fGone
Has Go() been called? Can&#39;t add more histograms after that.
std::unique_ptr< SpillCut > fSpillCut
Cut applied to the spill branch for every event.
const Cut cut
Definition: exporter_fd.C:30
assert(nhit_max >=nhit_nbins)
void ana::FileReducer::UpdateMetadata ( std::map< std::string, std::string > &  meta,
const std::set< std::string > &  mask,
const std::vector< std::string > &  fnames 
) const
protected

Definition at line 611 of file FileReducer.cxx.

References ana::assert(), om::cerr, allTimeWatchdog::endl, MakeMiniprodValidationCuts::f, fMetaMap, fNuTruthReductionFuncs, fReductionFuncs, fReductionFuncsWithProxy, fSpillReductionFuncs, cet::getenv(), it, m, shutoffs::rel, string, and runNovaSAM::tier.

Referenced by Go().

614  {
615  for(const std::string& m: mask){
616  std::cerr << "Warning: metadata parameter '" << m << "' differs between input files and has been dropped from the output." << std::endl;
617  meta.erase(m);
618  }
619 
620  // change caf -> decaf in the metadata field,
621  // if we actually reduced anything
622  // and if parents have data_tier
623  if ( (!fReductionFuncs.empty() || !fReductionFuncsWithProxy.empty() || !fSpillReductionFuncs.empty() || !fNuTruthReductionFuncs.empty())
624  && meta.find("data_tier") != meta.end())
625  {
626  std::string decaf_tier = meta["data_tier"];
627  assert(decaf_tier.size() >= 3);
628  assert(decaf_tier.substr(decaf_tier.size()-3,3) == "caf");
629  // don't make 'decaf' into 'dedecaf', however
630  if (decaf_tier.size() < 5 || decaf_tier.substr(decaf_tier.size()-5,5) != "decaf")
631  decaf_tier.replace(decaf_tier.size()-3,3,"decaf");
632  meta["data_tier"] = decaf_tier;
633  }
634 
635  const char* rel = getenv("SRT_BASE_RELEASE");
636  if(rel) meta["decaf.base_release"] = rel;
637 
638  std::string parents = "[";
639  for(const std::string& f: fnames){
640  parents += "{\"file_name\":\""+std::string(basename((char *)f.c_str()))+"\"},";
641  }
642  if(parents[parents.size()-1] == ',') parents.resize(parents.size()-1);
643  parents += "]";
644 
645  meta["parents"] = parents;
646 
647  // if there's one more than one parent, this is a concat.
648  // then we need "sumcaf", "sumdecaf", etc. as appropriate
649  if (fnames.size() > 1 && meta.find("data_tier") != meta.end())
650  {
651  auto & tier = meta["data_tier"];
652  // if it ends with 'caf' and doesn't start with 'sum',
653  // it should now start with 'sum'
654  if (tier.substr(tier.size()-3, 3) == "caf" && tier.substr(0, 3) != "sum")
655  tier = "sum" + tier;
656  }
657 
658  // Allow user to override any metadata
659  for(auto it: fMetaMap) meta[it.first] = it.second;
660  }
set< int >::iterator it
OStream cerr
Definition: OStream.cxx:7
std::string getenv(std::string const &name)
std::map< std::string, std::string > fMetaMap
Definition: FileReducer.h:130
Class for parsing *.fcl files for their metadata information.
string rel
Definition: shutoffs.py:11
std::vector< std::function< ReductionFunc > > fReductionFuncs
Definition: FileReducer.h:124
std::vector< std::function< NuTruthReductionFunc > > fNuTruthReductionFuncs
Definition: FileReducer.h:127
assert(nhit_max >=nhit_nbins)
std::vector< std::function< SpillReductionFunc > > fSpillReductionFuncs
Definition: FileReducer.h:126
std::vector< std::function< ReductionFuncWithProxy > > fReductionFuncsWithProxy
Definition: FileReducer.h:125
enum BeamMode string
IFileSource * ana::SpectrumLoaderBase::WildcardOrSAMQuery ( const std::string str) const
protectedinherited

Figure out if str is a wildcard or SAM query and return a source.

Definition at line 72 of file SpectrumLoaderBase.cxx.

References e, cet::getenv(), MECModelEnuComparisons::i, ProjMan::ifdh, ana::Limit(), ana::FileListSource::NFiles(), ana::Offset(), runNovaSAM::ret, and ana::Stride().

Referenced by ProducingSA(), ana::SpectrumLoaderBase::SetSpillCut(), ana::SpectrumLoaderBase::SpectrumLoaderBase(), and useBDTG().

73  {
74  // stat() blows up on strings with spaces
75  if(str.find(' ') == std::string::npos){
76  WildcardSource* ret = new WildcardSource(str, Stride(), Offset(), Limit());
77  if(ret->NFiles() > 0) return ret;
78  delete ret;
79  }
80 
81  // Maybe this the name of a SAM project?
82  {
83  IFDHSilent silent; // the usual case is for this to fail
85 
86  // findProject always gives back an address just by gluing bits together.
87  // (a tad annoying, because it _does_ go and look for the project,
88  // and even would print its 'didn't-find-this-project' error out to stderr
89  // if not for the IFDHSilent, but without scraping its stderr
90  // there's no way to know whether the project is there or not--
91  // you still get the URL.)
92  // however, the WebAPI call looking for the /status will return a 404 if
93  // the project doesn't exist. (suggested by Robert I. in INC000000925362.)
94  try
95  {
96  ifdh_util_ns::WebAPI webapi(i.findProject(str, getenv("SAM_STATION")) + "/status");
97  return new SAMProjectSource(str);
98  }
99  catch (ifdh_util_ns::WebAPIException &e)
100  {
101  ;
102  }
103  }
104 
105  // Maybe this is a SAM dataset or query?
106  return new SAMQuerySource(str, Stride(), Offset(), Limit());
107  }
int Limit()
Value passed to –limit, or -1 if not specified.
Definition: Utilities.cxx:420
std::string getenv(std::string const &name)
size_t Stride(bool allow_default)
Value passed to –stride, or 1 if not specified.
Definition: Utilities.cxx:372
ifdh
Definition: ProjMan.py:8
size_t Offset(bool allow_default)
Value passed to –offset, or 0 if not specified.
Definition: Utilities.cxx:396
Float_t e
Definition: plot.C:35

Member Data Documentation

double ana::SpectrumLoaderBase::fBeamNue
protectedinherited
bool ana::FileReducer::fCopyMetadata
protected

Definition at line 131 of file FileReducer.h.

Referenced by Go().

bool ana::FileReducer::fCopyNuTree
protected

Definition at line 132 of file FileReducer.h.

Referenced by Go().

Cut* ana::FileReducer::fEventCut
protected

Definition at line 120 of file FileReducer.h.

Referenced by AddEventCut(), Go(), and ~FileReducer().

std::set<std::tuple<int, int, int> > ana::FileReducer::fEventList
protected

Definition at line 122 of file FileReducer.h.

Referenced by Go(), and SetEventList().

ExposureSource ana::SpectrumLoaderBase::fExposureSource
protectedinherited
std::unique_ptr<IFileSource> ana::SpectrumLoaderBase::fFileSource
protectedinherited
bool ana::SpectrumLoaderBase::fGone
protectedinherited
std::map<std::string, std::string> ana::FileReducer::fMetaMap
protected

Definition at line 130 of file FileReducer.h.

Referenced by UpdateMetadata().

std::list<NuHistDef> ana::SpectrumLoaderBase::fNuHistDefs
protectedinherited
NuTruthCut* ana::FileReducer::fNuTruthCut
protected

Definition at line 121 of file FileReducer.h.

Referenced by AddNuTruthCut(), Go(), and ~FileReducer().

std::vector<std::function<NuTruthReductionFunc> > ana::FileReducer::fNuTruthReductionFuncs
protected

Definition at line 127 of file FileReducer.h.

Referenced by Go(), and UpdateMetadata().

bool ana::FileReducer::fOmitBadFiles
protected

Definition at line 123 of file FileReducer.h.

Referenced by Go().

std::string ana::FileReducer::fOutfile
protected

Definition at line 119 of file FileReducer.h.

Referenced by Go().

double ana::SpectrumLoaderBase::fPOT
protectedinherited

Accumulated by calls to GetNextFile.

Definition at line 160 of file SpectrumLoaderBase.h.

Referenced by ana::SpectrumLoaderBase::GetNextFile(), and ana::SpectrumLoaderBase::GetWeightedPOT().

std::vector<std::function<ReductionFunc> > ana::FileReducer::fReductionFuncs
protected

Definition at line 124 of file FileReducer.h.

Referenced by Go(), and UpdateMetadata().

std::vector<std::function<ReductionFuncWithProxy> > ana::FileReducer::fReductionFuncsWithProxy
protected

Definition at line 125 of file FileReducer.h.

Referenced by Go(), and UpdateMetadata().

double ana::SpectrumLoaderBase::fRunPOT
protectedinherited

Crude measure, not including spill cuts.

Definition at line 161 of file SpectrumLoaderBase.h.

Referenced by ana::SpectrumLoaderBase::GetNextFile(), ana::SpectrumLoader::Go(), and DecTreeLoader::HandlePOT().

double ana::SpectrumLoaderBase::fSingleNue
protectedinherited
double ana::SpectrumLoaderBase::fSinglePOT
protectedinherited
RecordMultiSink ana::SpectrumLoaderBase::fSink
protectedinherited
std::unordered_map<int, SystApplier*> ana::SpectrumLoaderBase::fSinksS
protectedinherited
std::unordered_map<std::pair<int, int>, CutApplier*> ana::SpectrumLoaderBase::fSinksSC
protectedinherited
std::unordered_map<std::tuple<int, int, int>, WeightApplier*> ana::SpectrumLoaderBase::fSinksSCW
protectedinherited
std::set<IRecordSink*> ana::SpectrumLoaderBase::fSpectrumSinks
protectedinherited
std::unique_ptr<SpillCut> ana::SpectrumLoaderBase::fSpillCut
protectedinherited
std::list<SpillHistDef> ana::SpectrumLoaderBase::fSpillHistDefs
protectedinherited
std::vector<std::function<SpillReductionFunc> > ana::FileReducer::fSpillReductionFuncs
protected

Definition at line 126 of file FileReducer.h.

Referenced by Go(), and UpdateMetadata().

double ana::SpectrumLoaderBase::fWeightedPOT
protectedinherited

Definition at line 166 of file SpectrumLoaderBase.h.

Referenced by ana::SpectrumLoaderBase::GetWeightedPOT().

std::string ana::SpectrumLoaderBase::fWildcard
protectedinherited

The documentation for this class was generated from the following files: