Public Member Functions | Protected Attributes | Static Protected Attributes | List of all members
ana::FileListSource Class Reference

Simple file source based on an explicit list provided by the user. More...

#include "/cvmfs/nova.opensciencegrid.org/externals/cafanacore/v01.11/src/CAFAna/Core/FileListSource.h"

Inheritance diagram for ana::FileListSource:
ana::IFileSource ana::SAMQuerySource ana::WildcardSource

Public Member Functions

 FileListSource (const std::vector< std::string > &files, int stride=-1, int offset=-1, int limit=-1)
 Default stride, offset, and limit mean obey cmd-line options. More...
 
virtual ~FileListSource ()
 
virtual TFile * GetNextFile () override
 Returns the next file in sequence, ready for reading. More...
 
int NFiles () const override
 May return -1 indicating the number of files is not known. More...
 
const std::vector< std::string > & GetFileNames () const
 

Protected Attributes

std::vector< std::stringfFileNames
 The list of files. More...
 
std::vector< std::string >::iterator fIt
 Iterator into fFileNames. More...
 
std::vector< std::stringfRetry
 List of files that failed 1st attempt. More...
 
bool fInRetry
 Did we finish fFileNames and are now in fRetry? More...
 
TFile * fFile
 The most-recently-returned file. More...
 

Static Protected Attributes

static bool fgGotTickets = false
 Have we renewed our tickets? More...
 

Detailed Description

Simple file source based on an explicit list provided by the user.

Definition at line 11 of file FileListSource.h.

Constructor & Destructor Documentation

ana::FileListSource::FileListSource ( const std::vector< std::string > &  files,
int  stride = -1,
int  offset = -1,
int  limit = -1 
)

Default stride, offset, and limit mean obey cmd-line options.

Definition at line 15 of file FileListSource.cxx.

References om::cerr, allTimeWatchdog::endl, fFileNames, fgGotTickets, fIt, cet::getenv(), MECModelEnuComparisons::i, makeTrainCVSamples::int, PandAna.Demos.demo0::loc, std::max(), update_sam_good_runs_metadata::stride, string, and system().

17  : fInRetry(false), fFile(0)
18  {
19  if(offset < 0){
20  if(getenv("CAFANA_OFFSET"))
21  offset = atoi(getenv("CAFANA_OFFSET"));
22 
23  offset = std::max(offset, 0);
24  }
25 
26  if(stride < 0){
27  if(getenv("CAFANA_STRIDE"))
28  stride = atoi(getenv("CAFANA_STRIDE"));
29 
30  stride = std::max(stride, 1);
31  }
32 
33  if(limit < 0){
34  if(getenv("CAFANA_LIMIT"))
35  limit = atoi(getenv("CAFANA_LIMIT"));
36  }
37 
38  if(!files.empty() && stride > int(files.size())){
39  std::cerr << "Warning: stride " << stride
40  << " is greater than the number of files: " << files.size()
41  << ". This is strange and inefficient." << std::endl;
42  }
43 
44  for(unsigned int i = offset; i < files.size(); i += stride){
45  fFileNames.push_back(files[i]);
46  if(limit > 0 && int(fFileNames.size()) == limit) break;
47  }
48 
49  fIt = fFileNames.begin();
50 
51  for(const std::string& loc: fFileNames){
52  if(loc.rfind("/pnfs/", 0) == 0){ // ie begins with
53  if(!fgGotTickets){
54  // No kerberos ticket means no point trying to voms-proxy-init. It
55  // likely also means we're in a grid job, where that would be
56  // counterproductive anyway.
57  if(system("klist -5 -s || klist -s") != 0) fgGotTickets = true;
58  }
59 
60  if(!fgGotTickets){
61  // This comes from NovaGridUtils or duneutil
62  system("setup_fnal_security -b");
63 
64  fgGotTickets = true;
65  break;
66  }
67  }
68  }
69  }
T max(const caf::Proxy< T > &a, T b)
system("rm -rf microbeam.root")
std::vector< std::string >::iterator fIt
Iterator into fFileNames.
OStream cerr
Definition: OStream.cxx:7
std::vector< std::string > fFileNames
The list of files.
std::string getenv(std::string const &name)
bool fInRetry
Did we finish fFileNames and are now in fRetry?
static bool fgGotTickets
Have we renewed our tickets?
TFile * fFile
The most-recently-returned file.
enum BeamMode string
ana::FileListSource::~FileListSource ( )
virtual

Definition at line 72 of file FileListSource.cxx.

References fFile.

73  {
74  delete fFile;
75  }
TFile * fFile
The most-recently-returned file.

Member Function Documentation

const std::vector<std::string>& ana::FileListSource::GetFileNames ( ) const
inline

Definition at line 22 of file FileListSource.h.

References fFileNames.

22 { return fFileNames; }
std::vector< std::string > fFileNames
The list of files.
TFile * ana::FileListSource::GetNextFile ( )
overridevirtual

Returns the next file in sequence, ready for reading.

A null return means that the end of the sequence has been reached. DO NOT close or delete the file that is returned.

Implements ana::IFileSource.

Definition at line 78 of file FileListSource.cxx.

References ana::assert(), om::cout, allTimeWatchdog::endl, fFile, fFileNames, fInRetry, fIt, fRetry, MECModelEnuComparisons::i, PandAna.Demos.demo0::loc, ana::pnfs2xrootd(), and string.

Referenced by make_dst_cosrejbdttrain(), trimmubarid(), and trimvar().

79  {
80  // Tidy up the last file we gave, which the caller no longer needs
81  delete fFile;
82  fFile = 0;
83 
84  // Did we run out of files?
85  if(fInRetry && fIt == fRetry.end()) return 0;
86  if(fIt == fFileNames.end()){
87  if(fRetry.empty()) return 0;
88  fIt = fRetry.begin();
89  fInRetry = true;
90  }
91 
92  // If the file is on pnfs rewrite it to an xrootd address
93  std::string loc = *fIt;
94  loc = pnfs2xrootd(loc); // no-op for non /pnfs locations
95 
96  if(fInRetry) std::cout << "Retrying " << loc << " which was previously deferred..." << std::endl;
97 
98  fFile = TFile::Open(loc.c_str()); // This pattern allows xrootd
99 
100  if(!fFile){
101  if(!fInRetry){
102  std::cout << "Unable to open " << loc << std::endl;
103  std::cout << "Will skip this file and try again later." << std::endl;
104  fRetry.push_back(*fIt);
105  ++fIt;
106  return GetNextFile();
107  }
108  else{
109  const int Nretry = 3;
110  std::cout << "Failed to open. Will retry " << Nretry << " more times" << std::endl;
111  for(int i = 0; i < Nretry; ++i){
112  std::cout << "Attempt " << i << std::endl;
113  fFile = TFile::Open(loc.c_str());
114  if(fFile){
115  std::cout << "Success!" << std::endl;
116  break;
117  }
118  } // end for i
119  if(!fFile){
120  std::cout << "Still unable to read " << loc << std::endl;
121  std::cout << "Aborting" << std::endl;
122  abort();
123  }
124  }
125  }
126 
127  // The above should have guaranteed this
128  assert(fFile);
129 
130  ++fIt; // Move on to the next file, for the subsequent call
131 
132  return fFile;
133  }
std::vector< std::string >::iterator fIt
Iterator into fFileNames.
std::string pnfs2xrootd(std::string loc, bool unauth)
Definition: UtilsExt.cxx:237
std::vector< std::string > fFileNames
The list of files.
bool fInRetry
Did we finish fFileNames and are now in fRetry?
OStream cout
Definition: OStream.cxx:6
std::vector< std::string > fRetry
List of files that failed 1st attempt.
virtual TFile * GetNextFile() override
Returns the next file in sequence, ready for reading.
assert(nhit_max >=nhit_nbins)
TFile * fFile
The most-recently-returned file.
enum BeamMode string
int ana::FileListSource::NFiles ( ) const
inlineoverridevirtual

May return -1 indicating the number of files is not known.

Reimplemented from ana::IFileSource.

Definition at line 20 of file FileListSource.h.

References fFileNames.

Referenced by trimmubarid(), trimvar(), and ana::SpectrumLoaderBase::WildcardOrSAMQuery().

20 {return fFileNames.size();}
std::vector< std::string > fFileNames
The list of files.

Member Data Documentation

TFile* ana::FileListSource::fFile
protected

The most-recently-returned file.

Definition at line 28 of file FileListSource.h.

Referenced by GetNextFile(), and ~FileListSource().

std::vector<std::string> ana::FileListSource::fFileNames
protected

The list of files.

Definition at line 24 of file FileListSource.h.

Referenced by FileListSource(), GetFileNames(), GetNextFile(), and NFiles().

bool ana::FileListSource::fgGotTickets = false
staticprotected

Have we renewed our tickets?

Definition at line 29 of file FileListSource.h.

Referenced by FileListSource().

bool ana::FileListSource::fInRetry
protected

Did we finish fFileNames and are now in fRetry?

Definition at line 27 of file FileListSource.h.

Referenced by GetNextFile().

std::vector<std::string>::iterator ana::FileListSource::fIt
protected

Iterator into fFileNames.

Definition at line 25 of file FileListSource.h.

Referenced by FileListSource(), and GetNextFile().

std::vector<std::string> ana::FileListSource::fRetry
protected

List of files that failed 1st attempt.

Definition at line 26 of file FileListSource.h.

Referenced by GetNextFile().


The documentation for this class was generated from the following files: