179 run(Counter run_cnt);
217 hash<string, PipelineQueue>
pmap;
394 abort(*
bool ignore_exceptions);
logError(string fmt)
Logs to the error log, if set.
hash< string, PipelineQueue > pmap
Hash of queues keyed by queue ID.
Definition: DataProviderPipeline.qc.dox.h:217
*code info_log
a closure or call reference for info logging
Definition: DataProviderPipeline.qc.dox.h:98
int tid
TID of the background thread.
Definition: DataProviderPipeline.qc.dox.h:144
waitDone()
Wait for the queue to be empty, then wait for all terminating pipelines to be empty.
int getId()
Returns the pipeline ID.
PipelineQueue copyPipeline(PipelineQueue old_queue)
Called by the copy constructor to copy the queues.
bool do_flush
Flush pipeline flag.
Definition: DataProviderPipeline.qc.dox.h:154
date start_time
run start time
Definition: DataProviderPipeline.qc.dox.h:234
int id
Queue ID.
Definition: DataProviderPipeline.qc.dox.h:114
bool stop_flag
Stop flag.
Definition: DataProviderPipeline.qc.dox.h:260
submit(auto _data)
Submits data for processing.
const PS_ABORTED
Pipeline status: ABORTED.
Definition: DataProviderPipeline.qc.dox.h:37
Defines the abstract class for data provider iterators; the destructor releases the iterator...
Definition: AbstractDataProviderRecordIterator.qc.dox.h:33
const PS_IDLE
Pipeline status: IDLE.
Definition: DataProviderPipeline.qc.dox.h:43
date duration
Total time processing end to end.
Definition: DataProviderPipeline.qc.dox.h:74
int size
Maximum queue size.
Definition: DataProviderPipeline.qc.dox.h:141
bool locked
Locked flag.
Definition: DataProviderPipeline.qc.dox.h:225
string name
The name of the pipeline.
Definition: DataProviderPipeline.qc.dox.h:49
string name
A descriptive name for logging purposes.
Definition: DataProviderPipeline.qc.dox.h:214
int num_queues
Number of pipeline queues.
Definition: DataProviderPipeline.qc.dox.h:66
Condition cond()
Queue condition variable.
*code error_log
Error log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:246
*date stop_time
Stop time for processing.
Definition: DataProviderPipeline.qc.dox.h:55
reset()
Resets the pipeline.
append(AbstractDataProcessor processor)
Appends a data processor to the default queue.
string getName()
Returns the pipeline name.
int appendQueue(int id)
Appends a new queue to an existing pipeline and returns the new queue ID.
float duration_secs
Total time processing end to end as a floating-point value in durationSecondsFloat.
Definition: DataProviderPipeline.qc.dox.h:77
bool do_bulk
Bulk flag.
Definition: DataProviderPipeline.qc.dox.h:222
bool bulk
Flag that indicates if the pipeline is capable of bulk record processing.
Definition: DataProviderPipeline.qc.dox.h:71
string status
Pipeline status.
Definition: DataProviderPipeline.qc.dox.h:60
*code debug_log
Debug log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:249
const PS_RUNNING
Pipeline status: RUNNING.
Definition: DataProviderPipeline.qc.dox.h:40
Defines an abstract class for accepting data and outputting optionally transformed or filtered data...
Definition: AbstractDataProcessor.qc.dox.h:33
list< auto > elems()
Pipeline elements.
Condition flush_cond()
Flush condition variable.
Pipeline info.
Definition: DataProviderPipeline.qc.dox.h:47
int flush_waiting
Number of threads waiting on the flush cond.
Definition: DataProviderPipeline.qc.dox.h:135
date stop_time
run stop time (set in waitDone())
Definition: DataProviderPipeline.qc.dox.h:237
bool isProcessing()
Returns True if the pipeline is processing data.
bool aborting()
Returns True if the object is aborting.
abort(*bool ignore_exceptions)
Aborts execution of a pipeline in progress.
copy()
Copy constructor; creates an empty pipeline with the same configuration as the original.
list< hash< ExceptionInfo > > error_list
list of exceptions in pipelines
Definition: DataProviderPipeline.qc.dox.h:231
float recs_per_sec
Records processed per second end to end.
Definition: DataProviderPipeline.qc.dox.h:80
*code thread_callback
a closure or call reference for setting thread-local data in new pipeline queue threads ...
Definition: DataProviderPipeline.qc.dox.h:254
*string name
the name of the pipeline for logging purposes; if this key is not included, a generic name will be ge...
Definition: DataProviderPipeline.qc.dox.h:106
Defines the abstract class for data provider iterators; the destructor releases the iterator...
Definition: AbstractDataProviderBulkRecordInterface.qc.dox.h:33
Pipeline option info.
Definition: DataProviderPipeline.qc.dox.h:84
DataProviderPipeline parent
Parent object.
Definition: DataProviderPipeline.qc.dox.h:160
Defines a class for passing data through record processors.
Definition: DataProviderPipeline.qc.dox.h:207
bool abort_flag
Abort flag.
Definition: DataProviderPipeline.qc.dox.h:263
Pipeline element.
Definition: DataProviderPipeline.qc.dox.h:110
int data_waiting
Number of threads waiting on data.
Definition: DataProviderPipeline.qc.dox.h:132
reportError(PipelineQueue queue, hash< ExceptionInfo > ex)
Called from a pipeline queue object to report a fatal error durring processing.
destructor()
Destroys the object.
waitDone()
Waits for all queues to have processed remaining data.
submitIntern(auto _data)
Submits data for processing.
*code info_log
Info log closure; takes a single format string and then arguments for format placeholders.
Definition: DataProviderPipeline.qc.dox.h:243
submitData(AbstractIterator i)
Submits data for processing.
int queue_waiting
Number of threads waiting data to be removed from the queue.
Definition: DataProviderPipeline.qc.dox.h:129
checkUpdatePipelineIntern(int id)
Check if the given queue exists.
*date start_time
Start of processing.
Definition: DataProviderPipeline.qc.dox.h:52
logDebug(string fmt)
Logs to the debug log, if set.
list< auto > queue
Data queue.
Definition: DataProviderPipeline.qc.dox.h:138
checkSubmitIntern()
Throws an exception if the pipeline cannot be used; locks the pipeline for changes otherwise...
checkLockedIntern()
Throws an exception if the pipeline is locked.
*code debug_log
a closure or call reference for debug logging
Definition: DataProviderPipeline.qc.dox.h:88
bool hasQueue(int id)
Returns True if the given queue exists, False if not.
constructor(*hash< PipelineOptionInfo > opts)
Creates the object with the given options.
Sequence seq(1)
Pipeline ID sequence generator.
hash< PipelineInfo > getInfo()
Returns pipeline info.
int record_count
Number of input records submitted.
Definition: DataProviderPipeline.qc.dox.h:63
Counter cnt
Parent counter.
Definition: DataProviderPipeline.qc.dox.h:120
throwPipelineException()
Throws an exception if errors occured in background pipeline processing.
run(Counter run_cnt)
Processing thread.
stopInternUnlocked()
Stops all background pipeline queues; lock must be held.
Qore AbstractDataField class definition.
Definition: AbstractDataField.qc.dox.h:27
*code thread_callback
a closure or call reference for setting thread-local data in new pipeline queue threads ...
Definition: DataProviderPipeline.qc.dox.h:103
int record_count
Record count.
Definition: DataProviderPipeline.qc.dox.h:240
logInfo(string fmt)
Logs to the info log, if set.
stopIntern()
Stops all background pipeline queues.
bool stopping()
Returns True if the object is stopping.
Counter cnt()
Thread counter.
*code error_log
a closure or call reference for error logging
Definition: DataProviderPipeline.qc.dox.h:93
submit(auto qdata)
Submits data for processing.
bool data_flushed
Data flushed confirmation.
Definition: DataProviderPipeline.qc.dox.h:157
resetIntern()
Resets the pipeline.
submitDataIntern(auto _data)
Submits a single record for processing.
submitBulkIntern(AbstractDataProviderBulkRecordInterface i)
Submits bulk data for processing.
constructor(DataProviderPipeline parent, Mutex lck, Counter cnt, int id, int size)
Creates the object.
Mutex lck
Parent lock.
Definition: DataProviderPipeline.qc.dox.h:117
registerThread(PipelineQueue queue)
Registers a new thread.