DBRunner discussion

[dnadeau-lanl]

I have been running, DBRunner.py with postgresql and got some issues with the following:

• NO INPUT with OUTPUT_TIMEBASE=RUN gives me an error

  File "/projects/sdndev/scripts/common/dbprocessing/dbprocessing/runMe.py", line 333, in __init__
    self.filename = 'RUN_{0}_{1}'.format(process_entry.process_name, self.input_files[0])

• I tried with OUTPUT_TIMEBASE=FILE and things work fine.

Would you know if there is a way to run DBRunner only where they are files in the processqueue table? Right now it's running every cronjob timing interval. I could write a wrapper to check if files were ingested and before launching it, but I thought you might have a DB way to do this. 😏

[jtniehof]

The first is definitely a bug. I hadn't thought about supporting processes that have no inputs and no outputs (which is basically what the RUN output timebase is), but I think it does make a certain amount of weird sense. (EDIT: Captured as issue #121.)

I'm not sure I see the use case for using DBRunner only when there's something in the process queue. That's basically what ProcessQueue -p does. (EDIT: Possibly addressed with #88.)

[dnadeau-lanl]

I have no required_input1 for that process, so there is no productprocesslink,

I ingested many files and there are file_id(s) in the process queue.

So when I run ProcessQueue -p or DBRunner the processqueue is emptied which is good.

I run DBRunner and my program processed all the ingested files found in the ingested directory at once.

ubet  --inputdir /projects/sdndev/dbingested/sabrs --outputdir /projects/sdndev/sabrs/data

The issues is that DBRunner.py is run by my cronjob every 5 minutes or every minute, and the DBRunner prints statements fill up the logs. Maybe we could make it more quiet...

[jtniehof]

I opened #121 to capture the first question. I think your use case for the second question was addressed by the changes in #88. Are we good to call this done?