-
Notifications
You must be signed in to change notification settings - Fork 2
Description
There are several issues when jumping to the end of some CSV files
- navigation keys are broken in that case (e.g. pressing "down" just goes back to the beginning of the file)
- blank data
I suppose this is when the file is large enough to not be entirely indexed at once AND the initially computed average line size is larger than the real average, which makes it try to read too far into the file, and thus gets a partial or even empty block.
We should generally implement a better behavior when read() returns fewer lines than what we expected, but could also special-case when the last "block of lines" is asked for and read backward (until we get enough lines) from the end of the file in that case. We should have a way from the adapter to signal that the shape and labels need to be updated immediately in that case.
Alternatively (or in addition to the above, I am unsure), we could implement a different behavior when we are entirely (ie including start_line) in non-indexed land: instead of trying to get the exact lines from start_line to stop_line (which could be beyond the end of the file), compute a stop_percent (a percentage of the file size in bytes) and then read as many chunks backward as necessary to get the desired number of lines (stop_line - start_line).