Is it possible to build a suffix-array in-memory?

[sebschmi]

I would like to construct suffix arrays of very short DNA strings (2 strings á 1000 characters), which should use only very little memory. How do I configure SuffixArray::new to make sure the algorithm does not write anything to disk?

[kyclark]

It's definitely possible! You should understand the difference in Sufr between creating and using/querying a suffix array. The creation step has to use at least as much memory as the input sequence plus some overhead, but the parallel construction algorithm manages to make this rather efficient and quick. The suffix array exists in pieces inside temporary files that are read into memory, sorted, and written back to disk. The final sorted suffix array is then created by concatenating the sorted temp data into a final output file.

The other actions (extract, list, count, locate) have low-memory options, but the default is to read the entire suffix array from disk into memory. You can use the libsufr crate directly in your code to manipulate this directly.

[traviswheeler]

To be precise about your question: sufr is designed with two principles in mind:

• a SuffixArray will be precomputed once, then re-used many times
• we're most interested in behavior for very large sequence inputs, since short ones all take essentially no time.

A result of the first principle is that we assume a file will be eventually be written. Since that's the intention, sufr makes use of batched file writes/reads as part of construction (allowing for low memory footprint during construction). In your use case, this is somewhat wasteful ... but probably not too bad because all file I/O is streaming.

That said, I'm going to re-open the issue. It'll be good for us to offer an option to perform entirely in-memory construction with no file I/O. It's not likely to be changed in the next few weeks...

[sebschmi]

Thanks, it would be great if sufr would support in-memory construction as well.

…

________________________________ From: Travis Wheeler ***@***.***> Sent: 10 June 2025 20:17 To: TravisWheelerLab/sufr ***@***.***> Cc: Schmidt, Sebastian S ***@***.***>; Author ***@***.***> Subject: Re: [TravisWheelerLab/sufr] Is it possible to build a suffix-array in-memory? (Issue #11) Reopened #11<#11>. — Reply to this email directly, view it on GitHub<#11 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AORQGR6JYPDB3S4DFJB6LO33C4HJZAVCNFSM6AAAAAB66YKBN6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJYGA3TSNZXGA3DIMA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[sriram98v]

I would also like to bring up some questions regarding SuffixArrayTrait. The fact that all the methods in the trait require a mutable reference seems to be a concern to me. Ideally, query methods like locate and count should not mutate the underlying struct, and hence should require only a mutable reference.

I can see that in the low-memory setting, you would need a mutable reference of the suffix array since you are reading only a portion of the full .sufr file. However, in the case where we have the full array in-memory, I would think that a method using a mutable reference would suffice.

Would you consider making a separate trait for in-memory applications?