perf (tsort) : avoid reading the whole input into memory and intern strings #9872

anastygnome · 2025-12-26T21:48:45Z

This PR is a WIP to get more performance out of tsort.

Move to usize is complete. Next step is to move to vec instead of hashmap

codspeed-hq · 2025-12-26T22:02:24Z

CodSpeed Performance Report

Merging #9872 will improve performance by 10.61%

_{Comparing anastygnome:tsort (4108130) with main (1eea517)}

Summary

⚡ 2 improvements
✅ 134 untouched
⏩ 24 skipped¹

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`tsort_input_parsing_heavy[5000]`	85.1 ms	76.9 ms	+10.61%
⚡	`tsort_wide_dag[100000]`	133.4 ms	129.3 ms	+3.16%

24 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

github-actions · 2025-12-26T22:07:22Z

GNU testsuite comparison:

GNU test failed: tests/misc/tsort. tests/misc/tsort is passing on 'main'. Maybe you have to rebase?

github-actions · 2025-12-27T22:04:56Z

GNU testsuite comparison:

GNU test failed: tests/cp/cp-mv-enotsup-xattr. tests/cp/cp-mv-enotsup-xattr is passing on 'main'. Maybe you have to rebase?
Note: The gnu test tests/csplit/csplit-io-err was skipped on 'main' but is now failing.

github-actions · 2025-12-27T22:38:10Z

GNU testsuite comparison:

Congrats! The gnu test tests/tail/follow-name is no longer failing!

github-actions · 2025-12-27T23:02:11Z

GNU testsuite comparison:

Congrats! The gnu test tests/tail/follow-name is no longer failing!

github-actions · 2025-12-27T23:59:47Z

GNU testsuite comparison:

GNU test failed: tests/misc/tsort. tests/misc/tsort is passing on 'main'. Maybe you have to rebase?

github-actions · 2025-12-28T01:14:42Z

GNU testsuite comparison:

GNU test failed: tests/cp/cp-mv-enotsup-xattr. tests/cp/cp-mv-enotsup-xattr is passing on 'main'. Maybe you have to rebase?
Note: The gnu test tests/csplit/csplit-io-err was skipped on 'main' but is now failing.

sylvestre · 2025-12-28T10:35:05Z

any idea why the perf regressed here?

sylvestre · 2025-12-28T11:22:46Z

Cargo.toml

 uu_base32 = { version = "0.5.0", path = "src/uu/base32" }
 uutests = { version = "0.5.0", package = "uutests", path = "tests/uutests" }
-
+string-interner = "0.19.0"


this isn't the right place to declare it

I'm sorry, where is that?

sylvestre · 2025-12-28T11:22:57Z

Cargo.toml

 whoami = { optional = true, version = "0.5.0", package = "uu_whoami", path = "src/uu/whoami" }
 yes = { optional = true, version = "0.5.0", package = "uu_yes", path = "src/uu/yes" }

+


please remove this change

anastygnome · 2025-12-28T11:23:59Z

Oh, @sylvestre , I just added a few changes :)
The perf regression is most likely due to using Usize symbols. U32 actually improves performance.

github-actions · 2025-12-28T11:51:43Z

GNU testsuite comparison:

GNU test failed: tests/sort/sort-stale-thread-mem. tests/sort/sort-stale-thread-mem is passing on 'main'. Maybe you have to rebase?

github-actions · 2025-12-28T12:13:50Z

GNU testsuite comparison:

Congrats! The gnu test tests/tail/assert is no longer failing!

anastygnome · 2025-12-28T13:29:53Z

@sylvestre I believe the regression noticed are due to cache misses because the metadata that goes with the token is just slightly bigger than usize.

Here's another benchmark that maybe of interest:

graph.tsort, 1 Gb random graph 
Massif output: 
Gnu tsort: 

    MB (Memory)
248.6^                                                                       #
     |                                                          ::::::::::@::#
     |                                        ::::::::::::::::::: ::: ::::@: #
     |                               :::::@:::: ::::: : ::::: ::: ::: ::::@: #
     |                            @@:: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |                        ::::@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |                     @@::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |                   ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |               ::::::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |              :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |            :::::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |         :::: :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |       :::: : :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |      :: :: : :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |    :::: :: : :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |    : :: :: : :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |   :: :: :: : :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     |   :: :: :: : :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     | @@:: :: :: : :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
     | @ :: :: :: : :::: ::@ ::: :@ :: : :@:: : ::::: : ::::: ::: ::: ::::@: #
   0 +----------------------------------------------------------------------->Gi (Giga Instructions) 
     0                                                                   23.47

Number of snapshots: 56
 Detailed snapshots: [1, 2, 16, 21, 26, 52, 55 (peak)]
 
 
 
uutils tsort 

     MB (Memory)
200.0^                                                                    #   
     |                                @::::@:::::::::::@::::@::::@::::@:::#:  
     |              :::::::::::@::::::@::::@::: :::::: @::::@::::@::::@:::#:  
     |           ::::::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:  
     |         :::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:: 
     |        ::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:: 
     |      @:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |     :@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |     :@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |     :@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |     :@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |     :@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |   @::@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |  :@::@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |  :@::@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |  :@::@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     |  :@::@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     | ::@::@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     | ::@::@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
     | ::@::@:::::: :::::::::::@:::: :@::::@::: :::::: @::::@::::@::::@:::#:::
   0 +----------------------------------------------------------------------->Gi (Giga Instructions) 
     0                                                                   35.83

Number of snapshots: 98
 Detailed snapshots: [3, 6, 24, 31, 37, 50, 51, 52, 62, 72, 82, 91 (peak)]

Hyperfine : 
 
Benchmark 1: target/release/coreutils tsort /home/bench_vm/graph.tsort
  Time (mean ± σ):      6.044 s ±  0.071 s    [User: 5.602 s, System: 0.394 s]
  Range (min … max):    5.974 s …  6.168 s    10 runs
 
Benchmark 2: tsort /home/bench_vm/graph.tsort 
  Time (mean ± σ):      7.115 s ±  0.147 s    [User: 6.738 s, System: 0.328 s]
  Range (min … max):    6.910 s …  7.390 s    10 runs
`
Summary: 'target/release/coreutils tsort /home/bench_vm/graph.tsort'
ran 1.18 ± 0.03 times faster than 'tsort /home/bench_vm/graph.tsort'
 `

github-actions · 2025-12-28T13:54:57Z

GNU testsuite comparison:

Congrats! The gnu test tests/tail/assert is no longer failing!

github-actions · 2025-12-28T18:15:10Z

GNU testsuite comparison:

Congrats! The gnu test tests/tty/tty-eof is no longer failing!

github-actions · 2025-12-28T21:33:01Z

GNU testsuite comparison:

Congrats! The gnu test tests/tty/tty-eof is no longer failing!

github-actions · 2025-12-28T23:50:22Z

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/tty/tty-eof is no longer failing!

github-actions · 2025-12-30T12:56:59Z

GNU testsuite comparison:

GNU test failed: tests/misc/tsort. tests/misc/tsort is passing on 'main'. Maybe you have to rebase?

github-actions · 2025-12-30T12:59:35Z

GNU testsuite comparison:

GNU test failed: tests/misc/tsort. tests/misc/tsort is passing on 'main'. Maybe you have to rebase?

github-actions · 2025-12-30T13:18:04Z

GNU testsuite comparison:

GNU test failed: tests/shuf/shuf-reservoir. tests/shuf/shuf-reservoir is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/sort/sort-stale-thread-mem. tests/sort/sort-stale-thread-mem is passing on 'main'. Maybe you have to rebase?

github-actions · 2025-12-30T15:28:03Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-12-30T18:39:49Z

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

github-actions · 2025-12-31T13:55:08Z

GNU testsuite comparison:

Congrats! The gnu test tests/factor/t26 is no longer failing!
Congrats! The gnu test tests/factor/t31 is no longer failing!

…ings

github-actions · 2025-12-31T23:43:38Z

GNU testsuite comparison:

GNU test failed: tests/tail/truncate. tests/tail/truncate is passing on 'main'. Maybe you have to rebase?

anastygnome force-pushed the tsort branch 3 times, most recently from 81ece9b to 0a56fe6 Compare December 27, 2025 21:48

anastygnome force-pushed the tsort branch from 0a56fe6 to bc88290 Compare December 27, 2025 22:22

anastygnome force-pushed the tsort branch from bc88290 to 4c0138c Compare December 27, 2025 22:46

anastygnome force-pushed the tsort branch 2 times, most recently from ddb2361 to b98950c Compare December 27, 2025 23:43

anastygnome force-pushed the tsort branch from b98950c to fdf5306 Compare December 28, 2025 00:59

anastygnome force-pushed the tsort branch from fdf5306 to 8908850 Compare December 28, 2025 11:21

sylvestre reviewed Dec 28, 2025

View reviewed changes

anastygnome force-pushed the tsort branch from c0f1452 to 6da22e3 Compare December 28, 2025 11:55

anastygnome force-pushed the tsort branch from 6da22e3 to 17c9efe Compare December 28, 2025 13:24

anastygnome marked this pull request as ready for review December 28, 2025 13:24

anastygnome requested a review from sylvestre December 28, 2025 13:36

anastygnome force-pushed the tsort branch from 17c9efe to c1dc245 Compare December 28, 2025 17:59

anastygnome force-pushed the tsort branch from c1dc245 to c82e26c Compare December 28, 2025 21:17

anastygnome force-pushed the tsort branch 9 times, most recently from 306c01c to 8439c5a Compare December 30, 2025 11:55

anastygnome force-pushed the tsort branch from 8439c5a to ef3266e Compare December 30, 2025 15:16

anastygnome force-pushed the tsort branch from ef3266e to 692bc7e Compare December 30, 2025 17:56

anastygnome force-pushed the tsort branch from 692bc7e to 46b970b Compare December 31, 2025 13:36

anastygnome force-pushed the tsort branch 4 times, most recently from da98aae to 380e2aa Compare December 31, 2025 23:33

anastygnome added 2 commits December 31, 2025 23:34

perf(tsort): avoid reading the whole input into memory and intern str…

2a96895

…ings

perf(tsort): avoid reading the whole input into memory and intern str…

4108130

…ings

anastygnome force-pushed the tsort branch from 380e2aa to 4108130 Compare December 31, 2025 23:34

		whoami = { optional = true, version = "0.5.0", package = "uu_whoami", path = "src/uu/whoami" }
		yes = { optional = true, version = "0.5.0", package = "uu_yes", path = "src/uu/yes" }

Uh oh!

perf (tsort) : avoid reading the whole input into memory and intern strings #9872

Are you sure you want to change the base?

perf (tsort) : avoid reading the whole input into memory and intern strings #9872

Conversation

anastygnome commented Dec 26, 2025

Uh oh!

codspeed-hq bot commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #9872 will improve performance by 10.61%

Summary

Benchmarks breakdown

Footnotes

Uh oh!

github-actions bot commented Dec 26, 2025

Uh oh!

github-actions bot commented Dec 27, 2025

Uh oh!

github-actions bot commented Dec 27, 2025

Uh oh!

github-actions bot commented Dec 27, 2025

Uh oh!

github-actions bot commented Dec 27, 2025

Uh oh!

github-actions bot commented Dec 28, 2025

Uh oh!

sylvestre commented Dec 28, 2025

Uh oh!

sylvestre Dec 28, 2025

Choose a reason for hiding this comment

Uh oh!

anastygnome Dec 28, 2025

Choose a reason for hiding this comment

Uh oh!

sylvestre Dec 28, 2025

Choose a reason for hiding this comment

Uh oh!

anastygnome commented Dec 28, 2025

Uh oh!

github-actions bot commented Dec 28, 2025

Uh oh!

github-actions bot commented Dec 28, 2025

Uh oh!

anastygnome commented Dec 28, 2025

Uh oh!

github-actions bot commented Dec 28, 2025

Uh oh!

github-actions bot commented Dec 28, 2025

Uh oh!

github-actions bot commented Dec 28, 2025

Uh oh!

github-actions bot commented Dec 28, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 30, 2025

Uh oh!

github-actions bot commented Dec 31, 2025

Uh oh!

github-actions bot commented Dec 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codspeed-hq bot commented Dec 26, 2025 •

edited

Loading