vr_cfg.sh use lots of memory failed to start if there are many firewall rules.

[echoidcf]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR

CLOUDSTACK VERSION

All version after 4.7 affected

CONFIGURATION

Vmware

OS / ENVIRONMENT

SUMMARY

Script writing style in /opt/cloud/bin/vr_cfg.sh in VR will consume HUGE memory if there are more than 1,000 firewall rules.

When VR is restarted, cloudstack will pass a aggregation of JSON files to VR and use vr_cfg.sh to unpack it and run python script to load them. The problem is cloudstack will generate each JSON file to one-line-file, which is ok. But vr_cfg.sh is written in following style:

while read line; do
...
done < $cfg

If JSON file is big, hundreds of kilo-byte for example, there will be a very LOOOOOOOOONG line for the script to read. This will use a lot of memory and result in memory exhausted and will fail the async job of start VR.
And of course , VR will not come up after that.

As a result of my test, 2k firewall rules will cause this problem on a VR with 1GB memory. 6K firewall rules will kill a VR with 8GB memory, and 10k firewall rules will kill the VR even with 16GB memory.

The fix is simple, just change it to

cat $cfg | while read line; do
...
done

will fix this problem.

STEPS TO REPRODUCE

1. add 2000 firewall rules to a VR with 1GB memory
2. stop and start it

it will fail to start.

EXPECTED RESULTS

VR failed to start

ACTUAL RESULTS

Job will failed and a memory out log in the management server log.

[DaanHoogland]

tnx for the report @echoidcf . Are you making a PR for this? and do you think this style of coding is present at more locations?

[yadvr]

This explains why read line is slow (https://stackoverflow.com/questions/13762625/bash-while-read-loop-extremely-slow-compared-to-cat-why); we can explore - https://stackoverflow.com/questions/4642191/read-line-by-line-in-bash-script

[DaanHoogland]

@echoidcf I added the PR for you as it is quite simple. Do you have any test results that proof this will help you, as in my understanding the amount of memory consumed in both variations won't differ much.

Also please see #4451 to check if this is the change you are suggesting.

[davidjumani]

From what I can see, vr_cfg.sh reads the command to be executed and not the contents of the json file itself. It's update_config.py which reads the contents of the potentially large file which could be causing the memory issues

[DaanHoogland]

@echoidcf your suggested code was merged and will be released with 4.15.1. I'm closing this. Please reopen and comment further if this doesn't solve the issue. We have no idea what is needed beyond this.