Add multi-container support

[zxhe-sean]

Description

Adding a flag --multi-container for supporting multi-container workloads.

With this flag enabled, running workload create will duplicate the container part in the final yaml file so that the resulting workload will be running in two containers running on one host.

Issue

b/459820949

Testing

https://cloudlogging.app.goo.gl/F81QQQ4Ca9Gew7hs8

[github-actions]

🤖 Hi @zxhe-sean, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[scaliby]

@zxhe-sean why do we need this change? Could you cover it with tests?

[jamOne-]

Thanks for the changes and the unit tests!
I left a couple of comments, PTAL.

[jamOne-]

Thanks for the changes, looks good! Left one more comment

[jamOne-]

In the FR, Victor said that not all tpu7x should support this, for instance 2x2x1 should not? Or am I wrong?

Could we add a support for that?

[zxhe-sean]

2x2x1 is 4 chips on a full host. I think it is the smallest topology where we can start using 2 containers @Obliviour

[Obliviour]

4 chips can support it but less than 4 can't

[jamOne-]

@Obliviour @FIoannides PTAL

[Obliviour]

So this PR provides us the ability from the workload to enable two container workloads.

There are two other dimensions that need to be covered that I think we need xpk team's help in:

1. In order for it to be numa aware, the gke nodepool topology manager needs to be configured with best effort. The default is none so when the two containers are scheduled, it won't be numa aware. GKE has an improvement on the way to make best effort the default so that this problem is solved. But in the short term, we need to create the nodepools with an additional topology manager set to best effort.

2. When nap is used, we need the nodepool to created with the same topology manager through nap.