pretrain + dryrun #7
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qhzhuang
requested changes
Dec 6, 2025
| @@ -0,0 +1,342 @@ | |||
| # 实验简介 | |||
|
|
|||
| 本实验在完成Qwen2.5-7B-Instruct预训练的基础上，使用内存模拟工具DryRun模拟相同配置下的预训练，对比内存占用情况，从而证明DryRun工具可以作为模型训练资源规划和问题诊断的有效手段，避免真实训练的资源消耗和时间成本。 | |||
Collaborator
There was a problem hiding this comment.
后半句"从而证明..."不要这么描述, 证明这个能力是你的隐性逻辑, 改为:“介绍DryRun工具的使用方法，帮助用户降低开发与调试成本”
|
|
||
| ``` | ||
|
|
||
| # RryRun模拟Qwen2.5-7B-Instruct预训练流程 |
| 本实验在完成Qwen2.5-7B-Instruct预训练的基础上，使用内存模拟工具DryRun模拟相同配置下的预训练，对比内存占用情况，从而证明DryRun工具可以作为模型训练资源规划和问题诊断的有效手段，避免真实训练的资源消耗和时间成本。 | ||
|
|
||
|
|
||
| # 环境准备 |
Collaborator
There was a problem hiding this comment.
这一部分不用重复介绍了吧, 直接引用Chapter2的东西就好了
|
|
||
| # RryRun模拟Qwen2.5-7B-Instruct预训练流程 | ||
|
|
||
| ## 权重转换 |
Collaborator
There was a problem hiding this comment.
权重转换, 数据预处理啥的也不用介绍了,反正你都和前面是一样的, 直接点一下, 在"参考xx章节,完成xx,xx,xx流程后"
|
|
||
| ## DryRun模拟预训练 | ||
|
|
||
| 1. 修改文件 /MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/training/training.py中的pretain函数，在pretain函数的开头部分添加以下代码行 |
|
|
||
|  | ||
|
|
||
| 2. 修改文件 /MindSpeed-Core-MS/MSAdapter/mmsadpter/distributed/distributed_c10d.py中的init_method为"tcp://ip:port"，此处ip和port根据实际情况修改即可 |
| </tr> | ||
| <tr> | ||
| <td>真实预训练+无重计算</td> | ||
| <td>Device MOC memory size: 62420M<br> |
| Used peak memory usage (without fragments): 55405M<br> | ||
| Actual peak memory usage (with fragments): 57400M | ||
| </td> | ||
| <td rowspan="2" style="vertical-align: middle;">1. DryRun可以模拟实际预训练显存使用<br> |
Collaborator
There was a problem hiding this comment.
结论的1改为: DryRun模拟的显存占用与实际显存占用差距较小, 后面的结论也是, "DryRun可以模拟"这种有点过于笃定
| ``` | ||
| 具体效果如下图所示 | ||
|
|
||
|  |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Changes
Testing & Benchmark
Checklist
Reviewers