switching_layer 层的参数恒为0，致使不能更新该层的参数

[gbl555]

下面是我从/src/modules文件摘抄出来的代码
262行 self.switching_layer = nn.Linear(2 * hidden_size + hidden_size, 1) 282行 switch_prob = torch.ge(torch.sigmoid(self.switching_layer(switch_input)), 0.5).to(torch.float32) # [batch x 1 x 1] 303行 output = torch.where(~switch_prob.byte(), vocab_output, pointer_output)

因为vocab_output和pointer_output中都不含有self.switching_layer层的参数，所以，output 对switching_layer 层的参数求导结果为0，这样一来，switching_layer 层无法更新参数

[amrufathy]

@gbl555 Hi, I have just seen this issue. I don't speak Chinese so I had to translate your comments.

Neither the vocab_output nor the pointer_output should contain the parameters of the switching layer. The switching layer is used as a decision function for the final output i.e. when to choose from the input sequence and when to choose from the softmax prediction. Hence, the output of the switching layer is included in the final output and the network in fact trains just fine.

[gbl555]

thanks for your reply.I agree with you that there is no problem in theory.But when I output the gradient of all the variables, I found that the gradient of switching_layer's parameter are None,that is to say，the parameters of switching_layer will not be updated.Do you have output the gradient of each variable and the problem I mentioned has appeared?

[amrufathy]

I am not very experienced with pytorch. How do I get the gradient of each variable? Maybe you could provide me a sample code?

And just to be sure, if you use the base ModelManager class, the gradients are expected to be None, because here I clip the gradient values by a None value. I overwrite this value in the child classes, which I use (not the base class).