model loading problem

[xiaocangnn]

Hello, when I was loading the pre-trained code generation model given in your github repository, the problem shown in the picture below occurred: the model is missing a word segmenter or the loading path conflicts. I have tried many methods but have not been able to solve it. Have you ever encountered this kind of problem? If so, how was it resolved?

[shunzh]

Hello, Thanks for your question! I quickly tried it on Google Colab and there doesn't seem to be a problem loading the GPT2Tokenizer.
Based on the error message, is there a local directory called "gpt2" in your workspace?

[xiaocangnn]

Thank you very much for your reply. I all used the code you gave on github and then ran it on the Auto DL server, but it prompted me missing the file vocab.json and merges.txt files, trying to download both files from Hugging Face and import, solved the above problem.
But I have a new question: can I take your PG-TD model as a pre-trained model and train it on my own dataset? Attempt to train into a code generation model for exploit exploits.

[shunzh]

Thank you very much for your reply. I all used the code you gave on github and then ran it on the Auto DL server, but it prompted me missing the file vocab.json and merges.txt files, trying to download both files from Hugging Face and import, solved the above problem.

Great to know that the problem is solved!

But I have a new question: can I take your PG-TD model as a pre-trained model and train it on my own dataset? Attempt to train into a code generation model for exploit exploits.

We don't own the pre-trained models. We use the models from this paper: https://arxiv.org/abs/2105.09938. You may check but I think it's okay to fine-tune their models.
However there have been newer code models since then, like CodeGen and Code Llama.