Using API's created instances gives error Permission denied (publickey) under the VPC network

[satishglondhe]

ISSUE TYPE

• Bug Report

COMPONENT NAME

API, VPC, SSH

CLOUDSTACK VERSION

4.16.0, 4.17.0.1 

CONFIGURATION

Advanced Networking - VPC

OS / ENVIRONMENT

N/A

SUMMARY

Not able to access the VM using the SSH keypair attached at the time of instance creation with VPC network using API's. If we reinstall such instance then it starts working. Also if we do this with the Cloud Stack UI it is working as expected in the first attempt.

STEPS TO REPRODUCE

Steps followed using APIs:
1. Create project
2. Create VPC network
3. Register SSH key pair
4. Create new instance
5. Attach this instance to the VPC network
6. Acquire and associate public ip address to the VPC network and static nat with the instance
7. Try to access this instance using public ip and ssh key
Everything working fine, We can see the registered ssh key pair in the Cloud Stack UI under the instance details. But at the time of logging in using ssh it gives error  **Permission denied (publickey).**

EXPECTED RESULTS

SSH login should work in the first attempt if instance created under the VPC network using API's

ACTUAL RESULTS

Permission denied (publickey)

ssh ubuntu@ip_address

[DaanHoogland]

@satishglondhe is it the ssh registering or the create new instance that is to be done by the API for this to fail, or both?
And what do you mean by
5. Attach this instance to the VPC network
is it created in another network first and then added to the vpc?
tnx

[satishglondhe]

is it the ssh registering or the create new instance that is to be done by the API for this to fail, or both?
Both is working, SSH registered using API and passed its name in deployVirtualMachine APIs keypair/ keypairs param. Then we can see the keypair attached to instance in details page at Cloud Stack UI level. But not able to access.

And what do you mean by
5. Attach this instance to the VPC network
My mistake, Network is created first and it's id passed in deployVirtualMachine API call in param iptonetworklist[0].networkid.

deployVirtualMachine API request params

{
"apikey": "API_KEY",
"bootmode": "LEGACY",
"boottype": "BIOS",
"command": "deployVirtualMachine",
"displayname": "vm-kimber-578e53",
"iptonetworklist[0].networkid": "6320d0b7-fd14-44b1-b32f-266169116424",
"keypairs": "satish",
"name": "vm-kimber-578e53",
"projectid": "adf82a41-3bf9-4798-8ac1-82d3169c6294",
"response": "json",
"serviceofferingid": "39f225df-5e9e-43ae-96ed-3ee8b8bb2717",
"startvm": "true",
"templateid": "aa94497a-4a4f-4373-8b9e-ec1f3c23e1fb",
"zoneid": "4b9ecd7c-04af-453a-8b86-88d864e59f71",
"signature": "mvrrdNbIr+XUHk/N4tYrHn/5WEY="
}

[weizhouapache]

@satishglondhe
the ssh key are saved in CloudStack virtual router.
The vm instance gets the ssh key only if cloud-init is installed or a script is executed in startup.

please refer to http://docs.cloudstack.apache.org/en/latest/adminguide/templates/_cloud_init.html
or use this script https://github.com/apache/cloudstack/blob/main/setup/bindir/cloud-set-guest-sshkey-password-userdata-configdrive.in

[satishglondhe]

Thanks @weizhouapache for replying.
I'll look into more details of cloud init setup. For now, we have added some delay while auto-provisioning the VM, which has resolved our issue. Let me explain the scenario. This may help someone, or if I'm wrong, you can give proper direction for this.
On a single click, we auto-provision the VM and related services by creating the network, creating egress rules if applicable, registering the SSH key, deploying the VM, acquiring an IP address, associating the acquired IP with the VM, creating firewall rules, and so on.
Steps are

1. User fill up the form (Region, Compute Offering, Network Type(Isolated/ VPC), Network(If available), SSH Key, Host Name)
2. Click on the deploy button, and then internally our system do the following using APIs
3. Register the SSH key for account/ project
4. If the network is not available in that account we create network
5. Create Virtual Machine
6. Acquire one available public IP address in network
7. Associate that public IP address to the VM
8. Add default firewall rules
9. And show the success message to User

For each API call, we wait for the ACS response using async job request APIs and then fire the next API. Everything works fine except the SSH key.
So we observed that we get a success response for deployVirtualMachine and then send the acquire IP address and associate IP address request APIs to the VM. But here is the gotcha: deployVirtualMachine gives the response, but still there are some OS installation operations going on and before this operation's completion, as per our flow, the next API gets executed because the OS is not fully ready. Sometimes the SSH key works, sometimes not. So we have added some delay before the associate public IP address in the VM API call, and then it started working fine.

[weizhouapache]

@satishglondhe
Thanks for the info.
It seems the userdata service does not work for a moment during acquiring public ip address.
I will review the process and see if the issue can be reproduced.

Reopen this issue