Session File Expiration?

[cosmicgumbo]

I am having an issue with the session file where it works for a time, but then the Nagios check will stop working. If I delete the session file or remove it from the command it is fixed. This has happened with both the host-service and datastores command. This is using -u and -p (not the environment variable). The owner is 'nagios' user and has rw permissions.

For example, I am using --sessionfile /usr/local/nagios/libexec/check_vsphere/datastore-session-file

[datamuc]

Sorry for the late reply. I failed to monitor these issues correctly. Does the problem still persists, what does "stop working" mean exactly? Is there some exception occuring?

[cosmicgumbo]

The error says:

UNKNOWN - Unhandled exception: (vim.fault.NotAuthenticated) {
dynamicType = ,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'The session is not authenticated.',
faultCause = ,
faultMessage = (vmodl.LocalizableMessage) [],
object = 'vim.Folder:group-d1',
privilegeId = 'System.View',
missingPrivileges = (vim.fault.NoPermission.EntityPrivileges) []
}

[datamuc]

I remember something, I guess you should create an about check (and maybe use it as a parent for all the other checks). The about check will check if the session has the required permissions and will delete the sessionfile if not.

https://omd.consol.de/docs/plugins/check_vsphere/cmd/about/

[cosmicgumbo]

OK, so I created an about check (check_vsphere about) and used the sessionfile parameter: --sessionfile /usr/local/nagios/libexec/check_vsphere/about-session-file

Now I should be able to use the session file on other checks?

[datamuc]

Correct. The about check implicitly checks if the session has the correct permissions (they sometimes get lost on certain action on the vcenter) and if not, the sessionfile is deleted. So any check that runs after that will login again and recreate the sessionfile.

[datamuc]

@cosmicgumbo did it help?

[cosmicgumbo]

I just added it, I'll check back to see if it sticks.

[cosmicgumbo]

After a bit, the other check (I'm using the perf disk:DeviceLatency as an example), started failing with "
UNKNOWN - Unhandled exception: (vim.fault.NotAuthenticated) {"

I found that the about check would fail (soft state) and recover quickly with the error: "CRITICAL: No System.View permission, VMware vCenter Server 7.0.3 build-24322018, api: VirtualCenter/7.0.3.0, product: VMware VirtualCenter Server 7.0"

I just added the --skip-permission to the about check.

Do you mean that the various checks should all use the same session file?

[datamuc]

After a bit, the other check (I'm using the perf disk:DeviceLatency as an example), started failing with " UNKNOWN - Unhandled exception: (vim.fault.NotAuthenticated) {"

I found that the about check would fail (soft state) and recover quickly with the error: "CRITICAL: No System.View permission, VMware vCenter Server 7.0.3 build-24322018, api: VirtualCenter/7.0.3.0, product: VMware VirtualCenter Server 7.0"

I just added the --skip-permission to the about check.

Yeah, it is meant to recover. If you add --skip-permission the whole thing won't work, because then the about check doesn't check for permission and will never delete the session file.

Do you mean that the various checks should all use the same session file?

One session file for each unique -s option.

[jstark1]

Interesting — I get this error during my tests today, and I also saw the code in the 'about' command. Wouldn't it be better to solve this during the initialisation phase of the command?

[datamuc]

Interesting — I get this error during my tests today, and I also saw the code in the 'about' command. Wouldn't it be better to solve this during the initialisation phase of the command?

To check if the session is valid requires an extra api call. If you check everything through the vcenter and have a gazillion of hosts and checks that makes a gazillion more API calls, every check_period.

I was thinking about adding a session timeout to the about check, that can be set to a value less than the real session timeout. So the session would get renewed before it actually expires.

But I don't even know if it actually expires. Maybe it gets invalid for other reasons. Maybe vcenter just looses it during reboot or something.

[jstark1]

main...jstark1:check_vsphere:check_session works for me.
A cleaner way my be vmware/pyvmomi#347 (comment)

[jstark1]

I added a second commit with a fix and some debugging. The monitoring user needs the privilege "Session -> Validate session" in its vCenter Role.

[datamuc]

service_instance.content.sessionManager.SessionIsActive(sessionId,args.user)

this makes the extra api call, I'm probably willing to merge if you can make this configurable

[jstark1]

I don't think the additional API call would have a significant impact on server performance. The notwork traffic is also minimal.
In my tests, the session expired after about 30 minutes. Therefore, if the check is run at regular intervals, the vim.fault.NotAuthenticated exception does not occur.
I'm considering only running SessionIsActive if the session file is older than 15 minutes, which would prevent the additional API call in real-world scenarios. What do you think?