grade motivation error - "ValueError: could not convert string to float: '-1\n= -1'"

[GenevieveBuckley]

I've just come across this error with grader, although I'm not sure how serious it is yet.
I pip installed grader from the blank-applied branch of @jni's fork

Note that I graded a few applications just fine before this one threw an error.

Your choice (-1, 0, 1)/s/d/l LABEL []? 0
motivation score set to 0

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 7.2% done, 244 left to go ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━┛


bad value: could not convert string to float: '-1\n= -1'
Traceback (most recent call last):
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/site-packages/grader/grader.py", line 1378, in main
    cmd.cmdloop()
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/cmd.py", line 217, in onecmd
    return func(arg)
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/site-packages/grader/grader.py", line 588, in do_grade
    if not self._grade(person, opts.disagreement is not None):
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/site-packages/grader/grader.py", line 662, in _grade
    self._dumpone(person, format='motivation')
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/site-packages/grader/grader.py", line 419, in _dumpone
    motivation_scores=self._gradings(p, 'motivation'),
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/site-packages/grader/grader.py", line 646, in _gradings
    return list_of_float(gen)
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/site-packages/grader/grader.py", line 645, in <genexpr>
    for identity in IDENTITIES)
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/site-packages/grader/configfile.py", line 26, in get
    return self.__getitem__(item)
  File "/anaconda3/envs/scipyschool-admin/lib/python3.6/site-packages/grader/configfile.py", line 18, in __getitem__
    value = self.type(value)
ValueError: could not convert string to float: '-1\n= -1'
grader> 

[otizonaizit]

@keszybz : Ah, so enabling more than 3 identities did not work after all!

[GenevieveBuckley]

We could probably just switch it back to 3 identities then - one of our organisers is very busy right now so won't be grading applications.

[otizonaizit]

Looking at the error it seems unrelated though... What was your identity?

…

On Wed 10 Oct, 13:05 +0000, Genevieve Buckley ***@***.***> wrote: We could probably just switch it back to 3 identities then - one of our organisers is very busy right now so won't be grading applications. — You are receiving this because you commented. Reply to this email directly, view it on GitHub¹, or mute the thread².☘ –––– ¹ #18 (comment) ² https://github.com/notifications/unsubscribe-auth/AAVMCzrpNfJU6VCx1_LEdhWsoeK76UXCks5ujfCDgaJpZM4XVMMs

[GenevieveBuckley]

Identity 1

[jni]

This is probably not related to the number of identities in the file, but rather to an incomplete application in the applications.csv. We found the following line in grader.conf (names changed to protect the innocent):

john smith = -1
	= -1
jane doe = 0

Tiziano, to answer your comment elsewhere, yes, we kinda screwed up the limesurvey settings, but at the same time fixed them, in that people can submit and resubmit based on the same token, so that they could save partial applications, even if all the questions are required. Of course, now we don't know whether an application is incomplete because the person gave up or because they forgot to answer one of the questions (and I don't want to penalise those that didn't answer e.g. the underrepresented minority question or something). Probably the solution is to have default answers for everything, but for now, we have to deal with incomplete applications. (It might mean that we will manually go and edit them.)

[otizonaizit]

This is probably not related to the number of identities in the file, but rather to an incomplete application in the applications.csv. We found the following line in grader.conf (names changed to protect the innocent):

john smith = -1
	= -1
jane doe = 0

I see. OK, I have do admit that before exportig the applications to CSV, I go through them using the limesurvey interface and clean up a bit. For example, some people write their name all lower case, but surely don't want their name to be printed lowercase in the website and in the name badge. This particular case seems to be someone who applied only with an email address but without specifying name and surname. I think I have never seen it before, I didn't even know it was possible. But, well, if the name is not written anywhere else in the application, I would just drop it (you can to it in limesurvey too). Or, if the name can be evinced from the application, just fix name and surname in the limesurvay interface before exporting

Tiziano, to answer your comment elsewhere, yes, we kinda screwed up the limesurvey settings, but at the same time fixed them, in that people can submit and resubmit based on the same token, so that they could save partial applications, even if all the questions are required.

But why, oh why? You can configure limesurvey to allow for resuming a partial application and even to re-edit an already submitted one by using the same token. This is completely unrelated to having the questions as mandatory or facultative. The setting is in Settings/Participant settings: Enable token-based response persistence, Allow multiple responses or update responses with one token. Both should be enabled.

Of course, now we don't know whether an application is incomplete because the person gave up or because they forgot to answer one of the questions (and I don't want to penalise those that didn't answer e.g. the underrepresented minority question or something).

In limesurvey you can export only the completed survey, so that at least all the mandatory questions are answered for sure. Then, I think, if you give the option "do not want to answer", it is perfectly fair to make that question mandatory, no?

Probably the solution is to have default answers for everything, but for now, we have to deal with incomplete applications. (It might mean that we will manually go and edit them.)

Well, long term the solution is to have only mandatory questions, allow for reuse of the token for completing or adjusting the application after the fact, and for the questions that you do not want the people to answer, allow for a "don't want to answer" answer. That is, by the way, what we were doing until now, so I am not sure why you changed that :)

[jni]

This particular case seems to be someone who applied only with an email address but without specifying name and surname. I think I have never seen it before, I didn't even know it was possible.

Me neither. But we at ASPP Apac like to be on the cutting edge. =P

The setting is in Settings/Participant settings: Enable token-based response persistence, Allow multiple responses or update responses with one token. Both should be enabled.

Those were my settings. I thought I had to change them to that for ASPP-Apac 1 so I don't think you were doing that prior to last year. Maybe I'm wrong, or maybe those settings don't get exported by limesurvey.

This is completely unrelated to having the questions as mandatory or facultative.

It's not unrelated. If you can submit partial applications, then when you come across a partial application, you have no idea whether the person saved it for later and never completed it, or simply (by accident) scrolled past a question. When I compared the completed-only and all applications exports, I found quite a few that had most stuff filled in, just missing one or two questions. I could just say, bad luck, those people should have paid more attention, but that feels unnecessarily mean.

So, I think the only way to make this work, if you want persistence, is for all responses to have a valid default value.

for the questions that you do not want the people to answer, allow for a "don't want to answer" answer. That is, by the way, what we were doing until now, so I am not sure why you changed that :)

We didn't change that. It's just that this still requires a click, and humans have extreme aversion to clicking on things.

[otizonaizit]

This is completely unrelated to having the questions as mandatory or facultative.

It's not unrelated. If you can submit partial applications, then when you come across a partial application, you have no idea whether the person saved it for later and never completed it, or simply (by accident) scrolled past a question. When I compared the completed-only and all applications exports, I found quite a few that had most stuff filled in, just missing one or two questions. I could just say, bad luck, those people should have paid more attention, but that feels unnecessarily mean.

I also check the partial response before exporting them to CSV. Until now I have had at most one/two per year which filled in most stuff and just forgot to click submit. If someone only filled in their generalities but didn't bother answering the programming experience or writing the motivation letter, I then just ignored the application. So, there must be something special about the people applying to ASPP-Apac. And, in any case, if there were just a few of them, well, why don't you manually edit the responses, add the missing stuff and re-export? It seems much easier and less dangerous than "fixing" grader.

We didn't change that. It's just that this still requires a click, and humans have extreme aversion to clicking on things.

Sure, but again, these have been a couple of applicants out of >150 consistently over the years, so I insist that you are over-estimating the problem, and fixing grader is the wrong approach.

[otizonaizit]

by the way, re-exporting doesn't break grader, so the work you have done until now does not get lost, unless you edit the name+surname combination, which is what grader uses as unique identifier.

[jni]

And, in any case, if there were just a few of them, well, why don't you manually edit the responses, add the missing stuff and re-export? It seems much easier and less dangerous than "fixing" grader.

This philosophy is surely the reason why the evaluation survey didn't have a markdown export until ASPP Apac 1. ;)

More seriously, I understand that the magnitude of the problem each year is relatively small, and indeed we've gone and fixed the part of the applications csv that was causing this problem. But (a) it is painful, particularly because reading a csv with newlines in the responses is horrible (and the limesurvey interface is not particularly amazing either), (b) there's been many ASPPs, and hopefully there will be many more, so these efforts might yield dividends, and (c) in another thread you were advocating for altering grader quite drastically! I only propose that missing data handling seems like a pretty basic feature and leaving this issue open as a reminder is not a bad thing, not that you need to update grader instantly so we can process our applications.

Oh, also (d), fixing limesurvey to have default values for the questions isn't hard, is it?

Incidentally, specifically for this issue, I have no idea what kind of parsing you're doing for the grades but why doesn't grader stop at the newline??? That's crazy.

So, there must be something special about the people applying to ASPP-Apac.

Not necessarily. I didn't go through them all that carefully, just stopped at the first couple of applications that falsified my hope that all incomplete applications had zero information.

re-exporting doesn't break grader, so the work you have done until now does not get lost,

We already made a bunch of manual edits to the .csv, which could be git-merged/cherry-picked to a new export, but I'd rather not do that.