botocore.vendored.requests.exceptions.SSLError: EOF occurred in violation of protocol

[kchen0x]

When I try to run python driver.py, it goes like following:

Dataset size: 26186978239.0, nKeys: 202, avg: 129638506.134
updating: mapper.py (deflated 54%)
updating: jobinfo.json (deflated 34%)
updating: lambdautils.py (deflated 64%)
updating: reducer.py (deflated 55%)
updating: jobinfo.json (deflated 34%)
updating: lambdautils.py (deflated 64%)
updating: reducerCoordinator.py (deflated 64%)
updating: jobinfo.json (deflated 34%)
updating: lambdautils.py (deflated 64%)
{u'TracingConfig': {u'Mode': u'PassThrough'}, u'CodeSha256': u'71RMdPKxqObAaOGTEIWEBewOYZ4gDRy/hzhfRLIKIwY=', u'FunctionName': u'BL-mapper-bl-release', 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '1c5907ad-8745-11e8-9ff2-c123f00e76d9', 'HTTPHeaders': {'date': 'Sat, 14 Jul 2018 09:05:53 GMT', 'x-amzn-requestid': '1c5907ad-8745-11e8-9ff2-c123f00e76d9', 'content-length': '626', 'content-type': 'application/json', 'connection': 'keep-alive'}}, u'CodeSize': 3777, u'RevisionId': u'cc218ffe-de04-4b1b-bea9-ef454225dcce', u'MemorySize': 1536, u'FunctionArn': u'arn:aws:lambda:us-west-1:238423183776:function:BL-mapper-bl-release:5', u'Version': u'5', u'Role': u'arn:aws:iam::238423183776:role/biglambda_role', u'Timeout': 300, u'LastModified': u'2018-07-14T09:05:53.331+0000', u'Handler': u'mapper.lambda_handler', u'Runtime': u'python2.7', u'Description': u'BL-mapper-bl-release'}
{u'TracingConfig': {u'Mode': u'PassThrough'}, u'CodeSha256': u'NejYpJPs9Kg2AzZW3lBIwZvPR+QUPtH7GWd+OW6TYjg=', u'FunctionName': u'BL-reducer-bl-release', 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '1ce717b1-8745-11e8-8cfa-49a139cb41d7', 'HTTPHeaders': {'date': 'Sat, 14 Jul 2018 09:05:54 GMT', 'x-amzn-requestid': '1ce717b1-8745-11e8-8cfa-49a139cb41d7', 'content-length': '630', 'content-type': 'application/json', 'connection': 'keep-alive'}}, u'CodeSize': 3859, u'RevisionId': u'c740f033-bbb4-42eb-8e6b-8885b1c4fdc6', u'MemorySize': 1536, u'FunctionArn': u'arn:aws:lambda:us-west-1:238423183776:function:BL-reducer-bl-release:5', u'Version': u'5', u'Role': u'arn:aws:iam::238423183776:role/biglambda_role', u'Timeout': 300, u'LastModified': u'2018-07-14T09:05:54.269+0000', u'Handler': u'reducer.lambda_handler', u'Runtime': u'python2.7', u'Description': u'BL-reducer-bl-release'}
{u'TracingConfig': {u'Mode': u'PassThrough'}, u'CodeSha256': u'w/dQHEKHQlKL8XTSjUJZcyDldUc+/QUYniJfYo+RNp8=', u'FunctionName': u'BL-rc-bl-release', 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '1df0e605-8745-11e8-96b5-692c4dd04c96', 'HTTPHeaders': {'date': 'Sat, 14 Jul 2018 09:05:56 GMT', 'x-amzn-requestid': '1df0e605-8745-11e8-96b5-692c4dd04c96', 'content-length': '626', 'content-type': 'application/json', 'connection': 'keep-alive'}}, u'CodeSize': 4997, u'RevisionId': u'b39a817b-1a18-4abf-9557-4fdbbd4c46e5', u'MemorySize': 1536, u'FunctionArn': u'arn:aws:lambda:us-west-1:238423183776:function:BL-rc-bl-release:5', u'Version': u'5', u'Role': u'arn:aws:iam::238423183776:role/biglambda_role', u'Timeout': 300, u'LastModified': u'2018-07-14T09:05:56.015+0000', u'Handler': u'reducerCoordinator.lambda_handler', u'Runtime': u'python2.7', u'Description': u'BL-rc-bl-release'}
{u'Statement': u'{"Sid":"82","Effect":"Allow","Principal":{"Service":"s3.amazonaws.com"},"Action":"lambda:InvokeFunction","Resource":"arn:aws:lambda:us-west-1:238423183776:function:BL-rc-bl-release","Condition":{"ArnLike":{"AWS:SourceArn":"arn:aws:s3:::mrtest-kchen"}}}', 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 201, 'RequestId': '1e3282ed-8745-11e8-81ce-130f46efef9d', 'HTTPHeaders': {'date': 'Sat, 14 Jul 2018 09:05:56 GMT', 'x-amzn-requestid': '1e3282ed-8745-11e8-81ce-130f46efef9d', 'content-length': '298', 'content-type': 'application/json', 'connection': 'keep-alive'}}}
# of Mappers  202
Traceback (most recent call last):
  File "driver.py", line 211, in <module>
    results = pool.map(invoke_lambda_partial, Ids[mappers_executed: mappers_executed + nm])
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
botocore.vendored.requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:726)

[Rayz0r]

Hi @kchen0x, I have been able to create a similar error. I believe this is to do with python version management and the AWS X-Ray SDK. If I remove the support for the SDK then the issue goes away, I was also able to make the issue go away with using pyenv and setting the python version to 2.7.10+. Can you test and I will dig further into this.