|
INFO:__main__:Worker started.
|
|
INFO:__main__:Parameters: Namespace(history_uuid='28292e22-e538-11ee-8935-1a5e2beb2457')
|
|
INFO:__main__:conversion_history
|
|
INFO:__main__:<app.tts_common.models.history.ConversionHistory object at 0x7f0677263fd0>
|
|
INFO:__main__:current_user from db Huy Nguyen
|
|
ERROR:__main__:processing tts_history: 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
ERROR:__main__:current user_id: 1288
|
|
INFO:__main__:Try to download and decrypt if needed for 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
INFO:__main__:File downloaded successfully to: tmp/contabo_uploaded/1288/20240318_145939_024737/5.docx
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:Convert TTS and send result START...
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:conversion_history_uuid 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:input_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/5.docx
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:model_price_rate 1
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:file_extension docx
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:current estimated_total_credits to calculate lock: 74424
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:current_estimate_credits: 312954
|
|
0-5 tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3
|
|
1-5 tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3
|
|
INFO:app.tts_common.services.speech:Parent process: Process-2 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
2-5 tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3
|
|
INFO:app.tts_common.services.speech:Parent process: Process-3 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
3-5 tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3
|
|
4-5 tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3
|
|
INFO:app.tts_common.services.speech:Parent process: Process-4 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
INFO:app.tts_common.services.speech:Parent process: Process-5 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
5-5 tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3
|
|
6-5 tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3
|
|
7-5 tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3
|
|
8-5 tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************8EUGxv3HEfJWxf5lWKyG0Y0lP
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************KomCkZdfAvYnpApCsRpIL7CRw
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************z7PCksXIM71r2gP94NEiwwU0h
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************Umq2MqToFgrLiMJu9bJcbZED2
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 22%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 22%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-3 END
|
|
INFO:app.tts_common.services.speech:Parent process: Process-6 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 1
|
|
total_pieces 9
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 22%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 22%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-2 END
|
|
INFO:app.tts_common.services.speech:Parent process: Process-7 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 1
|
|
total_pieces 9
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************pOaMwGkOjLrZMQwYdhjreYdhL
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************tKmcKWpU6i2IW9CYxb2lsoTt2
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 33%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 33%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-5 END
|
|
INFO:app.tts_common.services.speech:Parent process: Process-8 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 2
|
|
total_pieces 9
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************DTeibkkw1PqurxQcjTo7sdDZE
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 44%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 44%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-4 END
|
|
INFO:app.tts_common.services.speech:Parent process: Process-9 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 3
|
|
total_pieces 9
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************EM5hANj80EDXVKB63detSeFkj
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 55%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 55%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-6 END
|
|
INFO:app.tts_common.services.speech:Parent process: Process-10 START
|
|
INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 4
|
|
total_pieces 9
|
|
INFO:app.tts_common.services.speech:count_all_active_key: 2060
|
|
INFO:app.tts_common.services.speech:Requesting openAI with key: ********************WOyWeIrX7EqzjS7I76k3sP6oU
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 66%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 66%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-9 END
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 5
|
|
total_pieces 9
|
|
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK"
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 77%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 77%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-8 END
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 6
|
|
total_pieces 9
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 88%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 88%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-7 END
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 7
|
|
total_pieces 9
|
|
INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 100%
|
|
INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 100%
|
|
INFO:app.tts_common.services.speech:Parent process: Process-10 END
|
|
stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3
|
|
output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3
|
|
The file tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457
|
|
complete_count 8
|
|
total_pieces 9
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3
|
|
INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3
|
|
ERROR:__main__:Convert and send result failed
|
|
ERROR:__main__:Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
Traceback (most recent call last):
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 292, in upload_file
|
|
future.result()
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result
|
|
return self._coordinator.result()
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 266, in result
|
|
raise self._exception
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 139, in __call__
|
|
return self._execute_main(kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 162, in _execute_main
|
|
return_value = self._main(**kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 381, in _main
|
|
client.complete_multipart_upload(
|
|
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call
|
|
return self._make_api_call(operation_name, kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 964, in _make_api_call
|
|
raise error_class(parsed_response, operation_name)
|
|
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/app/worker.py", line 156, in main
|
|
convert_result = await convert_and_send_result(
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/app/app/tts_common/jobs/convert_and_send_result.py", line 176, in convert_and_send_result
|
|
is_uploaded = upload_file_to_s3(file_path=output_file_path, s3_object_name=s3_object_name)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/app/app/tts_common/services/s3_client.py", line 32, in upload_file_to_s3
|
|
s3_client.upload_file(file_path, bucket_name, s3_object_name)
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/inject.py", line 143, in upload_file
|
|
return transfer.upload_file(
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 298, in upload_file
|
|
raise S3UploadFailedError(
|
|
boto3.exceptions.S3UploadFailedError: Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
ERROR:__main__:Traceback (most recent call last):
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 292, in upload_file
|
|
future.result()
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result
|
|
return self._coordinator.result()
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 266, in result
|
|
raise self._exception
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 139, in __call__
|
|
return self._execute_main(kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 162, in _execute_main
|
|
return_value = self._main(**kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 381, in _main
|
|
client.complete_multipart_upload(
|
|
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call
|
|
return self._make_api_call(operation_name, kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 964, in _make_api_call
|
|
raise error_class(parsed_response, operation_name)
|
|
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/app/worker.py", line 156, in main
|
|
convert_result = await convert_and_send_result(
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/app/app/tts_common/jobs/convert_and_send_result.py", line 176, in convert_and_send_result
|
|
is_uploaded = upload_file_to_s3(file_path=output_file_path, s3_object_name=s3_object_name)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/app/app/tts_common/services/s3_client.py", line 32, in upload_file_to_s3
|
|
s3_client.upload_file(file_path, bucket_name, s3_object_name)
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/inject.py", line 143, in upload_file
|
|
return transfer.upload_file(
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 298, in upload_file
|
|
raise S3UploadFailedError(
|
|
boto3.exceptions.S3UploadFailedError: Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
Traceback (most recent call last):
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 292, in upload_file
|
|
future.result()
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result
|
|
return self._coordinator.result()
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 266, in result
|
|
raise self._exception
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 139, in __call__
|
|
return self._execute_main(kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 162, in _execute_main
|
|
return_value = self._main(**kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 381, in _main
|
|
client.complete_multipart_upload(
|
|
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call
|
|
return self._make_api_call(operation_name, kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 964, in _make_api_call
|
|
raise error_class(parsed_response, operation_name)
|
|
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/app/worker.py", line 156, in main
|
|
convert_result = await convert_and_send_result(
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/app/app/tts_common/jobs/convert_and_send_result.py", line 176, in convert_and_send_result
|
|
is_uploaded = upload_file_to_s3(file_path=output_file_path, s3_object_name=s3_object_name)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/app/app/tts_common/services/s3_client.py", line 32, in upload_file_to_s3
|
|
s3_client.upload_file(file_path, bucket_name, s3_object_name)
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/inject.py", line 143, in upload_file
|
|
return transfer.upload_file(
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 298, in upload_file
|
|
raise S3UploadFailedError(
|
|
boto3.exceptions.S3UploadFailedError: Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
ERROR:__main__:28292e22-e538-11ee-8935-1a5e2beb2457
|
|
Traceback (most recent call last):
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 292, in upload_file
|
|
future.result()
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result
|
|
return self._coordinator.result()
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 266, in result
|
|
raise self._exception
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 139, in __call__
|
|
return self._execute_main(kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 162, in _execute_main
|
|
return_value = self._main(**kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 381, in _main
|
|
client.complete_multipart_upload(
|
|
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call
|
|
return self._make_api_call(operation_name, kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 964, in _make_api_call
|
|
raise error_class(parsed_response, operation_name)
|
|
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/app/worker.py", line 156, in main
|
|
convert_result = await convert_and_send_result(
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/app/app/tts_common/jobs/convert_and_send_result.py", line 176, in convert_and_send_result
|
|
is_uploaded = upload_file_to_s3(file_path=output_file_path, s3_object_name=s3_object_name)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/app/app/tts_common/services/s3_client.py", line 32, in upload_file_to_s3
|
|
s3_client.upload_file(file_path, bucket_name, s3_object_name)
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/inject.py", line 143, in upload_file
|
|
return transfer.upload_file(
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 298, in upload_file
|
|
raise S3UploadFailedError(
|
|
boto3.exceptions.S3UploadFailedError: Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress
|
|
INFO:app.tts_common.helpers.email_helper:Send vocalized file failed notice email
|
|
INFO:app.tts_common.helpers.email_helper:sent to message to kafka topic: tts-emails-prod
|
|
INFO:Firebase client:Add notification to firebase START
|
|
INFO:Firebase client:Add notification to firebase END
|
|
INFO:__main__:Recalculate user locked credit
|
|
INFO:__main__:Convert and send result END
|
|
results save text pieces to files prepare to join audios
|
|
checking temporary audio files
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3
|
|
Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3
|
|
Joined MP3 files saved to: tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3
|
|
pre-signed url https://eu2.contabostorage.com/cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3?AWSAccessKeyId=24db761ab7758be8109fcc212513ea39&Signature=LCZsAeFgWkCoZ%2FNOAhckICaNr4E%3D&Expires=1711378914
|
|
Produced event to topic tts-emails-prod: value = {"function": "send_vocalized_file_failed_notice", "email_to": "nguyenhuy31ns@gmail.com", "recipient_name": "Huy Nguyen", "file_name": "5.docx"}
|