INFO:__main__:Worker started. INFO:__main__:Parameters: Namespace(history_uuid='28292e22-e538-11ee-8935-1a5e2beb2457') INFO:__main__:conversion_history INFO:__main__: INFO:__main__:current_user from db Huy Nguyen ERROR:__main__:processing tts_history: 28292e22-e538-11ee-8935-1a5e2beb2457 ERROR:__main__:current user_id: 1288 INFO:__main__:Try to download and decrypt if needed for 28292e22-e538-11ee-8935-1a5e2beb2457 INFO:__main__:File downloaded successfully to: tmp/contabo_uploaded/1288/20240318_145939_024737/5.docx INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:Convert TTS and send result START... INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:conversion_history_uuid 28292e22-e538-11ee-8935-1a5e2beb2457 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:input_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/5.docx INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:model_price_rate 1 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:file_extension docx INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:current estimated_total_credits to calculate lock: 74424 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:current_estimate_credits: 312954 0-5 tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3 1-5 tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3 INFO:app.tts_common.services.speech:Parent process: Process-2 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 2-5 tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3 INFO:app.tts_common.services.speech:Parent process: Process-3 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 3-5 tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3 4-5 tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3 INFO:app.tts_common.services.speech:Parent process: Process-4 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 INFO:app.tts_common.services.speech:Parent process: Process-5 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 5-5 tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3 6-5 tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3 7-5 tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3 8-5 tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3 INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:Requesting openAI with key: ********************8EUGxv3HEfJWxf5lWKyG0Y0lP INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:Requesting openAI with key: ********************KomCkZdfAvYnpApCsRpIL7CRw INFO:app.tts_common.services.speech:Requesting openAI with key: ********************z7PCksXIM71r2gP94NEiwwU0h INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:Requesting openAI with key: ********************Umq2MqToFgrLiMJu9bJcbZED2 INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 22% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 22% INFO:app.tts_common.services.speech:Parent process: Process-3 END INFO:app.tts_common.services.speech:Parent process: Process-6 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 1 total_pieces 9 INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 22% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 22% INFO:app.tts_common.services.speech:Parent process: Process-2 END INFO:app.tts_common.services.speech:Parent process: Process-7 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 1 total_pieces 9 INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:Requesting openAI with key: ********************pOaMwGkOjLrZMQwYdhjreYdhL INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:Requesting openAI with key: ********************tKmcKWpU6i2IW9CYxb2lsoTt2 INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 33% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 33% INFO:app.tts_common.services.speech:Parent process: Process-5 END INFO:app.tts_common.services.speech:Parent process: Process-8 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 2 total_pieces 9 INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:Requesting openAI with key: ********************DTeibkkw1PqurxQcjTo7sdDZE INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 44% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 44% INFO:app.tts_common.services.speech:Parent process: Process-4 END INFO:app.tts_common.services.speech:Parent process: Process-9 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 3 total_pieces 9 INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:Requesting openAI with key: ********************EM5hANj80EDXVKB63detSeFkj INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 55% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 55% INFO:app.tts_common.services.speech:Parent process: Process-6 END INFO:app.tts_common.services.speech:Parent process: Process-10 START INFO:app.tts_common.services.speech:setup_api_key of model: gpt-3.5 stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 4 total_pieces 9 INFO:app.tts_common.services.speech:count_all_active_key: 2060 INFO:app.tts_common.services.speech:Requesting openAI with key: ********************WOyWeIrX7EqzjS7I76k3sP6oU INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 66% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 66% INFO:app.tts_common.services.speech:Parent process: Process-9 END stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 5 total_pieces 9 INFO:httpx:HTTP Request: POST https://api.openai.com/v1/audio/speech "HTTP/1.1 200 OK" INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 77% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 77% INFO:app.tts_common.services.speech:Parent process: Process-8 END stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 6 total_pieces 9 INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 88% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 88% INFO:app.tts_common.services.speech:Parent process: Process-7 END stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 7 total_pieces 9 INFO:app.tts_common.helpers.tts_history_helper:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 100% INFO:[CRUD History]:add_percent_complete: 28292e22-e538-11ee-8935-1a5e2beb2457 - 100% INFO:app.tts_common.services.speech:Parent process: Process-10 END stream_to_file from openAI tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3 output_file_path tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3 The file tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3 exists. exiting waiting tts-history 28292e22-e538-11ee-8935-1a5e2beb2457 complete_count 8 total_pieces 9 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3 INFO:Job: convert_and_send_result 28292e22-e538-11ee-8935-1a5e2beb2457:join file tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3 ERROR:__main__:Convert and send result failed ERROR:__main__:Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 292, in upload_file future.result() File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result return self._coordinator.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 266, in result raise self._exception File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 139, in __call__ return self._execute_main(kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 162, in _execute_main return_value = self._main(**kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 381, in _main client.complete_multipart_upload( File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call return self._make_api_call(operation_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 964, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/worker.py", line 156, in main convert_result = await convert_and_send_result( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/app/tts_common/jobs/convert_and_send_result.py", line 176, in convert_and_send_result is_uploaded = upload_file_to_s3(file_path=output_file_path, s3_object_name=s3_object_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/app/tts_common/services/s3_client.py", line 32, in upload_file_to_s3 s3_client.upload_file(file_path, bucket_name, s3_object_name) File "/usr/local/lib/python3.11/site-packages/boto3/s3/inject.py", line 143, in upload_file return transfer.upload_file( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 298, in upload_file raise S3UploadFailedError( boto3.exceptions.S3UploadFailedError: Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress ERROR:__main__:Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 292, in upload_file future.result() File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result return self._coordinator.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 266, in result raise self._exception File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 139, in __call__ return self._execute_main(kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 162, in _execute_main return_value = self._main(**kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 381, in _main client.complete_multipart_upload( File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call return self._make_api_call(operation_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 964, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/worker.py", line 156, in main convert_result = await convert_and_send_result( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/app/tts_common/jobs/convert_and_send_result.py", line 176, in convert_and_send_result is_uploaded = upload_file_to_s3(file_path=output_file_path, s3_object_name=s3_object_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/app/tts_common/services/s3_client.py", line 32, in upload_file_to_s3 s3_client.upload_file(file_path, bucket_name, s3_object_name) File "/usr/local/lib/python3.11/site-packages/boto3/s3/inject.py", line 143, in upload_file return transfer.upload_file( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 298, in upload_file raise S3UploadFailedError( boto3.exceptions.S3UploadFailedError: Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 292, in upload_file future.result() File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result return self._coordinator.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 266, in result raise self._exception File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 139, in __call__ return self._execute_main(kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 162, in _execute_main return_value = self._main(**kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 381, in _main client.complete_multipart_upload( File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call return self._make_api_call(operation_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 964, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/worker.py", line 156, in main convert_result = await convert_and_send_result( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/app/tts_common/jobs/convert_and_send_result.py", line 176, in convert_and_send_result is_uploaded = upload_file_to_s3(file_path=output_file_path, s3_object_name=s3_object_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/app/tts_common/services/s3_client.py", line 32, in upload_file_to_s3 s3_client.upload_file(file_path, bucket_name, s3_object_name) File "/usr/local/lib/python3.11/site-packages/boto3/s3/inject.py", line 143, in upload_file return transfer.upload_file( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 298, in upload_file raise S3UploadFailedError( boto3.exceptions.S3UploadFailedError: Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress ERROR:__main__:28292e22-e538-11ee-8935-1a5e2beb2457 Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 292, in upload_file future.result() File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result return self._coordinator.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/futures.py", line 266, in result raise self._exception File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 139, in __call__ return self._execute_main(kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 162, in _execute_main return_value = self._main(**kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3transfer/tasks.py", line 381, in _main client.complete_multipart_upload( File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 530, in _api_call return self._make_api_call(operation_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 964, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/worker.py", line 156, in main convert_result = await convert_and_send_result( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/app/tts_common/jobs/convert_and_send_result.py", line 176, in convert_and_send_result is_uploaded = upload_file_to_s3(file_path=output_file_path, s3_object_name=s3_object_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/app/tts_common/services/s3_client.py", line 32, in upload_file_to_s3 s3_client.upload_file(file_path, bucket_name, s3_object_name) File "/usr/local/lib/python3.11/site-packages/boto3/s3/inject.py", line 143, in upload_file return transfer.upload_file( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 298, in upload_file raise S3UploadFailedError( boto3.exceptions.S3UploadFailedError: Failed to upload tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 to cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress INFO:app.tts_common.helpers.email_helper:Send vocalized file failed notice email INFO:app.tts_common.helpers.email_helper:sent to message to kafka topic: tts-emails-prod INFO:Firebase client:Add notification to firebase START INFO:Firebase client:Add notification to firebase END INFO:__main__:Recalculate user locked credit INFO:__main__:Convert and send result END results save text pieces to files prepare to join audios checking temporary audio files Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/0-5.mp3 Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/1-5.mp3 Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/2-5.mp3 Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/3-5.mp3 Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/4-5.mp3 Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/5-5.mp3 Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/6-5.mp3 Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/7-5.mp3 Checking File: tmp/contabo_uploaded/1288/20240318_145939_024737/8-5.mp3 Joined MP3 files saved to: tmp/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3 pre-signed url https://eu2.contabostorage.com/cdk-ttsopenai-gpt-prod-upload-bucket/contabo_uploaded/1288/20240318_145939_024737/5_converted.mp3?AWSAccessKeyId=24db761ab7758be8109fcc212513ea39&Signature=LCZsAeFgWkCoZ%2FNOAhckICaNr4E%3D&Expires=1711378914 Produced event to topic tts-emails-prod: value = {"function": "send_vocalized_file_failed_notice", "email_to": "nguyenhuy31ns@gmail.com", "recipient_name": "Huy Nguyen", "file_name": "5.docx"}