I do this all the time. Post the results for each row to a multiprocessing.Queue, and spawn a single process that gets from the queue and writes to the file. It'll post some code when I get to work.

Code a Llama 2 API with multiprocessing support using Python 7:39 am August 15, 2023 By Julian Horsey Meta has made a significant contribution with the release of the Llama 2 Large Language Model ...