one question about the semaphore in model_worker.py

[xiguiw]

I run into semaphore not released issue.

From the code, the semaphore is acquired when generate_stream, and release_model_semaphore is added as a background task. But in my envrionment (Ubuntu 22.04) it is never called.
Besides, the global_counter is not restored when semaphore released. Not get what's the trigger condition to call it.

In my test, it's blocked by semaphore after submit --limit-model-concurrency times of query of a model.

I put the release semaphore at the end of streamer the generated the text. There the model end of text generate. It works.

Could you tell me how the semapore is released in the background task?
Is my change acceptable?

Thanks!

     @torch.inference_mode()
     def generate_stream(self, params):
+        global model_semaphore, global_counter
         tokenizer, model, image_processor = self.tokenizer, self.model, self.image_processor

         prompt = params["prompt"]
@@ -189,12 +191,25 @@ class ModelWorker:
         ))
         thread.start()

         generated_text = ori_prompt

         for new_text in streamer:
             generated_text += new_text
             if generated_text.endswith(stop_str):
                 generated_text = generated_text[:-len(stop_str)]
            yield json.dumps({"text": generated_text, "error_code": 0}).encode() + b"\0"

+        global_counter -= 1
+        model_semaphore.release()

 def release_model_semaphore(fn=None):
     model_semaphore.release()
     if fn is not None:
         fn()

@@ async def generate_stream(request: Request):
     global model_semaphore, global_counter
     global_counter += 1
     params = await request.json()

     if model_semaphore is None:
         model_semaphore = asyncio.Semaphore(args.limit_model_concurrency)

     await model_semaphore.acquire()
     worker.send_heart_beat()
     generator = worker.generate_stream_gate(params)

     background_tasks = BackgroundTasks()
     background_tasks.add_task(partial(release_model_semaphore, fn=worker.send_heart_beat))
    return StreamingResponse(generator, background=background_tasks)

Xigui