feat(backend): Track LLM token usage + LLM blocks cleanup

[majdyz]

Background

We are not tracking LLM's token usage.

Changes 🏗️

• Added LLM token usage tracking on AIStructuredResponseGeneratorBlock
• Make other LLM block utilizes AIStructuredResponseGeneratorBlock as the base LLM call.

Testing 🔍

Note

Only for the new autogpt platform, currently in autogpt_platform/

• Create from scratch and execute an agent with at least 3 blocks
• Import an agent from file upload, and confirm it executes correctly
• Upload agent to marketplace
• Import an agent from marketplace and confirm it executes correctly
• Edit an agent from monitor, and confirm it executes correctly

[codiumai-pr-agent-pro]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Performance Issue The AIStructuredResponseGeneratorBlock's llm_call method now returns token usage information, but this information is not being used effectively in all cases. For example, in the Ollama provider case, it returns placeholder values (0, 0) for input and output tokens. This could lead to inaccurate token usage tracking. Code Duplication The AITextGeneratorBlock, AISummarizerBlock, and AIConversationBlock all implement similar llm_call methods that create an AIStructuredResponseGeneratorBlock and call its run_once method. This duplication could be refactored into a shared method or base class. Potential Bug The merge_stats method in the Block class doesn't handle nested dictionaries correctly. It only updates the top-level keys, which could lead to data loss or unexpected behavior if stats contain nested structures.

Addressed