The in-memory storage for tracking rate limits #29908
Comments
|
To resolve the warning about using in-memory storage for tracking rate limits in Superset during startup, you need to configure a persistent storage backend for rate limiting. This can be done by setting the Here is how you can set it up in your # A storage location conforming to the scheme in storage-scheme. See the limits
# library for allowed values: https://limits.readthedocs.io/en/stable/storage.html
RATELIMIT_STORAGE_URI = "redis://host:port"Replace Additionally, you can refer to the import os
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PORT = os.getenv("REDIS_PORT", "6379")
REDIS_CELERY_DB = os.getenv("REDIS_CELERY_DB", "0")
REDIS_RESULTS_DB = os.getenv("REDIS_RESULTS_DB", "1")
CACHE_CONFIG = {
"CACHE_TYPE": "RedisCache",
"CACHE_DEFAULT_TIMEOUT": 300,
"CACHE_KEY_PREFIX": "superset_",
"CACHE_REDIS_HOST": REDIS_HOST,
"CACHE_REDIS_PORT": REDIS_PORT,
"CACHE_REDIS_DB": REDIS_RESULTS_DB,
}
DATA_CACHE_CONFIG = CACHE_CONFIG
class CeleryConfig:
broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_CELERY_DB}"
result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_RESULTS_DB}"
CELERY_CONFIG = CeleryConfigEnsure that your Redis instance is running and accessible at the specified host and port. This will help you avoid the in-memory storage warning and improve the reliability of rate limiting in Superset [1][2]. |
|
Issue resolved and configured successfully. Awesome! |
|
++ @dosu could you help me with the warning also:
|
|
The warning "Class 'werkzeug.local.LocalProxy' is not mapped" is likely due to the use of # Instead of passing the proxy directly, access the underlying object
actual_results_backend = results_backend._get_current_object()
# Use actual_results_backend in your ORM operationsAdditionally, there has been a recent update in the Superset codebase that addresses this warning. The update is mentioned in the - [#27164](https://github.com/apache/superset/pull/27164) fix: unlock and bump werkzeug (@dpgaspar)This update includes changes that address the warning related to |
|
@dosu how to pass this line? where |
|
To pass the line
Here is the modified code: class ResultsBackendManager:
def __init__(self) -> None:
self._results_backend = None
self._use_msgpack = False
self._actual_results_backend = None
def init_app(self, app: Flask) -> None:
self._results_backend = app.config["RESULTS_BACKEND"]
self._use_msgpack = app.config["RESULTS_BACKEND_USE_MSGPACK"]
self._actual_results_backend = app.config.get("ACTUAL_RESULTS_BACKEND")
@property
def results_backend(self) -> Optional[BaseCache]:
return self._results_backend
@property
def should_use_msgpack(self) -> bool:
return self._use_msgpack
@property
def actual_results_backend(self) -> Optional[BaseCache]:
return self._actual_results_backend
Example usage in your ORM operations: from superset.extensions import results_backend_manager
def some_orm_operation():
backend = results_backend_manager.actual_results_backend
if backend:
# Perform operations using actual_results_backend
passIn the context of class SqlExecutionResultsCommand(BaseCommand):
_key: str
_rows: int | None
_blob: Any
_query: Query
def __init__(
self,
key: str,
rows: int | None = None,
) -> None:
self._key = key
self._rows = rows
def validate(self) -> None:
if not results_backend_manager.actual_results_backend:
raise SupersetErrorException(
SupersetError(
message=__("Actual results backend is not configured."),
error_type=SupersetErrorType.RESULTS_BACKEND_NOT_CONFIGURED_ERROR,
level=ErrorLevel.ERROR,
)
)
read_from_results_backend_start = now_as_float()
self._blob = results_backend_manager.actual_results_backend.get(self._key)
stats_logger.timing(
"sqllab.query.results_backend_read",
now_as_float() - read_from_results_backend_start,
)
if not self._blob:
raise SupersetErrorException(
SupersetError(
message=__(
"Data could not be retrieved from the actual results backend. You "
"need to re-run the original query."
),
error_type=SupersetErrorType.RESULTS_BACKEND_ERROR,
level=ErrorLevel.ERROR,
),
status=410,
)
self._query = (
db.session.query(Query).filter_by(results_key=self._key).one_or_none()
)
if self._query is None:
raise SupersetErrorException(
SupersetError(
message=__(
"The query associated with these results could not be found. "
"You need to re-run the original query."
),
error_type=SupersetErrorType.RESULTS_BACKEND_ERROR,
level=ErrorLevel.ERROR,
),
status=404,
)
def run(self) -> dict[str, Any]:
"""Runs arbitrary sql and returns data as json"""
self.validate()
payload = utils.zlib_decompress(
self._blob, decode=not results_backend_use_msgpack
)
try:
obj = _deserialize_results_payload(
payload, self._query, cast(bool, results_backend_use_msgpack)
)
except SerializationError as ex:
raise SupersetErrorException(
SupersetError(
message=__(
"Data could not be deserialized from the actual results backend. The "
"storage format might have changed, rendering the old data "
"stake. You need to re-run the original query."
),
error_type=SupersetErrorType.RESULTS_BACKEND_ERROR,
level=ErrorLevel.ERROR,
),
status=404,
) from ex
if self._rows:
obj = apply_display_max_row_configuration_if_require(obj, self._rows)
return objBy following these steps, you can integrate |
|
@dosu find the below error: |
|
To resolve the "NameError: name 'results_backend' is not defined" error in your Here is how you can define and initialize from flask_caching.backends.filesystemcache import FileSystemCache
# Initialize the results_backend
RESULTS_BACKEND = FileSystemCache("/app/superset_home/sqllab")
# Access the actual object behind the proxy
actual_results_backend = RESULTS_BACKEND._get_current_object()This ensures that Additionally, ensure you have the necessary dependencies installed for the caching backend you are using. For pip install flask-cachingThis should resolve the |
|
@dosu find this: |
|
The Here’s how you can modify your
from flask_caching.backends.filesystemcache import FileSystemCache
# Initialize the results_backend
RESULTS_BACKEND = FileSystemCache("/app/superset_home/sqllab")
# Directly use RESULTS_BACKEND without _get_current_object
actual_results_backend = RESULTS_BACKEND
This should resolve the |
|
@dosu but it doesn't solve the warning:
|
|
To address the warning Here are steps to resolve this:
Here is an example of how you can handle from flask import Flask
from werkzeug.local import LocalProxy
from superset.extensions import cache_manager, results_backend_manager
def create_app():
app = Flask(__name__)
# Initialize your app and extensions here
# ...
# Explicitly map or handle LocalProxy objects
app.config['CACHE'] = cache_manager.cache
app.config['CONF'] = LocalProxy(lambda: app.config)
app.config['RESULTS_BACKEND'] = LocalProxy(lambda: results_backend_manager.results_backend)
app.config['RESULTS_BACKEND_USE_MSGPACK'] = LocalProxy(lambda: results_backend_manager.should_use_msgpack)
app.config['DATA_CACHE'] = LocalProxy(lambda: cache_manager.data_cache)
app.config['THUMBNAIL_CACHE'] = LocalProxy(lambda: cache_manager.thumbnail_cache)
app.config['ASYNC_QUERY_MANAGER'] = LocalProxy(lambda: async_query_manager_factory.instance)
app.config['EVENT_LOGGER'] = LocalProxy(lambda: _event_logger.get("event_logger"))
app.config['SECURITY_MANAGER'] = LocalProxy(lambda: appbuilder.sm)
return appBy explicitly mapping these proxies in your application configuration, you can help ensure that they are properly recognized and handled, which should address the warning [1]. |
|
@dosu still the below warning exists, how to skip it.
|
|
To skip the warning
import warnings
# Suppress specific warning
warnings.filterwarnings("ignore", message="Class 'werkzeug.local.LocalProxy' is not mapped")
# superset/__init__.py or your main script
import warnings
# Suppress specific warning
warnings.filterwarnings("ignore", message="Class 'werkzeug.local.LocalProxy' is not mapped")This will specifically ignore warnings that match the given message and should prevent the warning from appearing in your logs [1]. |
Bug description
The below warnings appears in logs when startup:
How to reproduce the bug
Fresh setup.
Screenshots/recordings
No response
Superset version
4.0.2
Python version
3.11
Node version
I don't know
Browser
Not applicable
Additional context
No response
Checklist
The text was updated successfully, but these errors were encountered: