Worker Indexing Changes

The goal of Project Stein is to shift the functions indexing responsibility from the host to the worker. Some changes were made to the host to enable this new role, so this wiki page will go over the details of what changes were made and why they were necessary.

Current host flow without worker indexing:

At ScriptHost startup, the host requests for HostFunctionMetadataProvider to index the functions.

---

New host flow with worker indexing:

At ScriptHost startup, the host requests for the worker to index the functions and send back the function metadata through WorkerFunctionMetadataProvider.

---

Changes that were made

• New metadata provider: WorkerFunctionMetadataProvider communicates with the worker to get the function metadata. It also validates the received metadata before sending it back to FunctionMetadataManager.

• FunctionMetadataManager.LoadFunctionMetadata: now checks to see if worker can index or not. Based on this result, either a new WorkerFunctionMetadatProvider is created or the DI HostFunctionMetadataProvider is used.

• New GRPC message types: FunctionsMetadataRequest, FunctionMetadataResponses

  FunctionsMetadataRequest sends the function app directory to the worker.

  FunctionMetadataResponses includes a list of FunctionLoadRequest messages and an overall StatusResult attribute. Each of these FunctionLoadRequest messages contains the function ID and corresponding RpcFunctionMetadata message.

• Added new initialization code path to RpcFunctionInvocationDispatcher: new functions break up the original InitializeAsync logic to allow the host to send the WorkerMetadataRequest in the middle of the channel initialization process. The logic also needed to be broken up because the invocation buffers and invocation requests can only be made once the host has metadata to operate with.

• Added feature flag, FeatureFlagEnableWorkerIndexing, in front of new worker indexing code path

• Added utility function Utility.CanWorkerIndex(RpcWorkerConfig workerConfig, IEnvironment environment) to check to see if the worker can index or not. It returns true if the feature flag is enabled and the worker.config.json file states that the worker can index. Otherwise, it returns false.

---

How to Set Up Local Worker Indexing Environment (feat. Python Worker)

1. Pull the latest version of dev locally.
2. Then, set the AzureWebJobsFeatureFlags environment variable to have this value: EnableWorkerIndexing.
3. To set up the worker, follow these instructions.
4. Set a "workerIndexing" : "true" property within the Python worker's worker.config.json file.