Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLI] Windows testing #2495

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 60 additions & 2 deletions .github/workflows/_integration_test.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: CLI integration test

Check notice on line 1 in .github/workflows/_integration_test.yml

View workflow job for this annotation

GitHub Actions / benchmark

Benchmark results

......................................... fanout_to_subgraph_10x: Mean +- std dev: 62.2 ms +- 1.9 ms ......................................... fanout_to_subgraph_10x_sync: Mean +- std dev: 53.6 ms +- 1.9 ms ......................................... WARNING: the benchmark result may be unstable * the standard deviation (11.2 ms) is 12% of the mean (96.8 ms) Try to rerun the benchmark with more runs, values and/or loops. Run 'python -m pyperf system tune' command to reduce the system jitter. Use pyperf stats, pyperf dump and pyperf hist to analyze results. Use --quiet option to hide these warnings. fanout_to_subgraph_10x_checkpoint: Mean +- std dev: 96.8 ms +- 11.2 ms ......................................... fanout_to_subgraph_10x_checkpoint_sync: Mean +- std dev: 98.7 ms +- 3.8 ms ......................................... fanout_to_subgraph_100x: Mean +- std dev: 644 ms +- 34 ms ......................................... fanout_to_subgraph_100x_sync: Mean +- std dev: 525 ms +- 14 ms ......................................... fanout_to_subgraph_100x_checkpoint: Mean +- std dev: 1.02 sec +- 0.05 sec ......................................... fanout_to_subgraph_100x_checkpoint_sync: Mean +- std dev: 964 ms +- 28 ms ......................................... react_agent_10x: Mean +- std dev: 31.3 ms +- 0.6 ms ......................................... react_agent_10x_sync: Mean +- std dev: 22.9 ms +- 0.5 ms ......................................... react_agent_10x_checkpoint: Mean +- std dev: 48.0 ms +- 1.0 ms ......................................... react_agent_10x_checkpoint_sync: Mean +- std dev: 37.7 ms +- 1.0 ms ......................................... react_agent_100x: Mean +- std dev: 351 ms +- 7 ms ......................................... react_agent_100x_sync: Mean +- std dev: 275 ms +- 5 ms ......................................... react_agent_100x_checkpoint: Mean +- std dev: 978 ms +- 17 ms ......................................... react_agent_100x_checkpoint_sync: Mean +- std dev: 873 ms +- 15 ms ......................................... wide_state_25x300: Mean +- std dev: 24.7 ms +- 0.5 ms ......................................... wide_state_25x300_sync: Mean +- std dev: 15.7 ms +- 0.2 ms ......................................... wide_state_25x300_checkpoint: Mean +- std dev: 279 ms +- 3 ms ......................................... wide_state_25x300_checkpoint_sync: Mean +- std dev: 268 ms +- 4 ms ......................................... wide_state_15x600: Mean +- std dev: 29.0 ms +- 0.6 ms ......................................... wide_state_15x600_sync: Mean +- std dev: 18.2 ms +- 0.3 ms ......................................... wide_state_15x600_checkpoint: Mean +- std dev: 483 ms +- 8 ms ......................................... wide_state_15x600_checkpoint_sync: Mean +- std dev: 463 ms +- 5 ms ......................................... wide_state_9x1200: Mean +- std dev: 28.7 ms +- 0.6 ms ......................................... wide_state_9x1200_sync: Mean +- std dev: 18.1 ms +- 0.2 ms ......................................... wide_state_9x1200_checkpoint: Mean +- std dev: 314 ms +- 3 ms ......................................... wide_state_9x1200_checkpoint_sync: Mean +- std dev: 300 ms +- 7 ms

Check notice on line 1 in .github/workflows/_integration_test.yml

View workflow job for this annotation

GitHub Actions / benchmark

Comparison against main

+-----------------------------------------+---------+-----------------------+ | Benchmark | main | changes | +=========================================+=========+=======================+ | fanout_to_subgraph_100x | 659 ms | 644 ms: 1.02x faster | +-----------------------------------------+---------+-----------------------+ | wide_state_9x1200_checkpoint | 316 ms | 314 ms: 1.01x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_100x_checkpoint_sync | 876 ms | 873 ms: 1.00x faster | +-----------------------------------------+---------+-----------------------+ | wide_state_15x600_checkpoint_sync | 465 ms | 463 ms: 1.00x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_10x_sync | 22.7 ms | 22.9 ms: 1.01x slower | +-----------------------------------------+---------+-----------------------+ | wide_state_25x300_checkpoint_sync | 266 ms | 268 ms: 1.01x slower | +-----------------------------------------+---------+-----------------------+ | react_agent_10x | 31.1 ms | 31.3 ms: 1.01x slower | +-----------------------------------------+---------+-----------------------+ | wide_state_15x600_sync | 18.0 ms | 18.2 ms: 1.01x slower | +-----------------------------------------+---------+-----------------------+ | wide_state_9x1200_sync | 18.0 ms | 18.1 ms: 1.01x slower | +-----------------------------------------+---------+-----------------------+ | react_agent_100x_checkpoint | 966 ms | 978 ms: 1.01x slower | +-----------------------------------------+---------+-----------------------+ | react_agent_10x_checkpoint_sync | 37.0 ms | 37.7 ms: 1.02x slower | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_100x_checkpoint_sync | 944 ms | 964 ms: 1.02x slower | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_10x_sync | 52.4 ms | 53.6 ms: 1.02x slower | +-----------------------------------------+---------+-----------------------+ | wide_state_15x600 | 28.3 ms | 29.0 ms: 1.02x slower | +-----------------------------------------+---------+-----------------------+ | react_agent_100x | 343 ms | 351 ms: 1.02x slower | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_10x_checkpoint_sync | 95.3 ms | 98.7 ms: 1.04x slower | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_100x_sync | 506 ms | 525 ms: 1.04x slower | +-----------------------------------------+---------+-----------------------+ | Geometric mean | (ref) | 1.01x slower | +-----------------------------------------+---------+-----------------------+ Benchmark hidden because not significant (11): fanout_to_subgraph_100x_checkpoint, fanout_to_subgraph_10x, wide_state_25x300_sync, wide_state_9x1200, wide_state_25x300_checkpoint, wide_state_15x600_checkpoint, react_agent_10x_checkpoint, wide_state_9x1200_checkpoint_sync, wide_state_25x300, react_agent_100x_sync, fanout_to_subgraph_10x_checkpoint

on:
workflow_call:
Expand All @@ -14,7 +14,7 @@
python-version:
- "3.10"
- "3.11"
name: "CLI integration test"
name: "CLI integration test (Linux)"
defaults:
run:
working-directory: libs/cli
Expand Down Expand Up @@ -71,4 +71,62 @@
working-directory: libs/cli/js-examples
run: |
langgraph build -t langgraph-test-e


windows-build:
runs-on: windows-latest
name: "CLI integration test (Windows)"
defaults:
run:
working-directory: libs/cli
shell: bash
steps:
- uses: actions/checkout@v4
- name: Get changed files
id: changed-files
uses: Ana06/[email protected]
with:
filter: "libs/cli/**"
- name: Set up Python 3.11 + Poetry ${{ env.POETRY_VERSION }}
if: steps.changed-files.outputs.all
uses: "./.github/actions/poetry_setup"
with:
python-version: "3.11"
poetry-version: ${{ env.POETRY_VERSION }}
cache-key: integration-test-cli-windows
- name: Setup env
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples
run: cp .env.example .env
- name: Install cli globally
if: steps.changed-files.outputs.all
run: pip install -e .
- name: Build and test service A
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples
run: |
langgraph build -t langgraph-test-a --base-image "langchain/langgraph-trial"
cp .env.example .envg
timeout 60 python ../../../.github/scripts/run_langgraph_cli_test.py -c langgraph.json -t langgraph-test-a
- name: Build and test service B
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs
run: |
langgraph build -t langgraph-test-b --base-image "langchain/langgraph-trial"
timeout 60 python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-b
- name: Build and test service C
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs_reqs_a
run: |
langgraph build -t langgraph-test-c --base-image "langchain/langgraph-trial"
timeout 60 python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-c
- name: Build and test service D
if: steps.changed-files.outputs.all
working-directory: libs/cli/examples/graphs_reqs_b
run: |
langgraph build -t langgraph-test-d --base-image "langchain/langgraph-trial"
timeout 60 python ../../../../.github/scripts/run_langgraph_cli_test.py -t langgraph-test-d
- name: Build JS service
if: steps.changed-files.outputs.all
working-directory: libs/cli/js-examples
run: |
langgraph build -t langgraph-test-e
154 changes: 82 additions & 72 deletions libs/cli/langgraph_cli/config.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import json
import os
import pathlib
import textwrap
from typing import NamedTuple, Optional, TypedDict, Union
Expand Down Expand Up @@ -192,66 +191,76 @@ def check_reserved(name: str, ref: str):
resolved = config_path.parent / local_dep

# validate local dependency
if not resolved.exists():
raise FileNotFoundError(f"Could not find local dependency: {resolved}")
elif not resolved.is_dir():
raise NotADirectoryError(
f"Local dependency must be a directory: {resolved}"
)
elif not resolved.is_relative_to(config_path.parent):
raise ValueError(
f"Local dependency '{resolved}' must be a subdirectory of '{config_path.parent}'"
)
try:
if not resolved.exists():
raise FileNotFoundError(f"Could not find local dependency: {resolved}")
elif not resolved.is_dir():
raise NotADirectoryError(
f"Local dependency must be a directory: {resolved}"
)
elif not resolved.is_relative_to(config_path.parent):
raise ValueError(
f"Local dependency '{resolved}' must be a subdirectory of '{config_path.parent}'"
)

# if it's installable, add it to local_pkgs
# otherwise, add it to faux_pkgs, and create a pyproject.toml
files = os.listdir(resolved)
if "pyproject.toml" in files:
real_pkgs[resolved] = local_dep
if local_dep == ".":
working_dir = f"/deps/{resolved.name}"
elif "setup.py" in files:
real_pkgs[resolved] = local_dep
if local_dep == ".":
working_dir = f"/deps/{resolved.name}"
else:
if any(file == "__init__.py" for file in files):
# flat layout
if "-" in resolved.name:
raise ValueError(
f"Package name '{resolved.name}' contains a hyphen. "
"Rename the directory to use it as flat-layout package."
)
check_reserved(resolved.name, local_dep)
container_path = f"/deps/__outer_{resolved.name}/{resolved.name}"
# if it's installable, add it to local_pkgs
# otherwise, add it to faux_pkgs, and create a pyproject.toml
try:
files = list(resolved.iterdir())
file_names = [f.name for f in files]
except (PermissionError, OSError) as e:
raise click.UsageError(
f"Cannot access directory {resolved}: {str(e)}"
) from None

if "pyproject.toml" in file_names:
real_pkgs[resolved] = local_dep
if local_dep == ".":
working_dir = f"/deps/{resolved.name}"
elif "setup.py" in file_names:
real_pkgs[resolved] = local_dep
if local_dep == ".":
working_dir = f"/deps/{resolved.name}"
else:
# src layout
container_path = f"/deps/__outer_{resolved.name}/src"
for file in files:
rfile = resolved / file
if (
rfile.is_dir()
and file != "__pycache__"
and not file.startswith(".")
):
try:
for subfile in os.listdir(rfile):
if subfile.endswith(".py"):
check_reserved(file, local_dep)
break
except PermissionError:
pass
faux_pkgs[resolved] = (local_dep, container_path)
if local_dep == ".":
working_dir = container_path
if "requirements.txt" in files:
rfile = resolved / "requirements.txt"
pip_reqs.append(
(
rfile.relative_to(config_path.parent),
f"{container_path}/requirements.txt",
if any(file == "__init__.py" for file in file_names):
# flat layout
if "-" in resolved.name:
raise ValueError(
f"Package name '{resolved.name}' contains a hyphen. "
"Rename the directory to use it as flat-layout package."
)
check_reserved(resolved.name, local_dep)
container_path = f"/deps/__outer_{resolved.name}/{resolved.name}"
else:
# src layout
container_path = f"/deps/__outer_{resolved.name}/src"
for file in files:
if (
file.is_dir()
and file.name != "__pycache__"
and not file.name.startswith(".")
):
try:
subfiles = list(file.iterdir())
if any(f.name.endswith(".py") for f in subfiles):
check_reserved(file.name, local_dep)
except (PermissionError, OSError):
continue
faux_pkgs[resolved] = (local_dep, container_path)
if local_dep == ".":
working_dir = container_path
if "requirements.txt" in file_names:
rfile = resolved / "requirements.txt"
pip_reqs.append(
(
pathlib.PurePosixPath(
rfile.relative_to(config_path.parent)
),
f"{container_path}/requirements.txt",
)
)
)
except (PermissionError, OSError) as e:
raise click.UsageError(f"Cannot access path {resolved}: {str(e)}") from e

return LocalDeps(pip_reqs, real_pkgs, faux_pkgs, working_dir)

Expand All @@ -275,12 +284,12 @@ def _update_graph_paths(
else:
for path in local_deps.real_pkgs:
if resolved.is_relative_to(path):
module_str = f"/deps/{path.name}/{resolved.relative_to(path)}"
module_str = f"/deps/{path.name}/{pathlib.PurePosixPath(resolved.relative_to(path))}"
break
else:
for faux_pkg, (_, destpath) in local_deps.faux_pkgs.items():
if resolved.is_relative_to(faux_pkg):
module_str = f"{destpath}/{resolved.relative_to(faux_pkg)}"
module_str = f"{destpath}/{pathlib.PurePosixPath(resolved.relative_to(faux_pkg))}"
break
else:
raise ValueError(
Expand Down Expand Up @@ -313,17 +322,17 @@ def python_config_to_docker(config_path: pathlib.Path, config: Config, base_imag

pip_pkgs_str = f"RUN {pip_install} {' '.join(pypi_deps)}" if pypi_deps else ""
if local_deps.pip_reqs:
pip_reqs_str = os.linesep.join(
pip_reqs_str = "\n".join(
f"ADD {reqpath} {destpath}" for reqpath, destpath in local_deps.pip_reqs
)
pip_reqs_str += f'{os.linesep}RUN {pip_install} {" ".join("-r " + r for _,r in local_deps.pip_reqs)}'

pip_reqs_str += (
f'\nRUN {pip_install} {" ".join("-r " + r for _,r in local_deps.pip_reqs)}'
)
else:
pip_reqs_str = ""

# https://setuptools.pypa.io/en/latest/userguide/datafiles.html#package-data
# https://til.simonwillison.net/python/pyproject
faux_pkgs_str = f"{os.linesep}{os.linesep}".join(
faux_pkgs_str = "\n\n".join(
f"""ADD {relpath} {destpath}
RUN set -ex && \\
for line in '[project]' \\
Expand All @@ -335,12 +344,12 @@ def python_config_to_docker(config_path: pathlib.Path, config: Config, base_imag
done"""
for fullpath, (relpath, destpath) in local_deps.faux_pkgs.items()
)
local_pkgs_str = os.linesep.join(
local_pkgs_str = "\n".join(
f"ADD {relpath} /deps/{fullpath.name}"
for fullpath, relpath in local_deps.real_pkgs.items()
)

installs = f"{os.linesep}{os.linesep}".join(
installs = "\n\n".join(
filter(
None,
[
Expand All @@ -352,18 +361,19 @@ def python_config_to_docker(config_path: pathlib.Path, config: Config, base_imag
],
)
)

_workdir = f"WORKDIR {local_deps.working_dir}" if local_deps.working_dir else ""
dockerfile_lines = "\n".join(config["dockerfile_lines"])
return f"""FROM {base_image}:{config['python_version']}

{os.linesep.join(config["dockerfile_lines"])}
{dockerfile_lines}

{installs}

RUN {pip_install} -e /deps/*

ENV LANGSERVE_GRAPHS='{json.dumps(config["graphs"])}'

{f"WORKDIR {local_deps.working_dir}" if local_deps.working_dir else ""}"""
{_workdir}"""


def node_config_to_docker(config_path: pathlib.Path, config: Config, base_image: str):
Expand All @@ -390,10 +400,10 @@ def test_file(file_name):
install_cmd = "npm ci"
else:
install_cmd = "npm i"

dockerfile_lines = "\n".join(config["dockerfile_lines"])
return f"""FROM {base_image}:{config['node_version']}

{os.linesep.join(config["dockerfile_lines"])}
{dockerfile_lines}

ADD . {faux_path}

Expand Down
Loading