Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe your environment
OS: (e.g, Ubuntu) Linux
Python version: (e.g., Python 3.9.10)
Package version: (e.g., 0.46.0) latest
What happened?
Using latest version of fsspec[gcs] in ADK agent for prompts and responses upload failed with error:
/usr/local/lib/python3.13/site-packages/google/adk/cli/adk_web_server.py:475: UserWarning: [EXPERIMENTAL] App: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
agentic_app = App(
/usr/local/lib/python3.13/site-packages/opentelemetry/sdk/_logs/_internal/__init__.py:337: LogDeprecatedInitWarning: LogRecord init with `trace_id`, `span_id`, and/or `trace_flags` is deprecated since 1.35.0. Use `context` instead.
return cls(
INFO: 34.162.183.88:40558 - "GET /debug/trace/session/faa1ef56-6649-440a-b431-1bf5d2422b49 HTTP/1.1" 200 OK
INFO: 10.138.15.196:40559 - "GET /apps/capital_agent/users/user/sessions/faa1ef56-6649-440a-b431-1bf5d2422b49 HTTP/1.1" 200 OK
uploader failed
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/opentelemetry/util/genai/_upload/completion_hook.py", line 142, in done
future.result()
~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.13/site-packages/opentelemetry/util/genai/_upload/completion_hook.py", line 240, in _do_upload
with self._fs.open(path, "w", content_type=content_type) as file:
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1337, in open
self.open(
~~~~~~~~~^
path,
^^^^^
...<4 lines>...
**kwargs,
^^^^^^^^^
),
^
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1349, in open
f = self._open(
path,
...<4 lines>...
**kwargs,
)
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1256, in _open
return GCSFile(
self,
...<8 lines>...
**kwargs,
)
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1406, in __init__
det = getattr(self, "details", {}) # only exists in read mode
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1936, in details
self._details = self.fs.info(self.path)
~~~~~~~~~~~~^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 103, in sync
raise return_result
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 56, in _runner
result[0] = await coro
^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 834, in _info
out = await self._ls(path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 870, in _ls
out = await self._list_objects(path, prefix=prefix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 658, in _list_objects
return [await self._get_object(path)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 612, in _get_object
res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 539, in _call
self.validate_response(status, contents, path, headers)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1311, in validate_response
raise FileNotFoundError
FileNotFoundError
uploader failed
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/opentelemetry/util/genai/_upload/completion_hook.py", line 142, in done
future.result()
~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.13/site-packages/opentelemetry/util/genai/_upload/completion_hook.py", line 240, in _do_upload
with self._fs.open(path, "w", content_type=content_type) as file:
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1337, in open
self.open(
~~~~~~~~~^
path,
^^^^^
...<4 lines>...
**kwargs,
^^^^^^^^^
),
^
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1349, in open
f = self._open(
path,
...<4 lines>...
**kwargs,
)
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1256, in _open
return GCSFile(
self,
...<8 lines>...
**kwargs,
)
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1406, in __init__
det = getattr(self, "details", {}) # only exists in read mode
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1936, in details
self._details = self.fs.info(self.path)
~~~~~~~~~~~~^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 103, in sync
raise return_result
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 56, in _runner
result[0] = await coro
^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 834, in _info
out = await self._ls(path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 870, in _ls
out = await self._list_objects(path, prefix=prefix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 658, in _list_objects
return [await self._get_object(path)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 612, in _get_object
res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 539, in _call
self.validate_response(status, contents, path, headers)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1311, in validate_response
raise FileNotFoundError
FileNotFoundError
uploader failed
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/opentelemetry/util/genai/_upload/completion_hook.py", line 142, in done
future.result()
~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.13/site-packages/opentelemetry/util/genai/_upload/completion_hook.py", line 240, in _do_upload
with self._fs.open(path, "w", content_type=content_type) as file:
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1337, in open
self.open(
~~~~~~~~~^
path,
^^^^^
...<4 lines>...
**kwargs,
^^^^^^^^^
),
^
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1349, in open
f = self._open(
path,
...<4 lines>...
**kwargs,
)
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1256, in _open
return GCSFile(
self,
...<8 lines>...
**kwargs,
)
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1406, in __init__
det = getattr(self, "details", {}) # only exists in read mode
File "/usr/local/lib/python3.13/site-packages/fsspec/spec.py", line 1936, in details
self._details = self.fs.info(self.path)
~~~~~~~~~~~~^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 118, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 103, in sync
raise return_result
File "/usr/local/lib/python3.13/site-packages/fsspec/asyn.py", line 56, in _runner
result[0] = await coro
^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 834, in _info
out = await self._ls(path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 870, in _ls
out = await self._list_objects(path, prefix=prefix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 658, in _list_objects
return [await self._get_object(path)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 612, in _get_object
res = await self._call("GET", "b/{}/o/{}", bucket, key, json_out=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 539, in _call
self.validate_response(status, contents, path, headers)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/gcsfs/core.py", line 1311, in validate_response
raise FileNotFoundError
FileNotFoundError
Steps to Reproduce
Run adk with:
opentelemetry-instrumentation-google-genai==0.4b0
google-adk==1.16.0
fastapi==0.115.13
uvicorn==0.34.0
pydantic==2.11.7
fsspec[gcs]
Expected Result
No error
Actual Result
See error logs above
Additional context
No response
Would you like to implement a fix?
None
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Reacted by Jeff Luo
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Type
Fields
Priority
None yet
Activity
JeffLuoo commentedon Feb 3, 2026
@DylanRussell @aabmass
DylanRussell commentedon Feb 9, 2026
Hmm I was not able to reproduce this.. I tried running a vanilla ADK agent with all the instrumentation env vars and dependencies set as per the docs but with
fsspecandgcsfsto2026.2.0and the upload worked.aabmass commentedon May 5, 2026
Good to close folks?
JeffLuoo commentedon May 5, 2026
We can close it for now, I will re-open the ticket if I run into the issue again.
DylanRussell commentedon May 5, 2026
SG... I did make a change to verify the upload works at startup, and if it doesn't to fail, instead of repeatedly failing each upload: #4390 -- also it could be the env var wasn't prefixed with
gs://- that is required to upload to GCS.. I don't know if that could have caused this stacktrace or not..