fix(gemini): support image conversion when content coerced to content…

[yt20chill]

… blocks by setting output_version = v1

Fixes # (issue)

Right now when LC_OUTPUT_VERSION=v1 is set, content will be mapped to ContentBlock[] (https://docs.langchain.com/oss/javascript/migrate/langchain-v1#standard-content)

However, in the current gemini.ts file, content type is still deprecated MessageContent, causing ContentBlock.Multimodal.Image cannot be converted to GeminiPart and throws unrecognized type error.

This fix only take care of the {type: "image", url: string} block because it's the only one I used in my project and I lack knowledge about all types. A thorough fix will require passing the standardized contentBlocks instead of content to those mapping functions.

I have tested the fix in my project by directly changing the gemini.js file in dependency and it convert the block correctly for gemini to see the attached image.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2c1bece

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[christian-bromann]

Thank you @yt20chill for the contribution. Can we have unit tests for this to ensure we don't regress?

[yt20chill]

@christian-bromann I am sorry that I am pretty new to this repo and I can't even run the unit tests using pnpm.

I got this error on both pnpm test:single and just pnpm test: > ReferenceError: You are trying to import a file after the Jest environment has been torn down.

It will be very helpful if you could point me to some solutions. Meanwhile, I have duplicate all the unit tests for testing image_url cases and change the type from image_url to image and url to string instead of object and they should pass. Sorry for the extra work.

[christian-bromann]

No worries at all. I think the problem here is that we haven't ported to Vitest for all Google packages as we are in the process of migrating them all into a single package in #9374, maybe worth double checking if this fix is correctly applied there too.

In terms of test execution you can run standalone package tests via:

pnpm --filter @langchain/google-common test

If I run this in current main I get:

Test Suites: 4 passed, 4 total
Tests:       200 passed, 200 total

Can you share some more information about the issue?

[yt20chill]

What I experienced in my current project is that after I migrate langchain to v1 and set the lc_output_version to v1. By setting the output version to v1, the message.content becomes the same type as contentBlocks which I like it more because it's standardized for all model providers, not just for openai.

But then for gemini model, I got a parser error saying "image is not supported".

When I look into the gemini part converting function, it is still using deprecated MessageContent and thus doesn't have the type: image case. So in this PR, I added a case to handle the standardize image block.

I know it's a workaround but before langchain migrates everything to parsing contentBlocks, I think it might still be helpful to patch this.

I hope I explain the issue better.